Minutes of the storage EVO meeting 27 Apr 2011 Present: Glasgow: Sam, David Sheffield: Elena Lancaster: Matt Liverpool: Stephen QMUL: Chris Oxford: Ewan RHUL: Govind RAL T1: James, Brian, Jens (chair+mins) Apologies: RAL T1: Brian Edinburgh: Wahid 1. James talked about his recent Hadoop install, eight nodes with 1.6 TB capacity, currently with 350 GB of data. Running a sort MapReduce program. Files are sliced into 128 MB slices (default.) 2. Upgrade issues. The main problem with DPM arises from the renaming of tables, as Sam has been describing on the list, causing the schema upgrade to fail. In addition, the new request pruning feature (which deletes ancient requests) appears to work, but as it runs in the background it may take a while for it to finish processing. At Glasgow, with quite an old database, it took over a week for the first pass. InnoDB has quite a large db on disk, never shrinking the physical file, unless you reimport (dump and restore), or optimise a locked database which obviously is not possible. As for StoRM, Chris is a happier bunny now (maybe because it's Easter and Easter always makes bunnies happy) with 1.6.1/2. To measure the used space however, Chris du'ed his filesystem which took hours the first time, and later only 10 mins. 3. Safety Deposit Box Jens mentioned RAL have a pilot project for SDB, a commercial product from Tessella aiming at archiving scientific data. Based on OAIS (Open Archival Information System, see e.g. http://public.ccsds.org/publications/archive/650x0b1.PDF), its aim is to archive data rather than to run a repository like we usually do. It can however sit in front of other datastores like DMF, so its main aim is to track the ingest metadata and the archive metadata. While this kind of stuff is not directly relevant to us, it is useful sometimes to look for interoperation, and occasionally we can also learn something. We have seen before how others sometimes build "data grids" from scratch with little input from people who know how to do it in practice :-) 4. Small VOs roundup T2K - are they happy? Unknown. Elena got in touch with Jon Perkins - they don't have much effort which of course is one of the quintessential features of small VOs, and also the reason why we need to be proactive - we don't want them to be stuck for lack of handholding. We can't hold everybody's hand all the time, but we can maybe hold everybody's hand some of the time. (Note: everybody has a hand each, it's not a common hand shared by everyone.) Small VOs, as Sam wrote in the support writeup, tend to use the vanilla grid so stuff that has decayed hits them a lot harder than the large ones who run their own top layers. Sam reports that integrity checking is still a problem for T2K (FTS not registering files, and files in catalogue not being in SE). Similarly, what are NeISS doing? Sam reports that NeISS is back from paternity leave and has had some sleep (in the office?), and will be poked again. Jens will see if he can find out who is currently the honcho in pheno and see if they are happi(er). Sam says he last talked to Peter Richardson. 5. AOB NOB NEW ACTIONS 431 27/04/2011 Contact NeISS Sam Med Open 432 27/04/2011 Find out who is currently pheno and get in touch Jens Med Open 433 27/04/2011 Contact T2K and see if talking to us a month ago helped Jens Med Open [09:59:03] David Crooks joined [09:59:05] Sam Skipsey joined [09:59:24] Stephen Jones joined [09:59:47] Jens Jensen Apologies for the double booking... [10:01:24] Elena Korolkova joined [10:03:40] Ewan Mac Mahon joined [10:03:40] Ewan Mac Mahon left [10:05:07] Govind Songara joined [10:05:59] Matthew Doidge joined [10:06:23] Ewan Mac Mahon Simon at Bristol is quite keen on Hadoop/Bestman too. [10:06:41] Ewan Mac Mahon He seems to think that doing the authentication with argus might help. [10:09:45] Christopher Walker joined [10:17:45] John Bland joined [10:34:35] John Bland left [10:34:37] Elena Korolkova left [10:34:38] Ewan Mac Mahon left [10:34:42] David Crooks left [10:34:43] James Adams left [10:34:44] Matthew Doidge left [10:34:45] Govind Songara left [10:34:47] Sam Skipsey left [10:34:49] Stephen Jones left [10:34:54] Christopher Walker left