Present: Jens (chair+mins), RobC, Sam, David, Dan, Govind

Regarding the pre-GDB, as we discussed earlier, many topics are T1-ish or even RALish: object stores, tape archives.  It would be useful to understand how experiments use
object stores: are they (the object stores) there because the experiments have a need for them or because certain sites thought they were cool?  As Sam points out,
if you write objects to a object-store emulator designed to write to a file system which is actually a file system emulator on an actual object store, this
can be potentially inefficient.

There are of course also object stores in public clouds, which Andrew Lahiff and Alastair Dewhurst looked at.  Sam points out there was a DPM-in-the-cloud which
worked but didn't get stellar performance.

Erasure codes can be useful if used right; but there are all sorts of things to worry about.  It could make recalls faster, even from tape, if you have enough drives.
One needs to worry about the time to encode and decode the file; the time to rebuild missing pieces (analogously to RAID), access patterns, etc.

Sam points out that Brian Bockelman did soemthing similar with xroot: using xroot's redirection facility, it recovers parts of files from different replicas,
thus making use of parallel recall, which could be useful for GridFTP parallel transfers.  Sam had been wondering about using erasure codes for parallel transfers;
Jens had previously had a discussion with David Salmon who had suggested compressing before transfers (with GridFTP) which also suggests other types of preprocessing.
ZFS has compression, now faster!, which could also be useful, maybe also if your disk is slow and the file can be uncompressed in memory; except perhaps when one needs 
a non-initial segment of a large file.  Sam experimented with compressing ROOT files but only got ~5%; Jens had previously looked at the compressions on CASTOR
where if memory serves a compression ratio was 1.1 (but that may have included things other than the ROOT file itself.)

If you compress the file first and then erasure code, at least you will get a fixed, known entropy (which is analogous to encryption of files which will also often
compress the file first.)

On a not entirely unrelated note, it has been suggested GridPP storage get a slot at GridPP39 and ZFS may be a useful topic.  It was agreed ZFS was a good topic
but it's our slot and we can do with it what we want... ZFS could also be a hepsysman topic but that'd probably be a more detailed hands on talk, e.g. tuning
and stuff.  The GridPP talk should be more high level looking at current experiences and future directions - and future potential.


Daniel remarks (see chat) that we have a new request from CERN@School about storing lots of small files, but if they are being accessed as well, and not just
archived/backed up like DiRAC, some more thought is needed.  Suggested to get a representative from CERN@School into one of these Wednesday calls to talk to us,
or do it by email, or meet at GridPP.  We are happy to work with them but experience has shown it requires some effort on their end as well, and as all VOs
are different (also shown by experiences...), it is necessary to have a bit of discussion.

Also makes one wonder why the transfer of lots of small files is inefficient - of course there would always be some inefficiencies but there are facilities in
the protocols - and often also in the software - that should help speed things up.  Like the Apache KeepAlive, GridFTP has a "pipeline", SRM has a multiple-files
feature (in fact for the latter it's the default, a single file request is a multifile request for one file; but using the multifile feature alone would not
solve the problem as it would just return multiple TURLs.)

How are C@S doing their transfers - with FTS, with GFAL, with GFAL2?  (Probably) GFAL2.  Which also does not update any LFC.


Finally, there is Winnie's peculiar BDII problem, first reported 23/08 which is a week ago.  Could it be because they are SRM-less, or maybe there is something
wrong with the validation (seems unlikely if *all* information is missing, but possible).




Jens Jensen: (30/08/2017 10:01:25)
https://indico.cern.ch/event/578974/
Daniel Peter Traynor: (10:19 AM)
does anyone know how to trasfer 1 million files, each afew kB in size efficienlty with gfal?
each file takes 3-5 seconds to trasfer in serial
Samuel Cadellin Skipsey: (10:20 AM)
Dan: well, I would tar them up
Daniel Peter Traynor: (10:20 AM)
how do you untar at the othre end?
Samuel Cadellin Skipsey: (10:20 AM)
(This was the solution used for, for example, DiRAC )
Okay, so what's the actual use case?
Daniel Peter Traynor: (10:20 AM)
this is a cern@schooll request
Samuel Cadellin Skipsey: (10:21 AM)
Essentially, transferring tiny files is inefficient, as you've found :)
If you can merge them together, it's the easiest performance win you can make
Daniel Peter Traynor: (10:29 AM)
ok
I'll follow up