Hi Jens, I don't know how to upload stuff to the Storage Group agenda and minutes page, so I'm just sending some stuff your way. I've also attached Alastair's talk if you wanted to stick it on the page. - General Discussion/Operational Issues Brian/Matt testing gfal2 releases in the tarball and cvmfs worker node images against FTS. Need to test earlier releases of the tarball version. StoRM baseline release is now 1.11.8 Discussion on migration from old SL5 disk servers. Why are people slow to do this? [Ewan and Sam both noted that they were mostly slow to upgrade their SL5 disk server nodes because it would be tricky (esp now with new Puppet DPM), not because we're tied to it. Testing Centos7 for DPM was discussed.] - CEPH talk Alastair gave his ceph talk. Most questions were handled by George V in the chat (attached). Post talk, Brian asked: 1) has the file open rate been tested for the access plugins (cf Castor rate issues)? A) it hasn't, yet. This is on our list of things to test, but bandwidth tests were first. 2) IPv6 or IPv4 for internal ceph network? A) IPv4 internally - hope is to make the gateway nodes dual stack so they talk to internal IPv4 network and external RAL IPv6. Chat log attached: Bruno @ RAL: (10:13 AM) The physical monitors are Dell R420s, not R430s :-) Yes, SSDs for physical monitors will be an after market upgrade. Ewan Mac Mahon: (10:16 AM) Call me Mr Obvious, but if you've got one set of machines with SSDs you're not using, and one set of machines that don't have SSDs but should do, why not just shift them from one to the other? Daniel Traynor: (10:18 AM) why are you using raid with ceph? I think you made a case in the past Terry Froy: (10:20 AM) Using RAID with Ceph and Erasure Coding means that you lose more raw storage than you need to. Samuel Cadellin Skipsey: (10:21 AM) Unless you're using RAID0 to reduce the number of OSDs needed, of course. George V @ RAL: (10:21 AM) @Daniel: the idea behind using (possibly) using RAIDed drives with Ceph is to bring the number of OSDs down, as they consume CPU and RAM resources on a per-OSD basis rather than drive or capacity. E.g. you need about 2 GB RAM for each of the OSD daemons. A 36-drive machine would need around 80 GB of RAM but might only have 64. @Terry: We intend to try it with linux software RAID in striping mode Samuel Cadellin Skipsey: (10:22 AM) I would honestly try it with RAID0 pairs or triples. 36 drives down to 12 OSD volumes. George V @ RAL: (10:22 AM) @Samuel that's pretty much what we're thinking of Samuel Cadellin Skipsey: (10:22 AM) (and no lost storage) Yeah, it was my main complaint about the CERN test (that they never considered consolidation with raid 0) Bruno @ RAL: (10:24 AM) Yes, I can hear. Terry Froy: (10:24 AM) @George That won't get you anything though - from the Hardware Recommendations on ceph.com - "Also, the larger the storage drive capacity, the more memory per Ceph OSD Daemon you will need, especially during rebalancing, backfilling and recovery. A general rule of thumb is ~1GB of RAM for 1TB of storage space". Samuel Cadellin Skipsey: (10:25 AM) Pragmatically, however, it isn't quite that bad. Ewan Mac Mahon: (10:26 AM) Which is just as well when you start looking at (e.g.) 36 bays x 10TB drives. A third of a terabyte of RAM is a bit much for a disk server. Bruno @ RAL: (10:27 AM) We have a 2 disk RAID 1 for the OS and many 1 disk RAID 0 (fake JBOD) for the Data drives. This setup is a first draft. We may consolidate the Data disks into a few disks per RAID 0 later. George V @ RAL: (10:27 AM) @Terry I'm not entirely sure about the accuracy of those suggestions. We're seeing around 1.5GB used by each 2.7TB OSD on our small cluster Samuel Cadellin Skipsey: (10:28 AM) I think the suggestions never tested at very high capacities, tbh. Bruno @ RAL: (10:28 AM) The RAID 1 pair for OS is for historical reasons and is being evaluated. Ewan Mac Mahon: (10:29 AM) I think these machines are a quintisential use case for booting from a tiny CF card/USB stick type 'SSD'. Samuel Cadellin Skipsey: (10:29 AM) Speaking of the voms auth - are you planning on encoding xattrs into objects to do object level permissions for voms? [The way I handled this in the gfal2 plugin for / was just to strip off leading /s] George V @ RAL: (10:33 AM) @Samuel I wouldn't know about that, I'm still fairly new to these auth. methods. I am assuming the use of xattrs for authentication/permissions would be up to the gateway. S3 does its own thing, I'm not sure what GFTP/XrD do but that would be entirely up to the gateway. Samuel Cadellin Skipsey: (10:34 AM) Well, my point was more that the built in permissions system for rados is pool level. You can emulate object level perms with xattrs if you add it to the GridFTP layer etc Ewan Mac Mahon: (10:34 AM) In particular if you add it to both the xrootd and gridftp layers in a compatible manner. Samuel Cadellin Skipsey: (10:35 AM) Quite so, and since we know both developers... Ewan Mac Mahon: (10:36 AM) I think this is somewhere between 'vital' and 'total no brainer' - for vaguely normal grid use we're going to need both remote and local protocols to agree on permissions. And, as you say, it should be possible to get the right people to talk to each other. Quite how that then integrates with other interfaces is a whole other thing, but you could do quite a lot for 'grid' users with just gridftp/xroot, and maybe restrict S3 etc. to non 'grid' users. Samuel Cadellin Skipsey: (10:38 AM) It doesn't integrate anyway, Ewan - the S3 gateway has its own object striping mechanism which is not compatible with libradosstriper at present. Daniel Traynor: (10:40 AM) (cern)centos7 or sl7? Andrew Lahiff: (10:41 AM) We're generally using SL7 at RAL Daniel Traynor: (10:43 AM) do e have a list of cerns work (talks) George V @ RAL: (10:43 AM) @Brian I am looking at providing Tiju with an S3 gateway that does IPv6 other tasks have taken precedence Samuel Cadellin Skipsey: (10:44 AM) Dan: I shall distribute something to the list Matt Doidge: (10:46 AM) In theory the only thing you have to change is the environment setup scripts? Gareth Douglas Roy: (10:48 AM) Matt is there a wiki page with instructions on using the tarball? Matt Doidge: (10:48 AM) Yeah - it's a real mess though https://www.gridpp.ac.uk/wiki/EMITarball Gareth Douglas Roy: (10:49 AM) Cool, thanks Daniel Traynor: (10:49 AM) ruuning storm 1.11.8/9 Peter Gronbech: (10:50 AM) Will Alastairs slides be available , I didn't see an indico entry for today's meeting. Gareth Douglas Roy: (10:50 AM) @Matt looking to potentially mount in a container... is the NFS mount you use RO? Ewan Mac Mahon: (10:50 AM) @Pete Sam posted them to the storage list. Matt Doidge: (10:50 AM) Yes, you'd want it RO Gareth Douglas Roy: (10:51 AM) Ooh that's even better :) cheers! Matt Doidge: (10:51 AM) Let me know how it goes, I'm always interested in hearing how my tarballs are doing. Paige Winslowe Lacesso: (10:52 AM) I thought SL5 had support till 2017? So is it wlcg support only that ends March 2016? (if I heard that aright) Gareth Douglas Roy: (10:53 AM) Yeah will do, trying to minimise the container size, so mounting the WN RO from a shared source as well as CVMFS makes the container much easier to work with Matt Doidge: (10:53 AM) The Lancaster plan is largely "upgrading" a lot of our SL5 pool nodes by throwing them into a skip.