Attending: Brian, Chris W, Duncan, Elena, Gareth, Jens (C+M), Jeremy, John B, John H, Matt D, Raja, Sam, Steve, Alessandra, David, Ewan, Govind, Raul Apologising: Wahid 0. Operational issues - *and* blog posts. How to provide resources for "small" VOs, particularly those which might later grow bigger (cf T2K), specifically HyperK in this instance. Getting them started with an easy setup - with no particular quota or space tokens - makes it harder to wean them off not using space tokens (wean them on using space tokens?) Growth, of course, is facilitated also by GridPP deciding to allocate more resources to the VO (funding helps, too). Also slightly unhelpful is the inconsistent treatment of space tokens in different SRM implementations - is it a quota, a "pointer" to an area, a disk pool, a reservation, is it a bird, is it a plane? DPM makes space tokens orthogonal to the disk pools. StoRM uses space tokens as a quota in StoRM (but depending on the underlying implementation). In CASTOR, a space token maps to a service class, which is a device for placing data into the right disk pools and tape pools, not easily understood by mortals. Chris had made a valiant attempt from earlier at describing the situation: https://www.gridpp.ac.uk/wiki/Storage/SpaceTokens ATLAS is a keen user of space tokens but also change how they use the space tokens more than other space token users. T2K may also want to copy data to/from iRODS - could they copy directly from the WN? Would they need iRODS clients - they would if they are not using GridFTP to move data into iRODS. It might be useful to have a staging area where it is understood that files can be deleted. However, deletion of VO data should not be done lightly, even when they are warned that it might happen - but requiring them to track their own data and copy off the stuff they want to keep may not be a significant burden on a small VO? Could we warn them for each file about-to-be-deleted or could we check whether the file has been copied off? If they don't pay for the staging area, where is otherwise the incentive to clean up? We should probably pull together the stuff we have and see if we can come up with better guides - a policy, even - of describing how to use gridPP resources without cutting the VO off from future growth. Without more control, an alternative approach is to allocate a disk server to a small VO and have them share the space and then forget about them (as it were). As another technical problem, ATLAS have problems with WebDAV at Cambridge. John H shared the problem with an ATLAS list but will mail the storage list. 1. End of quarter stuff -questions mostly for the usual suspects (a) did we do anything? (b) did we tell anyone? (c) did we achieve anything? (d) what shall be our goals for the coming quarter? Specific questions we had for this quarter: - how is the WebFTS evaluation coming along - HTTP testing for DPM (and space tokens) Not much progress on these, due to software deployments being delayed. Also the generic questions: - interesting publications, meetings, conferences, that I don't already know about. Possible goals for the coming quarter: Sounds like the small VO guidelines/policy business is a good one to get stuck into. 2. There is a WLCG workshop coming up: http://indico.cern.ch/event/305362/ Both Sam and Wahid are going; Wahid is, as you can see, speaking, and would be interested in suggestions - theme is approximately storage, from ~today to a few years ahead; having WLCG covered, Wahid is interested in sites' non-WLCG perspective. There are draft slides but Wahid hasn't shared them yet. 3. Other topics to revisit - procurement, information systems, technology, small VOs, EGI 4. AOB Chris spoke with Tom re resilience of WebDAV for CERN@School. They are using "DFC" instead of LFC. Christopher John Walker: (02/07/2014 10:03:16) https://www.gridpp.ac.uk/wiki/Storage/SpaceTokens Is what I wrote up when I last had this discussion Elena Korolkova: (10:04 AM) also there was a problem for webdav access for some sites. Cambridge, for example. I'm not sure that John solved the problem (at least it was not obvious from email exchange last week) John Hill: (10:04 AM) No I haven't It worked several months ago, but now doesn't and I haven't knowlingly changed the config Elena Korolkova: (10:06 AM) The only hope are Sam or Wahid Ewan Mac Mahon: (10:06 AM) It's sortof both. On our system at least there's very very little 'free for all' space any more. John Hill: (10:06 AM) Wahid was going to look at it - not sure whether he's had time yet Ewan Mac Mahon: (10:06 AM) A token works as both a reservation and limits them. If they use the free-for-all space they'll probably fill it and then things will break, not least fo them. Steve Jones: (10:07 AM) Yes, but only if they use it. If there is any way not to use it, it's tempting. Ewan Mac Mahon: (10:07 AM) That too. Steve Jones: (10:07 AM) Sorry, "hve to use it". Ewan Mac Mahon: (10:08 AM) If they're trying to write everything to a central iRods, I do think they should just do it until or unless they can't any more. Duncan Rand: (10:08 AM) Can't hear you Chris Ewan Mac Mahon: (10:09 AM) t2k or hyperk? Samuel Cadellin Skipsey: (10:09 AM) hyperk Ewan Mac Mahon: (10:21 AM) More like PRODDISK, I think. SCRATCHDISK is sortof storage-y, PRODDISK is very much a staging area. John Bland: (10:25 AM) we do auto-cleanup on local storage. Has to be widely publicised this happens, and they get a a 7day warning Gareth Douglas Roy: (10:26 AM) If it fills up you just stop running jobs, same as with ATLAS, we're providing a service not managing the VO... John Bland: (10:26 AM) this does sometimes lead to files being left for auto-deletion, rather than tidying up as they go along Ewan Mac Mahon: (10:29 AM) It's also the 'and then give a warning' bit that bothers me a little - I don't want the hassle if I can avoid me. Er, avoid it. Gareth Douglas Roy: (10:30 AM) Well and by sad experience not everyone listens to the warning and you have the fall out of missing _vital_ data Christopher John Walker: (10:33 AM) Perhaps scratch spacetokens should be named HyperK_Scratchdisk or something Matt Doidge: (10:33 AM) It's important to also drill into the VOs that Tier 2s *aren't* for custodial data keeping of vital files. Samuel Cadellin Skipsey: (10:33 AM) Indeed, Matt's point is very valid Ewan Mac Mahon: (10:34 AM) True. So, I think the reccomendations could be: Samuel Cadellin Skipsey: (10:34 AM) Or at least, if you're using them for that, you need copies at more than 1 site Ewan Mac Mahon: (10:34 AM) - Tiny VO - no local storage Jens Jensen: (10:34 AM) http://indico.cern.ch/event/305362/ Ewan Mac Mahon: (10:34 AM) - Medium VO - one space token for FTS stageout - Big VO - whatever you want. Gareth Douglas Roy: (10:35 AM) Sorry have to leave Ewan Mac Mahon: (10:36 AM) That doesn't excude the possibility of people coming along knowing that they want different things and having a conversation about it, but those are the defaults for the people that don't know they want something else. Elena Korolkova: (10:38 AM) Many thanks, Sam There was also Manchester which had a problem Matt Doidge: (10:38 AM) John - we had a dodgey /etc/dmlile.conf.d/mysql.conf on our headnode after our upgrade *dmlite.conf.d On top of the dodgey ssl.conf Alessandra Forti: (10:39 AM) what problem did manchester have? Ewan Mac Mahon: (10:40 AM) Incedentally, the IP Range for Cambridge in the gocdb is 0.0.0.0/255.255.255.255 which is probably not right. John Hill: (10:40 AM) Matt - define "dodgy" Elena Korolkova: (10:40 AM) wevdav you said you will have a look Matt Doidge: (10:40 AM) It was replaced and didn't have the correct details in it - causing transfers to fail for a bit with database errors We actually just removed it and then things worked...