Attending: Dan (special guest star), David, Gareth, Jens (C+M), John B, Wahid, Elena, Duncan, Steve, Robert, Sam, Ewan, Matt D, Chris W, Adam, Pete Apologies: Brian (leave), Tom (whose apology got caught initially in STFC's recently even less impressive corporate spamfilter) 0. Operational John asked about HTTP access to DPM (ie anonymous and unauthenticated): in which case unauthenticated people can read world readable files without authenticating and they can read them anonymously. The head node, if using HTTP, generates a token which allows HTTP access (via redirect) on the pool node, so the pool node must also have HTTP open and accessible. Previously, we'd seen problems with rampaging robots, as DPM would not be able to offer a robots.txt file in /, but this will now be fixed in 1.8.9 or if not, in 1.8.10. [See discussion in chat as to what the default is.] 1. An audience with NA62 DIRAC interface is set up but not used yet. NA62 currently taking "pilot physics" data, over 60 days, due to finish soon. The PP data will be about 300TB, corresponding to a full (expected) data rate of 1 PB/yr. Data is written at CERN, where it will be processed, but it may be worth while copying some of the data - in January - to RAL and do some processing there, and from there copy it to the T2s. Alternatively, T2s can fetch data directly from CERN - or these data models can be mixed. How is the T1 allocation (and T2 allocations) managed? It's still an application to the UB, Pete will follow up. The data volumes for the skim data will be about 10-20% of the raw data. User analysis will take place at the T2s. How is the data analysed at the T2s - is it accessed remotely (federatedly) from each T2, or will it be copied to the T2 first as in the traditional grid models? Not sure yet: expecting to try both and see which one is better - a T2D, T2 with data and federate the rest. Also depends on the performance of the networking at the T2s, to RAL (or to CERN): the perfsonar stuff should provide useful input into this. Sam suggested an erasure coded model where data is split across several T2s and will need to be accessed at a number of them - say a 7/4 model where 7 T2s together hold it and any four will be sufficient - but then four *will* have to be accessed to get the data. Data models also have analysis data from T2s copied back to CERN (or to T1?) How is the data tracked - if it's all embarrassingly parallel, perhaps it doesn't matter so much, but it is still required to track where the data is - this was also highlighted at Monday's big data event - DIRAC have DFC, the DIRAC File Catalogue, and also Dan's File Catalogue, as Dan is making sure that the data can be tracked. What should a T2 do (L'pool, B'ham, IC, GLA) to support NA62? Nothing, other than the usual stuff of providing resources and advertising them; the DIRAC server at Imperial will do the rest. Currently NA62 have 90 TB at RAL but haven't used anything - NA62 had intended to use them for simulations - but will follow up with the UB regarding their future expansions. 2. Next week, Brian will volunteer to give us feedback on the "big data" workshop at Wellcome; Chris was there, too, and will provide feedback also, if pos. Also, there is a GDB today which seems to be more about cloud accounting than storage accounting, and an ARGUS meeting tomorrow; and there's the HTCondor workshop which is apparently more about the jobs than the HTCondor storage. Robert Frank: (10/12/2014 10:09:03) # Redirect using SSL or plain HTTP? Default is On NSSecureRedirect On wahid: (10:11 AM) so for me etc/httpd/conf.d/zlcgdm-dav.conf:NSSecureRedirect Off /etc/httpd/conf.d/zlcgdm-dav.conf.rpmnew: NSSecureRedirect On so it seems something (like Yaim changes it from the setting in the rpm) Robert Frank: (10:13 AM) in yaim: DPM_DAV_SECURE_REDIRECT="On" # Enable redirection from head to disk using plain HTTP. John Bland: (10:14 AM) wahid: so if you had poru 80 open on your headnode you'd probably have the same setup as us Matt Doidge: (10:14 AM) On our headnode we have SecureRedirect On. On most of our pool nodes it's off. For the pools I recently installed it's set to On though John Bland: (10:14 AM) poru=port wahid: (10:16 AM) So I have the DPM_DAV_SECURE_REDIRECT="On" commented out in yaim - so it does the yaim default which is Off I think and Atlas will prefer that for Davix direct access performance which will be appaling with https I rather suspect they for one don't operational care if people can read. But politically allowing even those without any cert to read will not be popular. John was Adrien actually unhappy with 80 closed outside Liv on the headnode (and open on the disk servers) Ewan Mac Mahon: (10:19 AM) Dang it :-) Though PS results to the Tier 1 aren't the greatest because their probe box is limited. Christopher John Walker: (10:19 AM) http://pprc.qmul.ac.uk/~lloyd/gridpp/nettest.html John Bland: (10:20 AM) wahid: the request was port 80 on disk, no mention was made of the headnode, but our headnode was one of the few that didn't have 80 open as well, so if other sites turn off secure redirect they'll hit the same issue Duncan Rand: (10:22 AM) Ewan: what do you mean by limited? Ewan Mac Mahon: (10:23 AM) Last I heard it was plugged into a 1G port. Duncan Rand: (10:24 AM) https://maddash.aglt2.org/maddash-webui/details.cgi?uri=/maddash/grids/UK+sites+-+UK+Cloud+BWCTL+Mesh+Test/lcgps02.gridpp.rl.ac.uk/heplnx130.pp.rl.ac.uk/Throughput 10 Gbps now I think. Rates to and from Oxford are nothing special though... Correction RAL data asymmetric: https://maddash.aglt2.org/maddash-webui/index.cgi?dashboard=UK%20sites Jens Jensen: (10:28 AM) DFC : Dan's File Catalogue http://indico.cern.ch/event/272780/ http://indico.cern.ch/event/348018/