Attending: Brian, Jens, Gareth, Steve, Wahid, John H, Duncan, Robert, David, Matt D, Elena, Chris W, Pete, Raja, Ewan, Jeremy, Govind Special guest stars: Matthew Viljoen (RALT1), Cheney Ketley (RALT1) on cloud storage and owncloud respectively. Apologies: Tom 0. Blogerational op posts Wahid reports an operational problem with DPM 1.8.9. two "little but significant" things First, SHA2 VOMS certs did not get deployed by the puppet module, so an update is needed. Secondly, the GridFTP server dumps a lot of stuff into /var/log/messages, basically a log of every read, so /var might fill up. This is fixed in EPEL testing; in the interim, it can be modified with rsyslog. Duncan changed the loglevel but this may affect only the GridFTP server log as opposed to messages. Duncan also seems to find passwords in the log files? (Matthew Viljoen, RAL) https://indico.cern.ch/event/336753/ Matt gave a summary of the event at CERN. Lots of people had built stuff on owncloud; also CERN's version of dropbox (sort of) featured (called cernbox of course). The latter has a 5GB avg usage per user, using ownbox clients but of course a modified server - based on EOS. Idea is to have "storage for science," allowing people to share data with each other. Clients exist for Linux, mac, Windows, Android. Surfdrive (of the Dutch NREN, SurfNet), provides space of 100GB/person; with a total of 200TB they then have thousands of users. Costed: needs a model that gets enough users paying for it. Also IBM had trialed owncloud, a scalability PoC with $10^4$ users. Using the "Enterprise Edition" of OwnCloud on GPFS. Was there anything on replication? People were replicating data using WebDAV and FTS3. The latter supports 3rd party copying on WebDAV but only with special magic that allows an endpoint to initiate a transfer (and no doubt some delegation juju). 2. Report from hepsysman (anyone-who-went) Hepsysman and HEPiX. Chris reported CEPH/Hadoop work, federated with xroot. Also a file "repair" process which is more like a replacement, as xroot is used to fetch a fresh copy from another site - so like "federation", only with one extra step. Jeremy points to Tom's presentation of DIRAC with CERN@School. Should we (the GridPP storage and data management) group do more DIRAC stuff? Probably, if there is increased uptake among non-LHC VOs and perhaps further resources added. We have "tried it" and even got the t-shirt (really) but otherwise don't have much experience. ElasticSearch is also interesting: Lancaster (Peter) using it, T1 (Rob Appleyard) using it for CASTOR logs, Edinburgh (Andy), and Glasgow (David,Gareth). Used for parsing and searching log files. Wahid points out that the sun will be shinier in DPM 1.8.9 as it will have a log format more amenable to parsing. Not storage but Lancaster and Edinburgh also looking at SGE logs. Peter has code in github but not the GridPP github. Should 3. Update on tasks and stuff (quick...) We now have more greenish tasks in KeyDocs at least (for Grid Storage docs), and one blog post. Still need more stuff though! 4. AOB Chris reports increased traffic - 6 Gb - at QMUL, thanks to brokering leading traffic to QMUL. Also, Chris is reducing his time doing GridPP stuff and doing more central IT; will eventually leave GridPP (but hopefully stay in touch). Matt asked about 36 bay storage with 4TB. RAL has had them for >1yr, partly serving as CASTOR disk servers and partly non-CASTOR D1T0. Also Glasgow. Ewan points out that RAID controllers used to not support 36 drives, in which case various shenanigans were needed, such as raiding as much as possible and leave some drives for stuff, or having 2x18 or something. Matt is getting LSI 9260 8i controllers. wahid: (19/11/2014 10:03:12) Nov 19 09:26:34 gridpp09 globus-gridftp-server[22523]: dmlite 1 write : count:131072 Nov 19 09:26:34 gridpp09 globus-gridftp-server[22523]: dmlite 1 write : Exiting. count:131072 res:131072 Duncan Rand: (10:10 AM) Wahid, I get those too. Jens Jensen: (10:10 AM) https://indico.cern.ch/event/336753/ wahid: (10:14 AM) thanks Duncan the epel- test version should stop them dmlite 0.7.2 you could check if it does for you - that would be interesting. It seems to for me - but also fts transfers haven't come in Matt Doidge: (10:16 AM) Peter's talk on Logstash included our first attempts to use it to monitor our pool nodes. Jens Jensen: (10:16 AM) https://indico.cern.ch/event/350917/ Jeremy Coles: (10:17 AM) Tom mentioned a few storage related items of interest when talking about DIRAC and CERN@School: https://indico.cern.ch/event/350917/session/0/contribution/11/material/slides/0.pdf. Gareth Douglas Roy: (10:23 AM) https://github.com/ptrlv/logstash Duncan Rand: (10:25 AM) Wahid: the emi metapackage complains... Did you uninstall it? wahid: (10:26 AM) ah right - I never use the metapackage sigh - thats a a bit rubbsih - it should allow a greater than version.. thats good feedback for the guys - what does it say Duncan Rand: (10:28 AM) Error: Package: dmlite-plugins-mysql-0.7.2-1.el6.x86_64 (epel-testing) Requires: dmlite-libs(x86-64) = 0.7.2-1.el6 Installed: dmlite-libs-0.7.1-1.el6.x86_64 (@epel) dmlite-libs(x86-64) = 0.7.1-1.el6 So possibly not the metapackage... This may be the old problem we had which is that IPv6 enabled repos don't have all the rpms... Jens Jensen: (10:37 AM) http://www.gridpp.ac.uk/php/KeyDocs.php Matt Doidge: (10:37 AM) I have an AOB kit-based question, that I might have asked before. Ewan Mac Mahon: (10:39 AM) Is 'how to install dcache' really a key doc? It's not something that happens very often, and quite possibly never again. Jeremy Coles: (10:39 AM) It depends on your region. wahid: (10:40 AM) oh - we'll miss you But congratulations and all the best too of course Ewan Mac Mahon: (10:45 AM) They weren't cheapest when we last bought, so we stuck with our favourite Dells. Gosh. That's pretty good. Jens Jensen: (10:49 AM) I'll have to check how ours are raided Matt Doidge: (10:51 AM) Thanks a lot of that info chaps! *for that info