Brian, Duncan, Elena, Jeremy, John B, John H, Matt, Steve, Wahid, Jens, Robert, Ewan, Chris W, Rob, Raul. 0. Conclusion - if any - of the two or three discussions last week. - Data day at GridPP - I am guessing that's a no. - As regards industry data products (as opposed to hardware), I am also concluding that's a "no go." - Dark data - I don't think we finished this discussion. Perhaps we can summarise it. Hm. LFC and FTS checksum - the client should say what it thinks the checksum of the file is (which it gets from the LFC). Checking before the client discover. But the LFC entry is created when the file is transferred... which is the authoritative. Thought was to make a check outside of transfer time, if the file hadn't been moved recently (or accessed), by comparing the checksums. Chris W discovered more corrupted files. 1. The purpose of dteam: not deployment team any more but debugging team. Cf. Ewan's FAX debugging problem. dteam is the team that debugs stuff, so people should be supporting dteam! In Ewan's case, you could tell the client tools not to authenticate, you can access the data! If you authenticate as an unknown person, you get kicked out! Sites in UK normally support dteam, although things may break unintentionally. 2. FTS stuff - what's happening and more importantly why is what is happening happening? Roadmap, I think I'm trying to say. Monitoring: if monitoring is turned on, there is a database load issue. Jens has a student looking at analysing FTS data. Roadmap also includes multiple FTS servers with the same "channels." Servers should be talking to each other, not admins. Maybe they look at how busy the others are and talk to each other: that'd be cunning. Not just for multiple VOs using the same channel, but also for redundancy. If one is seeing failures on one channel, the other will know about it - but does that mean it should try, or not try? 3. GLUE2 support revisited - what's happening? All the storage and data management tools are using 1.3...? Total and reserved may redefine in GLUE2. With a CASTOR hat, does it break something to not have GLUE2? Probably best not to ignore it though. 4. Checksumming chunks - let's call it chunksumming. Jens' summer project (or one of them.) GridFTP may already be doing this, although by default isn't, or is that just CASTOR? 5. ATLAS file renaming update (Brian?), WebDAV status Wahid was talking to DPM developers. Needed a fix after earlier tests. Still needs more testing before pushing it out. On 1.8.7, it's in EPEL for testing. Ewan willing to provide SE for testing - ATLAS could accept the risk for one or two sites. Also being tested on CASTOR, ongoing. WebDAV tested against QMUL and it seems to work, generally. 6. Any SHA2 questions (kind of storage related, as CASTOR doesn't support SHA2 AFAWK) 7. AOB [10:01:46] Wahid Bhimji hello [10:03:25] Jens Jensen http://www.ngs.ac.uk/ukca/certificates/supported-internet-browsers [10:07:13] Wahid Bhimji O the industry "product" I semi-disagree but can discuss another time as it depends what products... [10:17:07] Steve Jones It's very lucky that Deployment and Debugging both start with D! [10:17:23] Wahid Bhimji yeah but we have a solution now [10:18:41] Ewan Mac Mahon export LFC_HOST=prod-lfc-atlas-ro.cern.ch ; export CSEC_MECH=ID ; lfc-ls / [10:18:48] Ewan Mac Mahon ^ this works. [10:18:54] Wahid Bhimji I think in this case - site would push back to Atlas if its not a site problem [10:19:09] Wahid Bhimji (or if they suspect that but can't check) [10:19:24] Wahid Bhimji feature - its world readable [10:20:27] Ewan Mac Mahon Indeed, but it was rather tricky to work out that it wasn't/isn't a site problem - it's not always clear. [10:21:06] Ewan Mac Mahon I'm very happy to throw things that I think are ATLAS' issue at ATLAS, but tbh, I assumed this really was us initially. [10:22:23] Ewan Mac Mahon And yes, Chris is correct on the authentication thing - if you leave out 'export CSEC_MECH=ID' and authenticate with a dteam proxy, the RO LFC refuses to talk to you, rather than just disregarding the authentication. [10:23:02] Wahid Bhimji indeed sometimes sites and atlas have to work together ... I agree that the behvaiour is not ideal (nor is the obscure setting for the env variable) [10:23:47] Wahid Bhimji I am sitting opposite Oliver now (and for the next week) at CERN - if you have an FTS feature request let me know ! [10:27:27] Jens Jensen I also have a student who is looking at analysing FTS3 data (just the file data). He wanted to try some data mining with MapReduce so I gave him a FTS dataset to look at, with 2.4 million entries. [10:30:11] Wahid Bhimji what resources do you have him to run on with MapReduce - is there a sectet RAL hadoop cluster I can have access to [10:30:30] Jens Jensen T1 has a non-production Hadoop cluster. [10:30:39] Jens Jensen Can check about access. [10:30:51] Christopher Walker So what information would I find useful? Finding and debugging channels with low speed - and working out which end is at fault. [10:31:06] Wahid Bhimji Please do Jens ! it would be very useful for my running physics analysis [10:31:16] Wahid Bhimji on hadoop testing (I have something to run now) [10:31:19] Jens Jensen OK. Contact is James Adams. [10:31:26] Jens Jensen I'll ask him anyway. [10:31:42] Wahid Bhimji ok - I have asked for permissions for things from him before (and not got them ) so if youc an ask that would be great [10:31:57] Christopher Walker Changes over time too. [10:32:37] Christopher Walker Finding dodgy disk servers [10:33:13] Wahid Bhimji sigh [10:33:51] Wahid Bhimji not for ATLAS... [10:34:21] Wahid Bhimji I would rather not bother with info provider on storage if we can avoid it. [10:35:58] Duncan Rand hdfs? [10:36:55] Raul Lopes you would need has trees, right? [10:37:04] Raul Lopes hash trees [10:37:36] Jens Jensen Hash trees (or Merkle trees) would also do it, but you have fixed chunks. [10:38:43] Wahid Bhimji Actually for DPM I am working on it [10:39:46] Wahid Bhimji in hand [10:40:19] Christopher Walker https://access.redhat.com/site/documentation//en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/storage.html [10:40:38] Duncan Rand working on what wahid? [10:40:42] Christopher Walker Support for raw ECKD access to the DASD driver Normal I/O operations through the DASD device driver give access only to the data fields of an ECKD device even for track based I/O. In Red Hat Enterprise Linux 6.2, the DASD device driver is extended to give access to whole ECKD tracks including count, key, and data fields. [10:43:18] Wahid Bhimji ah - thats good to know Chris - I was going to ask that . [10:44:31] Wahid Bhimji duncan - I meant working on renaming in DPM with webdav (sadly- not anything intesting - though I am working on interesting things too [10:47:00] Wahid Bhimji I am travelling next week as well... [10:47:26] Wahid Bhimji can't remember what time - probably could dial in actually .. ok see you