Brian, Chris W, David, Gareth, John B, Ewan, Matt, Sam, Steve, Wahid, Jens, Duncan People in Friday meeting, also for their benefit. Useful as well in general. Currently leading data activities, data transfers, compared to what others find difficult. Future directions? How much we should engage with other projects - eg those using iRODS. Easy-to-use tools, management of tools, long tail science. Liaising with other big data projects activities? SKA, EBI Eventually we should end up using the same tech? Or at least it would be good to be interoperable (using open standards). OpenStack has a very large user community. Proposals for running stuff and engaging with others. Consistency checking: remote consistency checking. WebDAV via GFAL to check against catalogues, as a separate tool, or preventing the inconsistencies in the first place - SE message. Need a consistent state before we can maintain the consistency. Maybe somehting without too much manual intervention - between the LFC and the SE it could run in the background. Originally meant to be loosely coupled, with automatically recovering, falling back to the available replicas, but not with much thought to repair. Small VOs also cannot always afford multiple copies of files, so working on their consistency; Sam is working on something like RAID for them, or RS-code. Good then that their LFC is the RAL one. Ewan discovered an issue with 1.8.7 - David Smith got it sussed: /dpm/ - xrootd with DMlite doesn't work in this condition (see chat). WebDAV either seems to work or return 404 from the head node. Or maybe there are two different problems. DMLite may not have worked if your poolnode filesystem was mounted on /dpm?? This is not a new thing?! Metadata for files in storgae - can be done in LFC with comments. Like AMGA: Glasgow ran, KISTI are doing work with AMGA. Separate from LFC and SEs: would be better because you want the same metadata from all the replicas. iRODS does something similar. DOI for future? DPM workshop - now more likely 13 Dec. [10:05:14] Wahid Bhimji https://www.gridpp.ac.uk/wiki/StorageGroupActivities [10:05:35] Wahid Bhimji please send it and I can attach it to the wiki page [10:05:44] Sam Skipsey (I should say, the background music does add a certain something to Jen's speaking ) [10:06:42] Jens Jensen Maybe I should have it as a permanent feature [10:11:43] Jens Jensen Mark is here as well, he says hi [10:12:17] Wahid Bhimji jens stop talking you arejust causing distortion [10:12:28] Jens Jensen I've muted! [10:13:26] Ewan Mac Mahon Indeed; our networking and transfer tuff is a LOT better than (apparently) anyone else's. [10:18:33] Wahid Bhimji I tried putting that at (Chris and Jens's at ) https://www.gridpp.ac.uk/wiki/StorageGroupActivities#Future_Activities [10:19:26] Sam Skipsey So, we are talking to some bioinformaticians here at Glasgow [10:19:32] Sam Skipsey About storage amongst other things. [10:19:48] Jens Jensen The workshop is a great example of this engagement [10:20:42] Jens Jensen But also there's a case for open standards! [10:22:02] Ewan Mac Mahon There is an argument, particularly within academia, that it's not us that's re-inventing things, it's everyone else. We did get to some of this stuff first. [10:22:44] Sam Skipsey There is an argument, mostly in business, that if you don't tell promote your tools, then someone else will get the market. [10:23:18] Sam Skipsey (And, we got to a lot of the stuff first with a toolset that only works on one particular flavour of Linux ) [10:24:12] Jens Jensen We're already supporting other VOs (albeit mainly HEPish ones) [10:24:18] Ewan Mac Mahon But it's the BEST flavour of Linux! [10:24:34] Ewan Mac Mahon And it's certainly the 'enterprise' friendly one. [10:24:55] Jens Jensen I think Sam hit the nail on the head: interoperation... [10:25:21] Christopher Walker I won't be here next week unfortunately. [10:27:08] Wahid Bhimji Yeah sorry - do send me any comemnts [10:31:01] Jens Jensen It's always a hard problem - consistency across distributed systems [10:31:14] Wahid Bhimji yeah we talk about it soooo much [10:32:16] Jens Jensen Keeps computer scientisits busy, writing papers [10:32:59] Jens Jensen Yep, there wasn't that much though about repair - failures were considered temporary [10:33:37] Ewan Mac Mahon So, back to our theme of interoperability, sounds like we're looking at ditching everything and replacing it with one honking great big ceph filesystem acrocc the entire country? [10:33:42] Ewan Mac Mahon That should do it. [10:33:50] Jens Jensen [10:35:54] Brian Davies any VO who only keep one copy of data they want ot keep is being naieve. "Two is one, one is none" [10:35:58] Ewan Mac Mahon I'm special. [10:35:59] Wahid Bhimji /dpm [10:36:04] Wahid Bhimji https://www.gridpp.ac.uk/wiki/DPMUpgradeTips [10:36:36] Sam Skipsey Brian: but some of them can't afford to. (So the idea is that you let them pick a partial redundancy per file "1.5" copies as chunks etc) [10:41:51] Ewan Mac Mahon I say again, I'm special [10:43:12] Wahid Bhimji So do you want to go back to xrootd for atlas now ewan [10:44:00] Ewan Mac Mahon Good question; not sure. It should be working fine, but there's a bit of a question of whether we're likely to break it again testing things. [10:44:08] Ewan Mac Mahon So I'm not sure what's best. [10:44:31] Ewan Mac Mahon We could, but we might well have to switch it back again at some point. [10:45:39] Ewan Mac Mahon We need an XML schema for putting extended attributes into LFC comments. [10:45:39] Wahid Bhimji up to you really... switching is not huge hassle... I doubt a proper fix is going to come out of the dpm guys that soon [10:46:01] Wahid Bhimji but we can wait a little while - i don't mind really [10:46:43] Ewan Mac Mahon Well, let's see if David comes up with anything today based on my last email about what the WebDAV interface is doing, and then re-assess. [10:46:54] Ewan Mac Mahon So, leave it until tomorrow, at least? [10:47:42] Ewan Mac Mahon If only there was some scheme for generating uniform resource identifiers without a central authority. [10:48:45] Wahid Bhimji no I'm ok - thanks