"Greig A Cowan"

Hi everyone,

Hope you have all recovered from GridPP13 ;-)

Have RAL got a work-a-round for the problem of publishing the correct available and used space in your dCache? We are having problems doing this in Edinburgh (and I think other sites are the same, namely IC and Manchester).

ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 -b mds-vo-name=RAL-LCG2,o=grid

seems to report reasonable looking information unlike the corresponding commands for the other sites. The bugs the Alessandra and Owen submitted to savannah regarding the dCache information system are still open. Is there anything more that we can do to address this issue?

Greig

"Steve Traylen"

On Thu, Jul 07, 2005 at 04:27:27PM +0100 or thereabouts, Greig A Cowan 
wrote:
> Hi everyone,
>
> Hope you have all recovered from GridPP13 ;-)
>
> Have RAL got a work-a-round for the problem of publishing the correct
> available and used space in your dCache? We are having problems doing 
this
> in Edinburgh (and I think other sites are the same, namely IC and
> Manchester).

I don't think we have a workaround, it just works?

I expect you have mentioned it before but what is the problem?

 Steve
>
> ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 -b
> mds-vo-name=RAL-LCG2,o=grid
>
> seems to report reasonable looking information unlike the 
corresponding
> commands for the other sites. The bugs the Alessandra and Owen 
submitted
> to savannah regarding the dCache information system are still open. Is 

> there anything more that we can do to address this issue?
>
> Greig
>
> --

"Alessandra Forti"

The srm command doesn't accept the host certificate. If I have to repeat

it another time I'll scream ;-)

cheers

alessandra

"Philip Clark"

I don't think we have a workaround, it just works? I expect you have mentioned it before but what is the problem?

http://savannah.cern.ch/bugs/?func=detailitem&item_id=8777

We need to understand why you are not seeing this bug. IC, Manchester and Edinburgh all seem to have it. If we try to monitor your storage through the lcg information system then I expect it will show up too.

-Phil

"Steve Thorn"

Steve

First we had the proxy problem. There was a discrepancy between /opt/d-cache/etc/dcache.kpwd and the host's DN. Running /opt/d-cache/bin/grid-mapfile2dcache-kpwd as root fixes it but something that I've yet to identify changes it back in a period of approximately 1 hour.

With the above fixed, the following commands all give sensible space output

# /opt/lcg/libexec/lcg-info-wrapper
# /opt/lcg/libexec/lcg-info-dynamic-se
# /opt/lcg/libexec/lcg-info-dynamic-dcache
/opt/lcg/var/gip/lcg-info-generic.conf

but the GRIS still reports incorrectly even after restarting globus-mds, killing any remaining slapd processes etc.

ldapsearch -x -H ldap://srm.epcc.ed.ac.uk:2135 -b
mds-vo-name=local,o=grid|grep Space
GlueSAStateAvailableSpace: 1
GlueSAStateUsedSpace: 1
...

Steve

"Jamie Kelvin Ferguson"

Whats the name of the srm/SE machine at RAL. If I use dcache.gridpp.rl.ac.uk in a standard query that I use successfully for all tier 2 sites, I get the following

$ ldapsearch -LLL -x -h dcache.gridpp.rl.ac.uk -p 2135 -b
"mds-vo-name=local,
o=grid" "GlueSAAccessControlBaseRule=dteam" 
GlueSAStateAvailableSpace
GlueSAStateUsedSpace

ldap_bind: Can't contact LDAP server

But thats after its been hanging for ages.

However if I try,

$ ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 -b
mds-vo-name=RAL-LCG2,o=grid "GlueSAAccessControlBaseRule=dteam"
GlueSAStateAvailableSpace GlueSAStateUsedSpace

I get the storages returned.

The second one clearly queries a different machine but I thought you could query an SE/srm directly? Why doesn't the first one work?

Jamie

"Alessandra Forti"

> that I've yet to identify changes it back in a period of approximately 
1
> hour.

that's the cron job.

/etc/cron.d/edg-mkgridmap

cheers
alessandra

On Thu, 7 Jul 2005, Steve Thorn wrote:

> Steve
>
> First we had the proxy problem. There was a discrepancy between
> /opt/d-cache/etc/dcache.kpwd and the host's DN. Running
> /opt/d-cache/bin/grid-mapfile2dcache-kpwd as root fixes it but 
something
>
> With the above fixed, the following commands all give sensible space
> output
>
> # /opt/lcg/libexec/lcg-info-wrapper
> # /opt/lcg/libexec/lcg-info-dynamic-se
> # /opt/lcg/libexec/lcg-info-dynamic-dcache
> /opt/lcg/var/gip/lcg-info-generic.conf
>
> but the GRIS still reports incorrectly even after restarting 
globus-mds,
> killing any remaining slapd processes etc.
>
> ldapsearch -x -H ldap://srm.epcc.ed.ac.uk:2135 -b
> mds-vo-name=local,o=grid|grep Space
> GlueSAStateAvailableSpace: 1
> GlueSAStateUsedSpace: 1
> ...
>
> Steve

"Steve Thorn"

Alessandra

That's what you'd think but running the cron by hand doesn't change it.

Steve

"Alessandra Forti"

Things are publicly published through the local BDII that normally runs on the CE. So IMO it is reasonable and perhaps also us (other sites) should do the same in order to reduce publicly available ports.

cheers

alessandra

Because port 2135 on our SRM isn't open at our site firewall. I'd have to defer to someone more knowledgeable about the information system that me to tell you if that's reasonable or not, but yours is the first complaint I've heard of. Derek

"Ross, D \(Derek\)"

Okay, I've had look at the scripts in the information system setup, and it looks as though we're not using srm-storage-element-info. Instead our lcg-dynamic-dcache script is this perl script:

#!/usr/bin/perl -w

use strict ;
use File::Basename ;

my $used  = '/var/lib/edginfo/used-space.dat' ;
my $total = '/var/lib/edginfo/available-space.dat' ;

my %space ;

open(USED,$used) or die "Could not open $used: $!\n" ;
while(<USED>) {
   if (/^(\d+)\s+(\S+)\s*/) {
       my $kb    = $1  ;
       my $path  = &basename($2)  ;
       $space{$path}{'used'} =  $kb ;
   }
}
close(USED) ;

open(TOTAL,$total) or die "Could not open $total: $!\n" ;
while(<TOTAL>) {
   if (/^(\d+)\s+(\S+)\s*/) {
       my $kb    = $1  ;
       my $path  = &basename($2)  ;
       $space{$path}{'total'} =  $kb ;
   }
}


foreach( qw/cms dteam atlas lhcb/ ){
  print "dn: 
GlueSARoot=$_:/pnfs/gridpp.rl.ac.uk/data/$_,GlueSEUniqueID=dcache.gri
dpp.rl.ac.uk,Mds-Vo-name=local,o=grid\n" ;
  print "GlueSAStateAvailableSpace: ".$space{$_}{'total'}."\n" ;
  print "GlueSAStateUsedSpace: ".$space{$_}{'used'}."\n\n" ;

}

We also have a cron job in /etc/cron.d/:

50 3 * * *  edginfo /usr/bin/du -s  /pnfs/gridpp.rl.ac.uk/data/* > /var/lib/edginfo/used-space.dat

And the /var/lib/edginfo/available-space.dat is a file consisting of

14252613123     /pnfs/gridpp.rl.ac.uk/data/atlas
14252613123     /pnfs/gridpp.rl.ac.uk/data/cms
14252613123     /pnfs/gridpp.rl.ac.uk/data/dteam
14252613123     /pnfs/gridpp.rl.ac.uk/data/lhcb

Derek

"Philip Clark"

Alessandra Forti writes:

> Things are publicly published through the local BDII that normally 
runs
> on the CE. So IMO it is reasonable and perhaps also us (other sites)
> should do the same in order to reduce publicly available ports.

Hi Alessandra,

If this happens would there be anyway for an external user/job to find out what storage you have available? In the long run we might be able to reserve storage via srm, but this is not in place yet, so this information is quite important.

-Phil

"Alessandra Forti"

Hi Phil,

you just have to do the same query to the site CE on port 2170.

For example to query RAL:

  ldapsearch -LLL -x -h lcgce01.gridpp.rl.ac.uk -p 2170 -b "mds-vo-name=RAL-LCG2, o=grid" "GlueSAAccessControlBaseRule=dteam" GlueSAStateAvailableSpace GlueSAStateUsedSpace

gives

dn: 
GlueSARoot=dteam:/pnfs/gridpp.rl.ac.uk/data/dteam,GlueSEUniqueID=dcac
he.gridpp.rl.ac.uk,mds-vo-name=RAL-LCG2,o=grid
GlueSAStateAvailableSpace: 14252613123
GlueSAStateUsedSpace: 79981449

cheers

alessandra

"Steve Traylen"

On Thu, Jul 07, 2005 at 04:54:33PM +0100 or thereabouts, Philip Clark 
wrote:
>
> >
> > I don't think we have a workaround, it just works?
> >
> > I expect you have mentioned it before but what is the problem?
>
> http://savannah.cern.ch/bugs/?func=detailitem&item_id=8777
>
> We need to understand why you are not seeing this bug. IC, Manchester
> and Edinburgh all seem to have it. If we try to monitor your storage
> through the lcg information system then I expect it will show up too.

Does your dcache.kpwd contain

/C=UK/O=eScience/OU=Manchester/L=HEP/CN=bohr0013.tier2.hep.man.
ac.uk/E=alessandra.forti@manchester.ac.uk

i.e are you seeing this one.

https://savannah.cern.ch/bugs/?func=detailitem&item_id=5295

but I'm sure I have already asked this question twice so feel free to
scream if we are going through the same loop?

 Steve
>
> -Phil

--
Steve Traylen
s.traylen@rl.ac.uk
http://www.gridpp.ac.uk/

"Steve Traylen"

On Thu, Jul 07, 2005 at 05:05:05PM +0100 or thereabouts, Jamie Kelvin 
Ferguson wrote:
> Whats the name of the srm/SE machine at RAL. If I use
> dcache.gridpp.rl.ac.uk in
> a standard query that I use successfully for all tier 2 sites, I get 
the
> following
>
> $ ldapsearch -LLL -x -h dcache.gridpp.rl.ac.uk -p 2135 -b
> "mds-vo-name=local,
> o=grid" "GlueSAAccessControlBaseRule=dteam" 
GlueSAStateAvailableSpace
> GlueSAStateUsedSpace
>
> ldap_bind: Can't contact LDAP server
>
> But thats after its been hanging for ages.
>
> However if I try,
>
> $ ldapsearch -x -H ldap://site-bdii.gridpp.rl.ac.uk:2170 -b
> mds-vo-name=RAL-LCG2,o=grid "GlueSAAccessControlBaseRule=dteam"
> GlueSAStateAvailableSpace GlueSAStateUsedSpace
>
> I get the storages returned.
>
> The second one clearly queries a different machine but I thought you 
could
> query an SE/srm directly?
> Why doesn't the first one work?

It is behind a firewall on purpose, to stop people writing monitoring
scripts against it :)

Please use the site-bdii. 

 Steve

"Steve Traylen"

Okay, I've had look at the scripts in the information system setup, and it looks as though we're not using srm-storage-element-info. Instead our lcg-dynamic-dcache script is this perl script:

This is true, I have completly replaced the script with something less intensive but it was working before I changed it.

Steve

"Greig A Cowan"

Hi Steve,

Does your dcache.kpwd contain

/C=UK/O=eScience/OU=Manchester/L=HEP/CN=bohr0013.tier2.hep.man.
ac.uk/E=alessandra.forti@manchester.ac.uk

No, dcache.kpwd does not contain this line referring to Manchester. It does contain the following line that refer to our dCache host:

mapping
"/C=UK/O=eScience/OU=Edinburgh/L=NeSC/CN=host/srm.epcc.ed.ac.uk
/emailAddress=g.cowan@ed.ac.uk" 
edginfo
       
/C=UK/O=eScience/OU=Edinburgh/L=NeSC/CN=host/srm.epcc.ed.ac.uk/
emailAddress=g.cowan@ed.ac.uk

I do not have Email= or E=, but emailAddress= in the DN. Is this what everyone has?

Greig

i.e are you seeing this one.

https://savannah.cern.ch/bugs/?func=detailitem&item_id=5295

but I'm sure I have already asked this question twice so feel free to scream if we are going through the same loop?

Steve

"Jensen, J \(Jens\)"

emailAddress is correct.

OpenSSL, as of version 0.9.7 (long time ago), uses emailAdress instead of Email when it *displays* the DN, so GT, being built on top of OpenSSL, does the same.

We shouldn't really have the email address in the DN at all, but that's another story.

-j

"Alessandra Forti"

Hi Steve, mybe there is something in th format of RAL host DN that makes it different from the other sites.

Just as an attmpts it would be perhaps useful to compare the host lines in dcache.kpwd. Can you send the RAL ones?

thanks

cheers alessandra

"Alessandra Forti"

no I have

mapping
"/C=UK/O=eScience/OU=Manchester/L=HEP/CN=bohr0013.tier2.hep.man
.ac.uk/emailAddress=alessandra.forti@manchester.ac.uk" edginfo

login edginfo read-write 18948 18948 / / /

/C=UK/O=eScience/OU=Manchester/L=HEP/CN=bohr0013.tier2.hep.man.
ac.uk/emailAddress=alessandra.forti@manchester.ac.uk

cheers
alessandra

"owen maroney"

And for IC we have:

mapping
"/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.u
k/emailAddress=lcg-site-admin@imperial.ac.uk"
edginfo

login edginfo read-write 19491 19491 / / /

/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.uk
/emailAddress=lcg-site-admin@imperial.ac.uk

So, I try replacing this with:

> mapping 
"/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.u
k/E=lcg-site-admin@imperial.ac.uk" edginfo
>
> login edginfo read-write 19491 19491 / / /
>         
/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.uk
/E=lcg-site-admin@imperial.ac.uk

then

su - edginfo
> [edginfo@gfe02 edginfo]$ /opt/d-cache/srm/bin/srm-storage-element-info 
https://gfe02.hep.ph.ic.ac.uk:8443/srm/infoProvider1_0.wsdl

produces stuff ending with:

> StorageElementInfo :
>                      totalSpace     =2541546897408 (2481979392 KB)
>                      usedSpace      =30397658395 (29685213 KB)
>                      availableSpace =2502704826621 (2444047682 KB)

Hurrah! (although this will get overwritten at the next edg-mkgridmap update...)

However, although now when I run, as edginfo:

> [edginfo@gfe02 edginfo]$ /opt/lcg/libexec/lcg-info-dynamic-se

I get a 3 second pause and output like:

> dn: 
GlueSARoot=lhcb:/pnfs/hep.ph.ic.ac.uk/data/lhcb,GlueSEUniqueID=gfe02.
hep.ph.ic.ac.uk,Mds-Vo-name=local,o=grid
> GlueSAStateAvailableSpace: 2444047682
> GlueSAStateUsedSpace: 37931710

when I run, as edginfo, /opt/lcg/libexec/lcg-info-wrapper, it takes less than a second and produces output including:

> GlueSAStateAvailableSpace: 00
> GlueSAStateUsedSpace: 00

in the output. I checked /opt/lcg/var/gip/tmp and the file lcg-info-dynamic-dcache.ldif.7010 is being updated but is only zero sized.

So the output of the dynamic-se script does not seem to be getting incorporated into the output of the wrapper script.

cheers,

Owen.

"Greig A Cowan"

I essentially find the same as Owen just reported.

Should we be making another bug report?

Greig

On Fri, 8 Jul 2005, owen maroney wrote:

> And for IC we have:
>
> mapping
> 
"/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.u
k/emailAddress=lcg-site-admin@imperial.ac.uk"
> edginfo
>
> login edginfo read-write 19491 19491 / / /
> 
> 
/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.uk
/emailAddress=lcg-site-admin@imperial.ac.uk
>
> So, I try replacing this with:
> > mapping 
"/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.u
k/E=lcg-site-admin@imperial.ac.uk" edginfo
> >
> > login edginfo read-write 19491 19491 / / /
> >         
/C=UK/O=eScience/OU=Imperial/L=Physics/CN=gfe02.hep.ph.ic.ac.uk
/E=lcg-site-admin@imperial.ac.uk
>
> then
>
> su - edginfo
> > [edginfo@gfe02 edginfo]$ 
/opt/d-cache/srm/bin/srm-storage-element-info 
https://gfe02.hep.ph.ic.ac.uk:8443/srm/infoProvider1_0.wsdl
>
> produces stuff ending with:
>
> > StorageElementInfo :
> >                      totalSpace     =2541546897408 (2481979392 KB)
> >                      usedSpace      =30397658395 (29685213 KB)
> >                      availableSpace =2502704826621 (2444047682 KB)
>
> Hurrah! (although this will get overwritten at the next edg-mkgridmap
> update...)
>
> However, although now when I run, as edginfo:
> > [edginfo@gfe02 edginfo]$ /opt/lcg/libexec/lcg-info-dynamic-se
>
> I get a 3 second pause and output like:
> > dn: 
GlueSARoot=lhcb:/pnfs/hep.ph.ic.ac.uk/data/lhcb,GlueSEUniqueID=gfe02.
hep.ph.ic.ac.uk,Mds-Vo-name=local,o=grid
> > GlueSAStateAvailableSpace: 2444047682
> > GlueSAStateUsedSpace: 37931710
>
> when I run, as edginfo, /opt/lcg/libexec/lcg-info-wrapper, it takes 
less
> than a second and produces output including:
> > GlueSAStateAvailableSpace: 00
> > GlueSAStateUsedSpace: 00
> in the output.  I checked /opt/lcg/var/gip/tmp and the file
> lcg-info-dynamic-dcache.ldif.7010 is being updated but is only zero 
sized.
>
> So the output of the dynamic-se script does not seem to be getting
> incorporated into the output of the wrapper script.
>
> cheers,
> Owen.

"Alessandra Forti"

Yes.

On Fri, 8 Jul 2005, Greig A Cowan wrote:

> I essentially find the same as Owen just reported.
>
> Should we be making another bug report?
>
> Greig