Grid Data Storage

----------------------------------------------------------------------------------------------------------

The aim of this tutorial is to introduce file management in EMI.

This is achieved by running two sets of commands:

"lfc-" commands interact with the LFC catalogue server that maps logical filenames to "site URLs".
"lcg-" commands include ones used to copy files to and from an SE, and to replicate files.

The lcg-commands performs also interaction with both the SEs and the catalogue server.

1. Environment variables

Several environment variables need to be set before you start to ensure that the correct catalog service is used.

$ echo $LCG_GFAL_INFOSYS

bdii.grid.arn.dz:2170

$ echo $LCG_CATALOG_TYPE;
lfc

$ echo $LFC_HOST;
lfc.grid.arn.dz

If one/more of them has different/empty value, please set it (them) in this way:

export LCG_GFAL_INFOSYS=bdii.grid.arn.dz:2170

export LCG_CATALOG_TYPE=lfc

export LFC_HOST=lfc.grid.arn.dz


Now, you are ready to start.

2. Listing file and directory

For each of the supported VO a separate "top level" directory exists under the "/grid" directory. You can see all the files that are stored for the dzgrid VO.
First ensure you have a running VOMS proxy and then type:

$ lfc-ls -l /grid/dzgrid

You will see a listing of the contents of the /grid/dzgrid directory.

3. Create directory

Before creating and uploading any of your own files create a personal directory for storage by typing:

$  lfc-mkdir /grid/dzgrid/userdirectory

To check that you have created your directory type:

$  lfc-ls -l

and you should see your directory (plus possibly those of other attendees).

4. Upload file into SE

The First example shows how to upload a file into a Storage Element. First create locally a simple text file:

$ echo "Hello users of Constantine School" > text_file.txt

The command used for this is lcg-cr (LCG copy and register). Type the following to store this file on the se01.grid.arn.dz storage element : (use lcg-infosites use lcg-infosites in order to find which the available SE are)

$ lcg-cr --vo dzgrid file:///home/userxx/text_file.txt -l lfn:/grid/dzgrid/userdirectory/text_file.txt -d se01.grid.arn.dz

The output should be something like this:

$ lcg-cr --vo dzgrid file:///home/bentaleb/text_file.txt -l lfn:/grid/dzgrid/bentaleb/text_file.txt -d se01.grid.arn.dz
guid:8c4c83d1-8fdf-4af2-b514-a5d872d4d9f7

Of course, the guid you will have is different, since it is an unique identifier for each file (except for the case when you are replicating a file, as we will see). Check that the file is there by listing the contents of your directory.

Before continuing it is worth noting the difference between the command used to store the file and the creation of the directory in previous. The directory created is just a virtual directory and only exists within the catalog of lfn's. On the other hand the file physically exists on an SE but has an additional "virtual" filename in the catalog. This is connected to why the commands just handling the lfn namespace tend to start with "lfc" whilst the commands manipulating the file directly tend to start with "lcg".

5. Get the file SURL

For several purpose, FTS for instance, is useful to know the file SURL (they can be many if the file has replicas somewhere). The appropriate command is lcg-lr (list-replicas) [lfn | guid]

$ lcg-lr --vo dzgrid  lfn:/grid/dzgrid/userdirectory/text_file.txt

exemple:

$ lcg-lr --vo dzgrid lfn:/grid/dzgrid/bentaleb/text_file.txt
srm://se01.grid.arn.dz/dpm/grid.arn.dz/home/dzgrid/generated/2013-06-17/file65cdc9c8-bd81-452c-8031-524b0e897cf4

6. Replicate file between SE

gLite supports file replication. A file can be stored on multiple SE's and then a running job can access the closest SE with the file on it, thus giving faster access times to the data. This also helps protect against failures/access difficulties with a particular SE. To find the list of SE'S available to you see the tutorial on lcg-infosites which can be found here. We will replicate the file just created to the SE se03.grid.arn.dz, and you verify that this variable is settings with the command:

$ export VO_DZGRID_DEFAULT_SE=se03.grid.arn.dz

Then you write:

$ lcg-rep --vo dzgrid -d se03.grid.arn.dz lfn:/grid/dzgrid/userdirectory/text_file.txt

There is no output from this command on success, but you can check that the replica was created by listing all the replicas of your file, that is done by using the LCG list replicas command:

$ lcg-lr --vo dzgrid lfn:/grid/dzgrid/userdirectory/text_file.txt

You should get two replicas listed, as here:

$lcg-lr --vo dzgrid lfn:/grid/dzgrid/bentaleb/text_file.txt

Note how the path to where each file is stored is different. This demonstrates how the use of a "lfn" avoids the need to understand the local filesystem where the replica is actually stored.

srm://se01.grid.arn.dz/dpm/grid.arn.dz/home/dzgrid/generated/2017-05-18/file5556d536-4279-4e01-9a55-0d211a500ac1
srm://se03.grid.arn.dz/dpm/grid.arn.dz/home/dzgrid/generated/2017-05-18/filed419de99-0237-4f89-b7a7-9d62bae2ad65

7. Download a file from SE to UI

Having already uploaded a file the next step is to show downloading a file. To download the file you already uploaded using the new lfn you have just created use the command:

$ lcg-cp -v --vo dzgrid lfn:/grid/dzgrid/userdirectory/text_file.txt  file:///home/userDir/text_file_copy.txt

You can check that the file you just downloaded is the same as the file you uploaded with cat command, you should find the same text you entered below.

$ cat text_file_copy.txt

8. Delete a file from SE

You can delete a file previously registered in SE with lcg-del

$ lcg-del -a --vo dzgrid lfn:/grid/dzgrid/userdirectory/text_file.txt

This will remove File Catalog entries as well

$ lfc-ls -l userdirectory

complete by removing the working directory on file catalog

$ lfc-rm -r /grid/dzgrid/userdirectory

$ lfc-ls /grid/dzgrid/ | grep userdirectory

How to

Links

-----------------------------