The dataStorage.pl script (v2.90)

The dataStorage.pl script runs like a deamon on all the storage farm machines from ST4 to ST13 (except for ST6 and ST9 that are archive-only hosts) at the port 9020 on the storage subnet (stXXrear:9020, where XX is a number from 04 to 13).

- the executable is located at /root/bin/dataStorage.pl
- the working directory with data and configuration files is: /var/datastor

• Short description about it's functionalities:

The dataStorage.pl script:

- send periodically information about all the managed disk to the dataManager.pl module (polling system);

- checks the files received from the dataManager.pl:rsync module, if it has less frames than normal it asks for a better replica to the dataFinder.pl running on stol2;

- extract required informations for building the raw.ffl file from files (by using the FrDump.exe utility) and sends the to the fflGen.pl modules running on stol2;

- delete files from the oldest volume (action triggered by the dataManager.pl module);

- if the backup is configured in automatic mode, it takes care about the link creation for the new, incoming files (after the verify and compare module result);

- if the md5dbase is created, it takes care about the md5 value calculation for each single, stored file;

- send emails to the operators when there are missing frames on a file, recovered files and for all other exceptions;

• Configuration parameters and files

The following files must be present into the working directory:

this files can be manually edited:
- /var/datastor/local_host_name : usually stXXrear, is the local host name (mapped on the storage subnet);
- /var/datastor/datafind_host_name : usually stol2rear, is the name of the machine that hosts the module dataFinder.pl (used for the file revovering);
- /var/datastor/fflgen_host_name : usually stol2rear, is the name of the machine that hosts the fflGen.pl module;
- /var/datastor/dataman_host_name : usually stol1rear, is the name of the machine that hosts the module dataManager.pl;

this files can be edited due to manage an exception:
- /var/datastor/available_volumes : contains a list of volumes that are included into the dataManager.pl monitoring (polling);

- other files like .db .list and .lock files must be leaved untouched, changing or deleting them can cause system inconsistency.

All files are automatically replicated each hour by a crond script into the directory /var/dataSoft.bkp/datastor.stXX located on the datasw machine.

The following variables are located into the head of the script: to change their value the script must be stopped and restarted:

- set this variable to 1 or 2 to increment the log level detail:
# verbose level (1/2)
our $verbose = 1;

- set this services variable to 0 (disable) or 1 (enable):
# active services (0/1)
our $srv_chekscan = 1;
our $srv_createlnk = 1;
our $srv_md5check = 1;

… other variable must not be changed!

• Start the script

In order to start the script like a deamon, follow this rules:

- check if no instance are running:

# ps -edaf | grep dataStor

if the list is not empty and there are processes like dataStorage.pl:main, please wait until they will finish before restart the script like a deamon. If not, a port conflict will occour. Please also chech that dataStorage.pl:polling is not running too. If yes, please kill the process (with PID also written in the lock file) and remove the lock file /var/datastor/polling.lock. To start the script on the STXX machine (from the root account), you must:

# cd /var/datastor

# nohup /root/bin/dataStorage.pl &

then check the log or/and the nohup.out file:

# tail -f /var/log/dataStorage.pl.log

If you are starting the script after a total shutdown, remember that the correct script activation sequence is the following:

- the fflGen.pl on stol2rear;

- the dataBackup.pl on datagwrear;

- all the dataStorage.pl on stXXrear;

- the dataFinder.pl on stol2rear;

- the dataManager.pl on stol1rear;

• Stop the script

From any machine in the storage farm network simply type:

# dataCommand.pl stXXrear:9020 "stop:<reasons>"

please note that the ":" is mandatory even no <reasons> are specified, than take a look to the log file or to the nohup.out:

# tail -f /var/log/dataStorage.pl.log

Verify that the dataStorage:polling process is no more running and that the lock file is automatically removed.

• Commands recognized by the script

Like other modules, is possible to send a command direct to the dataStorage.pl module:

- force a volume to be deleted (permanently). In order to clean the entry into the ffl file related to that volume, another command must be send to do this (see fflGen.pl howto). Is not possible to delete more than one volume on same host at same time, in order to preserve disk performance, a lock will be created and the next delete commends will be ignored:

# dataCommand.pl stXXrear:9020 "delete:hostName::hostVolume"

- turn on (1) or off (0) the backup file link creation:

# dataCommand.pl stXX:9020 "makelink:0/1"

- turn on (1) or off (0) the md5sum calculation for each file:

# dataCommand.pl stXX:9020 "makemd5s:0/1"

… other commands present into the dataStorage.pl code are not intended to be execute manually.