From Duncan MacLeod:

I hope what follows are fairly straightforward instructions on how to deploy the igwn-accounting condor tool.

## Installing the tool

The official collector tool is called ‘igwn-accounting’ and is available from PyPI, as well as the LIGO (LSCSoft) YUM and Debian repositories. My understanding is that at least at Cascina you run a RHEL7 derivative, so using the RPM is probably easiest.

RPM option 1 --→ we choose this one

· Configure the LSCSoft RPM repo, using instructions here: https://computing.docs.ligo.org/guide/software/installation/#sl7

· Install the `igwn-accounting` package

RPM option 2

· Manually download and install the relevant RPMs from https://software.igwn.org/lscsoft/scientific/7/x86_64/production:

o python36-igwn-accounting-1.2.1

o igwn-accounting-1.2.1

Alternative option 3:

· `pip` install the package directly from pypi.python.org

For the record, the source code is here: https://git.ligo.org/computing/accounting/collection-client. The code is Python compatible with Python >=3.5 and requires htcondor (including the python bindings), and python-dateutil.

If you manually install, please ensure to install at least version 1.2.1.

## Running the tool

Once installed, the tool should be configured via ‘cron’ (or similar) to run once per day using the following invocation:

/usr/bin/igwn-accounting-report --cluster EGO-Cascina --utc $(date --utc +\%F --date='1 day ago') --output-file /var/www/html/accounting/EGO-Cascina-$(date --utc +\%F --date='1 day ago').txt

Notes:

· The `date …` command just formats ‘yesterday’ as YYYY-MM-DD

· The ‘EGO-Cascina’ string is just a standard name for the site, it can probably be whatever you want as long as it has no whitespace

This will write a file that includes only lines of the following format:

USERNAME ACCOUNTING_TAG CPU_HOURS YYYY-MM-DD CLUSTER

e.g.

duncan.macleod igwn.dev.o4.computing.accounting 12345 2021-01-01 EGO-Cascina

These files then need to be made available (currently) over HTTP(S) with a URL that is readable (at least) from the Caltech LDAS system. How you make these available within your site’s security policy is none of my business…

## What the tool actually does

In essence the tool just calls out to the condor_history executable with a call of the following form:

/usr/bin/condor_history -constraint 'JobUniverse = 7 && Owner = "igwn-pilot" && JobFinishedHookDone >= 1615766400.0 && JobFinishedHookDone < 1615852800.0' -jsonl -af LigoSearchUser LigoSearchTag CompletionDate CpusProvisioned CumulativeSuspensionTime EnteredCurrentStatus MATCH_GLIDEIN_Site Owner RemoteWallClockTime RequestCpus

where 1615766400 and 1615852800 are the Unix epochs for the start and end of the query interval (here, March 15th 2021). If you are running condor 8.8.x (as opposed to 8.9.x) the call will be subtly different (condor 8.8.x doesn’t support `-jsonl`) but will do the same job.

The tool then aggregates jobs across user and tag so there is only one entry in the output per (user, job, date) key.

on submit1:

needed expand life time of history files:
#>/etc/condor/condor/condor_config\
HISTORY         = <b>/var/lib/condor/spool/history
</b>ENABLE_HISTORY_ROTATION = True
MAX_HISTORY_LOG = 104857600 (100MB)
MAX_HISTORY_ROTATIONS = 5
#>cat /etc/crontab
50 23  * * * root /usr/bin/python3 -m igwn_accounting.report --cluster EGO-Cascina --utc $(date --utc +\%F --date='1 day ago') --output-file /var/www/html/accounting/EGO-Cascina-$(date --utc +\%F --date='1 day ago').txt
55 23  * * * root rsync -var --progress /var/www/html/accounting/* 193.205.74.251::history

on igwnacct-ego.virgo-gw.eu

Web Server configuration
#>cat /etc/httpd/conf.d/igwacc-ego.conf

<VirtualHost *:80>
        ServerName igwnacct-ego.virgo-gw.eu:80
        Redirect  /   https://igwnacct-ego.virgo-gw.eu/
</VirtualHost>

<VirtualHost *:443>
    ServerName igwnacct-ego.virgo-gw.eu:443
    ServerAlias igwnacct-ego.virgo-gw.eu:443
    DocumentRoot /var/www/html/
    SSLEngine on
    SSLCertificateChainFile /etc/httpd/conf/ssl.crt/sectigo-root-plus-geant-intermediate.pem
    SSLCertificateFile /etc/httpd/conf/ssl.crt/igwnacct-ego_virgo-gw_eu_cert.pem
    SSLCertificateKeyFile /etc/httpd/conf/ssl.crt/igwnacct-ego_virgo-gw_eu.key
#Ligo ssl client Authentication
  SSLVerifyClient none
  # ho unito i due certificati che hai mandato in CA_cert.pem
  #SSLCACertificateFile /etc/httpd/conf/ssl.crt/CA_cert.pem
  SSLCACertificateFile /etc/httpd/conf/ssl.crt/combined_CA.pem
  <Location /history>
        SSLVerifyClient require
        SSLVerifyDepth 10
        #SSOptions +StdEnvVars
        SSLRequire %{SSL_CLIENT_S_DN_CN} eq "Giuseppe Di Biase" or  %{SSL_CLIENT_S_DN_CN} eq "Stefano Cortese"  or %{SSL_CLIENT_S_DN_CN} eq "Stefano Cortese stefano.cortese@ligo.org" or %{SSL_CLIENT_S_DN_CN} eq "Duncan Macleod duncan.macleod@ligo.org" or  %{SSL_CLIENT_I_DN_CN} eq "ldas-grid.ligo.caltech.edu"
  </Location>
</VirtualHost>

RSYNC configuration

[root@igwnacct-ego ~]# cat /etc/rsyncd.conf 
# /etc/rsyncd: configuration file for rsync daemon mode

motd file = /etc/motd
max connections = 3

[history]
        comment = used by igwnacc-ego for accounting population
        path = /var/www/html/history
        read only = no
        list = yes
        hosts allow = 90.147.139.39 127.0.0.0/8
        uid = root
        gid = root

Tests

test WEB;
To test the X.509 authenticated access:
- created a proxy as usual with "ligo-proxy-init giuseppe.dibiase"
- create a .p12 bag from the pem credentials with: openssl pkcs12 -export -in /tmp/x509up_u1047 -out giuseppe_dibiase_cilogon_mics.p12 (in your case you must change the 1047 number with your unix uid on the system where you issued ligo-proxy-init)
- import the .p12 certificate in your browser (it will be valid for 11 days) 

Promemoria

Questo pull non funzionava almeno dal 19 aprile 2022, quando avevo abilitato l'https inspection per igwnacct-ego.virgo-gw.eu non ricordandomi che l'autenticazione e' basata su SSL client authentication.Ora ho disabilitato l'https inspection e messo una ACL per i client coinvolti (ldas-grid.ligo.caltech.edu, accounting-dev.ligo.org e per i test lscgw.virgo.infn.it).
esempio:
curl -L https://igwnacct-ego.virgo-gw.eu/history/EGO-Cascina-2022-09-11.txt --cert /tmp/x509up_u1280 -v