How to guide for BNL AF nodes

You can find the main SDCC page with more details

Interactive Connection to BNL

ssh Connection to the interactive nodes

TODO: do we have attsub nodes for FCC?

At BNL, you need login to ssh.sdcc.bnl.gov first, then login to interactive machines attsub01, ..., attsub08 with command ssh or "rterm -i" without argument (please refer to the following section for the rterm usage help ). You can add the following to file $HOME/.ssh/config on your laptop or local machine, so you can login attsub* machines directly.

host atlasgw
     hostname ssh.sdcc.bnl.gov
     user yourNameAtBNL

host attsub*
   user yourNameAtBNL
   proxycommand ssh atlasgw nc %h %p

Replace "yourNameAtBNL" with your own username at BNL. So you should be able to run the following to login any machine such as attsub01 directly,

ssh attsub01

Please note that: the gateway was just changed to ssh.sdcc.bnl.gov.

These nodes should be used for debugging and testing code. To run your complete analysis code you should take advantage of the batch system.

Setup FCC software environment

Once you are on the interactive nodes, you can simply run:

source /cvmfs/sw.hsf.org/key4hep/setup.sh

and follow the tutorials avalible at FCC tutorials

Disk space

Disk space is provided at

/gpfs/mnt/gpfs01/usfcc

Please create a folder using your username and store your data there.

Currently, there is no quota limit on user storage. Usage is based on an honor system, so users are expected to store data appropriate to their specific analysis needs. This flexible policy is intentional and aims to encourage users to explore a wide range of workflows without restrictions.

However, users with excessive or unreasonable usage will be contacted.

Condor batch system at BNL

RACF uses the Condor batch submission system. Detailed information about using this system (submitting jobs, killing jobs, etc.) is described here:
https://www.sdcc.bnl.gov/information/services/htcondor-sdcc. You can also get the HTCondor Quick Start Guide.

Since the upgrade of dCache in January 2019, xrootd access to dCache file would require a valid grid proxy. You need specify x509userproxy in the job description file with the following line:

x509userproxy = $ENV(X509_USER_PROXY)

after you have generated a valid grid proxy in an interactive machine.

Please notice that the envvar X509_USER_PROXY has already been defined to $HOME/x509_u$UID upon login. So actually you can also just pass all the env including the evnvar X509_USER_PROXY to the condor jobs without specifying x509userproxy, with the following line in the job description file:

GetEnv          = True

Notice on using bash script in Condor

Please be aware that aliases will not be expanded by default within script. To expand those aliases (such as asetup, athena), you need add the following line to your script prior to running setupATLAS:

shopt -s expand_aliases

TODO: Add information about the group here and how to add to the condor submission job?

Instruction to submit on Condor batch system at BNL

To submit jobs on the FCC AF using condor, first step is to request being added to the account group. Open a ticket emailing rt-racf-useraccounts@bnl.gov, and request your BNL username to be added to the HFCC LDAP (group_usfcc) group.

Once this is completed, you can submit jobs from the attsub* nodes using the typical condor workflow, and adding in the following lines to the condor script

accounting_group = group_usfcc