How to guide for BNL AF nodes¶

You can find the main SCDF page with more details

Interactive Connection to BNL¶

ssh Connection to the interactive nodes¶

SSH access requires a full SCDF account. If you only hold a Federated Identity account, you cannot use SSH — use JupyterHub instead. See Apply for computer accounts for the difference.

Set up your SSH key first

The SCDF gateways do not accept password logins. You must generate an SSH key pair and upload your public key before your first login, and set up ssh-agent so the key follows your connection through the gateway to the interactive nodes.

At BNL you first log in to the gateway ssh.sdcc.bnl.gov, then on to one of the FCC interactive machines, spoolsub01 or spoolsub02:

ssh yourNameAtBNL@ssh.sdcc.bnl.gov
ssh spoolsub01

To reach the interactive machines in a single step, add the following to $HOME/.ssh/config on your laptop or local machine:

Host spoolsub*
    User yourNameAtBNL
    ProxyCommand ssh yourNameAtBNL@ssh.sdcc.bnl.gov -W %h:%p

Replace yourNameAtBNL with your own BNL username. You can then log in directly:

ssh spoolsub01

These nodes are for interactive work: debugging, testing and short jobs. Run your full analysis through the batch system.

Setup FCC software environment¶

Once you are on the interactive nodes, you can simply run:

source /cvmfs/sw.hsf.org/key4hep/setup.sh

and follow the tutorials avalible at FCC tutorials

Disk space¶

Disk space is provided at

/gpfs/mnt/gpfs01/usfcc

Please create a folder using your username and store your data there.

Currently, there is no quota limit on user storage. Usage is based on an honor system, so users are expected to store data appropriate to their specific analysis needs. This flexible policy is intentional and aims to encourage users to explore a wide range of workflows without restrictions.

However, users with excessive or unreasonable usage will be contacted.

In addition to per-user directories, roughly 100 TB of common disk space is available for sharing datasets between users of the facility.

Condor batch system at BNL¶

The SCDF uses the HTCondor batch submission system. Detailed information about using this system (submitting jobs, killing jobs, etc.) is described here:
https://docs.sdcc.bnl.gov/services/computing/htcondor/. You can also get the HTCondor Quick Start Guide.

Since the upgrade of dCache in January 2019, xrootd access to dCache file would require a valid grid proxy. You need specify x509userproxy in the job description file with the following line:

x509userproxy = $ENV(X509_USER_PROXY)

after you have generated a valid grid proxy in an interactive machine.

Please notice that the envvar X509_USER_PROXY has already been defined to $HOME/x509_u$UID upon login. So actually you can also just pass all the env including the evnvar X509_USER_PROXY to the condor jobs without specifying x509userproxy, with the following line in the job description file:

GetEnv          = True

Notice on using bash script in Condor¶

Please be aware that aliases will not be expanded by default within script. To expand those aliases, you need add the following line to your script prior to running k4run, fccanalysis, etc:

shopt -s expand_aliases

Instruction to submit on Condor batch system at BNL¶

The FCC accounting group¶

FCC batch work runs against the accounting group group_usfcc. You were added to this group automatically when you selected FCC as your experiment on your account application — there is nothing to request.

What is not automatic is tagging your jobs. Every job description file must set accounting_group yourself, or the job will not run against the FCC allocation.

Submitting from an interactive node¶

Submit jobs from spoolsub01 or spoolsub02 using the usual HTCondor workflow. A minimal FCC job description file looks like:

universe                = vanilla
executable              = my_analysis.sh
accounting_group        = group_usfcc
request_memory          = 2000
GetEnv                  = True
output                  = job.$(Cluster).$(Process).out
error                   = job.$(Cluster).$(Process).err
log                     = job.$(Cluster).log
queue 1

accounting_group = group_usfcc is required — jobs submitted without it will not run against the FCC allocation.

Submit and monitor with:

condor_submit my_analysis.job
condor_q
condor_rm <cluster-id>

Requesting GPUs¶

The facility provides NVidia L40S GPUs. To request one, add to your job description file:

+want_gpu               = True
request_GPUs            = 1

Job limits¶

Batch jobs have a maximum runtime of 3 days. Jobs that exceed this, or that exceed their requested resources, may be evicted automatically. Split long workloads into shorter jobs, and checkpoint intermediate results to your directory under /gpfs/mnt/gpfs01/usfcc so an eviction does not lose work.