Skip to content

FCC AF Onboarding Checklist

This page is the ordered set of steps for a new user of the FCC Analysis Facility at BNL. Work through it top to bottom. Steps 1–3 involve BNL staff and can take a few business days, so start them as early as you can.

Step 0: Choose your account type

Two account types exist, and they grant different access:

Federated Identity account Full SCDF account
JupyterHub access Yes Yes
SSH to interactive nodes No Yes
Direct HTCondor submission No Yes
Batch compute Indirectly — HTC Jupyter sessions run as HTCondor jobs Yes, directly
Typical time to obtain 1–2 business days Longer — requires Guest/Life number and training
  • If you only need JupyterLab (notebooks, PyROOT, ML packages, a terminal console inside Jupyter), a Federated Identity account is enough and is much faster to obtain.
  • If you need SSH to the interactive nodes or to submit HTCondor jobs directly, you need a full SCDF account.

You can start with a Federated Identity account and apply for a full SCDF account later.

Step 1: Apply for your account

Select FCC on the application form

Whichever account type you apply for, select FCC as your experiment on the form. That is what adds you to the group_usfcc accounting group, which your batch jobs and HTC Jupyter sessions run against. There is no separate request to make afterwards.

Federated Identity account

  1. Obtain an ORCID if you do not have one — it is required on the registration form.
  2. Make sure MFA is enabled at your home institution (CERN, Fermilab or SLAC).
  3. Register at https://federated.sdcc.bnl.gov/, selecting FCC as your experiment.
  4. You will need the name of the designated point of contact as a verification step. Request it from sabidi@bnl.gov before you start the form.

Full instructions with screenshots: BNL Federated ID Account.

Full SCDF account

  1. Register for a Guest Number at the BNL GUV center if you do not already have a valid Life or Guest Number. Expect to upload photo identification; non-citizens also upload a CV.
  2. Complete CyberSecurity training and the Computer Use agreement.
  3. Sign up at BNL New User Account, selecting FCC as your experiment.

Full instructions: Apply for computer accounts.

Step 2: Set up MFA

BNL requires multi-factor authentication.

  • SCDF accounts: follow the Unified MFA procedure.
  • Federated accounts: MFA is handled by your home institution (CERN, Fermilab, SLAC). Confirm it is enabled there.

Step 3: Set up your SSH key (full SCDF accounts only)

You cannot log in with a password. The SCDF gateways require an SSH key pair, and your public key must be uploaded to the facility before your first login. Skip this step and you will be refused at the gateway.

  1. Generate an SSH key pair
  2. Upload your public key to the facility
  3. Set up ssh-agent so your key follows you through the gateway to the interactive nodes

Federated Identity users can skip this — JupyterHub uses your institutional login, not SSH keys.

Step 4: First login

Via JupyterHub (works for both account types)

Open https://jupyter-fedid.sdcc.bnl.gov/ and sign in. Choose the HTC JupyterHub for FCC work. See Jupyter at the FCC AF for the available kernels and extensions.

Via SSH (full SCDF accounts only)

ssh yourNameAtBNL@ssh.sdcc.bnl.gov
ssh spoolsub01

The FCC interactive nodes are spoolsub01 and spoolsub02.

To reach them in one step, add to $HOME/.ssh/config on your laptop:

Host spoolsub*
    User yourNameAtBNL
    ProxyCommand ssh yourNameAtBNL@ssh.sdcc.bnl.gov -W %h:%p

Then ssh spoolsub01 connects directly. Details: SSH access.

Step 5: Set up the FCC software environment

On an interactive node or in a Jupyter terminal:

source /cvmfs/sw.hsf.org/key4hep/setup.sh

This gives you the Key4hep stack, which includes the FCC software. Tutorials are at FCC tutorials.

Step 6: Create your work directory

mkdir -p /gpfs/mnt/gpfs01/usfcc/$USER
cd /gpfs/mnt/gpfs01/usfcc/$USER

There is no hard quota — storage runs on an honor system. Roughly 100 TB of common space is also available for sharing datasets between users.

Step 7: Run your first batch job

Create a job description file, for example first.job:

universe                = vanilla
executable              = first.sh
accounting_group        = group_usfcc
request_memory          = 2000
GetEnv                  = True
output                  = first.$(Cluster).out
error                   = first.$(Cluster).err
log                     = first.$(Cluster).log
queue 1

accounting_group = group_usfcc is required. Your membership of the group is automatic (you selected FCC when you applied), but jobs are not tagged with it automatically — you must set this line yourself in every job description file. Submit and monitor:

condor_submit first.job
condor_q

To request a GPU, add:

+want_gpu               = True
request_GPUs            = 1

Jobs have a maximum runtime of 3 days and may be evicted if they exceed it or their requested resources. More detail: Condor batch system.

Step 8: Know where to get help

What you need Where to go
Account creation and access problems rt-racf-useraccounts@bnl.gov
Questions, discussion, announcements AF Discourse Forum — "BNL FCC analysis facility" section
Facility requests and feedback sabidi@bnl.gov

Where to go next