HTCondor for Batch Analysis¶

HTCondor is the primary batch system at BNL for scaling Belle II analysis beyond interactive sessions. Instead of manually writing HTCondor submit files, we recommend using the b2af-submit tool which simplifies job submission and handles Belle II-specific configurations automatically.

Overview¶

The b2af-submit tool enables you to:

Submit Belle II analysis jobs with a simple command-line interface
Automatically handle x509 proxy authentication for grid file access
Create one job per input ROOT file for optimal parallelization
Monitor job progress with built-in tracking capabilities
Use pre-configured Belle II software environment via CVMFS

Recommended Approach

Use the b2af-submit tool instead of writing HTCondor submit files manually. It handles all the Belle II-specific setup and authentication automatically.

Installation Required

Before using b2af-submit, make sure you have installed the tools package:

# Clone the repository and install
git clone https://gitlab.desy.de/belle2/computing/computing-facilities/bnl-analysis-facility.git
cd bnl-analysis-facility
pip install -e .

HTCondor Background

For users who want to understand the underlying HTCondor system, visit the SDCC HTCondor documentation and the quick start guide.

Quick Start with b2af-submit¶

Basic Usage¶

Submit Belle II analysis jobs using the b2af-submit tool:

# Basic job submission
b2af-submit -i file_list.txt -s my_analysis_script.py --release light-2601-hyperion -o /gpfs/mnt/atlasgpfs01/belle2/users/$USER/output

# With custom resource requirements
b2af-submit -i files.txt -s analysis.py -o results/ --memory 8GB --cpus 2 --disk 10GB

# Dry run to check configuration without submitting
b2af-submit -i files.txt -s analysis.py -o results/ --dry-run

Input File Format¶

Create a text file containing the LPN paths of your ROOT files:

# file_list.txt
/belle/data/MC/release-05-02-00/DB00002172/MC15rd_TDCPV_e000000000r005699/mdst/mdst_000001.root
/belle/data/MC/release-05-02-00/DB00002172/MC15rd_TDCPV_e000000000r005699/mdst/mdst_000002.root
/belle/data/MC/release-05-02-00/DB00002172/MC15rd_TDCPV_e000000000r005699/mdst/mdst_000003.root

Automatic xrootd Prefix

Lines that do not start with root:// are automatically prepended with the BNL xrootd endpoint:

root://dcbldoor.sdcc.bnl.gov:1096/pnfs/sdcc.bnl.gov/...

This means you can use logical path names (LPNs) directly from Belle II tools like gb2_ds_list. Lines starting with root:// are used as-is. Lines starting with # are treated as comments and blank lines are skipped.

Analysis Script Requirements¶

Your analysis script should be a standard basf2 steering file. The job wrapper passes the input file to basf2 via the -i flag, so you do not need to parse sys.argv — basf2 handles it automatically:

# my_analysis_script.py
import basf2 as b2

# Create basf2 path
main_path = b2.create_path()

# Input file is provided by the job wrapper via basf2 -i flag
main_path.add_module('RootInput')

# Add your analysis modules
# ... your analysis code ...

main_path.add_module('RootOutput')

# Process events
b2.process(main_path)

Output File Naming

The job wrapper passes -o output_<job_index>.root to basf2, which overrides the RootOutput filename automatically (e.g., output_0.root, output_1.root). You do not need to set outputFileName in your script.

b2af-submit Features¶

Automatic Configuration¶

The b2af-submit tool handles many Belle II-specific configurations automatically:

X509 Proxy Management: Automatically discovers and validates your grid proxy (searches /tmp/x509up_* and $X509_USER_PROXY)
CVMFS Integration: Sources Belle II environment via /cvmfs/belle.cern.ch/tools/b2setup (configurable release, default: light-2503-ceres)
Job Parallelization: Creates one HTCondor job per input file for optimal resource utilization
Output Organization: Structured output directory with job logs, submit files, and per-job output

What Happens Inside Each Job¶

Each job runs inside the b2af_job_wrapper.sh wrapper script, which:

Sources the Belle II environment via CVMFS (configurable with --release, default: light-2503-ceres)
Runs your script with basf2 your_script.py -i <input_file> -o output_<job_index>.root -- <b2opt args>
Logs timestamps and worker node information for debugging

Any arguments provided via --b2opt are appended after -- in the basf2 command, allowing you to pass standard basf2 options such as -n (number of events) or --skip-events.

Output Directory Structure¶

After submission, b2af-submit creates the following structure:

output_dir/
├── logs/                          # HTCondor job logs
│   ├── b2af_analysis_UUID_0000.out
│   ├── b2af_analysis_UUID_0000.err
│   └── b2af_analysis_UUID_0000.log
├── submit_files/                  # Generated submit files
│   ├── b2af_analysis_UUID_0000.sub
│   └── b2af_job_wrapper.sh
├── output_0.root                  # Output from job 0
├── output_1.root                  # Output from job 1
├── ...
└── job_info_UUID.txt              # Summary: cluster IDs, submit time, paths

The job_info_UUID.txt file contains cluster IDs, timestamps, and submission parameters — useful for tracking and debugging.

Complete Option Reference¶

Option	Default	Description
`-i, --input-list`	required	Text file with input ROOT file paths (one per line)
`-s, --script`	required	Basf2 steering script to run on each file
`-o, --output-dir`	required	Output directory for results and logs
`--proxy-path`	auto-detect	Path to x509 proxy (searches `/tmp/x509up_*` and `$X509_USER_PROXY`)
`--job-name`	`b2af_analysis`	Base name prefix for HTCondor jobs
`--memory`	`4GB`	Memory per job
`--cpus`	`1`	CPU cores per job
`--disk`	`5GB`	Disk space per job
`--max-runtime`	`24h`	Maximum runtime (supports `h`, `m`, `s` suffixes)
`--release`	`light-2503-ceres`	Belle II software release to use
`--b2opt`	none	Options passed to basf2 after `--` (repeatable, e.g. `--b2opt '-n 100'`)
`--transfer-files`	none	Additional files to transfer (repeatable)
`--environment`	none	Environment variables as `VAR=value` (repeatable)
`--dry-run`	off	Generate submit files without submitting
`--monitor`	off	Poll job status after submission (~30s intervals)
`-v, --verbose`	off	Enable detailed output for debugging

Advanced Options¶

# Monitor jobs with built-in tracking
b2af-submit -i files.txt -s analysis.py -o results/ --monitor

# Specify custom proxy file location
b2af-submit -i files.txt -s analysis.py -o results/ --proxy-path ~/.globus/custom_proxy

# Transfer additional files to jobs (repeat flag for each file)
b2af-submit -i files.txt -s analysis.py -o results/ --transfer-files config.yaml --transfer-files helper.py

# Set environment variables (repeat flag for each variable)
b2af-submit -i files.txt -s analysis.py -o results/ --environment BELLE2_OPTION=value

# Use a specific Belle II software release
b2af-submit -i files.txt -s analysis.py -o results/ --release light-2411-bengal

# Pass options to basf2 (e.g., limit to 100 events per job)
b2af-submit -i files.txt -s analysis.py -o results/ --b2opt '-n 100'

# Pass multiple basf2 options (repeat the flag for each option)
b2af-submit -i files.txt -s analysis.py -o results/ --b2opt '-n 100' --b2opt '--skip-events 50'

# Custom job naming and runtime limits
b2af-submit -i files.txt -s analysis.py -o results/ --job-name my_analysis --max-runtime 12h

# Verbose output for debugging submission issues
b2af-submit -i files.txt -s analysis.py -o results/ --dry-run -v

Prerequisites¶

Before using b2af-submit, ensure you have:

Valid x509 proxy certificate (voms-proxy-info to check)
Access to CVMFS Belle II software stack
HTCondor client tools installed
Analysis script that accepts ROOT file path as first argument

Job Management¶

Monitor Jobs¶

# Check job status
condor_q

# Check specific user jobs
condor_q username

# Detailed job information
condor_q -analyze job_id

# Check completed jobs
condor_history username

Job Control¶

# Remove specific job
condor_rm job_id

# Remove all your jobs
condor_rm username

# Hold/release jobs
condor_hold job_id
condor_release job_id

Best Practices¶

Testing and Development¶

Test interactively first: Run your analysis script on a single file interactively before submitting jobs
Use dry run: Always use --dry-run to check configuration before actual submission
Start small: Begin with a small file list to test your workflow

# Test your script interactively first
python my_analysis.py /path/to/single/file.root

# Then use dry run to check b2af-submit configuration
b2af-submit -i small_file_list.txt -s my_analysis.py -o test_output/ --dry-run

Resource Management¶

Estimate resources: Monitor resource usage from test jobs to set appropriate limits
Right-size requests: Avoid over-requesting memory/CPU to improve scheduling
Monitor efficiency: Use condor_q -better-analyze to check job efficiency

# Start with conservative resource requests
b2af-submit -i files.txt -s analysis.py -o results/ --memory 4GB --cpus 1

# Scale up if needed based on monitoring
b2af-submit -i files.txt -s analysis.py -o results/ --memory 8GB --cpus 2

Error Handling and Debugging¶

Check proxy validity: Ensure x509 proxy is valid before submission
Review log files: Check output and error logs in the job directory
Use job analysis: Use HTCondor tools to diagnose job issues

# Check proxy before submitting
voms-proxy-info

# Check for held jobs and their reasons
condor_q -held
condor_q -analyze <job_id>

Common Analysis Patterns¶

Processing Large Datasets¶

For large datasets, create file lists and use b2af-submit for parallel processing:

# Create file list from dirac/gb2 tools
gb2_ds_list /belle/MC/release-XX-XX-XX/DB123456/MC15rd/* > large_dataset.txt

# Submit jobs with appropriate resources
b2af-submit -i large_dataset.txt -s my_analysis.py -o /gpfs/mnt/atlasgpfs01/belle2/users/$USER/results/ \
  --memory 6GB --disk 8GB --job-name large_analysis

Monte Carlo Studies¶

For MC studies requiring different configurations:

# Create separate file lists for different MC samples
ls /belle/MC/release-XX-XX-XX/signal_sample/*.root > signal_files.txt
ls /belle/MC/release-XX-XX-XX/background_sample/*.root > background_files.txt

# Submit separate jobs for each sample type
b2af-submit -i signal_files.txt -s mc_analysis.py -o results/signal/ --job-name signal_mc
b2af-submit -i background_files.txt -s mc_analysis.py -o results/background/ --job-name bkg_mc

Systematic Studies¶

For analyses requiring multiple systematic variations:

# Use environment variables to pass systematic parameters
b2af-submit -i nominal_files.txt -s systematic_analysis.py -o results/nominal/ \
  --environment "SYSTEMATIC=nominal" --job-name sys_nominal

b2af-submit -i nominal_files.txt -s systematic_analysis.py -o results/variation1/ \
  --environment "SYSTEMATIC=variation1" --job-name sys_var1

Getting Help¶

b2af-submit Help¶

# Get help for b2af-submit command
b2af-submit --help

# Check tool version and status
b2af-submit --version

Troubleshooting¶

Common issues and solutions:

Issue	Solution
"Proxy not found"	Check `voms-proxy-info` and generate new proxy if expired
Jobs on hold	Use `condor_q -held` and `condor_q -analyze <job_id>` to diagnose
Permission denied	Verify output directory exists and has write permissions
Script not found	Ensure analysis script path is correct and accessible

Additional Resources¶

b2af-submit documentation: See tools/README.md in the repository
BNL HTCondor support: Contact SDCC support for HTCondor-specific issues
Belle II software: Visit Belle II software documentation
Job debugging: Use condor_q -better-analyze for detailed HTCondor diagnostics