HPSS Archiving Tools

RHIC users can put archive data into HPSS using one of the following native HPSS archive client tools. hsi & htar outlined below require a valid Kerberos TGT for authentication purposes. Instructions on how to get a Kerberos TGT can be found on the Overview of Kerberos Services web page.

Users should note that the HPSS system is specifically designed to handle large files (> 2 GB) files; small files will result in extremely poor HPSS performance. For reference, small file performance of HPSS is measured at a FEW FILES PER MINUTE per tape drive. In contrast, for large files, HPSS performance is measured in excess of 50MB per second per tape drive. Since HPSS is a shared facility that handles high bandwidth data streams from the RHIC and LHC accelerators, in addition to archived user files, users are strongly encouraged to aggregate small files into larger files using tools like htar before archiving files. Files less than 2 GB in size may be rejected and deleted. Users who repeatedly inject small files may have their account disabled and access to HPSS revoked. It is recommended to inject files 10 GB in size or larger for optimal performance.

Tape access is for archival use only. Users should not inject files that require frequent read/write activity.

PFTP

PFTP (/usr/local/bin/pftp) is a FTP-like program that can be used to put data into HPSS, take data out of HPSS, and manipulate the HPSS file system namespace. PFTP supports transfers via TCP/IP and communicates directly with HPSS Movers to transfer data. Documentation for the native HPSS PFTP command can be found here.

HSI

HSI (/usr/local/bin/hsi) is an interactive shell interface to HPSS. Since it is a relatively heavyweight application, this interface should be used sparingly. DO NOT use HSI in any scripts and avoid any operations that may involve many file/file system meta data lookups. Documentation for HSI can be found here.

HTAR

HTAR (/usr/local/bin/htar) is a tar-like program that aggregates multiple files in the local file system into a single file that resides in HPSS. If the data you wish to upload contains many small files, use this utility to aggregate the upload into a single file and avoid rejection due to the 2 GB file size minimum. Documentation for HTAR can be found here. See the documentation for examples.