Data transfer policy

All Hoffman2 Cluster nodes provide access to network-connected home, scratch, and project directories, and to local scratch space.

Data ingress and egress are supported by numerous data transfer utilities.

Dedicated data transfer (“dtn”) nodes are available for bulk, high-speed transfers, and caching and non-caching proxy servers support limited connectivity (e.g., HTTPS and CVMFS) for retrieving runtime data on compute nodes.

The use of external file or block storage (e.g., NFS or CIFS) is unsupported on the Hoffman2 Cluster for reasons of performance, data integrity, and data security.

It is recommended that data are transferred to and from the Hoffman2 Cluster using one or more of the following methods:

  • Using supported data transfer utilities on a dtn node

  • Utilizing a caching proxy server when using the HTTPS protocol to download identical data to multiple compute nodes

  • Downloading small, unique remote data sets directly to compute nodes using HTTPS protocol transfers

Staging data, to storage systems connected the Hoffman2 Cluster, prior to submitting compute jobs will typically result in the best performance due to reduced storage latency and increased network bandwidth.

All data transfers to or from the Hoffman2 Cluster must be encrypted end-to-end for data, metadata, and protocol data. Unencrypted data transfers are prohibited.

Data processed on the Hoffman2 Cluster must comply with the Hoffman2 Cluster Data use and Security policies and are governed by the Hoffman2 Cluster Data access and retention policy.