Globus is a software tool to transfer files across the web in a reliable, high-performance and secure way. It provides fault-tolerant, fire-and-forget data transfer using simple web or command line interfaces. It is approriate for transferring very large files either between your desktop machine and a remote machine like the Hoffman2 Cluster, or between two remote machines on which you have accounts; both remote machines need to be part of the Globus project. All XSEDE resources are configured as Globus endpoints.
See High-speed file transfer with Globus for information on using Globus.
For security reasons, Hoffman2 Cluster allows file transfer only with scp or sftp or grid-ftp. For the same reason, you should use an scp or sftp client on your local machine. You should not use the scp command on the cluster.
The scp and sftp commands transfer files using the secure shell protocol (ssh) in which data is encrypted during transfer. The use of scp requires that an scp client be run on the machine that you use to initiate the transfer and that it communicate with a server run on any other machines which participate in the transfer. The Hoffman2 Cluster, like most Linux and Unix systems, runs both a client and a server.
There is an scp client command on desktop Linux/Unix systems and on Macs (use Terminal). On Windows, you usually have to install an ssh client which comes with an scp program.
The syntax of the Linux/Unix scp command is very similar to the cp command. For complete scp syntax, enter:
Here is a simplified scp syntax that accomplishes most transfers:
scp [-r] source target
where source is the name of the file on your local machine, and target will be the name of the file on the cluster.
For the source on your local machine, specify an absolute or relative file name or directory name. You can use wild cards to transfer multiple files to an existing target directory. Specify -r to transfer a whole source directory and its files.
For the target on the cluster, specify your login_id and the Hoffman2 address, followed by a colon (:), followed by the file specification. You can specify the directory where the file is to be saved, or a dot “.” meaning the same name in your home directory, or an absolute or relative path including a new file name. For large files or large amounts of data, use the Hoffman2 data transfer node dtn2.hoffman2.idre.ucla.edu
scp myfile email@example.com:.
will transfer the file named myfile from your current directory on your local machine to your home directory on the Hoffman2 Cluster. Its name on the cluster will be $HOME/myfile
The rsync command uses the SSH2 protocol to efficiently transfer files. It is perhaps most useful in keeping groups of files on different computers up to date with each other. For large files or large amounts of data, use the Hoffman2 data transfer node dtn2.hoffman2.idre.ucla.edu
Here is a 2-part example of discovering the status of files in a common directory named mydir. It is comparing files in your Hoffman2 $HOME/mydir directory with those on your local machine mydir directory. You need both parts to ensure any new files from either source are synchronized.
Part 1: Run this on your local machine:
rsync -an --itemize-changes firstname.lastname@example.org:mydir .
Any files prefixed with > in the ouput are different on Hoffman2 and you may want to download them from Hoffman2 (get):
rsync -av email@example.com:mydir .
Part 2: Run this on your local machine:
rsync -an --itemize-changes mydir firstname.lastname@example.org:
Any files prefixed with < in the output are different on your local machine and you may want to upload them to Hoffman2 (put):
rsync -av mydir email@example.com:
For more information about the rsync command and additional options, enter
at the shell prompt.
You can use the UCLA Grid Portal to:
- upload files from your local machine to the cluster.
- download files and job output from the cluster to your local machine.
- transfer files between the cluster and other clusters at UCLA.
The grid portal makes Hoffman2 Cluster accessible from any web browser.
You can use the UC Grid Portal to:
- transfer files between the cluster and clusters at other campuses of the University of California.
There is a file size limit of 1GB for files uploaded through the grid portals.