************* Data transfer ************* .. note:: Personal information and other sensitive data, including statutory, regulatory, and contractually protected data — for example, human subjects research, restricted research, student and educational data, and personal health information (PHI) — are **prohibited** on the Hoffman2 Cluster. .. include:: Data-transfer/DT-nodes.inc Tools ===== There are several methods to transfer data between a local computer and the Hoffman2 Cluster and we will cover some on this page. Depending on your network connection and the amount of data to transfer between your local computer and the Hoffman2 Cluster, you have options under graphical or command-line interface utilities. .. tip:: To move large files and/or large datasets, we recommend using other parallel transfer tools, e.g. :ref:`Globus`, rclone_. Graphical utilities ------------------- Many of the graphical utilities support a file manager with "drag and drop" functionality between your local and remote computers. You may use whichever tool you prefer... .. csv-table:: Graphical Transfer Utilities :header: "Application","Website", "Transfer Protocols", "Platform" :widths: 30, 40, 40, 10 "CyberDuck", "``_", "SFTP", "Windows, macOS" "FileZilla", "``_", "SFTP", "Windows, macOS, Linux" ":ref:`MobaXterm`", "``_ ", "SFTP", "Windows" "WinSCP", "``_", "SFTP", "Windows" ":ref:`Globus`", "``_", "`GridFTP and UDT `_", "Windows, macOS, Linux" Command-line utilities ---------------------- .. csv-table:: Command-line interface Transfer Utilities :header: "Application", "Platform", "Features" :widths: 20, 20, 20 "`scp`_", "macOS, Linux, Windows [#FN4]_ [#FN5]_ ", "secure copy" "`sftp`_", "macOS, Linux, Windows [#FN4]_ [#FN5]_ ", "secure file transfer" "`rsync`_", "macOS and Linux", "sync files to and from" "curl", "macOS and Linux", "xfr data using various protocols" "wget", "macOS and Linux", "download via HTTP/HTTPS" "`rclone`_", "Windows, macOS, Linux", "rsync for cloud storage" .. rubric:: Footnotes .. [#FN4] Git for Windows provides a BASH emulation with SSH/SCP/SFTP clients. `Git for Windows `_ .. [#FN5] `Windows Subsystem for Linux (WSL). WSL is supported on Windows 10 64-bit (Build 16215 or later) `_ Cloud storage services ====================== .. important:: If you have data use questions, please contact UCLA IT Services client support, via email - clientsupport@it.ucla.edu Faculty and staff use of cloud storage services must comply with applicable University policies, notably policies relating to the protection of University data and the `University of California Electronic Communications Policy `_. This includes the data use requirements, which are based on University-negotiated agreements established to help safeguard information about individuals and other confidential information for which the campus is a steward. Always employ due care when processing, transmitting, or storing sensitive information. Violation of these data use requirements or other campus policies may result in disciplinary action up to and including termination. .. include:: Data-transfer/box.inc .. include:: Data-transfer/google-drive.inc .. include:: Data-transfer/globus.inc .. include:: Data-transfer/rclone.inc scp === .. sidebar:: Attention Windows Users Please look under :ref:`Tools` for a list of SCP options For security reasons, the Hoffman2 Cluster allows file transfer only with scp or sftp or grid-ftp. For the same reason, you should use an scp or sftp client on your local machine. You should not use the scp command on the cluster. The :program:`scp` and :program:`sftp` commands transfer files using the secure shell protocol (ssh) in which data is encrypted during transfer. The use of scp requires that an scp client be run on the machine that you use to initiate the transfer and that it communicate with a server run on any other machines which participate in the transfer. The Hoffman2 Cluster, like most Linux and Unix systems, runs both a client and a server. There is an scp client command on desktop Linux/Unix systems and on macOS (use Terminal). On Windows, you usually have to install an ssh client which comes with an scp program. The syntax of the Linux/Unix scp command is very similar to the cp command. For complete scp syntax, enter: :: man scp Here is a simplified scp syntax that accomplishes most transfers: :: scp [-r] source target where source is the name of the file on your local machine, and target will be the name of the file on the cluster. For the source on your local machine, specify an absolute or relative file name or directory name. You can use wild cards to transfer multiple files to an existing target directory. Specify -r to transfer a whole source directory and its files. For the target on the cluster, specify your login_id and the Hoffman2 Cluster address, followed by a colon (:), followed by the file specification. You can specify the directory where the file is to be saved, or a dot “.” meaning the same name in your home directory, or an absolute or relative path including a new file name. For large files or large amounts of data, use the Hoffman2 Cluster data transfer nodes (dtn.hoffman2.idre.ucla.edu). You can connect directly to dtn1.hoffman2.idre.ucla.edu or dtn2.hoffman2.idre.ucla.edu or use the Domain Name Service (DNS) round-robin address, dtn.hoffman2.idre.ucla.edu, which load balances requests between the two servers. :: login_id@dtn2.hoffman2.idre.ucla.edu:filespec For example: :: scp myfile login_id@dtn2.hoffman2.idre.ucla.edu:. will transfer the file named myfile from your current directory on your local machine to your home directory on the Hoffman2 Cluster. Its name on the cluster will be $HOME/myfile sftp ==== .. sidebar:: If you would prefer to use a graphical SFTP client... Please look under :ref:`Tools` for a list of SFTP GUI applications. *secure file transfer program* :program:`sftp` is a file transfer program, similar to ftp, which performs all operations over an encrypted ssh transport. It may also use many features of ssh, such as public key authentication and compression. .. csv-table:: SFTP Interactive Commands :header: "**Command**", "**Function**", "**Example**" :widths: 20, 20, 40 "cd", "Change remote directory to path", "cd [path]" "lcd", "Change local directory to path", "lcd [path]" "ls", "Display remote directory listing", "ls" "lls", "Display local directory listing", "lls" "pwd", "Display remote working directory", "pwd" "lpwd", "Print local working directory", "lpwd" "mkdir", "Create remote directory specified by path", "mkdir [path]" "get", "Retrieve the remote path and store it on the local machine", "get remote_path [local_path]" "put", "Upload local-path and store it on the remote machine", "put local_path [remote_path]" "exit", "Quit SFTP", "exit" "quit", "Quit SFTP", "quit" "help", "Display help text", "sftp help" For complete syntax, please refer to the man page. :: $ man scp **Let's establish an SFTP connection** .. note:: For large files or large amounts of data, use the Hoffman2 Cluster data transfer nodes (dtn.hoffman2.idre.ucla.edu). You can connect directly to dtn1.hoffman2.idre.ucla.edu or dtn2.hoffman2.idre.ucla.edu or use the Domain Name Service (DNS) round-robin address, dtn.hoffman2.idre.ucla.edu, which load balances requests between the two servers. Replace ``login_id`` with your cluster user name below. This is an example of using the sftp client on macOS Terminal:: $ sftp login_id@dtn.hoffman2.idre.ucla.edu login_id@dtn.hoffman2.idre.ucla.edu's password: Connected to login_id@dtn.hoffman2.idre.ucla.edu. sftp> **Now let's move a file from our local computer to the Hoffman2 Cluster** Replace ``login_id`` with the user name on your local computer. What is my current local working directory? and what files are listed:: sftp> lpwd Local working directory: /Users/login_id/share/ sftp> lls a.out index.html What is my remote working directory? :: sftp> pwd Remote working directory: /u/home/l/login_id Let's create a new directory on the remote computer and change our working directory to it :: sftp> mkdir uploads sftp> cd uploads Copy file, "a.out" from local computer to the Hoffman2 Cluster :: sftp> put a.out Uploading a.out to /u/home/l/login_id/uploads/a.out a.out 100% 3125 703.0KB/s 00:00 sftp> ls a.out rsync ===== The :program:`rsync` command uses the SSH2 protocol to efficiently transfer files. It is perhaps most useful in keeping groups of files on different computers up to date with each other. Here is a 2-part example of discovering the status of files in a common directory named mydir. It is comparing files in your Hoffman2 Cluster $HOME/mydir directory with those on your local machine mydir directory. You need both parts to ensure any new files from either source are synchronized. .. note:: For large files or large amounts of data, use the Hoffman2 Cluster data transfer nodes (dtn.hoffman2.idre.ucla.edu). You can connect directly to dtn1.hoffman2.idre.ucla.edu or dtn2.hoffman2.idre.ucla.edu or use the Domain Name Service (DNS) round-robin address, dtn.hoffman2.idre.ucla.edu, which load balances requests between the two servers. Please replace ``login_id`` with your cluster user name. Part 1: Run this on your local machine: .. code-block:: console $ rsync -an --itemize-changes login_id@dtn2.hoffman2.idre.ucla.edu:mydir . Any files prefixed with > in the output are different on the Hoffman2 Cluster and you may want to download them from the Hoffman2 Cluster (get): .. code-block:: console $ rsync -av login_id@dtn2.hoffman2.idre.ucla.edu:mydir . Part 2: Run this on your local machine: .. code-block:: console $ rsync -an --itemize-changes mydir login_id@dtn2.hoffman2.idre.ucla.edu: Any files prefixed with < in the output are different on your local machine and you may want to upload them to the Hoffman2 Cluster (put): .. code-block:: console $ rsync -av mydir login_id@dtn2.hoffman2.idre.ucla.edu: For more information about the rsync command and additional options, enter **man rsync** at the :ref:`shell prompt ` .. code-block:: console $ man rsync MobaXterm ========= :program:`MobaXterm` is a Windows application that provides an enhanced terminal with an embedded X11 server for accessing remote computers using various protocols, such as SSH. * For more information on the various protocols and features, please refer to their `website `_ * To see a `live demo `_ After you start an SSH session to a remote computer, you have the option to display remote GUI applications. To see this in action, please see the `live demo `_ Installing MobaXterm -------------------- You can download and use MobaXterm Home Edition for free. :ref:`Instructions for MobaXterm` documents how to download and configure the application to connect to the Hoffman2 Cluster. Using MobaXterm for data transfer --------------------------------- When you start an SSH session to a remote computer, a graphical SFTP browser pane opens to the left of the command-line pane and allows you to: * browse the remote computer's filesystem * transfer files using a secure SFTP connection * or even open and edit your remote files using the many Unix tools packaged with MobaXterm The graphical SFTP browser pane allows you to drag and drop files (similar to `File Explorer `_ in Windows) to transfer to/from the remote computer. Click on :guilabel:`Sessions > SFTP` from the menu bar... <--- Sample text