Command Line Data Transfer
Standard Linux tools can efficiently transfer a small to moderate quantity of data to or from Rivanna/Afton.
scp
scp
uses the secure shell (SSH) protocol to transfer files between your local machine and a remote host, or between two remote hosts.
The following syntax enables copying from local to remote or vice versa. In both cases we are starting from the local system.
By default, scp
works from the level of the directory in which it is invoked.
- Copying from local to remote:
scp source_file mst3k@hostaddress:target_file
- Copying from remote to local:
scp mst3k@hostaddress:source_file target_file
The following examples detail how to transfer data between your local computer and /project storage on Rivanna/Afton. In these examples
my_file
is the file you would like to transfermst3k
is your computing IDmygroup_name
is the name of your/project
directory.my_directory
is the directory to which you wish to copy the file.
To copy a file
From your computer to /project
storage:
scp my_file mst3k@login.hpc.virginia.edu:/project/mygroup_name
From /project
storage to my_directory
on your computer:
scp mst3k@login.hpc.virginia.edu:/project/mygroup_name/my_file /my_directory
scp
accepts wildcards. In this example, the mycode directory must exist in your scratch directory.
scp *cxx mst3k@login.hpc.virginia.edu:/scratch/mst3k/mycode
scp
Options
The -r
option recursively copies directories.
From your computer to /project storage:
scp -r my_directory mst3k@login.hpc.virginia.edu:/project/mygroup_name
From /project
storage to your computer:
scp -r mst3k@login.hpc.virginia.edu:/project/mygroup_name /target_directory
The -p
option preserves modification time, access time, and ownership from the original file.
The -q
option suppresses the progress and debugging messages. Useful for scripts.
sftp
Secure FTP or sfpt
is an interface built on top of scp
to mimic the behavior of ftp
.
To connect to UVA HPC with sftp
, execute the following in the command line interface:
sftp mst3k@login.hpc.virginia.edu
When prompted, enter your password. Once the connection succeeds, you will see the sftp
prompt:
sftp>
Navigating Directories
You can access both your local and remote file systems with sftp
. The following table lists how to execute the following commands for both your local and remote systems.
Action | On Remote System | On Local System |
---|---|---|
Print Working Directory | pwd | lpwd |
List Contents of Directory | ls | lls |
Change Directory | cd | lcd |
File Transfer from Local to Remote
To transfer files from your computer to the Rivanna/Afton file system, use the put command:
sftp> put my_file
To transfer a folder from your computer to Rivanna/Afton, use put -r
. A folder with the same name must also exist on Rivanna. An example is shown below:
sftp> mkdir /project/mygroup_name/my_folder
sftp> cd /project/mygroup_name
sftp> put -r my_folder
File Transfer from Remote to Local
To transfer files from Rivanna/Afton to your computer, use the get command:
sftp> get my_file
To transfer a folder from Rivanna/Afton to your computer, use get -r
:
sftp> get my_folder
Terminating the Connection
To terminate the sftp
connection, use exit
.
sftp> exit
rsync
Remote sync is a powerful tool for copying files. It is most widely used to transfer multiple files and/or directories.
In this example, we have a local directory ldir
and a remote directory rdir
and we wish to copy the contents of ldir to rdir.
rsync -r ldir/ mst3k@login.hpc.virginia.edu:rdir
The trailing /
after ldir is important. Without it, ldir and its contents would be placed under rdir
.
Unlike scp
, if the target directory does not exist, rsync
will create it.
It is more common to use the -a
(archive) option. This option preserves symbolic links, special files, ownership, permissions, and timestamps.
rsync -a ldir/ mst3k@login.hpc.virginia.edu:rdir
Show a progress bar and keep partially transferred files
rsync -Pa ldir/ mst3k@login.hpc.virginia.edu:rdir
Delete files not present on the source directory if they are present on the target directory
rsync -Pa --delete ldir/ mst3k@login.hpc.virginia.edu:rdir
Have rsync print the list it will transfer without carrying out the transfers. Especially important when using --delete
.
rsync -Pa --delete --dry-run ldir/ mst3k@login.hpc.virginia.edu:rdir
AWS CLI
The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services, including data transfer.
Learn more about the AWS CLI tools.
Globus CLI
The Globus command-line interface can also be used to orchestrate the transfer of large datasets, or to script regular transfers in or out of systems.
Read more about the Globus CLI.
Usage from Off Grounds
Off Campus?
Connecting to Rivanna and Afton HPC systems from off Grounds via Secure Shell Access (SSH) or FastX requires a VPN connection. We recommend using the UVA More Secure Network if available. The UVA Anywhere VPN can be used if the UVA More Secure Network is not available. Only Windows and Mac OSX operating systems are supported by the Cisco client provided by ITS. Linux users should refer to these unsupported instructions to install and configure a VPN. The More Secure Network requires authentication through Duo; users should follow the instructions on the dialog box to enter "push" as the password.
Open OnDemand users do not need a VPN to access Rivanna.