/tag/data-transfer
Distribution and Discovery of Digital Information
A team of UVA Commerce faculty partnered with the DAC to explore how information about personal health and finance propagates on online platforms and influences decisions. The DAC developed for them a process to merge and standardize data from CoreLogic and ComScore, two sources of consumer and real estate information. The DAC assisted by developing a method to map real estate data to Designated Market Areas (DMAs), which are geographic regions where the population receives similar types of information through the media. The DMA mapping allowed the researchers to link consumer behavior in the real estate market with demographic characteristics. To make the data analysis more efficient, the DAC automated the data ingestion and analysis processes using high-performance computing.
Globus Data Transfer
Globus is a simple, reliable, and fast way to access and move your research data between systems. Globus allows you to transfer data to and from systems such as:
Laptops & personal workstations Rivanna/Afton HPC clusters High-Security Research Standard Storage Lab / departmental storage Tape archives Cloud storage Off-campus resources (ACCESS, National Labs) Globus can help you share research data with colleagues and co-investigators, or to move data back and forth between a lab workstation and Rivanna/Afton or your personal computer.
Are your data stored at a different institution? At a supercomputing facility? All you need is your institution’s login credentials.
Scratch Recovery with Globus
Globus is a simple, reliable, and fast way to access and move your research data between systems. Researchers can transfer data from their old scratch directories to new scratch via a single web interface - no software download or installation is necessary.
Command Line Data Transfer
Standard Linux tools can efficiently transfer a small to moderate quantity of data to or from Rivanna/Afton.
scp scp uses the secure shell (SSH) protocol to transfer files between your local machine and a remote host, or between two remote hosts.
The following syntax enables copying from local to remote or vice versa. In both cases we are starting from the local system.
By default, scp works from the level of the directory in which it is invoked.
Copying from local to remote: scp source_file mst3k@hostaddress:target_file Copying from remote to local: scp mst3k@hostaddress:source_file target_file The following examples detail how to transfer data between your local computer and /project storage on Rivanna/Afton.
Cyberduck
Cyberduck is a transfer tool for Windows and Mac. It supports a large number of transfer targets and protocols. Only SFTP can be used with Rivanna/Afton. The free version will pop up donation requests.
Download Download Cyberduck
Connecting to the HPC System and File Transfer Launch Cyberduck. After launching Cyberduck, the user interface will open. To initiate a connection to UVA HPC, click the Open Connection button.
Enter Your Credentials. From the drop-down menu, select SFTP (SSH File Transfer Protocol). Then enter the appropriate information in the following fields:
Host: login.hpc.virginia.edu Username: your computing ID Password: your UVA HPC password Port: 22 When completed, click Connect.
Filezilla
Filezilla is a cross-platform data transfer tool. The free version supports FTP, FTPS, and SFTP. Only SFTP can be used with UVA HPC.
Download Download Filezilla
Connecting to the HPC System and File Transfer Launch FileZilla. After launching FileZilla, the user interface will open. In the left panel, you should see your local file system and files listed in the left side panels. You will enter your login credentials in the fields highlighted in the figure below.
Enter Your Credentials. Fill in the Host, Username, Password, and Port fields.
Host: login.hpc.virginia.edu Username: your computing ID Password: your Eservices password Port: 22 When completed, click Quickconnect.
Data Transfer
Efficient and reliable data transfer is a critical component of research computing. A variety of useful tools is available for rapid data transfer, whether you are transferring data from an external site or within different computing environments at UVA. Common Scenarios Transfer public or internal use data between local workstation/laptop and UVA storage Transfer sensitive or highly sensitive data to Ivy storage Transfer data between external institutions/supercomputing centers and UVA Transfer between UVA HPC and Cloud storage The data transfer method you choose heavily relies on the data sensitivity classification, where the data are currently located and to where you want to transfer the data.
Data Transfer
Data transfer Public & Moderately Sensitive Data Transfer Secure Copy (scp) scp uses secure shell (SSH) protocol to transfer files between your local machine and a remote host. scp can be used with the following syntax:
scp [source] [destination]
scp SourceFile mst3k@login.hpc.virginia.edu:/scratch/mst3k
scp SourceFile mst3k@login.hpc.virginia.edu:/project/Grouper_group_name
Detailed instructions and examples for using scp are listed here.
Secure File Transfer Protocol (sftp) sftp is a network protocol for secure file management. Instructions and examples for using sftp are located here.
Graphical File-Transfer Applications Filezilla and Cyberduck, and MobaXterm are examples of open source SFTP client software for file management through an interactive graphical user interface.
