Transferring Files: Difference between revisions

From TRACC Wiki
Jump to navigation Jump to search
Amiot (talk | contribs)
534 edits
Created page with "== Transferring Files == * Files can be transferred to and from the cluster using a client that supports the SFTP (SSH File Transfer Protocol) or with scp (Secure Copy Protoc..."
 
Ley (talk | contribs)
277 edits
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Transferring Files ==


* Files can be transferred to and from the cluster using a client that supports the SFTP (SSH File Transfer Protocol) or with scp (Secure Copy Protocol). [[Free SSH clients | Free SFTP clients]] are available for Windows, and are included with or available for MacOS, and most varieties of Linux/UNIX. Be sure your client is using port 22.
==File Transfer==
Files can be transferred to and from the cluster using a client that supports the SFTP (SSH File Transfer Protocol) or with scp (Secure Copy Protocol). Free SSH or SFTP clients are available from https://www.putty.org for Windows, and are included with or available for MacOS, and most varieties of Linux/UNIX. Be sure your client is using port 22 on the login node to connect. This should be the default for most terminal and file transfer applications, but has proven to be a problem for some users unaware of the port number.


*Popular file transfer clients are WinSCP and FileZilla. You can download WinSCP from http://winscp.net and FileZilla from http://filezilla-project.org.
Popular file transfer clients are WinSCP and FileZilla. You can download WinSCP from http://winscp.net and FileZilla from http://filezilla-project.org. Even the "rsync" utility on a Linux or WSL machine can be used with the appropriate options. Contact us to learn more about the various options for file transfers.


* Some users have also reported success with [http://www.itefix.no/i2/cwrsync cwrsync], a prepackaged version of Cgywin and rsync.
Some users have also reported success with [http://www.itefix.no/i2/cwrsync cwrsync], a prepackaged version of Cgywin and rsync on Windows. There are numerous other SSH and SFTP clients available, and all of them should work with the login nodes.


=== Windows text files ===
==Text File Compatibility==
 
Note that text files created under Windows may cause subtle issues. This is because Windows, MacOS, and Linux have different conventions for representing a [http://en.wikipedia.org/wiki/Newline newline] in ASCII text files. Some Windows text editors will allow you to save a file in the UNIX/Mac/Linux format before transferring the file to the cluster. Alternatively, you can use the Linux command dos2unix to remove the CR characters that Windows uses at the ends of lines, or the reverse utility unix2dos:
Note that text files created under Windows may cause problems on the cluster. This is because Windows and Linux have different conventions for representing a [http://en.wikipedia.org/wiki/Newline newline] in ASCII text files. Some Windows text editors will allow you to save a file in the UNIX/Linux format before transferring the file to the cluster. Alternatively, you can use the Linux command dos2unix to remove the CR characters that Windows uses at the ends of lines:


  dos2unix somefile
  dos2unix somefile
Line 15: Line 14:
dos2unix will have no effect on a Linux text file, so it's safe to use on any text file.
dos2unix will have no effect on a Linux text file, so it's safe to use on any text file.


You can determine whether a file is in Windows or Linux format with the command
==GLOBUS Connect==
 
cat -v somefile
 
Windows text files will have a ^M at the end of each line.
<!--
=== Transferring files from Phoenix to Zephyr ===
I don't think we need to transfer files within the cluster anymore so I commented out all the code. Take a look at what is commented out and if we all agree we can remove this whole section........Larry ...........I agree, Steve
 
 
An easy and efficient way to transfer files is using the tool '''rsync''' over ssh.
 
The following example creates a "jobs" directory on Zephyr and transfers the '''hpl_benchmark''' directory along with its content from the jobs directory located in the users "home" directory on the Phoenix cluster to the created "jobs" directory on the Zephyr cluster.
 
# Login to login.zephyr.tracc.anl.gov
# rsync -avze ssh login.phoenix.tracc.anl.gov:~/jobs/hpl_benchmark ~/jobs/
 
For more complicated transfers, i.e collaborative groups, please contact us. 
 
 
 
The following example shows how to transfer a project file called BridgeModel.cad. First login to Zephyr:
 
 
'''Then you must sftp to:'''
 
    login.phoenix.tracc.anl.gov
 
 
'''To Establish the connection:'''
 
Connecting to login.phoenix.tracc.anl.gov... <br>
[email protected]'s password: <br>
 
'''Once connected, change to desired directory:''' <br>
sftp> cd project <br>
sftp> ls <br>
BridgeModel.cad <br>
sftp> get BridgeModel.cad <br>
Fetching /mnt/gluster/home/jdoe/project/BridgeModel.cad to BridgeModel.cad <br>
/mnt/gluster/home/jdoe/project/BridgeModel.cad 100% 100KB 100.0KB/s 00:00 <br>
sftp> quit <br>


[jdoe@login1 ~]$ ls -l BridgeModel.cad <br>
This is being evaluated and tested as of late 2025.
-rw-r----- 1 jdoe jdoe 102400 Sep 24 20:30 BridgeModel.cad
-->

Latest revision as of 01:04, December 6, 2025

File Transfer

Files can be transferred to and from the cluster using a client that supports the SFTP (SSH File Transfer Protocol) or with scp (Secure Copy Protocol). Free SSH or SFTP clients are available from https://www.putty.org for Windows, and are included with or available for MacOS, and most varieties of Linux/UNIX. Be sure your client is using port 22 on the login node to connect. This should be the default for most terminal and file transfer applications, but has proven to be a problem for some users unaware of the port number.

Popular file transfer clients are WinSCP and FileZilla. You can download WinSCP from http://winscp.net and FileZilla from http://filezilla-project.org. Even the "rsync" utility on a Linux or WSL machine can be used with the appropriate options. Contact us to learn more about the various options for file transfers.

Some users have also reported success with cwrsync, a prepackaged version of Cgywin and rsync on Windows. There are numerous other SSH and SFTP clients available, and all of them should work with the login nodes.

Text File Compatibility

Note that text files created under Windows may cause subtle issues. This is because Windows, MacOS, and Linux have different conventions for representing a newline in ASCII text files. Some Windows text editors will allow you to save a file in the UNIX/Mac/Linux format before transferring the file to the cluster. Alternatively, you can use the Linux command dos2unix to remove the CR characters that Windows uses at the ends of lines, or the reverse utility unix2dos: 

dos2unix somefile

dos2unix will have no effect on a Linux text file, so it's safe to use on any text file.

GLOBUS Connect

This is being evaluated and tested as of late 2025.

Retrieved from "https://wiki.anl.gov/wiki_tracc/index.php?title=Transferring_Files&oldid=2445"