HPC/Network Access/SSH Tunnel Setup on Linux and MacOS

The following sections document how to establish ssh tunneling on a Linux or MacOS X machine running OpenSSH. Numerous Windows solutions are available to provide the same functionality, but details vary considerably.

First steps

Assumptions

For the purpose of the following examples, I'll use the following names:

• your external system is homebase,
• your user-id on homebase is home_id,
• your Argonne domain login is argonne_id,
• your home directory on your machine is /Users/home_id.

Registering the gateway

First, test that you can reach the gateway from your current machine (homebase). More importantly, this command will fetch and store the host key of mega on homebase:

homebase:/Users/home_id$ ssh [email protected]
The authenticity of host 'mega.cnm.anl.gov (146.139.1.6)' can't be established.
RSA key fingerprint is 0e:c5:2a:49:e9:27:6a:2e:b0:9c:ed:3b:5e:8f:28:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mega.cnm.anl.gov,146.139.1.6' (RSA) to the list of known hosts.
 
                       NOTICE TO USERS
                [... DOE warning banner ...]
 
Password: *********
Last login: Mon Feb 18 16:57:55 2008 from 1234.comcast.net
Could not chdir to home directory /Users/argonne_id: No such file or directory          <-- This is expected.
mega:/ argonne_id$ exit

This step is only needed once for each computer you try to connect from. The public key for mega is stored on homebase in home_id's .ssh/known_hosts file and will be verified at each subsequent connection. There is no need to be alarmed by the absence of a home directory.

Establishing a test tunnel

To establish a tunnel ad hoc or for testing purposes, use the following commands on your workstation, best in a newly opened terminal window:

homebase:/Users/home_id$ ssh -v -L 33301:clogin:22  [email protected]
 
                       NOTICE TO USERS
                [… DOE warning banner …]
…  
debug1: Next authentication method: keyboard-interactive
Password: *********
…
debug1: Local connections to LOCALHOST:33301 forwarded to remote address clogin1:22
debug1: Local forwarding listening on ::1 port 33301.
…
debug1: Entering interactive session.
Could not chdir to home directory /Users/argonne_id: No such file or directory
mega:/ argonne_id$ _

Note for experts

ssh can use the -N option to run without a shell and the -f option to background itself after the password request. Note, however, that the use of these options tends to leave a bunch of forgotten ssh processes hanging around which hog ports. These processes would have to be tracked down by ps and be killed manually.

Connecting through the test tunnel

Now you are ready to connect through the tunnel. This means opening an ssh session to localhost, which may look odd at first. However, rather than using the standard port (which would get you right back into homebase), use the port specified above at the -L option. Roughly speaking, that will kick sshd on mega into gear to initiate the connection to the internal tunnel target (clogin) on your behalf, and then hand the connection over to your workstation.

 homebase:/Users/home_id$ ssh -p 33301 argonne_id@localhost 
 argonne_id@localhost's password: *********
 Last login: Mon Feb 18 18:56:32 2008 from mega.cnm.anl.gov
 *******************************************************
 For documentation, see:
 	http://wiki.inside.anl.gov/cnm/HPC
 *******************************************************
 [21:24] argonne_id@login2:/home/argonne_id$ 

The port number (33301 above) must of course agree between the two commands, but is otherwise largely arbitrary within the range of about 1024 … 65000. It refers to a port on your machine (homebase). If, at the first command, you get a message "Address already in use", check with ps -ef if perhaps this is due to one of your own, still active, tunnels. If not, choose another port. Do not change the clogin port; it is always 22.

When done, exit the interactive shell.

[21:54] argonne_id@login2:/home/argonne_id$ exit
logout
Connection to localhost closed.

Then, exit the original connection to mega, which hosted the tunnel:

[21:00] argonne_id@mega:/Users/argonne_id$ exit
logout
Connection to mega.cnm.anl.gov closed.

You may also choose to leave it running in the background to connect back later.

Permanent setup

For more routine access, I recommend using ssh public keys and the ssh configuration file. Run the following commands on your workstation, homebase:

Create your ssh keys

If you do not already have an ssh private/public key pair, create one:

 homebase:/Users/home_id$ ssh-keygen -t rsa
 Generating public/private rsa key pair.
 Enter file in which to save the key (/Users/home_id/.ssh/id_rsa): 
 Enter passphrase (empty for no passphrase): ************
 Enter same passphrase again: ************
 Your identification has been saved in /Users/home_id/.ssh/id_rsa.
 Your public key has been saved in /Users/home_id/.ssh/id_rsa.pub.
 The key fingerprint is:
 42:c3:72:fb:4c:be:c6:80:81:bc:8b:73:d8:88:84:2e [email protected]

On modern systems, the mere presence of private keys with their default names triggers handling through ssh-agent and brings up a passphrase dialog at the start of the desktop session or when a key is first requested for an outgoing connection. This is the case on MacOS X Leopard, where key management is largely automated through its general keychain concept; on Tiger, use SSHKeychain.app.

To verify that an ssh-agent is running and visible to your shell, check for the presence of the environment variable SSH_AUTH_SOCK. To list the keys it has loaded, use ssh-add -l (lowercase "L"). Typical output is as follows:

Linux or MacOS X Leopard

 homebase:/Users/home_id$ echo $SSH_AUTH_SOCK
 /tmp/ssh-HCvZKt5478/agent.5478
 homebase:/Users/home_id$ ssh-add -l
 1024 b9:65:98:c0:7c:25:da:b6:86:48:98:6b:2a:5a:9d:21 /Users/home_id/.ssh/id_rsa (RSA)

MacOS X Tiger using SSHKeychain.app

 homebase:/Users/home_id $ echo $SSH_AUTH_SOCK
 /tmp/55522/SSHKeychain.socket
 homebase:/Users/home_id$ ssh-add -l
 1024 b9:65:98:c0:7c:25:da:b6:86:48:98:6b:2a:5a:9d:21 /Users/home_id/.ssh/id_rsa (RSA)
 

On older systems, you need to (a) set up the session to run under ssh agent and (b) request the passphrase upon login. Circumstances vary widely. Ask Google or your local admin for help. See also the page Getting started with SSH.

Configure ~/.ssh/config

Create your ~/.ssh/config file or, if you already have one (in which case you might already know the procedure anyway), add the following lines. The first line, to be placed before any other Host entries, is optional. Use it if your home is shared across machines or you use many of these type of forwards.

 # optional
 NoHostAuthenticationForLocalhost yes

 Host mega
       Hostname mega.cnm.anl.gov
       User  argonne_id
       LocalForward  33301  clogin:22
       LocalForward  33343  wiki.inside.anl.gov:443
       LocalForward  33380  cmgmt1:80
 
 Host carbon
       Hostname localhost
       User argonne_id
       Port          33301
       ForwardX11Trusted yes

As with 33301, the other port numbers refer to ports on your machine (homebase) and are mostly arbitrary. If you are on a shared machine, it is entirely possible that these ports are busy, especially if someone else follows these very instructions. Consult /etc/services or IANA and pick ports above 1024 that are unassigned.

Establish the tunnel - simple version

Open a separate terminal window:

ssh mega

This command now uses the host alias mega from your .ssh/config file to address the gateway and even set your remote user name. You can minimize the window at this point and essentially forget about it until you log out.

Alternatively, you may choose to run:

ssh -N mega

This will give no more output after the password prompt, but will disallow adding further tunneled connections.

Place your public key on Carbon

To put your ssh key pair to use, you need to append your ssh public key to an authorized_keys file on Carbon. To do this, while you are on your external workstation, type:

 cat ~/.ssh/id_rsa.pub | ssh carbon "umask 033; mkdir -p .ssh; cat >> .ssh/authorized_keys"

The command above is more reliable than cut&paste across interactive terminal sessions. If you created a dsa-type key, or chose your own name for the key pair, you'll have to cat the appropriate file instead. Remember to use the public part of the key, i.e. the file id_foo.pub.

Troubleshooting

If public keys do not work, use your password to gain access, then check the following:

Permissions

The directories and the authorized_keys file must, at the least, not be writable by anyone except yourself; other permissions are recommended to be restrictive as well:

 /home/argonne_id$ ls -ld ~ ~/.ssh ~/.ssh/authorized_keys
 drwxr-xr-x 28 argonne_id users 4096 Feb 22 13:52 /home/argonne_id
 drwx------  4 argonne_id users 4096 Feb 18 21:25 /home/argonne_id/.ssh
 -rw-------  1 argonne_id users  829 Oct  9 10:55 /home/argonne_id/.ssh/authorized_keys

To change:

 /home/argonne_id$ chmod go-w ~
 /home/argonne_id$ chmod go-rwx ~/.ssh ~/.ssh/authorized_keys

Corruption of authorized_keys file

The file should contain public keys in OpenSSH format, one per line, without additional line breaks. One way to check is:

 /home/argonne_id $ cut -c1-50 ~/.ssh/authorized_keys
 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEjdsfjJSAHkjh88y
 ssh-dss AAAAB3NzaC1kc3MAAACBAkkjsadfJFg7EFjDJhWEjk

The output should look similar to the sample above. Use your favorite text editor to join erroneously broken lines. Note that there should not be anything like BEGIN RSA PRIVATE KEY, which would obviously indicate a private, rather than public key.

Routine use

The commands above describe a one-off setup process. Once all the necessary files are in place, all that is needed for routine access is to open the tunnel once per login session, and then use it as often as needed. The host aliases defined in the config file allow the tunnel "entrance" to appear like a regularly named machine.

I - Establish the tunnel - advanced version

If a tunnel session is not already running, open one. You can use one of the ways described above, or add a twist as follows:

 ssh -v mega  iostat -w 60

This command will give you a life sign from the tunnel machine every minute. This may be helpful with choppy network connections. The actual output (the load average on the gateway machine) is useful only for an admin who will note if the machine should get bogged down. The -v option will give you some ssh chatter, especially when the tunnel is being used. As before, you can still minimize this window and forget about it until the end of your desktop session. Note that you will always have to type your domain password to reach mega.

Keep in mind that either one of the above methods to establish the tunnel is necessary only once for each desktop session on your workstation.

II - Access the cluster

Once the tunnel is established, you can access the cluster through the tunnel as often as you like. If you've set up the ssh agent on your external machine, you no longer need to type your domain password for any of the following commands. These commands are for use on your external workstation.

Open an interactive shell

 ssh carbon

Execute a single remote command

 ssh carbon qstat

Copy a file to Carbon (push from outside)

 scp -p foo carbon:remote/path/

Copy a whole directory to Carbon

Short of using tar or zip:

 scp -rp local/path/ carbon:remote/path/

This method does not preserve symbolic links.

Copy a file from Carbon (pull from outside)

 scp -p carbon:remote/path/foo  local/path/
 scp -p carbon:remote/path/foo  .

Note the trailing "." in the last example to copy to your current directory.

Reverse tunneling

The file copy operations above must all be initiated on the outside host. It is possible to set up tunnels to be able to initiate file operations on the inside. However, this is more complex and will be left as an exercise for advanced users. It involves setting up:

1. agent forwarding
2. a .ssh/config file on Carbon, and
3. specifying -R command line options or RemoteForward config file directives when accessing carbon at the second stage.

Bonus – Intranet web access

Open the following links in a browser on your external workstation:

 https://localhost:33343/cnm/HPC
 http://localhost:33380/ganglia

--stern 00:24, February 20, 2008 (CST)