SSH

• 1: SSH Agent
• 2: SSH Gateways
• 3: SSH Keys
• 4: Troubleshooting SSH Connections
• 5: Using SSH Keys
• 6: Working With SSH Key Pairs

1 - SSH Agent

Each time you log into a remote host using SSH keys, you will be prompted to supply the passphrase used to lock your private key. There is an SSH agent that will hold your private key for you, respond to the SSH transaction, and supply the key as needed. 

You start the SSH agent by opening a Terminal window and entering the command:  ssh-agent [command [args ...]] Where command is an optional application to run that will have access to the agent. Most modern versions of Linux have the agent being started when your X-windows session starts, so the agent will be available to all applications under X.

You add keys to the agent with the ssh-add command. Typing ssh-add -l (where the argument is a lower-case L) will tell you whether the agent is running and which keys are loaded into it. 

If the ssh-add -l command displays:

Could not open a connection to your authentication agent.

then the agent is not running. If the response is:

The agent has no identities.

then no keys are loaded into the agent.

To load a key into the agent, use the command ssh-add [file ... ] If no file name is provided, then an attempt is made to load the default files; otherwise, the named private key file will be loaded. You will be prompted for the passphrase for the private key file.

Once your key is loaded in your agent, you can forward it along your SSH connections by including a forwarding flag (-X for X forwarding, or preferably -Y for authenticated X forwarding, e.g., ssh -Y).

SSH Agent in Windows (PuTTY)

To use the SSH agent under Windows, start Pageant from the PuTTY menu. Starting the agent will put an icon of a computer terminal with a hat on it in the system tray.

To load a key into the agent, right-click on the icon, and then click Add Key. A file dialog will come up, and you should open the private key file that you want to load. You will be prompted for the key’s passphrase.

If you want the agent to start automatically and load your private key when you Windows starts, simply add the .ppk file to your Startup folder in your Start menu (this assumes that you have installed the PuTTY programs using the Windows installer method, so that the association between the .ppk extension and the Pageant program was made) as: C:\Documents and Settings\<user>\id_rsa.ppk where <user> is replaced by your username, and the .ppk file should be the name of your private key file. You will be prompted to browse to the key file during the process of adding the file to the Startup folder.

If you want to add more than one private key at startup, then put a script like the following in your Startup folder:
C:\Program Files\PuTTY\pageant.exe <keyfile> <keyfile> where <keyfile> is replaced by the names and paths to the key files that you want to load.

If you do this, then when your machine starts up, a dialog box will prompt you for the passphrase for each of the private key files in the Startup folder. After you enter the passphrase, the Pageant icon will appear in the system tray and contain your private key.

Forwarding Keys in PuTTY

Once your private key has been loaded into Pageant, you’ll need to instruct PuTTY to forward the key as necessary.  To do this, in the main PuTTY Configuration window, in Connection -> SSH -> Auth, enable the options for Attempt authorization using Pageant and Allow agent forwarding.

2 - SSH Gateways

Information on the SSH Gateways at the SDCC

The SSH gateways at the SDCC are the primary “doors” into the SDCC facility. They are only to be used to gain access to the facility, and are not designed for data transfers or interactive sessions. Once you have logged into an SSH gateway, you can continue your connection through to another internal system via SSH. The legacy RHIC and US ATLAS SSH gateways have been deprecated and consolidated into the single set of SDCC SSH gateways.

Before starting your SSH session, enable SSH agent forwarding to have your public SSH key follow your session through each connection.

The SDCC SSH gateways can be accessed via the ssh.sdcc.bnl.gov alias, which automatically maps your connection to one of the multiple backend servers to provide load balancing and higher availability. A specific backend SSH server can be accessed explicitly by using its specific hostname, but it is recommended, whenever possible, to use the load-balanced ‘ssh’ alias. Note that each backend server has its own set of home directories, that is the home directories are not shared among gateway systems. At this time, the available backend servers are as follows:

Hostname	Backend Hostnames
ssh.sdcc.bnl.gov	ssh01.sdcc.bnl.gov ssh02.sdcc.bnl.gov ssh03.sdcc.bnl.gov ssh04.sdcc.bnl.gov

For slightly easier navigation, the facility supports Kerberos-based single sign-on. Running kinit and providing your Kerberos password will provide you with a Kerberos “ticket-granting ticket” (TGT)., which can used for all subsequent requests for authentication when connecting via SSH to other internal systems.

For BNL campus network access, we also provide another set of SSH gateways (note that the campus network is somewhat isolated and does not have access to most of the SDCC computing resources):

Hostname	Backend Hostnames
cssh.sdcc.bnl.gov	cssh01.sdcc.bnl.gov cssh02.sdcc.bnl.gov

User shell sessions on the SSH gateways are restricted, as they are intended as gateways to access other resources, and not to be used as general-purpose machines.

Once on an SSH gateway, you can use the rterm command to start an xterm session to one of the destination interactive nodes allocated to your experiment, or if you prefer, rterm -i to connect within your existing session and without launching an xterm window. If you require direct access to a specific experiment node for some reason, please consult your experiment’s documentation for a list of interactive nodes appropriate for your use, or contact your experiment liaison.

See our cluster gateway information to access to any of the SDCC computing clusters.

3 - SSH Keys

SSH keys are a public/private key pair that allows login to an SSH daemon (sshd) without using a password. SSH keys are generated using the ssh-keygen program on Linux/Unix/ macOS/Cygwin, or with PuTTYgen on Windows.

Note: SSH keys are checked within the sshd program itself, thereby taking precedence over the other authentication methods.

When fully implemented, CryptoCards authentication will automatically obtain Kerberos and AFS tokens and tickets.

SSH Key Utilities

SDCC has several web resources available to help you manage your SSH keys in the SDCC environment. 

• View the SSH key associated with your account
• Upload a new SSH key to SDCC
• Delete the SSH key associated with your account

4 - Troubleshooting SSH Connections

There are a number of problems that can cause failures when connecting to the SDCC. Here are some things to look for and try in order to resolve your problem.

Private and Public Key Issues

• Have you uploaded your public key to the SDCC via the key file upload form (requires your Kerberos user name and password)?
  • If your browser dialog does not display your .ssh directory or files because they are hidden, see these instructions.
• Most problems with a public key missing from a connection can be mitigated by using SSH agent.
• Are you connecting to our SSH gateway from the same system on which you generated your key pair?  If not, you will have to copy the private key to this additional system.  If this system is using a different SSH client, you may need to convert or import the private key.  See Using SSH Keys. Do not generate and upload another public key from this additional system; uploading another public key will overwrite your existing public key and create more problems. Even using different SSH clients on the same system may require the conversion or import of the private key.
• Are you asked for a passphrase when you activate your SSH agent or when you connect?  If not, then your client is not using the private key for some reason. It could be the private key doesn’t exist, is not in the default filename, access rights of the file are incorrect, the file is not in a directory the client is searching, or some other reason.
• If you have uploaded another public key, then that key pair is the only usable pair that will work, and all other pairs are now obsolete.  Also, the private key from this pair must be copied and possibly converted or imported to any other SSH clients you are using.

Username Issues

If your username on your local system is different from your username at the SDCC, then you must specify your SDCC username when you connect to the SDCC, using the -l option to the ssh command:

ssh -l [username] [SDCC-hostname]

or prepending @username to the SSH gateway system name (no space between the @ and the SSH gateway system name):

ssh [username]@[SDCC-hostname]

In Windows SSH clients, there is typically a text box in which you type in your username.

Ownership/Access Rights Issues

If you are using a Linux/UNIX-based SSH client, please check the ownership and access rights of your ~/.ssh/ directory and
the private key file in that directory.  Both must be owned by your local user account (not necessarily the same as your
SDCC user account). The rights on your ~/.ssh/ directory should be 700, and the rights on the private key file (possibly, but not definitely, named ~/.ssh/id_rsa), should be 600. The important thing here is that “group” and “other” access rights must be 00.

PuTTY Issues

If you are using PuTTY in Windows, then you have to either import your private key or somehow tell PuTTY where the key file is.

In the main PuTTY Configuration, click on SSH and then Auth.  The window will have a text box where you can put the path of the key or browse for it.  See the Generate a New Key Pair Using PuTTY for Windows Users section of this document for more information on generating SSH keys for use with PuTTY.

You may also need to forward your private key through a remote gateway machine to another server.  See our document on SSH Agent for more information on storing and forwarding your private key.

Viewing Your Public Key

You can view the contents of the public key you uploaded to the SDCC with our SSH Public Key File Viewing Utility. You can check this against any public key that may be on your local system (the public key is not required to be on your local system; the private key is required to be there).  If they are not the same, then the private key on your local system may not be paired with the public key you uploaded to the SDCC.

If you have both private and public keys on your local system, check the date/time stamps on them; they should be the same.  If they are not the same, then the private key on your local system may not be paired with the public key that you uploaded to the SDCC.  If you are using the openssh client, then you can also check to see if your local private key is paired to the public key that you uploaded to the SDCC.  From a Terminal window, run the command:

ssh-keygen -y

on your local system.  It will ask for the filename of your private key and its passphrase and will display the public key (without the trailing comment field) that is paired with it.  Check this against the results of viewing the public key
you uploaded to the SDCC as described above.

Frozen Sessions and Terminals

If your connection or session intermittently freezes, try adding a server keep-alive option to your usual SSH command:

ssh ... -o ServerAliveInterval=120

This ensures that a set of request and acknowledgment packets will be sent between the connection every two minutes, even when no other data has been requested. You can also add this option to your SSH configuration file (~/.ssh/config) instead of specifying it with each SSH command:

 ServerAliveInterval 120

Host Key Issues

Sometimes host key problems can close the ssh connection before login completes.  If you see an error like this:

ssh_exchange_identification: Connection closed by remote host

Then you might try removing the offending host key from your ~/.ssh/known_hosts file and try again.

Error: Agent admitted failure to sign using the key

This error might occur if you accidentally load the wrong SSH identity for a specific key, if you’ve uploaded a new public key that hasn’t yet been synced with your account (or uploaded multiple or invalid keys), or if you’re trying to load too many SSH identities at one time.  Your best recourse is usually to:

1. Log out of all current sessions
2. Log back in
3. Add your identity with the ssh-add command.

IPv6 and Disconnections

If you’re connecting to our SSH gateways via IP version 6 and notice problems with frequent disconnections, try forcing connections with IPv4 instead with the ‘-4’ option, i.e.:

ssh -4 …. 

Passwords vs. Passphrases

Your SSH key passphrase and your SDCC user account password are two distinct entities. Your personal SSH key passphrase, not your SDCC user account password, protects your private key and is required in order to connect to the SSH gateway with your public key. Account password changes do not affect your SSH key passphrase, which is private to you and known to you alone.

If you forget and can not determine your SSH passphrase, you can:

• Generate a new passphrase for your existing key, via:

ssh-keygen -p [existing-key-file]

• Or generate an entirely new key with another passphrase, and upload its fingerprint.

To help eliminate many common issues with SSH keys and passphrases, see the next section.

SSH Agent

We highly recommend using an SSH agent to forward your public key and passphrase ahead to remote sessions, which will help avoid most problems with the key and obviate the need to enter your passphrase at a remote prompt. See our SSH Agent page for details.

X Session Forwarding

You may have an issue with X11 session forwarding or corrupt session files. See our X11 troubleshooting page for help.

DSS Keys

The use of SSH key pairs generated via the DSA encryption algorithm, also known as DSS keys, is prohibited by the facility, as the algorithm is known to be weak in comparison to stronger, more modern encryption algorithms. Please do not attempt to generate and use DSA/DSS keys; instead, generate key pairs using the RSA algorithm, as described in our page on working with SSH key pairs.

RSA-Signed Keys

Newer OSes and versions of OpenSSH may reject the older, stable version of OpenSSH running on the SSH gateways, returning errors such as signing failed for RSA ... agent refused operation and restricting connections with keys signed with an RSA-based algorithm. You can override this by adding ssh-rsa to the list of accepted public key types and host key signing algorithms accepted by your SSH client, either in your remote host configuration, or adding it as an option to your SSH commands when connecting to the SDCC, e.g.:

ssh -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostkeyAlgorithms=+ssh-rsa you@ssh.sdcc.bnl.gov

All options on the command line

ssh -i /path/to/your/private/key -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostkeyAlgorithms=+ssh-rsa -A your_username@ssh.sdcc.bnl.gov

Further Troubleshooting

Some additional SSH-related sites and resources:

• The OpenSSH FAQ
• The PuTTY client FAQ

If all else fails, try running this command, substituting your account user name for [username]:

ssh -vvv [username]@ssh.sdcc.bnl.gov

Include the output of this command in an RT ticket in the UserAccounts queue.  Be sure to also include the operating system and SSH client software you are using.

5 - Using SSH Keys

To use your SSH key to log in to another remote machine, you must copy the public key file to the remote machine. For use at the SDCC, you can use the SSH key file upload form to upload your public key file (the one with the .pub extension) to the facility. Your public key will be stored in LDAP and will be available on all of the gateway machines. When you log into one of the gateway machines after uploading your key, you will be prompted to enter the passphrase for your private key (the passphrase and private key remain on your local machine and never be transmitted over the network). After successfully entering your passphrase, you will be logged into the gateway. To avoid typing your passphrase each time you log in, please refer to the SSH agent page.

If your browser dialog does not display your .ssh directory or files because they are hidden, see these instructions.

SSH Keys on Multiple Machines

You can only upload and use one SSH key at a time at the SDCC.  If you own multiple machines (e.g., a desktop and a laptop), then you can generate a public/private key pair on one machine, upload the public key to the LDAP server, and copy the private key to your other machines.

If your public key ends with a [username]@[host] string that does not directly follow an equals character (’=’), the [username]@[host] string serves as a host restriction and must be removed from the public key before the key can be used from another machine. A [username]@[host] string that immediately follows ’=’ at the end of the public key is merely a comment and can be ignored or removed.

Note: If you want to move keys between a Windows machine running PuTTY and a Linux machine, then you will have to convert the format of the key since PuTTY uses a different format from OpenSSH. To do so, see the Using An Existing Windows Key Pair on a Linux Machine section of the Working With SSH Key Pairs page. If you use Cygwin, the key files should be fully compatible, since OpenSSH is the version of SSH implemented in Cygwin.

Using Multiple SSH Keys

If you need to use multiple different SSH keys in order to access remote locations in addition to SDCC, a few options are available:

• Add the -i flag to your SSH client command and specify the path to your desired private key (e.g., ssh -i ~/.ssh/id_rsa_2).
• Load all necessary keys into your SSH agent.
• Configure your SSH client to load the required key when connecting to each host, via the IdentityFile configuration directive in your .ssh/config file, along with the host name, your user name, and an alias, if desired, e.g.:

Host your_alias yourhost.sdcc.bnl.gov
HostName yourhost.sdcc.bnl.gov
    IdentityFile ~/.ssh/your-private_key
    User your_user_name ```

6 - Working With SSH Key Pairs

SSH keys) are generated using the ssh-keygen program on Linux/UNIX/macOS/Cygwin, or with PuTTYgen on Windows. Once an SSH key has been generated, you should upload your public key to the SDCC’s servers.

Generate SSH Keys

Generate a Key Pair on Linux/macOS

1. Open a terminal window on the desktop machine or laptop that you will be using to log in to the SDCC.
2. At the prompt, type: ssh-keygen -t rsa You will see output similar to the following:

   Generating public/private rsa key pair.  
   
   Enter file in which to save the key (/home/<user>/.ssh/id_rsa):
   

   where <user> is replaced by your user name.
   This command will generate an RSA key of the default length (2048 bits).
3. To accept the default file name and location ~/.ssh/id_rsa, press Return.
4. At the Enter passphrase prompt, type in a pass phrase, which will not be echoed as you type, and then press Return. This passphrase will be used to unlock your private key file (failing to enter a passphrase for your key will, of course, defeat all security related to the key pair).
5. You will be prompted to verify the passphrase by entering it again. Retype your passphrase, and then press Return. The key pair will be generated, and you will see output similar to the following:

    Your identification has been saved in /home/<user>/.ssh/id_rsa.  
   
    Your public key has been saved in /home/<user>/.ssh/id_rsa.pub.  
   
    The key fingerprint is:  
    SHA256:PFCKP1tBWnyxxXX+GM4g+ldsJ9/upHPNzloiQLX467c <user>@<machine.name>
   

   … where the actual fingerprint for your key will be displayed (not the one shown in the example above), and where the terms in brackets (<>) are replaced by the values appropriate for your machine.
   Two files will be created:

• The first file is the private key, with the default name (or the name you entered above).
• The second file is the public key, with .pub appended to the file name.

6. Copy the key fingerprint value from the output above.

Generate A New Key Pair using PuTTY for Windows

These instructions document the use of the free PuTTY client. PuTTY is an implementation of SSH for Windows and UNIX platforms, written and primarily maintained by Simon Tatham.

By default the PuTTYgen program used to generate key pairs will generate a 1024-bit RSA key. Ensure that the “Number of bits in a generated key” value at the bottom of the GUI is set to 2048 bits or larger, or else your key will be rejected by SDCC.

Additional information on using SSH keys with PuTTY can be found at http://the.earth.li/~sgtatham/putty/0.60/htmldoc/Chapter8.html.

1. On the desktop machine or laptop that you will be using to login to the SDCC, from the PuTTY submenu in the Start menu, open the PuTTYgen program.
2. Using the defaults, click the Generate button, and then move your mouse around in the space above the Generate button.
3. When the key pair generation has completed, you will see a result similar to the following image, with both the Save public key and Save private key buttons now active.

![PuTTY Key Generator][/docs/ssh/putty_key.png]

If desired, you can change the value of the **Key comment:** field.   

4. Enter a Key passphrase and Confirm passphrase for password protection on the generated key.
5. To save your new keys, click the Save public key and Save private key buttons.
   The default save location for both files is the folder just above your My Documents folder, and your private key will have the file extension .ppk.
   Keep this window open so that you can copy and paste the key fingerprint later in the procedure.

Uploading Public Key to SDCC Servers

1. To upload your key file, visit the SDCC’s SSH Key Upload form. You will be prompted for your Kerberos user name and password.
2. On the form, click the Browse button. In the dialog box that appears, navigate to your .ssh directory (or the directory in which your public key file is stored).
   Linux/macOS: If your browser does not display hidden directories (directories whose names are preceded by a period, e.g. .ssh), you will need to type in, or copy and paste, the name of the public key file into the dialog box. Enter the full name of the public key file as displayed in output earlier, including the full path and the .pub file extension. If you copy and paste the name of the path, take care to leave off the period at the end of the line with the public key file name.
   PuTTY/Windows: The dialog box will most likely open in the default location for the key files. If not, you will need to navigate to the appropriate folder. Once in the correct folder, select the public key file and click Open.
3. Copy and paste the fingerprint of your public key file (as displayed in output earlier) into the second box in the form, or type it into the dialog box.
   Linux/macOS: The fingerprint starts with (and contains) the SHA256:
   PuTTY/Windows: The key is comprised of 16 2-digit hexadecimal numbers, separated by colons (:)
4. To upload your key file, click the Send File button
5. You can now log in to the gateway machines using SSH keys. You will be prompted for the passphrase for your private key during the login process. This passphrase will not leave your local machine. If you wish to avoid typing in your passphrase at every login, you may wish to consider adding your keys to your SSH agent.

Using an Existing Key Pair

Linux/macOS

1. To obtain the fingerprint of an existing public key in your system’s default format, open a Terminal window and type this command: ssh-keygen -l -f [public_key_file_name] where the first argument is a lower-case L, and <public_key_file_name> is the full path to your public key file.
   This command will return the default fingerprint hash of your public key, in output similar to the following:

   2048 SHA256:PFCKP1tBWnyxxXX+GM4g+IdsJ9/upHPNzloiQLX467c you@yourhost.com (RSA)
   

   where your default fingerprint is prefixed by the text SHA256:.
   Note: If you are using a version of OpenSSH older than v6.8, the default fingerprint is in MD5.
2. Proceed to upload your key as specified above.

PuTTY for Windows

1. To load an existing key into the PuTTY Key Generator, and to obtain the fingerprint of an existing public key, click on the Load button.
2. Proceed to upload your key as specified above.

Use an Existing Linux Key Pair on Windows

1. If you have already uploaded a Linux public key to LDAP, you can use the same private key on your Windows machine by copying the Linux private key to your Windows machine, and converting the key to PuTTY format.
2. After copying the private key to your Windows machine, launch the PuTTY Key Generator, and from the Conversions menu, choose Import.
3. In the dialog box, browse to and select the private key file. As the file is imported, you will be prompted to enter its passphrase.
4. After importing the key, to save it in .ppk format and use it on your Windows machine, click Save private key.
5. It may be necessary to convert your public key as well: to do so, click Save public key.

Use an Existing Windows Key Pair on a Linux Machine

1. To use a key that was generated in Windows on a Linux machine, from the PuTTY Key Generator’s Conversions menu, choose Export OpenSSH Key, and then copy the resulting files into the .ssh directory of the Linux machine.