Demystifying SSH connections

I decided today that I was going to finally deal with a problem that has been bugging me for a good few months. I have two main dev setups. The first is my iMac in my office in the city and the second is my Mac Mini at home. We use Bitbucket to manage our code base at the company I work for and I make an effort to have the two environments pretty much the same (I know, I know, it's impossible to have the two exactly the same but you get my drift or my aspiration at least).

Anyway, I use SSH to push builds up to a DigitalOcean staging server and I package these various calls up into shell scripts. The problem I was meeting was that even though everything seemed to be the same on both machines, I couldn't log in as root into the DigitalOcean droplet when working from home but I could do this at the office (I'm not going to get into the rights and wrongs of logging into a system as root. It makes running commands such as chmod and chown a lot easier and if you are going to the trouble of using key-authentication SSH, then why not reap the benefits?).

Anyway, the eventual reason for this inability once I managed to track it down turned out to be pretty simple in the end but by working through the problem and using some of SSH's client and server-side debug features, I finally got a much better sense of what SSH is all about and how it works. This SSH revelation happens to me every couple of months. I then gradually forget the key points and I have to drag myself through the learning process again the next time. Anyway, to save myself the hassle and some hours next time around, and hopefully to be of some use to my reader, I am going to set out exactly how it was that I solved my problem and what I learnt about the operation of SSH in the process.

First steps — the basic setup for SSH key-based authentication

SSH key-based authentication is a mechanism to allow for secure (i.e. encrypted) communication between two machines. SSH in its OpenSSH manifestation, which is the one that a lot of servers use, is an open standard and because of that clients exist for a lot of operating systems. In my case, my client was running OS X Sierra and my server Ubuntu 14.

You can do the authentication part using passwords with SSH but that can get a bit tedious and in some cases unusable (if you are running SSH commands using scripts), so the better solution is to use keys instead. When you mint an SSH key, you create a key pair: one private, which should never be moved away from the machine that it was created on and one public, which you send to peers that you wish to communicate with.

Because of the sensitivity of the private key in particular (if someone else got hold of your private key, they could impersonate you), the key files themselves and the folders that hold them need to be properly secured. On Linux-based systems, SSH keys are usually stored in the user's .ssh folder and this needs to have its permissions set to 700 (drwx------). The public key in turn should be 644 (-rw-r--r--) and the private key 600 (-rw------).

So the first thing you want to do is to fire up the terminal on your client and cd into your ~/.ssh folder and then run the following command to create your public and private keys.

ssh-keygen -t rsa

You will be asked if you wish to enter a passphrase. This improves security but will cause a problem if you are running SSH commands in a script. Lastly, you need to specify a filename for the key pair. The default filenames are id_rsa and id_rsa.pub. Now ensure that the file permissions are correct.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa
chmod 644 ~/.ssh/id_rsa.pub

Next, you need to log into your server and setup the .ssh folder for the user that you will use to construct the SSH connection. This is a key point and it turns out that it was the reason why I couldn't log in as root previously on my home machine. When you want to SSH into a machine, you enter the following command on the machine that is establishing the connection:

ssh -i PATH_TO_PRIVATE_KEY USER@IP_ADDRESS

This tells the SSH client to use the private key specified by the -i flag to connect to the server at the specified IP address and that you should then log into the remote system as the specified user. If you enter 'root' here, that means the remote SSH server will use the root user's .ssh folder contents to handle the request.

So that means that back on the server, you should have edited the contents of the root user's .ssh folder. To do this, you will need superuser access to the server, which you can get by running the following command and entering the root user's password:

sudo su

Ensure that the root user's .ssh folder has 700 (drwx------) permissions set and if there is not an authorized_keys file, create it with 600 (-rw------) permissions. Now you need to add the public key that you created on the client to a single line in the authorized_keys file. You can do this by running the cat command on the public key and copying the output and pasting it into the authorized_keys file but just ensure that your terminals support copying and pasting and are not adding extra carriage returns or something else that might cause you a problem. Save the file and then restart the SSH daemon on the server to enact the change.

service ssh restart

Finally you are ready to try it out on the client machine. Run the following command and if all goes well, you should be logged in as root into the server.

ssh -i PATH_TO_PRIVATE_KEY root@IP_ADDRESS

If that doesn't work, it's time to dig a bit deeper

SSH key-based authentication is a classic handshaking operation and so you really need to see both sides if you are to successfully debug a problem. On the client side, add the -v flag to your ssh command as follows:

ssh -v -i PATH_TO_PRIVATE_KEY root@IP_ADDRESS

This outputs verbose debug messages to the console as the SSH client attempts to establish a connection with the server. This will include a huge amount of really useful information that will help you diagnose any problems that the connection might be having. For instance, it will tell you if the SSH client cannot find the SSH private key file that you have specified. Anyway, there's a wealth of detail here that should point you in the right direction.

The second thing that you can and should do if you are experiencing SSH connection problems is to run the SSH daemon on the server in debug mode — I appreciate that you may not have access to the server given that you are having SSH connection problems but often cloud computing operators such as DigitalOcean provide access to servers via a web-based console. It makes sense to make the debug mode SSH daemon listen on a custom port. You can run this as follows:

sudo /usr/sbin/sshd -ddd -p 33333

All going well, the console output should tell you that the daemon is listening on port 33333. Next, you need to attempt to connect again to the server from your client using the same command as before with one change — you want to specify the port number of the debug SSH daemon. You do this as follows:

ssh -v -i PATH_TO_PRIVATE_KEY root@IP_ADDRESS -p 33333

And there you have it. If you are having problems that are to do with file system permissions, if the SSH daemon is looking in one folder for files when you thought it should be looking in another folder, the SSH daemon debug messages will be able to tell you that. There's lots here that I can't deal with in this post but at least if you get this setup working you will be able to see what the problems that the SSH client and server are having. Once you know the problem, you can set about fixing it. The other advantage of viewing each step of the SSH connection process is that it helps to demystify a service that we all use every day and is so vital to setting up proper dev and sysadmin environments.