Resilient SSH with autossh and/or systemd

SSH is a great tool, but it’s not much use to use when the connection craps out. This is often OK as you can just “ssh foo@bar” again, but it is a major problem if you were relying on a reverse tunnel (“ssh -R 1234:localhost:22 foo@bar”) and the machine is 2,000+ miles away. I found myself is just such a situation due to a rather poor router supplied by an ISP. I won’t mention any double-barrelled names due to the chilling effect of English libel laws on the freedom of speech.

To get around this I had a RasPi create a reverse tunnel back to a server here. Once connected, I could remote administer and set up other tunnels for services I might need (router access, RDP etc). This works well, but due to the appalling Internet connection, things would crap-out and I’d have to walk a user through re-starting it. Not a great solution as the users have zero IT knowledge.
The obvious answer is to switch to passwordless certs and start autossh at boot (“autossh -M 0 -f -N -T -q -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -R 1234:localhost:22 foo@bar”). autossh Will deal with the connection crapping-out and restore SSH.

Great…but how to run that at boot? One option is init.d and that’s probably the way to go on many older or stable distributions. I didn’t use that as it wasn’t appropriate in my case and I have always struggled to get init.d scripts to run at the right time during boot. No point in trying to SSH before the network is up, for example. systemd to the rescue!

[Unit]
 Description=AutoSSH service for a reverse tunnel from foo to bar
 After=network.target

[Service]
 User=foo
 ExecStart=autossh -M 0 -f -N -T -q -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -R 1234:localhost:22 foo@bar

[Install]
 WantedBy=multi-user.target

Save that as /etc/systemd/system/foo-autossh.service and run “systemctl daemon-reload && systemctl start foo-autossh”. This looks pretty good. It’ll start autossh after networking is ready and you should be able to connect, right? Wrong. This won’t work. If you check via “systemctl status foo-autossh” you’ll see autossh thrashing as it starts, ssh starts, exits with a code 15 and then gets started again by autossh. Continually.

This is because the “-f” flag does not work with systemd. Maybe that’s a bug in systemd or autossh, maybe it’s just something to do with trying to run as a daemon; I don’t know. The fix is rather easy though:

[Service]
 User=foo
 Environment="AUTOSSH_GATETIME=0"
 ExecStart=/usr/bin/autossh -M 0 -N -T -q -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -R 1234:localhost:22 foo@bar

“systemctl daemon-reload && systemctrl start foo-autossh” and BOOM, your ssh connection is up. I initially tested this by having it create a second tunnel and then killing the relevant processes, before moving it to the desired settings and doing a reboot.

There’s also another way; that’s to use plain SSH and have systemd fuss with the restarting.

[Service]
 User=foo
 ExecStart=/usr/bin/ssh -N -T -q -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -R 1234:localhost:22 foo@bar
 Restart=always
 RestartSec=60

What this will do is have systemd itself re-run the SSH should the process exit.

Which is better? Depends really. autossh knows all about ssh but has little clue on the system. systemd knows all about the system, but (auto)ssh is just another PID. You will almost certainly find there are subtle differences in behaviour that will be desirable in some situations an not in others (e.g. systemd’s start limit). I’m going with the first form just now to see how things go. Don’t forget to enable your new service at booth with “systemctl enable foo-autossh”.

You could go all belt-n-braces and combine the two:

[Service]
 User=foo
 Environment="AUTOSSH_GATETIME=0"
 ExecStart=/usr/bin/autossh -M 0 -N -T -q -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -R 1234:localhost:22 foo@bar
 Restart=always
 RestartSec=60

Finally, that’s a long-ass autossh string; simplify it with a ~.ssh/config file

Host bar_tun
        HostName        bar
        User    foo
        IdentityFile    /home/foo/.ssh/id_foo
        RemoteForward   1234 localhost:22
        ServerAliveInterval     30
        ServerAliveCountMax     3

And then update the systemd service:

[Service]
 User=foo
 Environment="AUTOSSH_GATETIME=0"
 ExecStart=/usr/bin/autossh -M 0 -NTq -F /home/foo/.ssh/config bar_tun

Any further tips or tricks? Let me know in the comments below.