Zero Downtime Deployments

My philosophy is to use simple, boring, open software. You won’t find me volunteering to use the (fancy) cloud unless I have a REALLY good reason. Websites have become more obese, not just in relation to their size, but also complexity, which includes deployment. I don’t believe it has to be like this.

Anyway, recently I’ve been getting into using systemd. I had a http server running on my machine and wanted it to automatically restart on crashes. My previous while true; do ./run; sleep 5; done script running inside of tmux wasn’t going to last forever.

I also started using systemd for tailing logs (journalctl), which worked great (REF).

The next thing I wanted to look into is how zero downtime deployments could be achieved without using a reverse proxy.

(Almost) zero downtime deployments

The simple solution (just restarting) worked really well. So well, in fact, that I didn’t need to do try to get all the way down to 0ms downtime. I have a tiny website that nobody but me uses. Literally nobody cares if the websites is down for 50ms each time I need to restart it.

zero downtime, with delayed requests

The simplest way of implementing zero downtime is using a reverse proxy. The proxy routes to localhost:8000 if up, otherwise it holds all requests until it's available again.

handle /api/* {
    reverse_proxy localhost:8000 {
        lb_try_duration 60s
        fail_duration 1s
        health_uri /health
        health_interval 1s
    }
}

I would’ve left it there, but one day I heard somewhere that you can use systemd for zero downtime server deployments. My curiosity peaked - I wanted to know how it works.

zero downtime, two servers at once

Systemd includes a feature called socket activation, which lets you specify which ports you want systemd to listen to on your behalf.

All you need is two files: server-https.socket and server.service.

The server-https.socket looks like this:

[Socket]
ListenStream = 0.0.0.0:443
BindIPv6Only = both
FileDescriptorName=https

[Install]
WantedBy = sockets.target

And the service:

[Unit]
Description=Server Service
After=network.target
Requires=server-https.socket

[Service]
Type=simple
ExecStart=...
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Now all you need to do is to enable them via (systemctl enable --now server-https.socket server.service).

You can also use systemd-socket-activate.