Continuous deployment for this website using GitLab CI/CD, SSH, Docker and systemd

(Last modified on )
#blog #docker #devops #gitlab #ssh #systemd

Almost two years ago I wrote that ideally I would not have to log in to my VPS to update this website. Well, that moment has finally arrived.

A couple of weeks ago I decided to pursue continuous deployment for this site. Not because it is such a hassle to deploy a new version myself and also not because it is needed that often, but because I wanted to explore the concept.

To summarize the most relevant parts of what I wrote in 2019:

  • The source code for this blog is hosted on GitLab.
  • Whenever I push a commit, GitLab CI/CD builds a new Docker image.
  • Once the build is done I SSH into the VPS this site is hosted from, pull the new image and use it to run this site.

That last, manual step is now automated. The hardest part was figuring out a method I was happy with; that is: not putting the keys to the kingdom in GitLab. Not that I distrust GitLab, but if someone would get access to my GitLab account, I would not want them to also have unlimited access to the VPS.

The solution I ended up with consists of three parts:

  • Configuration on GitLab to trigger a deployment.
  • A user on the VPS so the GitLab job can log into the VPS.
  • A monitoring service on the VPS to redeploy when a trigger is detected.

GitLab configuration

This part is basically a summary of what David Négrier wrote in his article Continuous delivery with GitLab, Docker and Traefik on a dedicated server. If you want a more detailed explanation, I can highly recommend reading his article.)

Before we can get into the job that I added to my pipeline, we need to prepare some things. Starting with adding a couple of GitLab CI/CD variables:

  • SSH_HOST and SSH_PORT: the SSH client needs to know how to connect to the VPS.
  • SSH_USER and SSH_PRIVATE_KEY: the SSH client needs authentication information.
  • SSH_KNOWN_HOST: the public SSH key of the server (SSH_HOST) so we can add it to the known_hosts file and prevent man-in-the-middle attacks. I got this value by running ssh-keyscan <hostname> on my laptop and pasting the output in GitLab.

In a moment we’ll see how these variables get used.

Since I want to use the digest of the Docker image that is built in this pipeline, I’ve added an artifact to store the digest so we can access it later on:

build_image:
  script:
    ...
    - docker image ls --filter "label=org.label-schema.vcs-url=https://gitlab.com/markvl/blog" --filter "label=org.label-schema.vcs-ref=$CI_COMMIT_SHA" --format "{{.Digest}}" > image-sha.txt
  artifacts:
    paths:
      - image-sha.txt

(The filters are probably not really necessary, but just in case there are multiple images present, I want to be reasonably sure that I’ve picked the right one.)

Now finally the job that triggers the deployment:

deploy_image:
  stage: deploy
  only:
    refs:
      - main
  services: []
  image: alpine:latest
  script:
    - apk add --no-cache openssh
    - mkdir ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/private_key
    - chmod 600 ~/.ssh/private_key
    # add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
    - eval $(ssh-agent -s)
    - ssh-add ~/.ssh/private_key
    - ssh -p $SSH_PORT $SSH_USER@$SSH_HOST $(cat image-sha.txt)

Most of the code is just to get SSH working. All the magic happens in the last line. Note that in contrast to David’s article I don’t actually execute commands on my VPS, instead I only send one string.

(For the full .gitlab-ci.yml file see the GitLab repo for this site.)

Now every time the pipeline is run on the main branch, the digest of the freshly built Docker image is sent to my VPS.

VPS SSH configuration

On the VPS we need to make sure that the GitLab job can SSH into the machine.

The first step is to create a user to be used by GitLab (the SSH_USER variable I mentioned above). Next we need to make sure that the SSH_PRIVATE_KEY stored in GitLab can be used to log in. To make this possible and to mitigate the risks of the SSH key in GitLab getting abused, I have added the following content to the file ~/.ssh/authorized_keys of the new user:

command="/home/<username>/.ssh/ssh_commands.sh",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding <public key> <comment>

Using the command option is an idea I got from a ServerFault answer and Mauricio Tavares' article Getting the SSH_ORIGINAL_COMMAND. In my case the ssh_commands.sh file stores the original command (in my case the digest) in a file called deployment.raw.

VPS monitoring service

To actually deploy the new image, we need just one more piece in this puzzle: a script to pull and use the Docker image.

I’ve opted for a systemd unit to monitor for the existence of the deployment.raw file by adding the file /etc/systemd/system/blog-deployment.path (note the “.path” at the end of the filename):

[Unit]
Description=Blog deployment path monitor
Wants=blog.service

[Path]
PathExists=/<path>/<to>/deployment.raw

[Install]
WantedBy=multi-user.target

This systemd unit configuration file is accompanied by the following service file (/etc/systemd/system/blog-deployment.service):

[Unit]
Description=Blog deployment service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/deploy_blog.sh

[Install]
WantedBy=multi-user.target

In the deploy_blog.sh script I do things like reading the deployment.raw file, checking its content, downloading the new Docker image, checking it and restarting this website with the new image.

Summary

To recap my continuous deployment solution:

  • I push a commit to the main branch of the repo of this site.
  • GitLab CI/CD builds a new Docker image and sends its digest to my server.
  • My server watches for the existence of the digest file and uses it as a trigger to deploy the new version of this website.

And now that I’ve written this down, I’m going to commit this article to Git, push it to GitLab and then sit back and wait (im)patiently for my website to update itself. ;-)