We at adjust recently started to use Elixir. We built a couple of small services using the Phoenix framework which successfully went live. In this blogpost I’d like to talk about, I’d say, the most undiscussed topic when it comes to Elixir — deployment.
Although you can find some blog posts about deploying Elixir applications, usually after reading them, it still remains unclear how to get the desired command which would deploy your code to production - and which would automate all the routines.
The first thing we’ve tried was
mina. I’d say, trying to use
Mina is an obvious choice if you come from the Ruby world. However, it becomes clear very quickly that the Capistrano way doesn’t fit well for Elixir apps. As you probably know, the preferred way to deploy Elixir applications is to use releases, which means you need a place where a release should be built. It’s possible to write a
Mina recipe to clone a project to the production host and build the release there, but that wouldn’t be very good idea. Compiling and building a release will take some resources (especially memory) which you don’t want to share on production.
Another option would be to build a release locally using the cross-compiling feature and copy it to production. There are a few gotchas with such approach:
- there might be some differences in environment (dependency versions, elixir version, etc) between different developers’ computers, so two developers might build two different builds based on the same codebase;
- it would be quite tricky to write such a recipe for
Capistrano(although much easier for
Mina); generally, using Capistrano just to copy one tarball to a server, unpack it and start it looks like overkill.
So using releases means that there should be a machine where every developer can build a release. Right, a build server! And the problem is that the concept of a build server isn’t something familiar for
Mina. So there should be a tool which is aware of the concept of a build server, which maybe even knows how to work with Elixir releases…
Thankfully such a tool does indeed exist.
Edeliver is a deployment tool for Elixir and Erlang projects. It knows how to work with releases and how to apply hot-upgrades, it’s aware of a build host and helps you to automate the deployment workflow.
Edeliver has very good and comprehensive documentation, including several wiki pages describing some edge cases as well. I don’t want to review
edelivers README in this blogpost, but rather I’d like to cover some of those edge cases and gotchas which we’ve discovered while using it.
There is a small issue with release names — they must be unique, so every time the
mix edeliver build release command finishes, a unique release should be generated.
Edeliver solves this issue by having a special config parameter with which it’s possible to append a Git revision, Git branch, build date, etc to a release name. So you don’t need to go to the
mix.exs file and change
project/0 function –
edeliver does it for you. We found that
AUTO_VERSION=git-branch+git-revision generates sufficiently unique release names. With this combination a release name would be something like “awesome_adjust_app_0.0.1+master-01b4601.release.tar.gz”.
edeliver provides only two environments to which it’s possible to deploy — staging and production. There is no easy way to add custom environments, but as it turned out it’s still possible to achieve that by overriding
STAGING_USER variables in
Let’s say we want to add
qa environments. To do so
.deliver/config should look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
As you can see, the
ENVNAME_NODES variables should be added and then based on
$DEPLOY_ENVIRONMENT, staging related variables should be overridden.
Also, it’s important to add the
.deliver/help file where these new environments should be added:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
With this config, it would be possible to deploy a release to the
qa hosts (in addition to
production) and to maintain these custom hosts. For example, in order to check the version of the
beta host, you’d run a command like this:
mix edeliver version beta.
It’s quite common to send notifications about successful deployments. For example, we might display such notifications in a Slack channel.
edeliver has hooks which can be implemented as bash functions. For example, there are two hook functions:
post_upgrade_release(). They are called exactly before applying an
upgrade and right after an
upgrade has been applied, respectively. Notifications about deployment usually contain information about the person who deployed, the Git branch and revision, and the environment name (staging/production).
The issue here is that you can’t get a Git branch and Git revision out of a release since a release is just a binary. With Capistrano, you can just run a couple of git commands on the target host to get the necessary data. With
edeliver it becomes a bit more tricky. The current workaround we use is to include the Git revision and Git branch into a release name using the following config:
AUTO_VERSION=git-branch+git-revision. This is as I described in the previous section on Auto-Versioning. Then in the project itself a
Notifier module might look as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
post_upgrade_release() hooks might look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
However, there are two flaws here. First, it works only when applying upgrades - not for releases. And second, when calling
Edeliver.release_version returns a git revision of the currently deployed release. So ‘deploying’ notification would have a git revision of the currently deployed version and the ‘deployed’ notification would have a git revision of the new version.
Different configurations on different deploy hosts
Most probably, your application has different settings for staging and production environments. Which means that you need either to build a release for each environment separately or somehow provide different settings on different hosts for the same release.
Edeliver, following a philosophy “build once, deploy everywhere” suggests to solve this problem by using
LINK_VM_ARGS config variables as described on this wiki page.
I’ll describe briefly how it works with
LINK_VM_ARGS variable. The logic is the same for
LINK_SYS_CONFIG. So it works as follows: you need to create a file which should have the same path on both
production hosts with config values specific for the target host. This could be
/home/deploy_user/my_app/vm.args, for example. Then in
.deliver/config you can specify
When making a release or an upgrade,
edeliver would put a symlink inside a release (instead of the real generated
vm.args) which will point to
/home/deploy_user/my_app/vm.args. So this tricky and sophisticated approach solves the issue. In theory. I couldn’t actually make it work. After a release deployment I see a symlink as expected, but on release start, my custom symlinked
vm.args file should replace
running-config which does not happen. However, if I remove the
running-config folder first and start a release afterwards, it works.
So since this approach didn’t fully work, we decided to build a release per environment, which is also suboptimal:
- you need to build a release per environment
- it violates a release philosophy: build once, deploy everywhere
- error prone: somebody can by mistake deploy a release on production, which has been built for staging
To partially fix the last bullet from the list above it’s possible to add a
mix-env parameter to the
AUTO_VERSION config value:
AUTO_VERSION=git-branch+git-revision+mix-env. So every build would have
-environment in its name to indicate for which environment a release has been built.
Usually, for Phoenix applications secret production settings (like database connection credentials for production DB) are stored in
prod.secret.exs. This file is not under version control, but it should be inside a release. To achieve that you might want to put this file manually into the build host, but the issue here is that a folder where a project is built is cleaned by
edeliver before every release build. The ‘cleaning’ means that everything which is not under version control will be removed before every build, so
config/prod.secret.exs will be gone. To avoid that there is an option to explicitly instruct
edeliver which folders should be cleaned. Having the config option
GIT_CLEAN_PATHS="_build rel deps" tells
edeliver to clean
deps folders before every release build, so
config folder stays untouched and therefore
prod.secret.exs stays alive between release builds.
Bonus: Change font color output
For light terminal themes
edeliver output by default looks as follows:
There is an option to change that by overriding the color of the font:
1 2 3
With the fix the output looks as follows:
Currently, there are not so many alternatives to
edeliver. But there is at least one: dicon. It’s in the early stages of development, it doesn’t have comprehensive readme, it’s not aware of build host and it does not support hot-upgrades yet. However,
Digital Conveyor has some niceties: it’s written completely in Elixir, it’s small and it supports configurations per target host out of the box. It will be interesting to see how
dicon will be evolving.
Edeliver is a great, ready-to-use deployment tool packed with a lot of useful features. It works with releases, supports hot-upgrades and build host concept, has very good documentation and gives you simple commands to automate deployment routines. Importantly, the project is in active development. I’d like to thank bharendt for amazing support, almost every tip or trick I’ve described in the post is a result of a detailed answer from him to an opened issue. Sometimes I had a feeling that I’m literally chatting with him in the
Issues tab, that’s amazing.
That’s it for today. Happy deploying!