How to run dockers on CI
No matter what project you have, there’s a high chance that at some point, you will want to test it in connection with other components/services.
The best way to do such a setup is to use docker or even docker-compose. As nobody wants to waste time running all the tests after each commit, the state of art is to have a continuous integration server that will take this burden from us.
But do we really know how to run containers on CI?
Note: There are many different CI servers and even a single one can be deployed and configured in countless ways. Depending on CI you use, the things I am describing here might or might not be applicable for you. My current setup is Jenkins on a physical machine with two agents, each one with its own machine.
Local setup
Before we start let’s take a look at the example docker-compose file for a local setup:
This will start two containers A and B and bind their ports to the respective ones on the host machine.
The naive approach would be to run this docker-compose directly on the CI, but as soon as we try to run more than one build concurrently, we will get an address already in use exception. That was to be expected.
The first job binds docker containers to the host’s ports and the next one cannot do that since they have already been bound. Since docker-compose files can be parameterized by environment variables, we could think about some ports randomization, but the truth is: we shouldn’t be binding to the host’s ports on our CI at all.
If we shouldn’t bind, then what?
Network investigation
The default docker-compose behavior is to create a single network in a bridge mode that wires all containers together. That means that they are isolated from the outside world by this network. So why can’t we just use the default behavior and remove binding?
We can, but we also have to have a way to access these containers from tests (at least sometimes, e.g. when they are mocks we might want to instrument them). Luckily, there is an easy way to achieve that.
The default network has a name (by default it is a name of the parent directory of the docker-compose file with _default postfix e.g. parent_directory_default), and we can connect to it. All we have to do is to run tests within another docker container and explicitly connect it to this network.
Assuming that our tests start by calling tests.sh it could look as follow:
Note: As we run tests inside the container we need to give it access to the source code, hence the directory binding.
Obviously we won’t be doing that locally, as we want to start our tests from IDE, therefore, we need a separate docker-compose with port bindings for local development.
Is that simple?
If we run multiple builds concurrently we won’t notice previous errors anymore, but it is very likely that tests start to fail randomly. If we take a closer look we will notice that we always use the same network and if two builds end up on the same machine, they will interfere (they won’t report any errors but they will silently reuse containers between themselves). We already know that we can influence docker-compose from outside by templating it. We could pass some unique build id to randomize the network name like that:
Notice that by the separation of the name and the identifier we have to template it only in a single place
Will it work?
Not necessarily. If you try it you will sometimes see a host unreachable exception. That is because a docker network is not a namespace.
We have randomized networks, but containers identifiers are still the same across builds and the containers will get reused. If we run two builds concurrently we will create two networks, but only a single set of components containers. Since our tests are bound to a particular network one of these builds will fail because tests won’t be able to reach components (components will be using another network).
We can go further with manual randomization as containers can have custom names like a network but with the growing amount of containers, this will become more and more cumbersome.
In the search for a namespace
What we actually need is to run all the things defined within a single docker-compose file in some kind of isolated namespace which would take care of all individual components isolation (both containers and networks).
Surprisingly, there is such a thing in docker-compose. Whenever you start a docker-compose it prepends the parent directory name to all its components which can be perceived as a primitive but working namespace implementation. We can alter this prefix by changing the — project-name parameter when starting docker-compose.
This will add ${BUILD_ID} prefix to both our containers’ names and networks’ names.
And that’s the whole story
One last thing you should keep in mind is not to forget to pass this parameter when you take your containers down, so you won’t end up with many zombie containers.
