In my previous post, Caching Docker Rails images. Please read it first. This post is a continuation describing new exciting Docker (aka Moby) addition: volume mounts during build, added in buildkit#442. It’s part of Buildkit project, future replacement for current Docker build engine. You can learn more about Buildkit by reading Tõnis Tiigi’s post, Introducing BuildKit.
To update source code in the repository my initial approach from previous post uses
COPY command has one fundamental problem for this use case.
COPY works similiarly to Unix
cp command - it only adds new files.
It is not capable of doing full sync and removing files from image when they are removed in the source.
Thanks to the latest update to Buildkit in buildkit#442 it is now possible to mount a volume during mount, including local filesystem. Combined with
rsync real syncing is finally available.
This feature is still in the proposal stage and there’s no stable version available. In order to build images you need to use custom
dockerd version. The good news is that images built this way are fully compatible with current stable Docker. This means that you only need the custom setup on a machine that creates images. However as it potentially is unstable use it carefully.
Preparing Docker for building images
In order to build Docker image using new feature you have to install Docker version that includes the latest change from buildkit#655.
The easiest way to do that is to build Docker yourself from source code. I use
master branch version to include the latest patches. Building process itself is very easy as Moby actually uses Docker to bootstrap itself and make the process even easier.
mobyrepository. What is Moby? The Moby Project is to Docker what Fedora is to Red Hat Enterprise Linux. - Solomon Hykes, Docker CTO/Founder
git clone firstname.lastname@example.org:moby/moby.git
- Run make to compile dockerd and all related processes. Make sure that you have running Docker. Moby actually uses Docker to build itself, so you don’t have to worry about any dependencies.
After the process finishes new binaries are available at
bundles/binary-daemon. These are portable go binray files, so you can install them by copying or adding to
$PATH. It’s important to copy all the binaries, not only
dockerd as they are direct dependencies.
The final step is to start
dockerd in experimental mode. In order to do that add
To save your time I created a Vagrantfile with all necessary steps included.
Vagrant.configure("2") do |config| config.vm.box = "ubuntu/xenial64" config.vm.provider "virtualbox" do |v| v.memory = 2048 # additional memory is required end config.vm.provision "shell", privileged: false, inline: <<-SHELL sudo apt-get update sudo apt-get install -y \ apt-transport-https \ ca-certificates \ curl \ software-properties-common \ make curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" sudo apt-get update sudo apt-get -y install docker-ce sudo gpasswd -a vagrant docker git clone https://github.com/moby/moby.git cd moby sg docker -c "make" sudo service docker stop sudo bundles/binary-daemon/* /usr/bin sudo mkdir -p /etc/systemd/system/docker.service.d/ echo -e "[Service]\nExecStart=\nExecStart=/usr/bin/dockerd -H fd:// --experimental" | sudo tee /etc/systemd/system/docker.service.d/override.conf > /dev/null sudo systemctl daemon-reload sudo service docker start SHELL end
Building images using RUN –mount
Buildkit is not yet default Docker’s builder. To use it you need set
DOCKER_BUILDKIT flag when running
Additionally as new
RUN --mount syntax is still in experimental phase
Dockerfile has to be prepended with a directive:
# syntax = tonistiigi/dockerfile:runmount20180618
To check that it works correctly create a
Dockerfile with content:
# syntax = tonistiigi/dockerfile:runmount20180618 FROM alpine RUN --mount=type=bind,source=.,target=/mounted-source cp /mounted-source/Dockerfile /copied-Dockerfile
DOCKER_BUILDKIT=1 docker build .
You will notice slightly different output of buildkit compared to standard Docker build engine.
Cached Dockerfile update
Finally, with everything set up, we are able to tune
Dockerfile.production.simplified.cached from the previous post.
COPY . . can be replaced with:
RUN apk add --no-cache --update rsync RUN --mount=type=bind,source=.,target=/mounted-app rsync -ra /mounted-app/ /app