Updated on May 21, 2024 in #docker

Mount Secure Build-Time Secrets with Docker and Docker Compose

You may want to use this to avoid leaking secrets in your Docker images such as using an API key or token to pull private packages.

Prefer video? Here it is on YouTube.

There’s a number of use cases for wanting build-time secrets but let’s focus on a very common one such as wanting to install a package from a private repository.

This could be a private git repo or a private package repository such as a self-hosted version of PyPI, RubyGems, etc.. This also applies to using commercial packages where you may have an access token to install a package from their private repo.

The basic idea here is you only need that secret API key or token to obtain access to the private repo. Once the package(s) have been downloaded and installed then there’s no reason to keep that secret key around.

There’s a couple of ways to do this but some of them may result in you accidentally (or knowingly) permanently including your build-time secrets in your image which isn’t ideal.

# Using ARG Is Insecure

Let’s quickly cover how using a build ARG is an insecure way of solving this problem:

FROM debian:stable-slim

ARG ACCESS_TOKEN
RUN echo ./private-install-script --access-token "${ACCESS_TOKEN}"

./private-install-script is a placeholder for whatever command you would run to install your packages. It doesn’t exist. That’s why I used echo to ensure the image will still build successfully without that script existing.

Then when building your image you can do something like this:

export ACCESS_TOKEN="supersecretvalue123"
docker image build --build-arg ACCESS_TOKEN="${ACCESS_TOKEN}" . -t myimage

You can optionally use Docker Compose too or wire things up to read that token from an .env file so you don’t have to manually export it every time you build your image.

Seems reasonable right? You’ve added a build time secret, used it and now your image has your private package(s) installed. The problem here is that RUN layer contains your ACCESS_TOKEN in its plain text evaluated form.

If you were to build an image this way you’d be able to see the secret value in the output:

$ docker image build --build-arg ACCESS_TOKEN="${ACCESS_TOKEN}" . -t myimage
[+] Building 1.2s (6/6) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 146B
 => [internal] load metadata for docker.io/library/debian:stable-slim
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [1/2] FROM docker.io/library/debian:stable-slim
 => [2/2] RUN echo ./private-install-script --access-token "supersecretvalue123"
 => exporting to image
 => => exporting layers
 => => writing image sha256:672eead902c28f4bb3a346d56adc21620ab14be34f3d8a153cebdaf88949f1d6
 => => naming to docker.io/library/myimage

You can also run docker image history myimage and find it in the output:

$ docker image history myimage
IMAGE          CREATED          CREATED BY
672eead902c2   39 seconds ago   RUN |1 ACCESS_TOKEN=supersecretvalue123 /bin...
<missing>      39 seconds ago   ARG ACCESS_TOKEN
<missing>      2 months ago     /bin/sh -c #(nop)  CMD ["bash"]
<missing>      2 months ago     /bin/sh -c #(nop) ADD file:d43b1d5d6f0054ace...

This demonstrates that your secret is permanently stored in your image.

If you’re self hosting everything you might think that’s not the end of the world, but in my opinion it’s still very much worth protecting your secrets. What if you add a security vulnerability scanning tool and it scans your image? Now they have access to your secret. Needlessly copying around secrets makes it easier to accidentally leak something.

Don’t feel too bad though, years ago before mounted secrets were available I did the above. I knew it was an issue long term but it was an easy solution that at least avoided committing the secret to your git history. I also only ever used it for private repo access tokens which are easy enough to re-roll.

Nowadays we can have the best of both worlds. Ease of use and it being secure.

# Securely Mounting Temporary Secrets

Before anything, it’s worth pointing out you must have BuildKit enabled. If you’re using Docker v23.0+ (Feb. 2023) on any platform then it’s enabled by default. If you’re using an older version of Docker you can likely enable it in a few ways, Docker’s docs cover that.

A slightly modified Dockerfile from the above example:

FROM debian:stable-slim

RUN --mount=type=secret,id=ACCESS_TOKEN \
    echo "./private-install-script --access-token $(cat /run/secrets/ACCESS_TOKEN)"

• Instead of using an ARG we’re using --mount within our RUN instruction
  • This secret will only be available for this specific RUN instruction
• type=secret says we want it to be a secret, other types are in Docker’s docs
• id=ACCESS_TOKEN lets us name our secret anything we want (we’ll reference it later)
• $(cat /run/secrets/ACCESS_TOKEN) lets us access the secret by its id
  • Mounted secrets end up in /run/secrets, catting it lets us get its value

Then we can build our image:

export ACCESS_TOKEN="supersecretvalue123"
$ docker image build --secret "id=ACCESS_TOKEN" . -t myimage
[+] Building 1.0s (6/6) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 306B
 => [internal] load metadata for docker.io/library/debian:stable-slim
 => [internal] load .dockerignore
 => => transferring context: 2B
 => CACHED [stage-0 1/2] FROM docker.io/library/debian:stable-slim
 => [stage-0 2/2] RUN --mount=type=secret,id=ACCESS_TOKEN     echo "./private-install-script --access-token $(cat /run/secrets/ACCESS_TOKEN)"
 => exporting to image
 => => exporting layers
 => => writing image sha256:b3e758370af54e7cc1612b33bb349bd8a5905a85276e70998a955899e27e4cdc
 => => naming to docker.io/library/myimage

• --secret "id=ACCESS_TOKEN" must match the id in the Dockerfile
  • Optionally we can set id=x,env=ACCESS_TOKEN to have an id that is different than the env var name, in this case you’d reference id=x in your Dockerfile
  • Personally I tend to avoid setting env= since using the same name is intuitive
• Notice how the secret value itself isn’t output anywhere

And here’s the history:

$ docker image history myimage
IMAGE          CREATED          CREATED BY
cc8bd26ab409   20 seconds ago   RUN /bin/sh -c echo "./private-install-scrip...
<missing>      2 months ago     /bin/sh -c #(nop)  CMD ["bash"]
<missing>      2 months ago     /bin/sh -c #(nop) ADD file:d43b1d5d6f0054ace...

You might think I’m lying because the value is truncated. You can run the above command with --no-trunc to avoid truncating the output, it will look messy here but here’s the non-truncated first line of output that includes our private command RUN /bin/sh -c echo "./private-install-script --access-token $(cat /run/secrets/ACCESS_TOKEN)".

Does It Really Work?

Given we’re not really doing anything in our RUN instruction (it’s a dummy echo command), how can we be sure things are really working?

I don’t suggest ever using this Dockerfile for anything other than local testing but here’s a quick way to verify that things are truly working. It’s going to completely defeat the purpose of using the secret mount:

FROM debian:stable-slim

RUN --mount=type=secret,id=ACCESS_TOKEN \
    cat /run/secrets/ACCESS_TOKEN > /tmp/dontdothisnormally

CMD ["cat", "/tmp/dontdothisnormally"]

Rebuild with docker image build --secret "id=ACCESS_TOKEN" . -t myimage and then:

$ docker container run --rm myimage
supersecretvalue123

Yep, it’s really being set! As an experiment you can try catting /run/secrets/ACCESS_TOKEN in the CMD and you’ll be happy to know it’s not accessible. You also can’t use --mount with CMD which makes sense since mounting is available for RUN.

What if my tool expects an environment variable to be set?

In our example we did ./x --access-token $(cat /run/secrets/ACCESS_TOKEN), having it set as an environment variable wouldn’t be any different than how you normally set them, such as ACCESS_TOKEN=$(cat /run/secrets/ACCESS_TOKEN) ./x.

You can also make it look nicer by formatting it like this:

RUN --mount=type=secret,id=ACCESS_TOKEN \
  ACCESS_TOKEN=$(cat /run/secrets/ACCESS_TOKEN) \
    ./x

What about multiple secrets?!

No problem, Docker has you covered:

RUN \
    --mount=type=secret,id=ACCESS_KEY \
    --mount=type=secret,id=ACCESS_TOKEN \
        ACCESS_KEY=$(cat /run/secrets/ACCESS_KEY) \
        ACCESS_TOKEN=$(cat /run/secrets/ACCESS_TOKEN) \
          ./x

Then you can build it with:

export ACCESS_KEY="somethingsecure"
export ACCESS_TOKEN="supersecretvalue123"
$ docker image build --secret "id=ACCESS_KEY" --secret "id=ACCESS_TOKEN" . -t myimage

What about mounting files, not environment variables?

This could be handy if you have something like a ~/.pypirc file or some type of config file that has sensitive API keys or tokens that is used for a command:

RUN --mount=type=secret,id=pypirc,target=/root/.pypirc \
  echo twine upload dist/*

• target= lets us define where this file will exist in our image
• That echo line demonstrates uploading a Python package, it’s not important here

Then we can build it with:

$ docker image build --secret "id=pypirc,src=${HOME}/.pypirc" . -t myimage
[+] Building 1.0s (6/6) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 600B
 => [internal] load metadata for docker.io/library/debian:stable-slim
 => [internal] load .dockerignore
 => => transferring context: 2B
 => CACHED [stage-0 1/2] FROM docker.io/library/debian:stable-slim
 => [stage-0 2/2] RUN --mount=type=secret,id=pypirc,target=/root/.pypirc   echo twine upload dist/*
 => exporting to image
 => => exporting layers
 => => writing image sha256:78d52c314c41ce45593e30d9c502c7a8105f6190e6ca84653277b94551bdc397
 => => naming to docker.io/library/myimage

• src= is where the file exists on your Docker host running the build command

If you want to confirm it works you can do the same tactic we used in the “does it really work?” section of this post.

You can also mount multiple files in the same way we mounted multiple environment variables. Just give each of them a unique id along with a src and target and you’re good to go.

# Docker Compose

In a larger project you might have a few secrets and are using Docker Compose. Here’s a few examples that convert the above into Docker Compose’s format.

Accessing Environment Variables

First, here’s an example Dockerfile. Going back to what we did earlier, we’re only catting out this file to demonstrate our secrets are being used. Don’t do this in a real project!

Nothing about this file is specific to Docker Compose by the way:

FROM debian:stable-slim

RUN --mount=type=secret,id=ACCESS_KEY \
    --mount=type=secret,id=ACCESS_TOKEN \
    echo "$(cat /run/secrets/ACCESS_KEY) | $(cat /run/secrets/ACCESS_TOKEN)" \
      > /tmp/dontdothisnormally

CMD ["cat", "/tmp/dontdothisnormally"]

Then we configure docker-compose.yml to set up our secrets. Notice the indentation of the services.myservice.build.secrets property, it’s important that it’s a build property:

services:
  myservice:
    build:
      context: "."
      secrets:
        - "ACCESS_KEY"
        - "ACCESS_TOKEN"

secrets:
  ACCESS_KEY:
    environment: "ACCESS_KEY"
  ACCESS_TOKEN:
    environment: "ACCESS_TOKEN"

Here we’re taking advantage of defining environment based secrets which is handled near the bottom of the file. That environment property is the name of the env var that will be read from your Docker host. The dictionary key name is the id of the secret.

Lastly, optionally create an .env file and Docker Compose will automatically read from it. If you don’t do this step then you’ll need to have these env vars exported in your current shell or your secrets will be empty:

export ACCESS_KEY=somethingsecure
export ACCESS_TOKEN=supersecretvalue123

Now you can see it all work:

$ docker compose build && docker compose run myservice
somethingsecure | supersecretvalue123

Accessing Files

This is the same Dockerfile as before in our file example:

FROM debian:stable-slim

RUN --mount=type=secret,id=pypirc,target=/root/.pypirc \
  echo twine upload dist/*

Then we configure docker-compose.yml to set up our secret:

services:
  myservice:
    build:
      context: "."
      secrets:
        - "pypirc"

secrets:
  pypirc:
    file: "${HOME}/.pypirc"

The only real difference from the other Docker Compose example is we’re doing a file based lookup near the bottom of the above file. Like the Docker command, that path is looked up on your Docker host.

Now you can see it all work:

$ docker compose build && docker compose run myservice
root@e491336de8f2:/#

Technically there’s nothing to see and it’ll drop you into a shell session since that’s what the Debian image does by default. You can always override the CMD and cat the file out to confirm it works (it does).

I’ve only covered a few common use cases here. Feel free to check the documentation for other options. You can even mount an SSH key. I haven’t used that because I typically use personal access tokens for images that need to clone private repos, in which case the above solutions work.

The video below demonstrates running all of these commands.

# Demo Video

Timestamps

• 0:37 – A couple of use cases
• 1:43 – An insecure way with build ARGs
• 6:14 – Make sure BuildKit is enabled
• 6:49 – Mounting a secure env variable secret
• 11:26 – Does it actually work?
• 14:04 – Passing it as an env var for a command
• 15:12 – Adding multiple secrets
• 16:02 – Mounting files instead of env vars
• 18:43 – Using Docker Compose for secret env vars
• 23:13 – Using Docker Compose for secret files and configs

How do you use Docker secrets? Let us know below.