Learn Docker With My Newest Course

Dive into Docker takes you from "What is Docker?" to confidently applying Docker to your own projects. It's packed with best practices and examples. Start Learning Docker →

Updated on June 22, 2018 in #docker

Docker Tip #60: What Really Happens When You Run docker-compose up?

docker-tips-and-tricks.jpg

In this tip you'll learn how to look under the hood of what Docker Compose is doing whenever you run any of its commands.

Quick Jump:

Without looking at Docker Compose’s source code, it would be reasonable to think that Docker Compose might just parse your docker-compose.yml file and then transform things like services, volumes and networks into various docker commands.

And that’s not far from the truth, but Docker Compose isn’t directly calling the docker binary to run these commands. Instead, it’s using Docker’s API.

Using --verbose to Take a Peek at What’s Happening

You can attach the --verbose flag to any docker-compose command, such as running docker-compose --verbose up to get the full view of what’s happening.

High level overview of running the up command:
  • Try to load any custom Docker config files
  • Connect to the configured Docker daemon
  • Attempt to inspect or create your project’s networks
  • Attempt to inspect or create your project’s volumes
  • Search for any containers already created for your project
  • Search for any images already built for your project
  • Create your project’s containers
  • Start your project’s containers
  • Periodically inspect your project’s running containers
  • When you press CTRL+C, then stop your project’s containers
The raw verbose output:

This is the output of running docker-compose --verbose up when running an example Flask application that is part of the Docker best practices project on GitHub.

nick@workstation:/e/src/github/docker-web-framework-examples/flask$ docker-compose --verbose up
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.utils.config.find_config_file: Trying paths: ['/home/nick/.docker/config.json', '/home/nick/.dockercfg']
docker.utils.config.find_config_file: No config file found
docker.utils.config.find_config_file: Trying paths: ['/home/nick/.docker/config.json', '/home/nick/.dockercfg']
docker.utils.config.find_config_file: No config file found
urllib3.connectionpool._new_conn: Starting new HTTP connection (1): 0.0.0.0
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/version HTTP/1.1" 200 568
compose.cli.command.get_client: docker-compose version 1.21.2, build a133471
docker-py version: 3.3.0
CPython version: 3.6.5
OpenSSL version: OpenSSL 1.0.1t  3 May 2016
compose.cli.command.get_client: Docker base_url: http://0.0.0.0:2375
compose.cli.command.get_client: Docker version: Platform={'Name': ''}, Components=[{'Name': 'Engine', 'Version': '18.05.0-ce', 'Details': {'ApiVersion': '1.37', 'Arch': 'amd64', 'BuildTime': '2018-05-09T22:20:16.000000000+00:00', 'Experimental': 'true', 'GitCommit': 'f150324', 'GoVersion': 'go1.10.1', 'KernelVersion': '4.9.87-linuxkit-aufs', 'MinAPIVersion': '1.12', 'Os': 'linux'}}], Version=18.05.0-ce, ApiVersion=1.37, MinAPIVersion=1.12, GitCommit=f150324, GoVersion=go1.10.1, Os=linux, Arch=amd64, KernelVersion=4.9.87-linuxkit-aufs, Experimental=True, BuildTime=2018-05-09T22:20:16.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('flaskhello_default')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/networks/flaskhello_default HTTP/1.1" 200 557
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {},
 'Created': '2018-06-21T13:42:50.1357015Z',
 'Driver': 'bridge',
 'EnableIPv6': False,
 'IPAM': {'Config': [{'Gateway': '172.18.0.1', 'Subnet': '172.18.0.0/16'}],
          'Driver': 'default',
          'Options': None},
...
compose.cli.verbose_proxy.proxy_callable: docker info <- ()
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/info HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64',
 'BridgeNfIp6tables': True,
 'BridgeNfIptables': True,
 'CPUSet': True,
 'CPUShares': True,
 'CgroupDriver': 'cgroupfs',
 'ClusterAdvertise': '',
 'ClusterStore': '',
 'ContainerdCommit': {'Expected': '773c489c9c1b21a6d78b5c538cd395416ec50f88',
                      'ID': '773c489c9c1b21a6d78b5c538cd395416ec50f88'},
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('flaskhello_default')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/networks/flaskhello_default HTTP/1.1" 200 557
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {},
 'Created': '2018-06-21T13:42:50.1357015Z',
 'Driver': 'bridge',
 'EnableIPv6': False,
 'IPAM': {'Config': [{'Gateway': '172.18.0.1', 'Subnet': '172.18.0.0/16'}],
          'Driver': 'default',
          'Options': None},
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=flaskhello', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dflaskhello%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 3
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=flaskhello', 'com.docker.compose.service=web', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/json?limit=-1&all=1&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dflaskhello%22%2C+%22com.docker.compose.service%3Dweb%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 1305
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('flaskhello_web')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/images/flaskhello_web/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'ArgsEscaped': True,
            'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '-c',
                    'python:config.gunicorn',
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=flaskhello', 'com.docker.compose.service=web', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/json?limit=-1&all=1&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dflaskhello%22%2C+%22com.docker.compose.service%3Dweb%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 1305
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('flaskhello_web')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/images/flaskhello_web/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'ArgsEscaped': True,
            'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '-c',
                    'python:config.gunicorn',
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
compose.parallel.feed_queue: Pending: {<Service: web>}
compose.parallel.feed_queue: Starting producer thread for <Service: web>
Starting flaskhello_web_1 ...
compose.parallel.feed_queue: Pending: {<Container: flaskhello_web_1 (c79799)>}
compose.parallel.feed_queue: Starting producer thread for <Container: flaskhello_web_1 (c79799)>
compose.cli.verbose_proxy.proxy_callable: docker attach <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9', stdout=True, stderr=True, stream=True)
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "POST /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/attach?logs=0&stdout=1&stderr=1&stream=1 HTTP/1.1" 101 0
urllib3.connectionpool._new_conn: Starting new HTTP connection (2): 0.0.0.0
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker attach -> <docker.types.daemon.CancellableStream object at 0x7fc9cc552cf8>
compose.cli.verbose_proxy.proxy_callable: docker start <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "POST /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/start HTTP/1.1" 204 0
compose.cli.verbose_proxy.proxy_callable: docker start -> None
Starting flaskhello_web_1 ... done
compose.parallel.feed_queue: Pending: set()
compose.parallel.parallel_execute_iter: Finished processing: <Service: web>
compose.parallel.feed_queue: Pending: set()
Attaching to flaskhello_web_1
compose.cli.verbose_proxy.proxy_callable: docker events <- (filters={'label': ['com.docker.compose.project=flaskhello', 'com.docker.compose.oneoff=False']}, decode=True)
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/events?filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dflaskhello%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker events -> <docker.types.daemon.CancellableStream object at 0x7fc9cc5bd2e8>
web_1  | [2018-06-21 13:54:07 +0000] [1] [INFO] Starting gunicorn 19.8.1
web_1  | [2018-06-21 13:54:07 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
web_1  | [2018-06-21 13:54:07 +0000] [1] [INFO] Using worker: sync
web_1  | [2018-06-21 13:54:07 +0000] [15] [INFO] Booting worker with pid: 15
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._new_conn: Starting new HTTP connection (3): 0.0.0.0
web_1  | /usr/local/lib/python2.7/site-packages/flask/sessions.py:208: UserWarning: "localhost" is not a valid cookie domain, it must contain a ".". Add an entry to your hosts file, for example "localhost.localdomain", and use that instead.web_1  |   ' "{rv}.localdomain", and use that instead.'.format(rv=rv)
web_1  | 127.0.0.1 - - [21/Jun/2018:13:54:37 +0000] "GET /healthy HTTP/1.1" 200 - "-" "Wget" in 951µs
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
^CGracefully stopping... (press Ctrl+C again to force)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=flaskhello', 'com.docker.compose.oneoff=False']})
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Dflaskhello%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D HTTP/1.1" 200 1472
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
Stopping flaskhello_web_1 ...
compose.parallel.feed_queue: Pending: {<Container: flaskhello_web_1 (c79799)>}
compose.parallel.feed_queue: Starting producer thread for <Container: flaskhello_web_1 (c79799)>
compose.cli.verbose_proxy.proxy_callable: docker stop <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9', timeout=10)
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._new_conn: Starting new HTTP connection (4): 0.0.0.0
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],
...
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker wait <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9')
urllib3.connectionpool._new_conn: Starting new HTTP connection (5): 0.0.0.0
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "POST /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/wait HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker wait -> {'Error': None, 'StatusCode': 0}
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "POST /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/stop?t=10 HTTP/1.1" 204 0
compose.cli.verbose_proxy.proxy_callable: docker stop -> None
Stopping flaskhello_web_1 ... done
urllib3.connectionpool._make_request: http://0.0.0.0:2375 "GET /v1.36/containers/c797998210352cb9596bbd2927133bc8f8cecc25021ee918e8c8802e4e753cd9/json HTTP/1.1" 200 None
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['--reload', '-c', 'python:config.gunicorn', 'hello.app:create_app()'],
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['gunicorn',
                    '--reload',
                    '-c',
                    'python:config.gunicorn',
                    'hello.app:create_app()'],

Free Intro to Docker Email Course

Over 5 days you'll get 1 email per day that includes video and text from the premium Dive Into Docker course. By the end of the 5 days you'll have hands on experience using Docker to serve a website.

Comments