Docker Compose and Volumes

In the first post of the container series Linux Containers and Docker, I introduced Linux Containers and Docker, as well as how to use the Docker CLI and basic management of containers. In this second post of the series I turn my focus to my favorite Docker management tool, Docker Compose.

Docker Compose is a container management tool for Docker, written in Python, and it is a game changer. While managing a single container with Docker is doable, throwing any additional containers in the mix quickly makes management tedious. With Compose, a YAML file is written for configuration, and a simple CLI interface is used to manage almost the entire life cycle of your containers.

Installation

Let’s begin by installing Compose

# apt update && apt install -y docker-compose

Note: The Docker Compose package in Ubuntu 18.04, currently version 1.17.1-2, lags behind the current release, 1.25.4. If you would prefer a newer version of Compose, I would recommend using Ubuntu 19.10 as a base, which currently includes Compose version 1.21.0-3. Alternatively you can directly download the binary if you would like the absolute latest version of Docker Compose as seen in the docs:

$ curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ chmod +x /usr/local/bin/docker-compose

Note: replace the version number with the latest version number.

YAML

In a bit we’ll be writing a docker-compose.yaml file so I thought I would briefly introduce YAML for those not familiar. YAML, (YAML Ain’t Markup Language) is a data-serialization language, commonly used for configuration files. YAML is a superset of JSON, trading curly brackets for Python-style indentation to indicate nesting. YAML is generally easy for humans to read, but any stray spaces or indentations can cause issues.

Volumes

At this point before continuing with Compose, I’d like to introduce another Docker concept, volumes. Volumes allow for persistent data storage and are managed completely by Docker. I introduced a related topic in the last post, bind mounts, which are dependent on the directory structure of the host machine as seen here:

$ docker run -d --name nginx -p 8080:80 -v ~/index.html:/usr/share/nginx/html/index.html:ro nginx

While both volumes and bind mounts are called with the -v flag and use the text:text structure, they are differentiated with the text contained before the colon. Bind mounts begin with a path on the host machine and volumes begin with a volume name. The path after the colon designates the mount point on the container file structure. Let’s illustrate the same container, but this time using a volume instead of a bind mount:

$ cd ~/ && echo 'Hello world!' > index.html
$ docker run -d --name nginx -p 8080:80 -v nginx_static:/usr/share/nginx/html nginx
$ docker cp ~/index.html nginx:/usr/share/nginx/html/
$ docker restart nginx

In a browser navigate to localhost:8080 or host_ip_address:8080 and you should find the Hello world! page (you might have to open a private browser tab to avoid cached content). Here we created a new Docker container, nginx, and a new volume, nginx_static. Nginx_static was populated with the default contents of Nginx’s static directory, /usr/share/nginx/html. Using the Docker copy command we then copied our custom html to nginx_static mounted in nginx.

A key feature of volumes is their enduring nature, as opposed to the ephemeral nature of containers. Let’s remove our nginx container and create a new one with the same parameters:

$ docker stop nginx
nginx
$ docker rm nginx
nginx
$ docker run -d --name nginx -p 8080:80 -v nginx_static:/usr/share/nginx/html nginx

If we had created the nginx container without the nginx_static volume, we would have found the default nginx landing page, but instead we found our custom html file intact (again you may have to open a new private browser window to avoid cached content).

All volumes on the system can be listed with docker volume ls and specific volumes can be deleted with docker volume rm:

$ docker volume ls
DRIVER VOLUME NAME
local nginx_static
$ docker stop nginx
nginx
$ docker rm nginx
nginx
$ docker volume rm nginx_static
nginx_static

From Docker to Docker Compose

Now that we know a bit about volumes let’s continue learning about Compose by making a new directory:

$ mkdir ~/compose && cd ~/compose

Again let’s revisit our simple Nginx container:

$ docker run -d --name nginx -p 8080:80 --restart unless-stopped nginx

Let’s create a docker-compse.yaml file with the equivalent Nginx container with the addition of a volume for the nginx static directory:

version: '2'
services:
  nginx:
    image: nginx
    container_name: nginx
    ports:
      - "8080:80"
    volumes:
      - "nginx_static:/usr/share/nginx/html"
    restart: unless-stopped
volumes:
  nginx_static:

Note: version ‘2’ at the beginning of the docker-compose.yaml file will be sufficient for most simple use cases. Docker Compose version ‘3’ and up incorporated Docker Swarm (Docker’s alternatie to Kubernetes) and made minimal changes to the version 2 YAML definitions. Docs reference.

Now with our config file created let’s start the container with our first Compose command:

$ docker-compose up -d
Creating network "compose_default" with the default driver
Creating volume "compose_nginx_static" with default driver
Creating nginx ... 
Creating nginx ... done

Compose uses a project name (the location of the docker-compose.yaml file, in this case compose) to isolate environments from each other. Networks will be created automatically following the projectName_default naming scheme.

Running docker-compose up -d created and started all containers in detached mode that were specified in the docker-compose.yaml file.

CLI

Generally when working with Compose I’ll just cd into the compose directory first, but you can use the -f flag to run Compose commands from any directory:

$ docker-compose -f /home/pete/compose/docker-compose.yaml up -d

This is especially useful for running Compose commands with scripts.

In order to update containers you’ll run the following:

$ docker-compose pull && docker-compose up -d
Pulling nginx (nginx:latest)...
latest: Pulling from library/nginx
Digest: sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
Status: Image is up to date for nginx:latest
test_nginx is up-to-date

With the docker-compose pull command, Compose will check if any containers specified in the docker-compose.yaml have an updated image, then download any new images as necessary. Then with docker-compose up -d, Compose removes and recreates only the containers with updated images. This is a massive improvement from checking for new container images, manually removing them, and then finally recreating from their original run command. Additionally, any volumes will be retained and attached to recreated containers.

In order to stop and destroy all running containers, use the following command:

$ docker-compose down
Stopping nginx ... done
Removing nginx ... done
Removing network compose_default

You can see all containers and networks were removed, but volumes remained intact. Volumes can also be deleted with the docker-compose down command by adding the -v flag.

$ docker-compose down -v
Stopping nginx ... done
Removing nginx ... done
Removing network compose_default
Removing volume compose_nginx_static

Additionally if you remove a container’s configuration from the docker-compose.yaml file and run docker-compose up -d without running docker-compose down first, the containers will not be automatically removed from the system:

$ docker-compose up -d
WARNING: Found orphan containers (nginx) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.

You will need to use the –remove-orphans flag:

$ docker-compose up -d --remove-orphans
Removing orphan container "nginx"

Lastly you can either remove orphaned volumes directly with docker volume rm or prune all local volumes not used by at least one container:

$ docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
compose_nginx_static

Total reclaimed space: 1.106kB

Check out the Compose manual page for additional commands.

$ man docker-compose

Next time we’ll continue with Docker by expanding our knowledge of Compose and build some containers from scratch with the docker build command.