Docker Compose and Volumes
Mar 19, 2020
In the first post of the container series Linux Containers and Docker, I introduced Linux Containers and Docker, as well as how to use the Docker CLI and basic management of containers. In this second post of the series I turn my focus to my favorite Docker management tool, Docker Compose.
Docker Compose is a container management tool for Docker, written in Python, and it is a game changer. While managing a single container with Docker is doable, throwing any additional containers in the mix quickly makes management tedious. With Compose, a YAML file is written for configuration, and a simple CLI interface is used to manage almost the entire life cycle of your containers.
Let’s begin by installing Compose
# apt update && apt install -y docker-compose
Note: The Docker Compose package in Ubuntu 18.04, currently version 1.17.1-2, lags behind the current release, 1.25.4. If you would prefer a newer version of Compose, I would recommend using Ubuntu 19.10 as a base, which currently includes Compose version 1.21.0-3. Alternatively you can directly download the binary if you would like the absolute latest version of Docker Compose as seen in the docs:
$ curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose $ chmod +x /usr/local/bin/docker-compose
Note: replace the version number with the latest version number.
In a bit we’ll be writing a docker-compose.yaml file so I thought I would briefly introduce YAML for those not familiar. YAML, (YAML Ain’t Markup Language) is a data-serialization language, commonly used for configuration files. YAML is a superset of JSON, trading curly brackets for Python-style indentation to indicate nesting. YAML is generally easy for humans to read, but any stray spaces or indentations can cause issues.
At this point before continuing with Compose, I’d like to introduce another Docker concept, volumes. Volumes allow for persistent data storage and are managed completely by Docker. I introduced a related topic in the last post, bind mounts, which are dependent on the directory structure of the host machine as seen here:
$ docker run -d --name nginx -p 8080:80 -v ~/index.html:/usr/share/nginx/html/index.html:ro nginx
While both volumes and bind mounts are called with the -v flag and use the text:text structure, they are differentiated with the text contained before the colon. Bind mounts begin with a path on the host machine and volumes begin with a volume name. The path after the colon designates the mount point on the container file structure. Let’s illustrate the same container, but this time using a volume instead of a bind mount:
$ cd ~/ && echo 'Hello world!' > index.html $ docker run -d --name nginx -p 8080:80 -v nginx_static:/usr/share/nginx/html nginx $ docker cp ~/index.html nginx:/usr/share/nginx/html/ $ docker restart nginx
In a browser navigate to localhost:8080 or host_ip_address:8080 and you should find the Hello world! page (you might have to open a private browser tab to avoid cached content). Here we created a new Docker container, nginx, and a new volume, nginx_static. Nginx_static was populated with the default contents of Nginx’s static directory, /usr/share/nginx/html. Using the Docker copy command we then copied our custom html to nginx_static mounted in nginx.
A key feature of volumes is their enduring nature, as opposed to the ephemeral nature of containers. Let’s remove our nginx container and create a new one with the same parameters:
$ docker stop nginx nginx $ docker rm nginx nginx $ docker run -d --name nginx -p 8080:80 -v nginx_static:/usr/share/nginx/html nginx
If we had created the nginx container without the nginx_static volume, we would have found the default nginx landing page, but instead we found our custom html file intact (again you may have to open a new private browser window to avoid cached content).
All volumes on the system can be listed with docker volume ls and specific volumes can be deleted with docker volume rm:
$ docker volume ls DRIVER VOLUME NAME local nginx_static $ docker stop nginx nginx $ docker rm nginx nginx $ docker volume rm nginx_static nginx_static
From Docker to Docker Compose
Now that we know a bit about volumes let’s continue learning about Compose by making a new directory:
$ mkdir ~/compose && cd ~/compose
Again let’s revisit our simple Nginx container:
$ docker run -d --name nginx -p 8080:80 --restart unless-stopped nginx
Let’s create a docker-compse.yaml file with the equivalent Nginx container with the addition of a volume for the nginx static directory:
version: '2' services: nginx: image: nginx container_name: nginx ports: - "8080:80" volumes: - "nginx_static:/usr/share/nginx/html" restart: unless-stopped volumes: nginx_static:
Note: version ‘2’ at the beginning of the docker-compose.yaml file will be sufficient for most simple use cases. Docker Compose version ‘3’ and up incorporated Docker Swarm (Docker’s alternatie to Kubernetes) and made minimal changes to the version 2 YAML definitions. Docs reference.
Now with our config file created let’s start the container with our first Compose command:
$ docker-compose up -d Creating network "compose_default" with the default driver Creating volume "compose_nginx_static" with default driver Creating nginx ... Creating nginx ... done
Compose uses a project name (the location of the docker-compose.yaml file, in this case compose) to isolate environments from each other. Networks will be created automatically following the projectName_default naming scheme.
Running docker-compose up -d created and started all containers in detached mode that were specified in the docker-compose.yaml file.
Generally when working with Compose I’ll just cd into the compose directory first, but you can use the -f flag to run Compose commands from any directory:
$ docker-compose -f /home/pete/compose/docker-compose.yaml up -d
This is especially useful for running Compose commands with scripts.
In order to update containers you’ll run the following:
$ docker-compose pull && docker-compose up -d Pulling nginx (nginx:latest)... latest: Pulling from library/nginx Digest: sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b Status: Image is up to date for nginx:latest test_nginx is up-to-date
With the docker-compose pull command, Compose will check if any containers specified in the docker-compose.yaml have an updated image, then download any new images as necessary. Then with docker-compose up -d, Compose removes and recreates only the containers with updated images. This is a massive improvement from checking for new container images, manually removing them, and then finally recreating from their original run command. Additionally, any volumes will be retained and attached to recreated containers.
In order to stop and destroy all running containers, use the following command:
$ docker-compose down Stopping nginx ... done Removing nginx ... done Removing network compose_default
You can see all containers and networks were removed, but volumes remained intact. Volumes can also be deleted with the docker-compose down command by adding the -v flag.
$ docker-compose down -v Stopping nginx ... done Removing nginx ... done Removing network compose_default Removing volume compose_nginx_static
Additionally if you remove a container’s configuration from the docker-compose.yaml file and run docker-compose up -d without running docker-compose down first, the containers will not be automatically removed from the system:
$ docker-compose up -d WARNING: Found orphan containers (nginx) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
You will need to use the –remove-orphans flag:
$ docker-compose up -d --remove-orphans Removing orphan container "nginx"
Lastly you can either remove orphaned volumes directly with docker volume rm or prune all local volumes not used by at least one container:
$ docker volume prune WARNING! This will remove all local volumes not used by at least one container. Are you sure you want to continue? [y/N] y Deleted Volumes: compose_nginx_static Total reclaimed space: 1.106kB
Check out the Compose manual page for additional commands.
$ man docker-compose
Next time we’ll continue with Docker by expanding our knowledge of Compose and build some containers from scratch with the docker build command.