Docker Image Storage - LVM Thinpool


Where should I store my Docker images?

In RHEl 7 or CentOS 7, the default Docker configuration uses devicemapper with loopback. This works perfect for testing or in a development environment...it's a bad idea for production.

Note: RHEL/CentOS Atomic Hosts are pre-configured for dedicated pools.

Steps

Edit the /etc/sysconfig/docker-storage-setup.

# vi /etc/sysconfig/docker-storage-setup

Add the following parameters to the /etc/sysconfig/docker-storage-setup file.

#STORAGE_DRIVER
#DEVS=/dev/sdd
VG=dockervg
GROWPART=enable
AUTO_EXTEND_POOL=enable
MIN_DATA_SIZE=8G
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=10

Stop Docker.

# systemctl stop docker

Remove any existing docker storage configuration file.

# rm -f /etc/sysconfig/docker-storage

Remove existing Docker location.

 # rm -f /var/lib/docker

Create a physical volume--in this example, /dev/sdd (and it is 20GB). Use the pvs and/or the fdisk -l command to verify you are working with the correct device/disk.

# pvcreate /dev/sdd

Create a volume group (i.e. dockervg). Volume group name must be the same name you specified in the /etc/sysconfig/docker-storage-setup file under the VG parameter.

# vgcreate dockervg /dev/sdd

Run the docker-storage-setup configuration tool.

# /bin/docker-storage-setup

Run the lvs command to check things out.

# lvs
LV          VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync
docker-pool dockervg twi-a-t---  8.00g             0.00   0.16   

Now we have a 8GB LVM thinpool for our Docker images. Wait...I thought the disk was 20GB.

Let's extend it. Run the vgdisplay command to check how much space is left in the volume group. In my case...3059 Extents are free

# vgdisplay dockervg
  --- Volume group ---
  ...
  Free  PE / Size       3059 / 12.00 GB

Now extend.

lvextend -l +3059 /dev/dockervg/docker-pool

Run the lvs command again.

# lvs
LV          VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync
docker-pool dockervg twi-a-t--- 19.95g             0.16   0.24 

Much better.

Crank Docker back up.

# systemctl start docker

Finally, verify.

# docker info
...
Storage Driver: devicemapper
Pool Name: dockervg-docker--pool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: 
Metadata file: 
Data Space Used: 34.6 MB
Data Space Total: 21.42 GB
Data Space Available: 21.39 GB
Metadata Space Used: 61.44 kB
Metadata Space Total: 25.17 MB
Metadata Space Available: 25.1 MB
Thin Pool Minimum Free Space: 2.142 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)

Naturally, you will want to keep your LVM thinpool "lean and clean". Check out How to delete unused Docker images for more details.

How to delete unused Docker images


To prevent your file system from filling up, one must purge old, unused Docker images.

There are many great references available on the web on how to accomplish this. This is what I have found to be most effective.

The goal is to query all running containers to determine whether an image is currently in use before attempting to delete (or remove) it.

Naturally, as a first step, you should remove all exited containers.

# docker rm -v $(docker ps -a -q -f status=exited)

You may also wish to remove created containers that are not running.

# docker rm -v $(docker ps -a -q -f status=created)

Then you can proceed with the Docker image cleanup.

Command

# for i in `docker images | tail -n +2 | awk '{print $1":"$2}'`; do if [[ `docker ps` != *$i* ]]; then docker rmi $i; fi; done

If you wish to leave specific images (say with the word "httpd" in the image name) untouched, then do the following by adding the grep command with the invert-match (-v) parameter.

# for i in `docker images | tail -n +2 | grep -i -v httpd | awk '{print $1":"$2}'`; do if [[ `docker ps` != *$i* ]]; then docker rmi $i; fi; done

Similarly, if you wish to retain multiple images with different words in the image name (i.e. httpd and nginx), do the following:

# for i in `docker images | tail -n +2 | grep -i -v 'httpd\|nginx' | awk '{print $1":"$2}'`; do if [[ `docker ps` != *$i* ]]; then docker rmi $i; fi; done

Using systemd to start Docker containers


Scenario

You want to automatically start Docker containers at boot and/or you wish to give your system administrators a familiar way to start and stop Docker containers using the systemctl command.

Prerequisites

You have a Docker container running with a unique name (i.e. my-web-server).

# docker run -d --name my-web-server --restart-always -p 80:80 -v /mnt/my-web-server-logs:/var/logs rhscl/httpd-24-rhel7

Step #1

Create a system service unit file as root in the "/etc/systemd/system" directory.

# vi /etc/systemd/system/docker-my-web-server.service

Add the following contents to the "docker-my-web-server.service" unit file.

[Unit]
Description=My Web Server Docker Container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a my-web-server
ExecStop=/usr/bin/docker stop -t 2 my-web-server

[Install]
WantedBy=default.target

Step #2

Change the permissions of the unit file.

# chmod 664 /etc/systemd/system/docker-my-web-server.service

Step #3

Reload systemd.

# systemctl daemon-reload

Step #4

Enable new service to start at boot.

# systemctl enable docker-my-web-server.service

Start/Stop new service using systemctl

# systemctl stop docker-my-web-server.service
# systemctl start docker-my-web-server.service

For additional information on using systemd to start and stop Docker containers, visit Red Hat's documentation regarding Creating Custom Unit Files or Docker's documentation Automatically start containers.

Security scan a RHEL7 Docker image & container


Scenario

You have a running Docker environment with a RHEL7 base image downloaded and running. The security folks are breathing down your neck for proof that the Docker images and containers are safe. Your challenge...prove it.

We will utilize the Open-Source Security Content Automation Protocol (OSCAP) tool specifically for Docker (oscap-docker).

We will install the packages provided through the Red Hat/CentOS channels but the packages are available at the link below if you prefer to download it direct.
https://github.com/OpenSCAP/container-compliance

Prerequisites

Install the openscap-utils package which contains the oscap-docker command.

# yum install openscap-utils -y

Additionally, install the SCAP Security Guide which provides predefined security policies (i.e. PCI DSS). You can also create custom security policies if you wish.

# yum install scap-security-guide -y

Create a directory where to store your scan results.

# mkdir /oscap

CVE Scans

Perform a Common Vulnerabilities and Exposures (CVE) scan of a Docker image.

# oscap-docker image-cve myprivatedockerregistry:5000/mydockerimage --results /oscap/mydockerimage-results-cve.xml --report /oscap/mydockerimage-report-cve.html

Perform the same CVE scan of a container.

# oscap-docker container-cve mycontainer --results /oscap/mycontainer-results-cve.xml --report /oscap/mycontainer-report-cve.html

PCI DSS Scans

Perform a Payment Card Industry Data Security Standard (PCI DSS) scan of a Docker image.

oscap-docker image myprivatedockerregistry:5000/mydockerimage xccdf eval --results /oscap/mydockerimage-results-pci-dss.xml --report /oscap/mydockerimage-report-pci-dss.html --profile xccdf_org.ssgproject.content_profile_pci-dss /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml

Perform the same PCI DSS scan of a container.

oscap-docker container mycontainer xccdf eval --results /oscap/mycontainer-results-pci-dss.xml --report /oscap/mycontainer-report-pci-dss.html --profile xccdf_org.ssgproject.content_profile_pci-dss /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml

The oscap-docker command uses the same switches and parameters as the oscap command.

For additional information, check the man page.

# man oscap-docker

I highly recommend patching your Docker image before running the scans (primarily the CVE scan). An all "green" scan equals a happy security department. To learn how to patch RHEL7 Docker images, click here.

Docker: purge exited containers and remove old images


Scenario

You have been using Docker for a while--downloading various Docker images and spinning containers up and down. Your list of "Exited" containers has grown and your file system space is dwindling because of all those Docker images.

Purge, Baby, Purge

To remove all "Exited" containers, use the following command:

# docker ps -a | grep 'Exited' | awk '{print $1}' | xargs --no-run-if-empty sudo docker rm

To remove all Docker images that are not tagged as "latest" (notice the invert-match grep option, -v), run this:

# docker images | grep -v 'latest' | awk '{print $3}' | xargs --no-run-if-empty sudo docker rmi

And to get them both in one swoop...

# docker rm $(docker ps --no-trunc -aq); docker rmi $(docker images --filter "dangling=true")

Keep it lean and mean. Edit crontab.

# crontab -e

Add this to purge containers and images every morning at 2:00 am.

0 2 * * * docker rm $(docker ps --no-trunc -aq) > /dev/null 2>&1; docker rmi $(docker images --filter "dangling=true") > /dev/null 2>&1