Docker Image Storage - LVM Thinpool


Where should I store my Docker images?

In RHEl 7 or CentOS 7, the default Docker configuration uses devicemapper with loopback. This works perfect for testing or in a development environment...it's a bad idea for production.

Note: RHEL/CentOS Atomic Hosts are pre-configured for dedicated pools.

Steps

Edit the /etc/sysconfig/docker-storage-setup.

# vi /etc/sysconfig/docker-storage-setup

Add the following parameters to the /etc/sysconfig/docker-storage-setup file.

#STORAGE_DRIVER
#DEVS=/dev/sdd
VG=dockervg
GROWPART=enable
AUTO_EXTEND_POOL=enable
MIN_DATA_SIZE=8G
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=10

Stop Docker.

# systemctl stop docker

Remove any existing docker storage configuration file.

# rm -f /etc/sysconfig/docker-storage

Remove existing Docker location.

 # rm -f /var/lib/docker

Create a physical volume--in this example, /dev/sdd (and it is 20GB). Use the pvs and/or the fdisk -l command to verify you are working with the correct device/disk.

# pvcreate /dev/sdd

Create a volume group (i.e. dockervg). Volume group name must be the same name you specified in the /etc/sysconfig/docker-storage-setup file under the VG parameter.

# vgcreate dockervg /dev/sdd

Run the docker-storage-setup configuration tool.

# /bin/docker-storage-setup

Run the lvs command to check things out.

# lvs
LV          VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync
docker-pool dockervg twi-a-t---  8.00g             0.00   0.16   

Now we have a 8GB LVM thinpool for our Docker images. Wait...I thought the disk was 20GB.

Let's extend it. Run the vgdisplay command to check how much space is left in the volume group. In my case...3059 Extents are free

# vgdisplay dockervg
  --- Volume group ---
  ...
  Free  PE / Size       3059 / 12.00 GB

Now extend.

lvextend -l +3059 /dev/dockervg/docker-pool

Run the lvs command again.

# lvs
LV          VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync
docker-pool dockervg twi-a-t--- 19.95g             0.16   0.24 

Much better.

Crank Docker back up.

# systemctl start docker

Finally, verify.

# docker info
...
Storage Driver: devicemapper
Pool Name: dockervg-docker--pool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: 
Metadata file: 
Data Space Used: 34.6 MB
Data Space Total: 21.42 GB
Data Space Available: 21.39 GB
Metadata Space Used: 61.44 kB
Metadata Space Total: 25.17 MB
Metadata Space Available: 25.1 MB
Thin Pool Minimum Free Space: 2.142 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)

Naturally, you will want to keep your LVM thinpool "lean and clean". Check out How to delete unused Docker images for more details.

How to delete unused Docker images


To prevent your file system from filling up, one must purge old, unused Docker images.

There are many great references available on the web on how to accomplish this. This is what I have found to be most effective.

The goal is to query all running containers to determine whether an image is currently in use before attempting to delete (or remove) it.

Naturally, as a first step, you should remove all exited containers.

# docker rm -v $(docker ps -a -q -f status=exited)

You may also wish to remove created containers that are not running.

# docker rm -v $(docker ps -a -q -f status=created)

Then you can proceed with the Docker image cleanup.

Command

# for i in `docker images | tail -n +2 | awk '{print $1":"$2}'`; do if [[ `docker ps` != *$i* ]]; then docker rmi $i; fi; done

If you wish to leave specific images (say with the word "httpd" in the image name) untouched, then do the following by adding the grep command with the invert-match (-v) parameter.

# for i in `docker images | tail -n +2 | grep -i -v httpd | awk '{print $1":"$2}'`; do if [[ `docker ps` != *$i* ]]; then docker rmi $i; fi; done

Similarly, if you wish to retain multiple images with different words in the image name (i.e. httpd and nginx), do the following:

# for i in `docker images | tail -n +2 | grep -i -v 'httpd\|nginx' | awk '{print $1":"$2}'`; do if [[ `docker ps` != *$i* ]]; then docker rmi $i; fi; done

Docker: purge exited containers and remove old images


Scenario

You have been using Docker for a while--downloading various Docker images and spinning containers up and down. Your list of "Exited" containers has grown and your file system space is dwindling because of all those Docker images.

Purge, Baby, Purge

To remove all "Exited" containers, use the following command:

# docker ps -a | grep 'Exited' | awk '{print $1}' | xargs --no-run-if-empty sudo docker rm

To remove all Docker images that are not tagged as "latest" (notice the invert-match grep option, -v), run this:

# docker images | grep -v 'latest' | awk '{print $3}' | xargs --no-run-if-empty sudo docker rmi

And to get them both in one swoop...

# docker rm $(docker ps --no-trunc -aq); docker rmi $(docker images --filter "dangling=true")

Keep it lean and mean. Edit crontab.

# crontab -e

Add this to purge containers and images every morning at 2:00 am.

0 2 * * * docker rm $(docker ps --no-trunc -aq) > /dev/null 2>&1; docker rmi $(docker images --filter "dangling=true") > /dev/null 2>&1

Experiences patching a RHEL 7.1 base Docker image


Scenario

You have Docker installed on a host system and you want to deploy a patched Red Hat Enterprise Linux base Docker image. I have presented two options and have shown several challenges that I had to overcome.


Option #1

Start an interactive shell into a RHEL 7.1 docker container.

# docker run -it registry.access.redhat.com/rhel7.1 bash

Run yum update inside the container.

 [root@0e439b6c0ec6 /]# yum update -y
Loaded plugins: product-id, subscription-manager

https://mysatelliteserver/pulp/repos/myorganization/myenvironment/mycontentview/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.


 One of the configured repositories failed (Red Hat Enterprise Linux 7 Server (RPMs)),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable rhel-7-server-rpms

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=rhel-7-server-rpms.skip_if_unavailable=true

failure: repodata/repomd.xml from rhel-7-server-rpms: [Errno 256] No more mirrors to try.

https://mysatelliteserver/pulp/repos/myorganization/myenvironment/mycontentview/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

Ugh! We hit an error. If you encounter the same error, then you need to specify the Red Hat release version using the "--releasever" yum parameter. See below.

[root@0e439b6c0ec6 /]# yum update -y --releasever=7.1
 Loaded plugins: product-id, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package bash.x86_64 0:4.2.46-12.el7 will be updated
 ...
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security@redhat.com>"
 Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
 Package    : redhat-release-server-7.1-1.el7.x86_64 (@koji-override-1/7.0)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0x2FA658E0:
 Userid     : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
 Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
 Package    : redhat-release-server-7.1-1.el7.x86_64 (@koji-override-1/7.0)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Running transaction check
Running transaction test


Transaction check error:
  file /usr/lib64/libsystemd-daemon.so.0 from install of systemd-libs-219-19.el7.x86_64 conflicts with file from package systemd-container-libs-208.20-6.el7.x86_64
  file /usr/lib64/libsystemd-id128.so.0 from install of systemd-libs-219-19.el7.x86_64 conflicts with file from package systemd-container-libs-208.20-6.el7.x86_64
  file /usr/lib64/libsystemd-journal.so.0 from install of systemd-libs-219-19.el7.x86_64 conflicts with file from package systemd-container-libs-208.20-6.el7.x86_64
  file /usr/lib64/libsystemd-login.so.0 from install of systemd-libs-219-19.el7.x86_64 conflicts with file from package systemd-container-libs-208.20-6.el7.x86_64
  file /usr/lib64/libudev.so.1 from install of systemd-libs-219-19.el7.x86_64 conflicts with file from package systemd-container-libs-208.20-6.el7.x86_64
  file /usr/lib64/security/pam_systemd.so from install of systemd-libs-219-19.el7.x86_64 conflicts with file from package systemd-container-libs-208.20-6.el7.x86_64

Error Summary
-------------

Again, we hit another error. This error is a known Red Hat Bug (1284056). The good news is that there is a work around.

[root@0e439b6c0ec6 /]# yum --releasever=7.1 swap -y -- remove systemd-container\* -- install systemd systemd-libs
 Loaded plugins: product-id, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package systemd.x86_64 0:219-19.el7 will be installed
...
 Complete!
 [root@0e439b6c0ec6 /]# yum update -y --releasever=7.1
 Loaded plugins: product-id, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package bash.x86_64 0:4.2.46-12.el7 will be updated
...
 Complete!

Bingo! A completely patched RHEL 7.1 Docker image. The final step is to commit the changes. Exit your running container by typing "exit" and then run a docker commit command.

# docker commit 0e439b6c0ec6 myprivatedockerregistry:5000/rhel7.1-patched

Golden! And the security department is happy!!


Option #2

Now for the easy way. Create a Dockerfile with the below contents.

FROM registry.access.redhat.com/rhel7.1

RUN yum clean all && \
    yum --releasever=7.1 swap -y -- remove systemd-container\* -- install systemd systemd-libs && \
    yum update -y --releasever=7.1 && \
    yum clean all

Run the docker build command and you're done!

# docker build -t myprivatedockerregistry:5000/rhel7.1-patched .

Run redis-cli in a container from kubectl command


Scenario

You have a Kubernetes cluster running a Redis pod or replication controller. You need to run a redis-cli command in the container utilizing the kubectl command. The grep/awk commands below may not be needed depending on how you have Redis configured. In this scenario, the Redis container binds to the container IP only (not localhost), so we must grep for the host. There are plenty of other ways to accomplish the same thing but wanted to pass this along in case anyone else ran into a similar situation.


Command

# kubectl --namespace=default exec -it `kubectl --namespace=default get pod | grep mypodname | awk '{print $1}'` -- /usr/bin/redis-cli -h `kubectl --namespace=default get pod | grep mypodname | awk '{print $1}'` -p 6379 -a mysecretpassword info server

Output

# Server
redis_version:3.0.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:33b0defdcca850ac
redis_mode:standalone
os:Linux 3.10.0-327.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:9
run_id:f58cb95b25f32519266d78f30fb92f800cf3e22d
tcp_port:6379
uptime_in_seconds:99775
uptime_in_days:1
hz:10
lru_clock:2077829
config_file:/etc/redis.conf

Advanced Command

If you need an administrative task to purge keys in the Redis cache or something similar, use this command with a pipe to a xargs command:

kubectl --namespace=default exec -i `kubectl --namespace=default get pod | grep my-redis-pod | awk '{print $1}'` -- /usr/bin/redis-cli -h `kubectl --namespace=default get pod | grep my-redis-pod | awk '{print $1}'` -p 6379 -a mysecretpassword KEYS "mykey:*" | xargs kubectl --namespace=default exec -i `kubectl --namespace=default get pod | grep my-redis-pod | awk '{print $1}'` -- /usr/bin/redis-cli -h `kubectl --namespace=default get pod | grep my-redis-pod | awk '{print $1}'` -p 6379 -a mysecretpassword DEL

This command can come in handy to allow developers to purge the Redis cache from a Jenkins Continuous Integration workflow.