Fun with cron


If you wish to run a script every second Tuesday of the month, add the following to crontab (crontab -e to edit).

0 0 8-14 * 2 myscript.sh

Using systemd to start Docker containers


Scenario

You want to automatically start Docker containers at boot and/or you wish to give your system administrators a familiar way to start and stop Docker containers using the systemctl command.

Prerequisites

You have a Docker container running with a unique name (i.e. my-web-server).

# docker run -d --name my-web-server --restart-always -p 80:80 -v /mnt/my-web-server-logs:/var/logs rhscl/httpd-24-rhel7

Step #1

Create a system service unit file as root in the "/etc/systemd/system" directory.

# vi /etc/systemd/system/docker-my-web-server.service

Add the following contents to the "docker-my-web-server.service" unit file.

[Unit]
Description=My Web Server Docker Container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a my-web-server
ExecStop=/usr/bin/docker stop -t 2 my-web-server

[Install]
WantedBy=default.target

Step #2

Change the permissions of the unit file.

# chmod 664 /etc/systemd/system/docker-my-web-server.service

Step #3

Reload systemd.

# systemctl daemon-reload

Step #4

Enable new service to start at boot.

# systemctl enable docker-my-web-server.service

Start/Stop new service using systemctl

# systemctl stop docker-my-web-server.service
# systemctl start docker-my-web-server.service

For additional information on using systemd to start and stop Docker containers, visit Red Hat's documentation regarding Creating Custom Unit Files or Docker's documentation Automatically start containers.

Fun with curl


Run curl and see what your response times are

# watch -n .5 curl -o /dev/null -s -w %{time_total} http://rootsquash.com

Check site response codes.

# for i in {0..100}; do echo -n `date` " "; curl -s -o /dev/null -w "%{http_code}" http://rootsquash.com; echo; done

Generate random string from Linux command line


Command

Use the following command to generate a random 16 character string with lower and upper case letters and "special" characters.

</dev/urandom tr -dc '0123456789!@#$%^&*abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' | head -c16; echo ""

Output

2kQjZ&iHSavVPj&U

If you prefer only eight alphanumeric characters, use:

</dev/urandom tr -dc '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' | head -c8; echo ""

TCxvA40s

Securing the Kubernetes API


Scenario

You have a functioning Kubernetes cluster that is running on a non-secure port with the API server exposed to everyone in your organization. You have been tasked with securing the Kubernetes API such that only the Kubernetes nodes and other defined users can call the API.

To secure your Kubernetes API, you must generate server and client certificates, specify a service account context using a token, configure the apiserver to listen on a secure port, and update the Kubernetes master and node configurations. Below are detailed instructions to accomplish this. Please note that while this is a working example, your mileage may vary depending on setup and configuration. I hope these instructions give you a head start on securing your Kubernetes cluster.

For additional information regarding authenticating/accessing the Kubernetes API, please visit http://kubernetes.io/docs/admin/authentication.


Kubernetes Master

Create a directory to store the certificates.

# mkdir /srv/kubernetes

Create a Certificate Authority (CA) key and certificate on the Kubernetes master.

# openssl genrsa -out /srv/kubernetes/ca.key 4096
# openssl req -x509 -new -nodes -key /srv/kubernetes/ca.key -subj "/CN=${HOSTNAME}" -days 10000 -out /srv/kubernetes/ca.crt

Create a server key on the Kubernetes master.

# openssl genrsa -out /srv/kubernetes/server.key 2048

Generate a Certificate Signing Request (CSR) for a server certificate on the Kubernetes master.

# openssl req -new -key /srv/kubernetes/server.key -subj "/CN=${HOSTNAME}" -out /srv/kubernetes/server.csr

Generate the server certificate from the CSR and sign it with the newly generated CA key and certificate.

# openssl x509 -req -in /srv/kubernetes/server.csr -CA /srv/kubernetes/ca.crt -CAkey /srv/kubernetes/ca.key -CAcreateserial -out /srv/kubernetes/server.crt -days 10000

View your server certificate (optional).

# openssl x509 -noout -text -in /srv/kubernetes/server.crt

Generate a random token that will be used by the "kubelet" service account and store it in a variable named "TOKEN".

# TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)

Create a "known tokens" file named /srv/kube-apiserver/known_tokens.csv (this will be referenced in the /etc/kubernetes/apiserver config).

# mkdir /srv/kube-apiserver
# echo "${TOKEN},kubelet,kubelet" > /srv/kube-apiserver/known_tokens.csv

Create a backup of your Kubernetes apiserver and controller-manager config files.

# cp /etc/kubernetes/apiserver /etc/kubernetes/apiserver.`date "+%Y%m%d"`.bak
# cp /etc/kubernetes/controller-manager /etc/kubernetes/controller-manager.`date "+%Y%m%d"`.bak

Edit the /etc/kubernetes/apiserver file.

# vi /etc/kubernetes/apiserver

Add the following flags to the KUBELET_API _ARGS parameter.

KUBE_API_ARGS="--cluster-name=mykubecluster --insecure-bind-address=127.0.0.1 --kubelet-https=true --client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.crt --tls-private-key-file=/srv/kubernetes/server.key --token_auth_file=/srv/kube-apiserver/known_tokens.csv"

Edit the /etc/kubernetes/controller-manager file.

# vi /etc/kubernetes/controller-manager

Add the following flags to the KUBE_CONTROLLER _MANAGER _ARGS parameter.

KUBE_CONTROLLER_MANAGER_ARGS="--root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --node-monitor-grace-period=20s --pod-eviction-timeout=20s"

Restart the kube-apiserver and kube-controller-manager service.

# systemctl restart kube-apiserver.service kube-controller-manager.service

Check the status to ensure things look good.

# systemctl status -l kube-apiserver.service 
# systemctl status -l kube-controller-manager.service

Create a variable called "NODES" that lists all of your Kubernetes nodes.

NODES="kubenode1.mydomain.com kubenode2.mydomain.com kubenode3.mydomain.com"

Generate a client certificate for each of your Kubernetes nodes.

# for NODE in $NODES; do
    openssl req -newkey rsa:2048 -nodes -keyout /srv/kubernetes/${NODE}.key -subj "/CN=${NODE}" -out $CERTDIR/${NODE}.csr
      openssl x509 -req -days 10000 -in /srv/kubernetes/${NODE}.csr -CA /srv/kubernetes/ca.crt -CAkey /srv/kubernetes/ca.key -CAcreateserial -out /srv/kubernetes/${NODE}.crt
  done

Copy the CA certificate as well as the client certificates to their respective nodes.

# for NODE in $NODES; do
ssh root@${NODE} mkdir /srv/kubernetes
scp /srv/kubernetes/ca.crt /srv/kubernetes/${NODE}.crt /srv/kubernetes/${NODE}.key root@${NODE}:/srv/kubernetes/
  done

Kubernetes Nodes

On each Kubernetes node, run the following kubectl commands to config the kubelet service to communicate securely to the kube-apiserver. You will need the token you generated earlier.

# kubectl config set-cluster mykubecluster --server=https://kubemaster.domain.com:6443 --insecure-skip-tls-verify=true
# kubectl config unset clusters
# kubectl config set-cluster mykubecluster --certificate-authority=/srv/kubernetes/ca.crt --embed-certs=true --server=https://kubemaster.domain.com:6443
# kubectl config set-credentials kubelet --client-certificate=/srv/kubernetes/${HOSTNAME}.crt --client-key=/srv/kubernetes/${HOSTNAME}.key --embed-certs=true --token=${TOKEN}
# kubectl config set-context service-account-context --cluster=mykubecluster --user=kubelet
# kubectl config use-context service-account-context

Copy the kubelet config file that was auto generated from the commands above (you can create/edit this file directly, if you prefer...the commands above just simplfy the configuration).

# cp /root/.kube/config /var/lib/kubelet/kubeconfig

Create a backup of your kubelet and kube-proxy config files.

# cp /etc/kubernetes/kubelet /etc/kubernetes/kubelet.`date "+%Y%m%d"`.bak
# cp /etc/kubernetes/proxy /etc/kubernetes/proxy.`date "+%Y%m%d"`.bak

Update the /etc/kubernetes/kubelet file to point to the new kubeconfig file.

# vi /etc/kubernetes/kubelet

Change the KUBELET_API _SERVER parameter to reflect the secure port (6443).

KUBELET_API_SERVER="--api_servers=https://kubemaster.domain.com:6443"

Add the "--kubeconfig" flag to the KUBELET_ARGS parameter.

KUBELET_ARGS="--enable_server=true --register-node=true --kubeconfig=/var/lib/kubelet/kubeconfig --node-status-update-frequency=5s"

Additionally, you will want to do the same thing for the kube-proxy configuration.

# mkdir /var/lib/kube-proxy
# cp /root/.kube/config /var/lib/kube-proxy/kubeconfig
# vi /etc/kubernetes/kube-proxy

Add the "--kubeconfig" flag to the KUBELET_PROXY _ARGS parameter.

KUBE_PROXY_ARGS="--kubeconfig=/var/lib/kube-proxy/kubeconfig"

Restart the kubelet and kube-proxy service.

# systemctl restart kubelet.service kube-proxy.service

Check the status to ensure things look good.

# systemctl status -l kubelet.service 
# systemctl status -l kube-proxy.service

Testing

Verify you can securely access the Kubernetes API using the client certificates. Run the following on one of the Kubernetes nodes.

# curl -k --key /srv/kubernetes/${HOSTNAME}.key  --cert /srv/kubernetes/${HOSTNAME}.crt  --cacert /srv/kubernetes/ca.crt https://kubemaster.domain.com:6443/version

Verify you can securely access the Kubernetes API using the token. Again, run the following on one of the Kubernetes nodes.

# curl -k -H "Authorization: Bearer ${TOKEN}" https://kubemaster.domain.com:6443/version