Private Docker Registries and the Canonical Distribution of Kubernetes

This blog post refers to an earlier version of Charmed Kubernetes. For the current methods of dealing with registries, please see the official documentation.

This originally appeared on Tim Van Steenburgh’s blog

How do I use a private image registry with my Kubernetes cluster? How do I set up my own registry? Let’s look at how to perform these tasks on the Canonical Distribution of Kubernetes (CDK).

Using an Existing Insecure Registry

In order to connect to an insecure registry, the Docker daemon must be reconfigured and an --insecure-registry option must be added.

This can be done directly via Juju, using the command:

juju config kubernetes-worker docker-config=”--insecure-registry registry.domain.com:5000"

Creating a Secure CDK Registry

CDK provides an option to deploy a secure Docker registry within the cluster, and expose it via an ingress.

Note: The registry provided is not a production grade registry, and should not be used in a production context.

Requirements

To deploy and use the provided registry, you will need:

• A DNS entry (registry.acme.com) pointing at the ingress of the cluster (directly, via DNS round robin or with a load balancer)
• A valid TLS certificate and key for registry.acme.com (registry.crt and registry.key)
• A set of usernames and passwords stored in a file for htpasswd authentication (format: username:password, one user per line)

Considering a htpasswd.cleartxt file filled with users, the following loop will generate an encoded version of it:

while read line
do
  USER=$(echo ${line} | cut -f1 -d':')
  PASS=$(echo ${line} | cut -f2 -d':')
  docker run \
    --rm \
    xmartlabs/htpasswd \
    ${USER} ${PASS} \
    | tee -a htpasswd.enc
done < htpasswd.cleartxt
sed -i "/^$/d" htpasswd.enc

Deployment

To deploy the registry, run:

juju run-action kubernetes-worker/0 registry \
  domain=registry.acme.com \
  htpasswd=”$(base64 -w0 htpasswd.enc)” \
  htpasswd-plain=”$(base64 -w0 htpasswd.cleartxt)” \
  tlscert=”$(base64 -w0 registry.crt)” \
  tlskey=”$(base64 -w0 registry.key)” \
  ingress=true

Tear down

To tear down the registry, run

juju run-action kubernetes-worker/0 registry \
  delete=true \
  ingress=true

Storage

The registry provided by CDK will use a /srv/registry hostPath to store the images. This means that in case of a rescheduling of the registry (failure, overload…), if the new pod is scheduled on a different host, you will lose your images.

Alternatively, you can use a network mount such as NFS on all workers to benefit from a single point of storage for the images.

Ingress Configuration

The CDK registry action makes the assumption that the ingress running is nginx and will enforce a change of the configuration to increase the client_max_body_size from 1MB to 1GB. This is done via a patch, hence will not overwrite other configuration keys.

If you are using another ingress, deploy with ingress=false and make sure your ingress will support image upload (typical images are ~300MB, and typical CUDA images are 1 to 4GB)

Alternatives

If you want a similar setup but with flexibility on the storage and management via native Kubernetes tools you will find a derived work delivered via a Helm chart on https://github.com/madeden/charts

If you’d like to follow along more closely with CDK development, you can do so in the following places:

• https://github.com/kubernetes/kubernetes (cluster/juju directory)
• https://github.com/juju-solutions/bundle-canonical-kubernetes
• Kubernetes Slack channels and SIG meetings
• #juju on Freenode IRC
• [email protected] mailing list

Until next time!