Using the Kubernetes Secrets API

This is the third part in my series on building and deploying a Rails app using Docker containers and Kubernetes. Here is the first part and the second part.

To follow along, you’ll need to complete the steps in the “Project Set Up” section of the previous blog post.

What are secrets?

The previous version of the ToDo app has some issues. One of the biggest is that the username and password for the database are included in plain text in several places. This isn’t a good idea for real applications. The Kubernetes Secret object was designed to handle securely sharing sensitive information between containers.

There are a couple ways to access secret objects. For this tutorial, I’m going to have the secret object mounted as a set of files on a volume the container can access. If you are used to accessing secrets in environment variables this may feel a little awkward at first but it quickly becomes familiar.

Creating the Secrets File

I have three secrets to store: the database username, the database password, and the Rails secret key. Secrets need to be base64 encoded and it is easy to do the encoding in Ruby.

require 'base64'

username = Base64.encode64('rails')
puts username

password = Base64.encode64('password')
puts password

secret_key = Base64.encode64('secret_key')
puts secret_key

Once I have the base64 encoded values I create a file for the secrets object. The yaml file to create a set of secrets looks very similar to the file that creates a pod or service.

#secrets.yml
apiVersion: v1
kind: Secret
metadata:
  name: secrets
type: Opaque
data:
  password: cGFzc3dvcmQ=
  username: cmFpbHM=
  secret-key: c2VjcmV0X2tleQ==

To create the secret I use kubectl create -f just like I do with other Kubernetes objects.

kubectl create -f secrets.yml

Modifying the Database Pod

The next step is to make the database container use the secrets. The official Postgres image doesn’t work with secrets as is so I need to make a Dockerfile to extend it. I put this Dockerfile in a separate pg directory.

FROM postgres:9.4
ENTRYPOINT []
CMD export POSTGRES_PASSWORD=$(cat /etc/secrets/password); export POSTGRES_USER=$(cat /etc/secrets/username); /docker-entrypoint.sh postgres

The last line of that Dockerfile contains three separate commands and is a bit hard to read. Here are the commands with some added whitespace:

export POSTGRES_PASSWORD=$(cat /etc/secrets/password)
export POSTGRES_USER=$(cat /etc/secrets/username)
/docker-entrypoint.sh postgres

The first line reads the password from /etc/secrets/password and puts it in the POSTGRES_PASSWORD environment variable. The second line does the same thing for the POSTGRES_USER environment variable. The last line runs the entrypoint script that is part of the Postgres image and starts up Postgres. Now I need to build this image and store it in the Google Container Registry.

docker build -t todo/pg pg/.
docker tag -f todo/pg gcr.io/my_project_id/pg:v1
gcloud docker push gcr.io/my_project_id/pg:v1

Now I need to modify the database pod to use the new image and access the secrets object. Here’s the old version:

# db-pod.yml
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: db
  name: db
spec:
  containers:
  - image: postgres
    name: db
    env:
    - name: POSTGRES_PASSWORD
      value: password
    - name: POSTGRES_USER
      value: rails
    ports:
    - name: pg
      containerPort: 5432
      hostPort: 5432

And here’s the new version:

# db-pod.yml
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: db
  name: db
spec:
  volumes:
  - name: secrets
    secret:
      secretName: secrets
  containers:
  - image: gcr.io/my_project_id/pg:v1
    name: db
    volumeMounts:
    - name: secrets
      mountPath: "/etc/secrets"
      readOnly: true
    ports:
    - name: cp
      containerPort: 5432
      hostPort: 5432

The first change is to remove the env: section. That information will come in via secrets now. After that I add a volume for the secrets (lines 9 - 12) and a volume mount (line 16 - 19). The last change is updating the image from the official Postgres image: postgres to the image I just built: gcr.io/my_project_id/pg:v1.

I create the database pod and database service using kubectl create -f.

kubectl create -f db-pod.yml
kubectl create -f db-service.yml
kubectl get pods

Now the database pods are using secrets to get the Postgres password and username.

Modifying the Rails Pods

Modifying the Rails pods to use secrets is also pretty straightforward. All the setup for the Rails pods is in init.sh so I just add lines to create environment variables from my three secrets. Here is the new version of init.sh:

export SECRET_KEY_BASE=$(cat /etc/secrets/secret_key)
export POSTGRES_PASSWORD=$(cat /etc/secrets/password)
export POSTGRES_USER=$(cat /etc/secrets/username)

bundle exec rake db:create db:migrate
bundle exec rake assets:precompile
bundle exec rails server -b 0.0.0.0

I also need to modify config/database.yml (the Rails database config file) to use the environment variables created in init.sh.

production:
  <<: *default
  adapter: postgresql
  encoding: unicode
  database: todo_production
  user: <%= ENV['POSTGRES_USER'] %>
  password: <%= ENV['POSTGRES_PASSWORD'] %>
  host:     <%= ENV['DB_SERVICE_HOST'] %>
  port:     <%= ENV['DB_SERVICE_PORT'] %>

Because I changed the Dockerfile, I need to rebuild the Docker image for the Rails container. I’m tagging this version of the image as v2. Once the image is built, I push it up to the Google Container Registry.

docker build -t todo .
docker tag -f todo gcr.io/my_project_id/todo:v2
gcloud docker push gcr.io/my_project_id/todo:v2

Just like with the database I need to modify the pods to use the new image and to access the secrets. The Rails pods are created by the web replication controller so I make my changes to web-controller.yml

# web-controller.yml
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: web
  name: web-controller
spec:
  replicas: 2
  selector:
    name: web
  template:
    metadata:
      labels:
        name: web
    spec:
      volumes:
      - name: secrets
        secret:
          secretName: secrets
      containers:
      - image: gcr.io/my_project_id/todo:v3
        name: web
        volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
        ports:
        - containerPort: 3000
          name: http-server

To start up the Rails pods and web service I follow the same steps as I did in the previous Kubernetes blog post.

kubectl create -f web-controller.yml

kubectl get rc
kubectl get pods

kubectl create -f web-service.yml

kubectl get services

The final step is opening up the firewall. This step doesn’t change at all:

gcloud compute firewall-rules create --allow=tcp:3000 --target-tags=your-node-name-here
gcloud compute forwarding-rules list

In the next post in this series, I’ll show how to make your database pod(s) more fault tolerant with a persistent disk.

Aja Hammerly

What are secrets?

Creating the Secrets File

Modifying the Database Pod

Modifying the Rails Pods