Kind + Gitlab CI

When developing Kubernetes Operators, testing becomes an interesting conundrum. Unit testing works great, but envtest (part of the k8s controller-runtime package) has some big limitations if your operator expects to interact with other standard controllers - or needs to delete something. We’ve jumped through a lot of hoops to try to get something to work, and in general always feel like we come up short or need to make too many exceptions for it to be useful.

So I set out to figure our a solution to test in a clean kubernetes environment. At present, my constraints are:

Use Gitlab CI (I generally like Gitlab CI, so it’s not a huge constraint)
Use linux docker containers as the CI runner base
Don’t spend money spinning up cloud clusters (in fact, try to do it with zero cloud spend)
Test Operators in isolation in a cluster

To get started, we’re going to use Kind. Kind stands for Kubernetes in Docker. It runs Kubernetes nodes as docker containers. The Kind tooling expects to interact with docker directly, which means we have to be able to Docker in Docker(dind) in Gitlab CI to make this work¹.

Let’s start building up the .gitlab-ci.yml file. In my case, I’m only using dind in a few jobs, so I will be defining all of the variables and services in a single job.

# .gitlab-ci.yml
kind test:
  stage: test
  image: golang:1.22.5
  tags:
    - linux
    - docker
  services:
    - name: docker:dind
  variables:
    # Docker in Docker variables
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""

This will start up the dind service in the background, and make it available to our job. The DOCKER_HOST variable is used by the Docker client to connect to the dind service. We’re playing fast and loose with security here, so we aren’t using encrypted connections to the docker host.

Our next order of business is to start kind and have it reference the dind service. By default, kind uses localhost as it’s API endpoint, which includes references in the certificates generated. But, we can control some of this with a configuration file for kind like this:

# test/test-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4

# This has the API server listen on all IP interfaces. In the Gitlab CI
# environment, it allows us to access the kubernetes API from outside
# of the dind service.
# WARNING: In normal use of kind, it is _strongly_ recommended that you
# keep this the default (127.0.0.1) for security reasons. However we
# need to change this to work in gitlab CI with Docker-in-docker running
# as a service.
networking:
  apiServerAddress: "0.0.0.0"

# This adds the hostname 'docker' to the Subject Alt Name (SAN) list on the
# certificates created by kubeadm for the kind cluster. This matches the
# alias that Gtlab CI uses to start up the dind service. The Kube API port
# is exposed on the dind service port, so we need to use that name.
kubeadmConfigPatchesJSON6902:
  - group: kubeadm.k8s.io
    version: v1beta3
    kind: ClusterConfiguration
    patch: |
      - op: add
        path: /apiServer/certSANs/-
        value: docker

With this configuration, we can add some before_script steps to the CI job to install tools and start the cluster.

# .gitlab-ci.yml
kind test:
  stage: test
  image: golang:1.22.5
  tags:
    - linux
    - docker
  services:
    - name: docker:dind
  variables:
    # Docker in Docker variables
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    KIND_VERSION: v0.23.0
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends docker.io
    - curl -SsL https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-$(go env GOOS)-$(go env GOARCH) -o /usr/local/bin/kind && chmod +x /usr/local/bin/kind
    - kind create cluster --name test-cluster --config ${CI_PROJECT_DIR}/test/test-cluster.yaml --wait 5m;

This should result in a test cluster ready to go. However, interacting with it involves a couple more steps. In the configuration above, when we set the network address to 0.0.0.0, that populates the kubeconfig generated by kind with a KubeAPI address of 0.0.0.0, and that isn’t going to work in the CI environment. So, we have to make some quick adjustments to swap out the IP with the docker alias name used in this job:

# .gitlab-ci.yml
kind test:
  stage: test
  image: golang:1.22.5
  tags:
    - linux
    - docker
  services:
    - name: docker:dind
  variables:
    # Docker in Docker variables
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    KIND_VERSION: v0.23.0
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends docker.io
    - curl -SsL https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-$(go env GOOS)-$(go env GOARCH) -o /usr/local/bin/kind && chmod +x /usr/local/bin/kind
    - kind create cluster --name test-cluster --config ${CI_PROJECT_DIR}/test/test-cluster.yaml --wait 5m;
    - kind get kubeconfig --name test-cluster > /tmp/kubeconfig
    - sed -E -i 's#https://0\.0\.0\.0:#https://docker:#' /tmp/kubeconfig

Okay. Now we should have a cluster ready to go with all of the pieces ready to connect. We just need to configure envtest to do so. Convenient for our purposes, this can be accomplished with a couple more environment variables. I prefer to control this in the job for CI purposes, and with other mechanisms for local development. So, our final job configuration looks like this:

# .gitlab-ci.yml
kind test:
  stage: test
  image: golang:1.22.5
  tags:
    - linux
    - docker
  services:
    - name: docker:dind
  variables:
    # Docker in Docker variables
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    KIND_VERSION: v0.23.0
    KUBECONFIG: /tmp/kubeconfig
    USE_EXISTING_CLUSTER: true
  before_script:
    - apt-get update && apt-get install -y --no-install-recommends docker.io
    - curl -SsL https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-$(go env GOOS)-$(go env GOARCH) -o /usr/local/bin/kind && chmod +x /usr/local/bin/kind
    - kind create cluster --name test-cluster --config ${CI_PROJECT_DIR}/test/test-cluster.yaml --wait 5m;
    - kind get kubeconfig --name test-cluster > /tmp/kubeconfig
    - sed -E -i 's#https://0\.0\.0\.0:#https://docker:#' /tmp/kubeconfig
  script:
    - make test-e2e

Of course, this doesn’t look at how to write tests for this setup, and maybe I will look at that another day. But with this² we are able to do some more proper end-to-end testing without a lot of extra spend. I’m sure there are refinements that could be made, and I’d love to hear if you’ve done something like this. Send me a message on mastodon to continue the conversation!

If you want to use Docker in Docker, please research and understand the security implications of doing so. In a shared environment, this might open up new threat vectors that docker alone does not. Security is always about layers³. No one layer will get you compromised, but no one layer can protect you either. Talk to the people in your sphere that know your environment to understand if there are issues that might prevent these techniques from being used safely. ↩
We actually have more of these steps in makefiles, including installing kind and starting the cluster. I pulled these steps out for clarity in this post. ↩
Like an ogre, or an onion, or like swiss-cheese. ↩

Kind + Gitlab CI

Footnotes