Kubeconfig Management

I work with a lot of different clusters in my day job. In addition to working with both local/dev and remote clusters, I’m regularly needing to compare resources in different clusters. Long ago I tried to make things work with kube context switching in a single kubeconfig file. I tried the newly forming ideas from kubectx project, but that didn’t work when I had multiple terminals open needing to look at different clusters side by side.

Instead I decided that I was going to just put everything in different kubeconfig files, one file per cluster access level. Every kubernetes CLI tool looks at the KUBECONFIG environment variable, and those are scoped to terminal sessions (or even specific commands). Now I can start a terminal, set the env var, and run whatever I want pointed at the right cluster config. Multiple terminals can be open with different clusters. Except, now I have a new problem: how do I know which cluster a terminal is referencing.

Display

I have a pretty extensively customized PS1 prompt¹. I include a section that checks to see if there is a current kube context, and if so displays the context name (which translates to a cluster name in my setup) and a namespace if one is set for the context. It ends up looking like:

09:57:43 ~/workspace [blip-k0s:apps]
$

In this case, the cluster context is blip-k0s and the namespace is apps. It’s right there for me before every command, easy to reference and verify. This also give me the option to color it for specific clusters, to make sure those prod clusters are known with visually obnoxious colors to catch my attention (want to take a guess at why I need this one?).

Here’s a snippet from the script that generates my PS1:

  KUBE_INFO=""
  kubecontext=$(${kubectl} config current-context 2>/dev/null)
  if [ $? -eq 0 ]; then
    color=$YELLOW
    case $kubecontext in
    *dev*) color="${GREEN}" ;;
    *preprod*) color=$YELLOW_BOLD ;;
    *prod*) color="${RED_BOLD}${YELLOW_BACKGROUND}" ;;
    *) color=$YELLOW ;;
    esac
    kubens=$(${kubectl} config view --minify --output 'jsonpath={..namespace}')
    KUBE_INFO=" ${color}[${kubecontext}:${kubens}]${RESET}"
  fi

Later in the script ${KUBE_INFO} is added to the PS1 definition.

Config File switching

Typing out KUBECONFIG=.... every time I want to do something, or exporting it as an environment variable can be a hassle. So I’ve also added a simple function that switches between different config files. I wanted this to be as short to type as possible². I settled on kc for kubeconfig.

An important part of the setup is a common naming convention for my config files. Everything lives in my ~/.kube directory. Each filename is prefixed with kubeconf- and is a ‘friendly name’ after that. It doesn’t have to be connected to the context name or anything else, just something that I use to reference the setup. I tend to not have any file extensions, but that isn’t a strict requirement. Sometimes I have one sneak in and everything still works.

The resulting bash function is:

# kc set kubeconfig based on what was passed in
kc() {
  export KUBECONFIG="${HOME}/.kube/kubeconf-$1"
}

With this, I don’t need to type the full path or even the full file name, I can just use the ‘friendly name’.

Autocomplete

As the list of clusters grew, with names that were really close in some cases, I got lazytired of typing as much and wanted some autocomplete functions. The full setup of the bash completions definitions is outside of the scope of this post, but I was able to whip up the following:

#!/usr/bin/env bash
_kc_kube_configs() {
  local curr_arg
  curr_arg=${COMP_WORDS[COMP_CWORD]}
  mapfile -t COMPREPLY < <(compgen -W "$(find ~/.kube -name 'kubeconf-*' -maxdepth 1 | cut -d'-' -f2-)" -- "${curr_arg}")

}

complete -F _kc_kube_configs kc

On my setup, files in ~/.completions are included, so this just lives as a file in that directory. It lets me type kc <tab> and get a list of the friendly names that match a given prefix.

End

This setup has been very handy and stood up to years of use. The number of situations that I hit where my setup just works and is safe only grows. I have never had a situation where I wished I had a single file context, and certain workflows are just impossible for colleagues who have a single file setup. This setup is obviously very bash heavy, I don’t know if everything can be translated to Powershell. Some of my team members have tried, but company policies have stopped us from getting something like this that works on Windows.

Footnotes