Working with multiple kubeconfig files
separate clean kubeconfig files to easily manage clusters
Working with multiple kubernetes clusters can be a hassle, starting with the kubeconfig file, where you have to add all your clusters configs into that file, and update it manually whenever you want to add a new cluster. That's what I thought is the only way, but then I found another way, much simpler and provides a better developer experience.
Surprisingly, there is no extra tools needed, its just a feature of kubectl, as it reads an environment variable tells it where to look for kubeconfig files.
Prepare kubeconfig files
create a new directory in my home folder, will call it
then copy our clusters kubeconfig files,
this will copy
kubeconfig2 to the new directory
we just created.
cp kubeconfig1 kubeconfig2 ~/clusters
For a better example we will assume that kubeconfig1 is a sandbox cluster, kubeconfig2 is a kubeconfig file for staging.
cd ~/clusters mv kubeconfig1 sandbox.yml mv kubeconfig2 staging.yml
clusters ├── exp.yml ├── test.yml ├── dev.yml ├── production.yml ├── sandbox.yml └── staging.yml
now we are ready to tell kubectl to use these files
When kubectl command is executed, it checks where to find kubeconfig files,
if nothing is specified, it falls back to the default file
We can tell kubectl to use the files we have prepared, by setting the environment
KUBECONFIG with a list (separated by a colon ":") of files, they can be anywhere,
they don't need to be in the same directory even.
Open your shell and set the variable
Now use kubectl to get list of contexts, I will use kubectx
note: the list of contexts will depend on the names used in the kubeconfig files (not the file names)
$ kubectx exp k8s.test.com k8s.dev.eg production.company.org sandbox staging
Make sure to set that variable every time your shell starts, you can do that by setting it in
.zshrc, depends on your shell.
// .bash_profile export ... export KUBECONFIG="~/clusters/sandbox.yml:~/clusters/staging.yml" export ...
Tada ! thats it, you are ready to get going and add as many files as you want.
Automate detection of kubeconfig files
Remember that directory we created in the first step? We can write a tiny script
that loops over the files in that directory and set the
# default kubeconfig FILES="$HOME/.kube/config" # create clusters directory if its not created CLUSTERS_DIR="$HOME/clusters" echo "creating clusters dir: $CLUSTERS_DIR" mkdir -p "$CLUSTERS_DIR" test CLUSTERS_DIR for cluster in `find $CLUSTERS_DIR -type f -name "*.yml"` do echo "found cluster: $cluster" FILES="$cluster:$FILES" done export KUBECONFIG=$FILES
Will not go through the code details in here, but you can copy that script into a file
and source that files in your
That's all folks!