KUBERNETESFirst impressions on GitOps and Kubernetes

This post was first published on Øysteins private blog blixhavn.dev

The last few months I have been setting up a Kubernetes infrastructure for my client. With no previous experience with Kubernetes it is safe to say the learning curve was quite steep, and as I am sure most others who have walked the same path have experienced, the amount of choices in terms of concepts, solutions, tooling and workflows is massively overwhelming. It is as if all I knew about software operations suddenly became obsolete, and I had to "learn anew" a lot of tools and concepts - because even the most basic concepts are just a little bit different in Kubernetes.

Between all I have learned the past months however, there is one concept that stands out to me as the most powerful - namely GitOps.

GitOps follows the principle of Infrastructure as Code, in that the desired state is declaratively described, with tooling software performing the necessary operations to achieve this state in the cluster. The upsides seems rather obvious:

1. You have an explicit definition of what your cluster is supposed to look like, which is actively referenced by the GitOps agent. This avoids configuration drift that might occur if developers are allowed to perform direct manipulations to the cluster.
2. All changes to the cluster are stored in version control, making it easy to revert to previous states (e.g. image tag deployments), as well as auditing changes.

The three current alternatives for GitOps controllers are ArgoCD, Flux and JenkinsX. ArgoCD and Flux are both GitOps specific tools, where the first one is more powerful and configurable, while the last one is simpler to get started with, but lacks some features and configurability (and a web interface). JenkinsX on the other hand, attempts to be a wholistic solution, with integrated CI/CD pipelines, preview environments, environment promotion (dev→stage→prod), as well as GitOps capabilities.

Not everything can be declarative

When I first discovered GitOps, I tried setting up my cluster configuration repository to be completely declarative, to the point where I wanted to be able to point it to a new cluster, run one install script from my computer, and that would be it. Unfortunately, I never achieved that.

There are some applications where the install procedure is not straight forward enought to be performed automatically. Maybe in theory, but it proved too convoluted for me. For example Linkerd generates a root certificate when installed into the cluster, and this needs to be used by Cert-Manager to create an intermediate certificate. The public key of this certificate then needs to be added back to the configuration of Linkerd. After a while, I settled for having a configuration that required some manual steps in the initial setup.

Environment handling is difficult

It took quite some experimenting, research and discussion to arrive at the setup we have for our development and production environments. It's tempting to use git branches, the tool we developers most often use for separating environments. However, this comes with a number of challenges, which are all neatly summarized here.

What we arrived at was in principle the solution proposed here. The GitOps controller is installed in the production cluster, configured to control both clusters. Instead of one environment per branch, both environments have configuration in the same branch, just in different folders. This makes environment promotion (almost) as simple as copying a file from one folder to another.

Apart from these issues, diving into GitOps has been an absolute joy. It is very satisfying to create a new application definition, push it to GitHub, and then watch as ArgoCD spins up the resources for me. With ArgoCD Notifications, it is also easy to be notified whenever a problem occurs with either updating the cluster to a new desired state, or if an application has trouble keeping its pods running.

I admit I am one sided on this, seeing as GitOps was such a fundamental part of my first Kubernetes experience, but I honestly can't imagine administering a Kubernetes cluster without using GitOps. I use it, and so should you!