Not too long after Kubernetes became popular, it became evident rather quickly that maintaining all the manifests involved in the deployment of a single application was going to be a cumbersome process. Having a different set of manifests for every environment you want to deploy to is simply unacceptable. People quickly realised that applying some DRY to the problem was very much needed.

That’s how HELM was born. The idea of HELM was to create highly-configurable “packages” that would contain all the dependencies of a particular application. So, if you wanted to deploy your own self-managed instance of Gitlab, the HELM chart for it would contain everything you needed: the database, the main application, the runners, the exporter, Gitaly, etc. You just needed to tweak some configuration values and that was it. Configurable and DRY Kubernetes was born.

HELM achieved this by making Kubernetes manifests a template (specifically, a Go template) and interpolating variables to render the actual manifests to be used by Kubernetes.

Quickly, we realized that there are several issues with this approach.

Increased learning curve

The first evident problem is that you are introducing a layer of abstraction on top of another layer of abstraction -- this is how Software works anyway. Now, you need to understand not only the something you are abstracting (in this case, k8s and manifests, which are quite full of learning already) but also the abstraction itself and its inner workings.

New concepts are introduced like Charts and Repositories. Now you need to learn how a chart is structured so you can build your own and then learn Golang template syntax if you don’t know it already. All these things, although not a major burden, hinder adoption by steeping the learning curve a bit.

Templating limits customisation

It should be fairly obvious that, in any solution to this problem that involves templating, only the values that are parametrised are the values that you can configure. Unfortunately, you can’t change the rest. Since every organisation or team has different needs, everyone needs to customise different aspects of the Chart. But again, the chart has to be parameterised for that specific thing you need to configure. In other words, if you want full control, a chart should parametrise every single possible yaml value, which is almost impossible and also very impractical. Suddenly, HELM charts that could have been simple K8s manifests become something unbearable to read.

This problem above is why you have like 20 different HELM charts per application. Everyone needs to customise different things or do things slightly differently.

Charts make choices for you

Because every chart has to contain all the units to make an application work, and because you cannot parametrise everything, then unavoidably some choices are made for you. Some charts deploy a Postgres instance using a StatefulSet with the official Postgres image, but your setup needs probably a PostgresCluster resource coming from the great Crunchydata Postgres Operator. Or maybe you have a database already available in Aurora and don’t need the chart to create one for you, but you cannot disable it.

Again, the impossibility to control these and other aspects of a chart is what leads to a chart explosion for the same applications. This for me is the strongest statement of the failure of HELM to solve the DRY problem Kubernetes Manifests have. It has created more complexity.

Kustomize: A better alternative

The Kubernetes team understood this quickly and moved to embrace a better approach natively inside kubectl. The kustomize tool proved to be simpler, more transparent and more effective at solving this problem than HELM.

The paradigm behind kustomize is that, instead of having a manifest template, we would have a normal Kubernetes manifest with some sensible defaults that we would use as a base, and then, in another manifest called overlay, we would apply operations to the base manifest to add, remove or change its values, or whole nodes of information if we so desire.

Kustomize does this by the use of common standards like JSON Path and JSON Patch.

This change of paradigm (towards composition) has several advantages over a templated approach.

First, base manifests do not need to contain every possible configuration and its parametrised form. In fact, the shorter they are, the better. Second, overlays only change and touch what is needed, which also keeps them thin. And third, overlays can be stacked one on top of the other to increase reusability.

Another benefit is that you don’t need another binary to use Kustomize. It is supported natively by using the -k flag on kubectl apply or the kustomize subcommand in kubectl.

Kustomize drawbacks

Not all that shines is gold. There are still some drawbacks to this approach, as with everything in engineering.

For starters, you need to learn how Kustomize reads your kustomization.yml files placed in your manifests, but also learn how JSON-Path and JSON-Patch work. So there is a cognitive load that increases the learning curve.

Having said that, this is much easier to make sense of than HELM. JSON Path is a very simple spec, as Json Patch is too -- you can learn both in less than a day. Also, only engineers involved in customisation need to know the advanced stuff; the application engineers can write their Kubernetes manifests as they see fit aided by the Kubernetes documentation. This will be overridden by the CD tool using Kustomize. This closes the gap between Development & Operations (SRE), furthering the DevOps culture across the organisation.

Final Thoughts

I greatly encourage you to give it a go, especially if you have a HomeLab Kubernetes cluster. I recently ported all stuff in my cluster from Helm to Kustomize and I’m enjoying how much simpler my cluster git repository is.