I didn’t plat to write this article but when I wrote “I always start with a Terraform module” lovely folks on Reddit spent kind words not about the idea of having a module on top of the default root one by default but complaining about the idea of using Terraform to pilot Kubernetes.

Something that was not at all part of the article. So I put in my backlog a dedicated one since it got a bit of traction.

First a little disclaimer that quickly turned to a mantra for me: “Tools are not solutions“, the right answer when it comes to computers is “it depends”, so I do not use Terraform to pilot Kubernetes as unique and final solution, everywhere, but it happened to be useful and to survive the test of time for long enough that it is a valuable solution for me.

So I know that there are other tools useful to manage Kubernetes, there is Flux, you can check out YAML and apply it via kubectl, I used operators in the past, I wrote custom resource definitions, at some point I have also experimenting with my own reactive planning on top of the go Kubernetes client but recently when I start a new project I go with Terraform. Recently my own friend David Rawkode shared a talk he gave at InfoQ just about this topic.

Terraform is usually already setup somewhere. Kubernetes is not something you onboard since day one, you start from provisioning your region with security groups, VPCs, connection with external regions and datacenters via peering, maybe databases, public ssh keys, VPN, DNS records… To do all of that I end up with a repository called infra hooked to CI/CD that runs Terraform plan/apply.

When it is time to hook Kubernetes into the mix the question is: should I use something else? Is the current setup enough?

This is part of the process of finding a solution, if the answer is YES what I have is enough you analyzed the pros and cons, and you are ready to jump onboard. It will survive forever? Will Terraform be the unique and forever solution? Probably not, because we are not married to our solutions.

Terraform provides a Kubernetes provider that proxies all the resources that you usually deal with in Kubernetes land: pods, services, deployment and so on to Terraform resources. For what it is not covered by the provider you can write plain manifests. This is for sure something to take under consideration if you rely on many CRDs, or resources that are not “default” in Kubernetes you will end up writing manifests, but at least for me, it was not a problem. Onboarding Flux or Pulumi is not for free. I mainly had to deal with it for cloud resources like on GKE autopilot (GCP) if you want to configure a custom health check for a service you end up with manifests for Backend Configs. It looks like this:

resource "kubernetes_manifest" "backend-config-api" {
  manifest = {
    "apiVersion" = "cloud.google.com/v1"
    "kind"       = "BackendConfig"
    "metadata" = {
      "name"      = "http-api"
      "namespace" = var.environment_name
    }
    "spec" = {
      "timeoutSec" = 3600
      "healthCheck" = {
        "checkIntervalSec" = 15
        "timeoutSec"       = 10
        "port"             = 9091
        "requestPath"      = "/healthz"
        "type"             = "HTTP"
      }
    }
  }
}

Nothing crazy but Kubernetes is a complex beast and the Terraform provider does not have a way to offer a structured Terraform resource for unknown resources.

What about continuous delivery?

Infra as code when it comes to continuous delivery turns to be controversial, for simple use case, usually the one that well fit to Terraform for Kubernetes the solution I used is to set the ImagePullPolicy to Always and instead of delivering the image with the git SHA as tag I use a branch name.

Let’s say that you want to put your Public API in continuous delivery from your main branch in Git. The Kubernetes deployment will have the image set to hub.docker.com/companyname/public-api:main and in continuous delivery you can authenticate against your Kubernetes cluster and run the command:

$ kubectl rollout restart deployment public-api

To force Kubernetes to check if a new image is available and deploy the new pod.

Pros and cons? For sure!