Since the dawn of time, man has sought to deploy web apps. We shoved boxes into colo racks and rawdogged nginx in front of LAMP stacks. It worked.
Mostly- until it didn't.
Then it just kind of sucked.
One day, some companies went: "hey, running racks sucks. what if we rented out slices of ours for a fee?"
Man was given massive warehouses full of servers, and a fancy UI to operate them. And from that, ClickOps was born.
Developers quickly discovered that ClickOps sucks too, and started scripting the hell out of it. Shell scripts, Python one-offs - and anything that could talk to the API. All of it imperative. Often, that code was not idempotent, so "just run it again" turned into having a few extra Load Balancers or databases, billed by the hour of course.
Eventually, people realized that rewriting the same fragile provisioning scripts over and over also sucked.
Out of that pain came Ansible, with its idempotent but still kind of imperative playbooks. Then came Terraform, and - for services - Kubernetes. Both with more declarative goals in mind.
These were a breath of fresh air. Until, predictably, we realized these all suck, too. So we started to build more tools for our tools.
See, Kubernetes had this grand idea: "We'll define a shared manifest format, and people will write proper programs to generate those manifests dynamically!"
In reality, no one did that. Imagine writing a custom script to generate a single application manifest package that only works in your specific k8s environment. Are you using istio or linkerd? or perhaps nginx ingress? It was _fucking insane, and very obviously not portable_.
Anyway - y'know how k8s manifests are YAML because it's "nice" and "human readable"? It's even got that cute whitespace thing going on, so you always know what depth you're at!
Allow me to introduce you to the seventh circle of hell: slapping Go templates on those YAML files. Because that whitespace thing will definitely never cause problems with multi-line string replace.
And This solution was so much gooder than the alternatives, that it became the de facto standard.
I mean, good luck ever crafting one of these templated YAML abominations, reading them, or accessing non-templated values in third-party ones - But what does that matter? it was decidedly so much gooder.
Now it's almost 2026, and afaict, Helm is still the de facto way to deploy web apps onto k8s, and I'm here to tell you... Helm sucks.
Terraform sucks too.
Pulumi is marginally better (but still sucks).
Kubernetes is decent, but... it sucks. Ask anyone who's used it; they'll tell you just how bad it sucks.
Ansible... actually, Ansible, you're cool. At least you try to be idempotent. Well done.
and VyOS... My sweet summer child...
VyOS is my favorite project that falls on its face in spectacularly mundane ways. Maybe it's a bit sunnier on the Juniper side of things? But alas, I'm stuck with you, dear VyOS. You, who implements a DSL for configuration, but can't handle NAT hairpinning despite every consumer router supporting it for decades...
You who brings lies of declarative configuration.
Applicable, error checked, idempotent... but not self-healing.
You see, when I remove a resource from my source of truth, I want that resource to no longer exist. But alas... the only way to remove a resource is to write a script and remove it manually.
And I understand why. The configuration we apply is merely a diff of desired state to wed. There is no observe-diff-apply loop. No runtime. Just observe, union, apply-once. Just to spite me.
I love you, nonetheless.
And as for my mortal enemy, Terraform... You were a disappointment from the moment you were born. Were your developers given time to think before acting, perhaps we could have had a little good in the world.
The problem with this abomination is that... it's not actually declarative. Not in practice. And not only that - it imposes restrictions on how you can even be imperative. It is, quite literally, the worst of both worlds. This is why Pulumi is kind of okay in my books, but honestly it still sucks at its core, and you can't fix that with more tooling.
And at this point, you may have realized: I'm simply not sane.
Who throws around all these concepts and technologies without explaining them in any way, shape, or form?
You see, dear reader, it is I who'st expecteth thee to read this twice. For now that you feel my pain, I can explain to you what it means to be declarative. For perhaps I was the most sanest all along - and thou wert mine fool.
hello previewers, this is where I stopped writing coherently, and most of after this is somewhat mishmashed and incomplete with no guarantee of a central thesis to guide it
important shit
As we've learned from Terraform, declarative configuration describes the state you want, but not the steps to get there.
A runtime controller/reconciler figures out the steps to get there.
Together, that buys you idempotence, meaning reapplying won't change state; auditability, a clear diff between desired and actual state; and self-healing via automatic drift reconciliation.
Most declarative tools (in name) don't abide by these philosophies due to "technical tradeoffs".
Case Studies:
Terraform (and the AWS Provider)
Terraform calls itself declarative, but it’s really a one-shot graph executor. It doesn’t watch or reconcile - it plans, applies, and leaves. As configuration changes over time, it effectively becomes imperative over time.
Imagine you have a configuration with declares resources Alice and Bob, where Bob is a child added to Alice.
Then later, a back-reference is added to Bob - perhaps something as mundane as a note or tag referencing Alice.id.
Now, imagine you took this "declarative" configuration and created a new environment.
Perhaps, for example, when creating a staging environment based off of the development environment. Terraform fails to reconcile the configuration, because neither Bob nor Alice can be fully resolved. They form a cyclical dependency.
Without partial reconciliation, we are stuck.
So when EKS shits the bed and rancher decides to delete your node pools because of that, you're fucked. Your cyclical dependencies will require deleting resources, making them from scratch, removing and re-importing state, and re-adding the cyclical dependencies.
This is a real experience I've had on a brownfield project I joined. It took 20+ hours, overnight - starting from ~10pm EST - to recover from that shitstorm, because we had to tear down the entire stack, and bring it back up piece by piece, finding alllll the cyclical dependencies former devs had made.
Mind you, it's entirely possible to organize your terraform to avoid cyclical dependencies, but it's not always intuitive or easy. It's certainly not a well-advertised bugbear.
And these aren't the only problems terraform has. Much of the blame I personally place on the Providers like AWS. Who, last I checked, often don't implement Read() or Update() correctly.
There's no partial reconciliation, which means resources that aren't fully-formed can't get made, which is made worse by the aforementioned Read/Update implementations, which forces resources to get deleted and remade because you can't just update the name of a Security Group, no - why would you ever want to update just the display name of a security group?
And if you've layered your terraform deployment to avoid cyclical dependencies, now you have to worry about all your downstream resources pointing to an old or even deleted resource.
at this point I'll mark that the previous paragraph(s) isn't written in good faith or good memory, as this chapter of my life was well over a year ago, and I don't remember the exact details of working within AWS/Terraform, but- I hope this paints a close enough picture for why I dislike terraform.
and all of this is made even worse by the difference between the state cache, the configuration files, and the actual state of infrastructure.
which one is the truth, Terraform?
and as an aside, why do we need to implement dynamic configuration semantics like ifs on top of for_each and count - if we're admitting that sometimes we can't live in pure declarative land, and realizing that due to providers and dependency-cycles, we are forced to behave imperatively, can we just-... allow that?
I mean I guess that's why pulumi exists...
*sigh...*
Kubernetes
The Kubernetes' reconciliation loop works - controllers actually watch and repair state - but what you declare is often too granular, and too manual.
I actually kind of love Kubernetes. I'll only tell you this buried deep within a blog post, or after a long night that ends in a dive bar - but, it's mostly good technology.
The definitions of Deployments, Services, ConfigMaps, Secrets, etc - are all very functional. But sometimes I just don't care to dig so deep to deploy a simple web app. There's no higher-level abstraction for "this is a web app with X replicas, ingress Y, config Z."
Hence: Helm, Kustomize, and every operator ever.
This is made worse by everything being string-y in yaml, where resources are linked together by labels and not something more semantic like Types or references.
If you rename something in one place, you should hope you haven't missed that rename anywhere else, because you're about to break routing for your database and lose a bunch of events.
But that's why we have helm, right? because templating fixes that issue! Well, at least the "renaming" issue. But then, obviously templating is gross and requires a lot of labor to do correctly. So then we have Kustomize, which is now built in to kubectl, but doesn't have quite the same "package manager" feeling to it that helm does.
And in another vein, we're still missing some static analysis that tells us whether an Ingress that we're defining points to a Service that doesn't exist. That failure mode happens at runtime only.
ArgoCD
I have a love/hate relationship with ArgoCD. I mean, it looks declarative. The idea of it's only true if it's in git sounds good on paper.
I'd had a list of complains here but I found out a lot of them have reasonable workarounds, like sync waves.
I still don't love the way you have to interact with it through git - sometimes I just want to save a file and test out a config without shitting all over my git history. (@GithubActions).
Ansible
Ansible at least tries to be idempotent...
- some modules check for state, others just blindly run commands
- no long-lived reconciliation
- inventory drift
Too Long; Didn't Read/Write
Kubernetes is almost good, Terraform is objectively bad - Crossplane exists because of that, but it's also... I mean, man. That's certainly an idea.
Anyway, we should be using KCL/Pkl for configuration - something with types, references, and actual semantics instead of string-typing relationships - with a nice package manager and tooling like cargo/crates.io.
If I can dnf install caddy why can't I kubectl install caddy and get static files vserved just as fast?