Our Logging Stack Worked For Two Years. It Was Also Shipping Logs As Admin.

We run our own logging. OpenSearch, driven by the OpenSearch Kubernetes Operator, with Fluent Bit on every node shipping container logs into it, on the cluster that hosts our internal infrastructure. It has been up for months. Dashboards loads, the searches return, retention keeps the disks from filling. By every signal anyone was looking at, it worked.

Last week we went to turn it into a stack profile, which is Ankra’s way of taking a stack that runs on one cluster and making it a versioned, parameterised thing anyone can launch on any other cluster. That should have been an afternoon of naming variables. Instead it turned into an audit, and the audit found that a good part of the stack had been quietly doing nothing at all.

None of it had ever caused an incident. That is the uncomfortable bit, and it is the reason this is worth writing down.

Parameterising a stack is an audit you did not schedule

Making a stack reusable forces one specific question over every line of it: does this value belong to the software, or to the cluster we happened to build it on? Answering that means reading the whole thing properly, probably for the first time since the day it went in.

The obvious answers came out fast. The Dashboards hostname was a literal in a Cloudflare tunnel binding, and the tunnel binding referenced a ClusterTunnel that exists on exactly one of our clusters. There was a LoadBalancer Service carrying Hetzner annotations, pinned to the fsn1 region, publishing the OpenSearch API to the internet. On any cluster without the Cloudflare operator installed, the first of those does not fail gracefully, it fails to apply at all, because the custom resource definition is simply not there. The second works only on Hetzner and is a bad idea everywhere.

That is what you expect to find. It is embarrassing in a mild way and takes ten minutes to fix.

Then we got to the credentials, and the questions stopped being cosmetic.

The credentials that went nowhere

The stack created a Kubernetes Secret called fluentbit-opensearch-credentials, with a username and a password. The profile we had captured from it correctly asked whoever launched it to supply both. It was the only thing the profile asked for.

Nothing read that Secret.

Fluent Bit’s Helm values pulled its OpenSearch username and password from a completely different Secret, infra-admin-password, which is the one the operator generates for the cluster administrator. So the log shipper on every node had been authenticating as the OpenSearch superuser for the entire life of the stack. Not a scoped ingest account with write access to the log indices. The account that can read every index, delete every index, and change the security configuration.

There was an intended design. A ConfigMap sat next to it named opensearch-fluentbit-security-config, holding a nicely written internal_users.yml that defined a fluentbit user, a roles.yml granting it create and write on an index pattern, and a roles_mapping.yml tying the two together. Someone had thought this through properly.

The OpenSearchCluster resource never referenced it. The operator applies security configuration when you point spec.security.config.securityConfigSecret at it, and that field was not set, so the ConfigMap was inert YAML sitting in a namespace. For extra confirmation that it had never run, the bcrypt hash in it was a placeholder with a visible run of repeated characters, the sort of thing you paste in intending to replace before you apply. And the index pattern in the role granted rights on fluent-bit-* while the indices Fluent Bit actually created were named infrastructure-*, so even if it had been applied, the user it created could not have written a single log line.

Three separate mistakes, stacked, and the stack worked anyway. It worked because of them. The fallback to admin credentials was covering for the fact that the intended user did not exist, and nothing anywhere reported an error, because from OpenSearch’s point of view a valid admin was writing valid documents.

This is the failure mode that quiet infrastructure specialises in. A thing that is broken and visible gets fixed on the day. A thing that is broken and compensated for by an over-permissioned fallback survives for years, and the only symptom is a blast radius nobody has looked at.

The retention policy that was aimed at everything

The other finding was closer to dangerous.

OpenSearch’s Index State Management deletes indices on a schedule, and an ISM policy can attach itself to new indices automatically through an ism_template with a list of index patterns. Ours was:

ismTemplate:
  indexPatterns:
    - '*'
  priority: 100

Delete anything older than fourteen days, where anything means anything. OpenSearch keeps its own state in indices: the security configuration, ISM’s own job store, the Dashboards saved objects and index patterns. A template matching * at priority 100 is aimed at those too.

We had been running that for months without losing the security index, which tells you the plugin’s handling of system indices was more forgiving than the policy deserved. It is not a guarantee, it is not documented behaviour to rely on, and it certainly is not something to publish for other people to launch on their clusters. A retention policy should be scoped to the indices you meant, and never expressed as “everything, and trust the platform to make exceptions on my behalf.”

What we changed, and what production actually means here

The rebuild is published now as version 2 of the profile, and version 1 is still in the history where it belongs.

The ingest identity is real. The operator ships custom resources for users, roles and role bindings, so the fluentbit user, its role and the binding between them are now declared as Kubernetes resources and reconciled like everything else, with the password taken from the one Secret the launcher supplies. That single input is now genuinely wired to the thing it names, and the role grants create and write on the log index pattern only. Fluent Bit can no longer read anything, and it cannot touch the security configuration.

The ingest connection verifies TLS. It had been running with verification off, which is a strange thing to do inside a cluster where the operator generates a certificate authority and puts it in a Secret specifically so you can trust it. Fluent Bit now mounts that CA and checks the certificate.

Retention is scoped to the log index prefix, so the policy can only ever match the indices this stack creates.

The topology changed too, and this is the part that separates a demo install from something you would put a retention promise on. Previously there was one node pool of three nodes carrying every role at once: cluster manager, data and ingest. That works right up until a data node gets busy or a disk fills, at which point the same JVM that is struggling is also the one holding cluster state. Now there are three dedicated cluster manager nodes forming a stable quorum, and a separate, independently sized pool of data and ingest nodes. Both pools have pod disruption budgets, so a drain can take at most one data node at a time and can never break manager quorum, and both spread across hosts rather than trusting the scheduler to do it by accident.

A few sizing decisions are worth stating because they are easy to get backwards. Memory requests equal memory limits, so the pods get guaranteed quality of service. CPU is requested but deliberately not limited, because throttling a JVM produces pauses that a cluster reads as a failing node, and the cure is worse than the disease. Heap is set to roughly half of a node’s memory and never above 31 gigabytes, past which the JVM gives up compressed object pointers and you get less usable heap out of more RAM.

Fluent Bit now buffers to the node filesystem instead of memory, so an OpenSearch restart costs you nothing rather than dropping whatever was in flight. And every log record is stamped with the name of the cluster it came from, which means several clusters can ship into one OpenSearch and stay distinguishable without inventing a different index prefix for each of them.

Dashboards is published through an ordinary Ingress with a cert-manager certificate, so it works with Traefik or anything else.

The part that generalises

Nineteen values that used to be literals are now inputs, each with a description written for somebody who has never seen the stack. Not “storage class”, but which one, why it has to support single-node read-write attach, and why you want it on SSD. Not “shards”, but the fact that you are aiming for shards in the ten to fifty gigabyte range and that more shards than data nodes just adds overhead.

That documentation only got written because publishing forced it. It is the same effect as the audit: the discipline of making a thing reusable is what surfaces both the defects and the knowledge, and neither of them shows up while the stack is merely running.

Which is the honest lesson from a week that was supposed to be an afternoon. A stack that works is not the same as a stack that is correct, and you cannot tell the difference by looking at whether it is up. Every mistake we found was invisible from the outside, because each one was being compensated for by something more permissive than it should have been. The compensations are what you are looking for.

If you have a stateful workload that has been quietly fine for a year or two, the cheapest way to find out what is actually holding it together is to try to hand it to somebody else. Make it launchable on a cluster that is not the one it grew up on. The parts that were only ever true of the original cluster will announce themselves, and so will the parts that were never true at all.

We wrote up how we think about Kubernetes logging choices a couple of months ago, and everything there still stands. This is the other half of it: picking the right log store is a decision you make once, and running it correctly is a decision you keep making, mostly without noticing.

Stack profiles are available on every Ankra organisation.

Get started: Create a free account on Ankra.

Join our community: Slack

Follow us on: LinkedIn | GitHub

Contact us: [email protected]