This is a public operator reference for people running Post Fiat-style validators and always-on agents on their own infrastructure. It catalogs failure modes I actually hit running a live validator and a paid agent — not hypotheticals — and for each one gives the tell, the root cause, and the response. The point is recognition: most operator mistakes come from misreading what you're looking at, so naming the failure fast is half the fix.

I run a Post Fiat testnet validator and, alongside it, a paid AI agent that answers validator questions. Between them, over months of live operation, I collected a specific set of failure modes — the ones that actually happened, several more than once. This is that catalog.

Each entry follows the same shape: what it looks like from the outside, what's actually causing it, and what to do. Identifiers, addresses, and hostnames are redacted throughout.

Failure 1 — Silent stale data behind a healthy process

The tell: Everything looks fine. The process is running, answering, producing well-formed output. But the data underneath is quietly out of date, and nothing is complaining.

Root cause: An upstream source I don't control reorganized its file paths. Every fetch started returning a 404 — which is indistinguishable from a normal quiet period with no new data. The collector polled, got nothing, recorded a miss, and waited, exactly as it does during a legitimately slow stretch. No error surfaced. The process was healthy the whole time; only the data had gone stale.

The response: Two layers. First, make the fetcher contract-aware — try the new path first, fall back to the old one per file, so it survives the reorganization. Second, and more important, monitor freshness as a signal separate from liveness. A liveness check confirms the process is alive; it cannot confirm the data is current. You need a monitor that asks "when did we last actually store something new?" and alerts when that answer gets too old.

The lesson: Liveness is not freshness. A process can be perfectly alive and completely wrong.

Failure 2 — The port your firewall didn't actually close

The tell: ufw is active, default-deny, and the admin port isn't in the allow list. By every check you'd normally run, the port is closed. It isn't.

Root cause: Docker publishes container ports by writing its own iptables rules below ufw. When a container publishes a port, that port can be reachable from the public internet even though ufw is denying by default and never lists it. The admin RPC port ended up exposed to the world, and ufw deny did nothing, because Docker never consults ufw for published ports.

The response: Restrict the port in Docker's own chain, DOCKER-USER, not in ufw. Drop external access to the admin port there while allowing localhost and the internal bridge network, and persist those rules across reboots with a systemd unit, since iptables rules don't survive a restart on their own.

The lesson: "The firewall is on" is not the same as "the port is closed." Verify from off the box, not from your assumptions.

Failure 3 — The sequence collision from acting before reading

The tell: A configuration action that should have been routine instead dropped consensus agreement, and recovery required bumping a sequence number to get back into a valid state.

Root cause: I ran a domain-attestation step twice without reading the docs on how the manifest sequence worked. The second run collided with the first — a sequence conflict that temporarily knocked the node out of agreement. It was self-inflicted, and entirely avoidable by reading the published behavior before acting.

The response: Recover to a clean sequence, then adopt a hard rule: when an action touches signed or sequenced state, read the upstream's own documented behavior first. Don't pattern-match from a similar system, and don't run a state-changing command twice to "make sure it took."

The lesson: For anything that touches signed or sequenced state, the docs are the source of truth, and guessing is a consensus risk.

Failure 4 — The orphaned processes eating the box

The tell: Memory usage climbs over time with no obvious cause until the box is under real pressure. Something is leaking, but the main process looks normal.

Root cause: A helper was being invoked in a way that spawned a short-lived wrapper process per call, and those wrappers were being orphaned instead of cleaned up. Enough of them accumulated to consume the majority of the machine's memory. The main agent was fine; the leak was in how it launched a subprocess.

The response: Stop spawning the wrapper. Invoke the underlying binary directly so there's no orphan to leak, and confirm the process count stays flat over time rather than climbing.

The lesson: A memory leak isn't always in your code's logic — sometimes it's in how your code launches other processes.

Failure 5 — The model's private reasoning leaking into a paid answer

The tell: Under a complex real-world query, a paid answer came back with the model's full internal monologue attached to it — the kind of output a paying user should never see.

Root cause: The first guard against this was a regex that tried to detect and retry when reasoning leaked. It failed, and worse, it failed by leaking a new phrase on the retry itself. Pattern-matching against messy model output is a losing game; there's always another phrasing.

The response: Replace the brittle regex retry with a deterministic cleanup pass — a separate, tool-less model call whose only job is to return the final answer cleaned of any reasoning. It doesn't try to detect the leak; it just reliably produces clean output regardless of what came in.

The lesson: When you're fighting the open-ended variety of model output, a deterministic transform beats a pattern-matching guard.

The thread running through all five

None of these announced themselves. The stale data looked like a quiet week. The open port looked closed. The sequence collision looked like a routine command. The memory leak looked like a healthy agent. The reasoning leak looked like a finished feature.

That's the real operator skill this catalog is about: the gap between what a failure looks like and what it is. A monitor that watches the wrong signal, a firewall check that trusts the wrong layer, a command run on the wrong assumption — the failure is usually one level removed from where you're looking. Naming it correctly is most of the work; the fix is often small once you do.

Written from live operation of a self-hosted Post Fiat validator and a paid agent. Specifics redacted; the patterns are the point.