The Validator Oracle (TVO) is a paid Post Fiat agent that answers questions about validators using historical data it collects itself. A snapshot collector pulls each round's validator data files from the explorer on a schedule and stores them in a local database. The agent then answers questions against that stored history.

During Round 7, the collector stopped capturing new data and the agent kept answering from the previous round — no errors, no alerts, process fully up. This memo covers what happened, why it was invisible, and the two-layer fix. It's a failure mode any data-backed agent inherits: alive and stale at the same time, with uptime monitoring blind to it.

Incident timeline

• The collector had been capturing each round's files reliably for weeks, requesting them from the explorer at a set of fixed paths.
• The explorer reorganized where it published per-round output files — the same data, moved to new paths.
• Round 7 went live under the new paths. The collector kept requesting the old paths and got 404 on every attempt.
• The agent continued answering questions using the last round it had successfully captured (Round 6). Every answer looked normal. No errors surfaced.
• The liveness monitor stayed green the entire time — the process was up, the work loop was running, the heartbeat was fresh.
• The failure was found only by chance: a manual spot-check of one answer against the live leaderboard showed the agent's data was a round behind.
• The fix was built and deployed the same session.

Root cause

The explorer changed where it published the files. The data TVO depends on moved to new paths, and the collector had the old paths hardcoded with no fallback. That alone would have been a simple bug. What made it dangerous was the second half: nothing in the system was watching whether the collected data was actually current.

Why the failure was invisible

A 404 from the explorer is not, by itself, an error condition. Rounds run weekly. For most of any given week there is no new round to capture, so the collector legitimately requests data and gets nothing back. A miss looks exactly like the normal waiting state between rounds.

So when the paths changed and every request returned 404, the collector recorded misses — exactly what it records during a normal quiet week. There was no anomaly to detect. It polled, got nothing, waited, and never noticed that "nothing" had stopped being temporary.

The agent, meanwhile, kept doing its job. Asked a question, it pulled the most recent round it had and answered — confident, well-formed, and wrong by exactly one round. To a user, and to every monitor watching the process, everything looked healthy.

The distinction the incident forces is between two different properties that are easy to conflate:

• Liveness — is the process running and doing work? Yes. The whole time.
• Freshness — is the data the process is working from actually current? No. And nothing was measuring it.

A liveness monitor watches the wrong signal for this failure class. It confirms the agent is alive. It cannot confirm the agent is right.

The fix, layer one: contract-aware fetching

The immediate break was the hardcoded paths. The fix is to try the new paths first and fall back to the old ones, per file, so the collector survives this contract change and the next one:

const fileKinds = [
  { kind: 'scores',   paths: ['outputs/validator_scores.json', 'scores.json'] },
  { kind: 'snapshot', paths: ['inputs/validator_evidence.json', 'snapshot.json'] },
  { kind: 'unl',      paths: ['outputs/selected_unl.json', 'unl.json'] }
];
// try each path in order; first 200 wins, else record a miss

When an upstream contract changes, the source's own published structure is the truth for the new paths, not guesswork. A fetcher that hardcodes one path is one reorganization away from silent failure.

But fallback only fixes the paths you already know about. It does nothing for the next silent break, which needs a separate signal: freshness.

The fix, layer two: a freshness monitor

The real gap was that nothing measured how old the newest captured data was. So the second layer is a separate monitor that does exactly that, and nothing else. It reads the timestamp of the most recent successful capture and alerts if that timestamp is older than it should be:

const STALE_MS = 8 * 24 * 60 * 60 * 1000; // 8 days (rounds run weekly)
let alerted = false;

function check() {
  const row = db.prepare('SELECT MAX(fetched_at) as latest FROM round_files').get();
  if (!row || !row.latest) return;
  const age = Date.now() - new Date(row.latest).getTime();
  if (age > STALE_MS && !alerted) {
    const days = Math.round(age / 86400000);
    notify(`TVO DATA STALE: no new round captured in ~${days} days. Source paths may have changed.`);
    alerted = true;
  } else if (age <= STALE_MS && alerted) {
    notify(`TVO data fresh again: new round captured.`);
    alerted = false;
  }
}

setInterval(check, 6 * 60 * 60 * 1000); // check every 6 hours
check();

It runs as its own service, independent of the agent and the liveness monitor, and edge-triggers — one alert when data goes stale, one when it recovers, no repeat spam.

The monitoring rule

The rule the incident produces is short: monitor the age of the newest successful source update, not process uptime and not polling activity.

Those last two are the traps. Uptime tells you the process is alive. Polling activity tells you it's trying. Neither tells you it's getting fresh data, and this failure lived precisely in the gap between trying and succeeding. The only signal that would have caught it is the one nothing was watching: how long since the last confirmed-good capture, measured against the known cadence of the source.

What this still does not catch

The 8-day window is a slow backstop. Because rounds are weekly, the threshold has to sit above a normal week-long gap to avoid false alarms, so a real stale-data problem can run for up to a week before it alerts. That is acceptable only because the cadence is weekly — it is not a model for a fast-moving source.

It also only knows whether something was captured recently, not whether what was captured is correct. A malformed but successfully-fetched payload would not trip it.

And the agent itself does not yet fail closed. When freshness can't be confirmed, the right behavior is to caveat or refuse rather than answer confidently from old data. That is the next step this incident points to, not something the current fix implements: the monitor tells the operator the data is stale; it does not stop the agent from serving it.

The takeaway

If your agent depends on an external data source, monitor the freshness of that data as a separate signal from process health. A 404 can look identical to a normal idle period, so uptime won't catch a source that has moved underneath you. Watch the age of the data, not the health of the process — the process will look fine right up until the moment it's confidently wrong.