Most builders coming to Post Fiat to build an agent assume the same thing I did: the explorer is the data layer. You see validators, scores, agreement percentages — so when you go to build an agent, you reach for the explorer. Or the RPC. Or some API you assume does what the explorer does.
None of those work the way you'd expect. The better architecture is sitting in plain sight, but only if you know to look for it.
This memo documents the validator-data backend gap and the architectural pivot that bridges it.
The setup
I'm building The Validator Oracle (TVO) — a Post Fiat Task Node agent that answers questions about validators the explorer can't. The explorer answers what (here's a validator's current score). TVO needs to answer why, compared to what, trending how, what's notable. A user types /vision and asks "why is the entire top 5 all under one parent domain?" — the agent needs to pull data across multiple validators, identify the pattern, and synthesize an answer.
For that to work, the agent needs three things:
- Programmatic access to per-validator state (scores, agreement, software version)
- Historical data to answer trend questions
- Cross-validator queryability to compute rankings and detect patterns
None of these are exposed by the obvious paths. Here's what happens when you reach for them.
The Forbidden door — direct RPC
The natural assumption is to ask the validator RPCs directly. Standard XRPL-style commands like validators and validator_list_sites are well-documented in the broader XRPL ecosystem. Run them, get the data, done.
{
"error" : "noPermission",
"request" : { "command" : "validators" },
"status" : "error"
}
Forbidden. On the public PFTL RPC, admin commands are blocked for anonymous operators. The official path to validator data is closed, and the docs don't tell you where it went instead.
The SPA dead end
Next you try the explorer's API. Surely if the explorer renders validator data, it's calling something:
<!DOCTYPE html>
<html lang="en">
<head>...
HTML. The explorer is a single-page application; the obvious validator endpoints return the page shell, not the data. To find the real path you have to open browser dev tools, watch the Network tab, refresh the page, and reverse-engineer which XHR loads what. Undocumented operator work — either you know to do this, or you don't have validator data.
The pivot — mirror what's already there
Here is where the architecture changes.
What you find when you reverse-engineer the explorer is that all the data you need already exists in a clean, machine-readable form. It just isn't where the docs point, and it isn't accessible the way you expect.
The data path is round-indexed JSON files, one per round, four file kinds per round:
/api/scoring/rounds/N/scores.json
/api/scoring/rounds/N/snapshot.json
/api/scoring/rounds/N/scoring_config.json
/api/scoring/rounds/N/unl.json
Where N is the round number. Together, the four files contain everything the explorer's UI displays: per-validator score breakdowns, network-level analysis, the raw validator data feeding the scorer (IP, ASN, country, domain, agreement windows, server version), the scoring algorithm config, and the trusted validator list.
Once you know the pattern, every round of validator data is one HTTP request away. No RPC permission needed. No client library required. No undocumented socket protocol.
This is the highest-leverage insight in the entire build, and I want to be precise about why it matters:
The right architecture for any agent that needs validator data is to mirror the explorer's round-indexed data into a local store — not to build collection from scratch.
Building collection from scratch means writing a system that connects to validator RPCs, polls them on some schedule, normalizes the responses, handles backoff and retries, and accumulates state. That is weeks of work for a system that will, on its best day, reproduce what the explorer already produces.
Mirroring the explorer means writing a script that fetches four JSON files per round and writes them to local SQLite. That is an evening of work for a system that captures every byte the explorer's analysis layer sees. The mirror approach also sidesteps the question of whether your independently-collected data is accurate against the network's own ground truth — by definition, your data is what the explorer sees.
Why mirroring is mandatory, not optional
Here's the catch that forces the issue: the explorer doesn't retain full history. It exposes a rolling window of recent rounds. When I probed at the time of build:
- Round 1: 404
- Round 2: 404
- Round 3: 200 (oldest retained)
- Round 4: 200
- Round 5: 200 (current)
- Round 6: 404 (not yet started)
Rounds 1 and 2 were already gone. Permanently. The explorer doesn't archive them, and no public Post Fiat service exposes them either. As future rounds come online, round 3 will age out, then round 4, and so on.
Any agent that needs historical context — "did this validator's score improve over the last month?", "who joined the network recently?", "is the network getting more centralized?" — needs that history before it disappears. Mirroring isn't a performance optimization. It's a hedge against the explorer's purge policy. Every day you delay starting the mirror is data your agent will permanently lack.
This is the structural argument for the mirror approach. Even if the explorer were a perfect query backend (which, as we'll see, it isn't), the bounded window alone would force local mirroring.
What you find once you can actually read the data
Open the files and you'll hit a series of schema realities the docs don't surface. Not bugs — undocumented design choices an agent backend has to defend against.
scores.json mixes network-level analysis with per-validator data in one file. The top level has two keys: network_report (network-wide commentary across five scoring categories) and validator_scores (a 39-entry list of per-validator score breakdowns). A naive parser written to expect a uniform list crashes on the first object, because the first object isn't a validator entry. Operators have to know to skip past the summary section to reach the actual scores.
Snapshot encodes three time windows per validator in one file. Each validator's record contains:
{
"agreement_1h": { "score": 1, "total": 1190, "missed": 0 },
"agreement_24h": { "score": 1, "total": 28600, "missed": 0 },
"agreement_30d": { "score": 0.9606, "total": 590828, "missed": 23281 }
}
Recent-vs-medium-term behavior is queryable from a single snapshot — you don't need historical snapshots to answer "is this validator currently performing better than its 30-day average?" The obvious assumption — that different time windows require different snapshots in time — is wrong. If you don't notice this, you'll over-engineer your historical layer to compute things that already exist in the current snapshot.
Validators can have completely null identity fields. Here's a real example from the current round:
{
"ip": null,
"asn": null,
"domain": null,
"geolocation": null,
"identity": null,
"master_key": "n949AEXAmiaZxcch9r8qgAZiPd2xFAGaR75D8yfSTf7mF3aWk12a",
"signing_key": "n949AEXAmiaZxcch9r8qgAZiPd2xFAGaR75D8yfSTf7mF3aWk12a",
"server_version": "1.0.4",
"agreement_30d": { "score": 0.00515, "total": 184785, "missed": 183834 }
}
Nine of 39 validators in the current round are anonymous this way — no IP, no ASN, no domain. They still get scored. They still run software. They still appear in rankings. But the explorer UI likely filters them out of the human-facing list, because they have nothing to display for the columns it renders.
For an agent backend, this means master_key is the only field guaranteed non-null across all validators, and it is the canonical primary key. Lookup-by-domain is convenient for human users but cannot be the system's primary access path. A user asking about an anonymous validator has to be able to pass its master_key directly.
The domain field is non-unique. Multiple validators can run under the same parent domain. One operator runs five validators under a single postfiat.org domain in the current round. A naive "find by domain" function that returns the first match silently hides the other four. Agent code that doesn't return arrays will lie to users.
There are undocumented fields that are always null. snapshot.json has a top-level snapshot_ledger_index field. In every snapshot I've collected, it's null. The field is in the schema but unused. Defensive code has to expect that, but you can only know to expect it by hitting it.
The mirror collector, in code
Here is the core of the mirror collector running in my live build. Runs every 30 minutes via systemd timer, deduplicates on payload hash so unchanged data isn't re-inserted, and accumulates a forever-retained local SQLite.
import Database from 'better-sqlite3';
import { createHash } from 'crypto';
const db = new Database('/var/lib/tvo/snapshots.db');
const FILE_KINDS = ['scores', 'scoring_config', 'snapshot', 'unl'];
const EXPLORER_BASE = 'https://<pft-explorer-host>/api/scoring/rounds';
async function collectRound(roundNumber) {
for (const kind of FILE_KINDS) {
const url = `${EXPLORER_BASE}/${roundNumber}/${kind}.json`;
const res = await fetch(url);
if (!res.ok) continue; // round/file not available
const text = await res.text();
const hash = createHash('sha256').update(text).digest('hex');
// dedup: only insert if this exact payload hasn't been seen for this round/kind
const existing = db.prepare(
`SELECT 1 FROM round_files
WHERE round_number=? AND file_kind=? AND payload_hash=? LIMIT 1`
).get(roundNumber, kind, hash);
if (existing) continue;
db.prepare(
`INSERT INTO round_files
(round_number, file_kind, fetched_at, payload, payload_hash)
VALUES (?, ?, ?, ?, ?)`
).run(roundNumber, kind, new Date().toISOString(), text, hash);
}
}
// pull current round and the two prior, so we cleanly bracket transitions
const current = await detectCurrentRound();
for (const r of [current - 2, current - 1, current]) {
await collectRound(r);
}
That's the whole architectural insight in roughly thirty lines. The work isn't writing the code — it's knowing this is the right code to write, which you only know after the Forbidden RPC, the SPA dead end, and the bounded-window discovery.
What another agent builder would get wrong
If you don't hit this gap before you start building, the failure modes are predictable:
- You try the RPC, get Forbidden, give up, and decide validator data is "not available." You build a less useful agent that can only answer questions the explorer's UI already answers.
- You discover the explorer endpoint and query it directly on every request. Your agent breaks the moment the rolling window purges a round it had been relying on. Your historical queries silently degrade as data ages out.
- You build collection from scratch against the validator RPCs. Weeks of work for an inferior version of what the explorer already produces. Your data drifts from the network's analysis layer because you're computing different things from different inputs.
- You write lookup-by-domain as your primary access path. Anonymous validators are invisible to your agent. The five validators under one parent domain appear as one. Your coverage is missing roughly a quarter of the network.
- You build a uniform parser for
scores.jsonexpecting a list of validator entries. It crashes on the first object, because the top of the file is the network-level analysis, not a validator.
Each of these is a real, specific way to lose a week — or worse, ship an agent that's quietly wrong and never notice.
The architectural pivot — the explorer already collects what you need; mirror it, don't rebuild it — is the entire game. Once you make that pivot, the rest of the data layer follows in an evening. Without it, you're building the wrong thing.
Closing
The Post Fiat explorer is a beautifully-rendered, human-readable, browser-targeted surface over an automated analysis pipeline. For human use, it's complete. For agent use, it's a UI on top of a backend you can't quite see — and the data path you actually need is sitting one folder over, undocumented, on a rolling window that's purging history while you read this.
Mirror it. Now.
