This is a public diagnostic reference for Post Fiat-style validator operators. It complements an earlier piece — the Validator Failure Response Manifest — by handling the step that comes before response: naming the failure. Every entry is derived from a real validator incident encountered during sustained operation. No invented categories, no postmortem prose, no hypothetical failures.

Purpose

Acting on a misidentified failure compounds damage. Restarting a node mid-sync regresses progress. Refreshing credentials on a version mismatch wastes the retry budget. Repairing config on a memory crash leaves the cause untouched.

This matrix makes the first step deterministic: identify the failure, then respond. The Response Manifest defined what to do for each class. This document defines how to recognize each class from observable evidence alone.

Severity Scale

Three levels. Severity drives the urgency of human attention, not the recovery action.

• warning — Node is operational but degraded; recovery may resolve without operator intervention.
• degraded — Node is partially functional; rewards or consensus participation impaired. Operator review expected.
• critical — Node is non-functional or actively damaging itself; immediate operator action required.

Failure Classification Matrix

Every entry is one real failure mode encountered in operation. Class IDs match the Response Manifest so the two documents combine into a single deterministic chain.

F-RES-01 — Resource Exhaustion (heap / memory)

Severity: critical

Observable Symptoms: Process exits unexpectedly; logs end with a memory exhaustion error pattern (e.g., OutOfMemoryError on JVM-based implementations); service auto-restarts but does not auto-rejoin cluster.

Likely Root Cause: Heap or memory allocation insufficient for sustained consensus workload.

Recovery Path: Increase memory allocation in unit file; restart; manually rejoin cluster. (See Manifest §F-RES-01.)

Prevention Rule: Size memory to peak observed working-set + 50% headroom. Verify after every version upgrade.

Proof / Evidence Signal: Memory exhaustion error string in service logs; process restart timestamp ≤ 5s after crash; node returns to pre-join state rather than Ready.

F-LOOP-01 — Crash Loop

Severity: critical

Observable Symptoms: Process exits and restarts ≥ 3 times within 10 minutes; service status shows rapid Active: activating → Active: failed cycles; no successful state transition between restarts.

Likely Root Cause: Underlying failure (config, version, missing dependency) prevents process from reaching steady state.

Recovery Path: Halt automated recovery; operator inspection required. (See Manifest §F-LOOP-01.)

Prevention Rule: Disable auto-restart only after exhausting other classifications. Crash loops mask the real failure.

Proof / Evidence Signal: Systemd Active: failed timestamps showing ≥ 3 restarts within 600 seconds; identical exit signature across restarts.

F-CONF-01 — Configuration Error

Severity: critical

Observable Symptoms: Process refuses to start, or starts and immediately exits with config parse error; common pattern after manual edits to unit files or config files.

Likely Root Cause: Malformed unit file (missing ExecStart, broken line continuation, unexpanded wildcards), invalid YAML/JSON, or required field missing.

Recovery Path: Halt automated recovery; operator must repair config. (See Manifest §F-CONF-01.)

Prevention Rule: Always reload init-system state after unit file changes. Validate YAML/JSON before save. Never edit live unit files with stream editors that may strip lines.

Proof / Evidence Signal: Init-system error indicating a missing or stripped command directive (e.g., systemd "Current command vanished from the unit file"), or config parser fatal in first 3 log lines after start.

F-VERS-01 — Version Incompatibility

Severity: critical

Observable Symptoms: Node reaches Ready locally but is excluded from consensus; peer count appears normal; logs show peer rejection or version negotiation failure post-handshake.

Likely Root Cause: Local jar/binary version differs from network-accepted version; cluster silently rejects participation.

Recovery Path: Halt automated recovery; perform manual version migration. (See Manifest §F-VERS-01.)

Prevention Rule: Subscribe to release announcements. Validate jar version before and after every upgrade. Never skip-update across two release windows.

Proof / Evidence Signal: Local node state == Ready AND consensus participation absent (no validated proposals attributed to node ID over ≥ 2 windows).

F-UPGRADE-01 — Incomplete Upgrade Artifact

Severity: critical

Observable Symptoms: Node fails to start or starts in degraded mode immediately after an upgrade procedure; symptoms resemble F-CONF-01 or F-VERS-01 but are caused by a missing artifact rather than a misconfiguration.

Likely Root Cause: Upgrade procedure copied a partial jar set (e.g., main jar replaced but a required secondary jar omitted); systemd starts the process but the process cannot resolve all required components.

Recovery Path: Inspect deployment directory; replace missing artifact from release; restart; rejoin.

Prevention Rule: Treat upgrade procedures as manifests, not commands. Verify file count and checksums against release before restarting service.

Proof / Evidence Signal: ClassNotFoundException, Could not load, or analogous "missing dependency" log line within 30s of start; deployment directory file count differs from release manifest.

F-AUTH-01 — Cluster Authorization Failure

Severity: degraded

Observable Symptoms: Node attempts to rejoin cluster and fails with HTTP 401 Unauthorized or session-not-found error; auto-rejoin script may report success but node remains rejected.

Likely Root Cause: Stale session credentials, expired bootstrap node identifiers, or auto-rejoin using outdated peer ID after upstream peer rotated keys.

Recovery Path: Refresh credentials / fetch current bootstrap IDs; restart; rejoin. (See Manifest §F-AUTH-01.)

Prevention Rule: Source bootstrap IDs from a live endpoint, not a hardcoded constant. Re-fetch on every rejoin attempt rather than reusing the prior session's IDs.

Proof / Evidence Signal: HTTP 401 or session-not-found response body on rejoin; rejoin call returns success but node state remains in pre-join.

F-BOOT-01 — Bootstrap Dependency Failure

Severity: degraded

Defining Axis: Bootstrap is unreachable.

Observable Symptoms: Node reaches pre-join state and stalls; bootstrap peer connection refused, timeout, or DNS failure; condition persists across restarts.

Likely Root Cause: Configured bootstrap peer (often the genesis or a primary validator) is offline or unreachable.

Recovery Path: Switch to next bootstrap peer in fallback list; restart. (See Manifest §F-BOOT-01.)

Prevention Rule: Maintain a fallback list ordered by observed reliability, not by topological position. Re-rank fallback list quarterly.

Proof / Evidence Signal: Connection attempts to bootstrap endpoint return refused/timeout while local node has confirmed external connectivity (e.g., other endpoints reachable).

F-JOIN-01 — Stalled Pre-Join

Severity: degraded

Defining Axis: Bootstrap is reachable but the node still does not advance.

Observable Symptoms: Node has been in pre-join state for > 30 minutes with no error pattern, no crash loop, and bootstrap confirmed reachable.

Likely Root Cause: Cluster has not yet accepted the node despite no observable error; commonly transient but can persist when paired with a silent root cause.

Recovery Path: Restart node; allow 15 minutes for state to advance. (See Manifest §F-JOIN-01.)

Prevention Rule: Enforce a maximum dwell time before triggering investigation rather than waiting indefinitely.

Proof / Evidence Signal: Node state unchanged across ≥ 6 polling windows of 5 minutes each, with no error pattern and stable restart behavior, while bootstrap peer responds to a direct connectivity check.

F-PEER-01 — Peer Isolation

Severity: degraded

Observable Symptoms: Node is Ready but connected peer count is zero or far below quorum; node otherwise stable.

Likely Root Cause: Network partition, firewall change, or local egress restriction; not a node-internal failure.

Recovery Path: No restart. Wait one cooldown window; re-evaluate. (See Manifest §F-PEER-01.)

Prevention Rule: Validate egress rules after every infrastructure change. Confirm peer endpoints reachable from the node before assuming local fault.

Proof / Evidence Signal: Connected peer count = 0 or < quorum threshold for ≥ 2 polling windows while process is up and Ready.

F-SYNC-01 — Post-Join Sync Stall

Severity: degraded

Observable Symptoms: Node is Ready with a healthy peer count, but local ordinal/height does not advance across multiple polling windows.

Likely Root Cause: Snapshot reconciliation hung, fork detection mid-resolve, or unresponsive peer monopolizing sync.

Recovery Path: Restart node; allow 15 minutes for sync to resume advancing. (See Manifest §F-SYNC-01.)

Prevention Rule: Distinguish RedownloadInProgress (normal) from sync stall (abnormal). Do not alert on the former.

Proof / Evidence Signal: Local ordinal unchanged across ≥ 3 consecutive polling windows; peer count healthy; no error pattern.

F-NOISE-01 — False-Positive Alert (not a failure)

Severity: warning

Observable Symptoms: Monitoring alerts fire during normal RedownloadInProgress cycling, transient peer disconnects, or expected state transitions.

Likely Root Cause: Alert thresholds set tighter than the protocol's normal operating envelope.

Recovery Path: No node action. Recalibrate monitoring thresholds.

Prevention Rule: Baseline normal-operation signal ranges before setting alert thresholds. Treat any single-window blip as noise unless paired with a state regression.

Proof / Evidence Signal: Alert fires but node state, peer status, and sync status all return to baseline within one polling window without intervention.

F-UNKNOWN — Unmatched Pattern

A signal pattern that does not match any class above resolves to F-UNKNOWN. Per the Response Manifest, F-UNKNOWN is itself an escalation trigger: an unmatched pattern indicates either incomplete vocabulary or a novel failure mode, and is treated as critical until reclassified.

Quick Identification Flow

A short decision tree for first-pass classification. Inspect in order; the first definitive answer narrows the matrix.

1. Is the process running? If Active: failed or cycling between activating and failed → check restart count in the last 10 min. If ≥ 3 → F-LOOP-01. If exit log shows memory exhaustion → F-RES-01. If exit log shows config parse error → F-CONF-01. If exit log shows missing class or dependency → F-UPGRADE-01.

2. Is the node in a transitional state for too long? If the rejoin response returns 401 or session-not-found → F-AUTH-01. If bootstrap endpoint is unreachable → F-BOOT-01. If bootstrap is reachable and dwell exceeds 30 minutes with no error pattern → F-JOIN-01.

3. Is the node Ready but excluded from consensus? Check whether validated proposals are attributed to the node ID. If absent over ≥ 2 windows → F-VERS-01.

4. Is the node Ready with peers but ordinal not advancing? If peer count is zero or below quorum → F-PEER-01. If peer count healthy and ordinal still stalled → F-SYNC-01.

5. Did an alert fire but everything looks normal? If node state, peer status, and sync status all return to baseline within one polling window → F-NOISE-01. Recalibrate, do not act on the node.

6. No row matches. → F-UNKNOWN. Escalate. Do not attempt automated recovery.

Implementation Notes

The matrix is designed to be wired into operator tooling rather than read from a screen during an incident. Four integration paths:

Monitoring alerts. Each entry's Proof / Evidence Signal is a literal alert condition. A monitoring system can fire pre-classified alerts (e.g., "F-AUTH-01 detected on validator-A") rather than generic "node down" alerts. Alert fatigue drops because every alert names the failure.

Automation rules. Classes flagged critical with retry budget 0 in the Response Manifest (F-LOOP-01, F-CONF-01, F-VERS-01) are never auto-recovered. Remaining classes auto-handle per the Manifest's retry/cooldown rules. The Matrix supplies the correct class label before the automation invokes recovery.

AI-assisted operator review. The matrix runs entirely on public observables — process state, log strings, peer counts, ordinal progression. No keys, wallets, or peer identity files required. An AI agent supplied with a node's current state and recent logs can apply this matrix to produce a class label.

Runbook integration. The Recovery Path entry references the Manifest by class ID. Classify here, act there. Deterministic chain from observation to resolution.

Closing

Identification before response. The Manifest tells operators what to do; this matrix tells them what they're looking at. Together: signal → classification → action → escalation.

Every entry corresponds to a real incident. Speculative failure modes and theoretical edge cases were excluded. A diagnostic reference is only as useful as the ground truth it's built from.