A node that looks alive but isn't is worse than a clean crash. The API answers, the process runs, the node joins the cluster and reports a normal state — every surface check comes back green while the node sits frozen and falls behind the network.

I hit this during a routine validator upgrade. The cause was a single closed firewall port the new version had silently started depending on. The fix was one command; finding it took two hours, because the symptom pointed everywhere except the real problem.

The pattern

A version upgrade can silently introduce a new network-surface requirement — a new port or peer-communication channel — and if your firewall doesn't know about it, the node fails in a way that looks like a sync or version problem rather than a connectivity problem.

The node binds the new port internally, so the software thinks it's configured. Peers can't reach that port externally, so consensus quietly breaks. You see a node that's "almost working" and reach for the wrong tools.

The setup

A routine point-release upgrade: stop the service, swap binaries, restart, rejoin. The node had run for days fully synced beforehand. After the upgrade it came up, connected, and entered an active-sounding state. By every check I ran first, it looked successful. It wasn't.

The misleading signature

Four symptoms, each individually pointing at a different benign explanation:

• It reached an active state — progressed through startup and landed in a normal "observing" peer state, not an error state.
• Its ledger position advanced, then froze — the ordinal climbed from zero to a few hundred and stopped dead, while the network's reference node kept advancing into the thousands. A stall, not a crash.
• The API stayed responsive — every status endpoint answered, balance queries returned, the process never died.
• No state-transition errors in the normal logs — peer handshakes showed as completing, and the standard log output showed no errors at all.

Together those scream "sync issue" or "new version counts the ledger differently." They do not scream "a firewall port is closed." That's the trap: a connectivity failure wearing the costume of a sync problem.

What didn't work

Because the symptom looked like a sync problem, I burned two hours on sync-problem tools:

• Fresh restarts with full data wipes — synced partially, froze in the same place, every cycle destructive and useless.
• Manual cluster-join requests — accepted the join, still stalled.
• Version verification — the reference node ran the exact same version, so it wasn't a mismatch, which only removed a suspect and deepened the confusion.

Every move was reasonable for the symptom. Every one was wrong, because the symptom was lying about the cause.

The diagnostic that isolated it

The breakthrough was filtering the logs for errors instead of reading the normal output:

grep -i "transition\|error\|fail" <logfile> | tail -20

The reassuring handshake-success lines had buried two things: repeated gossip-round failures with Connection refused against a peer on a port one higher than the ones I had open, and recurring ClosedChannelException. The errors named a port that wasn't in my mental model of the node's networking.

That made the diagnostic a two-command comparison — and this should be the first thing you run when a node looks half-alive:

ss -tlnp | grep 900     # what the node is listening on
ufw status              # what the firewall allows

The node was listening on the new port internally. The firewall had allow-rules for the old ports and not the new one. There's the contradiction in two commands: listening internally, dropped externally. Peers try to open the new channel, the firewall refuses, the gossip round fails, and the node can't make the consensus progress that advances its ordinal — so it sits in observing forever, looking alive, doing nothing.

Root cause

The new version added a peer-communication channel on a new port (a libp2p gossip channel on the port just above the existing peer port). The previous version never used it, so it had never needed a firewall rule. The upgrade added the requirement without adding the rule, and nothing in the normal startup output flagged it — from the node's own perspective it had bound the port and was listening. Only the peers, on the outside, knew the door was locked. That's also why version-matching gave a false all-clear: it genuinely wasn't a version mismatch. The mismatch was between the new version's network requirements and the host firewall, which the upgrade never touched.

The fix and the before/after

One command opened the port, then a clean restart:

sudo ufw allow 9003/tcp
sudo systemctl stop <node-service>
rm -rf <data-dir> <log-dir>
sudo systemctl start <node-service>

  Node ordinal Network reference State Before (2+ hrs) frozen ~720 1,200+ observing, stalled +1 min after fix ~87, climbing — syncing +90 sec after fix ~218, climbing — syncing toward reference

Two hours stuck versus ninety seconds to obvious recovery — one firewall rule.

What another operator would get wrong

The core lesson: a node that looks partially healthy after an upgrade is more likely a connectivity problem than a sync problem — rule out the cheap connectivity check before any destructive recovery.

The specific mistakes this invites:

1. Trusting "the API responds" as proof of health. That only proves the process is up, not that peer communication is reaching peers. Liveness of the management surface isn't liveness of the consensus surface.
2. Reading "observing / no errors" as a sync issue and resyncing. Wiping and resyncing is purely destructive here — you rebuild state that was never the problem and freeze at the same point.
3. Stopping at version-matching. An upgrade can change the host's network requirements with no version mismatch between nodes.
4. Reading only the normal log stream. The error-filtered view is where the truth was.

The runbook check

Before destructive recovery on a post-upgrade node that looks half-alive:

ss -tlnp | grep <port-range>     # listening
ufw status                        # allowed

If the node is listening on a port the firewall doesn't permit, you've found it in thirty seconds instead of two hours. This generalizes to any validator upgrade where the software's network-surface requirements change: treat every upgrade as a potential change to the node's required ports, check release notes for new ones, and diff listening ports against firewall rules as a standard post-upgrade step. The new requirement is invisible until a peer tries to use it and gets refused — so the partial-health failure lets you believe the node is fine while it costs you uptime and hours of misdirected effort. Knowing the signature — listening internally, blocked externally, alive on every surface check, frozen where it counts — turns that into a one-line fix.