One of my Pluralis Agora boxes went offline overnight — a steady RTX 5090 that had been earning for days. My Telegram monitor never fired. It had been running on every box for over a week and had alerted on nothing — not the death, not the prior transitions through sync into earning.

What follows is the diagnostic sequence and the rebuild. Full script at the bottom.

The First Look

The original bot was a small Bash script in a tmux window on each Vast.ai box. It tailed the agora_cli.py log file, read the last 5 lines every 30 seconds, and matched against three strings:

• "Sync complete" — supposed to mean the box was now earning
• "[SYNC]" — supposed to mean the box had caught a seat and started syncing
• "Maximum number of active nodes" — the capacity wall, meaning still looping

When any matched, it sent a Telegram message and updated a state file. That was the whole design.

I SSHed into my long-running earning box and ran a grep against the live log file for those strings. Empty. Zero matches across thousands of lines of training activity.

The Pluralis software prints those strings to the foreground terminal — not to the log file the bot reads. First lesson: screen output is not log content. If you build a monitor that reads a log file, you grep that file first. Documentation isn't ground truth; the live file is.

What the Log Actually Contains

Wider grep — sync, auth, error, training. The picture cleared up fast.

When the box is actively earning, the log writes lines like:

May 31 01:03:47.311 [INFO] Beginning optimizer step #22711
May 31 01:03:49.648 [INFO] Transitioning to epoch 22712

The "Beginning optimizer step" line is the real earning indicator — appears every 17 seconds or so during active training.

When the box is looping for a seat, the log writes a repeating pair every 20 seconds:

May 31 00:45:40.141 [INFO] Authorization started...
May 31 00:45:40.680 [ERROR] Authorization failed: Maximum number of active nodes reached, please try again later.. Exiting run.

When the box catches a seat, you see "Authorization started" with no Maximum failure after it — then a long sync phase before any "Beginning optimizer step" appears.

Fatal errors (the kind that kill a box) are distinct strings, each ending with "Exiting run.":

• "An unknown connection error occurred"
• "Failed to connect to CUDA"
• "Caught invalid value in gradients"
• "Too many failed all-reduce attempts"
• "RAM usage exceeded threshold"
• "Could not fetch global progress"
• "Caught CUDA error" (matched as a regex)
• "Averaging parameters failed with TimeoutError"
• "Averaging step failed: could not find a group"

That's the real surface area. The original bot's strings weren't on it because they don't exist in the log.

v2: The First Rebuild

I rewrote the bot using the real strings. Same overall design — tail the log, check for state-indicator strings, alert on transitions — with the improvements:

• Use "Beginning optimizer step" for earning detection instead of the nonexistent "Sync complete"
• Use "Authorization started" with no following Maximum for sync detection
• Watch a wider window — 50 lines back instead of 5 — so a fast burst of post-death output couldn't scroll past
• Scan the last 300 lines for any of the fatal error strings, with deduplication

I deployed v2 on the long-running earning box first. First check: bot correctly identified the state as "earning". No false alerts. Confidence built.

I deployed v2 on a second box that was actively looping for a seat. Within minutes, two Telegram alerts came through. The first said the box had caught a seat and was syncing. The second, sixty seconds later, said the box had dropped during sync and was back in the loop.

Neither was true. The box was still looping, unchanged.

That was an immediate, visible failure of the fix. Better visible than blind, but v2 had a different bug.

Why v2 Failed

The clue was the timing. The false "caught a seat" alert fired on one 60-second check, and the false "dropped" fired on the next check 60 seconds later. The bot saw different things on two adjacent checks while the box's actual state hadn't changed.

The root cause was in the design itself. The bot looked at the last 50 lines of the log to decide the current state. During looping, each retry cycle prints the "Maximum" line, then about 20 lines of JSON config dump from the next agora_cli.py launch, then the next "Maximum" line. Roughly one Maximum line every 22 lines of log content.

Sometimes the 50-line window contained two Maximum lines (detection works). Sometimes only one. And sometimes — the bug — the window straddled a boundary in a way that contained zero Maximum lines, even though the box was unambiguously still looping.

When that happened, the bot's logic ran: "no Maximum recently, no Beginning optimizer step recently, previous state was loop, so it must have transitioned to sync." False sync alert. A minute later when the next Maximum scrolled into the window, the bot saw it as a fresh transition. False drop alert.

This is the deeper lesson and it applies to any tail-window log monitor: events that recur at irregular intervals across log content of variable volume are not reliably detectable by looking at the last N lines. The window will sometimes catch them and sometimes not. Widening the window narrows the gap but doesn't close it.

v1 used 5 lines and 30 seconds. v2 used 50 lines and 60 seconds. Same failure mode in different sizes. The fix isn't to widen the window again — it's a different detection pattern.

v3: Count-Based Detection

The fix is simple once you see it. Instead of looking at recent log content, count the total number of state-indicator lines in the whole log file. Every 60 seconds, count again. Compare to the previous count.

• If the count of "Beginning optimizer step" went up since the last check, the box is earning right now.
• Else if the count of "Maximum number of active nodes" went up, the box is still looping.
• Else if neither went up and the previous state was loop or sync, the box must be syncing (transitioned out of loop, not yet earning).
• Else keep the previous state (handles brief log lulls during steady earning).

This removes the timing-window failure mode because growing counts are unambiguous. It doesn't matter how much filler text is between the events, how fast the log writes, or what cycle phase the bot catches. A count that went up means activity happened. A count that stayed the same means it didn't.

The fatal-error scan stays as a tail-window check, but the window is wide (300 lines) and each unique error is recorded so it doesn't repeat-alert.

I deployed v3 on the previously-misbehaving box first. Two consecutive 90-second verification cycles confirmed the state stayed "loop" as the Maximum-count climbed from 154 to 159. No false alerts. Then v3 went onto the earning box (first check correctly read "earning" with 2287 optimizer steps logged), and onto a fresh box that had been looping for about 20 hours (first check correctly read "loop", 1150 Maximum lines accumulated, 0 optimizer steps).

Three boxes, three correct detections, no spurious notifications.

The Reusable Lessons

Three things from this incident generalize to any production monitor, not just Agora.

A monitor that is running is not a monitor that is working. The v1 bot ran continuously for over a week. It produced no errors, no crashes, no signs of trouble. It was also alerting on nothing because the strings it watched for didn't exist in the log. Production-running and actually-firing are not the same property. Verifying a monitor means forcing a known state change and confirming the alert fires.

Tail-window detection is fragile when event spacing is uneven across log content of variable volume. Whether you watch 5 lines or 50 or 500, you have the same failure mode in different sizes. Counts and timestamps are more robust because they're not subject to window-position luck.

Screen output is not log content. The official documentation for the software you're monitoring will describe what it prints. Some of that prints to the terminal only and never reaches the log file. Always grep the actual log file before designing detection logic.

Also: an alert that fires too often becomes invisible. v3 stays silent during steady-state operation and only fires on real transitions or fatal errors. A monitor pinging your phone every hour stops being read within a week.

The Boundary

On-box monitors can only see what the on-box log shows them. They cannot detect a host that drops off the network entirely — when the box itself dies, the log stops being written, and the bot has nothing to read. For that failure mode, you need a separate off-box checker that pings the Pluralis dashboard or a heartbeat endpoint from somewhere outside the box. Different infrastructure for a different category of failure, worth building if you operate more than a single node.

The v3 Script

Drop this on any Agora node. Replace the placeholder values at the top with your own Telegram bot token, your chat ID, the location label you want in your alerts, and your log path if it differs. Save it as /root/agora_monitor.sh, make it executable, and start it with nohup in a tmux window. On first run it silently notes the current state without alerting, so deploying it on a steadily earning box won't spam you. From there it only fires on real state transitions and on the first occurrence of any fatal error.

#!/bin/bash
TOKEN="YOUR_TELEGRAM_BOT_TOKEN_HERE"
CHAT="YOUR_TELEGRAM_CHAT_ID_HERE"
LOG="/workspace/agora/logs/server_gpu0.log"
BOX="YOUR_BOX_LOCATION_LABEL_HERE"
STATE_FILE="/root/agora_monitor.state"
COUNTS_FILE="/root/agora_monitor.counts"
ERROR_FILE="/root/agora_monitor.last_error"
TG_PROTO="https"
TG_HOST="api.telegram.org"

send(){
  curl -s "${TG_PROTO}://${TG_HOST}/bot${TOKEN}/sendMessage" \
    -d chat_id="$CHAT" -d text="$1" >/dev/null
}

PREV_STATE=$(cat "$STATE_FILE" 2>/dev/null)
PREV_STATE=${PREV_STATE:-unknown}
PREV_MAX_COUNT=$(awk 'NR==1' "$COUNTS_FILE" 2>/dev/null)
PREV_OPT_COUNT=$(awk 'NR==2' "$COUNTS_FILE" 2>/dev/null)
PREV_MAX_COUNT=${PREV_MAX_COUNT:-0}
PREV_OPT_COUNT=${PREV_OPT_COUNT:-0}
LAST_ERROR=$(cat "$ERROR_FILE" 2>/dev/null)

while true; do
  CUR_MAX_COUNT=$(grep -c "Maximum number of active nodes" "$LOG" 2>/dev/null)
  CUR_OPT_COUNT=$(grep -c "Beginning optimizer step" "$LOG" 2>/dev/null)
  CUR_MAX_COUNT=${CUR_MAX_COUNT:-0}
  CUR_OPT_COUNT=${CUR_OPT_COUNT:-0}

  if [ "$CUR_OPT_COUNT" -gt "$PREV_OPT_COUNT" ]; then
    CUR_STATE="earning"
  elif [ "$CUR_MAX_COUNT" -gt "$PREV_MAX_COUNT" ]; then
    CUR_STATE="loop"
  elif [ "$PREV_STATE" = "loop" ] || [ "$PREV_STATE" = "sync" ]; then
    CUR_STATE="sync"
  else
    CUR_STATE="$PREV_STATE"
  fi

  WINDOW=$(tail -n 300 "$LOG" 2>/dev/null)
  FATAL=$(echo "$WINDOW" | grep " \[ERROR\] " | grep -E "An unknown connection error occurred|Failed to connect to CUDA|Caught invalid value in gradients|Too many failed all-reduce attempts|RAM usage exceeded threshold|Could not fetch global progress|Caught CUDA error|Averaging parameters failed|Averaging step failed" | tail -n 1)

  if [ -n "$FATAL" ] && [ "$FATAL" != "$LAST_ERROR" ]; then
    send "ERROR $BOX: $FATAL"
    echo "$FATAL" > "$ERROR_FILE"
    LAST_ERROR="$FATAL"
  fi

  if [ "$CUR_STATE" != "$PREV_STATE" ] && [ "$PREV_STATE" != "unknown" ]; then
    case "$PREV_STATE-$CUR_STATE" in
      loop-sync) send "$BOX: caught a seat - syncing now" ;;
      sync-earning) send "$BOX: sync complete - earning now" ;;
      earning-loop) send "$BOX: dropped from earning - back in the loop" ;;
      sync-loop) send "$BOX: dropped during sync - back in the loop" ;;
    esac
  fi

  echo "$CUR_STATE" > "$STATE_FILE"
  printf "%s\n%s\n" "$CUR_MAX_COUNT" "$CUR_OPT_COUNT" > "$COUNTS_FILE"
  PREV_STATE="$CUR_STATE"
  PREV_MAX_COUNT="$CUR_MAX_COUNT"
  PREV_OPT_COUNT="$CUR_OPT_COUNT"

  sleep 60
done

Three boxes running on v3. No false alerts, no missed events. v1's bug took a week of silent failure to expose. v2's took three minutes on a looping box. The lesson from both: deploy a monitor onto a state that exposes the bugs you don't know about yet, not just the one that confirms the design you already trust.