I built a scoring system for my DeFi bot, then built a tool to grade it. The grade came back: my highest-scored signals were my least accurate ones.
This is Part 4 of the Arb Series, about Agent 4, the Coordinator. It’s the agent that takes everything Agents 1 through 3 report and decides what’s actually worth acting on. It’s also the story of the day my own scoreboard told me I’d been ranking the wrong signals on top.
What the Coordinator actually does
Agent 4 is the decision layer: it reads the yield signals from Agent 2, the arbitrage signals from Agent 3, and the whale signals from Agent 5, then gives every signal one of four verdicts.
Before a signal earns a verdict, it runs through risk filters. Agent 4 marks down APYs that are too high to be believable, data that looks like an error, profit that’s too thin after slippage, windows that close too fast to act on, and tokens whose liquidity is scattered across too many pools.
There’s also a manual blocklist for confirmed rug pulls and emission traps. A blocked token gets an ABORT before any scoring logic runs, which keeps the code simple and the logs quiet:
if token in BLOCKED_TOKENS:
return {”verdict”: “ABORT”, “score”: 0}
Two more behaviors make it feel like a coordinator and not just a filter. Agent 5 watches large wallets, and Agent 4 nudges a score up when whales are adding liquidity and cuts it when they’re leaving. And when the same token shows up in both the yield and arb signals at once, Agent 4 flags it as a combined play.
I’m deliberately not publishing the exact thresholds. That’s the part I tuned, and it stays on the Pi.
Problem one: 82% of my signals were “top tier”
When I got Agent 6 sending alerts to Discord, I finally saw the volume of PRIME signals in one place. Of roughly 530 logged signals, 437 were PRIME. That’s 82%.
A top tier that holds four out of five signals isn’t a top tier. PRIME had become a synonym for EXECUTE, and the alerts channel was going to be noise.
The fix was to stop relying on a single score and require several conditions to line up at once, like how stable the opportunity window is and how much profit is left after costs. On the first cycle after the change, PRIME dropped to about 7% of signals. That’s the point of a top tier: rare and worth a look.
The small mistake I’m glad I caught
Wiring up Discord alerts meant using a webhook URL, which is basically a password for posting into a channel. While testing, I pasted that URL straight into my terminal and it ended up exposed.
I regenerated it in Discord right away, so the old one is dead. Then I did what I should have done first: the URL now lives in a config file that’s in .gitignore, so it can never be committed to GitHub. The repo only carries a template file with a placeholder.
Lesson: treat any URL with a token in it like a password. Rotate first, then fix your setup.
Grading my own homework
By late June I had close to 1,000 signals sitting in a scorecard marked PENDING, with no idea whether any were any good. So I wrote scorecard_checker.py, a standalone script that revisits old signals and labels each one WIN, PARTIAL, or LOSS.
For yield signals, it checks whether the APY held up or collapsed after the signal. For arb signals, it checks whether the price gap closed (someone else took it, so it was real), stayed open, or turned out to be phantom data.
I ran it against about 925 signals. Overall accuracy came back around 66%, against a 33% chance baseline for three outcomes. Not bad. Then I split it by score:
The higher the score, the worse the accuracy. The same pattern held when I split by verdict: EXECUTE beat PRIME on both yield and arb. And yield signals overall were the weak spot, around 44%.
My read on why: the scoring rewarded extremity. A huge APY, a massive delta, a giant gap. But extreme conditions are outliers, and outliers revert. In yield especially, the biggest numbers usually come from token emissions that dry up fast. The modest signals that cleared the baseline checks without any “too good to be true” flags were the ones that actually held up.
What this scorecard can’t tell you
I don’t want to oversell my own findings, especially after Part 3.
First, WIN and LOSS here are proxies, not profit and loss. “The APY held up” and “the gap closed” are reasonable signs a signal was real, but they aren’t money in an account. Nothing in this series is a track record.
Second, the arb accuracy figure was measured using Agent 3’s own gap logic, the same logic with the bug I wrote about last time. If the measuring stick is bent, the arb numbers deserve extra suspicion until that root fix lands. I’d treat the arb result as a hint, and the yield result, which doesn’t depend on Agent 3’s gap logic, as the more trustworthy finding.
I’d rather publish a finding with its limits showing than a clean number that falls apart when someone checks it.
What I’m changing
The scoreboard gave me a to-do list:
-
Redesign PRIME to reward reliability, not just extremity.
-
Add a stability check for yield signals, so a signal counts for more when its APY has held up over time, not just when it’s high.
-
Run the scorecard checker on a daily schedule so the grades never go stale.
-
Let these results shape how the future execution engine prioritizes signals, with the modest, well-checked ones first and the extreme ones last.
The bigger lesson is one I keep relearning on this project: a score only means something once you’ve measured it against what actually happened. It took a scorecard to find out my scoring was wrong.
Next time: Agent 5, the whale watcher, and what happens when your bot starts following the biggest wallets in the room.
Whatever you’re building today, keep goin’! 🚀🔥 — Voytek.ai
If you enjoy following this build, consider supporting it: ko-fi.com/askvoytek
