Well, when it rains, it pours! What a week, Vault Staking had the worst string of missed checkpoints in the 8 months I've been running this validator. Lots of delegators have reached out wondering why and if they should be concerned, so I want to write exactly what happened, my response to the issue, and way to improve.

First, a quick note on terminology. A "validator" is comprised of 2 servers, a sentry node and a validator node. The validator node mines blocks, handles transactions, and keeps state of the network. The sentry node, also called a full node, keeps state of the network, can take requests from dapps, users, and other sentries, and is responsible for being the intermediary between the validator node and the internet. The validator can only talk to an assigned sentry node; it is not able to talk to other sentries and is not able to receive requests directly from the internet. 

Issue 1:

What happened: The first issue of the week occurred on May 20, which was the first checkpoint I've missed since April 17! Luckily, since I have both email and discord notifications for any issues with my servers, I was immediately notified and was able to respond to start debugging within 15 minutes. By the time we resolved the issues, we missed just a few checkpoints. The only issue I could find was the validator was under extreme load and couldn't keep up with demand.

Solution: After a lot of trial and error, I increased my validator server's resources by double. Instead of the recommended 4 core and 16GB RAM server, my validator runs on an 8 core and 32 GB RAM server which seems to keep up with any burst demand.

Way to improve: I will be keeping the server at twice the recommended hardware specs to make sure we can keep up with demand. My delegators are great for sticking with me, and it's worth spending the commission the generously give me on beefier hardware.

Issue 2:

What happened: Issue 2 was just an unfortunate timing. In the middle of the night, my sentry node went down due to a "too many files open" exception in the Heimdall process. Since the sentry is between the Validator and the rest of the network, that means my validator couldn't perform its duty. The issue was easy to solve, but I missed more checkpoints than needed since I didn't catch the error early. 

Solution: I worked closely with the Polygon team and other validators to see what the issue was, and how to fix the "too many files open" error. Just to get the sentry running again, I restarted the server, and started up the services, which seemed to be a temporary fix. Talking to the team, the following can be added to /etc/systemd/system/heimdalld.service for both the sentry and validator nodes:

[Unit]
  Description=heimdalld

[Service]
  WorkingDirectory=/root/node
  ExecStart=/root/go/bin/heimdalld start
  Restart=always
  RestartSec=3
  Type=simple
  User=root
  LimitNOFILE=4096

[Install]
  WantedBy=multi-user.target

Way to improve: I've added the code above to my sentry and validator to prevent this error from reoccurring. With my increased commission, I'm adding a second sentry node, so that if one sentry fails, the validator still has another sentry to talk to. Each server isn't cheap, but this will help uptime and prevent outages, especially when sleeping or when my computer isn't readily available.

Issue 3:

What happened: This one was tough, and took a lot of time and effort to solve. There was an issue on the network that caused multiple validators to fail (not just Vault), and as I diligently worked over the hours of the outage try fixes and provide error messages to get help. The team documented what happened, and you can read more about the issue here. In short, there was an error at Heimdall block 5237861, causing mine and other validators to throw an "invalid merkle root" error.

Solution: Several of the validators and I posted logs, explained issues, and tried several different fixes before the team came up with a hot-fix. Essentially, the best fix would be to rewind the validator to before that block, and resync with the network. The hot fix is as follows:

cd ~/bor
git pull
git checkout arpit/hot-fix-26-05-2021
sudo service bor stop
make bor
sudo service bor start

Worth noting, normally during an "invalid merkle root" error, if just restarting the node doesn't work the following can be run with bor still running, where <hex of current block> is the hexadecimal notation of the block number:

bor attach ~/.bor/data/bor.ipc

> eth.getBlock("latest").number
> debug.setHead(<hex of above block number - 1000>)

Way to improve: There isn't much I can do here to prevent this from occurring, but I'm documenting some fixes here an in my notes that if it comes up again I will be able to correct it quicker. I've set up backups of my validator so that if there is an issue, I can revert to a previous time quicker. 

Going forward

The Checkpoints Signed percentage is based on the last 200 checkpoints, and since I missed a lot within a short amount of time, I fell to the low 80%. I completely understand my delegators' frustration and I will strive to be better. Since 200 checkpoints is such a small window, we are already back to 90.5%, and should be at 100% within a day or two. Hopefully we stay there longer this time. As I mentioned before, This was the worst Vault has performed since we started, and the only time we were out of the 90% range. As always, let me know if you have any questions. You can always reach out in the comments, find me active in the Polygon/Matic Discord server, or reach on on DM (AlgoRhythm#2931). Thanks for reading!