Cross-chain exchange and proof-of-bond network THORChain was halted [October 27] as the result of a bug causing non-determinism between individual nodes.’ At around 8:00 pm UTC, the THORChain team initially tweeted that developers were aware of a chain outage and were working to find the root cause of the issue. Roughly four hours later, the team posted a further update noting that ‘consensus halts in a distributed state machine are from sources of non-determinism between individual nodes and prevent the ledger from becoming corrupted.’ It stated that the next steps were to find the source of the non-determinism, release an update and restart the state machine, and while noting that step one was close, there have been no updates since, suggesting the developers are dealing with a fair bit of a headache.
[Quarmby, B. THORChain network halted following software bug.(Accessed October 29, 2022)].
.png)
“The team reported that they had identified the problem as string manipulation after another three hours as traders and investors awaited the firm’s response. According to their statement, the developers should have noticed the problem because the incorrect memo was switched out immediately and never entered into the block. The faulty memo was written in the block, which impacted the mainnet since it contains a queue that prevents the swap from synthesizing on the same block. The team had announced the halting of operations due to the incident and asked for patience from its customers as it prepared instructions while the fix was being delivered right away” [Brenda Mary THORChain Network Back Up After a 20-Hour Chain Pause. (Accessed October 29, 2022)].
“In an email to CoinDesk on Friday morning, a representative for THORChain said '[a]s of 10:20 AM (ET) on Friday, the network pause has been lifted and THORChain mainnet is once again producing blocks. Trading will remain paused until the outbound queue has been cleared. Once all pending outbound transactions have been processed, trading will be resumed” [Kessler, S. THORChain Resumes Operation After 20-hour Outage. (Accessed October 29, 2022)
Furthermore, “[i]n an Oct. 28 tweet, the THORChain team said the network was ‘back online and producing blocks’ and had reenabled trading” [Wright, T. THORChain network resumes following 20-hour chain halt. (Accessed October 29, 2022)].
.png)
“Turns out to be string manipulation: code was pushing a cosmos.Uint (instead of a uint64) into a string, which causes the string to get the point of the big int instead of the actual value, causing the memo string to be different on different nodes,’ said the THORChain team following the chain halt. 'Didn’t see this in stagenet because the bad memo never gets written to disk/block, because it gets swapped immediately” [Id].
“The chain was stopped for safety concerns, but according to a THORChain spokeswoman, the company intended to ‘revert once the source of non-determinism is uncovered’. However, during the downtime, the token-swapping platform THORSwap confirmed that Ethereum and ERC-20 swaps were still supported” [Brenda Mary, supra].
This blog will be updated should THORChain’s post mortem, due next week, disclose information contrary to that contained herein.
