In an old article of mine, I mentioned that it was Axie Infinity's media exposure during the Covid-19 pandemic that ushered the popularity of blockchain gaming. These "Play to Earn" scheme of blockchain games attracts users who wanted to earn some profit while gaming. There have been mixed reactions from professional gamers since the quality of these blockchain games are not comparable to existing video games in the market, but the promise of a profit is too tempting for ordinary users.
Last March 23, 2022, this promise of profit crumbled for most players of Axie Infinity when the Ethereum bridge Ronin was hacked for $625 Million worth of digital assets. That is, $600 Million (173,600) ETH and $25 Million USDC stablecoin. The Ronin network is a Layer 2 (Ethereum sidechain) network used to power the Axie Infinity economy focused on user transactions. The hack was said to be an exploit of an inherent vulnerability of blockchain bridges.
The Ronin bridge is currently closed as of this article. See how to get the BlogToon NFT below at the end of this article. Get a BlogToon NFT, own a piece of history.
The bridge-rooted vulnerability is not new. The PolyNetwork hack last December 2, 2021 was an example of a bridge-rooted vulnerability. Per Badger DAO technical post mortem report, the cause of the PolyNetwork hack was an API key that was generated and deployed without authorization. This API key gave the attackers access to the Badger application, resulting to the hack.
For the Ronin network hack, the Sky Mavis team posted a community warning here that was updated as of April 2, 2022. Sky Mavis is the company behind Axie Infinity.
Details of the Hack
Sky Mavis' Ronin validator nodes and Axie DAO validator node were compromised resulting to the siphoning of the digital assets from the Ronin bridge. Per community warning, the attackers used hacked private keys to forge fake withdrawals. The attack was only discovered when a user reported unable to withdraw 5,000 ETH.
At the time of the attack, Sky Mavis' Ronin network has 9 validator nodes. Five (5) out of nine (9) validators are required to sign off on a deposit or withdrawal event. The attacker managed to have control over four (4) validator nodes and a third party validator run by Axie DAO. Validator nodes are used to verify information circulating in the blockhain.
Only two transactions were needed to siphon out the digital assets to the hacker’s main address “0x098B716B8Aaf21512996dC57EB0615e2383E2f96”.
Sky Mavis Response to the Incident
Sky Mavis released a community warning six (6) days after the hack. Sky Mavis took the following actions in response to the incident:
1. The Ronin Bridge and Katana DEX were closed until further notice.
2. It claims to have reached out to law enforcement officials, forensic cryptographers and investors to ensure the recovery of funds.
3. Increased validator threshold from five (5) to eight (8).
4. They are working with Chainanalysis to monitor the stolen funds.
An additional action they claimed to have taken was that they are in the process of migrating their nodes separate from their old infrastructure. Personally, I do not think this is an action. They are just informing users about it but from how it was phrased, migration is not yet done. More on this later.
Binance, a separate entity from Sky Mavis, has disabled their bridge to/from Ronin as a response as well.
The Newsletter was subsequently updated on March 30, March 31, and April 2.
Gaps and Recommendations
The Root Cause
Per community warning, Sky Mavis advised, "the root cause of our attack was the small validator set which made it much easier to compromise the network".
I do not fully agree. Why was the hacker able to access five (5) validator nodes in the first place? What difference does it make if I add three (3) or ten (10) more validator nodes if the hacker will still be able to gain unauthorised access to these nodes anyway?
We need to establish the underlying symptoms to have a true understanding of the problem. For me, the root cause is the compromise of the private keys resulting to the malicious withdrawals.
Gaps and Recommendations
Below are my personal opinion (not of Sky Mavis) of potential gaps related to the root cause of the incident based on what information I have gathered from listed references. Please check official statements from Sky Mavis for updated information about the matter based on advise from their internal and external pool of hired experts.
1. Weak security of validator nodes.
Sky Mavis only has nine (9) validator nodes, of which only five (5) validator nodes are required to agree to approve a transaction (deposit or withdrawal). The attacker was able to gain access to the five (5) validators resulting to a successful hack. So, what do we mean by this? If the attacker gained access to four (4) validators, the ETH transfer will not be successful, because 5/9 signature verification is required.
As pointed out above - does increasing the number of threshold validator nodes from five (5) to eight (8) make the Ronin network more secure? Yes and no. Yes, if the number of threshold validator nodes is increased, consensus of more nodes will be required to approve a transaction. But, as long as hackers can gain unauthorised access to Sky Mavis network, they can always gain control of the required validator nodes! In short, we must note that the success of this will still depend on Sky Mavis' overall security.
The validator nodes needs to be secure and segregated from each other, such that even if one validator nodes gets compromised, it does not mean all other nodes are compromised as well.
2. Weak segregation controls.
The fact that the attackers were able to gain control of five (5) out of nine (9) validator nodes says a lot about the weak implementation of segregation in the network and per host. Each validator node must be secure and segregated from each other, in case one validator node falls under a hacker's control.
There is also the concept of Multi-Party Computation (MPC), wherein private keys can be sharded and distributed to multiple parties and restructured dynamically. The loss or theft of one shard of a private key will not mean the private key has been compromised since it will require all shards to be available.
3. Weak/ lack of process to manage third party integration.
Per Sky Mavis, it sought assistance from the Axie DAO last November 2021 to distribute free transactions to alleviate an immense user load. Sky Mavis allowlisted the Axie DAO to sign various transactions on its behalf. This was stopped last December 2021 but the Axie DAO IP was still on the allowlist and was not removed.
While Axie DAO is a separate entity from Sky Mavis, by including Axie DAO in the allowlist, Sky Mavis just opened a potential vector of attack via Axie DAO or vice versa. Sky Mavis was responsible in ensuring the "integration" was secure and necessary.
4. Weak/ lack of process to manage access and permissions.
Sure, the allowlisting was necessary to address an immense user load - we don't want irate users breathing down our necks. But not removing the allowlist once it was no longer necessary has created an opening for the attacker who was just waiting for an opportune time to strike.
Sky Mavis must ensure appropriate roles and permissions are defined. If not necessary, access should be removed.
5. Persistent use of old infrastructure.
I did not fully understand why Sky Mavis even mentioned as part of their actions taken after the hack that they are "in the process" of migrating their nodes. They may have been migrating from old to new infrastructure, but I do not think that counts as a response. Maybe what they mean is that they are facilitating the migration because of the attack.
Anyway, old infrastructure means more vulnerabilities to exploit. Hence, I understood why Sky Mavis wanted to inform their users about it. Sky Mavis needs to continue with the migration as soon as possible.
6. Weak triaging, logging and monitoring controls.
As per community warning, Sky Mavis only discovered the hack on March 29, six (6) days after the hack. The hack was only discovered when a user reported it was unable to withdraw 5,000 ETH from the Ronin Bridge.
There is something wrong when you only rely on your users to report or file complaints. Sky Mavis must have their own monitoring and logging solutions and processes in place. They must have a certain threshold that will trigger an investigation. Valid thresholds can be the number of transaction failures across a certain number of users or failed transactions of a certain big amount of ETH.
Although the attacker did the hack in one day, shouldn't the team be informed if someone just transferred 173,600 ETH and $25 Million USDC stablecoin to another wallet, or are we saying these big of a transaction is commonplace in the Ronin network that is why we don't need to monitor these events? I do not think so.
Also, if you are providing a service, most often than not, there will be a certain percentage x of complaints that are fake or spams. Your team cannot handle all complaints at once. You will normally have a ticketing process for each complaint, but having no triage or prioritization increases the risk that your team will miss a real complaint that may indicate a security breach has happened.
I believe the weak monitoring controls has attributed to the delays on identifying the hack. Sky Mavis needs to establish real-time logging and monitoring controls and establish effective triaging of events to prioritize potential incidents.
7. Weak social engineering controls.
Sky Mavis insisted that the hack happened when an attacker "found a backdoor through our gas-free RPC node". RPC stands for Remote Procedure Call and compared to the validator node, is a different type of PoS node used only by developers. There were no details shared on how the hackers gained the private keys or how a backdoor could have existed in the Ronin network. However, as per March 30, 2022 tweet of Aleksander Larson, COO of Axie Infinity, he confirmed that it was a social engineering attack combined with human error on December 2021.
A social engineering attack is the use of psychological manipulation of people into divulging confidential information or performing actions. A social engineering attack can be done using different mediums, e.g., via a phishing email or even an SMS with a malicious URL link. Sky Mavis needs to establish a robust security awareness plan for its employees, establish a security-aware culture., and deploy anti-phishing tools to protect its emails and communication channels.
The Ronin network hack is first and foremost a security problem. We can talk all day about the blockchain trilemma (decentralization, security, scalability). But, regardless of how you look at it, you cannot dumb down security when you have a market cap of $5.15 Billion.
Please stay tuned for my follow-up article about the Ronin network hack that will focus on the aftermath.
BlogToon NFT Project
Hope I was able to provide some useful insights. If you reached the end of this article, thank you. Please show your support with a like, comment or a tip. It would be greatly appreciated. Please follow Marcusblue Instagram and Twitter pages. Thank you for your support and appreciation.