What is sharding?

Sharding is a method of splitting and storing a single logical data set in the form of multiple databases. Another definition of sharding is the horizontal separation of data.

When and who invented sharding?

The concept of sharding has been used in managing traditional centralized databases since the late 1990s. The term “shard” (fragment) was spread thanks to one of the first online multiplayer role-playing online games, Ultima Online , in which developers distributed players to different servers (different “worlds” in the game) to cope with traffic.

A popular scenario for using sharding in business is dividing a user database by geographic location. Users belonging to the same geographical location are combined into one group and placed on a unique server.

What is sharding in the context of the blockchain?

Blockchain is a database with nodes representing individual servers. In relation to the blockchain, sharding implies the separation of the blockchain network into individual segments (shards). Each shard contains a unique set of smart contracts and account balances.

Each shard is assigned a node that verifies transactions and operations, in contrast to the scheme in which each node is responsible for verifying each transaction in the entire network.

Dividing the blockchain into more manageable segments can increase transaction throughput and thereby solve the scalability problem that most modern blockchains face.

How does sharding work?

Explanation on the example of Ethereum:

The Ethereum blockchain consists of thousands of computers or nodes, each of which “lends” a certain amount of hash to the network. It is this hashrate that allows the Ethereum Virtual Machine (EVM) to function - execute smart contracts and manage decentralized applications (DApps).

Ethereum currently operates on a sequential basis, in which each node must calculate each operation and process each transaction. Therefore, the transaction passing through the verification process requires considerable time: Ethereum performs approximately 10 transactions per second, while Visa, for example, has this figure in the region of 24,000.

Adding computers to the network does not necessarily increase efficiency, since the entire registry is stored on each device, and the verification chain simply becomes longer.

The idea of ​​sharding is to abandon the model in which each node must calculate each operation, in favor of the parallel execution model, in which the nodes process only certain calculations. This allows you to process multiple transactions in parallel.

The blockchain is divided into separate shards (subdomains or segments). Nodes only manage the part of the registry to which they are attached (they execute processes and confirm transactions), and they do not support the entire registry.

What problems does sharding solve?

Sharding is a potential solution to the problem of scaling.

The more popular blockchain becomes, the more users initiate transactions, launch decentralized applications and other processes on the network. As a result, transaction speed drops, which prevents blockchain expansion in the long run. The growth of transactional activity requires nodes to intensify the transaction verification process. There is a risk that these blockchains may become clogged, as happened with Ethereum during the CryptoKitties boom, when the game accounted for 11% of network transactions.

If groups of nodes are responsible for individual segments, then each node does not need to maintain the entire registry to perform each operation. Therefore, transaction validation can be performed in a parallel rather than a linear manner, which increases network speed. Thus, the problem of scaling is solved.

What are the disadvantages of sharding?

The main challenges of sharding are communication and security. If you divide the blockchain into isolated segments, then each shard will become a separate network. Users and applications of one subdomain will not be able to communicate with users and applications of another subdomain without using a special communication mechanism.

A segmented blockchain also raises a security problem, since it’s easier for hackers to grab one shard, due to the lower hash required to control individual segments (the so-called 1% attack ).

After capturing a segment, attackers can send invalid transactions to the main network. Also, data in this particular segment may become invalid and may be irretrievably lost. Ethereum offers a randomized sampling solution - shard protocols are randomly assigned to different sections to confirm block authentication.

What are the alternatives to sharding?

The developers proposed two solutions to improve the performance and speed of transactions in blockchains.

The first solution is to increase the block size. The key idea is the larger the block size, the more transactions can be placed in it and, therefore, the greater the number of transactions per second.

However, the larger the block, the more processing power is needed to verify it. If the block size is increased significantly, then only the most powerful computers will be able to control the processing power needed to operate as nodes.

The high cost of such computer equipment means that the node pools will inevitably become smaller and more centralized, which increases the risk of attack by 51% . Increasing the block size also requires a hard fork , which threatens to split the community: if not all users accept the update, then two different chains will appear that use different coins. Increasing block size may not be a long-term solution.

The second suggestion is to use altcoins so that various functions and applications are implemented on their own networks with their own coins.

Such a model will increase productivity, as a single blockchain will not be overloaded, but will also increase security risks, since computing power will be distributed across several blockchains. Again, the risk of network hacking will increase because the computing power needed to carry out a 51% attack will be much less.

Who uses sharding?

Zilliqa is the first platform to implement sharding. At the testnet stage, she managed to reach a figure of 2828 transactions per second.

The Near Blockchain Ecosystem allows developers to create and implement decentralized applications. Near calls himself a “Shard Blockchain on PoS” and claims that its sharding technology allows nodes to remain small enough to function on low-performance devices — potentially even on mobile phones.

Ethereum offers a blockchain ecosystem for implementing DApps based on smart contracts. The Ethereum Foundation plans to include sharding in an updated version of the Ethereum 2.0 protocol .

Other sharding projects include Cardano , QuarkChain, and PChain.

What is the future of sharding?

Sharding technology is featured in the white paper of the digital currency Libra. In anticipation of the launch, Facebook acquired Chainspace, whose development team specializes in sharding. The specific details are not yet known, but it can be assumed that a kind of sharding will be introduced in the Libra blockchain.

Sharding can theoretically be the solution to the so-called blockchain trilemma.

The trilemma of the blockchain, as Vitalik Buterin explained, is that at the same time you can save only two of the three key features of the blockchain - security, decentralization and scalability. If you overcome the difficulties that sharding faces, you can scale distributed networks without sacrificing decentralization or security.