Down the Rabbit Hole (Part 1): From Distributed Systems to the Scaling Problems

This is a rabbit hole.

This article is not about explaining blockchains, nor about promoting a specific architecture. It is about understanding how we ended up here. Why distributed systems evolved the way they did, why blockchains look the way they do today, and why scaling them turned out to be harder than expected.

This is Part 1. In this part, the goal is simple: to follow the path from classical distributed systems to peer-to-peer networks, then to blockchains, and finally to the scaling wall they inevitably hit.

Coordination is at the center of this path.

As long as a system runs on a single machine (no matter how strong the machine is), coordination is the fundamental problem encountered. All micro-services running concurrently consume the same resources in a certain location and data center, leading to a performance bottleneck for the system. However, the moment that system gets distributed across machines, data centers, or organizations, coordination gets to be not as implicit as before but becomes costly and fragile. This has always been a wall. Long before the blockchain technology.

Blockchains did not invent this problem, they didn’t start the fire. They inherited it.

What blockchains did change (e.g. Bitcoin) is the environment that coordination happens in. Instead of cooperating the machines, nodes of a system owned by a single operator, these chains run in open and challengeable networks where participants cannot trust each other to process any input and are economically unmotivated to cheat. Since blockchains are designed to survive in a world where no one trusts anyone, they are incredibly secure but this security comes with hard limits that they quickly run into.

In this article, we will explore the evolution from classical distributed systems to peer-to-peer networks, to blockchains, and finally to the scaling wall that forced us to rethink how blockchains are built at all.

Distributed Systems: The Coordination Cost

Let’s consider a globally deployed application backed by highly available databases.

• Writes must be replicated.

• Reads must stay consistent.

• Failures must be hidden from users.

In such an environment, the engineers rely on leader-election protocols, replication strategies, and carefully tuned timeouts to keep the system coherent and one. These systems assume a trusted operator. When a conflict arises (a corrupted replica, a network partition, etc.), there is always an organization that owns the infrastructure. They intervene, rollback the database, and reimburse the customers.

When thought this way, this assumption keeps coordination costs manageable. You can centralize decision making when needed and sacrifice availability for consistency, or vice versa, depending on the business structure. However when you remove the trusted operator, the problem changes entirely.

Peer-to-Peer Networks: Stateless Decentralization

Before blockchains, peer-to-peer (P2P) networks like BitTorrent were an attempt to remove centralized control over the distributed system without getting affected by the coordination costs.

There is no central server that hosts files in BitTorrent. Peers discover each other, exchange data, and leave. This system scales because the peers only care about the data they get and do not need a global shared agreement, a “state” as called in blockchain networks. These peers look only at the file hashes of the movies they are downloading. If they match, then they match.

It’s a working design because the problem, file hashes, are static. How can we change a file once published? There is no mutable state, no concept of ownership, balances, permissions, etc. At that point, this model starts to not work any more.

Blockchains: Order and Verify

The reason blockchains exist is to solve the problem P2P networks hit: global agreement on mutable state without trust.

Participants, “nodes” from now on, must agree on what happened and in what order even when some of them are dishonest. These early monolithic blockchain nodes have to do every single job:

1. Listen to the transaction

2. Do the math (Execution).

3. Vote on the order (Consensus).

4. Store the receipt forever (Data).

Consensus protocols establish ordering, replicated state machines ensure deterministic outcomes, and economic incentives discourage cheating. Every node independently verified everything, leaving less room for cheating. And for a time, this approach worked!

But this too had its own bottlenecks.

The Scalability Problem

Every node in the blockchain network must process and execute every transaction, and store the state. The same verification steps get duplicated thousands of times. Depending on use cases (RPC indexer, polling, etc.), blockchains are live databases that serve data in real time. It is hard to imagine for a database to re-execute every request input given they are usually lazy and slow, and if that same database joins low-latency consensus rounds, it ceases to be a mere database and becomes an agent that verifies.

And as the blockchain adoption and usage increases, coordination costs increase linearly. The throughput is bounded by the nodes slowest in latency. Transaction fees rise as block space becomes scarce and latency grows as consensus rounds multiply.

Unfortunately, these are unavoidable prices of maintaining a single, globally consistent state machine without trust.

Then engineers had a change in their perspectives and started to ask “What actually needs to happen on-chain?” instead of making the blockchains faster.

Layer 2s: Keeping the Execution Off-Chain

One of the first answers to this question was layer 2 solutions.

Not all computation requires global verification. In traditional systems we see that execution and settlement are routinely separated: payments are authorized instantly and settled later. Think of layer 2 like a bar tab: instead of the bank (the blockchain) processing every single drink ordered, the bartender (the L2) keeps a tally on a piece of paper. At the end of the night, the bar just sends one final total to the bank.

Layer 2s apply this pattern to the blockchains keeping the execution off-chain. This way, the base layer becomes a settlement and security provider rather than an execution engine. This system reduced congestion, lowered costs, and preserved trust assumptions. For the first time since its inception, the blockchain technology began to scale in practice.

An Incomplete Separation

Thanks to Layer 2 solutions’ success, execution moved elsewhere but data availability, consensus, and settlement remained tightly coupled. A monolithic blockchain has these four crucial layers: execution, data availability, consensus and settlement. And L2s handle only one, leaving the base chain still struggling with the other three. Then we started to realize a blockchain is not one solid block but rather a stack of different functionalities that do not necessarily have to live together while still maintaining the integrity of the network.

In Part 2, we will explore how separating data availability and consensus leads to modular blockchains and why app-specific chains follow from that separation.