I still remember running my first blockchain node years ago: it was a Celestia testnet node, and I spent most of my time figuring out how to get it running. Configuring the client, keeping it synchronized, dealing with errors I had never seen before. I was quite happy once it started working, and I suppose most people running their first node would feel the same.
Things are a bit different when you operate infrastructure that other people actually depend on. An RPC endpoint might respond to every request and still serve outdated information, while a validator might have excellent uptime without revealing much about how its signing keys are managed or what happens if something goes wrong.
I find this distinction particularly interesting because we tend to judge infrastructure by what we can observe. We see whether a validator is participating in consensus or whether an endpoint is responding within its promised SLA, but much of what determines its reliability happens somewhere else.
And that’s what I want to discuss here: the work that goes into operating blockchain infrastructure beyond simply keeping the nodes running, why some familiar engineering practices become more complicated in blockchain, and why I believe that the less visible parts of an operation deserve more attention.
When Good Engineering Is Not Enough
Blockchain infrastructure relies on much of the same engineering knowledge as any other distributed system, there’s no reason to pretend otherwise. You still need to understand the operating system, networking, storage, monitoring, and quite a few other things before you can operate anything reliably.
Yet there are situations where following a perfectly reasonable engineering practice without considering the underlying protocol can cause more trouble than it solves.
Take redundancy, for example. Running multiple instances of a service is normally a good way to improve availability. But if two validator instances using the same consensus key sign the same block, the validator may be slashed with some serious financial consequences. The usual approach to avoid a single point of failure suddenly requires a few additional precautions, some of which are very specific to the network you’re operating.
Similar challenges exist on the RPC side as well. Keeping an endpoint responsive is one thing, but making sure that the node behind it is synchronized, following the canonical chain, and providing the data a customer needs is another. And if a node loses its peers or ends up following the wrong fork, restarting it might not help much unless you’ve understood why it got there in the first place.
None of this means conventional DevOps practices are somehow inadequate for blockchain. On the contrary, they need to be applied with an understanding of what the protocol is doing underneath. Every network has its own assumptions, and even two clients serving the same blockchain can behave quite differently when something goes wrong.
Making Reliability a Habit
An upgrade is a good example of how much work can sit behind an otherwise ordinary operation. Replacing a binary and restarting a service might take a few minutes. Figuring out whether the release is safe to deploy, whether the database remains compatible, what needs to be backed up, and how to recover if the upgrade fails can take considerably longer.
Of course, none of these concerns are unique to blockchain! Change management, access control, documentation, and recovery planning have been part of infrastructure engineering for a long time. Still, when an unsuccessful upgrade can interrupt a customer’s access to blockchain data or affect a validator’s participation in consensus, it’s worth being particularly careful about them.
At node101, quite a lot of our work goes into these less visible parts of the operation. Alongside running validators and RPC services, we’ve invested time in reviewing our security procedures, improving how changes are managed, and having our operational practices examined externally. Our engineering team has also put considerable effort into areas such as infrastructure architecture, automated deployments, and troubleshooting, which are not necessarily blockchain-specific but are indispensable to operating these services properly.
I don’t think this is the kind of work anyone gets particularly excited about from the outside. There’s not much to announce when an upgrade goes as planned, an access policy does its job, or a recovery procedure turns out to be useful. But these things matter, especially when the alternative is finding out during an incident that something was never properly considered.
Trust Needs Evidence
Something else worth considering is how little information customers or delegators actually have about the infrastructure they rely on.
If you’re delegating to a validator, you can look at its uptime, commission, and on-chain performance. If you’re using an RPC provider, you can measure its latency, throughput, and availability. These are perfectly reasonable things to examine, although I would not say they tell you enough about the operation itself.
For example, how are the validator’s signing keys protected? Who has access to production infrastructure? How are incidents handled, and are the recovery procedures ever tested? Unless you have access to the operator’s internal systems and processes, you probably have no way of answering most of these questions.
This is partly why independent evaluations are useful. They allow operational practices to be examined against a set of criteria rather than relying entirely on what an operator says about itself. Security controls, internal procedures, infrastructure design, and even the way responsibilities are distributed within a team can all be subject to external scrutiny.
We’ve put our own operations through such evaluations at node101, and I consider them useful precisely because they’re not limited to whether our nodes happen to be performing well. They examine parts of the operation that would otherwise be difficult for someone outside the company to assess.
Naturally, none of this means an operator cannot fail after passing an assessment. But there is still a meaningful difference between saying that an infrastructure is reliable and being able to show how it is operated.
The Work That Never Ends
I wouldn’t say there’s a particular point at which blockchain infrastructure can be considered completely figured out. Client implementations change, protocols introduce new requirements, and different networks occasionally present problems that don’t resemble anything you’ve dealt with before.
The more interesting question is how operators can maintain a consistent standard of service despite all of this. And there’s something slightly ironic about it: blockchain is built around the idea that participants shouldn’t have to trust one another, yet most people accessing a blockchain through an RPC provider or delegating to a validator still know relatively little about the infrastructure they’re relying on.
We spend quite a lot of time at node101 thinking about that gap, and I suspect it’ll become more important as blockchain infrastructure continues to find uses beyond its original audience.
After all, there’s a lot more to operating a node than making sure it’s running.