What a beginner validator of blockchain networks needs to know

Key Terms

Client-validator — software that stores and uses private keys for signing and verifying the state of the blockchain.

Node operator — an individual or legal entity that launches and manages the software and hardware of the client-validator.

Stake — a certain amount of cryptocurrency that is locked up as collateral to become a validator.

Introduction

To become a validator, there are two paths:

1. Purchasing a sufficient amount of tokens — this option is not suitable for everyone because, besides skills, one needs to spend a significant amount of money (depending on the network and token price, ranging from tens to hundreds of thousands of dollars).
2. Take a place in validator games — this option is much cheaper but requires some skills, and I will discuss this in this article.

Most networks launched recently are Proof of Stake networks, and the article will focus on these types of networks. To explain this without the specifics of particular networks and very simply, each validator makes their stake and is responsible for the work done by that stake. If we make an analogy with the real world, it’s like a company that is responsible for its work with its registered capital. If the work is done well, the company earns income, but if the work is done poorly or not done at all, the company faces penalty sanctions.

When a new POS network is launched, it needs validators, and initially validators are usually nodes of the network creators themselves and project partners. However, for the network to achieve the required performance, be decentralized, and secure, it should have a large number of validators not bound by any commitments or geographical location, otherwise, there is a high chance of collusion among validators or influence on the network physically or legally.

Validator Games

To populate the network with validation nodes, network creators organize “validator games” where a test network or several networks are launched, and anyone interested can receive a stake and participate in the game. During the games, the network and its validators undergo various tests, such as increased load, network stoppage followed by its restart, selector refusal to accept stake for any reason, sudden service failure of the validation node, correct choice of reinvestment strategy. Participants need to cope with all of this as quickly as possible to earn as many points/tokens of the test network as possible. At the end of the games, which last from several weeks to several months, the winners receive a token reward in the main network and, along with this, the opportunity to connect to validation and start earning real tokens.

Required Knowledge

Now let’s understand what specific knowledge is needed to be a successful validator node operator. Regardless of which network you start with, you will need these skills:

Basic understanding of server hardware

Since it’s crucial to know which server to rent/buy, whether clock speed of the processor or the number of input/output operations of the ssd disk is more important, otherwise you may constantly experience slashing or server maintenance costs will outweigh the node’s income. If we talk about the first stage — validator games, then with incorrectly chosen server configuration, you can simply be eliminated from the competition.

Knowledge of Linux systems

Most modern blockchain networks’ nodes run on the Linux operating system. It’s not always enough to download and run the node application; often it needs to be compiled for your processor architecture, OS, etc.

After the application is compiled, it’s necessary to ensure that it works correctly (the node has joined the network, started syncing, and operates without errors). Otherwise, you need to find out why the node didn’t start or frequently restarts. There can be many reasons: the port is occupied by another application, the node was compiled with errors, there is insufficient disk or network speed, and the network is filling up faster than the node can catch up with these changes, the node is constantly killed by OOMK, etc. In short, knowledge of Linux systems is necessary to correctly launch the node and quickly identify and fix the problem.

Ability to work with monitoring systems

The first thing to monitor is the operating system, we must understand how many resources the node consumes at any given time, including CPU, memory, disk, and other resources. Depending on the network load and node uptime, the situation may change, and we must receive timely notifications about this because resource shortages or frequent restarts will affect the node’s performance and, consequently, its ranking in the competition results table. For these purposes, prometheus nodexporter is excellent, it provides enough information about the system to understand its state and identify main problems.

The next part that should be monitored is on-chain metrics: how many blocks you have signed, which epoch is ongoing, whether we participate in elections, what balance we have, whether there is slashing, etc. Often nodes provide most of these metrics (in the “OpenMetrics Text Format”), and all we need to do is to specify to the prometheus server where to collect them.

Sometimes node software is too raw or does not provide all the necessary information, in this case, we can write our prometheus exporter that will collect on-chain data and provide it to the prometheus server in a understandable format.

Next, we can set up rules for monitoring; that is, if any of important metrics in goes beyond the allowed boundaries, we can send a notification to a convenient communication channel, such as email, chat, or incident manager service(I recommend PagerDuty). AlertManager will help us with sending notifications to various communication channels.

Now, the last piece of our monitoring system is a data visualization system, and again, the mainstream choice is Grafana; it allows us to gather all the data we are interested in on one dashboard or make convenient jumps between dashboards if we have many. So, our metrics monitoring stack is Prometheus + AlertManager + Grafana.

Troubleshooting problems

Now a little about why this is so important. For example, you see that you are dropping in the game ranking; you check the state of the client-validator and see that it restarts every 5 minutes or even doesn’t restart but crashes after 5 minutes of operation. How to find out the reason? Here monitoring systems and knowledge of Linux systems will come in handy.

First, we can study the logs of the node itself and see that it says it cannot establish more connections, so it makes sense to check the net.core.somaxconn parameter.

Another option is to study the system logs; for example, we can see that OOMK is killing our client-validator, so it makes sense to look at the memory usage in the system; perhaps it’s insufficient or another process consumes too much.

And perhaps the most banal example is when the disk runs out of space, and the client-validator can no longer replenish its database. In this case, we can either increase our disk or start the node so that it stores not the entire state but only N days.

As you can see, the aforementioned knowledge is absolutely necessary if you want to be a successful Node operator in a POS network or, even more so, a winner of validator games.