The purpose of this project is to perform an external, objective and unbiased evaluation of the different Ethereum Consensus Layer Clients in order to create targets for client decentralization. The project will create a number of outcomes in the form of technical reports to provide transparent and reliable data to the ecosystem in order to educate node operators and drive decisions towards a more robust, secure and decentralized network.
We will evaluate the hardware resource consumption of all Ethereum Consensus Layer Clients as well as many other different metrics related to their network bandwidth consumption, number of peers, syncing speed, storage requirements, among many other metrics. We will then produce a detailed technical report describing our research methodology, showing all our findings and drawing conclusions. The technical report will be publicly available on the Internet for everyone to read. Moreover, we will write an easy-to-digest shorter post summarizing our main findings. We also plan to extend the technical report into a scientific paper and submit it to an international conference on blockchain technology. As customary, the scientific paper will explicitly acknowledge the funds received and the support organizations. We will also use social media to disseminate the results of our research.
In our previous work we have demonstrated how consensus bugs could translate into catastrophic failures due to the lack of finality leading to extreme resource consumption. While finality bugs are not frequent, and the Beacon Mainnet has been running stable for over a year, this kind of studies do show why it is important to characterize the resource consumption of the different clients and to be able to spot outlier behavior. This also helps the ecosystem to understand the different trade-offs existing between different clients.
There are a number of performance counters that we can analyze in this study and that will provide valuable information to the ecosystem. In addition to the previously studied performance counters (i.e., CPU usage, memory usage, disk usage, network bandwidth, synchronization time and number of peers) of the main five consensus clients (i.e., Prysm, Lighthouse, Teku, Nimbus and Lodestar) we will also perform a number of new experiments:
- We will study the performance of the new available clients such as Grandine
- We will also study the performance of validator clients such as Vouch
- Study new combinations of Beacon Node - Validator couple
- Compare multiple client modes (i.e., fast mode vs archival mode)
- Analyze the different interoperability pairings between clients (e.g., Common metrics)
It is important to define a clear and fair methodology for the study, as do not want the results to be biased in either positive or negative way. To guarantee fairness in our evaluation we will follow the next steps:
- We will contact each client team and discuss the ideal conditions (in terms of computer hardware requirements) in which their client should be executed.
- We will discuss the different configurations in which each client can be run (e.g., archival mode, fast mode) and test and report multiple configurations.
- When comparing clients we should make sure that they were tested in similar conditions, such as the same number of CPU cores, DRAM memory, etc.
- When reporting results, all details regarding hardware configuration, software environment, time and duration of the experiments will be reported.
- We will verify by multiple means that the numbers obtained are correct, for instance measuring CPU consumption internally and with an external script.