Summary
This update serves as the first performance report for the 12 Lido x Obol Cohort 1 clusters active on mainnet via the Simple DVT Module. In light of overall satisfactory performance, it is proposed that the key limits for Obol Cohort 1 are raised to 40 validators in a phased approach, and the key limits for Obol Cohort 2 are raised to 5 validators once they have completed onboarding.
Performance Analysis
Since the May 2nd activation date of the Lido x Obol Cohort 1 validators, overall cluster performance has been strong, with a single temporary caveat.
During the first four days following activation (most of which were over the weekend), a string of four missed block proposals was observed across a range of Lido x Obol mainnet clusters.
Upon looking into the logs from clusters, it was noted that a flag was missing from most participants using Lodestar to set useProduceBlockV3 to False due to configuration errors. A fix was rolled out to the Lodestart component of lido-charon-distributed-validator-node and participants were asked to update their configurations, a process that took approximately 48 hours to fully complete across the 84 participants. However, during this time, another block was missed as more than 3 nodes of the cluster had not yet updated their configurations.
Since May 7th when all clusters had rolled out the update, the Block Proposal Success Rate for Lido x Obol mainnet validators has been 100%. Over the course of the total initial monitoring period of May 2nd to June 6th, the aggregate block proposal success rate stands at 73.68%, with 14 proposed blocks and 5 missed blocks.
During this time, as per Rated average uptime across clusters stands at 99.95% and the Average Validator Effectiveness is 96.48%. This compares to the 30 day overall Ethereum validator set performance with an average Uptime of 99.7% and Average Validator Effectiveness of 96.36%.
Despite the initial issue with Block Proposals, it is clear from the 100% Block Proposal Success Rate since the fix, as well as the above network average performance metrics, that performance to-date is more than acceptable to continue scaling Lido x Obol mainnet clusters.
For a detailed overview of cluster level metrics & the list of validator keys, see the analysis: Obol Cohort 1 Performance Metrics - Google Sheets
Proposed Next Steps
As a result, it is proposed that the key limits for Lido x Obol Cohort 1 clusters be raised to a maximum of 40 validators each until the next performance monitoring period (which will end no earlier than 37 days after the publication date of this update), in a phased approach of raising the key limits to 20, followed by an additional raise to 40 after the initial 15 additional keys have been deposited to and observed to activate without issues.
It is also proposed that the key limits for the Lido x Obol Cohort 2 clusters are raised to 5 validators each, with a performance monitoring period and follow up performance report to be shared no earlier than 30 days after the keys are deposited to.
The relevant Easy Track motions for these key limit raises would be communicated in this thread and would follow the default approach of being veteoable by LDO holders over a 72 hour period.
This begins a 7 day discussion period for the DAO to consider the performance report and proposal to raise key limits to 40 validators in a phased approach for Obol Cohort 1, and raise key limits to the 5 initial validators for Obol Cohort 2.