The following deliverable is the completion of this research into the data available for a node operator scoring system. This research examined available datasets, both on-chain and off-chain, for feasibility and usefulness. We critiqued existing scoring systems and suggested a structure for a future Lido system.
Our focus was on identifying what data exists for the development of a NO scoring system that effectively balances stakeholder rewards with risk mitigation. While NO’s can influence the rewards their validators earn, they have far greater control and responsibility for minimizing penalties and slashing. Therefore, optimizing for these risk mitigation factors should be a better proxy for long-term performance.
By viewing a scoring system through the lens of risk, it will allow us to craft a healthy and resilient validator set, that should necessitate higher performance over infinite epochs.
Our investigation centered on identifying potential data sources and metrics crucial for an effective NO scoring system. We explore both on-chain and off-chain factors, recognizing their distinct impacts on overall NO performance and risk management. On-chain, we analyzed millions of data points to identify whether the datasets have the characteristics necessary to be used in scoring.
Limitations of On-Chain Data-Only Systems
While it is feasible to create a NO scoring system solely based on on-chain performance data, such a system would be substantially deficient. It would fail to comprehensively account for the myriad of risk factors, which are pivotal in ensuring a robust and reliable scoring system. We found that overwhelmingly risk mitigation data was necessary for the creation of any NO scoring system. Without it, any other system would optimize for another outcome, without any insight and transparency into the unknown accumulation of these risk factors. Therefore, an on-chain data-only approach would be considerably limited in its effectiveness.
Priority of Off-Chain Risk Data
Our research underscores the paramount importance of off-chain risk data in any effective scoring system. While this necessitates a departure from a fully trustless scoring system, it’s a necessary compromise to achieve a realistic assessment of NO performance.
Risk-Based Scoring Framework
The scoring system should prioritize minimizing penalties and slashing, key factors in long-term performance and stability. Key risk factors include internal processes, hardware, client and server locations, jurisdiction, and operator concentration.
Incorporating On-Chain and Off-Chain Data
The system should utilize a blend of on-chain data and critical off-chain risk data. This approach acknowledges the necessity of human involvement and increased transparency from NOs for a comprehensive risk assessment. Gathering this data is likely to conflict with a transition to permissionless anonymous NO’s. We find that it will be critical to create an incentive structure for NO’s to truthfully disclose information and a remediation system to investigate discrepancies. Without this information, the DAO has little transparency into the accumulation of risk in these factors and hence cannot properly maintain the health of the set.
MEV Data Exploration
Our study has identified MEV (Maximal Extractable Value) data as an intriguing area for future exploration. This includes potential optimizations for capturing MEV and tracking/preventing MEV theft by operators. However, currently, MEV data is not a viable metric for the scoring system due to implementation challenges and its relative unreliability as a dataset.
Data Source Reliability and Selection
Rated.network is identified as a suitable source for on-chain data, given its relatively high accuracy and robust API.
Engaging with the Lido DAO community is crucial, especially around areas like client diversity, MEV strategies, and key management, to ensure the scoring system aligns with community values and risk tolerance. We find that the DAO may benefit from creating stricter mandates for NO’s regarding systems, internal processes, and information disclosure. The economic value to NO’s from participation in Lido is immense and hence the DAO has significant power to enforce standards that will allow for the creation of a stronger scoring system and a healthier validator set.