As mentioned in a recent post, one challenge for Lido to decentralize the process of onboarding new validators is measuring validator performance over time and across different node operators.
Luckily, this is something that can be done today, for Lido’s current set of node operators. Having these measurements would make explicit where, in practice, the “bar” is set today, and can be used as a benchmark for a future, more open, node operator selection process.
LEGO would therefore like to request proposals to build an easily accessible dashboard where the Lido community can measure the performance of validators.
The dashboard should have information on all current Lido node operators, and track the following metrics:
Attestation Percentage;
Block Proposal Percentage;
Balance;
Income earned (average daily/last week/last month/since live);
Other relevant metrics at your discretion.
The dashboard should also keep historical records for all of these metrics, and ideally allow viewers to export the raw data for the pupose of further analysis.
If you would like to apply for this bounty, please share your proposal here. You can reach out to @timbeiko or @vshvsh for questions.
One other metric to include: ‘attestation inclusion delay’. i.e. how long after possible to include an attestation it gets included (fee is reduced accordingly for attester).
Might be some nuance in defining as both the generation/propagation of the single attestation and the generation/propagation of the aggregate could have delays but imo worth considering b/c affects the reward.
A bit of color here: for validation in Ethereum, we have an internal dashboard (even two of them) to that nature that are flaky as hell, or too crude for publishing.
One is based on chaind (GitHub - wealdtech/chaind) and this super article by Adrian Sutton, fair dose of regularly refreshed materialized views, and redash as the frontend.
We use GitHub - lidofinance/lido-python to get Lido’s public keys into the same database to be able to join it with pre-processed chaind data. It’s pretty flaky because chaind is not trivial to maintain and we don’t have the capacity to do it well right now. Other parts of the solution served us well, and redash as a frontend is a really great idea for things like this. It’s not open source for the moment but we will share if you want to build upon it.
The other is just regularly ingesting all validators’ balances in PostgreSQL and we’re using SQL to find the ones that belong to lido and underperform, and redash again to make a dashboard. It’s far less precise and less data-rich than the chaind based version but works well for the purpose of monitoring/alerting/crude analysis, and is less brittle.
One is too flaky and the other too crude to publish, plus I think it’s good if public monitoring of Lido’s validators is maintained by independent actors hence the RFP.
What’s the takeaway from our experience here:
we really want stable, rich, near real-time data from the beacon chain that can be SQL-queried or exported to run some analysis on. If we have only that, we can build the dashboard by ourselves, though it’s not my preferred way to do it. The minimal deliverable here can be a Google BigQuery dataset based on a well-maintained, fault-tolerant chaind deployment.
the stability of it is very important, as is the ability to build custom dashboards on top in a non-blocking way.
I think we can fund multiple solutions here, but at least one of them should be well-maintained; and all should be open source. Minimal deliverable is a public dataset; the desired one has a cool user interface to it as well.
Quick update: we’ve had 4 interested parties so far in various draft stages of RFPs. If you had an intention to submit a bit and aren’t in discussions with me or @vsh about it, please reach out now, even if your proposal isn’t done. Assuming no one else comes forward, I expect we’ll have a decision in the next week or so
We felt this proposal met all of the needs for the project, was priced fairly, and that the team’s previous work was a good indicator of the quality of their work.
Given the cost (35,000$, paid over 3 tranches), this will require a LEGO vote with majority approval. I will report back here once the vote as occurred.
Other applicants who submitted a proposal have been offered a sandgrain (100 LDO) grant for their effort.
Status update: we sent out the final payment for the project today, and plan to keep working with @abstract_df for maintenance and hosting of the project in the foreseeable future. LEGO committee members have +1’d the Pebble-sized addition to the grant for the next 3 months of hosting/maintenance.
Since starting the project (back in pre-history, late 2021) the tooling landscape around the beacon chain has grown and improved substantially. While the dashboard that we put up at stakerops.io as part of this project was somewhat unique in the those early days, that is no longer the case.
In other words, it has served its purpose, and late Jan 2023 we agreed to take it down and save the hosting costs and redirect its (minimal) maintenance time elsewhere.
Will post updates for the work in this thread as it gets started/paid/completed/etc.