Metrika Relay Monitor grant proposal

TLDR

Metrika is proposing a MEV Relay Dashboards and Alerts solution to help Node Operators understand the performance of Relays in real time and make adjustments in case a Relay is not performing as expected. Lido members will have free access to dashboards and alerts for 6 months after delivery. After the 6 month period Metrika will ensure that access will be free for any Node Operators with ≤5 validators. Compared to the existing MEV Relay Dashboards in the community, Metrika’s solution has some key differentiators:

  • We provide metrics on Bid Faults to ensure that Relays are performing their duties (e.g., propagate valid blocks, return bids within 1 second, process validator registrations without errors)
  • We provide Alerts on important events to prevent proposers from missing out on any MEV
  • Most dashboards collect data from on-chain Blocks by Relays. Metrika uses Relay Monitors to collect Bid (off-chain) data from all Relays. Our method produces a more comprehensive picture of the performance of Relays, instead of only displaying on-chain data.

Background

After speaking with several Node Operators we have recognized a pattern of problems in the Ethereum ecosystem due to gaps in existing solutions. One of the main opportunities for improvement is MEV Relay Transparency.

MEV Boost is used by all Lido Node Operators, but at the moment there is little transparency to the reliability and performance of Relays. If a Relay were to stop providing bids, Node Operators and Builders would be unaware and unable to take corrective actions. For example, on November 10th, the Flashbots Mainet Relay experienced a DoS attack which resulted in a drop in MEV-Boost blocks between 12:00 - 16:00 UTC. Furthermore, existing dashboards are incomplete due to limited datasources, lack of real time data, and insufficient metrics. After connecting with Chris Hager from Flashbots, regarding Metrika’s Ethereum Data Challenge Blog, we discussed potential solutions to the Relay transparency problem that Flashbots had outlined. Metrika is proposing a MEV Relay Transparency solution which includes dashboards and alerts to help Node Operators better choose Relays and react quickly in the event a Relay is not performant.

Scope of Work

Given the opportunity stated above, Metrika is proposing a solution for this grant: MEV Relay Transparency.

MEV Relay Transparency

There are two components to our solution: a public MEV Relay Dashboard and MEV Relay Alerts. The MEV Relay Dashboard will provide charts and visualizations of several metrics listed below grouped by Relays. The MEV Relay Alerts will notify users when Relays are not submitting bids and alerts on the most relevant metrics from the MEV Relay Dashboard. The data for these dashboards and alerts will be collected via a modified version of open source Relay Monitors. Metrika will push any relevant modifications to the open source project. Additionally, we will maintain connections to Relay APIs to minimize interruption of dashboards and alerts.

MEV Relay Dashboard

The dashboard will display the following metrics:

  • Total bids - Ping each Relay for a bid in each Slot
  • Bid faults - The number of faulty bids submitted by a Relay. Currently we are tracking the following bid faults:
    • Malformed
    • Consensus invalid
    • Payment invalid
  • Relay latency - the latency of each Relay is dependent on the location of a Validator or Relay Monitor. We provide the latency for several regions.
  • Bid Value Distribution
    • Average Rewards - The average of Bid values for each Relay
    • Median Rewards - The median of Bid values for each Relay
    • Minimum Rewards - The minimum of Bid values for each Relay
    • Maximum Rewards - The maximum of Bid values for each Relay
    • Percentiles of Rewards - The the 25th and 75th percentile of Bid values for each Relay
  • Block Count - The number of blocks that landed on chain
  • Total Rewards - Total Rewards paid to proposer by Relay
  • Unique Builders - The number of unique builders sending bids to the Relay
  • Time series of MEV blocks - The percentage of blocks on the chain that are MEV-Boost blocks vs natively built blocks

Users will be able to view the metrics by these 10 Relays (and we will add any additional Relays that come online):

  1. Flashbots
  2. BloXroute Max Profit
  3. Eden
  4. Blocknative
  5. BloXroute Ethical
  6. BloXroute Regulated
  7. Manifold
  8. Relayooor
  9. Agnostic
  10. Ultrasound
  11. Aestus

MEV Relay Alerts

Users will be able to set up a free account on the Metrika platform. They will be able to subscribe to several alerts listed below that will provide information about Relays.

Alerts that will be provided:

  • Consecutive Missed Bid - Alert if a Relay misses five consecutive bids
  • Bid Fault Alert - Alert if a Bid Fault occurs
  • Relay Latency - Alert if a bid request takes more than 0.5 second, MEV-Boost times-out after 1 second
  • Intended Fee Recipient - Alert if Proposer fee recipient ≠ intended fee recipient. Users would need to input their validator address in order to receive this alert.

Alerts to be delivered via the following options:

  • Opsgenie
  • Pagerduty
  • Email
  • Webhook
  • Slack
  • Discord
  • Telegram

What is Metrika

Metrika, founded in 2019, is a Series A startup building a monitoring and analytics platform to enable transparency, performance, and reliability to all operational stakeholders across blockchain ecosystems. Our key achievements to date is the launch of the Metrika Platform, which is informed by our deep analytics engagements with blockchain teams over the past three years, including with Ethereum, Solana, Algorand, Flow/Dapper Labs, Hedera Hashgraph and Axelar. Validators have used the Metrika Platform to discover and resolve issues, such as misconfigured validators, to increase performance based rewards.

Grant Request

Timeline

Breakdown Timeline for each of the following components:

  • MEV Relay Dashboard - 4 weeks
  • MEV Relay Alerts - 5 weeks
    Screenshot 2022-12-13 at 9.32.54 AM

Total Project Duration => (2 months)
(1 Front End Engineer, 1 SRE, 1 Analytics, 1 Backend Engineer, 0.5 Test Engineer)

Access

Access and ongoing support relating to all features outlined in the Scope of Work section above is timeboxed to six months after delivery of the full scope of the proposal.

Fees and Payment

For the Scope of Work proposed above, Metrika is requesting a $75,000 grant with the following structure for execution:

  • 25% upon agreement
  • 50% upon delivery
  • 25% up to 30 days after delivery

We are open to accommodating DAI, at the suggestion of the LEGO committee.

6 Likes

Hi @slal12,

Thank you for this proposal, I’m happy to tell that LEGO council voted for this, but where would you like to receive a first part of it?

2 Likes

Hi @Alex_L please send the payment to 0xbff4815F393dFcf327C85b105662fa3eded951A2.

1 Like

@slal12 1-st grant is disbursed! Looking forward for the project development :partying_face:

The funds are transferred in two transactions:

2 Likes

Hi Everyone, we are at the midpoint of the grant so I want to share an update on the progress we’ve made so far:

  • We have implemented the Relay Monitor and deployed them in 5 different regions
  • We have added several charts and tables Dashboards including Total Bids, Bid Faults, Relay Latency, Block Count, Total Rewards, and Unique Builders

Over the next month we will focus on:

  • Implementing the 4 Alerts
  • Adding the remaining charts and tables
  • Reorganizing the layout
  • Increasing reliability

We are on track to deliver by Feb 19 and you can see our progress in the live dashboard. Any feedback or questions are always welcomed!

2 Likes

Thank you for the update Sachin! Looking forward to the live dashboard and seeing this in action.

3 Likes

Great update! Big fans of the Metrika team

2 Likes

Metrika dashboard is fantastic. Looking forward to the final product!

1 Like