Lido on Ethereum: Relay Voting Proposal

Proposals
1

During the “soft-rollout” of MEV-Boost across Lido Node Operators as voted upon by the DAO in v1.0 of the Lido on Ethereum Block Proposer Rewards Policy, Node Operators have tested multiple relays in the six weeks following the merge.

Node Operators have been asked to produce blocks via MEV-Boost infrastructure by obtaining sealed blocks from the maximum possible number of relays from the providers that expressed interest during Lido’s initial call for relay providers.

With the end of the “soft-rollout” period now approaching and initial tests of relays undertaken (nearly all Lido related block proposals are now being delivered through MEV-Boost infrastructure), we believe the timing is appropriate to vote upon the relays that will be included in the respective “must-include” and “allow-list” of relays to be populated on-chain as described in LIP-17. As a reminder:

  • The “must-include list” of relays will be DAO-vetted relays that are considered trustworthy, well-operated, and reliable.

  • The “allow list” of relays will be DAO-approved, but perhaps less well-known or battle-tested relays which can be trialled prior to inclusion in the “must-include list”.

Per the Block Proposer policy, Node Operators will be required to use as many of the “must-include list” relays as possible, allowing for flexibility as determined by each Node Operator’s internal risk management and legal requirements.

As additional monitoring capabilities are developed over time, analysis will be undertaken and presented to the DAO to determine whether Lido Node Operators are finding an appropriate equilibrium between network and protocol health, staker rewards, and Operator specific compliance requirements.

Relay performance, consisting of factors such as timely unblinding of blocks, block inclusion time, accurate bid delivery, etc., is another important element that will need to be considered when information is fully available from the relevant API endpoints. For further information regarding the impact of MEV relays on the network see Attestant’s recent analysis. More detail regarding Lido’s planned implementation for monitoring is available in the Monitoring & Penalties Specification of the Block Proposer Rewards policy.

Relay Provider Background:

During Lido’s call for relay providers, five entities publicly expressed interest for Lido Node Operators to use their relays to propose blocks: Flashbots, bloXroute, Blocknative, Manifold Finance, and Eden.

As of 26/10/22, over the past seven days 63% of Lido block proposals have come through the Flashbots relay, 12% from bloXroute (including 6% from the ethical relay, 5% from the max profit relay, and 1% from the regulated relay), 6% from Eden, 3% from Blocknative, and <1% from Manifold**.

As discussed in the Block Proposer policy, relays suggested for inclusion in these lists should be:

  1. publicly available,
  2. publicly listed & maintained,
  3. open source,
  4. transparent about what (if any) transaction or address filtering they enforce.

In the call for relays thread, each provider had the opportunity to disclose their compliance with these suggested attributes. As a summary:

Flashbots:

  1. publicly available - Yes
  2. publicly listed & maintained - Yes
  3. open source - Yes
  4. transparent about what (if any) transaction or address filtering they enforce - Yes:

Flashbots filters transactions that interact with OFAC sanctioned addresses.

bloXroute:

  1. publicly available - Yes
  2. publicly listed & maintained - Yes
  3. open source – No: Planning to open source in 1-2 months (note: per the team this is still on track).
  4. transparent about what (if any) transaction or address filtering they enforce - Yes:

a. Max profit: Relay that provide blocks with all available transactions/bundles with no filtering.

b. Ethical: Relay that provide blocks that have no frontrunning bundles. bloXroute tries its best to filter out MEV bundles that execute strategies like generalized frontrunning and sandwiching.

c. Regulated: Relay that provide blocks with all available transactions/bundles except the ones sent from/to wallet addresses that are sanctioned by OFAC.

Blocknative:

  1. publicly available - Yes
  2. publicly listed & maintained - Yes
  3. open source - Yes
  4. transparent about what (if any) transaction or address filtering they enforce. - Yes:

Blocknative’s initial relay service will serve blocks that meet the OFAC compliance requirements of our early customers. We are monitoring the ecosystem in real time and will adjust to less filtering as soon as it makes sense.

Manifold:

  1. publicly available - Yes
  2. publicly listed & maintained - Yes
  3. open source - No
  4. transparent about what (if any) transaction or address filtering they enforce. - Yes:

Manifold does not filter OFAC sanctioned addresses (more detail: Here)

Eden:

  1. publicly available - Yes
  2. publicly listed & maintained - Yes
  3. open source - Yes
  4. transparent about what (if any) transaction or address filtering they enforce. - Yes:

a. Eden Relay provides blocks with transactions/bundles prepared by our own Eden Block Builder that is set up to always ensure regulatory compliance and respect all applicable regulations, including the OFAC sanction list (meaning no wallet addresses allowed through if they have been registered on the OFAC list).

b. All third party builders supplying transactions/bundles will need to accept ToS that disallow them to share transactions/bundles that have been sanctioned by OFAC (This will be enforced and penalized by removal/blacklisting of block builder if detected).

Although not included in v1 of the Block Proposer Policy, an additional criteria that may be considered for relay providers going forward is whether they allow for external block builders to submit blocks to their relay. Currently the Flashbots, bloXroute, and Eden relays allow for 3rd party builders to submit blocks, Manifold is in the process of re-implementing the feature, and Blocknative is considering what approach to take.

Notable Incidents:

  1. On September 21st, a number of Lido Node Operators missed block proposals due to an issue with the bloXroute max profit relay failing to unblind blocks. Later that day, the bloXroute team put out a public statement disclosing 88 total missed blocks due to the service outage and later repaid validators for each missed slot.
  2. On September 27th, bloXroute disclosed an issue where the ethical relay began receiving and returning bad blocks to validators from an internal block builder running an experimental build strategy that resulted in missed slots. BloXroute remedied this by repaying all missed slots for the bids received and added functionality to ensure all bloXroute relays simulate block submissions regardless of the source.
  3. On October 8th, Eden’s API reported the wrong values for certain blocks due to a bug in the reference builder implementation. The team publicly shared a fix for the issue and repaid the missed fee to the validator.
  4. ** On October 15th, an issue was discovered with the Manifold relay where block rewards were not matching the declared reward in the blinded block. This was due to a 3rd party block builder noticing that the verification of block rewards for blocks submitted to the relay had been disabled, allowing for blocks to be submitted with wrongly declared block rewards. Details of the incident were later shared in a post mortem report through the Manifold Finance telegram channel on October 22nd: Postmortem of incident on 2022-10-15 - HackMD. The Manifold team has notified Lido that they intend to reimburse validators for the missed rewards. Given pending reimbursement and the magnitude of the incident, most Lido Node Operators have disabled the Manifold relay for the time being.

Relay Proposals:

Given conversations with Lido Node operators and the prior context, including the suggested requirements of allowed relays, relay provider’s introductions to the Lido community, and the incidents discussed above, the following relays are suggested for the initial “must-include” and “allow” lists.

Must-Include List:

  • Blocknative
  • bloXroute
  • Eden
  • Flashbots

Allow List:

  • Manifold Finance

The proposal to include Manifold’s relay in the “Allow-list” reflects discussions with Lido Node Operators, pending resolution, and additional time needed to stress-test the relay post mitigation.

After appropriate time for discussion over the next week, the following lists will be put up for snapshot vote, followed by a vote through Aragon to populate the on-chain contract deployed under LIP-17.

Placement on either the “must-include” or “allow” lists are not final. Going forward these lists will be reviewed and updated on an at-least yearly basis, though given the early-stage of MEV-Boost infrastructure, the list will likely be reviewed before the end of Q1’23.

We invite all Lido stakeholders and the Ethereum community to weigh in on these proposed lists. Additional updates have also been proposed in the Lido on Ethereum Block Proposer Rewards Policy v2.0 where we invite broader discussion on the protocol’s MEV related policies.

11 Likes
2

Nice work. One question I did have was what is the process for changing the relay lists?

1 Like
3

I am uneasy about Manifold on the allow list. From the 2.0 policy:

that any and all communications are made timely and professionally.

and

Relays which exhibit poor or otherwise unacceptable communication and especially performance (such as, but not limited to, inability to performantly and timely register validators, inability to performantly and timely send/receive bids and payloads, not verify bids, not simulate payload validity etc.), and/or fail to conform with the expectations outlined above with regards to incident management (as determined by the DAO and Node Operator communities) will be subject to removal from the Lido-approved relay lists.

Following Manifold’s issues, they behaved unprofessionally on Twitter through their official account. Samples:

“There are a litany of bugs with the relay implementation, however you continue spread lies and disinformation about our relay. This isnt a bug in our relay, flashbots has blocked us on all accounts so how can you purport to be for the community?”
https://twitter.com/metachris/status/1581329826111455234

This ties also into AGPL v3 and open-sourcing, more on that further down.

and

Blame Flashbots for having a poorly implemented API specification.
https://twitter.com/foldfinance/status/1581472268982841344?s=20&t=bDqu61pFsaNqEqmLCAlXIg

I find the communication 5x more damning than the actual bug / issue that arose. What is more, this was not addressed at all in their post mortem, even after they received feedback that it should be included when they circulated an early draft.

As for open-source: AGPL v3 requires open-sourcing derivative works. Not only is the Manifold relay not open-source, it also has no published plans to become open-source. Manifold’s assertion that the relay code was “full of bugs” was challenged:

I’d love to see even a single bug report from you. Also, you are using our open source relay, but not making your fork public. Isn’t that in violation of the AGPL license?

They did not respond to this challenge.

“If it’s not open-source, it is not secure and it is not Ethereum”. This is true for all projects on Ethereum, and especially for trusted actors like relays.

Until and unless Manifold address the communications and FOSS issues, and explain what concrete steps they are taking to ensure professional communications in future - this likely involves removing Sam from their official Twitter account - I don’t see how they are a good candidate for inclusion in the allow list.

2 Likes
4

Good question! Generally the same process we follow for any on-chain governance thing that’s not Easy Track’d (I think we should ET it eventually, but I’m not sure how fast we could make that happen @TheDZhon @kadmil ?).

Normal governance process (current)

  • Forum post (anyone can post, an NO, a community member, a relay operator) “hey guys we should try out this relay, here are its details” and a discussion follows (2-3 days)
  • Snapshot vote (“Should we add “Bob the Relayor” to the “Allow list”?”`)
  • On-chain (Aragon) vote “add Bob the Relayor relay to Allow list”
  • On-chain execution of above (if it passes)

repeat the process if we want to move a relay from the “allow list” to the “must-include list” or add another new relay to list.

Estimated time to get a relay added/removed/moved from the list(s): 1-2 weeks.

Easy Track process

To set up Easy Track

  • Have a forum discussion about the DAO delegating “default” operational management of the Relay Lists to authorized groups (e.g. a committee, the LNOSG, any node operator, some combination, etc.)
  • Vote via Snapshot and then (if it passes), on-chain vote to give the delegated-to parties ability to manage relay lists via easy track

To manage relays

  • Forum post (anyone can post, an NO, a community member, a relay operator) “hey guys we should try out this relay, here are its details” and a discussion follows (2-3 days) (this step could be optional or sped up, given that Easy Track allows for very easy to use objections)
  • delegates can propose on-chain-effect vote to add relay to optional list (can be vetoed via easy-track objection)
  • if a relay has been in “allow list” for X period of time, delegates can propose on-chain-effect vote to move to “must include” list (again, can be objected to)

Estimated time to get a relay added/removed/moved: ~3-6 days (happy flow), or 1-2 weeks (after an objection and reversion to normal governance process).

5

I really like the proposal, clearly thought through & articulated reasoning.
With regards to Manifold, I share the concern with regards to communications. I would personally prefer a much more professional approach and until seen some proof for this, not engage.
But given the importance of as much competition on all stages of the “MEV supply chain” and the fact that they do not censor their blocks, I hope that putting them “on probation” via the allow-list is a sensible compromise that leaves the door open to professionalize. To me, that is what the allow-list is for, so let’s see if it has the desired effects.

4 Likes
6

Although I am sure most already know and maybe LIDO has already established contact, I’d suggest to actively engage with new relays/builders in this early days of the market evolving, in order to help possible smaller players gain traction quickly and ideally enriching the whole ecosystem.
E.g. https://relayooor.wtf/ by https://twitter.com/builder0x69 (some details First 10,000 blocks and new features | by builder0x69 | Oct, 2022 | Medium ) - started Oct26th

1 Like
7

Thank you for this input and great that we’re having this discussion openly. I hope that manifold replies here so that we can constructively move forward on this.

I think the license issue is important but murky. Manifold have raised an interesting point re: the CLA that Flashbots has employed and apparent reticence by FB to accept Manifold to contribute upstream directly. One can argue that just pushing their changes to a forked repo is probably enough here to “comply”, and I think they should do it (and if they have and we’ve missed it we should update accordingly), but the behavior by FB here is also kinda weird. However I don’t think it’s productive to litigate license compliance , so instead what if we do something like discuss what we think acceptable way forward may be here:

  • Open sourcing of current derivative changes to fb relay & builder-geth
  • Clear plan for open sourcing relay implementation (if own is to be built)
  • Update on external builder integration and reward simulation

I agree with your points re: communication and speed of action.

That said, I believe that the over-arching concern here is “what’s best for the ecosystem” and having an additional relay is net positive, because ultimately what we want is relay and builder diversity.

Freddy makes a good point that we can see the “allow list” usage as an expression of dissatisfaction with the status quo but also allowing for a speedier path to amelioration and best overall outcome. While there’s room for improvement, we’re still early days here and I’d suggest that we consider working through this stuff optimistically rather than conservatively.

1 Like
8

Definitely! We have reached out to relayooor and 2 other relay operators who have expressed a desire to run independent/neutral relays and figuring out how fast we can get them connected to testnet validators. If anyone else is bumping into relay operators, please ask them to post in the Lido on Ethereum: Call for Relay Providers topic!

9

It’s a good thing to be Easy-Track-ed, indeed. The contract was designed with the ability to be integrated into the Easy Track governance process.

Speaking about the capacity of some better-to-be-involved contributors, I may suggest putting it on the table for Q1 '23. As far as I know, we expect huge deployments of the regular DAO payroll committees (having the higher priority, basically).

1 Like
10

It looks like we have rough consensus here we all agree with. I’d like to see as much relays on the must-include list as possible, on the sole principle that community is decently sure they won’t steal or lose MEV from our stakers. I think that Manifold has to do a few things to get to the “decently sure” level: compensate the loss incurred and address the comms issue.

1 Like
11

Detailed response for the Lido Community

This response is in two main parts:

  • a detailed summary with some commentary
  • a personal statement

FOSS Relay

We have a monorepo, our entire infrastructure is designed, built, deployed and managed that way. Though this argument is a strawman: just because the code is open source does not mean that it is being ran 1:1 (i.e. it is not being ran modified, etc).

Post Mortem response

This issue in response was entirely my fault, I had taken a work break after devcon and was traveling. I did not understand the issue at the time, and had mistook it for a previous service disruption that had occurred earlier in the week which was not comparable in magnitude. We had already provided a summary to the operations team at Lido at this event (in which we voluntarily disclosed to them). I confused it for a repeat of the same event and dismissed the need for a timely public statement.

If it is not clear, we are providing 100% restitution. We are currently working with authorities and an external investigator in this matter. The restitution can be coordinated with Lido’s operations team. Note that this does not mean we admit fault in this manner.

Relay and Network Reliability

Building a reliable, robust service often means building something that can keep working when some parts fail. A service where not every feature is available is often better than one that’s entirely offline.

Doing this in a meaningful way is not obvious.

The usual response is to hire more engineering, more support, and even more managers. Error handling, or making components that can recover from faults, often feels like the option of last resort—especially in blockchain networks.

The usual response to error handling is optimism. Unfortunately, the other choices aren’t exactly clear, and often difficult to choose from too. If you have two services, what do you do when one of them is offline: Try again later? Give up entirely? Or just ignore it and hope the problem goes away?

This was my thinking in making those frustrated comments. I no longer interact on Twitter either personally or professionally, nor care to.

Communications and Engagement

We have been considering for more than two months how best to approach this role, and found someone we think will be invaluable to the entire team. We suggest a community call participation at the next regularly scheduled Lido community event or if desired we can do a one-off meeting at whatever time the community finds desirable to have such a call.

Summary

  • We have created a new dedicated Statuspage strictly for SecureRpc

  • Create a dedicated Ethereum Ecosystem RSS aggregation and notification alert page for defect response and alerts. This is meant to provide notifications to node operators of any potential issue affecting them as it relates to software they are running themselves.

  • We have recruited a new business and community lead for coordinating and facilitating our engagements across communities and different projects.

  • Attend and interact with the Lido community during a live community call/gathering within the next 7-14 days. A time that is scheduled can be attended by us.

  • 100% Restitution, within the next 7-14 days, as was originally implied in the post mortem.

  • In the coming days we will be making public a few on-chain and off chain solutions that should provide not only improved recovery it will also provide validators with a type of insurance bond protecting both potential losses as well as surplus reward payouts to them.

  • Manifold is also (and has been) ready to operate as a node operator as well, we see no meaningful reason for the relay/operator separation.

  • A 3rd party forensic auditor has been engaged and is helping assist authorities in our internal investigation.

Personal Remarks

This was an offensive attack. An attack on a service that most Ethereum developers will never even work on. This service also happens to offer Ethereum a non-censoring transaction relay that is operating against a potential counter party that is primarily concerned with maintaining its international rules-based system by financial controls for purposes of managing the global economy. We consider this event as strong evidence of battle-space preparation. We know the potential outcomes, we have seen the tactics being used thus far.

Cheers,

Sam

4 Likes
12

Also, Manifold has run on average the most profitable relay since the merge till 10/22/2022, this is from a 3rd party that will be publishing a rigorous analysis of relay performance:

IMAGE 2022-10-31 19:27:21
IMAGE 2022-10-31 19:27:211280×842 49.8 KB

2 Likes
13

Is it this: https://status.manifoldfinance.com/?

Would it be valuable in your mind to have an open-source repo to serve as a devops primer or knowledge base to collect learnings & best practices from various MEV relay runners + to help prospective relay operators bootstrap & get started?

This is good news and congrats. And yes that is a great idea. I would also suggest inviting the ethstaker community to that call or perhaps holding a separate one with them, to the extent the Lido community would prefer to keep it within Lido.

Is the idea behind this that it would be an aggregated feed covering all known MEV relays, public RPC endpoints, and execution & consensus client updates? I like the idea and would like something like that (more surprised it doesn’t exist already)

Funded presumably out of treasury? Or would it be via a portion of your builder’s profits?

Do you mean relay operator, builder, proposer, or all of those things?

Would be eager to see results of their investigation once finished, especially if the Manifold team is comfortable publishing in its entirety.

Same comment as above, would be eager to see if the 3P auditor can corroborate this, in which case alarm bells should be going off for all other relay operators (particularly the “unregulated” ones).

14

As someone who runs blockchain services: Troubleshoot the failing service, find root cause, fix whatever went wrong, and think on how to improve things so it doesn’t fail again. If that’s an option. Sometimes it’s an upstream github issue to fix a bug.

I’m not quite sure what the message is, here. “Ops is hard, sometimes it’s best to just do nothing”? Or, “we don’t have enough staff”? Both would require action and a change in operational behavior, in my opinion.

I am uneasy with this answer. This sounds entirely dismissive: “No, we won’t open-source; and anyway, if we did, what guarantee do you have that we’d actually run that code?”

No guarantee but your word. No evidence but that of an open-source repo and issues and PRs opened upstream. Relays are trusted entities, more so than other ecosystem participants. That requires behavior that goes above and beyond when it comes to transparency, community engagement and good faith.

15

We are weird humans here at Flashbots :disappointed: :person_shrugging: :laughing: I’m curious to hear more about this. We are coming back from conference+meeting+rest, with a stronger commitment, better structure, and more time to engage with our community. If there are issues around our free software, I’m happy to get them resolved.

I want to bring back attention to this: What makes a relay trust-worthy? - Relays - The Flashbots Collective

In particular, that after a brief discussion in the PBS developers roundtable in Bogotá I’ve added:

  • In case of missing blocks, does not retroactively pay to the affected validators.

When designing the side-car approach for mev-boost we took into account that getting extra MEV rewards comes with extra risks. I didn’t like when bloxroute paid the affected validators after an incident. I don’t like this to become a thing, because it introduces weird economic dynamics that we haven’t understood yet. If you have thoughts about this, please share them.

1 Like
16

While I agree that the idea of restitution can have weird economic dynamics, I think the current trust assumptions unfortunately skew risk in such a way that restitution is one of the few viable mechanisms for trust to be regained when incidents occur.

I think the risk here is disproportionate to rewards, at least in the current status quo, especially when you consider the damage that might be caused by a mal-performing relay (or set of relays) to the network.

Validators need to implicitly trust relays until such a time that payout is guaranteed (e.g. via enshrined PBS) and/or validators are able to effectively operationally reduce the impact of operational failure of the MEV Boost stack at a relatively quick speed (which to be honest isn’t really feasible currently or practical – features like circuit breakers are still not implemented and even in such a case they’re “local” (i.e. if a relay has an issue then each validator needs to process that issue on its own via issues somehow being communicated across validator and operator sets)).

Especially when taken into consideration with the coming of new and less eponymous relays (which we obviously want to support), consider that something goes wrong with a relay and a lot of validators are affected. What reason would these validators have to re-trust that relay if they are not made whole?

17

I hear you. I know I’m not making an economic argument in here, which might be naive. I still like my point :slight_smile:

Two facts are that there will be bugs and that the reward boost is tempting.

With that, I think that we can make sure we never lose trust by following:

  • Take down the server when a high-severity bug is found, so validators switch to local building and don’t miss slots.
  • Publishes a post-mortem after every incident.

(from https://collective.flashbots.net/t/what-makes-a-relay-trust-worthy/145)

If a team fails to do that, I don’t think it should be re-trusted, ever. In the Lido case, I see that as down-grading from must to allow. Currently, I don’t see a way from must to allow to must again.

I think this will change once the relay monitor is operating, because then we can quantify the risk and the reward of every relay. At that time, maybe getting into the must list means having a high score in the monitor, and it’s no longer subjective.

18

This is deliberately misleading. Manifold counts regular mempool transactions to the proposer feeRecipient as “additional value provided”, using such transactions to inflate the bid value. In one instance, there was a 170 ETH transaction to the proposer, which Manifold counts towards “being the most profitable relay”, but in reality that value would have been part of any mempool block as well, and other relays wouldn’t have counted that as part of the bid value at all.

There is currently ongoing discussions around the block-scoring mechanism, and full balance diff is a possible option, but it’s disingenuous to claim being the most profit relay while comparing inflated numbers (concentrated in only few blocks too).

We welcome upstream contributions, but haven’t seen any attempts to do so from Manifold.

More importantly, Manifold claiming the CLA being an issue is a straw-man argument! Our relay is released under the AGPL license, in the hope it’s useful to others, with the simple requirement to make their changes available to the public again. Manifold has chosen to use our codebase but deliberately rejects to simply make their fork public, which would be enough to satisfy the AGPL license requirements.

2 Likes
19

Sigma Prime will use the “must-include” list of relays.

We do not believe that Ethereum should achieve censorship resistance by pressuring validators to potentially expose themselves to the legal risks of other entities on the network. We believe that Ethereum is capable of specifying and implementing novel forms of cryptography which clearly place the responsibility of transacting on those that are transacting, rather than the block-signing “middle-men” (middle-people). This is not evasion of the law, rather ensuring that individuals are solely responsible for their own actions. This is what we will work towards at Sigma Prime.

We fully support node operators that wish to run only non-censoring relays and it is our wish that Lido makes room for these operators, too. We believe that if Lido wishes to engage with operators that have a high degree of moral integrity, then it should also make allowances for those operators to exercise their own moral judgement. We believe that Lido can survive perfectly well in a world where some operators use exclusively non-censoring relays and it would speak volumes to Lido’s credibility to allow those operators the freedom to operate within their own moral framework.

6 Likes
20

Looking forward to the outcomes of your efforts here. Neutral infrastructure is critical and something we aim to build and your work here will help immensely.

One factor worth bringing up is relay diversity. Relay diversity, like client diversity, is critical to the health of the ethereum ecosystem. It is one thing to be connected to multiple relays - but if they all use the same source code you’re vulnerable to invalid and empty blocks. Source code diversity is critical.

To-date there are two verifiably unique codes, Flashbots and Blocknative’s Dreamboat. For operational redundancy validators should ensure that they have - at a minimum - one relay on Flashbots source code and one on Dreamboat source code hooked up to their MEV-boost.Transparency in this is something we think all validators (and relays) should consider because code diversity ensures operational redundancy.

3 Likes