Lido on Ethereum: Relay Voting Proposal

[yorickdowne]

I am uneasy about Manifold on the allow list. From the 2.0 policy:

that any and all communications are made timely and professionally.

and

Relays which exhibit poor or otherwise unacceptable communication and especially performance (such as, but not limited to, inability to performantly and timely register validators, inability to performantly and timely send/receive bids and payloads, not verify bids, not simulate payload validity etc.), and/or fail to conform with the expectations outlined above with regards to incident management (as determined by the DAO and Node Operator communities) will be subject to removal from the Lido-approved relay lists.

Following Manifold’s issues, they behaved unprofessionally on Twitter through their official account. Samples:

“There are a litany of bugs with the relay implementation, however you continue spread lies and disinformation about our relay. This isnt a bug in our relay, flashbots has blocked us on all accounts so how can you purport to be for the community?”
https://twitter.com/metachris/status/1581329826111455234

This ties also into AGPL v3 and open-sourcing, more on that further down.

and

Blame Flashbots for having a poorly implemented API specification.
https://twitter.com/foldfinance/status/1581472268982841344?s=20&t=bDqu61pFsaNqEqmLCAlXIg

I find the communication 5x more damning than the actual bug / issue that arose. What is more, this was not addressed at all in their post mortem, even after they received feedback that it should be included when they circulated an early draft.

As for open-source: AGPL v3 requires open-sourcing derivative works. Not only is the Manifold relay not open-source, it also has no published plans to become open-source. Manifold’s assertion that the relay code was “full of bugs” was challenged:

I’d love to see even a single bug report from you. Also, you are using our open source relay, but not making your fork public. Isn’t that in violation of the AGPL license?

They did not respond to this challenge.

“If it’s not open-source, it is not secure and it is not Ethereum”. This is true for all projects on Ethereum, and especially for trusted actors like relays.

Until and unless Manifold address the communications and FOSS issues, and explain what concrete steps they are taking to ensure professional communications in future - this likely involves removing Sam from their official Twitter account - I don’t see how they are a good candidate for inclusion in the allow list.

[Izzy]

Good question! Generally the same process we follow for any on-chain governance thing that’s not Easy Track’d (I think we should ET it eventually, but I’m not sure how fast we could make that happen @TheDZhon @kadmil ?).

Normal governance process (current)

• Forum post (anyone can post, an NO, a community member, a relay operator) “hey guys we should try out this relay, here are its details” and a discussion follows (2-3 days)
• Snapshot vote (“Should we add “Bob the Relayor” to the “Allow list”?”`)
• On-chain (Aragon) vote “add Bob the Relayor relay to Allow list”
• On-chain execution of above (if it passes)

repeat the process if we want to move a relay from the “allow list” to the “must-include list” or add another new relay to list.

Estimated time to get a relay added/removed/moved from the list(s): 1-2 weeks.

Easy Track process

To set up Easy Track

• Have a forum discussion about the DAO delegating “default” operational management of the Relay Lists to authorized groups (e.g. a committee, the LNOSG, any node operator, some combination, etc.)
• Vote via Snapshot and then (if it passes), on-chain vote to give the delegated-to parties ability to manage relay lists via easy track

To manage relays

• Forum post (anyone can post, an NO, a community member, a relay operator) “hey guys we should try out this relay, here are its details” and a discussion follows (2-3 days) (this step could be optional or sped up, given that Easy Track allows for very easy to use objections)
• delegates can propose on-chain-effect vote to add relay to optional list (can be vetoed via easy-track objection)
• if a relay has been in “allow list” for X period of time, delegates can propose on-chain-effect vote to move to “must include” list (again, can be objected to)

Estimated time to get a relay added/removed/moved: ~3-6 days (happy flow), or 1-2 weeks (after an objection and reversion to normal governance process).

[Freddy_Blockdaemon]

I really like the proposal, clearly thought through & articulated reasoning.
With regards to Manifold, I share the concern with regards to communications. I would personally prefer a much more professional approach and until seen some proof for this, not engage.
But given the importance of as much competition on all stages of the “MEV supply chain” and the fact that they do not censor their blocks, I hope that putting them “on probation” via the allow-list is a sensible compromise that leaves the door open to professionalize. To me, that is what the allow-list is for, so let’s see if it has the desired effects.

[Freddy_Blockdaemon]

Although I am sure most already know and maybe LIDO has already established contact, I’d suggest to actively engage with new relays/builders in this early days of the market evolving, in order to help possible smaller players gain traction quickly and ideally enriching the whole ecosystem.
E.g. https://relayooor.wtf/ by https://twitter.com/builder0x69 (some details First 10,000 blocks and new features | by builder0x69 | Oct, 2022 | Medium ) - started Oct26th

[Izzy]

Thank you for this input and great that we’re having this discussion openly. I hope that manifold replies here so that we can constructively move forward on this.

I think the license issue is important but murky. Manifold have raised an interesting point re: the CLA that Flashbots has employed and apparent reticence by FB to accept Manifold to contribute upstream directly. One can argue that just pushing their changes to a forked repo is probably enough here to “comply”, and I think they should do it (and if they have and we’ve missed it we should update accordingly), but the behavior by FB here is also kinda weird. However I don’t think it’s productive to litigate license compliance , so instead what if we do something like discuss what we think acceptable way forward may be here:

• Open sourcing of current derivative changes to fb relay & builder-geth
• Clear plan for open sourcing relay implementation (if own is to be built)
• Update on external builder integration and reward simulation

I agree with your points re: communication and speed of action.

That said, I believe that the over-arching concern here is “what’s best for the ecosystem” and having an additional relay is net positive, because ultimately what we want is relay and builder diversity.

Freddy makes a good point that we can see the “allow list” usage as an expression of dissatisfaction with the status quo but also allowing for a speedier path to amelioration and best overall outcome. While there’s room for improvement, we’re still early days here and I’d suggest that we consider working through this stuff optimistically rather than conservatively.

[Izzy]

Definitely! We have reached out to relayooor and 2 other relay operators who have expressed a desire to run independent/neutral relays and figuring out how fast we can get them connected to testnet validators. If anyone else is bumping into relay operators, please ask them to post in the Lido on Ethereum: Call for Relay Providers topic!

[TheDZhon]

It’s a good thing to be Easy-Track-ed, indeed. The contract was designed with the ability to be integrated into the Easy Track governance process.

Speaking about the capacity of some better-to-be-involved contributors, I may suggest putting it on the table for Q1 '23. As far as I know, we expect huge deployments of the regular DAO payroll committees (having the higher priority, basically).

[vsh]

It looks like we have rough consensus here we all agree with. I’d like to see as much relays on the must-include list as possible, on the sole principle that community is decently sure they won’t steal or lose MEV from our stakers. I think that Manifold has to do a few things to get to the “decently sure” level: compensate the loss incurred and address the comms issue.

[sambacha]

Detailed response for the Lido Community

This response is in two main parts:

• a detailed summary with some commentary
• a personal statement

FOSS Relay

We have a monorepo, our entire infrastructure is designed, built, deployed and managed that way. Though this argument is a strawman: just because the code is open source does not mean that it is being ran 1:1 (i.e. it is not being ran modified, etc).

Post Mortem response

This issue in response was entirely my fault, I had taken a work break after devcon and was traveling. I did not understand the issue at the time, and had mistook it for a previous service disruption that had occurred earlier in the week which was not comparable in magnitude. We had already provided a summary to the operations team at Lido at this event (in which we voluntarily disclosed to them). I confused it for a repeat of the same event and dismissed the need for a timely public statement.

If it is not clear, we are providing 100% restitution. We are currently working with authorities and an external investigator in this matter. The restitution can be coordinated with Lido’s operations team. Note that this does not mean we admit fault in this manner.

Relay and Network Reliability

Building a reliable, robust service often means building something that can keep working when some parts fail. A service where not every feature is available is often better than one that’s entirely offline.

Doing this in a meaningful way is not obvious.

The usual response is to hire more engineering, more support, and even more managers. Error handling, or making components that can recover from faults, often feels like the option of last resort—especially in blockchain networks.

The usual response to error handling is optimism. Unfortunately, the other choices aren’t exactly clear, and often difficult to choose from too. If you have two services, what do you do when one of them is offline: Try again later? Give up entirely? Or just ignore it and hope the problem goes away?

This was my thinking in making those frustrated comments. I no longer interact on Twitter either personally or professionally, nor care to.

Communications and Engagement

We have been considering for more than two months how best to approach this role, and found someone we think will be invaluable to the entire team. We suggest a community call participation at the next regularly scheduled Lido community event or if desired we can do a one-off meeting at whatever time the community finds desirable to have such a call.

Summary

• We have created a new dedicated Statuspage strictly for SecureRpc

• Create a dedicated Ethereum Ecosystem RSS aggregation and notification alert page for defect response and alerts. This is meant to provide notifications to node operators of any potential issue affecting them as it relates to software they are running themselves.

• We have recruited a new business and community lead for coordinating and facilitating our engagements across communities and different projects.

• Attend and interact with the Lido community during a live community call/gathering within the next 7-14 days. A time that is scheduled can be attended by us.

• 100% Restitution, within the next 7-14 days, as was originally implied in the post mortem.

• In the coming days we will be making public a few on-chain and off chain solutions that should provide not only improved recovery it will also provide validators with a type of insurance bond protecting both potential losses as well as surplus reward payouts to them.

• Manifold is also (and has been) ready to operate as a node operator as well, we see no meaningful reason for the relay/operator separation.

• A 3rd party forensic auditor has been engaged and is helping assist authorities in our internal investigation.

Personal Remarks

This was an offensive attack. An attack on a service that most Ethereum developers will never even work on. This service also happens to offer Ethereum a non-censoring transaction relay that is operating against a potential counter party that is primarily concerned with maintaining its international rules-based system by financial controls for purposes of managing the global economy. We consider this event as strong evidence of battle-space preparation. We know the potential outcomes, we have seen the tactics being used thus far.

Cheers,

Sam

[sambacha]

Also, Manifold has run on average the most profitable relay since the merge till 10/22/2022, this is from a 3rd party that will be publishing a rigorous analysis of relay performance:

[randomishwalk]

@sambacha

• We have created a new dedicated Statuspage strictly for SecureRpc

Is it this: https://status.manifoldfinance.com/?

@sambacha

The usual response is to hire more engineering, more support, and even more managers. Error handling, or making components that can recover from faults, often feels like the option of last resort—especially in blockchain networks.

The usual response to error handling is optimism. Unfortunately, the other choices aren’t exactly clear, and often difficult to choose from too. If you have two services, what do you do when one of them is offline: Try again later? Give up entirely? Or just ignore it and hope the problem goes away?

Would it be valuable in your mind to have an open-source repo to serve as a devops primer or knowledge base to collect learnings & best practices from various MEV relay runners + to help prospective relay operators bootstrap & get started?

@sambacha

We have been considering for more than two months how best to approach this role, and found someone we think will be invaluable to the entire team. We suggest a community call participation at the next regularly scheduled Lido community event or if desired we can do a one-off meeting at whatever time the community finds desirable to have such a call.

This is good news and congrats. And yes that is a great idea. I would also suggest inviting the ethstaker community to that call or perhaps holding a separate one with them, to the extent the Lido community would prefer to keep it within Lido.

@sambacha

• Create a dedicated Ethereum Ecosystem RSS aggregation and notification alert page for defect response and alerts. This is meant to provide notifications to node operators of any potential issue affecting them as it relates to software they are running themselves.

Is the idea behind this that it would be an aggregated feed covering all known MEV relays, public RPC endpoints, and execution & consensus client updates? I like the idea and would like something like that (more surprised it doesn’t exist already)

@sambacha

it will also provide validators with a type of insurance bond protecting both potential losses as well as surplus reward payouts to them.

Funded presumably out of treasury? Or would it be via a portion of your builder’s profits?

@sambacha

• Manifold is also (and has been) ready to operate as a node operator as well, we see no meaningful reason for the relay/operator separation.

Do you mean relay operator, builder, proposer, or all of those things?

@sambacha

• A 3rd party forensic auditor has been engaged and is helping assist authorities in our internal investigation.

Would be eager to see results of their investigation once finished, especially if the Manifold team is comfortable publishing in its entirety.

@sambacha

This was an offensive attack. An attack on a service that most Ethereum developers will never even work on. This service also happens to offer Ethereum a non-censoring transaction relay that is operating against a potential counter party that is primarily concerned with maintaining its international rules-based system by financial controls for purposes of managing the global economy. We consider this event as strong evidence of battle-space preparation. We know the potential outcomes, we have seen the tactics being used thus far.

Same comment as above, would be eager to see if the 3P auditor can corroborate this, in which case alarm bells should be going off for all other relay operators (particularly the “unregulated” ones).

[yorickdowne]

@sambacha

If you have two services, what do you do when one of them is offline: Try again later? Give up entirely? Or just ignore it and hope the problem goes away?

As someone who runs blockchain services: Troubleshoot the failing service, find root cause, fix whatever went wrong, and think on how to improve things so it doesn’t fail again. If that’s an option. Sometimes it’s an upstream github issue to fix a bug.

I’m not quite sure what the message is, here. “Ops is hard, sometimes it’s best to just do nothing”? Or, “we don’t have enough staff”? Both would require action and a change in operational behavior, in my opinion.

We have a monorepo, our entire infrastructure is designed, built, deployed and managed that way. Though this argument is a strawman: just because the code is open source does not mean that it is being ran 1:1 (i.e. it is not being ran modified, etc).

I am uneasy with this answer. This sounds entirely dismissive: “No, we won’t open-source; and anyway, if we did, what guarantee do you have that we’d actually run that code?”

No guarantee but your word. No evidence but that of an open-source repo and issues and PRs opened upstream. Relays are trusted entities, more so than other ecosystem participants. That requires behavior that goes above and beyond when it comes to transparency, community engagement and good faith.

[crece-el-rio]

@Izzy

the behavior by FB here is also kinda weird

We are weird humans here at Flashbots I’m curious to hear more about this. We are coming back from conference+meeting+rest, with a stronger commitment, better structure, and more time to engage with our community. If there are issues around our free software, I’m happy to get them resolved.

@sambacha

100% Restitution

I want to bring back attention to this: What makes a relay trust-worthy? - Relays - The Flashbots Collective

In particular, that after a brief discussion in the PBS developers roundtable in Bogotá I’ve added:

• In case of missing blocks, does not retroactively pay to the affected validators.

When designing the side-car approach for mev-boost we took into account that getting extra MEV rewards comes with extra risks. I didn’t like when bloxroute paid the affected validators after an incident. I don’t like this to become a thing, because it introduces weird economic dynamics that we haven’t understood yet. If you have thoughts about this, please share them.

[Izzy]

@crece-el-rio

When designing the side-car approach for mev-boost we took into account that getting extra MEV rewards comes with extra risks. I didn’t like when bloxroute paid the affected validators after an incident. I don’t like this to become a thing, because it introduces weird economic dynamics that we haven’t understood yet. If you have thoughts about this, please share them.

While I agree that the idea of restitution can have weird economic dynamics, I think the current trust assumptions unfortunately skew risk in such a way that restitution is one of the few viable mechanisms for trust to be regained when incidents occur.

I think the risk here is disproportionate to rewards, at least in the current status quo, especially when you consider the damage that might be caused by a mal-performing relay (or set of relays) to the network.

Validators need to implicitly trust relays until such a time that payout is guaranteed (e.g. via enshrined PBS) and/or validators are able to effectively operationally reduce the impact of operational failure of the MEV Boost stack at a relatively quick speed (which to be honest isn’t really feasible currently or practical – features like circuit breakers are still not implemented and even in such a case they’re “local” (i.e. if a relay has an issue then each validator needs to process that issue on its own via issues somehow being communicated across validator and operator sets)).

Especially when taken into consideration with the coming of new and less eponymous relays (which we obviously want to support), consider that something goes wrong with a relay and a lot of validators are affected. What reason would these validators have to re-trust that relay if they are not made whole?

[crece-el-rio]

@Izzy

What reason would these validators have to re-trust that relay if they are not made whole?

I hear you. I know I’m not making an economic argument in here, which might be naive. I still like my point

Two facts are that there will be bugs and that the reward boost is tempting.

With that, I think that we can make sure we never lose trust by following:

• Take down the server when a high-severity bug is found, so validators switch to local building and don’t miss slots.
• Publishes a post-mortem after every incident.

(from What makes a relay trust-worthy? - Relays - The Flashbots Collective)

If a team fails to do that, I don’t think it should be re-trusted, ever. In the Lido case, I see that as down-grading from must to allow. Currently, I don’t see a way from must to allow to must again.

I think this will change once the relay monitor is operating, because then we can quantify the risk and the reward of every relay. At that time, maybe getting into the must list means having a high score in the monitor, and it’s no longer subjective.

[metachris]

@sambacha

Also, Manifold has run on average the most profitable relay since the merge till 10/22/2022, this is from a 3rd party that will be publishing a rigorous analysis of relay performance:

This is deliberately misleading. Manifold counts regular mempool transactions to the proposer feeRecipient as “additional value provided”, using such transactions to inflate the bid value. In one instance, there was a 170 ETH transaction to the proposer, which Manifold counts towards “being the most profitable relay”, but in reality that value would have been part of any mempool block as well, and other relays wouldn’t have counted that as part of the bid value at all.

There is currently ongoing discussions around the block-scoring mechanism, and full balance diff is a possible option, but it’s disingenuous to claim being the most profit relay while comparing inflated numbers (concentrated in only few blocks too).

@Izzy

I think the license issue is important but murky. Manifold have raised an interesting point re: the CLA that Flashbots has employed and apparent reticence by FB to accept Manifold to contribute upstream directly.

We welcome upstream contributions, but haven’t seen any attempts to do so from Manifold.

More importantly, Manifold claiming the CLA being an issue is a straw-man argument! Our relay is released under the AGPL license, in the hope it’s useful to others, with the simple requirement to make their changes available to the public again. Manifold has chosen to use our codebase but deliberately rejects to simply make their fork public, which would be enough to satisfy the AGPL license requirements.

[paulhauner]

Sigma Prime will use the “must-include” list of relays.

We do not believe that Ethereum should achieve censorship resistance by pressuring validators to potentially expose themselves to the legal risks of other entities on the network. We believe that Ethereum is capable of specifying and implementing novel forms of cryptography which clearly place the responsibility of transacting on those that are transacting, rather than the block-signing “middle-men” (middle-people). This is not evasion of the law, rather ensuring that individuals are solely responsible for their own actions. This is what we will work towards at Sigma Prime.

We fully support node operators that wish to run only non-censoring relays and it is our wish that Lido makes room for these operators, too. We believe that if Lido wishes to engage with operators that have a high degree of moral integrity, then it should also make allowances for those operators to exercise their own moral judgement. We believe that Lido can survive perfectly well in a world where some operators use exclusively non-censoring relays and it would speak volumes to Lido’s credibility to allow those operators the freedom to operate within their own moral framework.

[ldelisle_blocknative]

@paulhauner

We do not believe that Ethereum should achieve censorship resistance by pressuring validators to potentially expose themselves to the legal risks of other entities on the network. We believe that Ethereum is capable of specifying and implementing novel forms of cryptography which clearly place the responsibility of transacting on those that are transacting, rather than the block-signing “middle-men” (middle-people). This is not evasion of the law, rather ensuring that individuals are solely responsible for their own actions. This is what we will work towards at Sigma Prime.

Looking forward to the outcomes of your efforts here. Neutral infrastructure is critical and something we aim to build and your work here will help immensely.

@KimonSh

As additional monitoring capabilities are developed over time, analysis will be undertaken and presented to the DAO to determine whether Lido Node Operators are finding an appropriate equilibrium between network and protocol health, staker rewards, and Operator specific compliance requirements.

One factor worth bringing up is relay diversity. Relay diversity, like client diversity, is critical to the health of the ethereum ecosystem. It is one thing to be connected to multiple relays - but if they all use the same source code you’re vulnerable to invalid and empty blocks. Source code diversity is critical.

To-date there are two verifiably unique codes, Flashbots and Blocknative’s Dreamboat. For operational redundancy validators should ensure that they have - at a minimum - one relay on Flashbots source code and one on Dreamboat source code hooked up to their MEV-boost.Transparency in this is something we think all validators (and relays) should consider because code diversity ensures operational redundancy.

[randomishwalk]

@ldelisle_blocknative

To-date there are two verifiably unique codes, Flashbots and Blocknative’s Dreamboat. For operational redundancy validators should ensure that they have - at a minimum - one relay on Flashbots source code and one on Dreamboat source code hooked up to their MEV-boost.Transparency in this is something we think all validators (and relays) should consider because code diversity ensures operational redundancy.

Strong agree. Do you think relay codebase diversity should be held up to the same standard as execution and consensus client diversity? Or should we demand more/less of the MEV relay ecosystem (which is quite small and concentrated at the moment)?

[aidan]

Hi everyone, Aidan from ChainSafe here.

The very limited discussion here on the impact of forcing censorship on a significant portion of the network is highly concerning. ChainSafe stands firmly alongside Sigma Prime in recognizing that this is an unnecessary action that does not contribute or align with the open values of the Ethereum community.

For some Node Operators being compliant is a necessity, and we fully support those operators in doing so. For those who have the choice, we urge you to consider the implications of censorship.

We would like to see further discourse on this topic, with the hope that it will lead to a healthy dialogue surrounding censorship at the protocol layer, and foster further options for validators.