Over the course of the day on 09/27 the bloXroute ethical relay was receiving and returning bad blocks to validators from one of our block builders running an experimental build strategy that resulted in missed slots.
Overview
This builder was improperly committing transactions and caused improperly formatted blocks. Because the relay was not simulating blocks from known builders these blocks were getting sent to validators causing them to miss slots due to proposing blocks with improper receipts or log blooms. A total of 15 slots were missed due to this issue.
Contributing Factors
BloXroute’s relay was not simulating blocks from internal builders and was instead relying on these builders to submit valid blocks.
Resolution
This was resolved as quickly as we were able to identify where the issue was located.
The experimental builder was taken offline and ensured that it was no longer submitting blocks to any of our relays.
All of the validators who missed slot’s were paid for the bids they received from the bloXroute relay.
Additionally:
On 09/30 all of the relays now simulate all block submissions regardless of source.
Timeline
Slot first seen
4790116
Slot last seen
4791561
Additional Request
While transferring funds to our builder to pay for the missed slots the builder considered the transfer of 10 ETH as an MEV tip for the validator and paid that 10 ETH to a validator on slot 4791729.
Payout to Validator:
0xb6b5c290bd9d7f8217defc048a24e0fa62d90dc35880272da12a73a2eee0f628
If it’s possible could the funds be delivered back to the sender of that transaction: 0xea48ba2edefae9e4ddd43ea565aa8b9aa22baf08
Total of 11ETH was sent in error to the validator during this with an additional 1 ETH occurrence as well in slot 4792026
Transfer to builder: 0xacfa882698c1d691d0375a5f851c91d0048de1df1c1f2a67bdf28e4bb4d1b675
Payout to Validator: 0xe6a126d0a95bfd787ef6401a6e85547251d2bae38457ce2dbcbf87c28076c90e
Another dev at bloXroute told me the same thing happened on this slot regarding tip:
Transfer to builder: 0xacfa882698c1d691d0375a5f851c91d0048de1df1c1f2a67bdf28e4bb4d1b675
Payout to Validator: 0xe6a126d0a95bfd787ef6401a6e85547251d2bae38457ce2dbcbf87c28076c90e
First of all, I personally appreciate your participation in the Lido protocol running the ethical relay. Also, it deserves the respect that you decided to provide disclosure details of the incident.
What I need to say here also, that funds previously sent to LidoExecutionLayerRewardsVault via the 0xb6b5c290bd9d7f8217defc048a24e0fa62d90dc35880272da12a73a2eee0f628 tx were re-staked back to the protocol on behalf of all stETH holders (effectively increasing their balance) and there is no practically feasible way to recover the funds directly from the vault (to lower the attack surface, the contract was designed without possibility to withdraw ether funds by anyone except the Lido contract).
Thus, it would be meaningful to consider another sources of refund if DAO members (maybe committees) engaged and decided to back the case.
In both cases where BloXroute relays didn’t perform as expected, the BloXroute team were quick to resolve and to reimburse Lido stakers for the missed proposals. Given this
Especially since the funds were sent in error it makes sense to have a proposal to send the funds back (eg snapshot to approve → if approved include in next available on-chain omnibus).
just try to understand the PBS workflow.
looks like the current PBS workflow is that before the block is added to the slot with a finality state, the builder will send MEV to the validator? so whether the block is finalized or not, VAs proposing the block will receive MEV?
Generally the payment either happens in the last transaction on the block the validator’s fee recipient is set as the fee recipient directly on the block. So as long as the block is accepted the payment will have gone through.
Hi! Sorry, it is not clear to me where to return
1 ETH. Also to 0xea48ba2edefae9e4ddd43ea565aa8b9aa22baf08?
Can’t dig out this data in the transaction history.
P.S.
7.10.2022 BloXroute confirmed that all 11ETH should be returned to address 0xeA48bA2eDefaE9e4ddD43eA565Aa8b9AA22baF08 as described in the proposal
The previous snapshot did not reach a quorum, although it had not many votes against. Perhaps the quorum was not reached due to the intersection in time with the DevCon.
Short clarification about the post & the proposal.
Due to the operation error on the BloXroute’s side (experimental block builder instance sending bad blocks to bloXroute’s ethical relay), extra 11 ETH have been sent to Lido’s Execution Rewards Vault contract. BloXroute team is asking whether Lido can help out with correcting this error. Those funds have been distributed along with other rewards to stETH holders, with 5% fee going to Node Operators & another 5% going to Lido Treasury. The only way Lido DAO can actually help is to spend funds from the Treasury, leaving Lido stakers & NOs better & BloXroute even. Whether to do so or not is for the DAO to decide, so please, leave comments & vote on the proposal!
The snapshot got the quorum & 11 eth transfer is pre-approved by the DAO! It’s to be included in the next Omnibus vote, starting ~2PM UTC Tue, 25 Oct Snapshot
the vote ends Oct 14, 2022, 5:00 PM UTC