Out of order Exits by Consensys node operator

Overview
On 2023-07-17 at approximately 1200 UTC, Consensys Staking erroneously submitted voluntary exit messages for 125 validators that were set up using Lido on Ethereum protocol. After confirmation, we reached out to Lido DAO contributors in the NOM workstream around 1520 UTC and proceeded with root cause analysis and remediation actions.

Impact
“Out of order exits” is the name given to validator exits that take place outside the expected exit request orders as determined by the protocol. This out-of-order exit removed 125 Lido validators from the active staking set, sending the original stake and any residual (after last skimming cycle) reward back to the Execution Layer side of the Lido protocol, where it was re-cycled into the Consensus Layer activation queue. Because of the long entry queue, this meant approximately 39 days of the validators not participating in duties and rewards.

Resolution
Consensys Staking strives to provide the best non-custodial technical support services for Ethereum staking, and we are constantly working on improving those services. In response to this event, we have added additional administrative safeguards to prevent the erroneous exiting of validators. This change is now in our testing environment, and we will not be submitting new validator keys to the Lido protocol until that change reaches production.

Although we are not required to do so under a service level agreement, in the interest in providing the best customer service to other users of the Lido protocol, we intend to submit the amount of 19.732 ETH to compensate stakers for the service downtime due to the 125 validators going through the Ethereum entry queue instead of being in the active validator set.

Our thanks to Lido DAO contributors for their aid in estimating the amount of value lost, which we will deposit to 0x388C818CA8B9251b393131C08a736A67ccB19297 , the “Lido Execution Layer Rewards Vault”. (edited with transaction id: to-be-added)

8 Likes

Thank you Kuhan. I confirm that the number matches calculations performed by NOM and analytics workstream contributors as well (will be made public for transparency soon), and thank you for the transparent comms and offer to compensate stakers.

4 Likes

Thank you Kuhan. As a contributor to the analytics workstream, i confirm that the number matches within our estimation of the effect.
The details on the methodology and exact calculations can be found here:
Exited validators impact calculating methodology

Although it’s worth mentioning, that apart from the effect above, representing rewards lost for the validators in question while their ETH is “unproductive”, which is a direct effect, there is also an indirect effect, caused by additional time for re-entering due to limited capacity of activating new validators.

The valuation of this Queue delaying effect on exiting & reentering Validators hinges upon a variety of assumptions and peculiarities of the way that the entry queue in Ethereum functions.
On top of that, as this effect appears due to the activation queue and stake demand (adjusted to withdrawals) surpassing the capability (represented by churn limit) it should be treated as the realization of risk due to market conditions affecting all entities (stalkers, DAO and NO), but evaluated aiming for full transparency.

6 Likes

Addendum: Here is the transaction hash for the 19 ETH: 0x2fceea7691018aa267106567e384e1d6d8bcf78c77fce60799301b6067910c5e

2 Likes

Can someone direct me as to whom authorized this address to be a vault and was thorough investigation done or even checked ?

Concerned asset owner

1 Like

Hi, this vault is the Lido Execution Layer Rewards Vault. You can check it here or if you want you can verify it on-chain here

5 Likes