On October 11, 2023 20 validator slashings related to validators operators by Launchnodes as a part of the Lido protocol were observed. The node operator has shut down the impacted node(s) and is investigating the root cause.
The Node Operator is currently assessing the full impact and working on a plan to bring back the remaining validators together with the assistance of Lido DAO NOM contributors.
More information and a detailed incident report will follow.
Update:
At this time, the situation is stable as the Launchnodes team worked through the night to slowly bring the rest of the validator infrastructure back up. The root cause of the slashing incident is understood, with small details still being investigated.
Slashed validator penalties currently amount to 20.04 ETH and are estimated to increase to 23.06 ETH by time of withdrawal. Infra downtime penalties and overall missed rewards (excl. EL rewards) amount to 5.663 ETH (including slashed validators).
Launchnodes immediately expressed their desire to compensate stakers at the soonest possible time, and so disbursed 25.663 ETH prior to today’s rebase, which means that stakers will see no reduced rewards for the day resulting from the outage and slashing.
Once future penalties and missed rewards for the slashed validators are finalised (i.e. the validators are withdrawn), a final calculation will be made and Launchnodes have indicated that they will also cover this amount resulting in no lost rewards for stETH holders.
The compensation transaction for the day’s reduced rewards can be found here.
A full post-mortem will be posted here in the coming days.
The blog with the post mortem which includes the full incident report by Launchnodes has been published.
Note: In the previous slashing incident, there was a DAO vote to lower the operator’s key limit as a precautionary measure to prevent additional stake being allocated.
In this case, while exploring options and considering functionality introduced in LoP V2, Launchnodes elected to remove an unused key (below their limit at the time, which was 4000 keys) which reset their vetted keys (i.e. validator limit) to their current number of used keys (2582) (details and relevant transaction are in the post mortem).
As a result, there is no current need to set a validator limit for the NO by explicit DAO vote.
EDIT:
Also including the relevant tweets here for posterity https://twitter.com/LidoFinance/status/1712898983998214645
An update on the slashing incident: as at Nov 16, 2023 the 20 validators in question are now withdrawable (and have thus stopped accumulating penalties).
The breakdown of the total penalties and missed rewards can be found in the below-attached image. Actual total penalties have been calculated as the change in balance of the relevant validators as of the slashing epoch for each validator up until the epoch that each became withdrawable. Missed rewards have been calculated in the same manner that they were in the post mortem. The actual amount of penalties was less than the initially projected amount per the post mortem.
The total amount of
| Total Penalties & Missed Rewards (initial estimate) | Total Penalties & Missed Rewards (actual) | Initially Compensated on day of incident | Remainder |
|---|---|---|---|
| 28.677 ETH | 28.463 ETH | 25.663 ETH | 28.463 - 25.663 = 2.8 ETH |
For those who wish to check or re-perform the calculations it can be done here:
Note that penalties for specific duties (attestations, proposals, etc.) are based on calculation and cannot be queried directly from chain data.
As indicated by Launchnodes, they will be compensating stakers for the remainder of this amount (i.e. less the compensation already sent). The compensation will occur by sending the ETH amount to the Lido Execution Layer Rewards Vault.
Many thanks to the whole Lido DAO community and the broader ecosystem of Operators and Stakers.
We really appreciate your support with this event, and with your diligence in helping to ensure that all Lido stakers have been made whole.
The Launchnodes team sent the outstanding balance to the Lido Execution Layer Rewards Vault today:
We’re looking forward to supporting and participating in the projects already underway to help ensure the robustness and security of Lido.
