Slashing Incident involving Launchnodes Validators - Oct 11, 2023

On October 11, 2023 20 validator slashings related to validators operators by Launchnodes as a part of the Lido protocol were observed. The node operator has shut down the impacted node(s) and is investigating the root cause.

The Node Operator is currently assessing the full impact and working on a plan to bring back the remaining validators together with the assistance of Lido DAO NOM contributors.

More information and a detailed incident report will follow.

https://twitter.com/LidoFinance/status/1712142945783013393

8 Likes

Update:

At this time, the situation is stable as the Launchnodes team worked through the night to slowly bring the rest of the validator infrastructure back up. The root cause of the slashing incident is understood, with small details still being investigated.

Slashed validator penalties currently amount to 20.04 ETH and are estimated to increase to 23.06 ETH by time of withdrawal. Infra downtime penalties and overall missed rewards (excl. EL rewards) amount to 5.663 ETH (including slashed validators).

Launchnodes immediately expressed their desire to compensate stakers at the soonest possible time, and so disbursed 25.663 ETH prior to today’s rebase, which means that stakers will see no reduced rewards for the day resulting from the outage and slashing.

Once future penalties and missed rewards for the slashed validators are finalised (i.e. the validators are withdrawn), a final calculation will be made and Launchnodes have indicated that they will also cover this amount resulting in no lost rewards for stETH holders.

The compensation transaction for the day’s reduced rewards can be found here.

A full post-mortem will be posted here in the coming days.

6 Likes

The blog with the post mortem which includes the full incident report by Launchnodes has been published.

Note: In the previous slashing incident, there was a DAO vote to lower the operator’s key limit as a precautionary measure to prevent additional stake being allocated.

In this case, while exploring options and considering functionality introduced in LoP V2, Launchnodes elected to remove an unused key (below their limit at the time, which was 4000 keys) which reset their vetted keys (i.e. validator limit) to their current number of used keys (2582) (details and relevant transaction are in the post mortem).

As a result, there is no current need to set a validator limit for the NO by explicit DAO vote.

EDIT:
Also including the relevant tweets here for posterity https://twitter.com/LidoFinance/status/1712898983998214645

6 Likes

An update on the slashing incident: as at Nov 16, 2023 the 20 validators in question are now withdrawable (and have thus stopped accumulating penalties).

The breakdown of the total penalties and missed rewards can be found in the below-attached image. Actual total penalties have been calculated as the change in balance of the relevant validators as of the slashing epoch for each validator up until the epoch that each became withdrawable. Missed rewards have been calculated in the same manner that they were in the post mortem. The actual amount of penalties was less than the initially projected amount per the post mortem.

The total amount of

Total Penalties & Missed Rewards (initial estimate) Total Penalties & Missed Rewards (actual) Initially Compensated on day of incident Remainder
28.677 ETH 28.463 ETH 25.663 ETH 28.463 - 25.663 = 2.8 ETH

For those who wish to check or re-perform the calculations it can be done here:

Note that penalties for specific duties (attestations, proposals, etc.) are based on calculation and cannot be queried directly from chain data.

As indicated by Launchnodes, they will be compensating stakers for the remainder of this amount (i.e. less the compensation already sent). The compensation will occur by sending the ETH amount to the Lido Execution Layer Rewards Vault.

2 Likes

Many thanks to the whole Lido DAO community and the broader ecosystem of Operators and Stakers.

We really appreciate your support with this event, and with your diligence in helping to ensure that all Lido stakers have been made whole.

The Launchnodes team sent the outstanding balance to the Lido Execution Layer Rewards Vault today:

We’re looking forward to supporting and participating in the projects already underway to help ensure the robustness and security of Lido.

:pray:

4 Likes

Hi All

We wanted to provide an update on progress and plans. We have been running our nodes in a backup environment while restoring and hardening our bare metal infrastructure in Africa.

The new infrastructure removes key dependencies and risks, and is being assessed for vulnerabilities by a leading web3 security body this week. After taking on board any of their findings and recommendations, we aim to switch back to our original data center as soon as possible, ideally from next week onwards.

We have been running test nodes on the new infrastructure, and are looking forward to being back with production nodes soon too.

In terms of a migration plan, we have very well rehearsed processes for safely moving between environments and avoiding double attestations or other issues. Key steps are included below, with feedback or questions very welcome.

We’ll keep you posted, thank you as always for your support. :pray:

1 Like