Following the incident from April 13, 2023 and the measures we have taken since to prevent something like this to happen again, we want to give you a status update as of today.
This also refers to the public call today which you can see here: Node Operator Community Call #6 - YouTube
SLASHING INCIDENT OVERVIEW
On April 13, 2023, 11 validators operated by RockLogic GmbH were slashed due to the duplication of validator keys in two different active clusters, causing a double vote. This was caused by an unforseeable bug, which since has been eliminated. But, at the time of incident, the full extent of measures to prevent key duplication was not taken, leading to the slashing.
When the slashing event was confirmed within 15 mins of its first occurrence, we prevented further slashings by bringing relevant clusters offline. The failover cluster was brought back online and the remaining keys were all incrementally activated within the next three hours. Both our’s and Prysmatic Labs’ technical investigations confirmed the bug the next day and we made it reproducible for further analysis. Lido released a post mortem that day.
The bug has been identified and fixed by Prysmatic Labs (GH Issue #12281) as of Prysm v4.0.3.
RockLogic GmbH updated the configuration of nodes (doppelganger used uniformly throughout) and key handling guides and issued those on Github.
We are updating monitoring & alerting and procedures & guides, which will be made publicly available soon (ongoing).
We also plan to create automated tests for such cases (not started yet).
As a consequence and further precaution following the slashing, Lido DAO took an on-chain Aragon vote to limit RockLogic keys until full remediation has taken place, which passed. There is also a Lido DAO Snapshot vote for staker compensation using the cover fund still in progress.
We plan a community discussion about which policies to implement for future reactions upon incidents like this, and related ones (eg large outages) from a governance and NO set management perspective. This will start later this week.