Reimbursement for RockLogic GmbH 2023-06-09 Ethereum Validator Incident

Node Operators
1

Overview

The RockLogic GmbH validators, run as part of the Lido liquid staking protocol, failed to validate for a period of approximately 9 hours 30 minutes on June 9 2023. This impacted staking rewards on the Lido protocol.

Incident Timeframe

9-6-2023 ~ 01:00 AM | ISP goes offline due to hardware failure
9-6-2023 10:29:59 AM | ISP fixed problem

Root Cause

ISP failure

Reimbursement

To compensate for the missed proposals and attestations during the outage, RockLogic GmbH reimbursed the Lido rewards vault 3.1531 ETH.

Reimbursement Calculation

Total value lost = 2x penalties + missed rewards

This is because penalties would have been rewards if not incurred, so total value lost is actual penalties plus opportunity cost of not operating correctly.

Penalties Summary (89 epochs range)
Operator: RockLogic GmbH
Total: 1.0765

Missed Rewards Summary (89 epochs range)
Operator: RockLogic GmbH
Total: 1.0001

Reimbursement Transaction Details

– RockLogic GmbH

7 Likes
2

Thank you for the detailed incident report

2 Likes
3

It was observed that stETH was sent to the EL rewards vault instead of ETH (which means it cannot be automatically handled by the protocol). Thus the reimbursement is not yet finalized.

On-chain ops and NOM contributors will work on a planned proposal to remedy the issue so that the reimbursement can be completed. As it will require an on-chain DAO vote, we will try to line it up with a scheduled omnibus vote.

5 Likes
4

The On-chain vote to complete the reimbursement is over and enacted! :tada:

Results:

  • For 55,279,179.485
  • Against 0.

Thank you for participating!

3 Likes