Reimbursement for Chorus One October 2021 Downtime

In mid October, Chorus One’s Lido eth2 validators experienced a downtime episode as a consequence of loss of an entire vendor (multiple datacenters) that led to a cascading failure in our storage layer, where slashing protection databases were located. Initially, we attempted to restore the failing storage cluster, but were unable to do so and migrated our nodes to alternative deployment strategy. Regrettably, the attempted restoration of the storage cluster led to an extended period of downtime before nodes were able to resume operation.

As this episode resulted in forgone rewards for stETH holders, we decided to investigate and reimburse the incident as part of our delegator protection pool program.

We identified the attestations missed by all our validators during the period in question (comparing to 100% effectiveness rating) and extrapolated the lost ETH rewards using an average loss of rewards per attestation of 0.0000185 ETH (an approximated average taken from beaconcha.in), which led stETH holders in aggregate to a loss of staking rewards of 33.827287 ETH (or ~$150k at the time of writing).

The calculation data for this analysis is available here.

We propose to reimburse the foregone ETH staking rewards by sending an equivalent amount of stETH to the Lido DAO, which will then burn this amount of stETH, lowering the total supply and effectively increasing the claimable ETH rewards for all existing stETH holders.

10 Likes

This is a great and commendable decision by Chorus One team. Love you folks!

3 Likes

Big props to Chorus One for this initiative. Although there’s no requirement for Node Operators to reimburse stETH holders in these cases, the existence of the delegator protection pool program and extending it to cover Chorus One’s participation in the Lido protocol is continued evidence of their exemplary ecosystem participation and their being a steward of the protocol.

6 Likes

We carried out the above discussed reimbursement to the Lido DAO treasury in 2 transactions:

6 Likes

The vote on the executing the reimbursement has successfully passed & is executed! Again, big props to the Chorus One team for the initiative! :muscle::champagne::desert_island:
https://mainnet.lido.fi/#/lido-dao/0x2e59a20f205bb85a89c53f1936454680651e618e/vote/106

3 Likes