# Reimbursement for Certus One / Jump Crypto 2023-02-11 Ethereum Validators Incident

**URL:** <https://research.lido.fi/t/reimbursement-for-certus-one-jump-crypto-2023-02-11-ethereum-validators-incident/4297>\
**Category:** Node Operators\
**Created:** [April 5, 2023, 9:37pm UTC](https://research.lido.fi/t/reimbursement-for-certus-one-jump-crypto-2023-02-11-ethereum-validators-incident/4297 "2023-04-05T21:37:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dragonfrond](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/dragonfrond/32/2675_2.png) [@dragonfrond](https://research.lido.fi/u/dragonfrond)\
**Post date:** [April 5, 2023, 9:37pm UTC](https://research.lido.fi/t/reimbursement-for-certus-one-jump-crypto-2023-02-11-ethereum-validators-incident/4297/1 "2023-04-05T21:37:13Z")

</div>

## Overview

The Certus One Ethereum validators, run as part of the Lido liquid staking protocol failed to validate for a period of approximately 6 hours on February 11 2023. This impacted staking rewards on the Lido protocol.

Certus One was aquired by Jump Crypto in 2021.

### Execution Environment

Inside Google Kubernetes Engine (GKE)

- We run one Lighthouse and one Prysm beacon client each connected to their own geth instance.
- A single Vouch instance communicates with the two beacon clients, and mevboost

Outside GKE:

- We have a cluster of three dirk threshold signers that Vouch uses

## Incident Timeline

All times UTC

- 2023-02-11 14:55 `geth-lighthouse` restarted and started resyncing, looked like it was catching up
- 2023-02:-1 21:40 `geth-prsym` instance rebooted, and started resyncing. The Prsym instance lost connection and stopped syncing
- 2023-02-11 21:48 Alerted that `dirk` instances were signing fewer than expected attestations, operator investigated Dirk node and seemed operational (all pods & process were reporting up, but not progressing as expected)
- 2023-02-11 22:30 Notified that Lido validators were down in Telegram room, alerting for down pods did not occur (as processes were all up)
- 2023-02-12 00:00 Operator started investigating, discovering issues on both nodes
- 2023-02-12 05:18 Validators back online

## Root Cause

- The Prysm Beacon client OOMed, restarted and stuck at block 16591518 (but reported up)
- The Lighthouse Beacon chain OOMed earlier in the day but was yet to catch up
- The geth pods were repeatedly OOMing because of `--cache` flag exceeding pod memory limit of 16gb

## Workaround

- Pointed Lighthouse client at geth node running on dedicated node. Lighthouse returned to operation and validator was able to start proposing blocks.
- Reduced `--cache` amount from 8192 to 4096 on geth nodes & increased pod memory limit.
- Adjust termination grace period for pods from 10 seconds to 5 minutes; this allows geth pods to flush sync data to persistent disk.

## Things to improve

- Alerting, escalation and knowledge transfer to wider team, from Telegram and Discord (_complete_).
- Improve alerting on specific components in the stack, rather than final attestation (_complete_).
- Investigate splitting stake across two instances or moving to a hot/warm vouch instance in multiple clusters (_under consideration_).
- Expand the number of Beacon/Consensus nodes that the vouch instance is using, including nodes outside the GKE cluster (_complete_).

## Reimbursement

To compensate for the missed proposals and attestations during the outage, Jump Crypto reimbursed the Lido rewards vault 2.2939 ETH:

`total value lost = 2*penalties + missed rewards` This is because penalties would have been rewards if not incurred, so total value lost is actual penalties plus opportunity cost of not operating correctly.

- 0.9549 in missed rewards
- 0.6695 in penalties

Reimbursement transactions:

- [Ethereum Transaction Hash: 0x7d74f441a3... | Etherscan](https://etherscan.io/tx/0x7d74f441a31d7d91455b5f14554596668d188e8b889415d8e48af97c61ebef5a)
- [Ethereum Transaction Hash: 0xa5c3e52587... | Etherscan](https://etherscan.io/tx/0xa5c3e525878759b5b7fb0ae87e39039ef91f0127b025691d39ca324d58251f04)

---

<div class="post-metadata">

**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\
**Post date:** [April 6, 2023, 6:40am UTC](https://research.lido.fi/t/reimbursement-for-certus-one-jump-crypto-2023-02-11-ethereum-validators-incident/4297/2 "2023-04-06T06:40:14Z")

</div>

Thanks for the transparency and I’m sure the stakers appreciate the reimbursement!