# Post Mortem: Develp Fleet Fusaka Fork Downtime 2025/12/03

**URL:** https://research.lido.fi/t/post-mortem-develp-fleet-fusaka-fork-downtime-2025-12-03/10967
**Category:** Node Operators
**Created:** [December 6, 2025, 12:26am UTC](https://research.lido.fi/t/post-mortem-develp-fleet-fusaka-fork-downtime-2025-12-03/10967 "2025-12-06T00:26:03Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![Develp.Devops](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/develp.devops/32/6492_2.png) [@Develp.Devops](https://research.lido.fi/u/Develp.Devops)
#### Post date: [December 6, 2025, 12:26am UTC](https://research.lido.fi/t/post-mortem-develp-fleet-fusaka-fork-downtime-2025-12-03/10967/1 "2025-12-06T00:26:03Z")

</div>

**Name:** Fusaka Fork Downtime  
**Date:** 2025/12/03  
**Impacted servers:** All hosts using Nethermind client  
**Impacted services:** Nethermind, nimbus-eth2

## Summary

On **2025/12/03** after Fusaka fork on epoch `411392`(`21:49:11` UTC) the LIDO fleet consensus nodes attached to Nethermind execution layer client experienced desynchronisation due to receiving fake invalid blocks from the EL clients.

 ![2025-12-05_21-02](https://europe1.discourse-cdn.com/flex013/uploads/lido/original/2X/1/118971bb1c29849fa0aeed66daf2a491c882253c.png)

Nodes attached to Go-Ethereum(Geth) execution layer client experienced no issues and DevOps engineers switched affected validator clients to use our available spare Geth nodes. Nearly full recovery was achieved after 52 minutes.

 ![image](https://europe1.discourse-cdn.com/flex013/uploads/lido/original/2X/a/ad24a032eb858747618497b1490a3b892baf9489.png)

In addition during that window two proposals were missed:

- [https://light-mainnet.beaconcha.in/slot/13164707](https://light-mainnet.beaconcha.in/slot/13164707)
- [https://light-mainnet.beaconcha.in/slot/13164592](https://light-mainnet.beaconcha.in/slot/13164592)

## Glossary

- **[execution layer node](https://nimbus.guide/execution-client.html)** - Nethermind or Geth client used, further referred to as **EL**.
- **[beacon node](https://nimbus.guide/quick-start.html)** - Nimbus client running, further referred to as **BN**.
- **[validator client](https://nimbus.guide/validator-client.html)** - Nimbus client providing signatures to **BN** , further called **VC**.

## Timeline (UTC)

- **21:49** - Fusaka fork activates on `411392` epoch.
- **21:50** - **IMPACT** : DevOps team notices first `InvalidBlockException` in Nethermind logs.
- **21:51** - Team notices spike in **BN** sync distance on multiple nodes.
- **21:55** - Lido team is informed about synchronization issues on Develp fleet.
- **21:59** - Team confirms that **VC** version being one hotfix version below **BN** is fine.
- **22:02** - Attempts to restart both **EL** and **BN** nodes do not yield any results.
- **22:09** - Team starts switching affected **VCs** to spare hosts running Geth **EL**.
- **22:21** - Recover is seen from first node switched to spares with Geth **ELs**.
- **22:32** - **RECOVERY** : Last remaining impacted **VC** shows recovery and startes attesting.
- **22:46** - Investigation of **BN** and **EL** logs by Nimbus team begins.
- **23:00** - Nimbus team confirms Nethermind **EL** marked a canonical block as fake invalid.

## Details

At `21:50:50` UTC the first fake invalid block was returned from Nethermind EL for canonical block [13164552](https://light-mainnet.beaconcha.in/slot/13164552):

```auto
INF 2025-12-03 21:50:50.866+00:00 execution payload invalid from EL client newPayload topics="gossip_blocks" executionPayloadStatus=INVALID executionPayload="(parent_hash: 46339486, fee_recipient: 0xdadb0d80178819f2319190d340ce9a924f783711, state_root: 8182db49, receipts_root: c9a20f42, prev_randao: 40515cc6, block_number: 23935699, gas_limit: 60000000, gas_used: 59973609, timestamp: 1764798647, extra_data: BuilderNet (Flashbots), base_fee_per_gas: 25588596, block_hash: 401f3ba5, num_transactions: 345, num_withdrawals: 16, blob_gas_used: 786432, excess_blob_gas: 1572864)" blck="(blck: (slot: 13164552, proposer_index: 2060282, parent_root: b0eac6a9, state_root: 71a2c3ec, eth1data: (deposit_root: 2ebc563c, deposit_count: 2045305, block_hash: 0958d835), graffiti: , proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 8, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 478, block_number: 23935699, block_hash: 0x401f3ba5d0c8240fb889981a8d80f4744409d21052a53df471c922beef54da68, parent_hash: 0x4633948676924db21b27780428aa7e3e7ca52cb43da85ab2df164a76c4311c61, fee_recipient: 0xdadb0d80178819f2319190d340ce9a924f783711, bls_to_execution_changes_len: 0, blob_kzg_commitments_len: 6), signature: 8518eaae)"

```

The Nethermind node produced `InvalidBlockException` errors due to `Block gas limit exceeded`:

```auto
03 Dec 21:50:50 | Encountered exception Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
 Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
03 Dec 21:59:41 | Encountered exception Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
 Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
04 Dec 11:39:49 | Encountered exception Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
 Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded

```

Since restarting both **EL** and **BN** nodes did not help with recovery the DevOps team decided to switch **VCs** to use spare hosts which ran Geth as **EL**. This process required initiating deployments and caused additional 12 minutes of downtime due to [doppelganger detection](https://nimbus.guide/doppelganger-detection.html) being enabled.

Normally restarting the **BN** and **EL** should allow both to recover, since the **BN** would at startup find the current canonical chain and follow it. But in this particular case [a bug in Nethermind](https://github.com/NethermindEth/nethermind/issues/9878) cause it to consistently and incorrectly inform the **BN** that block number [13164552](https://light-mainnet.beaconcha.in/slot/13164552) is invalid. Initially we believed that Nethermind was simply caching this invalid block and returning to that state after a restart. As it turns out allowing the **EL** to re-sync from scratch would also consistently cause the **BN** to exhibiting the same sync issues as right after Mainnet fork:

 ![image](https://europe1.discourse-cdn.com/flex013/uploads/lido/original/2X/0/08ba8b98929bffa6780e5139bb3358e3788ac2f3.png)

In response the team replaced half of the Nethermind nodes with Geth and started syncing them from scratch. Sync completed in ~6 hours which allowed engineers to spread the load more evenly to stabilize the fleet and continue working on more long term solutions and helping Nethermind team with debugging.

### Previous Hoodi Issue

It is worth noting that a similar issue was detected on Hoodi on 2025/11/21 where both **EL** and **BN** abruptly stopped syncing and could not be re-synced:

```auto
INF 2025-11-21 12:38:01.309+00:00 execution payload invalid from EL client newPayload topics="gossip_blocks" executionPayloadStatus=INVALID executionPayload="(parent_hash: f93e32a4, fee_recipient: 0x0420004de65359da2ed522a7a5a2b74245585123, state_root: 0e5bafc4, receipts_root: cc4b0a0d, prev_randao: 938b6973, block_number: 1666323, gas_limit: 60000000, gas_used: 59485286, timestamp: 1763728680, extra_data: reth/v1.9.3/linux, base_fee_per_gas: 993096998, block_hash: 2a59a6a0, num_transactions: 81, num_withdrawals: 16, blob_gas_used: 2752512, excess_blob_gas: 238195599)" blck="(blck: (slot: 1792940, proposer_index: 749660, parent_root: 8c627f6d, state_root: ec89b431, eth1data: (deposit_root: b53d0890, deposit_count: 37025, block_hash: 5adeb26c), graffiti: reth, proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 8, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 455, block_number: 1666323, block_hash: 0x2a59a6a03ff06b081ca56f4ef4a25e37be89ee1259bc23af7da25b95bbd48358, parent_hash: 0xf93e32a425b83537f2f31efc665c0ae82facfc8251374b1df4f574460a72a015, fee_recipient: 0x0420004de65359da2ed522a7a5a2b74245585123, bls_to_execution_changes_len: 0, blob_kzg_commitments_len: 21), signature: 8ce36a9a)"

```

For which a [Discord thread was opened](https://discord.com/channels/629004402170134531/1441444979133251658) and Nethermind team was provided with database dump files but no conclusion was reached other than the suggestion to re-sync the node.

### Nethermind Bug

Currently it has been confirmed based on suggestion of Nethermind team that the [66986e57](https://github.com/NethermindEth/nethermind/commit/66986e575ebbe23afe7f3c68720562572523cb82) commit from `master` branch does resolve the issue and allows us to re-sync the **ELs** successfully. Unfortunately that does not provide the root cause of the issue and how it was resolved.

Because our setup is based on Nix and NixOS our build of Nethermind is non-standard which in the opinion of the client development team is most probably the cause. While it is true that this issue appears to not affect other operators of note, it is not actually clear that that is the real root cause. Considering the issue can be consistently reproduced on releases `1.35.2` and `1.35.3` it should be possible to bisect the commit history and identify the commit that actually fixed the issue.

One possible candidate that would actually increase the probability of Nix being the root cause is the [#9391](https://github.com/NethermindEth/nethermind/pull/9391) PR which upgrades .NET from 9 to 10. If this is indeed the commit which fixed the bug, it could be indeed true that the way `nixpkgs` provide .NET 9 is in some way broken and could have caused this bug. Our team will attempt to identify the fix and possibly root cause.

## Conclusion

Several improvements can be applied to the fleet:

- Debug Nethermind issue of producing fake invalid blocks.
  - [https://github.com/NethermindEth/nethermind/issues/9878](https://github.com/NethermindEth/nethermind/issues/9878)

- Introducing more **EL** client diversity. (In progress)
  - Both increase **BN** to **VC** ratio and use more **EL** types.

- Separate **VCs** from the **BN** and **EL** pairs. (In progress)
- Reduce downtime caused by Doppelganger Detection via dynamic **VC** updates.
  - [https://github.com/status-im/nimbus-eth2/issues/7668](https://github.com/status-im/nimbus-eth2/issues/7668)