Name: Fusaka Fork Downtime
Date: 2025/12/03
Impacted servers: All hosts using Nethermind client
Impacted services: Nethermind, nimbus-eth2
Summary
On 2025/12/03 after Fusaka fork on epoch 411392(21:49:11 UTC) the LIDO fleet consensus nodes attached to Nethermind execution layer client experienced desynchronisation due to receiving fake invalid blocks from the EL clients.
Nodes attached to Go-Ethereum(Geth) execution layer client experienced no issues and DevOps engineers switched affected validator clients to use our available spare Geth nodes. Nearly full recovery was achieved after 52 minutes.
In addition during that window two proposals were missed:
Glossary
- execution layer node - Nethermind or Geth client used, further referred to as EL.
- beacon node - Nimbus client running, further referred to as BN.
- validator client - Nimbus client providing signatures to BN, further called VC.
Timeline (UTC)
- 21:49 - Fusaka fork activates on
411392epoch. - 21:50 - IMPACT: DevOps team notices first
InvalidBlockExceptionin Nethermind logs. - 21:51 - Team notices spike in BN sync distance on multiple nodes.
- 21:55 - Lido team is informed about synchronization issues on Develp fleet.
- 21:59 - Team confirms that VC version being one hotfix version below BN is fine.
- 22:02 - Attempts to restart both EL and BN nodes do not yield any results.
- 22:09 - Team starts switching affected VCs to spare hosts running Geth EL.
- 22:21 - Recover is seen from first node switched to spares with Geth ELs.
- 22:32 - RECOVERY: Last remaining impacted VC shows recovery and startes attesting.
- 22:46 - Investigation of BN and EL logs by Nimbus team begins.
- 23:00 - Nimbus team confirms Nethermind EL marked a canonical block as fake invalid.
Details
At 21:50:50 UTC the first fake invalid block was returned from Nethermind EL for canonical block 13164552:
INF 2025-12-03 21:50:50.866+00:00 execution payload invalid from EL client newPayload topics="gossip_blocks" executionPayloadStatus=INVALID executionPayload="(parent_hash: 46339486, fee_recipient: 0xdadb0d80178819f2319190d340ce9a924f783711, state_root: 8182db49, receipts_root: c9a20f42, prev_randao: 40515cc6, block_number: 23935699, gas_limit: 60000000, gas_used: 59973609, timestamp: 1764798647, extra_data: BuilderNet (Flashbots), base_fee_per_gas: 25588596, block_hash: 401f3ba5, num_transactions: 345, num_withdrawals: 16, blob_gas_used: 786432, excess_blob_gas: 1572864)" blck="(blck: (slot: 13164552, proposer_index: 2060282, parent_root: b0eac6a9, state_root: 71a2c3ec, eth1data: (deposit_root: 2ebc563c, deposit_count: 2045305, block_hash: 0958d835), graffiti: , proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 8, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 478, block_number: 23935699, block_hash: 0x401f3ba5d0c8240fb889981a8d80f4744409d21052a53df471c922beef54da68, parent_hash: 0x4633948676924db21b27780428aa7e3e7ca52cb43da85ab2df164a76c4311c61, fee_recipient: 0xdadb0d80178819f2319190d340ce9a924f783711, bls_to_execution_changes_len: 0, blob_kzg_commitments_len: 6), signature: 8518eaae)"
The Nethermind node produced InvalidBlockException errors due to Block gas limit exceeded:
03 Dec 21:50:50 | Encountered exception Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
03 Dec 21:59:41 | Encountered exception Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
04 Dec 11:39:49 | Encountered exception Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
Nethermind.Blockchain.InvalidBlockException: Transaction 0x59ef86a5fdc7a9fd136201c36b6499538b32887ddfda3f7a572e5b98d4779d1b at index 319 failed with error Block gas limit exceeded
Since restarting both EL and BN nodes did not help with recovery the DevOps team decided to switch VCs to use spare hosts which ran Geth as EL. This process required initiating deployments and caused additional 12 minutes of downtime due to doppelganger detection being enabled.
Normally restarting the BN and EL should allow both to recover, since the BN would at startup find the current canonical chain and follow it. But in this particular case a bug in Nethermind cause it to consistently and incorrectly inform the BN that block number 13164552 is invalid. Initially we believed that Nethermind was simply caching this invalid block and returning to that state after a restart. As it turns out allowing the EL to re-sync from scratch would also consistently cause the BN to exhibiting the same sync issues as right after Mainnet fork:
In response the team replaced half of the Nethermind nodes with Geth and started syncing them from scratch. Sync completed in ~6 hours which allowed engineers to spread the load more evenly to stabilize the fleet and continue working on more long term solutions and helping Nethermind team with debugging.
Previous Hoodi Issue
It is worth noting that a similar issue was detected on Hoodi on 2025/11/21 where both EL and BN abruptly stopped syncing and could not be re-synced:
INF 2025-11-21 12:38:01.309+00:00 execution payload invalid from EL client newPayload topics="gossip_blocks" executionPayloadStatus=INVALID executionPayload="(parent_hash: f93e32a4, fee_recipient: 0x0420004de65359da2ed522a7a5a2b74245585123, state_root: 0e5bafc4, receipts_root: cc4b0a0d, prev_randao: 938b6973, block_number: 1666323, gas_limit: 60000000, gas_used: 59485286, timestamp: 1763728680, extra_data: reth/v1.9.3/linux, base_fee_per_gas: 993096998, block_hash: 2a59a6a0, num_transactions: 81, num_withdrawals: 16, blob_gas_used: 2752512, excess_blob_gas: 238195599)" blck="(blck: (slot: 1792940, proposer_index: 749660, parent_root: 8c627f6d, state_root: ec89b431, eth1data: (deposit_root: b53d0890, deposit_count: 37025, block_hash: 5adeb26c), graffiti: reth, proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 8, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 455, block_number: 1666323, block_hash: 0x2a59a6a03ff06b081ca56f4ef4a25e37be89ee1259bc23af7da25b95bbd48358, parent_hash: 0xf93e32a425b83537f2f31efc665c0ae82facfc8251374b1df4f574460a72a015, fee_recipient: 0x0420004de65359da2ed522a7a5a2b74245585123, bls_to_execution_changes_len: 0, blob_kzg_commitments_len: 21), signature: 8ce36a9a)"
For which a Discord thread was opened and Nethermind team was provided with database dump files but no conclusion was reached other than the suggestion to re-sync the node.
Nethermind Bug
Currently it has been confirmed based on suggestion of Nethermind team that the 66986e57 commit from master branch does resolve the issue and allows us to re-sync the ELs successfully. Unfortunately that does not provide the root cause of the issue and how it was resolved.
Because our setup is based on Nix and NixOS our build of Nethermind is non-standard which in the opinion of the client development team is most probably the cause. While it is true that this issue appears to not affect other operators of note, it is not actually clear that that is the real root cause. Considering the issue can be consistently reproduced on releases 1.35.2 and 1.35.3 it should be possible to bisect the commit history and identify the commit that actually fixed the issue.
One possible candidate that would actually increase the probability of Nix being the root cause is the #9391 PR which upgrades .NET from 9 to 10. If this is indeed the commit which fixed the bug, it could be indeed true that the way nixpkgs provide .NET 9 is in some way broken and could have caused this bug. Our team will attempt to identify the fix and possibly root cause.
Conclusion
Several improvements can be applied to the fleet:
- Debug Nethermind issue of producing fake invalid blocks.
- Introducing more EL client diversity. (In progress)
- Both increase BN to VC ratio and use more EL types.
- Separate VCs from the BN and EL pairs. (In progress)
- Reduce downtime caused by Doppelganger Detection via dynamic VC updates.