[Post Mortem] Staking Router v3: Accounting Oracle & VEBO Incident
Incident Date: 25.07.2026
Summary
Following the SRv3 release, several issues were detected in the Lido Accounting Oracle (AO) and the Validators ExitBus Oracle (VEBO):
-
BLS signature verification degraded in performance across both oracles, delaying report delivery by 3–4 hours within a frame.
-
Some pending deposits were accounted for incorrectly in the AO report, and a 32 ETH TVL drop was submitted as a result for one day.
-
An edge-case related to calculating the number of validators needing to be exited to fulfil withdrawals caused by a zero division error resulted in the possibility of VEBO reports which would require validator exits to not be compiled properly.
Impact
-
No funds were lost or frozen.
-
AO and VEBO report delivery was delayed by 3–6 hours between July 24 and July 28.
-
The AO report submitted on July 25 was incorrect: one in-flight 32 ETH pending deposit was missing from the reported protocol TVL. Reported extrapolated APR came in at 2.04% against 2.15% expected. A manually checked report on July 26 resolved the issue and corrected the figure to an extrapolated 2.29%. The issue never occurred in the following reports.
-
Some VEBO reports between July 26th to July 28th were not produced, resulting in slightly longer than nominal times for stETH withdrawal request finalization, but still below the network average.
Root causes
Performance degradation: SRv3 introduced an updated method for calculating transient balances in the protocol (ETH moving from the Execution Layer to the Consensus Layer and vice versa). This method, while more accurate, requires traversal of the state of all protocol validators and BLS signature verification of pending deposit objects. This required roughly 40,000 BLS signatures to be scanned for every report. The Python library used for BLS signature verification proved unacceptably slow at that volume, and combined with the large entry queue, AO and VEBO reports took 3–4 hours to build.
VEBO: The zero division error was caused by incorrect edge case handling in the VEBO code, which did not account for CMv2 being attached to the protocol while possibly holding no registered node operators with non-zero weight.
AO: The exact cause of a single 32 ETH pending deposit not being included on the AO of July 24th was not established, as the issue cannot be reproduced using historical data, and the logs provided by Oracle holders are insufficient to confirm any of the hypotheses. Our working assumption is a race condition that desynced CL, EL, and KAPI responses, but neither Lido contributors nor external auditors were able to identify or confirm the exact root cause despite the fact that MixBytes team came up with a good assumption regarding KAPI integration – the issue has not resurfaced, and three-way match with logging and matching between the input sources have been introduced, meaning that, should it, pinpointing the root cause should be much easier.
Detection
All three issues were detected independently by Lido on-chain alerting software.
-
The performance issue surfaced through Lido On-chain Alerts, which flagged the initial VEBO report delay and a drop in CL rewards relative to the previous AO report. VEBO report delay detected 17:21 UTC, July 24th. Incorrect AO report variant detected 13:41 UTC, July 25th
-
The TVL drop was also confirmed by Lido On-chain Alerts, though only once the report version carrying the 32 ETH drop reached quorum.
-
The missing VEBO reports were caught by On-chain Alerts overdue alarms. Detected 19:19 UTC, July 26th
Resolution
The performance issue was resolved by updating the Oracle code to verify BLS signatures using a C++ library, which outperforms the previous Python library by several orders of magnitude and has resulted in much faster reporting for both the AO and the VEBO.
The VEBO code was updated to handle the edge case where all Node Operators in CMv2, or all Node Operators within a given operator group in CMv2, carry zero weight.
The AO issue was not resolved explicitly, as the cause of the missing pending deposit remains unidentified. The performance improvements did, however, produce consistent report construction. Invariant and sanity checks were added to the AO code alongside circuit breakers, so that a report with a mismatched number of deposits cannot be submitted. Logging was also extended across the codebase to support future debugging.