Date: November 24 – December 1, 2025
Affected Service: SSV Distributed Validators
Impact: 1,000 SSV Validators experiencing degraded block proposal performance
Duration: November 24, 2025 – December 1, 2025
Incident Summary
Starting November 24, 2025, Blockscape observed degraded block proposal performance across 1,000 SSV validators following the Ethereum gas limit increase to 60M. The issue manifested as missed block proposals despite healthy attestation performance. After extensive investigation and collaboration with SSV, the root cause was identified as resource constraints in the Teku and Nimbus consensus clients under the increased computational load from higher gas limits.
Timeline of Events
November 24, 2025: Initial performance degradation observed coinciding with gas limit increase to 60M. Monitoring and observation phase initiated. Checked and updated all clients to latest release.
November 27, 2025: Removed secondary client pair from each SSV node to reduce load. Minor improvement observed but missed proposals continued.
November 27, 2025: Reached out to SSV team for trace analysis. SSV traces showed no issues with proposer duties.
November 28, 2025: Multiple remediation steps taken:
- Updated to latest SSV-node binary v2.3.8
- Ran SSV-pulse diagnostic tool – all healthy except Besu max peer count
- Increased Besu max-peers from 25 to 50
- Fixed SSV-node trace exporting issue
Performance remained below baseline despite changes.
December 1, 2025: Identified relay connection timeouts causing proposal delays. Removed high-latency relays from MEV-boost configuration.
Further investigation revealed late block proposals in Teku logs. Increased JVM heap size from 28GB to 32GB.
Applied Nimbus configuration optimizations:
- –subscribe-all-subnets
- –doppelganger-detection=false
- –in-process-validators=false
- –rest-statecache-size=8
- –rest-statecache-ttl=120
December 4, 2025: No more missed proposals. Performance returned to normal operational baseline.
Root Cause Analysis
The root cause was identified as resource constraints in the consensus clients (Teku and Nimbus) triggered by the increased computational requirements following the Ethereum gas limit increase to 60M.
Teku: The JVM heap size of 28GB proved most-likely an insufficient under the higher block processing demands, causing delayed block proposals.
Nimbus: Default configuration parameters were not optimized for high-load SSV operations. The combination of doppelganger detection overhead, in-process validator handling, and inadequate state cache sizing contributed to proposal timing issues.
Contributing Factor: High-latency MEV-boost relay connections caused additional delays in the block proposal pipeline, exacerbating the timing issues.
Impact Assessment
Validators Affected: 1,000 SSV validators
Duration: ~7 days of degraded performance
Missed Proposals: 57
Missed Rewards: 3.0572 ETH
Preventive Measures & Next Steps
Immediate Actions Taken:
Increased Teku JVM heap size to 32GB
Optimized Nimbus configuration for SSV workload
Removed high-latency MEV-boost relays
Updated to latest SSV-node version
Future Preventive Measures:
Implement proactive resource monitoring with alerts for JVM memory pressure
Establish relay latency monitoring
Conclusion
The performance degradation was most-likely caused by insufficient resource allocation in consensus clients that became apparent after the Ethereum gas limit increase to 60M. The increased block sizes and processing requirements exceeded the configured limits for both Teku (JVM heap) and Nimbus (state cache and validator processing). After systematic investigation and configuration adjustments, validator performance has returned to normal.
A payment of 3.0572 ETH has been made by Blockscape to the Lido Execution Layer Rewards vault, to cover rewards missed due to this incident: