Welcome to our monthly news, where we provide you with updates on our efforts to establish a secure Ethereum environment. Here’s a summary of notable developments since our last update.
General
-
Upgraded firmwares on network devices without interruption (seamless failover).
-
Extensive maintenance of a data center provider led to multiple planned tests of ISP uplink failovers and power outage. All of them were successful.
-
Further improving network security (isolation)
-
Resolving storage bottleneck
-
Successful restore tests (system containing no validator keys)
Holesky
Goodbye, Holesky. You’ve served us well through every deploy, fork, and bug.
Hoodi
-
Fixed an incident where 1000 Hoodi validators was offline
-
Upgraded Validator Ejector to 1.9.0
-
Updated EASY_TRACK config for the Validator Ejector
Mainnet
All scheduled mainnet upgrades for our Lido systems, including fullnodes and validator clients, were successfully completed. Also:
-
Relay list updated as instructed by Lido NOM
-
Issue with sync committee discovered with multi-fullnode setup. Reordering fullnodes in validator-client resolved the issue.
-
Time drift detected and resolved on specific nodes.
-
Resynced a Teku node after it got stuck. No service affected (seamless failover).
-
Issue with upgrading Besu from 25.8.0 to 25.9.0 led to instabilities which forced us to downgrade.
-
Preparing Curated Set intra-DVT key migration
-
Lighthouse stability is still questionable with sync committee participation. Ongoing monitoring and setup adjustments.
-
Improving monitoring and alerting systems
-
Updated Oracle Allowlist
Lido x SSV: Agile Aye-Aye
Completed further tasks instructed by the LIDO team:
-
Temporarily shut down SSV node for investigation due to slashing-incident of another participant
-
Upgraded SSV nodes to v2.3.6
-
Fixed an issue with underperforming SSV node after failover switch due to Besu 25.9.0 upgrade-bug
-
Fixed an issue with low correctness score of SSV node (monitoring and measuring still in progress)
-
Managed our cluster Agile Aye-Aye (we are Cluster Coordinator)
- Monitored health of SSV nodes and DKG services of all cluster members
Lido x SSV: Thoughtful Tiger
Completed further tasks instructed by the LIDO team:
-
Temporarily shut down SSV node for investigation due to slashing-incident of another participant
-
Upgraded SSV nodes to v2.3.6
-
Fixed an issue with underperforming SSV node after failover switch due to Besu 25.9.0 upgrade-bug
-
Fixed an issue with low correctness score of SSV node (monitoring and measuring still in progress)
-
Managed our cluster Thoughtful Tiger (we are Cluster Coordinator)
- Monitored health of SSV nodes and DKG services of all cluster members
Lido x Obol: Frolicsome Frog
- Regular Node Updates
Lido x Obol: Divine Dragon
-
Regular Node Updates
-
Migrate from Lodestar VC to Teku VC to ensure diversity among the cluster