# Slashing Incident involving RockLogic GmbH Validators - April 13, 2023 **URL:** \ **Category:** Node Operators\ **Created:** [April 13, 2023, 2:00pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399 "2023-04-13T14:00:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\ **Post date:** [April 13, 2023, 2:00pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/1 "2023-04-13T14:00:42Z")
> On April 13, 2023 DAO Contributors identified 11 slashings related to validators operators by RockLogic GmbH as a part of the Lido protocol and notified the relevant node operator to shut down the impact node(s) and investigate the cause. Contributors are currently working together with the Node Operator to assess the full impact, if any other validators may be affected, and root cause. More information and a detailed incident report will follow. ---
**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\ **Post date:** [April 14, 2023, 6:35pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/2 "2023-04-14T18:35:45Z")
A post mortem with updates has been posted to the Lido blog, including identification of the root cause and an outline for possible next steps. > **[Post Mortem: Lido on Ethereum RockLogic GmbH Slashing Incident](https://blog.lido.fi/loe-rocklogic-gmbh-slashing-incident/)** > > At April 13 a slashing incident occurred affecting 11 validators in the Lido on Ethereum protocol. This post mortem details what occurred and the identified root cause. ---
**Author:** ![ccitizen](https://avatars.discourse-cdn.com/v4/letter/c/4af34b/32.png) [@ccitizen](https://research.lido.fi/u/ccitizen)\ **Post date:** [April 17, 2023, 1:43pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/3 "2023-04-17T13:43:10Z")
Great post mortem, thanks for sharing. I’d suggest that the DAO use the cover fund for reimbursement, as that’s what it’s there for. I don’t like the precedent or assumption of operators reimbursing for losses. The cover fund exists for this purpose and by using it we exemplify its importance, which I believe has been overlooked. ---
**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\ **Post date:** [April 18, 2023, 12:26pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/4 "2023-04-18T12:26:28Z")
Since lowering key limits cannot be done via EasyTrack, I would like to share that NOM contributors are proposing to lower RockLogic’s current key limit (9000) to a number close to their currently active amount (5456), e.g. something like 5800 (given that this on-chain vote would take 3 days to finalize and be enacted). The exact number will be finalized as close to the omnibus vote launch as possible. This would effectively limit new stake distribution to RockLogic until a) remediation of internal processes and setups can occur and be communicated, and b) a further action is discussed and voted on by the DAO. EDIT: The omni bus vote including the above proposal is live here: [Lido DAO Voting UI](https://vote.lido.fi/vote/154) ---
**Author:** ![stefa2k](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/stefa2k/32/6940_2.png) [@stefa2k](https://research.lido.fi/u/stefa2k)\ **Post date:** [April 18, 2023, 2:59pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/5 "2023-04-18T14:59:49Z")
We sincerely apologize for the incident and the damage it has caused to all those directly involved. We outline more of the details of the bug, as well as a way to reproduce it & our learnings from it in the following incident report: > **[RockLogic-Statement-LIDO-Incident-13-April-2023.pdf](https://rocklogic.at/wp-content/uploads/2023/04/RockLogic-Statement-LIDO-Incident-13-April-2023.pdf)** > > 349.39 KB If you have any further questions concerning this incident please feel free to contact us at any time. ---
**Author:** ![zuzu\_eeka](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/zuzu_eeka/32/1157_2.png) [@zuzu\_eeka](https://research.lido.fi/u/zuzu_eeka)\ **Post date:** [April 18, 2023, 4:02pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/6 "2023-04-18T16:02:15Z")
Aragon vote is started [Lido DAO Voting UI](https://vote.lido.fi/vote/154) The main phase will last for 48 hours. Please cast your votes ---
**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\ **Post date:** [April 18, 2023, 4:28pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/7 "2023-04-18T16:28:23Z")
Thanks for providing additional details and, importantly, the reproduction instructions. As a follow-up, it seems that the Prysmatic Labs team has been working hard on this confirmed issue already and it appears as closed in their github ([keystore: Deleting keys via keymanager API may not fully delete keys · Issue #12281 · prysmaticlabs/prysm · GitHub](https://github.com/prysmaticlabs/prysm/issues/12281)), so I hope it’s included in the next release. ---
**Author:** ![stefa2k](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/stefa2k/32/6940_2.png) [@stefa2k](https://research.lido.fi/u/stefa2k)\ **Post date:** [April 19, 2023, 7:59am UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/8 "2023-04-19T07:59:27Z")
We would like to request that the 11 slashed validators be reimbursed by the cover fund for reimbursement. We think that this incident qualifies to the core purpose of this fund, strengthening how important it is to have an instrument like this to mitigate risk. > **[20230419\_Insurance\_Covering\_Request.pdf](https://rocklogic.at/wp-content/uploads/2023/04/20230419_Insurance_Covering_Request.pdf)** > > 109.38 KB ---
**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\ **Post date:** [April 19, 2023, 8:36pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/10 "2023-04-19T20:36:57Z")
The below is a summary for purposes of pointing the upcoming snapshot vote to. * * * #### Incident Summary On April 13, 2023, 11 validators operated by RockLogic GmbH were slashed due to the duplication of validator keys in two different active clusters, causing a double vote. Lido DAO contributors alerted the RockLogic team to the slashings and RockLogic subsequently brought the affected cluster offline to mitigate potential further risk. The root cause was identified by the RockLogic team and the remaining validators were steadily brought back online. The estimated impact on stETH holders, from the time of the incident until the validators fully exit and are automatically withdrawn from the network on May 20th, is estimated to be ~13.77 ETH. For more detail regarding the incident’s root cause, timeline, impact, and response, please see the [Lido post mortem](https://blog.lido.fi/loe-rocklogic-gmbh-slashing-incident/) and [RockLogic post mortem](https://rocklogic.at/wp-content/uploads/2023/04/RockLogic-Statement-LIDO-Incident-13-April-2023.pdf). #### Staker Compensation On April 19, 2023, RockLogic GmbH[requested](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/8)that the Lido DAO utilize the [slashing cover fund](https://etherscan.io/address/0x8B3f33234ABD88493c0Cd28De33D583B70beDe35#tokentxns)to provide compensation to stakers (more [detail regarding the cover fund](https://research.lido.fi/t/redirecting-incoming-revenue-stream-from-insurance-fund-to-dao-treasury/2528)). The actual amount of compensation will be determined following the full exit and withdrawal of the 11 validators from the network, but has been estimated to be close to ~13.77 ETH. #### Next Steps If there’s no objections, I propose that a snapshot vote commence on Apr 20 (and run until Apr 27th). DAO participants will vote whether the slashing cover fund should be utilized to source compensation for this incident. If the snapshot vote passes, an on-chain Aragon vote will be held after May 20th (the date the validators will be withdrawn) to proceed with on-chain execution in order to compensate stETH holders via usage of the slashing cover fund. Lido DAO and community members are encouraged to voice their opinions regarding this matter in this thread. ---
**Author:** ![kadmil](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/kadmil/32/1958_2.png) [@kadmil](https://research.lido.fi/u/kadmil)\ **Post date:** [April 19, 2023, 8:40pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/11 "2023-04-19T20:40:02Z")
Would fully support the issue going to the snapshot as soon as possible, thank you for posting! ---
**Author:** ![stefa2k](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/stefa2k/32/6940_2.png) [@stefa2k](https://research.lido.fi/u/stefa2k)\ **Post date:** [April 20, 2023, 2:36pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/12 "2023-04-20T14:36:08Z")
[**Slashing Incident involving RockLogic GmbH Validators - April 13, 2023**](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399) **Measures Already Implemented and Further Action Points** **As of April 20, 2023** Following the incident from April 13, 2023 we took several actions to understand its causes and to prevent them from happening ever again. Here is an overview of 1) what has been carried out so far, 2) what we are currently working on and will finalise in the next days and 3) what we plan for the coming weeks. We want to share this with the community and are happy for your feedback! 1. ALREADY DONE - Reproduced the bug causing the failure and shared this information with the community (see earlier statement) - Expanded internal monitoring - Security checks of client configurations (eg. if doppelgänger is enabled) - Documented and clear instructions for the migration of keys 1. ONGOING ACTIVITIES - Prepared key handling guides (import keys/delete keys/move keys) - will be shared as github gist markdown for NO and LIDO - Prepared a Node update guide - will be shared as github gist markdown for NO and LIDO - Doppelgänger protection of Nodes is active on new and unassigned servers for standby; on the productive Nodes it is being rolled out today. - Slashing alerts tool (fast and reliable alerts to all Devs) is in preparation 1. FURTHER ACTION POINTS - Make all guides public to everyone on our website - Review other processes to see if bugs like this could possibly cause similar outcomes. - Schedule additional automized tests for such cases - Make use of lessons learned: Tighten up the process of moving keys and configs to eliminate the risk of running into a bug If you have further questions or ideas, please contact us at any time. ---
**Author:** ![governance-data-bot](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/governance-data-bot/32/2204_2.png) [@governance-data-bot](https://research.lido.fi/u/governance-data-bot)\ **Post date:** [April 20, 2023, 5:15pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/13 "2023-04-20T17:15:40Z")
## Snapshot vote started We’re starting the [RockLogic Slashing Incident Staker Compensation](https://snapshot.org/#/lido-snapshot.eth/proposal/0x78bbc81011457ffcc0d2183de2a813869708d4f9996f4af3df8b669510950cf3) Snapshot, active till Thu, 27 Apr 2023 17:00:00 GMT . Please don’t forget to cast your vote! ---
**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\ **Post date:** [April 21, 2023, 7:20am UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/14 "2023-04-21T07:20:44Z")
The relevant identified bug in Prysm’s keymanager has been addressed by the Prysmatic Labs team in the newest release [Release v4.0.3 · prysmaticlabs/prysm · GitHub](https://github.com/prysmaticlabs/prysm/releases/tag/v4.0.3), as a part of [PR 12284](https://github.com/prysmaticlabs/prysm/pull/12284) ---
**Author:** ![stefa2k](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/stefa2k/32/6940_2.png) [@stefa2k](https://research.lido.fi/u/stefa2k)\ **Post date:** [April 21, 2023, 9:17am UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/15 "2023-04-21T09:17:03Z")
You will now find our key handling guides, node update guide and Doppelgänger protection of nodes on github gist. Please provide us with feedback for the processes we propose! Import keys on node > Delete keys on node > Move keys (migration) > Update node > Doppelgänger detection for clients > ---
**Author:** ![zuzu\_eeka](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/zuzu_eeka/32/1157_2.png) [@zuzu\_eeka](https://research.lido.fi/u/zuzu_eeka)\ **Post date:** [April 21, 2023, 4:04pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/16 "2023-04-21T16:04:43Z")
Aragon intended to setting RockLogic’s limit to 5800 was enacted [Ethereum Transaction Hash (Txhash) Details | Etherscan](https://etherscan.io/tx/0xc1e45db477c3502cf975927d982f79771d206c7b30afb31b8211c17184465e95) ---
**Author:** ![MF\_DROO](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/mf_droo/32/2286_2.png) [@MF\_DROO](https://research.lido.fi/u/MF_DROO)\ **Post date:** [April 25, 2023, 3:28pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/17 "2023-04-25T15:28:05Z")
To hopefully add to the discussion beyond the operational~~ The node operator in question **is not ‘faultless’** , as these errors and slower response-time (i.e. contributors informing the operator and not the other way around) feel like symptoms of a larger issue. If you provide another entity that ‘specializes’ in Devops an opportunity to make ~decent $$ a year, wouldn’t you expect them to have: - strong monitoring, - & the ability to proportionally reimburse, when fault is admitted? There are plenty of candidates eagerly waiting to join the validator set, and the standards for being a part of the curated professional validator set should be set to a professional level. Lowering the operators key limit seems to be the best middle-ground for the parties involved. I see Lido’s mission is to deliver on decentralization and making staking simple – not to provide operators who are operating shakily a profitable business model. For every $ we spend in the latter direction we take away from the DAOs ability to forward it’s overarching goals. ---
**Author:** ![stefa2k](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/stefa2k/32/6940_2.png) [@stefa2k](https://research.lido.fi/u/stefa2k)\ **Post date:** [April 25, 2023, 7:43pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/18 "2023-04-25T19:43:46Z")
Following the incident from April 13, 2023 and the measures we have taken since to prevent something like this to happen again, we want to give you a status update as of today. This also refers to the public call today which you can see here: [Node Operator Community Call #6 - YouTube](https://www.youtube.com/watch?v=VGqMmAgGcg0) SLASHING INCIDENT OVERVIEW On April 13, 2023, 11 validators operated by RockLogic GmbH were slashed due to the duplication of validator keys in two different active clusters, causing a double vote. This was caused by an unforseeable bug, which since has been eliminated. But, at the time of incident, the full extent of measures to prevent key duplication was not taken, leading to the slashing. INCIDENT RESPONSE When the slashing event was confirmed within 15 mins of its first occurrence, we prevented further slashings by bringing relevant clusters offline. The failover cluster was brought back online and the remaining keys were all incrementally activated within the next three hours. Both our’s and Prysmatic Labs’ technical investigations confirmed the bug the next day and we made it reproducible for further analysis. Lido released a post mortem that day. REMEDIATION ACTIVITIES The bug has been identified and fixed by Prysmatic Labs (GH Issue #12281) as of Prysm v4.0.3. RockLogic GmbH updated the configuration of nodes (doppelganger used uniformly throughout) and key handling guides and issued those on Github. We are updating monitoring & alerting and procedures & guides, which will be made publicly available soon (ongoing). We also plan to create automated tests for such cases (not started yet). INCIDENT FOLLOW-UP As a consequence and further precaution following the slashing, Lido DAO took an on-chain Aragon vote to limit RockLogic keys until full remediation has taken place, which passed. There is also a Lido DAO Snapshot vote for staker compensation using the cover fund still in progress. FUTURE PROCEDURES We plan a community discussion about which policies to implement for future reactions upon incidents like this, and related ones (eg large outages) from a governance and NO set management perspective. This will start later this week. ---
**Author:** ![governance-data-bot](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/governance-data-bot/32/2204_2.png) [@governance-data-bot](https://research.lido.fi/u/governance-data-bot)\ **Post date:** [April 27, 2023, 5:05pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/19 "2023-04-27T17:05:10Z")
## Snapshot vote ended Thank you all who participated in the [RockLogic Slashing Incident Staker Compensation](https://snapshot.org/#/lido-snapshot.eth/proposal/0x78bbc81011457ffcc0d2183de2a813869708d4f9996f4af3df8b669510950cf3) Snapshot, the proposal passed! 🙏 The results are: **For** : 55.2M LDO **Against** : 27 LDO ---
**Author:** ![Izzy](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/izzy/32/320_2.png) [@Izzy](https://research.lido.fi/u/Izzy)\ **Post date:** [April 28, 2023, 11:05am UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/20 "2023-04-28T11:05:29Z")
The proposal to use the cover fund for compensation has passed. Once the validators have fully exited (~May 20th) and the actual amount of penalties can be finalized, another vote will be set up so that the compensation can occur. ---
**Author:** ![stefa2k](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/stefa2k/32/6940_2.png) [@stefa2k](https://research.lido.fi/u/stefa2k)\ **Post date:** [May 3, 2023, 5:25pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/21 "2023-05-03T17:25:51Z")
**Lido on Ethereum RockLogic Weekly News Post** “Making things better by the week” First Update 03.05.2023 INTRODUCTION This is a new format that Lido DAO contributors and RockLogic set up to communicate in a more transparent manner with the Ethereum blockchain community. What happened until this week: MINOR BUG A 1.000 keys were moved, following a Lighthouse bug - no harm here. > > > \## Description > > Setup with Besu, LH BC & LH VC. VC seems to lose connection to… BC. > > I tried: > \- restarting all services > \- resyncing BC with checkpoint sync > > \## Version > > \`v4.0.2-rc.0\` > > \## Present Behaviour > > BC logs don't show anything unusual. > > Errors in VC: > \`\`\` > Apr 30 16:05:47.878 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 54, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[\], count: 0, service: attestation > Apr 30 16:05:47.879 DEBG Request to beacon node failed error: "No aggregate available for AttestationData { slot: Slot(6337226), index: 54, beacon\_block\_root: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, source: Checkpoint { epoch: Epoch(198037), root: 0x42e925532fce00fc81beca5da88319a82d038a5ab0e2671981e9f12abc776871 }, target: Checkpoint { epoch: Epoch(198038), root: 0xf505cdac4b7379cc260c6b58b30531cee18dd05ec968e380f60241f7f81a35a8 } }", node: http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/ > Apr 30 16:05:47.879 CRIT Error during attestation routine slot: 6337226, committee\_index: 54, error: "Some endpoints failed, num\_failed: 1 http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/ =\> RequestFailed(\\"No aggregate available for AttestationData { slot: Slot(6337226), index: 54, beacon\_block\_root: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, source: Checkpoint { epoch: Epoch(198037), root: 0x42e925532fce00fc81beca5da88319a82d038a5ab0e2671981e9f12abc776871 }, target: Checkpoint { epoch: Epoch(198038), root: 0xf505cdac4b7379cc260c6b58b30531cee18dd05ec968e380f60241f7f81a35a8 } }\\")", service: attestation > Apr 30 16:05:47.881 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 32, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[430031\], count: 1, service: attestation > Apr 30 16:05:47.882 INFO Connected to beacon node endpoint: http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/, version: Lighthouse/v4.0.2-rc.0-35d8c98/x86\_64-linux > Apr 30 16:05:48.584 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 49, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[432238\], count: 1, service: attestation > Apr 30 16:05:48.585 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 20, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[431688\], count: 1, service: attestation > Apr 30 16:05:51.001 DEBG No local validators in current sync committee, slot: 6337227, service: sync\_committee > Apr 30 16:05:51.001 DEBG Fetching subscription duties current\_slot: 6337227, duty\_slot: 6337227, service: sync\_committee > Apr 30 16:05:51.001 DEBG No sync subscriptions to send slot: 6337227, service: sync\_committee > Apr 30 16:05:51.338 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 32, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[432237\], count: 1, service: attestation > Apr 30 16:05:51.447 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 43, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433094\], count: 1, service: attestation > Apr 30 16:05:51.448 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 52, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[436430\], count: 1, service: attestation > Apr 30 16:05:51.518 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 15, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[431902\], count: 1, service: attestation > Apr 30 16:05:51.519 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 25, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[436008\], count: 1, service: attestation > Apr 30 16:05:51.566 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 45, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[418892\], count: 1, service: attestation > Apr 30 16:05:51.569 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 20, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[432869\], count: 1, service: attestation > Apr 30 16:05:53.001 INFO Connected to beacon node(s) synced: 1, available: 1, total: 1, service: notifier > Apr 30 16:05:53.001 INFO All validators active slot: 6337227, epoch: 198038, total\_validators: 500, active\_validators: 500, current\_epoch\_proposers: 0, service: notifier > Apr 30 16:05:53.374 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 5, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433089\], count: 1, service: attestation > Apr 30 16:05:53.374 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 36, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433090\], count: 1, service: attestation > Apr 30 16:05:55.006 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 52, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 436430, service: attestation > Apr 30 16:05:55.006 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 5, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 433089, service: attestation > Apr 30 16:05:55.006 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 36, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 433090, service: attestation > Apr 30 16:05:55.007 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 25, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 436008, service: attestation > Apr 30 16:05:55.788 DEBG Measured BN latency latency: 0, node: http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/ > Apr 30 16:05:56.388 CRIT Not signing slashable attestation error: SQLPoolError("timed out waiting for connection"), attestation: AttestationData { slot: Slot(6337227), index: 31, beacon\_block\_root: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, source: Checkpoint { epoch: Epoch(198037), root: 0x42e925532fce00fc81beca5da88319a82d038a5ab0e2671981e9f12abc776871 }, target: Checkpoint { epoch: Epoch(198038), root: 0xf505cdac4b7379cc260c6b58b30531cee18dd05ec968e380f60241f7f81a35a8 } } > Apr 30 16:05:56.388 CRIT Failed to sign attestation slot: 6337227, committee\_index: 31, validator: 0x94787d7167e21db1e629e29153c48cd53035bdf2cb2d34bcf0428702267717cc300af9eb5fd981ca109798db8718c6b8, error: Slashable(SQLPoolError("timed out waiting for connection")), service: attestation > Apr 30 16:05:56.391 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 31, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[436422\], count: 1, service: attestation > Apr 30 16:05:56.391 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 27, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[411179\], count: 1, service: attestation > Apr 30 16:05:56.963 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 12, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433005\], count: 1, service: attestation > Apr 30 16:05:56.965 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 62, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[435326, 414154\], count: 2, service: attestation > Apr 30 16:05:59.016 DEBG Computed attestation selection proofs time\_taken\_ms: 15, lookahead\_slot: 6337236, batch\_size: 21, service: duties > \`\`\` > > BC logs: > \`\`\` > Apr 30 16:05:00.163 INFO New block received root: 0xc0c51a21376570ff84a3d9cfb02a68e10aa9138084b16d9657e5faa5f96cf53b, slot: 6337223 > Apr 30 16:05:05.000 INFO Downloading historical blocks est\_time: 2 days 2 hrs, speed: 34.67 slots/sec, distance: 6319489 slots (125 weeks 2 days), service: slot\_notifier > Apr 30 16:05:05.000 INFO Synced slot: 6337223, block: 0xc0c5…f53b, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x8959…8c47 (verified), peers: 59, service: slot\_notifier > Apr 30 16:05:05.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:12.488 INFO New block received root: 0x6fb51c5d7f917b61d1f0e0f1445a0431096f0f09a77bc040f3a7c55d0d9f9a8a, slot: 6337224 > Apr 30 16:05:17.000 INFO Synced slot: 6337224, block: 0x6fb5…9a8a, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0xaaf9…e50f (verified), peers: 60, service: slot\_notifier > Apr 30 16:05:17.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:24.303 INFO New block received root: 0x39294f957aae4539512e6793bedf30eeb35319f7a18717198c52eb5bcebfc9e6, slot: 6337225 > Apr 30 16:05:29.000 INFO Synced slot: 6337225, block: 0x3929…c9e6, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x4543…8f6a (verified), peers: 61, service: slot\_notifier > Apr 30 16:05:29.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:36.724 INFO New block received root: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, slot: 6337226 > Apr 30 16:05:41.000 INFO Synced slot: 6337226, block: 0xebdc…edd8, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0xda67…e3ba (verified), peers: 62, service: slot\_notifier > Apr 30 16:05:41.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:48.293 INFO New block received root: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, slot: 6337227 > Apr 30 16:05:53.000 INFO Synced slot: 6337227, block: 0x0a49…d99d, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x6c36…d739 (verified), peers: 62, service: slot\_notifier > Apr 30 16:05:53.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:06:00.632 INFO New block received root: 0xe6a78949a9d0cd8bb0c5aeb4bd020227f7c284bb04344e07586a76d83939e059, slot: 6337228 > Apr 30 16:06:05.000 INFO Downloading historical blocks est\_time: 1 day 11 hrs, speed: 49.34 slots/sec, distance: 6315969 slots (125 weeks 2 days), service: slot\_notifier > Apr 30 16:06:05.000 INFO Synced slot: 6337228, block: 0xe6a7…e059, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x5d34…181c (verified), peers: 62, service: slot\_notifier > Apr 30 16:06:05.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:06:12.984 INFO New block received root: 0xd30b68f3d81b7a773ccca0ea25d9638f1d6b27dcc40abb992d6ab04b26da18c8, slot: 6337229 > Apr 30 16:06:17.000 INFO Synced slot: 6337229, block: 0xd30b…18c8, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x27fc…7ac2 (verified), peers: 63, service: slot\_notifier > Apr 30 16:06:17.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:06:24.074 INFO New block received root: 0x6fd092d4186832871d6ded6f0bf9337d5a9791405aeb6ba735cf2f20937870c3, slot: 6337230 > Apr 30 16:06:29.001 INFO Synced slot: 6337230, block: 0x6fd0…70c3, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x854c…32db (verified), peers: 63, service: slot\_notifier > \`\`\` > > \## Expected Behaviour > > VC not losing connection to BC every slot. > > \## Steps to resolve > > Unknown. INFRASTRUCTURE EXTENSION We moved 500 keys in accordance with an extension of our infrastructure. SLASHING ALERT SERVICE As announced previously, we will establish a slashing alert service - this is due to be released May 4th, 2023. LIDO V2 COMPLIANCE RockLogic will be fully compliant to Lido V2 on mainnet by May 8th, 2023. So far, it has been fully tested on the testnet. COMMUNITY DISCUSSION ON INCIDENT TREATMENT We are still keen on input of the community following our thread on how to deal with future incidents and if/how to set up a universal procedure of how to deal with things: > [@Discussion - Treatment of Potentially Harmful Incidents](https://research.lido.fi/t/discussion-treatment-of-potentially-harmful-incidents/4498): > > The [slashing incident](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023) of April 13, 2023 has got us all thinking about its causes and how to remedy them - and perhaps even more importantly, about how to prevent incidents like this happening to Ethereum and its contributors in the future. Whether you call this particular incident a mistake, a reluctant reaction, human error, system failure or anything else, there will always be minor and major incidents that are unforeseeable; we need a proper strategy to deal with these. Since mistakes provi… [Next page](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399.md?page=2)