# Slashing Incident involving RockLogic GmbH Validators - April 13, 2023 **URL:** \ **Category:** Node Operators\ **Created:** [April 13, 2023, 2:00pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399 "2023-04-13T14:00:42Z")\ **Posts on this page:** 1\ **Showing post:** 21
**Author:** ![stefa2k](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/stefa2k/32/6940_2.png) [@stefa2k](https://research.lido.fi/u/stefa2k)\ **Post date:** [May 3, 2023, 5:25pm UTC](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399/21 "2023-05-03T17:25:51Z")
**Lido on Ethereum RockLogic Weekly News Post** “Making things better by the week” First Update 03.05.2023 INTRODUCTION This is a new format that Lido DAO contributors and RockLogic set up to communicate in a more transparent manner with the Ethereum blockchain community. What happened until this week: MINOR BUG A 1.000 keys were moved, following a Lighthouse bug - no harm here. > > > \## Description > > Setup with Besu, LH BC & LH VC. VC seems to lose connection to… BC. > > I tried: > \- restarting all services > \- resyncing BC with checkpoint sync > > \## Version > > \`v4.0.2-rc.0\` > > \## Present Behaviour > > BC logs don't show anything unusual. > > Errors in VC: > \`\`\` > Apr 30 16:05:47.878 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 54, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[\], count: 0, service: attestation > Apr 30 16:05:47.879 DEBG Request to beacon node failed error: "No aggregate available for AttestationData { slot: Slot(6337226), index: 54, beacon\_block\_root: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, source: Checkpoint { epoch: Epoch(198037), root: 0x42e925532fce00fc81beca5da88319a82d038a5ab0e2671981e9f12abc776871 }, target: Checkpoint { epoch: Epoch(198038), root: 0xf505cdac4b7379cc260c6b58b30531cee18dd05ec968e380f60241f7f81a35a8 } }", node: http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/ > Apr 30 16:05:47.879 CRIT Error during attestation routine slot: 6337226, committee\_index: 54, error: "Some endpoints failed, num\_failed: 1 http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/ =\> RequestFailed(\\"No aggregate available for AttestationData { slot: Slot(6337226), index: 54, beacon\_block\_root: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, source: Checkpoint { epoch: Epoch(198037), root: 0x42e925532fce00fc81beca5da88319a82d038a5ab0e2671981e9f12abc776871 }, target: Checkpoint { epoch: Epoch(198038), root: 0xf505cdac4b7379cc260c6b58b30531cee18dd05ec968e380f60241f7f81a35a8 } }\\")", service: attestation > Apr 30 16:05:47.881 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 32, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[430031\], count: 1, service: attestation > Apr 30 16:05:47.882 INFO Connected to beacon node endpoint: http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/, version: Lighthouse/v4.0.2-rc.0-35d8c98/x86\_64-linux > Apr 30 16:05:48.584 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 49, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[432238\], count: 1, service: attestation > Apr 30 16:05:48.585 INFO Successfully published attestations type: unaggregated, slot: 6337226, committee\_index: 20, head\_block: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, validator\_indices: \[431688\], count: 1, service: attestation > Apr 30 16:05:51.001 DEBG No local validators in current sync committee, slot: 6337227, service: sync\_committee > Apr 30 16:05:51.001 DEBG Fetching subscription duties current\_slot: 6337227, duty\_slot: 6337227, service: sync\_committee > Apr 30 16:05:51.001 DEBG No sync subscriptions to send slot: 6337227, service: sync\_committee > Apr 30 16:05:51.338 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 32, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[432237\], count: 1, service: attestation > Apr 30 16:05:51.447 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 43, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433094\], count: 1, service: attestation > Apr 30 16:05:51.448 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 52, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[436430\], count: 1, service: attestation > Apr 30 16:05:51.518 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 15, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[431902\], count: 1, service: attestation > Apr 30 16:05:51.519 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 25, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[436008\], count: 1, service: attestation > Apr 30 16:05:51.566 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 45, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[418892\], count: 1, service: attestation > Apr 30 16:05:51.569 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 20, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[432869\], count: 1, service: attestation > Apr 30 16:05:53.001 INFO Connected to beacon node(s) synced: 1, available: 1, total: 1, service: notifier > Apr 30 16:05:53.001 INFO All validators active slot: 6337227, epoch: 198038, total\_validators: 500, active\_validators: 500, current\_epoch\_proposers: 0, service: notifier > Apr 30 16:05:53.374 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 5, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433089\], count: 1, service: attestation > Apr 30 16:05:53.374 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 36, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433090\], count: 1, service: attestation > Apr 30 16:05:55.006 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 52, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 436430, service: attestation > Apr 30 16:05:55.006 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 5, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 433089, service: attestation > Apr 30 16:05:55.006 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 36, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 433090, service: attestation > Apr 30 16:05:55.007 INFO Successfully published attestation type: aggregated, slot: 6337227, committee\_index: 25, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, signatures: 1, aggregator: 436008, service: attestation > Apr 30 16:05:55.788 DEBG Measured BN latency latency: 0, node: http://stereum-ac4c61eb-6976-9de3-b6f1-48c29a215355:5052/ > Apr 30 16:05:56.388 CRIT Not signing slashable attestation error: SQLPoolError("timed out waiting for connection"), attestation: AttestationData { slot: Slot(6337227), index: 31, beacon\_block\_root: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, source: Checkpoint { epoch: Epoch(198037), root: 0x42e925532fce00fc81beca5da88319a82d038a5ab0e2671981e9f12abc776871 }, target: Checkpoint { epoch: Epoch(198038), root: 0xf505cdac4b7379cc260c6b58b30531cee18dd05ec968e380f60241f7f81a35a8 } } > Apr 30 16:05:56.388 CRIT Failed to sign attestation slot: 6337227, committee\_index: 31, validator: 0x94787d7167e21db1e629e29153c48cd53035bdf2cb2d34bcf0428702267717cc300af9eb5fd981ca109798db8718c6b8, error: Slashable(SQLPoolError("timed out waiting for connection")), service: attestation > Apr 30 16:05:56.391 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 31, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[436422\], count: 1, service: attestation > Apr 30 16:05:56.391 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 27, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[411179\], count: 1, service: attestation > Apr 30 16:05:56.963 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 12, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[433005\], count: 1, service: attestation > Apr 30 16:05:56.965 INFO Successfully published attestations type: unaggregated, slot: 6337227, committee\_index: 62, head\_block: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, validator\_indices: \[435326, 414154\], count: 2, service: attestation > Apr 30 16:05:59.016 DEBG Computed attestation selection proofs time\_taken\_ms: 15, lookahead\_slot: 6337236, batch\_size: 21, service: duties > \`\`\` > > BC logs: > \`\`\` > Apr 30 16:05:00.163 INFO New block received root: 0xc0c51a21376570ff84a3d9cfb02a68e10aa9138084b16d9657e5faa5f96cf53b, slot: 6337223 > Apr 30 16:05:05.000 INFO Downloading historical blocks est\_time: 2 days 2 hrs, speed: 34.67 slots/sec, distance: 6319489 slots (125 weeks 2 days), service: slot\_notifier > Apr 30 16:05:05.000 INFO Synced slot: 6337223, block: 0xc0c5…f53b, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x8959…8c47 (verified), peers: 59, service: slot\_notifier > Apr 30 16:05:05.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:12.488 INFO New block received root: 0x6fb51c5d7f917b61d1f0e0f1445a0431096f0f09a77bc040f3a7c55d0d9f9a8a, slot: 6337224 > Apr 30 16:05:17.000 INFO Synced slot: 6337224, block: 0x6fb5…9a8a, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0xaaf9…e50f (verified), peers: 60, service: slot\_notifier > Apr 30 16:05:17.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:24.303 INFO New block received root: 0x39294f957aae4539512e6793bedf30eeb35319f7a18717198c52eb5bcebfc9e6, slot: 6337225 > Apr 30 16:05:29.000 INFO Synced slot: 6337225, block: 0x3929…c9e6, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x4543…8f6a (verified), peers: 61, service: slot\_notifier > Apr 30 16:05:29.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:36.724 INFO New block received root: 0xebdcf96f86d61001585454dd2b4a3086ae4e38f6da7ea25de61729a9bf8aedd8, slot: 6337226 > Apr 30 16:05:41.000 INFO Synced slot: 6337226, block: 0xebdc…edd8, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0xda67…e3ba (verified), peers: 62, service: slot\_notifier > Apr 30 16:05:41.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:05:48.293 INFO New block received root: 0x0a49f36fe421e2f855ef0449dd22203d055a5c00b21478098c454f86df25d99d, slot: 6337227 > Apr 30 16:05:53.000 INFO Synced slot: 6337227, block: 0x0a49…d99d, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x6c36…d739 (verified), peers: 62, service: slot\_notifier > Apr 30 16:05:53.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:06:00.632 INFO New block received root: 0xe6a78949a9d0cd8bb0c5aeb4bd020227f7c284bb04344e07586a76d83939e059, slot: 6337228 > Apr 30 16:06:05.000 INFO Downloading historical blocks est\_time: 1 day 11 hrs, speed: 49.34 slots/sec, distance: 6315969 slots (125 weeks 2 days), service: slot\_notifier > Apr 30 16:06:05.000 INFO Synced slot: 6337228, block: 0xe6a7…e059, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x5d34…181c (verified), peers: 62, service: slot\_notifier > Apr 30 16:06:05.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:06:12.984 INFO New block received root: 0xd30b68f3d81b7a773ccca0ea25d9638f1d6b27dcc40abb992d6ab04b26da18c8, slot: 6337229 > Apr 30 16:06:17.000 INFO Synced slot: 6337229, block: 0xd30b…18c8, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x27fc…7ac2 (verified), peers: 63, service: slot\_notifier > Apr 30 16:06:17.000 WARN Syncing deposit contract block cache est\_blocks\_remaining: initializing deposits, service: slot\_notifier > Apr 30 16:06:24.074 INFO New block received root: 0x6fd092d4186832871d6ded6f0bf9337d5a9791405aeb6ba735cf2f20937870c3, slot: 6337230 > Apr 30 16:06:29.001 INFO Synced slot: 6337230, block: 0x6fd0…70c3, epoch: 198038, finalized\_epoch: 198036, finalized\_root: 0x2c27…c055, exec\_hash: 0x854c…32db (verified), peers: 63, service: slot\_notifier > \`\`\` > > \## Expected Behaviour > > VC not losing connection to BC every slot. > > \## Steps to resolve > > Unknown. INFRASTRUCTURE EXTENSION We moved 500 keys in accordance with an extension of our infrastructure. SLASHING ALERT SERVICE As announced previously, we will establish a slashing alert service - this is due to be released May 4th, 2023. LIDO V2 COMPLIANCE RockLogic will be fully compliant to Lido V2 on mainnet by May 8th, 2023. So far, it has been fully tested on the testnet. COMMUNITY DISCUSSION ON INCIDENT TREATMENT We are still keen on input of the community following our thread on how to deal with future incidents and if/how to set up a universal procedure of how to deal with things: > [@Discussion - Treatment of Potentially Harmful Incidents](https://research.lido.fi/t/discussion-treatment-of-potentially-harmful-incidents/4498): > > The [slashing incident](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023) of April 13, 2023 has got us all thinking about its causes and how to remedy them - and perhaps even more importantly, about how to prevent incidents like this happening to Ethereum and its contributors in the future. Whether you call this particular incident a mistake, a reluctant reaction, human error, system failure or anything else, there will always be minor and major incidents that are unforeseeable; we need a proper strategy to deal with these. Since mistakes provi… --- _[View the full topic](https://research.lido.fi/t/slashing-incident-involving-rocklogic-gmbh-validators-april-13-2023/4399)._