# RockLogic Monthly Notes - 09/2023
**URL:** https://research.lido.fi/t/rocklogic-monthly-notes-09-2023/5395
**Category:** Node Operators
**Created:** [September 7, 2023, 9:52am UTC](https://research.lido.fi/t/rocklogic-monthly-notes-09-2023/5395 "2023-09-07T09:52:23Z")
**Posts on this page:** 1
**Page:** 1
### Author:  [@RRoland](https://research.lido.fi/u/RRoland)
#### Post date: [September 7, 2023, 9:52am UTC](https://research.lido.fi/t/rocklogic-monthly-notes-09-2023/5395/1 "2023-09-07T09:52:23Z")
Dear Lido Community,
welcome to our monthly notes, updating you about our activities to maintain a safe Ethereum environment with Stereum.
SOFTWARE UPDATES
All full nodes and validator clients have been updated.
We started investigations regarding a Nimbus validator client performing poorly with non-Nimbus fallback nodes. More information about this Nimbus crash investigation:
>
>
> \*\*Describe the bug\*\*
> Nimbus VC recognizing time offset followed by missed dutie…s and resulting in a crash.
> The VC was running just fine beforehand (no time offsets, no missed duties)
>
> \*\*To Reproduce\*\*
> Nimbus is running in Docker with \`statusim/nimbus-validator-client:multiarch-v23.7.0\` image
>
> Logs:
> \`\`\`
> {"log":"INF 2023-08-29 03:56:47.004+00:00 Slot start slot=7204782 epoch=225149 attestationIn=4s blockIn=\\u003cunknown\\u003e validators=500 good\_nodes=2 viable\_nodes=0 bad\_nodes=4 delay=4ms800us139ns\\n","stream":"stdout","time":"2023-08-29T03:56:47.005291
> 403Z"}
> {"log":"INF 2023-08-29 03:56:59.003+00:00 Slot start slot=7204783 epoch=225149 attestationIn=4s blockIn=\\u003cunknown\\u003e validators=500 good\_nodes=2 viable\_nodes=0 bad\_nodes=4 delay=3ms40us618ns\\n","stream":"stdout","time":"2023-08-29T03:56:59.0035494
> 45Z"}
> {"log":"INF 2023-08-29 03:57:04.676+00:00 Beacon node has notable time offset time\_offset=596ms699us238ns service=fallback\_service node=http://10.12.1.62:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=5 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:04.6765380
> 71Z"}
> {"log":"WRN 2023-08-29 03:57:05.877+00:00 Beacon node has significant time offset time\_offset=2s832ms261us118ns service=fallback\_service node=http://10.12.1.55:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:05.878121314Z"}
> {"log":"WRN 2023-08-29 03:57:05.877+00:00 Beacon node has enormous time offset time\_offset=2s832ms261us118ns service=fallback\_service node=http://10.12.1.55:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:05.878157704Z"}
> {"log":"WRN 2023-08-29 03:57:05.878+00:00 Beacon node's clock is out of order, (beacon node is unusable) node=http://10.12.1.55:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:05.878164366Z"}
> {"log":"INF 2023-08-29 03:57:11.542+00:00 Slot start slot=7204784 epoch=225149 attestationIn=4s blockIn=\\u003cunknown\\u003e validators=500 good\_nodes=1 viable\_nodes=0 bad\_nodes=4 delay=542ms471us289ns\\n","stream":"stdout","time":"2023-08-29T03:57:11.542851935Z"}
> {"log":"WRN 2023-08-29 03:57:11.670+00:00 Beacon node down reason=\\"Beacon node has acceptable time offset;200;checkTimeOffsetStatus()();status-update\\" node=http://10.12.1.55:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:11.670298428Z"}
> {"log":"INF 2023-08-29 03:57:12.853+00:00 Beacon node has notable time offset time\_offset=913ms852us823ns service=fallback\_service node=http://10.12.1.62:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=5 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:12.853519981Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=17 duties\_count=3 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007327417Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=23 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007369927Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=30 duties\_count=2 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007375277Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=11 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007386307Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=47 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007391237Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=0 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.00739827Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=39 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007406305Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=27 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007472891Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=45 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007479824Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=25 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007518005Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=32 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007534186Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=43 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007586223Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=4 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007598677Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=9 duties\_count=2 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007638783Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=38 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007676734Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=1 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007739091Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=55 duties\_count=1 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007782843Z"}
> {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee\_index=20 duties\_count=2 reason=\\"No online beacon node(s)\\" service=attestation\_service\\n","stream":"stdout","time":"2023-08-29T03:57:19.007795578Z"}
> {"log":"NOT 2023-08-29 03:57:21.410+00:00 Beacon node is online agent\_version=Nimbus/v23.7.0-60f0a2-stateofus node=http://10.12.1.62:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=5 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:21.41085529Z"}
> {"log":"NOT 2023-08-29 03:57:21.428+00:00 Beacon node is online agent\_version=Nimbus/v23.7.0-60f0a2-stateofus node=http://10.12.1.55:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:21.428594517Z"}
> {"log":"NOT 2023-08-29 03:57:21.480+00:00 Beacon node is compatible node=http://10.12.1.62:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=5 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:21.480468129Z"}
> {"log":"NOT 2023-08-29 03:57:21.482+00:00 Beacon node is compatible node=http://10.12.1.55:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:21.482453775Z"}
> {"log":"NOT 2023-08-29 03:57:21.503+00:00 Beacon node is in sync head\_slot=7204784 sync\_distance=0 is\_optimistic=false node=http://10.12.1.55:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:21.503179259Z"}
> {"log":"NOT 2023-08-29 03:57:21.505+00:00 Beacon node is in sync head\_slot=7204784 sync\_distance=0 is\_optimistic=false node=http://10.12.1.62:5052\[Nimbus/v23.7.0-60f0a2-stateofus\] node\_index=5 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-29T03:57:21.505819437Z"}
> {"log":"INF 2023-08-29 03:57:23.322+00:00 Slot start slot=7204785 epoch=225149 attestationIn=4s blockIn=\\u003cunknown\\u003e validators=500 good\_nodes=2 viable\_nodes=0 bad\_nodes=4 delay=322ms496us772ns\\n","stream":"stdout","time":"2023-08-29T03:57:23.322832844Z"}
> {"log":"WRN 2023-08-29 03:57:23.328+00:00 Unable to publish sync committee messages and contributions in time slot=7204784 timeout=11s456ms906us700ns service=sync\_committee\_service\\n","stream":"stdout","time":"2023-08-29T03:57:23.328831842Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(865) main\\n","stream":"stderr","time":"2023-08-29T03:57:25.021216691Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(858) NimMain\\n","stream":"stderr","time":"2023-08-29T03:57:25.021256956Z"}
> {"log":"/home/user/nimbus-eth2/beacon\_chain/nimbus\_validator\_client.nim(521) main\\n","stream":"stderr","time":"2023-08-29T03:57:25.021261705Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(1504) waitFor\\n","stream":"stderr","time":"2023-08-29T03:57:25.021265562Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(258) poll\\n","stream":"stderr","time":"2023-08-29T03:57:25.021268959Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(318) futureContinue\\n","stream":"stderr","time":"2023-08-29T03:57:25.021272505Z"}
> {"log":"/home/user/nimbus-eth2/beacon\_chain/validator\_client/api.nim(233) getAttesterDuties\\n","stream":"stderr","time":"2023-08-29T03:57:25.021276212Z"}
> {"log":"/home/user/nimbus-eth2/beacon\_chain/validator\_client/api.nim(117) apiResponseOr\\n","stream":"stderr","time":"2023-08-29T03:57:25.02127999Z"}
> {"log":"/home/user/nimbus-eth2/beacon\_chain/validator\_client/api.nim(120) apiResponseOr\\n","stream":"stderr","time":"2023-08-29T03:57:25.021283446Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/assertions.nim(38) failedAssertImpl\\n","stream":"stderr","time":"2023-08-29T03:57:25.021287013Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/assertions.nim(28) raiseAssert\\n","stream":"stderr","time":"2023-08-29T03:57:25.021290509Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(54) sysFatal\\n","stream":"stderr","time":"2023-08-29T03:57:25.021294137Z"}
> {"log":"\[\[reraised from:\\n","stream":"stderr","time":"2023-08-29T03:57:25.021297473Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(865) main\\n","stream":"stderr","time":"2023-08-29T03:57:25.021300749Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(858) NimMain\\n","stream":"stderr","time":"2023-08-29T03:57:25.021304026Z"}
> {"log":"/home/user/nimbus-eth2/beacon\_chain/nimbus\_validator\_client.nim(521) main\\n","stream":"stderr","time":"2023-08-29T03:57:25.021307452Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(1504) waitFor\\n","stream":"stderr","time":"2023-08-29T03:57:25.021310929Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(258) poll\\n","stream":"stderr","time":"2023-08-29T03:57:25.021314245Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(318) futureContinue\\n","stream":"stderr","time":"2023-08-29T03:57:25.021317641Z"}
> {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncmacro2.nim(197) getAttesterDuties\\n","stream":"stderr","time":"2023-08-29T03:57:25.021320867Z"}
> {"log":"\]\]\\n","stream":"stderr","time":"2023-08-29T03:57:25.021324264Z"}
> {"log":"Error: unhandled exception: /home/user/nimbus-eth2/beacon\_chain/validator\_client/api.nim(120, 13) \`not (future.cancelled())\` \[AssertionDefect\]\\n","stream":"stderr","time":"2023-08-29T03:57:25.02132757Z"}
>
> \`\`\`
>
>
> \*\*Describe the bug\*\*
> Nimbus VC continuously drops Prysm and Teku BN connections…. For Prysm this normally happens after a lot of attestations are published and after VC receives new attester duties. For Teku this only happens after receiving attester duties.
>
> This is on Mainnet holding 500 keys.
>
> Same setup on testnet with 1 key works fine.
>
> running in docker:
> \`consensys/teku:23.8.0\`
> \`prysmaticlabs/prysm-beacon-chain:v4.0.8\`
> \`nimbus-validator-client:multiarch-v23.7.0\`
>
> Logs:
> \`\`\`
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=a6dc555e slot=7223569 validator\_index=514033 committee\_index=31 committee\_length=374 committees\_at\_slot=64 validator\_committee\_index=119 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.735339392Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=81874181 slot=7223567 validator\_index=514044 committee\_index=18 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=294 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.735366031Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=abbd30f1 slot=7223557 validator\_index=506733 committee\_index=30 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=136 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.735421667Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=883e1861 slot=7223582 validator\_index=517741 committee\_index=36 committee\_length=374 committees\_at\_slot=64 validator\_committee\_index=275 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.735427488Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=a5755039 slot=7223559 validator\_index=506140 committee\_index=8 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=223 epoch=225736 dependent\_root=da7ad193 service=duties\_servic
> e\\n","stream":"stdout","time":"2023-08-31T18:24:23.735471561Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=af495a90 slot=7223564 validator\_index=506898 committee\_index=33 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=357 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.73551859Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=81ba9640 slot=7223554 validator\_index=506896 committee\_index=62 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=24 epoch=225736 dependent\_root=da7ad193 service=duties\_servic
> e\\n","stream":"stdout","time":"2023-08-31T18:24:23.735532486Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=982a9814 slot=7223579 validator\_index=514172 committee\_index=10 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=363 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.735574475Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=878460da slot=7223581 validator\_index=486544 committee\_index=15 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=318 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.735625832Z"}
> {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=8c935d35 slot=7223582 validator\_index=519759 committee\_index=20 committee\_length=373 committees\_at\_slot=64 validator\_committee\_index=362 epoch=225736 dependent\_root=da7ad193 service=duties\_servi
> ce\\n","stream":"stdout","time":"2023-08-31T18:24:23.735631562Z"}
> {"log":"ERR 2023-08-31 18:24:24.123+00:00 Beacon node provides unexpected response reason=\\"Serialization error;200;getValidatorsActivity();unexpected-data\\" node=http://10.12.1.63:5051\[teku/v23.8.0/linux-x86\_64/-eclipseadoptium-openjdk64bitservervm-java-17\] node\_index=0 node\_roles=A
> GBSDT\\n","stream":"stdout","time":"2023-08-31T18:24:24.123851353Z"}
> {"log":"NOT 2023-08-31 18:24:25.604+00:00 Beacon node is online agent\_version=teku/v23.8.0/linux-x86\_64/-eclipseadoptium-openjdk64bitservervm-java-17 node=http://10.12.1.63:5051\[teku/v23.8.0/linux-x86\_64/-eclipseadoptium-openjdk64bitservervm-java-17\] node\_index=0 n
> ode\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-31T18:24:25.604645362Z"}
> {"log":"NOT 2023-08-31 18:24:25.608+00:00 Beacon node is compatible node=http://10.12.1.63:5051\[teku/v23.8.0/linux-x86\_64/-eclipseadoptium-openjdk64bitservervm-java-17\] node\_index=0 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-31T18:24:25.608599933Z"}
> {"log":"NOT 2023-08-31 18:24:25.611+00:00 Beacon node is in sync head\_slot=7223520 sync\_distance=0 is\_optimistic=false node=http://10.12.1.63:5051\[teku/v23.8.0/linux-x86\_64/-eclipseadoptium-openjdk64bitservervm-java-17\] node\_index=0 node\_roles=AGBSDT\\n","stream":"st
> dout","time":"2023-08-31T18:24:25.612457281Z"}
> {"log":"NOT 2023-08-31 18:24:27.018+00:00 Sync committee message published message=\\"(slot: 7223520, beacon\_block\_root: \\\\\\"3c756965\\\\\\", validator\_index: 486083, signature: \\\\\\"b1cf4ea6\\\\\\")\\" validator=9133d249 validator\_index=486083 delay=18ms301us808ns service=sync\_commit
> tee\_service\\n","stream":"stdout","time":"2023-08-31T18:24:27.018828953Z"}
> {"log":"NOT 2023-08-31 18:24:27.741+00:00 Attestation published delay=741ms469us863ns service=attestation\_service validator=b33f3ae5@506454 attestation=\\"(aggregation\_bits: 0b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> 0000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, d
> ata: (slot: 7223520, index: 46, beacon\_block\_root: \\\\\\"3c756965\\\\\\", source: \\\\\\"225734:4a744323\\\\\\", target: \\\\\\"225735:3c756965\\\\\\"), signature: \\\\\\"8b100faf\\\\\\")\\"\\n","stream":"stdout","time":"2023-08-31T18:24:27.741906386Z"}
> {"log":"WRN 2023-08-31 18:24:29.291+00:00 Beacon node down reason=\\"Communication failed while sending request's body, stream error \[AsyncStreamIncompleteError\]: Incomplete data sent or received;n/a;submitPoolAttestations(first);communication\\" node=\\"http://1
> 0.12.1.66:3500\[prysm/v4.0.8 (linux amd64)\]\\" node\_index=3 node\_roles=AGBSDT\\n","stream":"stdout","time":"2023-08-31T18:24:29.292162114Z"}
> {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms187us602ns service=attestation\_service validator=8606ba05@506455 attestation=\\"(aggregation\_bits: 0b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
> data: (slot: 7223520, index: 33, beacon\_block\_root: \\\\\\"3c756965\\\\\\", source: \\\\\\"225734:4a744323\\\\\\", target: \\\\\\"225735:3c756965\\\\\\"), signature: \\\\\\"b0418aa3\\\\\\")\\"\\n","stream":"stdout","time":"2023-08-31T18:24:29.292304612Z"}
> {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms485us634ns service=attestation\_service validator=ac64a945@515716 attestation=\\"(aggregation\_bits: 0b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
> data: (slot: 7223520, index: 43, beacon\_block\_root: \\\\\\"3c756965\\\\\\", source: \\\\\\"225734:4a744323\\\\\\", target: \\\\\\"225735:3c756965\\\\\\"), signature: \\\\\\"b2119b76\\\\\\")\\"\\n","stream":"stdout","time":"2023-08-31T18:24:29.292635326Z"}
> {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms574us222ns service=attestation\_service validator=b3ff1ce0@517195 attestation=\\"(aggregation\_bits: 0b0000000000000000000000000000000000000000000000000001000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
> data: (slot: 7223520, index: 3, beacon\_block\_root: \\\\\\"3c756965\\\\\\", source: \\\\\\"225734:4a744323\\\\\\", target: \\\\\\"225735:3c756965\\\\\\"), signature: \\\\\\"952a259e\\\\\\")\\"\\n","stream":"stdout","time":"2023-08-31T18:24:29.292651758Z"}
> {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms650us966ns service=attestation\_service validator=833a8de9@506588 attestation=\\"(aggregation\_bits: 0b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
> data: (slot: 7223520, index: 32, beacon\_block\_root: \\\\\\"3c756965\\\\\\", source: \\\\\\"225734:4a744323\\\\\\", target: \\\\\\"225735:3c756965\\\\\\"), signature: \\\\\\"a415ee51\\\\\\")\\"\\n","stream":"stdout","time":"2023-08-31T18:24:29.292814604Z"}
> {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms957us645ns service=attestation\_service validator=83365bed@519763 attestation=\\"(aggregation\_bits: 0b0000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
> 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
> , data: (slot: 7223520, index: 17, beacon\_block\_root: \\\\\\"3c756965\\\\\\", source: \\\\\\"225734:4a744323\\\\\\", target: \\\\\\"225735:3c756965\\\\\\"), signature: \\\\\\"a0308734\\\\\\")\\"\\n","stream":"stdout","time":"2023-08-31T18:24:29.293072411Z"}
> \`\`\`
OBSERVATIONS OF POOR COMPATIBILITY
We also observed this bad sync participation with non-Nimbus fallback nodes on our systems, and a generally bad compatibility with non-Nimbus nodes.
These observations lead us to planning new fallback combinations, bearing especially those combinations in mind that show poor compatibility.
INCREASE OF HARDWARE PERFORMANCE
By optimizing the configuration of our hardware, we managed to increase its performance.
Also, we exchanged faulty hardware - without any downtime, due to a redundant setup.
CLIENT COMPATIBILITY
Due to the experiences in the past months, we redesign failover setups mixing validator clients and consensus clients right now. The changes will be made public in our monthly notes as we roll them out.
OBOL
Just in case we already started an Obol mainnet cluster and hope to use this new technology soon for Lido too.
SSV
For the SSV mainnet we prepared a mainnet setup after having participated in the testnets for some time now. For this we see the possibility for using it for Lido, too.
Thank you for your interest in our work!
Roland for the RockLogic Node Operator Team