RockLogic Monthly Notes - 09/2023

RRoland · September 7, 2023, 9:52am

Dear Lido Community,

welcome to our monthly notes, updating you about our activities to maintain a safe Ethereum environment with Stereum.

SOFTWARE UPDATES
All full nodes and validator clients have been updated.
We started investigations regarding a Nimbus validator client performing poorly with non-Nimbus fallback nodes. More information about this Nimbus crash investigation:

github.com/status-im/nimbus-eth2

Missed Duties followed by a crash of nimbus VC

opened 01:23PM - 29 Aug 23 UTC

closed 05:32PM - 29 Aug 23 UTC

NeoPlays

**Describe the bug** Nimbus VC recognizing time offset followed by missed dutie…s and resulting in a crash. The VC was running just fine beforehand (no time offsets, no missed duties) **To Reproduce** Nimbus is running in Docker with `statusim/nimbus-validator-client:multiarch-v23.7.0` image Logs: ``` {"log":"INF 2023-08-29 03:56:47.004+00:00 Slot start slot=7204782 epoch=225149 attestationIn=4s blockIn=\u003cunknown\u003e validators=500 good_nodes=2 viable_nodes=0 bad_nodes=4 delay=4ms800us139ns\n","stream":"stdout","time":"2023-08-29T03:56:47.005291 403Z"} {"log":"INF 2023-08-29 03:56:59.003+00:00 Slot start slot=7204783 epoch=225149 attestationIn=4s blockIn=\u003cunknown\u003e validators=500 good_nodes=2 viable_nodes=0 bad_nodes=4 delay=3ms40us618ns\n","stream":"stdout","time":"2023-08-29T03:56:59.0035494 45Z"} {"log":"INF 2023-08-29 03:57:04.676+00:00 Beacon node has notable time offset time_offset=596ms699us238ns service=fallback_service node=http://10.12.1.62:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=5 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:04.6765380 71Z"} {"log":"WRN 2023-08-29 03:57:05.877+00:00 Beacon node has significant time offset time_offset=2s832ms261us118ns service=fallback_service node=http://10.12.1.55:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:05.878121314Z"} {"log":"WRN 2023-08-29 03:57:05.877+00:00 Beacon node has enormous time offset time_offset=2s832ms261us118ns service=fallback_service node=http://10.12.1.55:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:05.878157704Z"} {"log":"WRN 2023-08-29 03:57:05.878+00:00 Beacon node's clock is out of order, (beacon node is unusable) node=http://10.12.1.55:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:05.878164366Z"} {"log":"INF 2023-08-29 03:57:11.542+00:00 Slot start slot=7204784 epoch=225149 attestationIn=4s blockIn=\u003cunknown\u003e validators=500 good_nodes=1 viable_nodes=0 bad_nodes=4 delay=542ms471us289ns\n","stream":"stdout","time":"2023-08-29T03:57:11.542851935Z"} {"log":"WRN 2023-08-29 03:57:11.670+00:00 Beacon node down reason=\"Beacon node has acceptable time offset;200;checkTimeOffsetStatus()();status-update\" node=http://10.12.1.55:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:11.670298428Z"} {"log":"INF 2023-08-29 03:57:12.853+00:00 Beacon node has notable time offset time_offset=913ms852us823ns service=fallback_service node=http://10.12.1.62:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=5 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:12.853519981Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=17 duties_count=3 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007327417Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=23 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007369927Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=30 duties_count=2 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007375277Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=11 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007386307Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=47 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007391237Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=0 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.00739827Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=39 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007406305Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=27 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007472891Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=45 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007479824Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=25 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007518005Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=32 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007534186Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=43 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007586223Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=4 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007598677Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=9 duties_count=2 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007638783Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=38 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007676734Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=1 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007739091Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=55 duties_count=1 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007782843Z"} {"log":"WRN 2023-08-29 03:57:19.007+00:00 Unable to proceed attestations slot=7204784 committee_index=20 duties_count=2 reason=\"No online beacon node(s)\" service=attestation_service\n","stream":"stdout","time":"2023-08-29T03:57:19.007795578Z"} {"log":"NOT 2023-08-29 03:57:21.410+00:00 Beacon node is online agent_version=Nimbus/v23.7.0-60f0a2-stateofus node=http://10.12.1.62:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=5 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:21.41085529Z"} {"log":"NOT 2023-08-29 03:57:21.428+00:00 Beacon node is online agent_version=Nimbus/v23.7.0-60f0a2-stateofus node=http://10.12.1.55:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:21.428594517Z"} {"log":"NOT 2023-08-29 03:57:21.480+00:00 Beacon node is compatible node=http://10.12.1.62:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=5 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:21.480468129Z"} {"log":"NOT 2023-08-29 03:57:21.482+00:00 Beacon node is compatible node=http://10.12.1.55:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:21.482453775Z"} {"log":"NOT 2023-08-29 03:57:21.503+00:00 Beacon node is in sync head_slot=7204784 sync_distance=0 is_optimistic=false node=http://10.12.1.55:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:21.503179259Z"} {"log":"NOT 2023-08-29 03:57:21.505+00:00 Beacon node is in sync head_slot=7204784 sync_distance=0 is_optimistic=false node=http://10.12.1.62:5052[Nimbus/v23.7.0-60f0a2-stateofus] node_index=5 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-29T03:57:21.505819437Z"} {"log":"INF 2023-08-29 03:57:23.322+00:00 Slot start slot=7204785 epoch=225149 attestationIn=4s blockIn=\u003cunknown\u003e validators=500 good_nodes=2 viable_nodes=0 bad_nodes=4 delay=322ms496us772ns\n","stream":"stdout","time":"2023-08-29T03:57:23.322832844Z"} {"log":"WRN 2023-08-29 03:57:23.328+00:00 Unable to publish sync committee messages and contributions in time slot=7204784 timeout=11s456ms906us700ns service=sync_committee_service\n","stream":"stdout","time":"2023-08-29T03:57:23.328831842Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(865) main\n","stream":"stderr","time":"2023-08-29T03:57:25.021216691Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(858) NimMain\n","stream":"stderr","time":"2023-08-29T03:57:25.021256956Z"} {"log":"/home/user/nimbus-eth2/beacon_chain/nimbus_validator_client.nim(521) main\n","stream":"stderr","time":"2023-08-29T03:57:25.021261705Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(1504) waitFor\n","stream":"stderr","time":"2023-08-29T03:57:25.021265562Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(258) poll\n","stream":"stderr","time":"2023-08-29T03:57:25.021268959Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(318) futureContinue\n","stream":"stderr","time":"2023-08-29T03:57:25.021272505Z"} {"log":"/home/user/nimbus-eth2/beacon_chain/validator_client/api.nim(233) getAttesterDuties\n","stream":"stderr","time":"2023-08-29T03:57:25.021276212Z"} {"log":"/home/user/nimbus-eth2/beacon_chain/validator_client/api.nim(117) apiResponseOr\n","stream":"stderr","time":"2023-08-29T03:57:25.02127999Z"} {"log":"/home/user/nimbus-eth2/beacon_chain/validator_client/api.nim(120) apiResponseOr\n","stream":"stderr","time":"2023-08-29T03:57:25.021283446Z"} {"log":"/home/user/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/assertions.nim(38) failedAssertImpl\n","stream":"stderr","time":"2023-08-29T03:57:25.021287013Z"} {"log":"/home/user/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/assertions.nim(28) raiseAssert\n","stream":"stderr","time":"2023-08-29T03:57:25.021290509Z"} {"log":"/home/user/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(54) sysFatal\n","stream":"stderr","time":"2023-08-29T03:57:25.021294137Z"} {"log":"[[reraised from:\n","stream":"stderr","time":"2023-08-29T03:57:25.021297473Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(865) main\n","stream":"stderr","time":"2023-08-29T03:57:25.021300749Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-libp2p/libp2p/protocols/pubsub/pubsub.nim(858) NimMain\n","stream":"stderr","time":"2023-08-29T03:57:25.021304026Z"} {"log":"/home/user/nimbus-eth2/beacon_chain/nimbus_validator_client.nim(521) main\n","stream":"stderr","time":"2023-08-29T03:57:25.021307452Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(1504) waitFor\n","stream":"stderr","time":"2023-08-29T03:57:25.021310929Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(258) poll\n","stream":"stderr","time":"2023-08-29T03:57:25.021314245Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(318) futureContinue\n","stream":"stderr","time":"2023-08-29T03:57:25.021317641Z"} {"log":"/home/user/nimbus-eth2/vendor/nim-chronos/chronos/asyncmacro2.nim(197) getAttesterDuties\n","stream":"stderr","time":"2023-08-29T03:57:25.021320867Z"} {"log":"]]\n","stream":"stderr","time":"2023-08-29T03:57:25.021324264Z"} {"log":"Error: unhandled exception: /home/user/nimbus-eth2/beacon_chain/validator_client/api.nim(120, 13) `not (future.cancelled())` [AssertionDefect]\n","stream":"stderr","time":"2023-08-29T03:57:25.02132757Z"} ```

github.com/status-im/nimbus-eth2

Nimbus VC continuously loses connection to Prysm and Teku BNs

opened 06:40PM - 31 Aug 23 UTC

NeoPlays

**Describe the bug** Nimbus VC continuously drops Prysm and Teku BN connections…. For Prysm this normally happens after a lot of attestations are published and after VC receives new attester duties. For Teku this only happens after receiving attester duties. This is on Mainnet holding 500 keys. Same setup on testnet with 1 key works fine. running in docker: `consensys/teku:23.8.0` `prysmaticlabs/prysm-beacon-chain:v4.0.8` `nimbus-validator-client:multiarch-v23.7.0` Logs: ``` {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=a6dc555e slot=7223569 validator_index=514033 committee_index=31 committee_length=374 committees_at_slot=64 validator_committee_index=119 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.735339392Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=81874181 slot=7223567 validator_index=514044 committee_index=18 committee_length=373 committees_at_slot=64 validator_committee_index=294 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.735366031Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=abbd30f1 slot=7223557 validator_index=506733 committee_index=30 committee_length=373 committees_at_slot=64 validator_committee_index=136 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.735421667Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=883e1861 slot=7223582 validator_index=517741 committee_index=36 committee_length=374 committees_at_slot=64 validator_committee_index=275 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.735427488Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=a5755039 slot=7223559 validator_index=506140 committee_index=8 committee_length=373 committees_at_slot=64 validator_committee_index=223 epoch=225736 dependent_root=da7ad193 service=duties_servic e\n","stream":"stdout","time":"2023-08-31T18:24:23.735471561Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=af495a90 slot=7223564 validator_index=506898 committee_index=33 committee_length=373 committees_at_slot=64 validator_committee_index=357 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.73551859Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=81ba9640 slot=7223554 validator_index=506896 committee_index=62 committee_length=373 committees_at_slot=64 validator_committee_index=24 epoch=225736 dependent_root=da7ad193 service=duties_servic e\n","stream":"stdout","time":"2023-08-31T18:24:23.735532486Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=982a9814 slot=7223579 validator_index=514172 committee_index=10 committee_length=373 committees_at_slot=64 validator_committee_index=363 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.735574475Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=878460da slot=7223581 validator_index=486544 committee_index=15 committee_length=373 committees_at_slot=64 validator_committee_index=318 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.735625832Z"} {"log":"INF 2023-08-31 18:24:23.735+00:00 Received new attester duty pubkey=8c935d35 slot=7223582 validator_index=519759 committee_index=20 committee_length=373 committees_at_slot=64 validator_committee_index=362 epoch=225736 dependent_root=da7ad193 service=duties_servi ce\n","stream":"stdout","time":"2023-08-31T18:24:23.735631562Z"} {"log":"ERR 2023-08-31 18:24:24.123+00:00 Beacon node provides unexpected response reason=\"Serialization error;200;getValidatorsActivity();unexpected-data\" node=http://10.12.1.63:5051[teku/v23.8.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17] node_index=0 node_roles=A GBSDT\n","stream":"stdout","time":"2023-08-31T18:24:24.123851353Z"} {"log":"NOT 2023-08-31 18:24:25.604+00:00 Beacon node is online agent_version=teku/v23.8.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17 node=http://10.12.1.63:5051[teku/v23.8.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17] node_index=0 n ode_roles=AGBSDT\n","stream":"stdout","time":"2023-08-31T18:24:25.604645362Z"} {"log":"NOT 2023-08-31 18:24:25.608+00:00 Beacon node is compatible node=http://10.12.1.63:5051[teku/v23.8.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17] node_index=0 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-31T18:24:25.608599933Z"} {"log":"NOT 2023-08-31 18:24:25.611+00:00 Beacon node is in sync head_slot=7223520 sync_distance=0 is_optimistic=false node=http://10.12.1.63:5051[teku/v23.8.0/linux-x86_64/-eclipseadoptium-openjdk64bitservervm-java-17] node_index=0 node_roles=AGBSDT\n","stream":"st dout","time":"2023-08-31T18:24:25.612457281Z"} {"log":"NOT 2023-08-31 18:24:27.018+00:00 Sync committee message published message=\"(slot: 7223520, beacon_block_root: \\\"3c756965\\\", validator_index: 486083, signature: \\\"b1cf4ea6\\\")\" validator=9133d249 validator_index=486083 delay=18ms301us808ns service=sync_commit tee_service\n","stream":"stdout","time":"2023-08-31T18:24:27.018828953Z"} {"log":"NOT 2023-08-31 18:24:27.741+00:00 Attestation published delay=741ms469us863ns service=attestation_service validator=b33f3ae5@506454 attestation=\"(aggregation_bits: 0b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, d ata: (slot: 7223520, index: 46, beacon_block_root: \\\"3c756965\\\", source: \\\"225734:4a744323\\\", target: \\\"225735:3c756965\\\"), signature: \\\"8b100faf\\\")\"\n","stream":"stdout","time":"2023-08-31T18:24:27.741906386Z"} {"log":"WRN 2023-08-31 18:24:29.291+00:00 Beacon node down reason=\"Communication failed while sending request's body, stream error [AsyncStreamIncompleteError]: Incomplete data sent or received;n/a;submitPoolAttestations(first);communication\" node=\"http://1 0.12.1.66:3500[prysm/v4.0.8 (linux amd64)]\" node_index=3 node_roles=AGBSDT\n","stream":"stdout","time":"2023-08-31T18:24:29.292162114Z"} {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms187us602ns service=attestation_service validator=8606ba05@506455 attestation=\"(aggregation_bits: 0b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, data: (slot: 7223520, index: 33, beacon_block_root: \\\"3c756965\\\", source: \\\"225734:4a744323\\\", target: \\\"225735:3c756965\\\"), signature: \\\"b0418aa3\\\")\"\n","stream":"stdout","time":"2023-08-31T18:24:29.292304612Z"} {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms485us634ns service=attestation_service validator=ac64a945@515716 attestation=\"(aggregation_bits: 0b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, data: (slot: 7223520, index: 43, beacon_block_root: \\\"3c756965\\\", source: \\\"225734:4a744323\\\", target: \\\"225735:3c756965\\\"), signature: \\\"b2119b76\\\")\"\n","stream":"stdout","time":"2023-08-31T18:24:29.292635326Z"} {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms574us222ns service=attestation_service validator=b3ff1ce0@517195 attestation=\"(aggregation_bits: 0b0000000000000000000000000000000000000000000000000001000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, data: (slot: 7223520, index: 3, beacon_block_root: \\\"3c756965\\\", source: \\\"225734:4a744323\\\", target: \\\"225735:3c756965\\\"), signature: \\\"952a259e\\\")\"\n","stream":"stdout","time":"2023-08-31T18:24:29.292651758Z"} {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms650us966ns service=attestation_service validator=833a8de9@506588 attestation=\"(aggregation_bits: 0b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, data: (slot: 7223520, index: 32, beacon_block_root: \\\"3c756965\\\", source: \\\"225734:4a744323\\\", target: \\\"225735:3c756965\\\"), signature: \\\"a415ee51\\\")\"\n","stream":"stdout","time":"2023-08-31T18:24:29.292814604Z"} {"log":"NOT 2023-08-31 18:24:29.292+00:00 Attestation published delay=2s292ms957us645ns service=attestation_service validator=83365bed@519763 attestation=\"(aggregation_bits: 0b0000000000000000000000000000000000000000100000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 , data: (slot: 7223520, index: 17, beacon_block_root: \\\"3c756965\\\", source: \\\"225734:4a744323\\\", target: \\\"225735:3c756965\\\"), signature: \\\"a0308734\\\")\"\n","stream":"stdout","time":"2023-08-31T18:24:29.293072411Z"} ```

OBSERVATIONS OF POOR COMPATIBILITY
We also observed this bad sync participation with non-Nimbus fallback nodes on our systems, and a generally bad compatibility with non-Nimbus nodes.
These observations lead us to planning new fallback combinations, bearing especially those combinations in mind that show poor compatibility.

INCREASE OF HARDWARE PERFORMANCE
By optimizing the configuration of our hardware, we managed to increase its performance.
Also, we exchanged faulty hardware - without any downtime, due to a redundant setup.

CLIENT COMPATIBILITY
Due to the experiences in the past months, we redesign failover setups mixing validator clients and consensus clients right now. The changes will be made public in our monthly notes as we roll them out.

OBOL
Just in case we already started an Obol mainnet cluster and hope to use this new technology soon for Lido too.

SSV
For the SSV mainnet we prepared a mainnet setup after having participated in the testnets for some time now. For this we see the possibility for using it for Lido, too.

Thank you for your interest in our work!
Roland for the RockLogic Node Operator Team

Topic	Replies	Views
RockLogic Monthly Notes - 11/2023 Node Operators	1062	November 8, 2023
RockLogic Monthly Notes - 10/2023 Node Operators	1149	October 4, 2023
RockLogic Monthly Notes - 05/2024 Node Operators	416	May 15, 2024
RockLogic Monthly Notes - 01/2024 Node Operators	809	January 10, 2024
RockLogic Monthly Notes - 06/2024 Node Operators	254	June 20, 2024

RockLogic Monthly Notes - 09/2023

Related topics