Blocknative Relay proposer public key update

Node Operators
1

Events

Earlier today, Friday, March 10, blocks sent through the Blocknative Relay began sending invalid proposer public keys. The blocks were committed by Validators but on getPayload requests the relay could not deliver a block as it was stored under an invalid proposer public key. A fix was developed ( Add proposer pubkey check in getheader by aratz-lasa · Pull Request #103 · blocknative/dreamboat · GitHub) and deployed to Goerli and Mainnet at 13:30 UTC on Friday, March 10, and the issue has been resolved. This fix prevents getHeader requests for blocks with invalid proposer public keys. The Blocknative Relay no longer delivers bids if the block does not have the correct proposer public key .

Impact

We are aware of several slots that were missed due to an invalid proposer public key . We are continuing to determine the impact on validators and slots.

Improvements

4 Likes
2

Thank you for the incident update. I can confirm that from our side it looks like normal operation has resumed, so NOs may consider re-enabling the blocknative relay.

4 Likes
3

Thank you for the write up. It doesn’t explain why the requests were being stored under an “invalid proposer public key” nor what had changed to cause this given everything was working fine previously?

1 Like
4

The Blocknative team analyzed the reported missed slots by Lido and determined that 17 slots were missed due to a bug in the validation of blocks. We reimbursed the total rewards (0.934127638 ETH) for those slots to the Lido Execution Layer Rewards Vault contract, 0x388C818CA8B9251b393131C08a736A67ccB19297, in the following transaction: 0xb9afce10d52fabb9065cbbae69947750fa5f15695c1073f50c077f015a93ae87.

We determined the expected rewards by looking at our bid at the end of the proposal period for each missed slot (12 second mark):

  • 5966288; 0.025005669 ETH
  • 5961902; 0.050399232 ETH
  • 5963874; 0.066077672 ETH
  • 5966857; 0.059810491 ETH
  • 5966915; 0.030232819 ETH
  • 5965996; 0.103487019 ETH
  • 5966989; 0.027248057 ETH
  • 5967046; 0.019968912 ETH
  • 5967087; 0.059889327 ETH
  • 5967821; 0.093399725 ETH
  • 5967826; 0.019235052 ETH
  • 5968113; 0.126925462 ETH
  • 5968129; 0.039550142 ETH
  • 5968146; 0.028233159 ETH
  • 5968151; 0.094648156 ETH
  • 5968239; 0.064705085 ETH
  • 5968937; 0.025311660 ETH

Total: 0.934127638 ETH

The fix was implemented within 7 hours of initial issue reporting, which you can see in the public repository here: Add proposer pubkey check in getheader by aratz-lasa · Pull Request #103 · blocknative/dreamboat · GitHub. This fix resolved responding to a getHeader request with an invalid block hash because the submitted block had an incorrect proposer public key.

The validation error was exposed during a phased release in alpha testing external (non-Blocknative) builders submitting to the Blocknative Relay. We suspended the alpha testing on Friday, March 17 at 17:00 UTC to conduct a comprehensive review of the relay code and further enhance validation of submitted blocks. We expect to re-enable external block builder submissions following successful completion of this evaluation.

2 Likes
6

Thank you @ldelisle_blocknative for the post-mortem and insight into the fix, we very much appreciate the transparency and the follow-up on this incident.

3 Likes