BloXroute - Feb 6th Post Mortem

Adding a post here for some more surface area of the issue that arose on the (EST) morning of Tuesday the 6th. There was a bug in the automatic demotion logic of our max-profit relay that resulted in 9 missed slots due to an improperly formatted block.

Of those 9 missed slots only one was a Lido validator and bloXroute has covered the cost of all ETH lost due to the bug.

Below is a couple links and info about the event but if you have any questions please feel free to reach out to us or reply to this post and I’ll fill in more details as available.

Here is the TX hash for the payment to Lido for the missed slot: 0xe9ac6633e870e554c336a26f3e3933b8dc563f11783dbc1abc68f210b033ac8d

A full post mortem is available on our discord here:

Invite: bloXroute Labs
Link: Discord

Also we have a gist with a bit more information:

5 Likes

Thank you Ben for the sharing the details of the incident, in line with the importance of the communication expectations from relay operators towards node operators, and the compensation to stakers, and the prompt response to investigating the incident despite the unfortunate bug.

3 Likes