Updates to the Blocknative Relay

Node Operators
1

On Friday, December 8th we noticed occasional spikes in our relay’s processing time on getPayload. Upon investigation, we discovered the following:

  1. Increased validator registration load near epoch boundaries was consuming enough resources to impact header/payload request processing.
  2. Some consensus clients (e.g. Prysm, Teku) have a hardcoded timeout on getPayload of 3 seconds vs. the MEV-Boost 4 second timeout. So our understanding of timeout budgets was incomplete.
  3. On slots where we saw the spike in getPayload, validator’s running consensus client’s with a 3 second timeout experienced a timeout error triggered by their consensus client because the Blocknative relay didn’t deliver within 3 seconds, causing them to miss the slot.

On Monday, Dec 12th ~1800 UTC, Blocknative deployed an update to our relay to better handle validator registration congestion on the Blocknative Relay at the edge of each epoch. This allows the Blocknative Relay to scale as we onboard more validators and deliver on a higher getPayload rate. Since this release, we are seeing significantly improved getPayload response times.

We continue to make improvements to our relay implementation and supporting infrastructure to ensure accurate and timely responses on all our APIs.

We have started a discussion in the Flashbots forum regarding the discrepancy in timeout for getPayload between the consensus client and MEV-Boost to reduce any confusion on where timeouts are happening. Feel free to let us know if you have any questions.

5 Likes
3

As a followup to the above:

When some validators experienced a timeout error triggered by their consensus client having a 3 second timeout, this was most likely due to the roundtrip of the getPayload taking longer than 3 seconds, causing them to miss the slot. This led to the following missed slots between Dec 8 and 12th:

  1. 5303360: 2032ms request duration
  2. 5311904: 2038ms request duration
  3. 5315808: 2491ms request duration
  4. 5321632: 2431ms request duration
  5. 5329824: 2664ms request duration
  6. 5334784: 1241ms request duration

Prior to this release, the Blocknative Relay had an average request duration of 28.627ms across blocks and a p99 of 1095.2ms (data from Dec 5-9). Since this release, the getPayload request duration averages at 7.76ms, a reduction of 72%, with a p99 of 33.77ms, a reduction of 96%.

1 Like