Hi Izzy, thanks for the questions, answers inline.
We went for a custom tool over using Vouch because Vouch was only validator oriented (to get the block proposals from the different blocks, you needed to have a validator with a valid duty in a slot to track the scores). In addition, having a custom tool allowed us to remove some of the complexity of Vouch, add custom SQL indexers for the results, and track the score of every single slot. Also, we thought that having our own software would enable us to understand better how the tool works, which would give more credibility to the study and make it easier for us to explain the results. We have added the functionality to track attestation reception for now, and in the future, we want to add reorg detection. For the rest, the software works the same way as Vouch and even uses the same scoring algorithm, as you may see the references in our GitHub repository.
Nothing conclusive yet, but it all has to do with which attestations you include in the block. Block proposal rewards come from sync committee aggregations (which can only be included in the block it was meant to, otherwise those aggregations are dismissed), and from attestation aggregations. So it seems that Lighthouse is choosing better which attestations to include when you have double the number of attestations from where to pick, probably due to better aggregation tracking. Once again, nothing proved yet, we didn’t dig into the code to demonstrate it.
Mostly Nethermind 1.14.7
Teku sometimes gets out of sync and after a couple of minutes, it recovers. Lodestar usually includes more attestations than the others but does not get higher rewards (probably redundant attestations or attestations already included). Lighthouse also sometimes does not answer but this hardly happens.
Also, sometimes if we restart the CL, then we also need to restart the Nethermind, because it does not detect anymore the CL.
Regarding EL+CL “pairs” we discussed with Teku developers about the difference in performance of Teku nodes in the wild, vs Teku nodes in the CIP program (significantly lower performance). And it seems to be related to the fact that the CIP Teku nodes run with Besu, which is very slow compared to Geth or Nethermind.
We continue working on this and gathering data, we will continue updating this thread with new posts.