Detailed response for the Lido Community
This response is in two main parts:
- a detailed summary with some commentary
- a personal statement
FOSS Relay
We have a monorepo, our entire infrastructure is designed, built, deployed and managed that way. Though this argument is a strawman: just because the code is open source does not mean that it is being ran 1:1 (i.e. it is not being ran modified, etc).
Post Mortem response
This issue in response was entirely my fault, I had taken a work break after devcon and was traveling. I did not understand the issue at the time, and had mistook it for a previous service disruption that had occurred earlier in the week which was not comparable in magnitude. We had already provided a summary to the operations team at Lido at this event (in which we voluntarily disclosed to them). I confused it for a repeat of the same event and dismissed the need for a timely public statement.
If it is not clear, we are providing 100% restitution. We are currently working with authorities and an external investigator in this matter. The restitution can be coordinated with Lido’s operations team. Note that this does not mean we admit fault in this manner.
Relay and Network Reliability
Building a reliable, robust service often means building something that can keep working when some parts fail. A service where not every feature is available is often better than one that’s entirely offline.
Doing this in a meaningful way is not obvious.
The usual response is to hire more engineering, more support, and even more managers. Error handling, or making components that can recover from faults, often feels like the option of last resort—especially in blockchain networks.
The usual response to error handling is optimism. Unfortunately, the other choices aren’t exactly clear, and often difficult to choose from too. If you have two services, what do you do when one of them is offline: Try again later? Give up entirely? Or just ignore it and hope the problem goes away?
This was my thinking in making those frustrated comments. I no longer interact on Twitter either personally or professionally, nor care to.
Communications and Engagement
We have been considering for more than two months how best to approach this role, and found someone we think will be invaluable to the entire team. We suggest a community call participation at the next regularly scheduled Lido community event or if desired we can do a one-off meeting at whatever time the community finds desirable to have such a call.
Summary
-
We have created a new dedicated Statuspage strictly for SecureRpc
-
Create a dedicated Ethereum Ecosystem RSS aggregation and notification alert page for defect response and alerts. This is meant to provide notifications to node operators of any potential issue affecting them as it relates to software they are running themselves.
-
We have recruited a new business and community lead for coordinating and facilitating our engagements across communities and different projects.
-
Attend and interact with the Lido community during a live community call/gathering within the next 7-14 days. A time that is scheduled can be attended by us.
-
100% Restitution, within the next 7-14 days, as was originally implied in the post mortem.
-
In the coming days we will be making public a few on-chain and off chain solutions that should provide not only improved recovery it will also provide validators with a type of insurance bond protecting both potential losses as well as surplus reward payouts to them.
-
Manifold is also (and has been) ready to operate as a node operator as well, we see no meaningful reason for the relay/operator separation.
-
A 3rd party forensic auditor has been engaged and is helping assist authorities in our internal investigation.
Personal Remarks
This was an offensive attack. An attack on a service that most Ethereum developers will never even work on. This service also happens to offer Ethereum a non-censoring transaction relay that is operating against a potential counter party that is primarily concerned with maintaining its international rules-based system by financial controls for purposes of managing the global economy. We consider this event as strong evidence of battle-space preparation. We know the potential outcomes, we have seen the tactics being used thus far.
Cheers,
Sam