Hey Friends, I am writing this post to reference a platform bug we have identified back in July this year as well as our immediate actions to mitigate and eradicate it since then. On top of the internal review, we have since engaged accredited external auditors to further assess our system and organization controls, as a precautionary measure.
The integrity of the Lido ecosystem is top of our priority and should any slashing happen because of this bug, our team will take responsibility and reimburse the vault.
In line with our commitment to continuous improvement and transparency, this post will address a potential vulnerability identified and resolved in July 2023. As we dive into the details, we want to assure everyone that there were no adverse effects on our operation for Lido following the resolution. We have conducted a thorough data integrity check as well as extensive protocol assessment and upgrade across all systems and nodes to ensure no compromise of sensitive information during the incident. On top of the internal review, we have since engaged accredited external auditors to further assess our system and organization controls, as a precautionary measure. This led to the attainment of our SOC 2 Type 1 attestation. This attestation, aligning with AICPA standards, confirms the strength of our security, availability, and processing integrity controls. In addition, it is also important to note that the non-custodial nature of our platform inherently safeguards our clients’ assets, preventing total exposure to any platform vulnerabilities.
2 The Scope
The vulnerability, associated with an open-source third-party library called Tailon, was disclosed to us in early July 2023. We immediately identified that 237 instances were in scope, of which 212 instances were deployed for our development and testing purposes, and 25 freshly deployed instances in the production environment. The instances identified in production constitute a fraction below 0.1% of the live nodes we have launched to date. But more importantly, out of the production instances affected, NONE was used to deploy keys related to Lido. Postmortem investigation reveals no exploitation of this vulnerability within our systems.
3 Our Immediate Actions
3.1 Actions In General
Following the discovery, our initial action swiftly removed a third party library named Tailon from our system, which was crucial in eliminating any immediate unauthorized access risks. Further, we conducted a thorough data integrity check across all systems and nodes to ensure no compromise of sensitive information during the incident. As a precaution, we performed an extensive protocol assessment and upgrades, including building binaries directly from the source and redeploying using fresh copies of blockchain data.
On top of addressing the issues directly, we conducted a thorough access control review and restricted system access to authorized personnel based on the need-to-know principles. Moreover, we invalidated and rotated all credentials on affected node instances to mitigate any potential exposure and secure our system against latent threats.
Beyond these measures, we craft security roadmaps, refine vulnerability reporting procedures, and enhance system controls, aiming not only to address immediate issues but also to integrate new best practices into our policies for the future.
3.2 Actions Related to Lido
Our postmortem investigation reveals that NO exploitation of this vulnerability has occurred in any Lido-related instances. Following the actions mentioned above, we conducted a complete redeployment of all instances hosting Lido-related keys and credentials as a precautionary measure. No further exploitation is possible for those keys post-redeployment.
4 Impact and Follow-up
a). Operation Integrity
We have previously mentioned that our postmortem investigation reveals NO exploitation of this vulnerability in any Lido-related instances. In addition to enhanced security measures, the ETH validator keys are not only secured by their own passphrase when generated but also by the wallet passphrase for the validator client. Based on the evidence we have gathered at this time, we believe that none of the Lido validator keys were leaked.
b). Asset Security
Without delving into the intricacies of blockchain delegation and ETH staking, as Lido operators, we can confirm that all validator withdraw credentials are mandated to be set to the Withdraw Vault. It is impossible to add a key with a different withdrawal credential to the registry. Therefore, none of the ETH staked with Lido can be withdrawn by potential attackers, despite some false claims that may suggest otherwise. For instance, claims such as ‘an attacker could have stolen around 145 million dollars’ from InfStones’ Aptos validator are factually incorrect, as validators do not control delegators’ asset keys. Such assertions are aimed at grabbing attention.
We are committed to working closely with the DAO to take any necessary further steps to ensure the community’s peace of mind.
4.2 Looking Into The Future
In response to the vulnerability and our initial containment measures, we have further strengthened our platform’s security through ongoing initiatives and preventive measures. Key among these is the attainment of SOC 2 Type I attestation, a significant milestone that confirms our compliance with AICPA standards. This certification not only demonstrates our adherence to stringent security and availability controls but also enhances client trust and assures transparency in our operations.
Additionally, we launched the BlocGuardians Bug Bounty Program, inviting cybersecurity experts and enthusiasts to help identify and mitigate potential security weaknesses. This program is a testament to our proactive approach in maintaining a resilient and secure platform, especially in the dynamic and complex Web3 landscape. It underscores our commitment to leveraging community expertise for continuous improvement in our security practices, thereby safeguarding user data and assets against emerging threats.
4.3 What’s next?
a) Temporary Lower Validator Cap On InfStones Account
We have reduced our validator number cap to match the currently active number of validators managed by InfStones on the same day as this post.
Transaction: Ethereum Transaction Hash (Txhash) Details | Etherscan
b) Key Rotation
We are going to start rotating ALL keys associated with InfStones. Meanwhile, we will be actively engaged with the DAO in the forum to reach a consensus for future steps.
If still interested in more details please check article here: From Vulnerability to Vigilance: Insights on a Recent Bug and Continued Safeguards