Lido on Ethereum Node Operator (InfStones) Platform Vulnerability Investigation - November 22, 2023

Node Operators
1

Over the course of the last 24 hours, Lido DAO contributors were made aware of a platform vulnerability that affected an active Node Operator using the Lido on Ethereum protocol (InfStones) sometime over the course of the previous few months. The vulnerability was disclosed to InfStones in July 2023 by security researchers dWallet Labs. The Node Operator has announced that the vulnerability has been addressed.

The vulnerability is related to the possible exposure of root-level access to 25 validator servers that may not be related to the Lido protocol, including possibly key material, to external attackers. It is not clear to contributors at this time if servers and/or keys related to Lido validators were included in the scope of affected systems or not.

Lido DAO contributors are now actively working with the Node Operator in investigating the incident to understand its full scope and potential impact.

More information will be shared in the coming hours.

Additional information:

15 Likes
2

Izzy,

Thank you for the detail and transparency provided to the community. We’re happy to see that the vulnerability has been addressed and that the DAO is continuing to investigate the incident.

Our research has found that off-chain attacks have been accelerating both in frequency and magnitude over the last few years, and we believe this latest vulnerability highlights the importance of regular infrastructure auditing to proactively identify and address off-chain vulnerabilities like this one.

We posted a proposal for infrastructure auditing services just two days ago, and we believe the InfStones vulnerability is something that would have been caught by our off-chain team with a simple port scanning process that we execute as part of our annual infrastructure auditing.

Proactive identification of vulnerabilities, particularly off-chain ones, can help ensure the Lido ecosystem remains as safe and secure as possible, and we look forward to the opportunity to leverage our substantial experience to help on this front.

Halborn

1 Like
3

While further investigation is underway, the InfStones team has made an on-chain transaction to reset their vetted validators limit to their currently active amount of validators (i.e. preventing any new deposits from potentially being routed to them), specifically to the current 10,001 validators.

4 Likes
4

Hey Friends, I am writing this post to reference a platform bug we have identified back in July this year as well as our immediate actions to mitigate and eradicate it since then. On top of the internal review, we have since engaged accredited external auditors to further assess our system and organization controls, as a precautionary measure.

The integrity of the Lido ecosystem is top of our priority and should any slashing happen because of this bug, our team will take responsibility and reimburse the vault.

1 Introduction

In line with our commitment to continuous improvement and transparency, this post will address a potential vulnerability identified and resolved in July 2023. As we dive into the details, we want to assure everyone that there were no adverse effects on our operation for Lido following the resolution. We have conducted a thorough data integrity check as well as extensive protocol assessment and upgrade across all systems and nodes to ensure no compromise of sensitive information during the incident. On top of the internal review, we have since engaged accredited external auditors to further assess our system and organization controls, as a precautionary measure. This led to the attainment of our SOC 2 Type 1 attestation. This attestation, aligning with AICPA standards, confirms the strength of our security, availability, and processing integrity controls. In addition, it is also important to note that the non-custodial nature of our platform inherently safeguards our clients’ assets, preventing total exposure to any platform vulnerabilities.

2 The Scope

The vulnerability, associated with an open-source third-party library called Tailon, was disclosed to us in early July 2023. We immediately identified that 237 instances were in scope, of which 212 instances were deployed for our development and testing purposes, and 25 freshly deployed instances in the production environment. The instances identified in production constitute a fraction below 0.1% of the live nodes we have launched to date. But more importantly, out of the production instances affected, NONE was used to deploy keys related to Lido. Postmortem investigation reveals no exploitation of this vulnerability within our systems.

3 Our Immediate Actions

3.1 Actions In General

Following the discovery, our initial action swiftly removed a third party library named Tailon from our system, which was crucial in eliminating any immediate unauthorized access risks. Further, we conducted a thorough data integrity check across all systems and nodes to ensure no compromise of sensitive information during the incident. As a precaution, we performed an extensive protocol assessment and upgrades, including building binaries directly from the source and redeploying using fresh copies of blockchain data.

On top of addressing the issues directly, we conducted a thorough access control review and restricted system access to authorized personnel based on the need-to-know principles. Moreover, we invalidated and rotated all credentials on affected node instances to mitigate any potential exposure and secure our system against latent threats.

Beyond these measures, we craft security roadmaps, refine vulnerability reporting procedures, and enhance system controls, aiming not only to address immediate issues but also to integrate new best practices into our policies for the future.

3.2 Actions Related to Lido

Our postmortem investigation reveals that NO exploitation of this vulnerability has occurred in any Lido-related instances. Following the actions mentioned above, we conducted a complete redeployment of all instances hosting Lido-related keys and credentials as a precautionary measure. No further exploitation is possible for those keys post-redeployment.

4 Impact and Follow-up

4.1 Impact

a). Operation Integrity

We have previously mentioned that our postmortem investigation reveals NO exploitation of this vulnerability in any Lido-related instances. In addition to enhanced security measures, the ETH validator keys are not only secured by their own passphrase when generated but also by the wallet passphrase for the validator client. Based on the evidence we have gathered at this time, we believe that none of the Lido validator keys were leaked.

b). Asset Security

Without delving into the intricacies of blockchain delegation and ETH staking, as Lido operators, we can confirm that all validator withdraw credentials are mandated to be set to the Withdraw Vault. It is impossible to add a key with a different withdrawal credential to the registry. Therefore, none of the ETH staked with Lido can be withdrawn by potential attackers, despite some false claims that may suggest otherwise. For instance, claims such as ‘an attacker could have stolen around 145 million dollars’ from InfStones’ Aptos validator are factually incorrect, as validators do not control delegators’ asset keys. Such assertions are aimed at grabbing attention.

We are committed to working closely with the DAO to take any necessary further steps to ensure the community’s peace of mind.

4.2 Looking Into The Future

In response to the vulnerability and our initial containment measures, we have further strengthened our platform’s security through ongoing initiatives and preventive measures. Key among these is the attainment of SOC 2 Type I attestation, a significant milestone that confirms our compliance with AICPA standards. This certification not only demonstrates our adherence to stringent security and availability controls but also enhances client trust and assures transparency in our operations​​​​.

Additionally, we launched the BlocGuardians Bug Bounty Program, inviting cybersecurity experts and enthusiasts to help identify and mitigate potential security weaknesses. This program is a testament to our proactive approach in maintaining a resilient and secure platform, especially in the dynamic and complex Web3 landscape. It underscores our commitment to leveraging community expertise for continuous improvement in our security practices, thereby safeguarding user data and assets against emerging threats​​.

4.3 What’s next?

a) Temporary Lower Validator Cap On InfStones Account

We have reduced our validator number cap to match the currently active number of validators managed by InfStones on the same day as this post.

Transaction: Ethereum Transaction Hash (Txhash) Details | Etherscan

b) Key Rotation

We are going to start rotating ALL keys associated with InfStones. Meanwhile, we will be actively engaged with the DAO in the forum to reach a consensus for future steps.

If still interested in more details please check article here: From Vulnerability to Vigilance: Insights on a Recent Bug and Continued Safeguards

6 Likes
5

Thank you for the details of the investigation, the impact analysis, and the summary of the actions taken especially the important initiative of setting up a bug bounty and commencing an external audit to assess the effectiveness of your IT security processes & controls.

Based on the understanding of a group of DAO contributors after discussions with InfStones and the researchers (dWallet labs), my personal opinion is that:
note, this is not a DAO edict as the DAO has not had time to consider and vote on the below, but is what I and fellow contributors I’ve spoken to would suggest

  • In line with the DAO’s mission, although there is currently no indication that any keys have been compromised, from a preponderance of caution and safety that all current (10,001) InfStones validators should be exited;

  • Following such an “out of order exit” process, the ETH from these validators will flow back into the Lido protocol via the Withdrawal Vault and be re-allocated to validator keys in the buffer based on the on-chain distribution mechanism (there are currently a sufficient number of keys in the buffer to absorb this cycling);

  • Exits can be processed in such a way as to not necessarily clog the exit queue, but hastily nonetheless.

Currently, InfStones would not be eligible to receive any of this “cycled” ETH as they do not have “depositable keys” in the registry, as a result of resetting their limit earlier. The only way for them to get depositable keys would be to request to increase their limit via easy track (would take three days and be subject to possible objection by LDO holders) or for a full DAO vote that would increase their limit.

image
image1046×480 31.4 KB

We suggest that a DAO discussion should follow regarding next steps:

  • are there any additional details or information required by the community to fully understand and assess the situation?

  • do we feel that the NO has sufficiently remediate the issues in the systems / processes, and is the DAO satisfied with the NO’s response?

  • should the Node Operator remain in the set? if so,

  • should the node operator submit new keys, and when should the node operator be allowed to increase their limit?

8 Likes
6

The node operator’s response demonstrates a high level of professionalism and precision in addressing the situation. Given this proactive approach, it makes sense to consider retaining them in the validator set to maintain network stability. And to uplift their limits after the keys rotation. To mitigate future risks, I suggest the idea of conducting infrastructure reviews for this and probably other node operators.

9 Likes
7

I am no DevOps nor IT security expert, so I just want to share my first impressions from what I read:

  • the severity of this sounds very different from reading the above linked disclosures and posts (paraphrased and pointed from my personal understanding: NO: “we have everything under control and no damage occured”, dWallet: “there were several severe vulnerabilities that cast doubt about the security culture both at the effected company as well as many blockchain protocols”) - this is frustrating and I’d like to at least try to get 1-3 independent “objective” assessments

  • imho abundance of caution is justified, so I am personally in favor of

  1. exiting/rotating Lido validator keys, plus
  2. reevaluating not only this NOs setup but also from other NOs (regularly),
  3. using the results of this analysis to reassess each NOs weight in the Lido protocol,
  4. creating/expanding bug bounties to include these types of attack vectors (again both from the NO and from the side of all blockchain protocols)

Let’s use this as a warning shot that will make us collectively stronger across the whole ecosystem!

9 Likes
8

I agree the response from the NO once they were engaged regarding the potential vulnerability has been good, especially in terms of taking into account contributor feedback and putting safety of stakers above all else; and I also really want to point out that the initiative to set up a bug bounty program to capture the possible risk (and cost) associated with infrastructure vulnerabilities and overall security is probably something very important for professional node operators.

Some additional thoughts:

  • Ideally, a public disclosure would have been made of the platform bug as soon as possible, but it’s understandable that this incident has the inherent complexity of affecting infrastructure that is cross-network, and thus it may not be possible to speedily remedy possibly affected systems across all networks at the same time, hampering the speed with which a public disclosure could have happened.
  • That said, I think there was probably sufficient time in between identification and now where disclosure should have happened sooner – and disclosure could have been made in private to DAO contributors to help assess the situation at an earlier timeframe (the tricky thing is here though that actions necessitated as a result may have pre-empted a disclosure, but I think we need to think a bit deeper and holistically (i.e. treating Lido as a wider protocol) when weighing the risks here, which wasn’t done explicitly)
  • The lack of a pre-existing bug bounty program for infra at the NO level (the more and more I think about it the more things like self-insurance and bug bounties are probably required for larger NOs, especially for uncollateralized validators) and perhaps some sort of “whistleblowing” program at the protocol level (i.e. to encourage disclosure to the DAO, or at least some contributors who can be entrusted with sensitive data before it can be fully publicly disclosed) probably made the disclosure process take longer than it should have (or not allowed for as many paths as necessary in order for it to happen timely).

Agree with this!

5 Likes
9

Unless there’s specific evidence of negligence, I don’t think we should penalize operators. I support rotating the keys back into the NO

This event is a good call of action for all Lido NOs and coordinators. While setup diversity is good there could exist some minimum guidelines of operational practices agreed by some majority of Lido community members.

6 Likes
10

Just a quick update on the progress of our proposed key rotation. As of this post, we have rotated out 3000 keys. We will keep monitoring the exit queue and gradually rotate out the rest, as discussed. Thank you all for taking the time to discuss and providing feedback!

4 Likes
11

How the key rotation works here? The withdrawed ETH would go back to Lido withdrawal vault and be allocated to the NOS with lower shares first?

12

Yes that’s how it works

1 Like
13

Hi I’m Omer, CEO at dWallet Labs.

A short response from our security research team 0d who made the discovery.

  1. As we demonstrated in the post, we assert that all InfStones validators, including Ethereum validators were vulnerable and their keys could have been extracted by attackers. InfStones, in their response on this thread and elsewhere, are continuously only referring to the first vulnerability that we discovered (regarding tailon), and don’t address the impact of the full discovery of the chain of vulnerabilities that we presented and specifically the one involving their proprietary “infd” management service.
  2. As such, and as we recommended directly to InfStones when the vulnerabilities were disclosed to them back in July, besides fixing the vulnerabilities, all validator keys must be rotated AND all validators must be redeployed. We also made additional recommendations including some on the internal security measures at InfStones, however we have no visibility into what the InfStones team actually implemented.
  3. As we write in the post, we believe that attack vectors involving validator infrastructure with traditional web2 attacks are at least as important to web3 security, if not more so, as traditional web3 attack vectors such as smart contracts. Taking over validators doesn’t only pose a risk for the stakers and delegators of that validator, it could have a devastating affect on the network with censorship, DoS and complete network takeovers.
  4. In our post, and in our discussion with the Lido DAO contributors that reached out to us, we suggested including NO vulnerabilities in the bug bounty program.
  5. Our position is that by doing that, not only will white hat hackers be incentivized to discover vulnerabilities before bad actors, but they will also have a framework to operate within without being worried about crossing any ethical or legal lines, seeing as the vast majority of NOs do not have their own public disclosure or bug bounty programs.
  6. A way to fund that part of the bug bounty program could be by allocating some of the fee received by each NO to a dedicated “vault” for that section of the bug bounty, and that will also solidify their continuing consent and endorsement of the bug bounty program’s terms and condition.

Finally, we wanted to thank the Lido DAO contributors and the Lido community for taking our report and those risks seriously, this is a good signal for the maturing of Web3 infrastructure and security.

10 Likes
14

Thank you for your comments Omer!

Your team’s security analysis and the subsequent discussion that we had was highly valuable and I think that there is a lot of room for identifying possible processes and mechanisms to further beef up the security profile of node operators while at the same time increasing the comfort that the DAO and users of the protocol can have through the use of 3rd party security researchers to uncover and continuously monitor things like this.

This is the only part I disagree with.

I don’t think it’s possible or sustainable for an open protocol like Lido (or the DAO that helps to support it) to have a bug bounty program that is inclusive of every user that may run infra that supports the protocol (imagine if the Lido bug bounty had to cover all possible integrations, bridges, user interfaces, node operators, etc).

IMO, the only sustainable path is that the DAO should have some sort of program to encourage and reward vulnerability disclosure of users of the protocol, but not of an all inclusive nature – I’m not exactly sure what shape that would take exactly, as the primary concern here is a sustainable funding mechanism and ensuring that the onus of responsibility is on the right agent (e.g. it may relay on bonds or rewards set aside by NOs that are released on a delay mechanism).I firmly believe that this responsibility (i.e. safeguarding of infra) should lie with the infrastructure provider themselves and the protocol (or the DAO) should foremost have a way to assess its sufficiency and effectiveness, and thereafter as a failsafe and last resort have some sort of “whistleblowing” mechanism (that for economic reasons cannot be robust enough to have a universal scope).

3 Likes
15

As InfStones have at this point exited 10000 of their 10001 keys, and most of the stake is in the process of re-cycling through the protocol, I’d like to advance the discussion to address the below proposed items (and any others that the community thinks important):

I think the input from @Omer_Sadika is important here in terms of understanding that the second vulnerability is also equally important to be addressed as the initial one. To that end, I have asked InfStones 1) also proceed with removing any additional previously submitted keys from the Node Operator Registry contract, as it is difficult to discern when these private keys were produced and if the material was at any point susceptible to the infra vulnerabilities described above, and 2) to share (to the extent currently possible) the results of the SOC I review performed by the external auditor, as well as the detailed list of actions (to be) taken regarding securing the infrastructure against the identified gaps.

I think this requires a DAO vote, which can probably happen during next week’s regularly scheduled Snapshot slot. My personal opinion is that if the above asks are met (i.e. all previous keys have been rotated and/or flushed, remediation actions have been fully identified and completed (and assessed as such), and that there is no further reason to believe that the infrastructure has issues), that the NO can remain in the set and begin to submit validators anew, probably at a somewhat guarded pace.

3 Likes
16

Thank you for the detailed suggestions and next steps! Here is a quick status report on our current progress:

1 All of our 10,001 keys are now confirmed exited, we have no active keys as of this post.

2 We will unregister the 998 unused keys currently associated with our operator. I want to assure the community that, as a precaution, we have already reset our key limit through the Operator Registry contract. Those 998 keys will NOT be used by the protocol. Here is the transaction we submitted about a week ago:

3 We will also share detailed action items we have performed with regard to securing the infrastructure in this thread.

3 Likes
17

I’m writing this post to provide updates on how we addressed the vulnerability and offer additional insights into the matter we discussed extensively. The fix we implemented for Tailon, as detailed in our previous post, also encompasses crucial enhancements for the infd services. The infd service, to put it simply, is an in-house developed tool that we use to manage instances launched on our platform. Functions, such as upgrade blockchain binanry, can be performed automatically for minimal human error. Since July this year, we have done the following:

1 We have restricted access to infd from our internal API server through security groups to prevent external access. Specifically, the port to access the infd service is now only open to limited ip addresses in a whitelist.

2 The infd code base was updated, and ALL requests sent to infd require JWT Tokens that are unique for each instance for authentication, no matter the access method.

3 ALL communication with our internal API servers now also requires such unique tokens for authentication. As previously mentioned, when we rotated the AWS related credentials on all the servers, we also rotated the JWT tokens on all instances on our platform. Our new workflow dictates that no matter which environment the instance was deployed, development or production, no authentication bypass is allowed.

4 Further, the infd service was refactored, so it no longer allows shell commands, and shell scripts were removed from all operations. The infd service now only accepts a few pre-defined actions; no other actions are executable through infd.

We understand that to have a third party verify the fix and system resilience is important. As the next step, InfStones will conduct a pentest with a third-party auditor, with the results to be shared with the DAO once available.

3 Likes
18

Thanks for the update Sili.

Hopefully with this information and the commitment for a full pentest there’s enough information for voters to form an opinion and for a vote to be held.

Given that I’d move to continue with the plans to have a snapshot go up tomorrow that will in general outline the three options discussed in this thread:

  • Allow InfStones to resume submitting keys
  • Remove InfStones as Operator
  • Hold for further information/analysis/discussion (time-boxing this to something like ~1 month)
19

The snapshot vote for this item is up for vote for the next 7 days.

As suggested yesterday, there are three options here. I think that there’s still room for discussion as this vote rolls on, so I would like to withhold my comments/personal opinion for a few more days.

5 Likes
20

Hey there,

As I see from the thread, InfStones performed the actions to prevent asset losses by stakers, though failed to disclose the vulnerability early, and answered on additional concerns regarding the infd service brought up by dWallet rather lately.

I am against removing the InfStones node operator from the currently maintained by the Lido DAO curated set.

Nevertheless, I would suggest waiting for:

  • vulnerability mitigation path tech description and artifacts (add a bit of transparency)
  • detailed technical reports (both audit and pentest results) confirming that mitigations are whole and sound

The point here is about making an informed decision.

5 Likes