Hi all,
As promised, now that all withdrawals have completed and Kiln is no longer operating any Lido validators, we are following up on two topics.
-
Postmortem on the incident. What happened, and lessons we have taken from it and are sharing with the community. This is a postmortem that goes into specific details on Ethereum infrastructure, unlike the more general writeup to all Kiln customers that was publicly shared here.
-
Recap on the lost rewards due to voluntary exits, and context on Kiln’s participation in Lido to date.
Thank you to the Lido community for your attention and understanding throughout this incident.
Part I: Incident postmortem
Background
On 8 September 2025, Kiln detected unauthorized activity on our platform that resulted in a malicious Solana transaction being constructed and signed by one Kiln customer’s custody quorum, leading to a loss of funds.
The root cause was a compromise of an infrastructure engineer’s GitHub access token (PAT), which was then used to execute CI workflows, harvest cloud credentials, and inject a malicious payload into a running Kiln Connect API Kubernetes pod. This payload altered a single API endpoint to return a malicious Solana transaction alongside the expected “deactivate stake” transaction.
There is no evidence of any other malicious transaction, unauthorized change to Kiln systems, or asset loss beyond the initially identified Solana incident. However, given the level of access in Kiln’s infrastructure achieved by the threat actor, we treated the incident as a full infrastructure compromise and acted accordingly.
Impact on Lido
-
Kiln, following notification towards Lido contributors and in coordination with Lido contributors and other staking ecosystem stakeholders, began an orderly exit of all Ethereum validators on 9 September 2025, including ETH validators operated for Lido.
-
The decision to exit all validators was a precautionary step to protect stakers and was taken in consultation with customers, Ethereum ecosystem stakeholders, and external security firms.
-
Exits followed Ethereum’s protocol rules and queues and could not be accelerated by Kiln. Each validator was expected to take approximately 10–40+ days to exit, with withdrawals following after.
-
For Lido, the exited ETH flowed back into the protocol and was utilized for withdrawals or re-allocated according to the protocol’s onchain stake allocation logic as exits and withdrawals completed.
-
Given Ethereum validators do not earn while waiting to be swept post-exit, and in the re-entry process, the stETH pool rewards were affected by this procedure, leading to slightly less APR than would have been expected in nominal conditions, as detailed in part II of this document.
No evidence has come to our attention that the threat actor ever attempted to use potentially privileged or private information related to Ethereum validators for nefarious purposes.
Why we exited all Ethereum validators and rotated keys
Threat model after the incident
Once we confirmed that:
-
A threat actor had leveraged CI/CD to harvest cloud credentials, and
-
Those credentials granted access to production workloads across our cloud service providers,
We treated all infrastructure that could plausibly reach validator-related workloads as potentially compromised.
Given that Lido and other partners rely on Kiln as a professional operator, the bar for acceptable residual risk is effectively zero when it comes to validator keys.
Why exited everything on Ethereum
We chose to exit all ETH validators for three reasons:
-
Best-practice remediation for a suspected infra compromise
- If infrastructure is treated as compromised, the correct response is to rebuild and refresh all secrets. For validators, that includes validator keys and the environments that use them.
-
Ethereum’s validator model makes key “rotation” very visible
-
On most PoS protocols we support, validator key rotation can be done without unstaking or user-visible downtime.
-
Unfortunately, on Ethereum today, rotating validator keys effectively means:
-
Exit the current validator via an exit message
-
Withdraw funds when they become withdrawable
-
Restake using new validators and fresh keys
-
Alignment with ecosystem stakeholders
- The plan to exit all ETH validators was developed and validated with major ecosystem stakeholders and security partners. Internal discussions and the Ethereum exit plan explicitly reference syncing and agreeing on this approach with the Ethereum Foundation and Lido contributors.
Ethereum infrastructure rebuild and hardening
In parallel with the Ethereum exit, we rebuilt the infrastructure that runs validators and Kiln’s control plane along six strategic security axes:
-
Zero-Trust Access Plane: Identity-driven access, enforced through predictable and controllable network boundaries. For instance, we now route all engineer and CI traffic through Tailscale with dedicated exit nodes and deterministic egress IPs; restrict cloud/K8s consoles to those egress addresses and Okta trusted zones.
-
Trusted (CI/CD) and Infrastructure as code (IaC) Execution: Only hardened, auditable pipelines can modify infrastructure, with no long-lived secrets. For instance, we banned long-lived credentials in pipelines: require scoped, short-lived tokens for infrastructure actions and harden runners and IaC: use self-hosted, audited runners with fixed egress, SAST scans on Terraform and builds.
-
Blast-Radius Isolation and Least Privilege: Strict segmentation and privilege minimization prevent lateral movement. For instance, we put workloads in dedicated cloud projects/accounts with minimal cross-project IAM and enforce granularity in roles and automated policy drift detection.
-
Application and Container Hardening: Immutable workloads and runtime integrity checks reduce tampering and exploitability. We moved Ethereum components (web3signer, validator agents) to distroless / minimal images, strip shells, and deploy as immutable read-only containers.
-
Continuous Monitoring and Response: Endpoints, clouds, and APIs are continuously scanned, logged, and defended by a 24/7 SOC. We deployed CNAPP sensors across cloud service providers and forwarded all telemetry to a centralized SIEM with dedicated detection rules. In addition, we now have a 24/7 SOC with real-time alerting and playbooks that trigger containment for anomalous access to critical components.
-
Validator Key Protection: Signing operations are isolated depending on protocol. Keys are secured through hardened workloads. Kiln’s upstream contributions patched Web3signer, which previously required private validation keys and their associated password to be accessible through the filesystem. This patch enables a fully in-memory mode, which significantly reduces the attack vector in case of unauthorized machine access.
Validator key protection
For Ethereum, we have specifically strengthened key isolation and usage:
-
Segregated signing environments:
Validator signing operations run in hardened, isolated environments with minimal privileges beyond what is strictly necessary for consensus participation.
-
Ongoing key renewal:
Following the incident we exited validators and issued new validator keys for new validators on our hardened infrastructure.
Suggested learnings for Lido and large operators
The incident, and the ETH exit that followed, surface several ecosystem-level lessons:
-
Ethereum’s current exit model makes industrial-scale key rotation noisy and painful. Large-scale key rotation is visible and disruptive at the protocol level.
-
Validators should be treated as disposable from day one. Architectures, processes and customer contracts should all assume that mass validator rotation (exit + restake) may be required as a normal security operation, not just a theoretical worst case.
-
Node operator / protocol coordination is critical. Proactive alignment with contributors, EF and other stakeholders was essential to executing the exit without destabilizing the network or surprising stETH holders. The governance thread and joint communications helped set expectations with the community.
Conclusion
From a security perspective, we treated this incident as an opportunity to rebuild our Ethereum validator stack on stronger foundations. The decision to exit all ETH validators was disruptive in the short term but aligned with fundamental risk principles: when infrastructure is suspected compromised, you rebuild, rotate keys, and leave nothing critical behind.
Part II: Recap on the lost rewards due to voluntary exits
Lost rewards
The Lido analytics workgroup has performed an evaluation of the impact of Kiln’s precautionary out of order exits due to the incident, which they will publish in this thread.
The analysis concludes on an evaluated total impact of 207.312 ETH in missed protocol rewards, with an average ~13.5 days of missed rewards for 5 726 deposited validators.
Kiln performance to date
One factor we feel is worth considering alongside these lost rewards due to the incident is the overperformance that Kiln has delivered since we are part of the curated node operator set, i.e extra rewards compared to other node operators in the same region and with similar setups [1].
Our analysis finds that Kiln has generated an additional 362.95 ETH compared to the mean rewards (EL + CL) of the comparison group, and 504.60 ETH compared to the median rewards, since January 1st 2023. It is worth noting that the outperformance is essentially entirely EL rewards driven. The full methodology details can be seen in this Github repository.
This is an imperfect argument when considering compensation, as the stETH holders in this period are different from the stETH holders in the incident period with the reduced APR due to out-of-order exits. We nonetheless feel it is worth mentioning to show the quality of the service Kiln has provided to Lido since joining the set in 2021.
Next steps
In a subsequent post we will outline a proposed approach regarding lost rewards and under what conditions Kiln might re-join the Lido curated set, following discussions with the Lido team.
[1] The list of node operators used for a representative comparison, as defined with the Lido Analytics team, is: RockawayX Infra, ParaFi Technologies LLC, Chorus One, P2P.ORG - P2P Validator, Blockscape, Gateway.fm AS, Staking Facilities
