# Post Mortem: Develp Fleet DNS Resolution Failure and Missed Proposal on 2026/05/07 **URL:** \ **Category:** Node Operators\ **Created:** [May 29, 2026, 10:52am UTC](https://research.lido.fi/t/post-mortem-develp-fleet-dns-resolution-failure-and-missed-proposal-on-2026-05-07/11603 "2026-05-29T10:52:21Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Develp.Devops](https://dub1.discourse-cdn.com/flex013/user_avatar/research.lido.fi/develp.devops/32/6492_2.png) [@Develp.Devops](https://research.lido.fi/u/Develp.Devops)\ **Post date:** [May 29, 2026, 10:52am UTC](https://research.lido.fi/t/post-mortem-develp-fleet-dns-resolution-failure-and-missed-proposal-on-2026-05-07/11603/1 "2026-05-29T10:52:21Z")
**Name** : DNS Resolution Failure and Missed Proposal **Impacted Servers** : `geth-06` and `node-06` **Impacted Services** : `nimbus-beacon-node`, `nimbus-validator-client`, `web3signer` ## Summary On **2026/05/07** after a Linux Kernel security upgrade, one of the LIDO server experienced a network issue leading to attestation delays and a missed proposal. The event happened a few days after a hardware incident where the Cloud Provider had to replace the network switch used by one of the servers running Validators. In addition during that window one proposal was missed: - [https://beaconcha.in/slot/14277711](https://beaconcha.in/slot/14277711) ## Glossary - **[execution layer node](https://nimbus.guide/execution-client.html)**- Nethermind or Geth client used, further referred to as **EL**. - **[beacon node](https://nimbus.guide/quick-start.html)** - Nimbus client running, further referred to as **BN**. - **[validator client](https://nimbus.guide/validator-client.html)** - Nimbus client providing signatures to **BN** , further referred as **VC**. - **[web3signer](https://docs.web3signer.consensys.io/)** - Remote signing service that loads keys, further referred as **Signer**. - **[Copy Fail](https://copy.fail/)** - Security Exploit detected in the Linux kernel. - **proposal** - The validator proposes a block to the network. - **attestation** - Action done by the validator to approve the proposed block. ## Timeline (UTC) - **2026/05/01** - **00:55** - Network incident at the Cloud Provider datacenter. - **06:00** - Switch replacement completed, end of the hardware incident. - **2026/05/06** - **11:00** - Host system security update. - **17:40** - Host reboot to apply kernel update. - **18:00** - DevOps team notices increased attestation delays. As issue is not critical, investigation delayed till next day. - **2026/06/07** - **07:10** - Manual update in DNS configuration to mitigate latency. - **07:30** - Enabled `systemd-resolved` service. Partial recovery. - **12:33** - Missed proposal at [14277711](https://beaconcha.in/slot/14277711) slot. - **13:15** - **VC** config changed to use IP address instead of DNS entries. - **14:50** - **Signer** config changed to use IP address instead of DNS entry. - **19:28** - Proper DNS fix deployed to less impacted `node-06` for testing. ## Details The incident is a combination of multiple events over multiples days: - [Networking switch incident](#networking-switch-incident) in Teraswitch data center. - [CopyFail kernel security patch](#copyfail-kernel-security-patch) and subsequent host reboot. - [DNS Server misconfiguration](#dns-server-misconfiguration) in Teraswitch data center. - [Missed proposal](#missed-proposal) caused by increased delays. ### Networking Switch Incident During the night of the 2026/05/01, the cloud provider had a hardware issue and changed the switch of the server `geth-06` that ran some of the validators. During the switch replacement, the Technical team misconfigured the internal DNS. ### CopyFail Kernel Security Patch The **CVE 2026-31431** - also know as [CopyFail](https://copy.fail/) - was detected at the end of April and patched in the Linux Kernel on the 29/04/2026. This security vulnerability allowed a user to gain administration privileges. The fix this security issue, the kernel of the server has to be upgraded and the host restarted. After fixing the most exposed servers of our infrastructure, it was decided to deploy the patch on the Validators fleet. Each host was restarted sequentially to reduce missed attestations. ### DNS server misconfiguration After the restart, the hosts `geth-06` and `node-06` started to exhibit high delay in the block attestation and missing attestations. Our layout of Ethereum nodes in the fleet uses servers in pairs to provide redundancy at the level of the **BN** : ![image](https://europe1.discourse-cdn.com/flex013/uploads/lido/original/2X/1/1f56ace58f81f052920f9bf911c99829bc147cb8.png) Because **VCs** on `geth-06` and `node-06` nodes are linked to both **BNs** the DNS resolution delays on `geth-06` affected metrics of both **BNs**. The DevOps team identified four separate issues which together caused the problem: 1. The cloud provider DNS misconfiguration that included an unreachable DNS server. 2. Sequential nature of `libc` DNS resolving used by Nimbus client which would get gets stuck on bad server. 3. Unexpected lower priority of `/etc/hosts` file in Name Service Switch configuration. 4. Usage of DNS entries for **BN** and **Signer** in **VC** config even for local ones. Initially the faulty cloud provider DNS server was removed manually from the hosts configuration which provided full recovery. In second step activation of `systemd-resolved` on the host intended to address the 4th issue undid the manual DNS sever removal. ![image](https://europe1.discourse-cdn.com/flex013/uploads/lido/original/2X/9/947fe5fc9d8eaa590bde4369361293b6f3605c42.png) The DNS removal happened around 07:10 which quickly resulted in a notable drop in the attestation delay, but then some re-appeared after `systemd-resolved` enabling. The situation started to stabilise and the DevOps team continued to investigate while the attestation delay was settling down. ### Missed proposal At 14:22 UTC, one validator missed a proposal slot due to delay in network submission. As a temporary measure the loopback interface IP address was used in **VC** configuration to mitigate any impact of DNS issues. This resulted in return to low attestation delays and confirmed DNS as the cause of the issue. By replacing all DNS entries(including `localhost`) with IP addresses in all service config the delay stopped completely, confirming DNS was the root cause. The permanent solution was to make `/etc/hosts` to be first in `libc` resolving priority by modify the host Name Service Switch configuration. The use of `systemd-resolved` was kept for record caching. ```bash > grep hosts: /etc/nsswitch.conf hosts: files mymachines myhostname resolve [!UNAVAIL=return] dns ``` ## Conclusions Key lessons are: - Order of DNS servers in `/etc/resolv.conf` matters since `libc` querying is sequential. - Priority of `/etc/hosts` is not the default on all systems and shouldn’t be assumed to be. - Using `127.0.0.1` instead of `localhost` is safer since it avoids DNS resoution entirely. - Nimbus **VC** behavior of querying DNS at every request exacerbates impact of DNS issues. - Monitoring of DNS server availability can help detect hard to debug issues easier. The issue has been resolved but additional DNS availability checks will be implemented.