Troubleshooting Intermittent DNS Failures: Packet Drops, EDNS0 Buffer Truncation, and Resolver Timeouts

When distributed microservices or internal applications experience intermittent resolution latency exceeding 1,000 ms, DNS is frequently misdiagnosed as generic packet loss. In environments running DNSSEC or extended resource records, the root cause frequently traces back to EDNS0 buffer misconfigurations, firewall fragmentation drops, or recursive resolver TCP fallback failures.

Resolving these issues requires systematic packet analysis across the query path rather than relying on dashboard status metrics.

Reproducing Resolver Timeouts with dig and tcpdump

The first step in diagnosing intermittent resolution failures is validating whether queries fail universally or only for specific record types (such as TXT, SRV, or DNSKEY) that return large payloads.

Standard iterative lookup verification with diagnostic flags:

# Query the local forwarder and inspect response flags and buffer advertisement
dig +nocmd +noall +stats +comments api.internal.corp @10.0.0.53

# Inspect root delegation and query traversal step by step
dig +trace +dnssec api.internal.corp @10.0.0.53

# Force a large payload test to evaluate EDNS0 buffer handling
dig +bufsize=4096 +dnssec TXT large-record.internal.corp @10.0.0.53

When intermittent timeouts occur, run simultaneous packet captures on the client interface and the recursive resolver ingress interface:

sudo tcpdump -ni eth0 'port 53' -s 0 -w /tmp/client_dns.pcap
sudo tcpdump -ni bond0 'port 53' -s 0 -w /tmp/resolver_ingress.pcap

Wireshark and Tshark Packet Analysis

Analyzing the captured trace files with tshark exposes query latency and flags dropped transactions:

tshark -r /tmp/client_dns.pcap -Y "dns.time > 1.0" -T fields -e frame.number -e ip.src -e ip.dst -e dns.qry.name -e dns.time
tshark -r /tmp/client_dns.pcap -Y "dns.flags.truncated == 1" -T fields -e frame.number -e dns.id -e dns.qry.name

In Wireshark, the display filter dns.time > 1.0 || dns.flags.truncated == 1 highlights responses requiring transport fallback. When the recursive resolver responds with the TC (Truncation) flag asserted in the UDP response header, RFC 1035 stipulates that the resolver client must immediately retry the query over TCP port 53.

If the client sends a TCP SYN to port 53 and receives an ICMP Destination Unreachable (Code 13: Communication Administratively Prohibited) or encounters continuous retransmissions followed by a TCP RST, intermediate stateful firewalls are blocking TCP/53.

EDNS0 Buffer Truncation and MTU Fragmentation

RFC 6891 defined Extension Mechanisms for DNS (EDNS0), allowing resolvers to advertise UDP buffer sizes larger than the legacy 512-byte limit (traditionally 4096 bytes). However, large UDP payloads introduce severe fragmentation risks:

1. When a DNS response payload exceeds the path Maximum Transmission Unit (MTU), typically 1500 bytes on Ethernet or 1420 bytes across IPsec and GRE tunnels, the IP layer fragments the datagram.
2. The initial fragment carries the UDP port headers, but subsequent fragments do not contain L4 port information.
3. Network firewalls, load balancers, or cloud security groups configured with strict L4 inspection frequently drop non-initial IP fragments.
4. The client resolver waits until timeout expiry before retrying, resulting in application connection stalls.

Following the DNS Flag Day standardisation, authoritative and recursive resolvers should clamp the EDNS0 UDP buffer size to 1232 bytes. This guarantees that total IP datagram size remains well within the IPv6 minimum link MTU (1280 bytes) and standard IPv4 fragmented topologies, eliminating fragmentation drops.

Hardening Resolver Configurations

BIND9 Configuration (named.conf.options)

Ensure BIND limits UDP buffer size and permits TCP fallback without rate-limiting legitimate retries:

options {
    directory "/var/cache/bind";

    edns-udp-size 1232;
    max-udp-size 1232;

    recursion yes;
    allow-recursion { 10.0.0.0/8; 172.16.0.0/12; 192.168.0.0/16; };

    resolver-query-timeout 3;
    tcp-listen-queue 128;
    tcp-clients 500;

    dnssec-validation auto;
    auth-nxdomain no;
    listen-on { 127.0.0.1; 10.0.0.53; };
    listen-on-v6 { none; };
};

Unbound Configuration (unbound.conf)

For deployments using Unbound as a validating recursive cache:

server:
    interface: 0.0.0.0
    interface: ::0
    port: 53

    do-udp: yes
    do-tcp: yes
    do-ip4: yes
    do-ip6: no

    edns-buffer-size: 1232
    msg-buffer-size: 1232

    num-threads: 4
    infra-cache-numhosts: 10000
    infra-host-ttl: 60
    infra-cache-min-rtt: 50

    prefetch: yes
    prefetch-key: yes

    access-control: 10.0.0.0/8 allow
    access-control: 127.0.0.0/8 allow

Automated Path Latency and Truncation Validation

To continuously detect packet loss and TCP fallback degradation before application failures occur, deploy an active query validation script:

#!/usr/bin/env python3
import time
import dns.message
import dns.query
import dns.rdatatype

RESOLVER_IP = "10.0.0.53"
TARGET_NAME = "api.internal.corp"
TIMEOUT_SECONDS = 2.0

def evaluate_resolver(ip: str, domain: str) -> None:
    query = dns.message.make_query(domain, dns.rdatatype.A, use_edns=True, payload=1232)
    
    start_udp = time.monotonic()
    try:
        response_udp = dns.query.udp(query, ip, timeout=TIMEOUT_SECONDS)
        elapsed_udp = (time.monotonic() - start_udp) * 1000
        print(f"[UDP] {ip} -> {domain}: {elapsed_udp:.2f}ms (Flags: {dns.flags.to_text(response_udp.flags)})")
        
        if response_udp.flags & dns.flags.TC:
            print("[WARN] Truncation bit asserted, testing TCP fallback...")
            start_tcp = time.monotonic()
            response_tcp = dns.query.tcp(query, ip, timeout=TIMEOUT_SECONDS)
            elapsed_tcp = (time.monotonic() - start_tcp) * 1000
            print(f"[TCP] {ip} -> {domain}: {elapsed_tcp:.2f}ms (Answers: {len(response_tcp.answer)})")
    except dns.exception.Timeout:
        print(f"[ERROR] UDP query to {ip} timed out after {TIMEOUT_SECONDS}s")

if __name__ == "__main__":
    evaluate_resolver(RESOLVER_IP, TARGET_NAME)

Summary Checklist for Outage Prevention

1. Standardise EDNS0 advertised buffer size to 1232 bytes on all caching resolvers and forwarders.
2. Confirm stateful firewalls permit both UDP and TCP egress on port 53 across all resolver routing boundaries.
3. Validate that intermediate security devices do not drop fragmented IP packets when legacy servers request larger buffers.
4. Implement active synthetic monitoring capturing query latency metrics separated by protocol transport (UDP versus TCP).