Diagnosing Ghost Disconnects: Stateful Firewall TCP Idle Timeouts in Industrial ERP Networks

Industrial plant networks rely on persistent database and terminal sessions between automated machinery, warehouse scanning terminals, manufacturing execution systems (MES), and enterprise resource planning (ERP) database servers. When client sessions terminate abruptly after brief idle periods, the incident is frequently misdiagnosed as wireless roaming instability or database thread locks.

In stateful firewall architectures, intermittent disconnects of persistent TCP sessions are overwhelmingly caused by connection table eviction resulting from timer mismatches between firewalls and operating systems.

Stateful Firewall Connection Tracking Mechanics

Stateful inspection firewalls (including Cisco ASA, Cisco Firepower/FTD, Palo Alto Networks, and Fortinet FortiGate) track active connections in a state table. A session entry is indexed by a 5-tuple: source IP, destination IP, source port, destination port, and protocol.

[Client 10.10.10.50:49152] ---> [Stateful Firewall] ---> [Database 10.20.20.100:1521]
State Table Entry:
TCP inside:10.10.10.50/49152 outside:10.20.20.100/1521 idle 0:00:15 timeout 1:00:00

Firewalls enforce an idle timeout on every state entry. If no packets traverse the connection within the configured timeout window, the firewall silently evicts the entry from memory to prevent state table exhaustion:

• Cisco ASA/FTD default TCP idle timeout: 3,600 seconds (60 minutes).
• Palo Alto Networks default TCP established timeout: 3,600 seconds (60 minutes).
• Aggressive profiles in high-concurrency environments: frequently lowered to 900 or 1,800 seconds (15 to 30 minutes).

When an ERP client remains quiet while a barcode scanner or terminal user processes physical inventory, no packets cross the wire. If the inactivity interval exceeds the firewall idle timeout, the firewall purges the session. When the client subsequently transmits its next query, the firewall evaluates the packet:

1. Because the SYN flag is not set and no matching connection exists in the state table, the packet is classified as out-of-state.
2. The firewall either silently drops the packet or injects a TCP RST segment back to the client.
3. The client application freezes, encounters a socket write error, and crashes.

Capturing Connection Resets: The Dual-Ended Capture Strategy

Proving that a stateful firewall is terminating an established connection requires synchronized packet captures on both sides of the firewall simultaneously.

Cisco ASA/FTD Capture Configuration

Configure inline captures on ingress and egress interfaces matching the affected flow:

access-list CAP_ERP_ACL extended permit tcp host 10.10.10.50 host 10.20.20.100 eq 1521
access-list CAP_ERP_ACL extended permit tcp host 10.20.20.100 eq 1521 host 10.10.10.50

capture CAP_INSIDE access-list CAP_ERP_ACL interface inside buffer 33554432 circular
capture CAP_OUTSIDE access-list CAP_ERP_ACL interface outside buffer 33554432 circular

Export and inspect the captures when a session freeze occurs:

tshark -r /tmp/cap_inside.pcap -Y "tcp.flags.reset == 1" -T fields -e frame.time -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport -e tcp.seq -e ip.ttl

Distinguishing Firewall-Generated RSTs from Host RSTs

Inspect the IP Time-to-Live (TTL) and TCP Window Size of the RST packet:

• Host-generated RST: Reflects standard OS initial TTL decremented by router hops (e.g. initial TTL of 64 on Linux decremented to 62; initial TTL of 128 on Windows decremented to 126).
• Firewall-generated RST: Often emitted with default platform TTL (e.g. TTL 255 on Cisco ASA, TTL 64 on FortiGate) and a TCP window size of 0.
• If the RST appears on CAP_INSIDE destined to the client, but does not appear on CAP_OUTSIDE originating from the server, the firewall originated the reset internally upon receiving out-of-state data.

TCP Keepalive vs Firewall State Table Misalignment

The fundamental architectural defect in these environments is the mismatch between OS-level TCP keepalive timers and firewall session eviction timers.

Most operating systems default to conservative keepalive intervals:

• Linux (/proc/sys/net/ipv4/tcp_keepalive_time): 7,200 seconds (2 hours).
• Windows Server (HKLM\\SYSTEM\\CurrentControlSet\\Services\\Tcpip\\Parameters\\KeepAliveTime): 7,200,000 ms (2 hours).

If the firewall idle timeout is set to 3,600 seconds, the firewall purges the connection after 1 hour of silence. The operating system will not send its first keepalive probe until 2 hours have elapsed. By the time the client OS sends the keepalive ACK probe (with sequence number SEG.SEQ = SND.NXT - 1), the firewall has already discarded the session state.

Resolving Asymmetric Routing Drops

Stateful firewalls require bidirectional traffic inspection. If client-to-server traffic traverses Firewall A, but asymmetric core routing routes server-to-client return traffic through Firewall B or an internal layer-3 switch bypass, both firewalls drop the connection.

Signs of asymmetric routing:

• Firewall logs reporting: Deny TCP (no connection) or TCP packet dropped: ACK without prior SYN.
• Capture on client-facing interface reveals outbound SYN and inbound SYN-ACK, but subsequent server data frames never traverse the firewall.

Diagnostic verification: Trace route both directions using TCP syn-probes rather than ICMP:

# From client to server
traceroute -T -p 1521 10.20.20.100

# From server to client
traceroute -T -p 49152 10.10.10.50

Verify whether intermediate IP hops differ between forward and reverse paths.

Remediation Protocol

To establish permanent resilience for long-lived industrial TCP connections:

1. Tune OS TCP Keepalive Intervals

On Linux ERP servers and database hosts, configure sysctl parameters to send keepalive probes well inside the firewall timeout:

sudo sysctl -w net.ipv4.tcp_keepalive_time=600
sudo sysctl -w net.ipv4.tcp_keepalive_intvl=30
sudo sysctl -w net.ipv4.tcp_keepalive_probes=5

Persist in /etc/sysctl.d/99-tcp-keepalive.conf:

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5

2. Configure Dedicated Firewall Service Policies

Rather than extending global idle timeouts across all traffic, apply targeted timeout policies to specific enterprise application ports.

On Cisco ASA:

class-map CM_ERP_DATABASE
 match access-list ACL_ERP_PORTS

policy-map PM_CUSTOM_TIMEOUTS
 class CM_ERP_DATABASE
  set connection timeout tcp 8:00:00

service-policy PM_CUSTOM_TIMEOUTS interface inside

On Palo Alto Networks: Create an Application Override or custom Service Object for port 1521 with TCP Established Timeout set to 28,800 seconds (8 hours).

By aligning OS keepalive timers to probe every 10 minutes and extending database firewall timeouts to 8 hours, ghost disconnects are permanently eliminated without weakening global firewall state table hygiene.