Systematic Network Triage: Layer-by-Layer Fault Isolation in Multi-Vendor Networks

Unstructured troubleshooting in complex enterprise topologies generates noise and prolongs mean time to resolution (MTTR). When links flap or application flows experience intermittent degradation across multi-vendor networks (Cisco IOS-XE, Arista EOS, Juniper Junos), an ex-TAC triage methodology requires isolating issues systematically along OSI boundaries: physical layer integrity, data link negotiation, address resolution, and transport layer state.

Bypassing lower-layer checks to speculate about application or routing bugs frequently overlooks basic physical and link-layer faults.

Layer 1: Physical Link and Optical Transceiver Health

Begin by interrogating the hardware layer on the intermediate switches and routers. Software counters reveal physical degradation before a link reports a hard down state.

Interpreting Interface Error Counters

On Cisco IOS-XE:

Switch# show interfaces GigabitEthernet1/0/1
GigabitEthernet1/0/1 is up, line protocol is up (connected)
  5 minute input rate 124000 bits/sec, 142 packets/sec
  5 minute output rate 88000 bits/sec, 96 packets/sec
     4219412 packets input, 312451921 bytes, 0 no buffer
     Received 1241 broadcasts (0 multicast)
     14 runts, 0 giants, 0 throttles
     8421 input errors, 8399 CRC, 8 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     2419512 packets output, 1941209512 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets

Metric interpretation:

• CRC Errors: Indicates bit corruption during transmission. Primary causes include faulty patch cables, unseated transceivers, dirty optical fiber end-faces, or electromagnetic interference.
• Runts: Frames smaller than the 64-byte minimum Ethernet standard. Occurs when collisions happen on half-duplex links or when a transmitting ASIC serialiser suffers voltage drops.
• Giants: Frames exceeding maximum configured MTU (typically 1518 bytes untagged, or 1522 bytes with 802.1Q tag, unless baby giants or jumbo frames are permitted).
• Input Errors: Aggregate counter. If input errors increment alongside CRC errors, focus entirely on the physical cable and optics.

Digital Optical Monitoring (DOM)

For fiber interfaces, evaluate received and transmitted optical power against calibrated thresholds:

Switch# show interfaces GigabitEthernet1/0/1 transceiver detail
Optical   High Alarm  High Warn   Low Warn    Low Alarm
Parameter Threshold   Threshold   Threshold   Threshold   Current Value
--------- ----------- ----------- ----------- ----------- -------------
Tx Power   -1.0 dBm    -2.0 dBm   -10.0 dBm   -11.9 dBm     -3.4 dBm
Rx Power   -1.0 dBm    -2.0 dBm   -17.0 dBm   -19.0 dBm    -22.4 dBm (ALARM)

An Rx power of -22.4 dBm falling below the Low Alarm threshold (-19.0 dBm) directly explains CRC error increments. Inspect the fiber strand with a scope and clean the ferrule before replacing the SFP module.

Layer 2: Speed, Duplex, and Spanning Tree Inconsistencies

Link autonegotiation failures produce subtle packet loss patterns that scale with traffic volume.

Duplex Mismatches

Under IEEE 802.3u autonegotiation, if one device is hard-coded to 100/Full and the link partner is configured for auto/auto, negotiation fails. The auto-negotiating end falls back to 100/Half according to standard default rules.

Resulting symptoms:

• The half-duplex side records late collisions (late collision counter in show interfaces). Late collisions occur when a collision is detected after the first 512 bits (64 bytes) of the frame have been transmitted.
• The full-duplex side records CRC and frame alignment errors.
• Throughput appears normal during low-traffic ping tests, but collapses under high-throughput TCP transfers due to collision-induced packet corruption.

Rule: Always set both sides to autonegotiation, or hard-code both sides identically.

Spanning Tree Topology Flaps

A flapping access port without spanning-tree portfast enabled triggers Topology Change Notifications (TCNs) throughout the broadcast domain.

Switch# show spanning-tree detail | include is executing|occurred
  Number of topology changes 142 last seen 00:00:12 ago
  from GigabitEthernet1/0/24

Each TCN forces switches across the VLAN to reduce their MAC address table aging timer from 300 seconds to the forward delay timer (typically 15 seconds). This causes widespread unicast flooding across the switch fabric, driving up CPU utilisation and saturating uplinks.

Remediation:

interface range GigabitEthernet1/0/1 - 48
 spanning-tree portfast
 spanning-tree bpduguard enable

Layer 3: ARP Resolution and Path Asymmetry

When physical and link layers are confirmed stable, isolate the network layer.

Inspecting ARP Cache Inconsistencies

Router# show ip arp 192.168.10.50
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  192.168.10.50           -   Incomplete      ARPA   Vlan10

An Incomplete ARP entry indicates:

1. The host is powered off or disconnected.
2. The host has an active local firewall dropping broadcast ARP requests.
3. Control Plane Policing (CoPP) on the upstream switch is dropping ARP traffic due to rate-limiting thresholds.

Verify CoPP dropped packets on Cisco IOS-XE:

Router# show policy-map control-plane | section class-default|copp-system-class-critical
  Class-map: copp-system-class-critical (match-any)
    541291 packets, 42104192 bytes
    5 minute offered rate 12000 bps, drop rate 0000 bps
    Match: access-group name copp-critical-acl
    police:
        cir 8000000 bps, bc 250000 bytes
        conformed 541291 packets, 42104192 bytes; actions: transmit
        exceeded 0 packets, 0 bytes; actions: drop

Layer 4: TCP Window Exhaustion vs Network Drops

When packet loss is suspected of degrading application throughput, inspect TCP window scaling and acknowledgments using packet captures.

In Wireshark:

1. Look for tcp.analysis.zero_window: The receiver buffer is completely full, advertising a window size of 0. The sender halts transmission until a window update arrives. This is an application host processing bottleneck, not network packet loss.
2. Look for tcp.analysis.duplicate_ack followed by tcp.analysis.fast_retransmission: Three duplicate ACKs signal that a single segment was lost in transit while subsequent segments arrived, triggering TCP Fast Retransmit.

tshark -r /tmp/app_flow.pcap -Y "tcp.analysis.zero_window || tcp.analysis.retransmission" -T fields -e frame.number -e ip.src -e ip.dst -e tcp.analysis.flags

Structured Triage Sequence

By systematically validating L1 through L4 in sequence, network engineers eliminate false assumptions and isolate root causes with deterministic precision.