Teaching Agents eBPF: Why We Built a Cilium MCP Server for Kubernetes Triage

When engineering teams connect large language models (LLMs) to Kubernetes clusters for operational triage, the initial design usually defaults to granting the model a generic shell execution tool. The rationale appears straightforward: give an autonomous agent access to kubectl, cilium-dbg, and hubble, and allow it to run whatever diagnostic commands an experienced engineer would type.

In production environments, delegating raw terminal access to autonomous reasoning loops introduces four fundamental failure modes:

1. Context Window Flooding: Commands such as kubectl get pods -A -o yaml or unfiltered Hubble streams emit tens of thousands of tokens of low-signal text. Once an agent context fills with raw JSON and ANSI escape sequences, its attention budget degrades rapidly, leading to forgotten constraints and hallucinations.
2. Fragile Unstructured Parsing: Terminal utilities format output for human eyes using variable-width whitespace tables, column truncation, and dynamic line wrapping. Large language models waste valuable reasoning capacity attempting to regex-parse alignment artifacts from cilium endpoint list or ip route show.
3. Absence of Execution Guardrails: An unrestricted subshell cannot differentiate between an inspection command (cilium-dbg bpf endpoint list) and an irreversible mutation (kubectl delete -f policy.yaml). Prompt-level instructions to "remain read-only" fail under multi-turn autonomous execution paths.
4. Zero Auditability: Standard shell history records ephemeral command strings without capturing the structured evidence the agent evaluated, the confidence score of its hypothesis, or the rollback procedure.

Resolving distributed Kubernetes networking issues with AI assistants requires replacing unconstrained terminal access with typed, contract-driven tool adapters. To deliver this in production, we built technoxi/cilium-mcp: a dedicated Model Context Protocol (MCP) server that provides AI agents with structured, read-only Cilium and Hubble telemetry, strict output contracts, and guarded dry-run action primitives.

Why Model Context Protocol Fits Kubernetes Networking

Anthropic's Model Context Protocol (MCP) establishes an open standard for connecting AI models to external systems via JSON-RPC 2.0. Rather than exposing arbitrary system commands, an MCP server exposes discrete functions defined by explicit JSON Schema parameter and return contracts.

In cilium-mcp, the LLM interacts exclusively with strongly typed tools. The agent never executes a raw subshell. Instead, it calls analytical endpoints and receives deterministic, curated JSON envelopes:

[AI Assistant / Coding Agent]
          │
          ▼ (JSON-RPC 2.0 over Streamable HTTP at /mcp or stdio)
┌────────────────────────────────────────────────────────┐
│                  technoxi/cilium-mcp                   │
├────────────────────────────────────────────────────────┤
│  • Token Authentication & Blast-Radius Clamps          │
│  • JSON Schema Validation (Pydantic / MCP SDK)        │
│  • Curated Telemetry Aggregators & Output Contracts   │
└────────────────────────────────────────────────────────┘
          │
          ├─► Hubble gRPC API (Flow Decision Stream, Drops, L7)
          ├─► Cilium Agent Daemonset (eBPF Maps, Endpoint State)
          └─► Kubernetes API (CNP/CCNP Resources, Pod Specs)

The toolset is organised into three distinct tiers based on operational risk:

1. Context Tools (High-Signal Scopes)

Context tools compress cluster state into compact, token-efficient summaries:

• context.namespace_inventory: Enumerates workloads, pod readiness, Cilium endpoint IDs, and active policy counts without emitting raw pod YAML.
• context.network_insights: Aggregates eBPF drop rates, DNS lookup latency percentiles, and cross-node Geneve encapsulation health across a specified time window.
• context.rca_pack: Assembles workload events, effective network policies, and recent Hubble packet drops into a unified incident dossier.

2. Investigation Tools (Deep-Dive Analysis)

When an agent identifies an anomaly, it pivots to targeted diagnostic probes:

• investigate.pod: Inspects container specs, restart counts, and Cilium endpoint identity metadata.
• investigate.service_path: Traces the complete data path from client pod through Kubernetes Service VIP, BPF kube-proxy replacement maps (cilium_lb4_services_v2), to backend pod endpoints.
• investigate.policy_impact: Simulates and evaluates whether a specific CiliumNetworkPolicy (CNP) or CiliumClusterwideNetworkPolicy (CCNP) selects an endpoint and blocks traffic.
• investigate.flow_query: Queries the Hubble flow buffer for real-time L3, L4, and L7 packet decisions with fine-grained filters (verdict: DROPPED, source pod, destination port).

3. Guarded Actions (Dry-Run First)

When remediation is necessary, actions are strictly constrained:

• act.rollout_restart: Triggers a staged rollout restart under server-side dry-run validation.
• act.scale: Adjusts workload replica counts within strict delta clamps.

eBPF Datapath Mechanics and Drop Reason Codes

In standard Kubernetes networking without Cilium, tracking down connectivity failures between pods requires executing tcpdump across overlay interfaces or navigating thousands of iptables rules generated by kube-proxy. This approach suffers from linear search latency and offers no visibility into socket-level terminations.

Cilium attaches compiled eBPF bytecode directly to Linux kernel hooks: socket operations (sockops), socket message programs (sockmsg), and Traffic Control (tc) filters on container virtual ethernet (veth) interfaces.

Every container endpoint is allocated a local numeric security identity (a 16-bit integer) derived from its Kubernetes labels and namespace. Network policy enforcement executes directly within kernel BPF hash maps (cilium_policy_*). When the kernel terminates a packet, Cilium's datapath emits a structured event into the cilium_events perf ring buffer, which Hubble reads over local UNIX domain sockets.

Key eBPF datapath drop reasons surfaced by cilium-mcp:

By querying Hubble over gRPC instead of parsing raw kernel messages, cilium-mcp delivers deterministic identity mappings directly to the model.

The Structured Output Contract

A core architectural principle of cilium-mcp is the mandatory output envelope. Every diagnostic tool in the server returns data formatted according to a shared JSON schema, preventing the LLM from hallucinating missing telemetry:

{
  "executive_summary": "Egress packets from 'order-service' to 'inventory-db' are rejected by Cilium eBPF policy engine.",
  "top_findings": [
    {
      "severity": "HIGH",
      "confidence": 0.98,
      "title": "CiliumNetworkPolicy 'restrict-db-egress' dropping traffic",
      "description": "Destination endpoint lacks required label 'tier: data', triggering drop reason 133."
    }
  ],
  "evidence": {
    "dropped_flows_count": 312,
    "drop_reason": "Policy denied (Identity 1042 -> 3105)",
    "policy_name": "restrict-db-egress",
    "namespace": "production"
  },
  "causal_graph": [
    "Workload label 'tier: data' removed from inventory-db pod template",
    "Cilium assigned new security identity 3105 to inventory-db",
    "CiliumNetworkPolicy rule #1 allows egress only to identity matching 'tier: data'",
    "eBPF tc egress filter dropped SYN packets at socket layer"
  ],
  "actions": [
    {
      "action_type": "patch_label",
      "target": "deployment/inventory-db",
      "patch": "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"tier\":\"data\"}}}}}"
    }
  ],
  "commands": [
    "hubble observe --namespace production --verdict DROPPED --to-pod inventory-db",
    "cilium-dbg endpoint get 1042"
  ],
  "rollback_and_verification": {
    "verification_command": "hubble observe --namespace production --follow --to-pod inventory-db",
    "expected_result": "Verdict: FORWARDED"
  },
  "meta": {
    "execution_time_ms": 118,
    "version": "1.0.0"
  }
}

This envelope provides three operational benefits:

1. Ranked Confidence: Findings carry explicit severity and confidence ratings, keeping the model focused on verifiable issues rather than benign informational logs.
2. Deterministic Causal Graph: Forces the agent to evaluate the sequence of dependency changes before proposing remediations.
3. Reproducible Operator Verification: The commands array supplies copy-pasteable CLI commands so that on-call engineers can independently verify the model conclusions in their own shell.

Packet Mechanics in Action: Diagnosing an eBPF Policy Drop

Consider a scenario where an API service deployed in Kubernetes cannot reach an internal database backend.

When an agent running with cilium-mcp investigates the path using investigate.flow_query, it receives structured flow records directly from the Hubble buffer:

{
  "time": "2026-10-11T14:22:08.412Z",
  "verdict": "DROPPED",
  "drop_reason_desc": "Policy denied",
  "drop_reason": 133,
  "source": {
    "identity": 1042,
    "namespace": "production",
    "pod_name": "order-service-78f99bc74-kd92x",
    "labels": ["app=order-service", "env=prod"]
  },
  "destination": {
    "identity": 3105,
    "namespace": "production",
    "pod_name": "inventory-db-54dc784cb-m28ql",
    "labels": ["app=inventory-db"]
  },
  "traffic": {
    "protocol": "TCP",
    "source_port": 54210,
    "destination_port": 5432
  }
}

Because the output is structured, the agent compares the destination pod labels (["app=inventory-db"]) against the active CiliumNetworkPolicy manifest:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "restrict-db-egress"
  namespace: "production"
spec:
  endpointSelector:
    matchLabels:
      app: order-service
  egress:
  - toEndpoints:
    - matchLabels:
        app: inventory-db
        tier: data        # Required label missing from target pod identity 3105
    toPorts:
    - ports:
      - port: "5432"
        protocol: TCP

In two tool invocations, the agent pinpoints that inventory-db lacks tier: data, causing the eBPF datapath to drop traffic with reason code 133 (Policy denied). The agent did not need to run unbounded log queries, grep through kernel rings, or guess at network interfaces.

Action Guardrails: Preventing Production Accidents

Allowing an AI model to analyse telemetry is low risk; allowing it to execute changes requires programmatic controls. In cilium-mcp, action tools (act.*) enforce three layers of defence:

1. Mandatory Server-Side Dry Run

Every mutation endpoint requires a boolean dry_run parameter. When dry_run: true is passed (the default), the server executes the mutation using Kubernetes server-side dry run (--dry-run=server). The server returns the simulated result, allowing the model to verify that its patch satisfies OpenAPI validation before live application.

2. Approval Tokens for Live Mutations

To prevent unvetted automated execution, the server can be configured with an approval secret:

export CILIUM_MCP_APPROVAL_TOKEN='sec-auth-98421-prod'

If an agent attempts to execute an action with dry_run: false without supplying the matching token, the request is rejected with 403 Forbidden. In production pipelines, a human reviewer inspects the proposed patch and unlocks the execution.

3. Blast-Radius Limits

To prevent compute exhaustion, the server enforces replica change thresholds:

export CILIUM_MCP_MAX_SCALE_DELTA=5

If an agent attempts to scale a deployment by more than 5 replicas in a single step, the call is rejected immediately.

4. Immutable Audit Trail

Every executed action appends a JSON record to an audit log:

export CILIUM_MCP_AUDIT_LOG_PATH='/var/log/cilium-mcp-audit.jsonl'

The log records the tool name, agent input arguments, target resource, timestamp, and the identity that authorised the execution.

Running cilium-mcp Locally

The server can be deployed locally for interactive debugging or containerised as a cluster-internal service.

1. Installation

Clone the repository and install dependencies in a virtual environment:

git clone https://github.com/technoxi/cilium-mcp.git
cd cilium-mcp
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

2. Execution Modes

For local execution with Claude Desktop or Cursor:

cilium-mcp-server

To run as an HTTP microservice for agent frameworks:

cilium-mcp-server --transport http --host 127.0.0.1 --port 8000 --path /mcp

3. Direct CLI Verification

The repository includes a test CLI in scripts/mcp_cli_test.py to test tool invocation directly:

# List registered MCP tools
python3 scripts/mcp_cli_test.py --url http://127.0.0.1:8000/mcp list-tools

# Query namespace inventory
python3 scripts/mcp_cli_test.py --url http://127.0.0.1:8000/mcp call \
  --tool context.namespace_inventory \
  --args '{"namespace":"production"}'

# Execute Root Cause Analysis dossier
python3 scripts/mcp_cli_test.py --url http://127.0.0.1:8000/mcp call \
  --tool context.rca_pack \
  --args '{"namespace":"production","workload":"order-service","since_minutes":10}'

Engineering Takeaways

1. Structured lenses beat raw shells: Language models make significantly better decisions when supplied with curated, typed telemetry envelopes rather than unconstrained command-line text.
2. eBPF data provides deterministic ground truth: Because Cilium captures network identities and packet drop reasons at the kernel layer, eBPF telemetry provides the exact data points models need to diagnose distributed microservice failures.
3. Safety must be programmatic: Prompt instructions to "be careful" cannot guarantee production safety. Guardrails require dry-run enforcement, parameter delta clamps, and cryptographic approval tokens built directly into the tool layer.