Monitoring Cilium Hubble with Vigilmon

Cilium Hubble gives you something no traditional CNI can: deep, kernel-level visibility into every network flow traversing your Kubernetes cluster — HTTP requests, DNS queries, TCP connections, and policy enforcement decisions — without touching application code. Cilium (developed by Isovalent, now part of Cisco) uses eBPF to intercept traffic at the Linux kernel level, and Hubble aggregates that data into a queryable, Prometheus-compatible observability layer. When Cilium or Hubble fails, you lose both your Kubernetes networking and your network observability simultaneously. Vigilmon gives you external health monitoring for every layer of the Cilium/Hubble stack — so you know before your network policy silently drops prod traffic or before Hubble Relay goes down and takes your service map with it.

What You'll Set Up

• Cilium Agent health check on each node via DaemonSet status
• Hubble Relay gRPC health monitor
• Hubble UI availability check
• Network policy drop rate alert via Prometheus scrape endpoint
• DNS failure rate monitoring
• HTTP error rate per service tracking
• eBPF map pressure alert
• Observability coverage check (nodes running Cilium)

Prerequisites

• Cilium deployed as the Kubernetes CNI (via Helm or cilium install)
• Hubble enabled (hubble.enabled=true in Helm values)
• Hubble Relay enabled (hubble.relay.enabled=true)
• Hubble UI enabled (optional, hubble.ui.enabled=true)
• Prometheus metrics enabled (hubble.metrics.enabled)
• A free Vigilmon account

Step 1: Monitor Cilium Agent Health

The Cilium Agent (cilium-agent) is a DaemonSet pod running on every Kubernetes node. It manages the eBPF programs that implement networking and enforce policies. If cilium-agent goes down on a node, that node loses all Cilium-managed networking and policy enforcement.

Deploy a monitoring endpoint that checks Cilium Agent health across all nodes:

#!/usr/bin/env python3
# cilium_health_probe.py — deploy in the monitoring namespace
from flask import Flask, jsonify
from kubernetes import client, config

app = Flask(__name__)
config.load_incluster_config()

@app.route('/cilium-agents')
def cilium_agents():
    v1 = client.CoreV1Api()
    pods = v1.list_namespaced_pod(
        namespace="kube-system",
        label_selector="k8s-app=cilium"
    )
    unhealthy = []
    for pod in pods.items:
        if pod.status.phase != "Running":
            unhealthy.append({"node": pod.spec.node_name, "phase": pod.status.phase})
        else:
            for cs in (pod.status.container_statuses or []):
                if not cs.ready:
                    unhealthy.append({"node": pod.spec.node_name, "container": cs.name, "ready": False})
    
    total = len(pods.items)
    if unhealthy:
        return jsonify({"status": "unhealthy", "total": total, "unhealthy": unhealthy}), 503
    return jsonify({"status": "ok", "total": total}), 200

Deploy as a Kubernetes Deployment and Service, then add a Vigilmon monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: http://cilium-probe.monitoring.svc.cluster.local/cilium-agents (or expose externally via Ingress).
4. Check interval: 1 minute
5. Expected HTTP status: 200
6. Under Keyword check, enter "ok".
7. Click Save.

Step 2: Monitor Hubble Relay Health

Hubble Relay aggregates flow data from all Cilium Agents across the cluster and exposes it via a single gRPC endpoint. When Relay goes down, cluster-wide flow querying fails, the Hubble UI loses all data, and any tooling that reads from Relay (security dashboards, compliance systems) goes blind.

Hubble Relay exposes a gRPC health endpoint. Check it via the HTTP health port:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://hubble-relay.kube-system.svc.cluster.local:4245 — Hubble Relay serves gRPC on port 4245. For an HTTP health check, use a sidecar probe or expose the gRPC health as HTTP via Envoy or grpc-health-probe.

For a simpler approach, check Hubble Relay's Kubernetes pod status through the probe service:

@app.route('/hubble-relay')
def hubble_relay():
    v1 = client.CoreV1Api()
    pods = v1.list_namespaced_pod(
        namespace="kube-system",
        label_selector="k8s-app=hubble-relay"
    )
    for pod in pods.items:
        if pod.status.phase == "Running":
            for cs in (pod.status.container_statuses or []):
                if cs.ready:
                    return jsonify({"status": "ok"}), 200
    return jsonify({"status": "unhealthy", "pods_found": len(pods.items)}), 503

Add a Vigilmon monitor:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://cilium-probe.monitoring.svc.cluster.local/hubble-relay
3. Check interval: 1 minute
4. Expected HTTP status: 200

Step 3: Monitor Hubble UI Availability

The Hubble UI is the graphical service map that shows real-time network flows between services. While not critical for cluster networking, its availability is important for your ops and security teams who use it for troubleshooting.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://hubble.yourdomain.com (or http://hubble-ui.kube-system.svc.cluster.local:80 internally).
3. Check interval: 2 minutes
4. Expected HTTP status: 200
5. Under Keyword check, enter Hubble to verify the React app loaded correctly.
6. Enable Monitor SSL certificate if using HTTPS, with a 21 day expiry alert.
7. Click Save.

Step 4: Monitor Network Policy Drop Rate

Cilium enforces Kubernetes NetworkPolicies via eBPF. Policy drops can indicate a misconfigured policy blocking legitimate traffic, or a malicious actor triggering policy enforcement. Hubble exports Prometheus metrics for policy verdicts.

Cilium exposes Prometheus metrics on each node's cilium-agent at port 9962. Scrape the network policy drop metric and expose it via the probe:

import requests as req

CILIUM_METRICS_PORT = 9962

@app.route('/policy-drops')
def policy_drops():
    try:
        # Query Prometheus for cilium policy drop rate
        # This endpoint is accessible from within the cluster
        resp = req.get(
            f"http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query",
            params={"query": "rate(cilium_drop_count_total[5m])"},
            timeout=5
        )
        data = resp.json()
        results = data.get("data", {}).get("result", [])
        
        # Sum drop rate across all nodes
        total_drops_per_sec = sum(float(r["value"][1]) for r in results)
        
        DROP_THRESHOLD = 100  # drops per second — adjust for your environment
        if total_drops_per_sec > DROP_THRESHOLD:
            return jsonify({"status": "spike", "drops_per_sec": round(total_drops_per_sec, 2)}), 503
        return jsonify({"status": "ok", "drops_per_sec": round(total_drops_per_sec, 2)}), 200
    except Exception as e:
        return jsonify({"status": "error", "error": str(e)}), 503

Add a monitor:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://cilium-probe.monitoring.svc.cluster.local/policy-drops
3. Check interval: 2 minutes
4. Expected HTTP status: 200

Step 5: Monitor DNS Query Failure Rate

Hubble captures all DNS queries from pods across the cluster. A high DNS failure rate (NXDOMAIN, SERVFAIL) can indicate CoreDNS issues, network policy misconfiguration blocking DNS traffic, or application misconfiguration querying non-existent domains.

@app.route('/dns-failures')
def dns_failures():
    try:
        resp = req.get(
            f"http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query",
            params={"query": "rate(hubble_dns_queries_total{rcode!=\"No Error\"}[5m]) / rate(hubble_dns_queries_total[5m])"},
            timeout=5
        )
        data = resp.json()
        results = data.get("data", {}).get("result", [])
        
        if not results:
            return jsonify({"status": "ok", "error_rate": 0}), 200
        
        # Average DNS error rate across all namespaces
        rates = [float(r["value"][1]) for r in results if r["value"][1] != "NaN"]
        avg_error_rate = sum(rates) / len(rates) if rates else 0
        
        DNS_ERROR_THRESHOLD = 0.05  # 5% error rate
        if avg_error_rate > DNS_ERROR_THRESHOLD:
            return jsonify({"status": "high_error_rate", "error_rate": round(avg_error_rate, 4)}), 503
        return jsonify({"status": "ok", "error_rate": round(avg_error_rate, 4)}), 200
    except Exception as e:
        return jsonify({"status": "error", "error": str(e)}), 503

Add a Vigilmon monitor on this endpoint with Check interval: 3 minutes.

Step 6: Monitor HTTP Error Rate by Service

Hubble captures Layer 7 HTTP traffic and exports per-service HTTP error rates via Prometheus. This gives you service-level SLI monitoring without any application code changes.

@app.route('/http-errors')
def http_errors():
    try:
        # Check for any service with >5% HTTP 5xx rate
        resp = req.get(
            f"http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query",
            params={
                "query": (
                    "sum by (destination_workload) ("
                    "  rate(hubble_http_responses_total{status=~\"5..\"}[5m])"
                    ") / sum by (destination_workload) ("
                    "  rate(hubble_http_responses_total[5m])"
                    ") > 0.05"
                )
            },
            timeout=5
        )
        data = resp.json()
        results = data.get("data", {}).get("result", [])
        
        if results:
            offenders = [
                {
                    "service": r["metric"].get("destination_workload", "unknown"),
                    "error_rate": round(float(r["value"][1]), 4)
                }
                for r in results
            ]
            return jsonify({"status": "high_error_rate", "services": offenders}), 503
        return jsonify({"status": "ok"}), 200
    except Exception as e:
        return jsonify({"status": "error", "error": str(e)}), 503

Step 7: Monitor eBPF Map Pressure

Cilium uses eBPF maps (kernel data structures) for policy enforcement, connection tracking, and load balancing. eBPF maps have fixed maximum sizes. When a map fills up, Cilium cannot enforce policies for new connections — a silent security and reliability failure.

@app.route('/ebpf-maps')
def ebpf_maps():
    try:
        # cilium_bpf_map_ops_total with "ERROR" in tags indicates map pressure
        resp = req.get(
            f"http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query",
            params={"query": "rate(cilium_bpf_map_ops_total{outcome=\"error\"}[5m])"},
            timeout=5
        )
        data = resp.json()
        results = data.get("data", {}).get("result", [])
        
        error_ops = sum(float(r["value"][1]) for r in results if r["value"][1] != "NaN")
        
        if error_ops > 0:
            return jsonify({"status": "map_errors", "error_ops_per_sec": round(error_ops, 4)}), 503
        return jsonify({"status": "ok"}), 200
    except Exception as e:
        return jsonify({"status": "error", "error": str(e)}), 503

eBPF map errors should be zero in a healthy cluster. Any non-zero value warrants immediate investigation.

Step 8: Monitor Observability Coverage

All nodes in your cluster should be running cilium-agent. Nodes where Cilium is not running create networking and observability blind spots — traffic to and from pods on those nodes bypasses all eBPF policies and Hubble capture.

@app.route('/coverage')
def coverage():
    v1 = client.CoreV1Api()
    
    all_nodes = v1.list_node()
    total_nodes = len(all_nodes.items)
    
    cilium_pods = v1.list_namespaced_pod(
        namespace="kube-system",
        label_selector="k8s-app=cilium"
    )
    nodes_with_cilium = set(pod.spec.node_name for pod in cilium_pods.items 
                            if pod.status.phase == "Running")
    
    coverage_count = len(nodes_with_cilium)
    missing_nodes = [n.metadata.name for n in all_nodes.items 
                     if n.metadata.name not in nodes_with_cilium]
    
    if missing_nodes:
        return jsonify({
            "status": "incomplete",
            "total_nodes": total_nodes,
            "covered_nodes": coverage_count,
            "missing": missing_nodes
        }), 503
    return jsonify({"status": "ok", "total_nodes": total_nodes, "covered_nodes": coverage_count}), 200

Step 9: Configure Alerting

In Vigilmon, go to Alert Channels and add your notification channels:

• Email: network/platform team
• Slack: #network-alerts or #kubernetes-ops via webhook
• PagerDuty: for Cilium Agent failures (a node losing networking is a severity-1 incident)

Recommended thresholds:

| Monitor | Recommended threshold | Severity | |---|---|---| | Cilium Agent health | Alert immediately — any unhealthy agent | P1 | | Hubble Relay health | Alert immediately | P2 | | Hubble UI | Alert after 2 consecutive failures | P3 | | Policy drop rate | Alert on spike above baseline | P2 | | DNS failure rate | Alert at >5% error rate | P2 | | HTTP error rate | Alert on any service >5% 5xx | P2 | | eBPF map errors | Alert immediately — any non-zero value | P1 | | Observability coverage | Alert immediately — missing node | P1 |

Conclusion

A complete Vigilmon setup for Cilium Hubble monitors the entire eBPF networking and observability stack:

• Cilium Agent — catches node-level networking failures before pods lose connectivity
• Hubble Relay — catches observability gaps before your service map goes dark
• Policy drop rate — detects misconfigured policies blocking legitimate traffic
• DNS failure rate — catches CoreDNS issues and policy misconfiguration blocking DNS
• HTTP error rates — gives per-service SLI monitoring without application instrumentation
• eBPF map pressure — detects policy enforcement failures before they become security gaps
• Coverage — ensures no node becomes an unmonitored, unenforced blind spot

With these monitors in place, Cilium/Hubble's visibility into your cluster is itself visible — and you're alerted before a network issue becomes a production incident.

Get started with a free Vigilmon account.