tutorial

How to Monitor KubeOVN with Vigilmon

KubeOVN is an enterprise-grade Kubernetes networking CNI plugin that leverages OVN (Open Virtual Network) and OVS (Open vSwitch) — the same networking stack ...

KubeOVN is an enterprise-grade Kubernetes networking CNI plugin that leverages OVN (Open Virtual Network) and OVS (Open vSwitch) — the same networking stack used in OpenStack and VMware NSX-T — to provide rich networking capabilities for Kubernetes clusters. Unlike simpler CNIs such as Flannel, KubeOVN delivers subnet management, static IP assignment, VPC tenant isolation, built-in L4 load balancing, per-pod QoS, and NAT policies all in one stack. That richness comes with a more complex failure surface: OVN NB/SB databases, OVS bridges, per-node daemon pods, and the kube-ovn-controller must all be healthy simultaneously for pod networking to work.

This tutorial shows you how to monitor KubeOVN's critical health layers with Vigilmon so you detect networking failures before they become pod connectivity outages.

Why KubeOVN needs external monitoring

KubeOVN's multi-layer architecture means failures can be localized (one node's OVS bridge goes down) or catastrophic (OVN Northbound DB leader election fails and all network provisioning stops). In either case, the failure is often not immediately visible through Kubernetes pod status — pods can show Running while OVN flow programming has stalled and new traffic rules are not being installed.

External monitoring catches what Kubernetes internal probes miss:

  • kube-ovn-controller crash — the controller managing logical OVN network objects exits; new pods, services, and subnets get no OVN programming; existing flows continue working temporarily but changes stop being applied
  • OVN NB DB leader election failure — the Northbound database stores logical network topology; a leader election failure stops all KubeOVN network provisioning operations
  • OVN SB DB failure — the Southbound database stores physical bindings and OpenFlow rules; an SB failure halts flow programming to all nodes
  • kube-ovn-daemon missing on a node — without the per-node daemon, that node's OVS bridge cannot be programmed; pods scheduled there get no working network interfaces
  • OVS bridge down — if the br-int or br-ovn bridge on any node goes down, all pods on that node lose network connectivity
  • Subnet IP pool exhaustion — a subnet approaching 100% IP utilization starts failing to schedule new pods in its namespace
  • OVN flow programming lag — high pod churn or load causes OVN southbound programming to fall behind; new pod flows are not installed within acceptable time

What you'll need

  • A Kubernetes cluster with KubeOVN installed (namespace: kube-system or kube-ovn)
  • kubectl access with permission to read KubeOVN resources
  • A free Vigilmon account

Step 1: Expose the kube-ovn-monitor health endpoint

KubeOVN ships a built-in monitoring component (kube-ovn-monitor) that exposes metrics and a health endpoint. Use it as your primary health probe:

# Verify kube-ovn-monitor is running
kubectl get pods -n kube-system -l app=kube-ovn-monitor

# Check the monitor service
kubectl get svc kube-ovn-monitor -n kube-system

# Test the metrics endpoint
kubectl port-forward -n kube-system svc/kube-ovn-monitor 10661:10661 &
curl http://localhost:10661/metrics | grep "^kube_ovn" | head -20

Expose the monitor externally:

# kube-ovn-monitor-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: kube-ovn-monitor-external
  namespace: kube-system
spec:
  type: NodePort
  selector:
    app: kube-ovn-monitor
  ports:
    - port: 10661
      targetPort: 10661
      nodePort: 30661
      protocol: TCP

Or route via Ingress:

# kube-ovn-monitor-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kube-ovn-monitor-health
  namespace: kube-system
spec:
  rules:
    - host: kube-ovn-monitor.internal.example.com
      http:
        paths:
          - path: /metrics
            pathType: Prefix
            backend:
              service:
                name: kube-ovn-monitor
                port:
                  number: 10661

Step 2: Monitor the kube-ovn-controller health

The controller is the heart of KubeOVN — it translates Kubernetes network objects into OVN logical topology. Add an HTTP monitor targeting its health endpoint:

# Find the controller health port
kubectl describe deployment kube-ovn-controller -n kube-system | grep -A5 Ports

# Expose it
kubectl expose deployment kube-ovn-controller \
  --name=kube-ovn-controller-health \
  --type=NodePort \
  --port=10660 \
  --target-port=10660 \
  -n kube-system

In Vigilmon:

  1. Go to Monitors → New Monitor → HTTP / HTTPS
  2. URL: http://kube-ovn.internal.example.com:10660/healthz
  3. Check interval: 1 minute
  4. Expected response: status code 200
  5. Name: kube-ovn-controller health

When the controller crashes, OVN logical network updates stop — no new Subnets, no IP allocations from new Pods, no VPC routing changes. Existing traffic flows continue briefly (OVN southbound flows persist), but the cluster becomes incapable of network provisioning changes.

Step 3: Monitor OVN database health

OVN's NB (Northbound) and SB (Southbound) databases are clustered OVSDB instances. Their health is fundamental — a leader election failure blocks everything.

Check OVN NB database

# Check NB DB pods
kubectl get pods -n kube-system -l app=ovn-central

# Check NB DB cluster status
kubectl exec -n kube-system \
  $(kubectl get pod -n kube-system -l app=ovn-central -o name | head -1) \
  -- ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound

# Check SB DB cluster status
kubectl exec -n kube-system \
  $(kubectl get pod -n kube-system -l app=ovn-central -o name | head -1) \
  -- ovn-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound

Set up a Vigilmon heartbeat to monitor OVN DB cluster health:

# ovn-db-health-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ovn-db-health-check
  namespace: kube-system
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kube-ovn-health-checker
          hostNetwork: true
          restartPolicy: OnFailure
          tolerations:
            - operator: Exists
          nodeSelector:
            ovn-nb-leader: "true"
          containers:
            - name: checker
              image: kubeovn/kube-ovn:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: ovn-db-heartbeat
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Check NB DB leader
                  NB_STATUS=$(ovn-appctl -t /var/run/ovn/ovnnb_db.ctl \
                    cluster/status OVN_Northbound 2>&1 | grep "^Status:")
                  echo "NB DB: $NB_STATUS"
                  echo "$NB_STATUS" | grep -q "leader" || exit 1

                  # Check SB DB leader
                  SB_STATUS=$(ovn-appctl -t /var/run/ovn/ovnsb_db.ctl \
                    cluster/status OVN_Southbound 2>&1 | grep "^Status:")
                  echo "SB DB: $SB_STATUS"
                  echo "$SB_STATUS" | grep -q "leader" || exit 1

                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
                  echo "OVN DBs healthy. Heartbeat sent."

In Vigilmon:

  1. Go to Monitors → New Monitor → Heartbeat
  2. Name: ovn-db-cluster-health
  3. Expected interval: 5 minutes
  4. Grace period: 3 minutes

Step 4: Monitor kube-ovn-daemon DaemonSet coverage

The kube-ovn-daemon runs on every node, managing OVS and local network programming. A missing daemon means pods on that node get no OVN flow installation.

# kube-ovn-daemon-coverage-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-ovn-daemon-coverage
  namespace: kube-system
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kube-ovn-health-checker
          restartPolicy: OnFailure
          containers:
            - name: checker
              image: bitnami/kubectl:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: kube-ovn-daemon-heartbeat
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  DESIRED=$(kubectl get daemonset kube-ovn-daemon -n kube-system \
                    -o jsonpath='{.status.desiredNumberScheduled}')
                  READY=$(kubectl get daemonset kube-ovn-daemon -n kube-system \
                    -o jsonpath='{.status.numberReady}')
                  if [ "$READY" -lt "$DESIRED" ]; then
                    echo "ERROR: kube-ovn-daemon coverage $READY/$DESIRED nodes"
                    exit 1
                  fi
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
                  echo "kube-ovn-daemon $READY/$DESIRED nodes covered."

Step 5: Monitor subnet IP pool utilization

KubeOVN manages per-subnet IP pools. When a subnet approaches full utilization, new pods in that namespace fail to start with no available IP errors. Set up a proactive alert before you hit the ceiling.

# List subnets and their IP utilization
kubectl get subnet -o json | jq -r \
  '.items[] | "\(.metadata.name): \(.status.v4UsingIPs)/\(.status.v4AvailableIPs) used"'

# Find subnets above 80% utilization
kubectl get subnet -o json | jq -r \
  '.items[] | select(
    (.status.v4UsingIPs + .status.v4AvailableIPs) > 0 and
    (.status.v4UsingIPs / (.status.v4UsingIPs + .status.v4AvailableIPs)) > 0.8
  ) | "\(.metadata.name): ALERT - high IP utilization"'

Add this to a CronJob that pings the heartbeat only when all subnets are below 90% utilization:

# subnet-ip-utilization-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: subnet-ip-utilization-check
  namespace: kube-system
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: kube-ovn-health-checker
          restartPolicy: OnFailure
          containers:
            - name: checker
              image: bitnami/kubectl:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: subnet-utilization-heartbeat
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  OVERLOADED=$(kubectl get subnet -o json | jq '
                    [.items[] |
                      select(
                        (.status.v4UsingIPs + .status.v4AvailableIPs) > 0 and
                        (.status.v4UsingIPs /
                          (.status.v4UsingIPs + .status.v4AvailableIPs)) > 0.9
                      )
                    ] | length')
                  if [ "$OVERLOADED" -gt 0 ]; then
                    echo "ERROR: $OVERLOADED subnets above 90% IP utilization"
                    exit 1
                  fi
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null

Step 6: Monitor pod network connectivity end-to-end

The ultimate test of KubeOVN health is whether cross-node pod-to-pod connectivity works. A pod networking test that runs periodically and pings the heartbeat on success gives you an authoritative end-to-end check.

# pod-connectivity-probe.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: pod-connectivity-probe
  namespace: kube-system
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: probe
              image: curlimages/curl:latest
              env:
                - name: TARGET_SVC
                  value: "kube-dns.kube-system.svc.cluster.local"
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: pod-connectivity-heartbeat
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Test DNS resolution (which requires pod network)
                  nslookup "$TARGET_SVC" > /dev/null
                  # Test HTTP connectivity within cluster
                  curl -fsS --max-time 5 \
                    "http://kube-ovn-monitor.kube-system.svc.cluster.local:10661/metrics" \
                    > /dev/null
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
                  echo "Pod connectivity healthy. Heartbeat sent."

Step 7: Configure alert channels

KubeOVN failures have direct user impact — they break pod networking. Route alerts to your network/platform on-call immediately.

Critical (page immediately)

Assign to your PagerDuty integration:

  • kube-ovn-controller health (HTTP)
  • ovn-db-cluster-health (heartbeat)
  • pod-connectivity-probe (heartbeat)

Warning (Slack notification)

Assign to your #platform-networking Slack channel:

  • kube-ovn-daemon-coverage (heartbeat)
  • subnet-ip-utilization-check (heartbeat)

In Vigilmon, go to Alert Channels → New Channel for each destination and assign to the appropriate monitors.

Step 8: Create a network infrastructure status page

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name: "KubeOVN Network Infrastructure"
  3. Add monitors grouped by layer:
    • Control Plane: kube-ovn-controller health, ovn-db-cluster-health
    • Node Layer: kube-ovn-daemon-coverage
    • Connectivity: pod-connectivity-probe
    • Capacity: subnet-ip-utilization-check
  4. Share with the network and platform engineering teams

Summary

| Monitor | Type | What it catches | |---|---|---| | kube-ovn-controller /healthz | HTTP | Controller crash, no OVN provisioning | | ovn-db-cluster-health | Heartbeat | NB/SB DB leader election failure | | kube-ovn-daemon-coverage | Heartbeat | Missing daemon on node(s) | | pod-connectivity-probe | Heartbeat | Cross-node pod networking broken | | subnet-ip-utilization-check | Heartbeat | Subnet IP pool >90% full | | kube-ovn-monitor metrics | HTTP | Monitor component crash |

KubeOVN's rich feature set means a deeper dependency graph than simpler CNIs. Any layer — controller, OVN databases, per-node daemon, or OVS bridges — can fail independently. Multi-layer monitoring with Vigilmon ensures you detect failures at each level and correlate them to their root cause quickly.

Get started free at vigilmon.online — no credit card required, first monitor running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles