Monitoring Open vSwitch with Vigilmon

Open vSwitch (OVS) is an open source virtual switch that has become the de facto standard for software-defined virtual networking in Linux environments. Originally developed by Nicira Networks (acquired by VMware in 2012) and donated to the open source community, OVS is integrated into KVM/QEMU hypervisors, Xen, OpenStack Neutron, Kubernetes via OVN (Open Virtual Network), and DPDK high-performance data path deployments. When OVS fails — ovs-vswitchd crashes, ovsdb-server becomes unreachable, or VXLAN tunnels go down — virtual machines and containers lose network connectivity, often silently and with no obvious error visible from the workload side. Vigilmon gives you proactive monitoring across every layer of the Open vSwitch stack.

What You'll Set Up

• ovs-vswitchd process health monitor
• ovsdb-server health monitor
• OVS management port TCP monitor
• Packet drop rate and flow table health via cron heartbeat
• VXLAN/Geneve tunnel health monitor
• OVN northbound and southbound database health check
• OVSDB transaction success rate monitor
• Bridge and port health heartbeat
• Alert channels with appropriate thresholds

Prerequisites

• Open vSwitch installed and running (OVS 2.x or later)
• ovs-vsctl, ovs-dpctl, and ovs-appctl CLI tools available
• OVS management socket accessible (default: /var/run/openvswitch/db.sock)
• A free Vigilmon account

Step 1: Monitor ovs-vswitchd Process Health

ovs-vswitchd is the main OVS daemon. If it crashes, packet forwarding may continue briefly on cached kernel flows but no new programming happens — VMs lose connectivity as flows age out.

Since OVS doesn't expose a built-in HTTP health endpoint, use a cron heartbeat to monitor the daemon:

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 2 minutes.
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
4. On your OVS host, create a health check script:

#!/bin/bash
# /usr/local/bin/ovs-vswitchd-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-vswitchd-heartbeat-id"

# Check ovs-vswitchd is running
if ! pgrep -x ovs-vswitchd > /dev/null; then
  exit 1  # Don't ping — Vigilmon alerts on missed heartbeat
fi

# Check the daemon is responsive via appctl
if ! ovs-appctl version > /dev/null 2>&1; then
  exit 1  # Daemon hung — don't ping
fi

curl -sf "$HEARTBEAT_URL" > /dev/null

chmod +x /usr/local/bin/ovs-vswitchd-check.sh

Add to crontab:

*/2 * * * * /usr/local/bin/ovs-vswitchd-check.sh

Step 2: Monitor ovsdb-server Health

ovsdb-server stores all OVS configuration — bridges, ports, flows, tunnel endpoints. If it crashes, you cannot modify OVS configuration until it recovers, and OVN controllers lose the ability to program logical networks.

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 2 minutes.
3. Copy the heartbeat URL.
4. Create a monitoring script:

#!/bin/bash
# /usr/local/bin/ovsdb-server-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-ovsdb-heartbeat-id"

# Check ovsdb-server is running
if ! pgrep -x ovsdb-server > /dev/null; then
  exit 1
fi

# Check the server is responding to queries
if ! ovs-vsctl show > /dev/null 2>&1; then
  exit 1  # ovsdb-server not responding to vsctl
fi

curl -sf "$HEARTBEAT_URL" > /dev/null

chmod +x /usr/local/bin/ovsdb-server-check.sh

Add to crontab:

*/2 * * * * /usr/local/bin/ovsdb-server-check.sh

Also monitor the OVSDB management port if you expose it for remote controller access:

1. Click Add Monitor → TCP Port.
2. Host: your-ovs-host
3. Port: 6640 (OVSDB passive listening port for SDN controllers)
4. Check interval: 1 minute
5. Click Save.

Step 3: Monitor Packet Drop Rates

Packet drops in OVS indicate flow table misses (packets hitting userspace), kernel data path overload, or tunnel errors. Monitor drops via a cron heartbeat that checks the OVS data path statistics.

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. Create a drop rate monitoring script:

#!/bin/bash
# /usr/local/bin/ovs-drops-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-drops-heartbeat-id"
DROP_THRESHOLD=1000  # Packets dropped per 5-minute window — adjust for your environment

# Get drop stats from the kernel datapath
# Using ovs-dpctl show which reports dropped packets per datapath
DROPS=$(ovs-dpctl show 2>/dev/null | grep -oP 'lost:\K[0-9]+' | paste -sd+ | bc 2>/dev/null)

if [ -z "$DROPS" ]; then
  # Can't read stats — check if OVS is even running
  pgrep -x ovs-vswitchd > /dev/null && curl -sf "$HEARTBEAT_URL" > /dev/null
  exit 0
fi

if [ "$DROPS" -lt "$DROP_THRESHOLD" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi
# Above threshold: don't ping — Vigilmon alerts on missed heartbeat

chmod +x /usr/local/bin/ovs-drops-check.sh

Add to crontab:

*/5 * * * * /usr/local/bin/ovs-drops-check.sh

For more granular per-bridge statistics:

# List stats for each bridge
for bridge in $(ovs-vsctl list-br); do
  echo "Bridge: $bridge"
  ovs-dpctl show "system@$bridge" 2>/dev/null | grep -E "packets|dropped|missed"
done

Step 4: Monitor Flow Table Health

OVS maintains OpenFlow flow tables per bridge. When the flow table approaches its maximum size, flow installation failures cause packet drops and connectivity issues.

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 10 minutes.
3. Copy the heartbeat URL.
4. Create a flow table monitoring script:

#!/bin/bash
# /usr/local/bin/ovs-flows-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-flows-heartbeat-id"
FLOW_LIMIT=250000   # OVS default kernel flow limit — adjust for your kernel config
FLOW_THRESHOLD=80   # Alert at 80% of flow limit

for bridge in $(ovs-vsctl list-br 2>/dev/null); do
  FLOW_COUNT=$(ovs-ofctl dump-flows "$bridge" 2>/dev/null | grep -c "cookie=")
  PCT=$((FLOW_COUNT * 100 / FLOW_LIMIT))
  
  if [ "$PCT" -ge "$FLOW_THRESHOLD" ]; then
    # Flow table filling up — don't ping heartbeat
    exit 0
  fi
done

curl -sf "$HEARTBEAT_URL" > /dev/null

chmod +x /usr/local/bin/ovs-flows-check.sh

Add to crontab:

*/10 * * * * /usr/local/bin/ovs-flows-check.sh

Step 5: Monitor VXLAN/Geneve Tunnel Health

OVS VXLAN and Geneve tunnels carry overlay network traffic between hypervisors. A tunnel going down silently breaks VM-to-VM connectivity across hosts.

For tunnel port availability, add a heartbeat that checks BFD session state:

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 3 minutes.
3. Copy the heartbeat URL.
4. Create a tunnel health script:

#!/bin/bash
# /usr/local/bin/ovs-tunnel-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-tunnel-heartbeat-id"

# Check all tunnel ports are present
BRIDGE=$(ovs-vsctl list-br 2>/dev/null | head -1)
if [ -z "$BRIDGE" ]; then
  exit 1  # No bridges — don't ping
fi

# Count tunnel ports
TUNNEL_COUNT=$(ovs-vsctl list-ports "$BRIDGE" 2>/dev/null | xargs -I{} \
  ovs-vsctl get interface {} type 2>/dev/null | grep -cE "vxlan|geneve|gre")

if [ "$TUNNEL_COUNT" -eq 0 ]; then
  # No tunnels configured — still healthy if tunnels aren't expected
  curl -sf "$HEARTBEAT_URL" > /dev/null
  exit 0
fi

# Check BFD session states (requires BFD configured on tunnel ports)
FAILED_BFD=$(ovs-vsctl find interface bfd_status:state!=up type=vxlan 2>/dev/null | wc -l)

if [ "$FAILED_BFD" -eq 0 ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi
# Any failed BFD sessions: don't ping

chmod +x /usr/local/bin/ovs-tunnel-check.sh

Add to crontab:

*/3 * * * * /usr/local/bin/ovs-tunnel-check.sh

Additionally, verify the VXLAN UDP port is open if you run a software VTEP:

1. Click Add Monitor → TCP Port (or use a UDP check if your version supports it).
2. Host: your-ovs-host
3. Port: 4789 (VXLAN standard port)
4. Check interval: 1 minute
5. Click Save.

Step 6: Monitor OVN Northbound and Southbound Databases

If you use OVN (Open Virtual Network) for Kubernetes or OpenStack logical networking, the OVN northbound and southbound databases are critical. If ovn-northd crashes, logical network programming stops.

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 2 minutes.
3. Copy the heartbeat URL.
4. Create an OVN health check script:

#!/bin/bash
# /usr/local/bin/ovn-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-ovn-heartbeat-id"

# Check ovn-northd is running
if ! pgrep -x ovn-northd > /dev/null; then
  exit 1
fi

# Check northbound database is accessible
if ! ovn-nbctl show > /dev/null 2>&1; then
  exit 1
fi

# Check southbound database is accessible
if ! ovn-sbctl show > /dev/null 2>&1; then
  exit 1
fi

curl -sf "$HEARTBEAT_URL" > /dev/null

chmod +x /usr/local/bin/ovn-check.sh

Add to crontab:

*/2 * * * * /usr/local/bin/ovn-check.sh

For OVN database TCP port monitoring:

1. Click Add Monitor → TCP Port.
2. Host: your-ovn-host
3. Port: 6641 (OVN northbound database) and 6642 (OVN southbound database)
4. Check interval: 1 minute
5. Click Save for each.

Step 7: Monitor OVSDB Transaction Success Rate

All OVS configuration changes — adding bridges, ports, modifying flows — go through OVSDB transactions. A high OVSDB transaction failure rate indicates database corruption, disk issues, or concurrent write contention.

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. Create a transaction health script:

#!/bin/bash
# /usr/local/bin/ovsdb-txn-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-ovsdb-txn-heartbeat-id"

# Test a read-only OVSDB transaction (list bridges)
if ovs-vsctl --timeout=5 list-br > /dev/null 2>&1; then
  # Test a no-op write transaction
  if ovs-vsctl --timeout=5 set open_vswitch . other_config:healthcheck="$(date +%s)" > /dev/null 2>&1; then
    curl -sf "$HEARTBEAT_URL" > /dev/null
  fi
fi
# Failed transactions: don't ping — Vigilmon alerts

chmod +x /usr/local/bin/ovsdb-txn-check.sh

Add to crontab:

*/5 * * * * /usr/local/bin/ovsdb-txn-check.sh

Step 8: Monitor Bridge and Port Count

Unexpected bridge or port disappearance indicates a configuration management failure (Puppet/Ansible/Chef removing OVS config) or a crash that wiped the database.

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 10 minutes.
3. Copy the heartbeat URL.
4. Create a bridge/port count monitor. First, record your expected baseline:

# Run this once to record your baseline bridge and port counts
echo "Bridges: $(ovs-vsctl list-br | wc -l)"
echo "Ports on br-int: $(ovs-vsctl list-ports br-int | wc -l)"

5. Create the monitoring script with your baseline values:

#!/bin/bash
# /usr/local/bin/ovs-topology-check.sh

HEARTBEAT_URL="https://vigilmon.online/heartbeat/your-topology-heartbeat-id"
EXPECTED_BRIDGES=3        # Set to your expected bridge count
MIN_PORTS_BR_INT=10       # Minimum ports expected on br-int (your integration bridge)

BRIDGE_COUNT=$(ovs-vsctl list-br 2>/dev/null | wc -l)
PORT_COUNT=$(ovs-vsctl list-ports br-int 2>/dev/null | wc -l)

if [ "$BRIDGE_COUNT" -ge "$EXPECTED_BRIDGES" ] && [ "$PORT_COUNT" -ge "$MIN_PORTS_BR_INT" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi
# Unexpected topology change: don't ping

chmod +x /usr/local/bin/ovs-topology-check.sh

Add to crontab:

*/10 * * * * /usr/local/bin/ovs-topology-check.sh

Step 9: Configure Alert Channels and Thresholds

1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
2. For ovs-vswitchd and ovsdb-server heartbeats, set the expected interval tight (2 minutes) — these daemons crashing is immediately critical.
3. For packet drop and flow table heartbeats, the 5-10 minute interval provides context without over-alerting on brief spikes.
4. For tunnel health, a 3-minute interval balances responsiveness with BFD convergence time.
5. Enable Recovery notifications on all monitors so you know when OVS comes back after a restart.
6. For OVSDB TCP port monitor, set Consecutive failures to 1 — if the OVSDB port stops accepting connections, SDN controllers cannot configure OVS.

Running on Multiple Hypervisors

If you run OVS across multiple KVM/QEMU or Xen hosts, deploy the heartbeat scripts on each host and create separate Vigilmon monitors for each. Use Monitor groups (or monitor naming conventions like ovs-vswitchd@hypervisor-01) to track each host independently. A single hypervisor losing OVS connectivity may not be visible from a central monitor unless you have per-host checks.

Conclusion

You now have end-to-end visibility across the Open vSwitch stack: ovs-vswitchd and ovsdb-server heartbeats catch daemon crashes before VMs lose connectivity, packet drop monitoring detects flow table misses and data path overload, tunnel health checks catch VXLAN/Geneve failures in overlay networks, and OVN database monitoring protects Kubernetes and OpenStack logical networking. The OVSDB transaction check gives early warning of database write failures before they cascade into network programming breakdowns. For more self-hosted infrastructure monitoring guides, see vigilmon.online.