tutorial

Monitoring Xen Project Hypervisor with Vigilmon

Xen Project is a Type 1 bare-metal hypervisor — Dom0 is your only window into hypervisor health. Here's how to monitor Dom0 resource utilization, DomU VM status, storage repositories, live migrations, and Xen toolstack health with Vigilmon.

Xen Project is a bare-metal (Type 1) hypervisor — it runs directly on server hardware with no host operating system underneath it. The privileged control domain, Dom0, is the only gateway you have to manage all DomU guest VMs, provide I/O virtualization, and interact with the Xen toolstack (xl/libvirt). When Dom0 becomes resource-starved — its CPU, memory, or disk I/O saturated by guest I/O demands — every guest VM degrades simultaneously. And if the Xen toolstack becomes unresponsive, you lose the ability to create, migrate, or destroy VMs even when the guests themselves are still running. Vigilmon lets you monitor Dom0 health, per-DomU resource usage from xentop, storage repository (SR) capacity, live migration success, and toolstack responsiveness through a combination of HTTP monitors and scripted cron heartbeats.

What You'll Set Up

  • Xen hypervisor health via Dom0 monitoring
  • Dom0 CPU, memory, and disk I/O resource monitoring
  • DomU (guest VM) count and unexpected destruction alerts
  • Per-DomU vCPU utilization via xentop
  • Guest VM memory balloon target tracking
  • Xen Storage Repository (SR) capacity monitoring
  • Live migration success rate tracking
  • Xen network bridge health
  • XL toolstack responsiveness monitoring
  • Xen security advisory (XSA) patch status

Prerequisites

  • Xen Project 4.x+ installed (Dom0 running Linux — Debian, Ubuntu, RHEL, or CentOS)
  • xl toolstack installed (or libvirt with Xen driver)
  • SSH access to Dom0
  • A free Vigilmon account

Step 1: Monitor Dom0 Health (Proxy for Hypervisor Health)

The Xen hypervisor itself has no management interface — you reach it only through Dom0. Dom0 health is the best external proxy for hypervisor health. If Dom0 goes dark, you cannot manage any guest VMs regardless of whether they're still running.

If Dom0 exposes a web interface (Xen Orchestra, XAPI, or XenCenter), add an HTTP monitor:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your Dom0 management URL (e.g., https://xen-host.example.com).
  4. Set Check interval to 1 minute.
  5. Enable Monitor SSL certificate with a 21 day alert window.
  6. Click Save.

For Dom0 with only SSH access, use a TCP Port monitor:

  1. Click Add Monitor → TCP Port.
  2. Enter Dom0's IP address and port 22.
  3. Set Check interval to 1 minute.

Additionally, deploy a minimal health endpoint on Dom0 to get richer status:

# Install a minimal HTTP server on Dom0
python3 -m http.server 8080 &
# Or use a simple Flask/Node health endpoint

Step 2: Track Dom0 Resource Health

Dom0 provides all I/O for DomU guests — network, storage, and PCI passthrough. Dom0 CPU or memory exhaustion causes guest I/O to queue and degrade. Set up a cron heartbeat that monitors Dom0 resource consumption:

  1. Click Add Monitor → Cron Heartbeat.
  2. Name it Xen Dom0 Resource Health.
  3. Set the expected interval to 5 minutes.
  4. Copy the heartbeat URL.

Script to check Dom0 resources:

#!/bin/bash
# /usr/local/bin/xen-dom0-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DOM0_HEARTBEAT_ID"
CPU_THRESHOLD=80    # Dom0 CPU utilization %
MEM_THRESHOLD=85    # Dom0 memory utilization %
IO_THRESHOLD=90     # Disk I/O utilization %

# Check Dom0 CPU (Dom0 is domain 0 in xentop)
DOM0_CPU=$(xentop -b -i 2 2>/dev/null | awk '/Domain-0/ {cpu=$3} END {print cpu}' | \
           tr -d '%')

# Check Dom0 memory
DOM0_MEM=$(free | awk '/Mem:/ {printf "%.0f", $3/$2*100}')

# Check disk I/O
IO_UTIL=$(iostat -dx 1 2 2>/dev/null | awk 'NR>4 && /^[sv]d/ {util=$NF} END {print util}' | \
          tr -d '%' | awk '{m=$1+0; if(m>max) max=m} END {print max}')

HEALTHY=1
[ -n "$DOM0_CPU" ] && [ "${DOM0_CPU%.*}" -ge "$CPU_THRESHOLD" ] && HEALTHY=0
[ "${DOM0_MEM:-0}" -ge "$MEM_THRESHOLD" ] && HEALTHY=0
[ -n "$IO_UTIL" ] && [ "${IO_UTIL%.*}" -ge "$IO_THRESHOLD" ] && HEALTHY=0

if [ "$HEALTHY" -eq 1 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Schedule every 5 minutes:

echo "*/5 * * * * root /usr/local/bin/xen-dom0-check.sh" > /etc/cron.d/xen-dom0-check

Step 3: Monitor DomU (Guest VM) Count and Status

Track the number of running guest domains and alert when a VM disappears unexpectedly:

# List all running domains
xl list

# Count running DomUs (excludes Dom0)
xl list | awk 'NR>1 && !/Domain-0/ {count++} END {print count}'

Create a cron heartbeat for DomU status:

  1. Add a Cron Heartbeat named Xen DomU Status.
  2. Set the expected interval to 5 minutes.

Script:

#!/bin/bash
# /usr/local/bin/xen-domu-status.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DOMU_HEARTBEAT_ID"
EXPECTED_DOMUS=4  # Set to your expected number of running guest VMs

RUNNING=$(xl list 2>/dev/null | awk 'NR>1 && !/Domain-0/ {count++} END {print count+0}')

if [ "$RUNNING" -ge "$EXPECTED_DOMUS" ]; then
    curl -s "$HEARTBEAT_URL"
fi

For guest VMs serving HTTP traffic, add individual Vigilmon HTTP monitors per DomU using each VM's IP address. This gives you fine-grained per-VM alerting independent of Dom0 management.

Step 4: Monitor Per-DomU vCPU Utilization

Each DomU is allocated vCPUs. Sustained 100% vCPU utilization means the guest workload is CPU-bound and may need additional vCPUs or host CPU capacity.

Use xentop for real-time per-domain CPU data:

# One-shot snapshot of all domain CPU usage
xentop -b -i 1 | awk 'NR>3 && !/^$/ {print $1, $3}'

Monitor high-CPU DomUs with a cron heartbeat:

#!/bin/bash
# /usr/local/bin/xen-cpu-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_CPU_HEARTBEAT_ID"
CPU_WARN=95  # Alert if any DomU is above this %

THROTTLED=$(xentop -b -i 2 2>/dev/null | awk -v threshold="$CPU_WARN" \
  'NR>3 && !/Domain-0/ && !/^$/ {
    cpu=$3+0; gsub(/%/,"",cpu)
    if (cpu >= threshold) count++
  } END {print count+0}')

if [ "$THROTTLED" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 5: Monitor Guest VM Memory and Balloon Driver

Xen uses a balloon driver to dynamically adjust DomU memory allocations. If the balloon driver fails to meet a memory target (for example during hot-plug memory changes), the guest may be running with less memory than intended.

Check DomU memory targets:

# Show memory allocation vs target for all domains
xl list -v
# Or via xenstore
xenstore-ls /local/domain | grep memory

Script to detect balloon driver failures:

#!/bin/bash
# /usr/local/bin/xen-memory-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_MEM_HEARTBEAT_ID"

# Check if any domain's current memory differs from target by >10%
BALLOON_FAIL=$(xl list 2>/dev/null | awk 'NR>1 && !/Domain-0/ {
    # xl list columns: name, id, mem(MB), vcpus, state, time
    # In practice, use "xl info" per domain for target vs actual
    print $1
}' | while read DOM; do
    TARGET=$(xl info "$DOM" 2>/dev/null | awk '/current_memory/ {print $3}')
    ACTUAL=$(xl info "$DOM" 2>/dev/null | awk '/total_memory/ {print $3}')
    if [ -n "$TARGET" ] && [ -n "$ACTUAL" ]; then
        DIFF=$(( (TARGET - ACTUAL) * 100 / TARGET ))
        [ "$DIFF" -gt 10 ] && echo "BALLOON_FAIL: $DOM"
    fi
done | wc -l)

if [ "$BALLOON_FAIL" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 6: Monitor Xen Storage Repositories (SR)

Xen stores VM disk images in Storage Repositories — NFS mounts, LVM volume groups, or iSCSI targets. A full SR causes VM disk writes to fail, which corrupts running VMs without giving them a graceful shutdown.

If using XAPI (XenServer/XCP-ng):

# List SR usage via xe
xe sr-list params=name-label,physical-utilisation,physical-size

If using xl without XAPI, monitor the underlying filesystem or LVM:

#!/bin/bash
# /usr/local/bin/xen-sr-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_SR_HEARTBEAT_ID"
SR_THRESHOLD=80  # Alert if SR filesystem >80% full

# For NFS-backed SR
NFS_USAGE=$(df /var/lib/xen/images 2>/dev/null | awk 'NR==2 {gsub("%",""); print $5}')

# For LVM-backed SR
LVM_USAGE=$(vgs --noheadings -o vg_free_count,vg_extent_count xen-vg 2>/dev/null | \
            awk '{printf "%.0f", (1-$1/$2)*100}')

OVERFULL=0
[ -n "$NFS_USAGE" ] && [ "$NFS_USAGE" -ge "$SR_THRESHOLD" ] && OVERFULL=1
[ -n "$LVM_USAGE" ] && [ "$LVM_USAGE" -ge "$SR_THRESHOLD" ] && OVERFULL=1

if [ "$OVERFULL" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 7: Monitor Live Migration and Network Bridges

Live Migration: Xen supports live VM migration between hosts. Failed migrations can leave VMs in an inconsistent state.

Wrap VM migrations with success tracking:

#!/bin/bash
migrate_domu() {
    local DOMAIN=$1
    local TARGET=$2
    local HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_MIGRATION_HEARTBEAT_ID"

    if xl migrate "$DOMAIN" "$TARGET" 2>&1; then
        curl -s "$HEARTBEAT_URL"
        echo "Migration of $DOMAIN to $TARGET: SUCCESS"
    else
        echo "Migration of $DOMAIN to $TARGET: FAILED"
        # No heartbeat ping — Vigilmon will alert on missed interval
    fi
}

Network Bridges: Dom0 provides virtual network bridges (xenbr0, etc.) that all DomUs use. A downed bridge disconnects all guests on that bridge.

#!/bin/bash
# /usr/local/bin/xen-bridge-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_BRIDGE_HEARTBEAT_ID"

# Check all xenbr* bridges are up
BRIDGES_DOWN=$(ip link show 2>/dev/null | grep -E "xenbr[0-9]" | grep -c "DOWN" || true)

if [ "$BRIDGES_DOWN" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL"
fi

Step 8: Monitor XL Toolstack Responsiveness

If xl becomes unresponsive, you cannot create, destroy, or migrate VMs. The toolstack can hang if the Xen event channel is exhausted or xenstore becomes corrupted.

Create a cron heartbeat that times the xl list command:

#!/bin/bash
# /usr/local/bin/xen-toolstack-check.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_TOOLSTACK_HEARTBEAT_ID"
TIMEOUT_SECONDS=10

if timeout "$TIMEOUT_SECONDS" xl list > /dev/null 2>&1; then
    curl -s "$HEARTBEAT_URL"
fi

Schedule every 5 minutes. If xl list hangs and the heartbeat misses its window, Vigilmon alerts — giving you an early warning before the toolstack lock affects operations.

Step 9: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add email, Slack, or PagerDuty.
  2. For Dom0 TCP/HTTP monitors, set Consecutive failures before alert to 2 — a single check during a brief network blip should not page on-call.
  3. For Dom0 resource saturation heartbeats, alert on the first missed ping — resource exhaustion on Dom0 is always a critical event given its blast radius.
  4. For toolstack responsiveness heartbeats, alert on the first missed ping — a hung xl toolstack is an emergency.
  5. For SR disk usage heartbeats, alert immediately when the threshold is crossed — a full SR causes VM write failures and potential corruption.
  6. Set up Maintenance Windows in Vigilmon before Xen host reboots or upgrades:
# Suppress monitors during Xen host maintenance
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"monitor_ids": ["dom0-id", "toolstack-id"], "duration_minutes": 60}'

Conclusion

Xen Project's Type 1 architecture gives you excellent isolation between guest VMs, but that thin hypervisor layer means Dom0 is the single point of observability for everything below the guest. By monitoring Dom0 resources, the XL toolstack, storage repositories, network bridges, and per-DomU vCPU/memory alongside external HTTP checks on guest VMs, Vigilmon gives you the full-stack visibility that Xen's architecture demands. You'll catch Dom0 I/O saturation before it manifests as guest degradation, detect SR capacity crises before VM writes fail, and confirm that live migrations completed cleanly — all in one monitoring dashboard.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles