Monitoring Apache CloudStack with Vigilmon

Apache CloudStack is an open source IaaS platform that lets you build and operate public, private, or hybrid clouds. It manages KVM, VMware, and XenServer hypervisors, handles VM lifecycle, networking, and multi-tenant storage — all through a central Java management server backed by MySQL. When CloudStack's management server goes down, VM provisioning stops, the API becomes unavailable, and tenants lose the ability to manage their own workloads. When MySQL fails, the entire cloud state is inaccessible. Vigilmon monitors every critical layer of your CloudStack deployment: the management plane, database, hypervisor agents, System VMs, and storage capacity.

What You'll Set Up

• HTTP uptime monitor for the CloudStack management server
• MySQL database health and connectivity monitoring
• Hypervisor agent heartbeat tracking
• System VM (CPVM and SSVM) health checks
• Primary and secondary storage capacity alerts
• CloudStack API response latency monitoring
• Web UI uptime check

Prerequisites

• Apache CloudStack 4.17+ management server running and accessible over HTTP/HTTPS
• MySQL 8.0+ database (local or remote)
• At least one hypervisor zone configured
• A free Vigilmon account

Step 1: Monitor the CloudStack Management Server

The CloudStack management server exposes a REST/query API and a web portal. Start with an HTTP health check:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: https://cloudstack.yourdomain.com:8080/client/api?command=listApis (or http://cloudstack.yourdomain.com:8080/client for the web portal)
4. Check interval: 1 minute.
5. Expected HTTP status: 200.
6. Click Save.

For a deeper API health check, use the listCapabilities command which does not require authentication:

GET https://cloudstack.yourdomain.com:8080/client/api?command=listCapabilities&response=json

Add Response must contain: "listcapabilitiesresponse" to confirm the API is processing requests, not just accepting TCP connections.

Step 2: Monitor MySQL Database Health

CloudStack stores all cloud state — VM inventory, network configuration, account data, resource allocations — in MySQL. A database failure immediately renders the management server non-functional.

Add a TCP port monitor for MySQL:

1. Click Add Monitor → TCP Port Check.
2. Host: your-mysql-host.internal (or localhost if co-located).
3. Port: 3306.
4. Check interval: 1 minute.
5. Save.

For deeper database health monitoring, add a cron heartbeat that runs a functional query:

#!/bin/bash
RESULT=$(mysql -h localhost -u cloud -pcloudpassword cloud \
  -e "SELECT COUNT(*) FROM vm_instance WHERE state='Running';" 2>/dev/null)

if [ $? -ne 0 ]; then
  echo "MySQL query failed — database unreachable or auth error"
  exit 1
fi

# Optionally alert if running VM count is unexpectedly zero
# (tune this threshold for your environment)
RUNNING_VMS=$(echo "$RESULT" | tail -1)
echo "Running VMs: $RUNNING_VMS"

curl -s "https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN"

This validates not just TCP connectivity but that the CloudStack schema is accessible and queryable. Create a Cron Heartbeat in Vigilmon with a 2-minute interval.

Step 3: Monitor Hypervisor Agent Connectivity

CloudStack deploys agents on each hypervisor host. These agents handle VM operations, compute resource reporting, and network provisioning. A host agent that loses its heartbeat to the management server means VMs on that host are no longer manageable.

Check agent status via the CloudStack API:

#!/bin/bash
# List all hosts and check for any in non-Up state
API_URL="https://cloudstack.yourdomain.com:8080/client/api"
API_KEY="your-api-key"
SECRET_KEY="your-secret-key"

# Build signed request (simplified — use a CloudStack API client for proper HMAC signing)
RESPONSE=$(curl -s "${API_URL}?command=listHosts&type=Routing&response=json&apikey=${API_KEY}&signature=...")

HOSTS_DOWN=$(echo "$RESPONSE" | jq '[.listhostsresponse.host[] | select(.state != "Up")] | length')

if [ "$HOSTS_DOWN" -gt 0 ]; then
  echo "${HOSTS_DOWN} hypervisor host(s) not in Up state"
  exit 1
fi

curl -s "https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN"

For production use, leverage a CloudStack API client library (Python cs, Java SDK, or cloudmonkey) to handle HMAC-SHA1 request signing:

#!/usr/bin/env python3
import cs
import sys

cloud = cs.CloudStack(
    endpoint="https://cloudstack.yourdomain.com:8080/client/api",
    key="your-api-key",
    secret="your-secret-key"
)

hosts = cloud.listHosts(type="Routing")
down_hosts = [h for h in hosts.get("host", []) if h["state"] != "Up"]

if down_hosts:
    print(f"{len(down_hosts)} host(s) not Up: {[h['name'] for h in down_hosts]}")
    sys.exit(1)

import urllib.request
urllib.request.urlopen("https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN")

Create a Cron Heartbeat with a 5-minute interval. A missed heartbeat or explicit failure means at least one hypervisor host is disconnected from the management plane.

Step 4: Monitor System VM Health

CloudStack deploys two types of System VMs that provide critical infrastructure services:

• Console Proxy VM (CPVM): Proxies web console access to running VMs. If CPVM is down, users lose console access.
• Secondary Storage VM (SSVM): Manages template downloads, ISO uploads, and volume snapshot operations. If SSVM is down, these operations fail.

#!/usr/bin/env python3
import cs
import sys

cloud = cs.CloudStack(
    endpoint="https://cloudstack.yourdomain.com:8080/client/api",
    key="your-api-key",
    secret="your-secret-key"
)

system_vms = cloud.listSystemVms()
vms = system_vms.get("systemvm", [])

unhealthy = [vm for vm in vms if vm["state"] != "Running"]
if unhealthy:
    types = [f"{vm['systemvmtype']}({vm['state']})" for vm in unhealthy]
    print(f"System VMs not Running: {types}")
    sys.exit(1)

import urllib.request
urllib.request.urlopen("https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN")

Create a Cron Heartbeat with a 3-minute interval. System VM failures are recoverable — CloudStack will attempt to restart them — but a CPVM that cannot start often indicates network or storage configuration issues that need attention.

Step 5: Monitor Storage Capacity

CloudStack manages two storage tiers:

• Primary storage: Hypervisor-attached storage for running VM disk volumes (NFS, Ceph, iSCSI, local storage)
• Secondary storage: Zone-level storage for VM templates, ISOs, and volume snapshots

Both tiers alert at 80% utilization.

#!/usr/bin/env python3
import cs
import sys

cloud = cs.CloudStack(
    endpoint="https://cloudstack.yourdomain.com:8080/client/api",
    key="your-api-key",
    secret="your-secret-key"
)

THRESHOLD = 80  # percent
alerts = []

# Primary storage pools
storage_pools = cloud.listStoragePools()
for pool in storage_pools.get("storagepool", []):
    capacity_bytes = pool.get("disksizeallocated", 0)
    total_bytes = pool.get("disksizetotal", 1)
    pct = int(capacity_bytes * 100 / total_bytes)
    if pct > THRESHOLD:
        alerts.append(f"Primary pool '{pool['name']}': {pct}% used")

# Secondary storage capacity
zones = cloud.listZones()
for zone in zones.get("zone", []):
    secondary = cloud.listCapacity(type=6, zoneid=zone["id"])  # type 6 = STORAGE_SECONDARY
    for cap in secondary.get("capacity", []):
        used = cap.get("capacityused", 0)
        total = cap.get("capacitytotal", 1)
        pct = int(used * 100 / total)
        if pct > THRESHOLD:
            alerts.append(f"Secondary storage zone '{zone['name']}': {pct}% used")

if alerts:
    print("\n".join(alerts))
    sys.exit(1)

import urllib.request
urllib.request.urlopen("https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN")

Create a Cron Heartbeat with a 30-minute interval. Primary storage exhaustion causes VM write operations to fail; secondary storage exhaustion blocks template downloads and snapshot creation.

Step 6: Monitor VM Provisioning Success Rate

VM deployment is the core function of CloudStack. A sustained failure rate means your cloud is not delivering its primary service.

#!/usr/bin/env python3
import cs
import sys
from datetime import datetime, timedelta

cloud = cs.CloudStack(
    endpoint="https://cloudstack.yourdomain.com:8080/client/api",
    key="your-api-key",
    secret="your-secret-key"
)

# Check async job results from the last hour
start_time = (datetime.utcnow() - timedelta(hours=1)).strftime("%Y-%m-%dT%H:%M:%S+0000")
jobs = cloud.listAsyncJobs(startdate=start_time)

deploy_jobs = [j for j in jobs.get("asyncjobs", [])
               if j.get("cmd", "").endswith("DeployVMCmd")]

if not deploy_jobs:
    # No VM deployments in the last hour — post heartbeat normally
    import urllib.request
    urllib.request.urlopen("https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN")
    sys.exit(0)

failed = [j for j in deploy_jobs if j.get("jobstatus") == 2]  # 2 = failed
total = len(deploy_jobs)
fail_rate = int(len(failed) * 100 / total) if total > 0 else 0

if fail_rate > 5:
    print(f"VM deployment failure rate: {fail_rate}% ({len(failed)}/{total}) in last hour")
    sys.exit(1)

import urllib.request
urllib.request.urlopen("https://vigilmon.online/api/heartbeat/YOUR_HEARTBEAT_TOKEN")

Create a Cron Heartbeat with a 15-minute interval.

Step 7: Monitor CloudStack API Response Latency

The CloudStack query API serves all management operations. Slow API responses indicate management server overload, database query slowness, or JVM heap pressure on the Tomcat/management server JVM.

Add a Vigilmon HTTP monitor with latency tracking:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://cloudstack.yourdomain.com:8080/client/api?command=listZones&available=true&response=json (listZones does not require authentication)
3. Check interval: 2 minutes.
4. Expected HTTP status: 200.
5. Response must contain: "listzoneresponse".
6. Response time alert: 3000ms.
7. Save.

Step 8: Monitor the CloudStack Web UI

The self-service portal (/client) is what your tenants use to manage VMs. An unavailable UI blocks end-user access even if the API is functional.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://cloudstack.yourdomain.com:8080/client
3. Check interval: 2 minutes.
4. Expected HTTP status: 200.
5. Response must contain: CloudStack (or any string reliably present in the portal HTML).
6. Save.

Step 9: Configure Alerting

In Vigilmon, configure thresholds and notification channels:

| Monitor | Alert Condition | Severity | |---|---|---| | Management server API | Status ≠ 200 | Critical | | MySQL TCP | Port 3306 unreachable | Critical | | MySQL heartbeat query | Heartbeat missed | Critical | | Hypervisor agent status | Any host not Up | High | | CPVM health | Not Running | High | | SSVM health | Not Running | Medium | | Primary storage | >80% full | High | | Secondary storage | >80% full | Medium | | VM provisioning failure | >5% fail rate | High | | API latency | p95 >3s | Medium | | Web UI | Status ≠ 200 | High |

Route Critical alerts (management server + MySQL) to page your on-call engineer immediately. These failures make the entire cloud unmanageable. High alerts (hypervisor agents, System VMs, storage) should alert within minutes — they are serious but do not immediately stop existing VMs from running.

For hypervisor agent alerts, set a 3-failure window to avoid false positives during brief network blips. System VM failures should alert after a single miss since CloudStack will already be attempting recovery.

Conclusion

CloudStack manages your entire private cloud — VMs, networks, storage, templates, and tenant accounts. When it fails, it fails at scale. With Vigilmon, you get management server uptime checks that verify the full API path, database connectivity monitoring at both TCP and query level, agent and System VM health tracking, storage capacity alerts before disks fill, and VM provisioning success rate tracking to catch cloud-wide failures before your tenants file support tickets. Set up these monitors and gain confidence that your private cloud infrastructure is functioning end-to-end.

Get started at vigilmon.online.