Monitoring Zenoss Core with Vigilmon

Zenoss Core is the open source edition of the Zenoss enterprise IT monitoring platform, built on Zope and a Python/ZODB stack. It unifies monitoring for servers, network devices, storage, applications, and virtual infrastructure using SNMP, WMI, SSH, and JMX. But Zenoss Core has a blind spot: when its own collection daemons crash or fall behind, you lose visibility across your entire environment — and Zenoss itself won't tell you. Vigilmon closes that loop by monitoring Zenoss from the outside, so you know the moment Zenoss stops knowing.

What You'll Set Up

• Zenoss daemon health checks (zenhub, zenmodeler, zenperfsnmp, zenping, zenstatus)
• Device modeling cycle time alerts
• SNMP collection cycle completion monitoring
• RabbitMQ queue depth alerts
• MySQL health monitoring
• Device availability coverage alerts

Prerequisites

• Zenoss Core 6.x installed and managing at least one device
• SSH access to the Zenoss server
• A free Vigilmon account

Step 1: Monitor the Zenoss Web Interface

Before monitoring the internals, confirm the Zenoss web application itself is up. Zenoss serves its UI on port 8080 by default.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Zenoss URL: http://your-zenoss-server:8080/zport/dmd.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

If Zenoss is behind a reverse proxy (nginx or Apache), use the proxy URL instead. A 200 response confirms the Zope application server and zenhub are responsive enough to serve the UI.

Step 2: Create a Daemon Health Check Script

Zenoss runs multiple background daemons. All must be healthy for monitoring to function. Create a health check endpoint that reports daemon status:

# /usr/local/bin/zenoss-health-check.sh
#!/bin/bash

DAEMONS="zenhub zenmodeler zenperfsnmp zenping zenstatus zenactions"
ALL_OK=true

for daemon in $DAEMONS; do
    if ! /opt/zenoss/bin/zenoss status $daemon 2>/dev/null | grep -q "is running"; then
        ALL_OK=false
        echo "FAIL: $daemon is not running"
    fi
done

if $ALL_OK; then
    echo "HTTP/1.1 200 OK"
    echo "Content-Type: text/plain"
    echo ""
    echo "all daemons running"
else
    echo "HTTP/1.1 503 Service Unavailable"
    echo "Content-Type: text/plain"
    echo ""
    echo "daemon failure detected"
fi

Expose this via a simple netcat listener or configure it as a CGI endpoint. Alternatively, use the Zenoss zenoss status command result in a heartbeat script (see Step 3).

Step 3: Daemon Health via Cron Heartbeat

The most reliable way to monitor Zenoss daemons from outside is a cron heartbeat. If any daemon is down, the heartbeat stops pinging Vigilmon.

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Create the heartbeat script on the Zenoss server:

# /usr/local/bin/zenoss-heartbeat.sh
#!/bin/bash

DAEMONS="zenhub zenmodeler zenperfsnmp zenping zenstatus"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

for daemon in $DAEMONS; do
    if ! /opt/zenoss/bin/zenoss status $daemon 2>/dev/null | grep -q "is running"; then
        # A daemon is down — don't ping heartbeat
        exit 1
    fi
done

# All daemons healthy — ping Vigilmon
curl -fsS "$HEARTBEAT_URL" --max-time 10

Add to root's crontab:

*/5 * * * * /usr/local/bin/zenoss-heartbeat.sh

If any Zenoss daemon crashes, the heartbeat stops. Vigilmon alerts you within 5 minutes.

Step 4: Monitor Device Modeling Cycle Time

Zenoss remodels devices to detect configuration changes (new interfaces, file systems, hardware changes). The zenmodeler daemon handles this. A modeling cycle that takes longer than 60 minutes indicates a queue backup.

Create a script that checks the last modeling completion time:

# /usr/local/bin/check-zenoss-modeling.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
LOG="/opt/zenoss/log/zenmodeler.log"
MAX_AGE_MINUTES=70  # Alert if no successful model in 70 minutes

# Find the last successful modeling completion
last_success=$(grep -i "finished modeling" "$LOG" 2>/dev/null | tail -1 | awk '{print $1, $2}')

if [ -z "$last_success" ]; then
    exit 1  # No modeling record found
fi

# Check if it's within the acceptable window
last_ts=$(date -d "$last_success" +%s 2>/dev/null)
now=$(date +%s)
age_minutes=$(( (now - last_ts) / 60 ))

if [ "$age_minutes" -lt "$MAX_AGE_MINUTES" ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Add a second Vigilmon heartbeat with a 70-minute timeout window for this check:

1. Click Add Monitor → Cron Heartbeat.
2. Set timeout to 70 minutes.
3. Run the script every 30 minutes via cron:

*/30 * * * * /usr/local/bin/check-zenoss-modeling.sh

Step 5: Monitor SNMP Collection Cycle Completion

zenperfsnmp collects SNMP performance data on a scheduled cycle. If the collection cycle falls behind, your Zenoss graphs will have gaps and threshold alerts may not fire.

# /usr/local/bin/check-zenoss-snmp-cycle.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
LOG="/opt/zenoss/log/zenperfsnmp.log"

# Check for collection cycle completion in the last 20 minutes
recent_completion=$(grep -i "cycle complete" "$LOG" 2>/dev/null | \
    awk -v cutoff="$(date -d '20 minutes ago' '+%Y-%m-%d %H:%M')" \
    '$0 >= cutoff' | tail -1)

if [ -n "$recent_completion" ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Add a Vigilmon heartbeat with a 20-minute timeout and schedule this check every 10 minutes.

Step 6: Monitor RabbitMQ Queue Depth

Zenoss uses RabbitMQ as its internal message bus between daemons. Deep queues indicate that a consuming daemon is falling behind.

Install the RabbitMQ management plugin if not already enabled:

rabbitmq-plugins enable rabbitmq_management

Create an HTTP monitor in Vigilmon that checks the management API:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-zenoss-server:15672/api/queues/%2F.
3. Enable keyword check and set it to confirm the response doesn't contain a queue with messages over 1000.
4. Set check interval to 2 minutes.

For a more targeted check, use a script-based heartbeat:

#!/bin/bash
RABBIT_API="http://localhost:15672/api/queues/%2F"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
MAX_DEPTH=1000

# Get max queue depth across all queues
max_depth=$(curl -fsS -u guest:guest "$RABBIT_API" | \
    python3 -c "import json,sys; queues=json.load(sys.stdin); print(max((q.get('messages',0) for q in queues), default=0))")

if [ "$max_depth" -lt "$MAX_DEPTH" ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Step 7: Monitor MySQL Health

Zenoss stores events in MySQL and reads them for the event console. A MySQL failure causes event loss.

1. In Vigilmon, click Add Monitor → TCP Port.
2. Enter the Zenoss server hostname.
3. Set Port to 3306.
4. Set check interval to 1 minute.
5. Click Save.

For deeper health checking, create a heartbeat script that verifies MySQL query performance:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"

# Run a simple query and check it completes within 5 seconds
result=$(timeout 5 mysql -u zenoss -pzenoss zenoss_zep \
    -e "SELECT COUNT(*) FROM status LIMIT 1;" 2>/dev/null)

if [ $? -eq 0 ] && [ -n "$result" ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Step 8: Configure Alerting

With monitors in place, configure Vigilmon alerts for your team:

1. Click Alerts in the Vigilmon dashboard.
2. Click Add Alert Channel.
3. Choose your notification method:
   • Email — on-call engineer address
   • Slack — #ops-alerts channel
   • PagerDuty — for P1 daemon-down events
   • Webhook — to your incident management tool

Recommended Alert Policy

| Monitor | Condition | Severity | |---------|-----------|----------| | Zenoss web UI | Down for 2 minutes | Critical | | Daemon heartbeat | Missing for 10 minutes | Critical | | Modeling cycle | Missing for 75 minutes | Warning | | SNMP collection | Missing for 25 minutes | Warning | | RabbitMQ queue | Heartbeat missing | Warning | | MySQL TCP | Down for 1 minute | Critical |

For daemon failures, set Notify immediately — every minute a daemon is down is a minute of blind infrastructure monitoring.

Step 9: Monitor zenhub Connection Count

All Zenoss daemons connect through zenhub. If zenhub drops connections, the entire collection pipeline stops.

Create a monitor that checks zenhub's connection metrics via the Zenoss API:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/pqr678"
ZENOSS_URL="http://localhost:8080"
ZENOSS_USER="admin"
ZENOSS_PASS="zenoss"

# Check zenhub connections via the Zenoss JSON API
conn_count=$(curl -fsS -u "${ZENOSS_USER}:${ZENOSS_PASS}" \
    "${ZENOSS_URL}/zport/dmd/Monitors/StatusMonitors/localhost/getHubDaemonList" \
    2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(len(d.get('result',{}).get('data',[])))" 2>/dev/null)

if [ -n "$conn_count" ] && [ "$conn_count" -gt 0 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Alert if this heartbeat goes missing for 5 minutes — zero zenhub connections means no data collection.

Step 10: Monitor Device Availability Coverage

Zenoss tracks device reachability. A sudden drop below 95% availability suggests a network issue or mass device failure.

Create a heartbeat that checks the Zenoss event console for active ping-down events:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/stu901"
ZENOSS_URL="http://localhost:8080"

# Count devices with active ping-down events
down_count=$(curl -fsS -u "admin:zenoss" \
    "${ZENOSS_URL}/zport/dmd/Events/getEventSummary" \
    --data '{"params":{"eventClass":"/Status/Ping","severity":5},"method":"getEventSummary","id":1}' \
    -H "Content-Type: application/json" 2>/dev/null | \
    python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('result',{}).get('totalCount',999))" 2>/dev/null)

# Only ping heartbeat if ping-down count is below threshold
if [ -n "$down_count" ] && [ "$down_count" -lt 10 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Adjust the threshold based on your environment's normal baseline.

Conclusion

Zenoss Core is a powerful monitoring platform, but it has no built-in self-monitoring. Daemons can crash silently, RabbitMQ queues can back up, and modeling cycles can stall — leaving you with a green dashboard that's no longer accurate. Vigilmon gives you an independent, external view of Zenoss health: if your collection pipeline stops, your heartbeats stop, and Vigilmon alerts you before your users notice.

Key things to monitor in Zenoss Core:

• All daemons (zenhub, zenmodeler, zenperfsnmp, zenping, zenstatus) must run continuously
• Modeling cycles under 60 minutes confirm device config discovery is current
• RabbitMQ queue depth under 1000 messages confirms daemon communication is healthy
• MySQL health protects event storage and correlation
• Device availability coverage above 95% is a leading indicator of infrastructure health

Set up these monitors now — before the next silent zenhub crash.

Start monitoring Zenoss Core with Vigilmon →