Monitoring Pandora FMS with Vigilmon

Pandora FMS (Pandora Flexible Monitoring System) is an enterprise-grade open source monitoring platform from Artica Soluciones Tecnologicas, widely deployed across government, education, and enterprise environments in Spain and Latin America. It handles agent-based monitoring, SNMP, WMI, network scanning, and custom plugins through a modular server architecture. But Pandora FMS is itself infrastructure — and like all infrastructure, it needs to be monitored. Vigilmon watches Pandora FMS from the outside so you know the moment your monitoring pipeline breaks.

What You'll Set Up

• Pandora FMS console web health monitoring
• Server daemon module health (DataServer, NetworkServer, PluginServer)
• Agent data reception rate monitoring
• Tentacle protocol server health
• MySQL health checks
• Alert queue depth monitoring

Prerequisites

• Pandora FMS 7.x or later installed
• SSH access to the Pandora FMS server
• A free Vigilmon account

Step 1: Monitor the Pandora FMS Console

The Pandora FMS console is a PHP/Apache web interface. A 200 response confirms Apache and PHP are working.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your Pandora console URL: http://your-pandora-server/pandora_console.
4. Set Check interval to 1 minute.
5. Enable Keyword check and enter Pandora FMS to verify the login page loads.
6. Click Save.

If your Pandora console uses SSL, monitor the HTTPS URL and enable Monitor SSL certificate with a 21-day expiry alert.

Step 2: Monitor Pandora FMS Server Processes via Heartbeat

Pandora FMS server runs multiple internal modules (DataServer, NetworkServer, PluginServer, EventServer, AlertServer, ReconServer, WMIServer). All must run for full monitoring capability.

In Vigilmon, create a Cron Heartbeat:

1. Click Add Monitor → Cron Heartbeat.
2. Set timeout to 5 minutes.
3. Copy the heartbeat URL.

Create the health script on the Pandora server:

# /usr/local/bin/pandora-heartbeat.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
PANDORA_LOG="/var/log/pandora/pandora_server.error"

# Check that the pandora_server process is running
if ! pgrep -x "pandora_server" > /dev/null 2>&1; then
    echo "CRITICAL: pandora_server process not running" >&2
    exit 1
fi

# Check server status via the Pandora API
STATUS=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT COUNT(*) FROM tserver WHERE status=1 AND server_keepalive > UNIX_TIMESTAMP()-300;" 2>/dev/null)

if [ -z "$STATUS" ] || [ "$STATUS" -eq 0 ]; then
    echo "CRITICAL: no active Pandora server modules" >&2
    exit 1
fi

curl -fsS "$HEARTBEAT_URL" --max-time 10

Add to crontab:

*/5 * * * * /usr/local/bin/pandora-heartbeat.sh

This checks both the OS process and the server keepalive heartbeat in MySQL.

Step 3: Monitor Agent Data Reception Rate

Pandora FMS agents send XML data files to the server on schedule. If the DataServer stops processing these files, agent data goes stale.

Create a heartbeat that checks the agent data queue:

# /usr/local/bin/check-pandora-agents.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
INCOMING_DIR="/var/spool/pandora/data_in"

# Check that agent files are arriving and being processed
# If the incoming directory has a large backlog, something is wrong
BACKLOG=$(find "$INCOMING_DIR" -name "*.data" -mmin +15 2>/dev/null | wc -l)

if [ "$BACKLOG" -lt 50 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
else
    echo "WARNING: $BACKLOG agent files older than 15 minutes in queue" >&2
fi

*/10 * * * * /usr/local/bin/check-pandora-agents.sh

Add a second check that confirms the agent count matches expectations:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"

# Count agents that have reported in the last 30 minutes
ACTIVE_AGENTS=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT COUNT(*) FROM tagente WHERE ultimo_contacto > DATE_SUB(NOW(), INTERVAL 30 MINUTE) AND disabled=0;" 2>/dev/null)

TOTAL_AGENTS=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT COUNT(*) FROM tagente WHERE disabled=0;" 2>/dev/null)

if [ -n "$ACTIVE_AGENTS" ] && [ -n "$TOTAL_AGENTS" ] && [ "$TOTAL_AGENTS" -gt 0 ]; then
    RATIO=$(( ACTIVE_AGENTS * 100 / TOTAL_AGENTS ))
    if [ "$RATIO" -ge 80 ]; then
        curl -fsS "$HEARTBEAT_URL" --max-time 10
    fi
fi

Step 4: Monitor the Tentacle Server

The Tentacle Protocol (port 41121) is Pandora FMS's custom secure file transfer protocol for agent data delivery. If Tentacle is down, agents can't send data files.

1. In Vigilmon, click Add Monitor → TCP Port.
2. Enter your Pandora FMS server hostname.
3. Set Port to 41121.
4. Set Check interval to 1 minute.
5. Click Save.

This confirms Tentacle is listening. If you want to verify it's accepting connections, also monitor the port with a custom script:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"

# Test Tentacle port connectivity
if timeout 5 bash -c "echo '' > /dev/tcp/localhost/41121" 2>/dev/null; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Step 5: Monitor MySQL Health

Pandora FMS stores all monitoring data, configuration, and events in MySQL. A MySQL failure stops data ingestion, alerting, and the console.

Add a TCP monitor for MySQL:

1. Click Add Monitor → TCP Port.
2. Port: 3306.
3. Check interval: 1 minute.
4. Click Save.

Add a deeper health heartbeat:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"

# Check MySQL is responsive and not overloaded
result=$(mysql -u pandora -ppandora pandora \
    -e "SELECT 1;" 2>/dev/null)

# Check for table growth that might indicate runaway event storage
db_size=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT ROUND(SUM(data_length + index_length) / 1024 / 1024) FROM information_schema.tables WHERE table_schema='pandora';" 2>/dev/null)

# Alert if DB exceeds 50GB (adjust threshold for your environment)
if [ $? -eq 0 ] && [ -n "$result" ] && [ "${db_size:-0}" -lt 51200 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Step 6: Monitor Alert Queue Processing

The AlertServer processes triggered alerts. A backed-up alert queue means incidents may not trigger notifications.

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"

# Check pending alert count
pending=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT COUNT(*) FROM talert_execution_queue WHERE execution_at < NOW();" 2>/dev/null)

# If more than 100 alerts are pending for >5 minutes, the AlertServer is behind
if [ -n "$pending" ] && [ "$pending" -lt 100 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Step 7: Monitor Network Check Latency

The NetworkServer performs ICMP/SNMP checks. If checks consistently take longer than the check interval, network data becomes stale.

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/pqr678"

# Check average network check duration from server stats
avg_ms=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT IFNULL(AVG(execution_time),0) FROM tserver WHERE server_type=1 AND server_keepalive > UNIX_TIMESTAMP()-300;" 2>/dev/null)

# Alert if average check duration exceeds 30 seconds
if [ -n "$avg_ms" ]; then
    threshold=30000
    if [ "${avg_ms%.*}" -lt "$threshold" ]; then
        curl -fsS "$HEARTBEAT_URL" --max-time 10
    fi
fi

Step 8: Detect Event Storms

More than 1000 events/minute usually indicates a monitoring misconfiguration — an alerting loop, a flapping device, or a broken threshold.

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/stu901"

# Count events generated in the last minute
events_per_minute=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT COUNT(*) FROM tevento WHERE utimestamp > UNIX_TIMESTAMP()-60;" 2>/dev/null)

if [ -n "$events_per_minute" ] && [ "$events_per_minute" -lt 1000 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
else
    echo "WARNING: event storm detected - ${events_per_minute} events/minute" >&2
fi

Create a 5-minute heartbeat for this check. Missing = event storm in progress.

Step 9: Configure Alerting

1. In Vigilmon, click Alerts → Add Alert Channel.
2. Choose your notification method:
   • Email — your on-call or NOC address
   • Slack — #monitoring-alerts channel
   • PagerDuty — for P1 outages
   • Webhook — to ITSM or incident management

Recommended Alert Policy

| Monitor | Condition | Severity | |---------|-----------|----------| | Pandora console HTTP | Down for 2 minutes | Critical | | Server process heartbeat | Missing for 10 minutes | Critical | | Agent data reception | Missing for 15 minutes | Warning | | Tentacle port 41121 | Down for 1 minute | Critical | | MySQL port 3306 | Down for 1 minute | Critical | | Alert queue depth | Heartbeat missing | Warning | | Event storm detector | Heartbeat missing | Warning |

Step 10: Monitor Agent Version Distribution

Agents on significantly outdated versions may report metrics incorrectly or fail silently.

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/vwx234"

# Get the most common (current) agent version
current_version=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
    -e "SELECT agent_version FROM tagente GROUP BY agent_version ORDER BY COUNT(*) DESC LIMIT 1;" 2>/dev/null)

# Count agents on old versions (simplistic major version check)
if [ -n "$current_version" ]; then
    current_major=$(echo "$current_version" | cut -d'.' -f1)
    old_agents=$(mysql -u pandora -ppandora -s --skip-column-names pandora \
        -e "SELECT COUNT(*) FROM tagente WHERE CAST(SUBSTRING_INDEX(agent_version,'.',1) AS UNSIGNED) < $((current_major - 2)) AND disabled=0;" 2>/dev/null)

    if [ "${old_agents:-0}" -lt 5 ]; then
        curl -fsS "$HEARTBEAT_URL" --max-time 10
    fi
fi

Conclusion

Pandora FMS gives you comprehensive visibility over your infrastructure — but it's only as valuable as its own health. When the DataServer falls behind, the Tentacle server drops connections, or MySQL slows under load, your monitoring data goes stale without any warning. Vigilmon provides the independent, external health layer that Pandora FMS itself can't provide for itself.

Key signals to monitor in Pandora FMS:

• All server modules (DataServer, NetworkServer, PluginServer, AlertServer) must run continuously
• Agent data queue should have no significant backlog — stale agent files mean stale dashboards
• Tentacle server on port 41121 is the critical path for agent data delivery
• MySQL health is foundational — everything in Pandora flows through it
• Event rate above 1000/minute indicates a misconfiguration to fix immediately

Set these monitors up today and let Vigilmon be the safety net under your safety net.

Start monitoring Pandora FMS with Vigilmon →