tutorial

Monitoring Percona PMM with Vigilmon

Percona PMM monitors your databases — but who monitors PMM itself? Here's how to keep your PMM Server, VictoriaMetrics ingestion, QAN pipeline, and Grafana dashboards healthy with Vigilmon.

Percona Monitoring and Management (PMM) is a free, open source database observability platform for MySQL, MariaDB, MongoDB, and PostgreSQL. Its combination of Grafana dashboards, VictoriaMetrics time-series storage, and Query Analytics (QAN) gives DBAs deep visibility into query performance and database health. But PMM is infrastructure itself — and when the PMM Server container crashes, VictoriaMetrics falls behind on ingestion, or ClickHouse fills its disk, your database visibility disappears. Vigilmon gives you the independent health layer that ensures PMM is always monitoring.

What You'll Set Up

  • PMM Server container and Nginx health monitoring
  • Grafana dashboard availability monitoring
  • VictoriaMetrics ingestion health via heartbeat
  • QAN data pipeline health monitoring
  • PMM Client connectivity monitoring
  • Disk usage alerts for VictoriaMetrics and ClickHouse

Prerequisites

  • PMM Server 2.x running (Docker or VM)
  • PMM Clients installed on database hosts
  • A free Vigilmon account

Step 1: Monitor the PMM Server and Grafana

PMM Server exposes Grafana on port 443 (or 80). A 200 response from the Grafana URL confirms the container, Nginx, and Grafana are all running.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter your PMM Server URL: https://your-pmm-server.
  4. Set Check interval to 1 minute.
  5. Enable Keyword check and enter Grafana to verify the Grafana UI loads.
  6. Click Save.

Enable Monitor SSL certificate with a 21-day alert threshold — PMM's Nginx uses TLS and a certificate expiry would lock out your DBAs.

Also add a monitor for the PMM API health endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: https://your-pmm-server/v1/version.
  3. Expected HTTP status: 200.
  4. Check interval: 1 minute.
  5. Save.

This endpoint returning a version JSON confirms the PMM API layer is responsive.

Step 2: Monitor VictoriaMetrics Ingestion Health

PMM 2.x uses VictoriaMetrics instead of Prometheus for time-series storage. If VictoriaMetrics falls behind on ingestion, your Grafana graphs develop gaps and alerts based on time-series data stop firing.

Create a heartbeat that checks the VictoriaMetrics health and ingestion rate:

# /usr/local/bin/check-pmm-victoriametrics.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
PMM_HOST="your-pmm-server"

# Check VictoriaMetrics health endpoint (runs on port 9090 internally, exposed via Nginx)
vm_status=$(curl -fsS "https://${PMM_HOST}/victoriametrics/health" \
    --max-time 10 2>/dev/null)

if echo "$vm_status" | grep -qi "ok\|healthy"; then
    # Also check ingestion rate is non-zero
    ingestion_rate=$(curl -fsS "https://${PMM_HOST}/victoriametrics/metrics" \
        --max-time 10 2>/dev/null | grep "vm_rows_inserted_total" | tail -1 | awk '{print $2}')

    if [ -n "$ingestion_rate" ] && [ "${ingestion_rate%.*}" -gt 0 ]; then
        curl -fsS "$HEARTBEAT_URL" --max-time 10
    fi
fi

If PMM is Docker-based, you can also check from the host:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"

# Check VictoriaMetrics container inside PMM
vm_health=$(docker exec pmm-server curl -fsS http://localhost:9090/health 2>/dev/null)

if echo "$vm_health" | grep -qi "ok"; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Schedule every 2 minutes:

*/2 * * * * /usr/local/bin/check-pmm-victoriametrics.sh

Step 3: Monitor QAN Data Pipeline Health

Query Analytics (QAN) is PMM's flagship feature — it shows per-query latency breakdowns, slow query analysis, and execution plans. QAN data flows from pmm-agent on each database host through pmm-managed to ClickHouse on the PMM Server.

Create a heartbeat that checks QAN data receipt:

# /usr/local/bin/check-pmm-qan.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"
PMM_HOST="your-pmm-server"
PMM_USER="admin"
PMM_PASS="admin"

# Query the PMM QAN API for recent data points
response=$(curl -fsS -u "${PMM_USER}:${PMM_PASS}" \
    "https://${PMM_HOST}/v0/qan/GetMetrics" \
    -H "Content-Type: application/json" \
    -d '{"period_start_from":"'"$(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%SZ)"'","period_start_to":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'","limit":1}' \
    --max-time 15 2>/dev/null)

if echo "$response" | python3 -c "import json,sys; d=json.load(sys.stdin); exit(0 if d.get('total_rows',0)>0 else 1)" 2>/dev/null; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

For a simpler ClickHouse-level check (if you have direct DB access):

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/def456"

# Check ClickHouse has recent QAN rows
recent=$(docker exec pmm-server clickhouse-client \
    --query "SELECT count() FROM pmm.metrics WHERE period_start > now() - INTERVAL 15 MINUTE" 2>/dev/null)

if [ -n "$recent" ] && [ "$recent" -gt 0 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Step 4: Monitor PMM Client Connectivity

Each monitored database host runs pmm-agent. If pmm-agent disconnects, that database's metrics stop flowing to PMM Server.

Create a heartbeat script that queries the PMM API for connected clients:

# /usr/local/bin/check-pmm-clients.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/ghi789"
PMM_HOST="your-pmm-server"
PMM_USER="admin"
PMM_PASS="admin"
MIN_CONNECTED=1  # Adjust to your expected minimum

# Get the list of PMM nodes and their agent status
connected=$(curl -fsS -u "${PMM_USER}:${PMM_PASS}" \
    "https://${PMM_HOST}/v1/inventory/agents?agent_type=PMM_AGENT" \
    --max-time 15 2>/dev/null | \
    python3 -c "import json,sys; agents=json.load(sys.stdin).get('pmm_agent',[]); print(sum(1 for a in agents if a.get('status')=='RUNNING'))" 2>/dev/null)

if [ -n "$connected" ] && [ "$connected" -ge "$MIN_CONNECTED" ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
else
    echo "WARNING: only $connected PMM agents connected (minimum: $MIN_CONNECTED)" >&2
fi

Schedule every 5 minutes. A dropped heartbeat means one or more database hosts has lost PMM visibility.

Step 5: Monitor Disk Usage for VictoriaMetrics and ClickHouse

PMM accumulates metric data fast. VictoriaMetrics stores time-series data and ClickHouse stores QAN data — both can fill their volumes and cause PMM to stop accepting new data.

If PMM runs in Docker:

# /usr/local/bin/check-pmm-disk.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"
MAX_PERCENT=80

# Check PMM data volume usage
pmm_usage=$(df /srv 2>/dev/null | awk 'NR==2{print $5}' | tr -d '%')

# If /srv is where PMM Docker volumes live
if [ -z "$pmm_usage" ]; then
    # Fallback: check from inside the container
    pmm_usage=$(docker exec pmm-server df /srv 2>/dev/null | awk 'NR==2{print $5}' | tr -d '%')
fi

if [ "${pmm_usage:-0}" -lt "$MAX_PERCENT" ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
else
    echo "WARNING: PMM data volume at ${pmm_usage}% (threshold: ${MAX_PERCENT}%)" >&2
fi

For a VM-based PMM installation, check the specific mount points:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/jkl012"

vm_usage=$(df /var/lib/victoriametrics 2>/dev/null | awk 'NR==2{print $5}' | tr -d '%')
ch_usage=$(df /var/lib/clickhouse 2>/dev/null | awk 'NR==2{print $5}' | tr -d '%')

if [ "${vm_usage:-0}" -lt 80 ] && [ "${ch_usage:-0}" -lt 80 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

Schedule every 15 minutes.

Step 6: Monitor Database Query Throughput Baseline

PMM tracks queries-per-second (QPS) per monitored database. A sudden QPS drop — more than 50% below baseline — indicates an application-side problem.

# /usr/local/bin/check-pmm-qps.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/mno345"
PMM_HOST="your-pmm-server"
PMM_USER="admin"
PMM_PASS="admin"

# Query VictoriaMetrics for the MySQL QPS over the last 5 minutes
# Uses the PromQL API exposed by PMM's VictoriaMetrics
qps=$(curl -fsS -u "${PMM_USER}:${PMM_PASS}" \
    "https://${PMM_HOST}/victoriametrics/api/v1/query?query=rate(mysql_global_status_queries[5m])" \
    --max-time 15 2>/dev/null | \
    python3 -c "import json,sys; d=json.load(sys.stdin); results=d.get('data',{}).get('result',[]); print(sum(float(r['value'][1]) for r in results) if results else 0)" 2>/dev/null)

# If QPS is above 0 (active traffic), the baseline is healthy
if [ -n "$qps" ] && (( $(echo "$qps > 0" | bc -l) )); then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
fi

This heartbeat only pings when there is measurable database traffic. If QPS drops to zero during business hours, the heartbeat stops — alerting you to an application issue.

Step 7: Monitor Exporter Health Per Database Type

PMM deploys per-database exporters (mysqld_exporter, postgres_exporter, mongodb_exporter) on each monitored host. An exporter with a high scrape error rate means PMM is collecting incomplete metrics.

# /usr/local/bin/check-pmm-exporters.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/pqr678"
PMM_HOST="your-pmm-server"
PMM_USER="admin"
PMM_PASS="admin"

# Query PMM for exporter scrape error rates (using PromQL via VictoriaMetrics)
error_rate=$(curl -fsS -u "${PMM_USER}:${PMM_PASS}" \
    "https://${PMM_HOST}/victoriametrics/api/v1/query?query=rate(up{job=~'mysql.*|postgres.*|mongodb.*'}[5m])" \
    --max-time 15 2>/dev/null | \
    python3 -c "
import json, sys
d = json.load(sys.stdin)
results = d.get('data', {}).get('result', [])
if not results:
    print(100)  # No data is itself an error
    sys.exit()
# Count how many exporters are down (up metric = 0)
down = sum(1 for r in results if float(r['value'][1]) < 0.5)
print(down)
" 2>/dev/null)

if [ -n "$error_rate" ] && [ "$error_rate" -lt 2 ]; then
    curl -fsS "$HEARTBEAT_URL" --max-time 10
else
    echo "WARNING: $error_rate exporters showing errors or missing" >&2
fi

Step 8: Monitor PMM Version Currency

PMM releases frequent updates with security fixes and bug patches. Running a version more than 2 releases behind is a risk.

# /usr/local/bin/check-pmm-version.sh
#!/bin/bash

HEARTBEAT_URL="https://vigilmon.online/heartbeat/stu901"
PMM_HOST="your-pmm-server"

# Get the installed PMM version
installed=$(curl -fsS "https://${PMM_HOST}/v1/version" --max-time 10 2>/dev/null | \
    python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('version','0'))" 2>/dev/null)

# Get the latest available version from PMM's own update endpoint
latest=$(curl -fsS "https://${PMM_HOST}/v1/updates/Check" \
    -H "Authorization: Basic $(echo -n 'admin:admin' | base64)" \
    --max-time 15 2>/dev/null | \
    python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('latest',{}).get('version','0'))" 2>/dev/null)

if [ -n "$installed" ] && [ -n "$latest" ]; then
    # Simple version comparison: extract major.minor
    inst_minor=$(echo "$installed" | cut -d'.' -f2)
    latest_minor=$(echo "$latest" | cut -d'.' -f2)
    behind=$(( latest_minor - inst_minor ))

    if [ "$behind" -le 2 ]; then
        curl -fsS "$HEARTBEAT_URL" --max-time 10
    else
        echo "WARNING: PMM is $behind minor versions behind (installed: $installed, latest: $latest)" >&2
    fi
fi

Schedule this check daily.

Step 9: Monitor Monitored Database Instance Availability

PMM tracks the up/down state of each monitored database. Use Vigilmon to alert independently when a critical database goes offline.

For each critical database, add a TCP port monitor:

  1. Click Add Monitor → TCP Port.
  2. Enter the database host.
  3. Port: 3306 (MySQL), 5432 (PostgreSQL), or 27017 (MongoDB).
  4. Check interval: 1 minute.
  5. Save.

This gives you a second opinion, independent of PMM — critical for the scenario where PMM itself is down and can't tell you the database is also down.

Step 10: Configure Alerting

  1. In Vigilmon, click Alerts → Add Alert Channel.
  2. Add your DBA team's notification channels:
    • Email — DBA on-call or database team list
    • Slack — #dba-alerts channel
    • PagerDuty — for database or PMM Server outages

Recommended Alert Policy

| Monitor | Condition | Severity | |---------|-----------|----------| | PMM Server / Grafana HTTP | Down for 2 minutes | Critical | | VictoriaMetrics heartbeat | Missing for 10 minutes | Critical | | QAN data pipeline | Missing for 20 minutes | Warning | | PMM Client connectivity | Missing for 10 minutes | Warning | | Disk usage heartbeat | Missing | Warning | | Exporter health | Missing for 10 minutes | Warning | | Database TCP port | Down for 1 minute | Critical |

PMM Server down and VictoriaMetrics ingestion stopped should both page immediately — they mean you have no database observability.

Conclusion

Percona PMM gives DBAs deep visibility into query performance and database health — but only when it's running correctly. VictoriaMetrics ingestion failures create metric gaps. ClickHouse disk exhaustion stops QAN data. PMM Client disconnects create blind spots per database host. Vigilmon provides the external, independent monitoring layer that ensures your database observability platform is always observing.

Key health signals for PMM:

  • PMM Server container — the foundation; monitor its Nginx and Grafana HTTP health
  • VictoriaMetrics ingestion — metric gaps appear silently; monitor actively
  • QAN pipeline — ClickHouse-backed; disk exhaustion kills it; monitor separately
  • PMM Clients — each disconnected agent is a blind database; count them
  • Disk usage — VictoriaMetrics and ClickHouse fill up fast; alert at 80%

Set these monitors up before your DBA team next asks "why does PMM show no data for the last hour?"

Start monitoring Percona PMM with Vigilmon →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles