Monitoring Proxmox Backup Server with Vigilmon

Proxmox Backup Server (PBS) is an open source enterprise backup solution from Proxmox Server Solutions, designed for efficient backup of virtual machines and containers in Proxmox VE environments. It uses content-defined chunking for deduplication, dirty-bitmap tracking for incremental backups, client-side AES-256-GCM encryption, and periodic verification to ensure backup integrity. When you run PBS in production, silent failures are the biggest risk: a full datastore that blocks new backups, a garbage collection job that never ran, verification failures that reveal corrupt chunks, or a sync job that stopped replicating to your off-site PBS. None of these surface as visible errors in PVE — they require monitoring PBS directly. Vigilmon keeps watch across every PBS health signal so you know before a restore event reveals the gap.

What You'll Set Up

PBS daemon health via the REST API health endpoint
Datastore disk usage alert at 80% capacity
Backup task success rate monitoring
Last successful backup age per VM/CT
Verification task health monitor
Garbage collection task health monitor
Chunk store integrity monitor
PBS web UI port 8007 reachability check
Deduplication ratio monitor
PBS sync job health monitor

Prerequisites

Proxmox Backup Server installed and running (web UI accessible on port 8007)
PBS REST API accessible (same port 8007, with valid API token)
proxmox-backup-manager CLI available on the PBS host
A free Vigilmon account

Step 1: Monitor PBS Daemon Health

PBS exposes a built-in health endpoint at /api2/json/nodes/localhost/health that returns daemon status. A failed or degraded response indicates the core backup service is unhealthy.

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
URL: https://your-pbs-host:8007/api2/json/nodes/localhost/health
Under Headers, add: Authorization: PBSAPIToken=user@realm!tokenid=<your-api-token> (Create an API token in PBS: Configuration → Access → API Tokens → Add)
Check interval: 1 minute.
Expected HTTP status: 200.
Click Save.

If you prefer not to expose the API token in a monitor header, use a cron heartbeat instead:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-pbs-health-heartbeat-id"

status=$(proxmox-backup-manager status 2>/dev/null | grep -c "running")

if [ "$status" -eq 0 ]; then
  echo "PBS daemon not running"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 2: Monitor Datastore Disk Usage

PBS backups are stored in datastores — directories on the PBS server, optionally on ZFS or LVM. When a datastore's disk fills above ~80%, new backup tasks will start failing. Alert before you hit the limit.

Click Add Monitor → Heartbeat / Cron.
Name: PBS Datastore Disk Usage.
Heartbeat interval: 15 minutes.
Copy the heartbeat URL.

Disk usage check script:

#!/bin/bash
# /usr/local/bin/check-pbs-datastore.sh
HEARTBEAT_URL="https://hb.vigilmon.online/your-datastore-heartbeat-id"
DATASTORE_PATH="/mnt/datastore/main"   # replace with your datastore mount path
WARN_THRESHOLD=80

used_pct=$(df --output=pcent "$DATASTORE_PATH" | tail -1 | tr -d ' %')

if [ "$used_pct" -gt "$WARN_THRESHOLD" ]; then
  echo "PBS datastore at ${used_pct}% — approaching capacity (threshold: ${WARN_THRESHOLD}%)"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Add to cron:

*/15 * * * * /usr/local/bin/check-pbs-datastore.sh

For multiple datastores, create one heartbeat monitor per datastore path and run separate check scripts for each.

Step 3: Monitor Backup Task Success Rate

PBS records every backup task — successful completions, failures, and warnings — in the task log. Alert when backup tasks for critical VMs or containers fail.

Click Add Monitor → Heartbeat / Cron.
Name: PBS Backup Task Success.
Heartbeat interval: 1 hour.

Task success check script:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-backup-tasks-heartbeat-id"
LOOKBACK_HOURS=26   # check last 26 hours

# Query PBS task log for failed backup tasks
failed=$(proxmox-backup-manager task list --limit 500 2>/dev/null \
  | awk -v cutoff="$(date -d "${LOOKBACK_HOURS} hours ago" '+%Y-%m-%dT%H:%M:%S')" \
    '$0 ~ /backup/ && $0 ~ /error/ && $4 > cutoff {count++} END {print count+0}')

if [ "$failed" -gt 0 ]; then
  echo "${failed} backup task(s) failed in the last ${LOOKBACK_HOURS} hours"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Alternative via PBS API:

failed=$(curl -s -k \
  -H "Authorization: PBSAPIToken=user@realm!tokenid=<token>" \
  "https://localhost:8007/api2/json/nodes/localhost/tasks?typefilter=backup&statusfilter=error&limit=100" \
  | python3 -c "import sys,json; tasks=json.load(sys.stdin)['data']; print(len([t for t in tasks if t.get('status','').startswith('TASK ERROR')]))")

Step 4: Monitor Last Successful Backup Age

The most operationally critical PBS metric: how old is the most recent successful backup for each critical VM or container? If a VM's backup schedule silently stopped running, you won't know until you need a restore.

Click Add Monitor → Heartbeat / Cron.
Name: PBS Last Backup Age - <VM Name>.
Heartbeat interval: 1 hour.

Per-VM backup age check:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-backup-age-heartbeat-id"
DATASTORE="main"           # your PBS datastore name
BACKUP_TYPE="vm"           # vm or ct
BACKUP_ID="100"            # VM/CT ID in Proxmox VE
RPO_HOURS=26               # alert if backup older than 26h

# Get the most recent backup snapshot for this VM
latest=$(proxmox-backup-client snapshots \
  --repository "user@pbs@pbs-host:$DATASTORE" \
  "${BACKUP_TYPE}/${BACKUP_ID}" 2>/dev/null \
  | grep -v "^Snapshots" | tail -1 | awk '{print $1}')

if [ -z "$latest" ]; then
  echo "No backups found for ${BACKUP_TYPE}/${BACKUP_ID}"
  exit 1
fi

# Parse backup timestamp from snapshot name (format: YYYY-MM-DDThh:mm:ssZ)
backup_epoch=$(date -d "$latest" +%s 2>/dev/null)
now_epoch=$(date +%s)
age_hours=$(( (now_epoch - backup_epoch) / 3600 ))

if [ "$age_hours" -gt "$RPO_HOURS" ]; then
  echo "Last backup for ${BACKUP_TYPE}/${BACKUP_ID} is ${age_hours}h old (RPO: ${RPO_HOURS}h)"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Create one monitor per critical VM or CT.

Step 5: Monitor Verification Task Health

PBS verification jobs read backup data from the datastore and verify all chunks can be read and match their stored hashes. Failed verification tasks indicate corrupt backups that cannot be restored. This is a silent failure — the backup appears to exist but the restore will fail.

Click Add Monitor → Heartbeat / Cron.
Name: PBS Verification Health.
Heartbeat interval: 24 hours (post heartbeat after each scheduled verify run).

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-verify-heartbeat-id"

# Check for failed verify tasks in the last 7 days
failed=$(proxmox-backup-manager task list --limit 500 2>/dev/null \
  | grep -c "verify.*error" || echo 0)

if [ "$failed" -gt 0 ]; then
  echo "${failed} PBS verification task(s) failed — backup integrity not confirmed"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Schedule your PBS verification job (in the PBS web UI under Datastore → Verify Jobs) at a consistent interval, then set the heartbeat interval to slightly longer than the verify job's expected runtime.

Step 6: Monitor Garbage Collection Health

PBS garbage collection (GC) reclaims disk space from expired or deleted backup snapshots by identifying and removing unreferenced chunks. If GC fails to run, the datastore will grow indefinitely even after you delete old snapshots.

Click Add Monitor → Heartbeat / Cron.
Name: PBS Garbage Collection Health.
Heartbeat interval: 24 hours.

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-gc-heartbeat-id"
MAX_GC_AGE_DAYS=3   # alert if GC hasn't succeeded in 3 days

# Check for successful GC task in the last N days
gc_ok=$(proxmox-backup-manager task list --limit 200 2>/dev/null \
  | awk -v cutoff="$(date -d "${MAX_GC_AGE_DAYS} days ago" '+%Y-%m-%dT%H:%M:%S')" \
    '$0 ~ /garbage_collect/ && $0 ~ /OK/ && $4 > cutoff {found=1} END {print found+0}')

if [ "$gc_ok" -eq 0 ]; then
  echo "PBS garbage collection has not completed successfully in ${MAX_GC_AGE_DAYS} days"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 7: Monitor Chunk Store Integrity

PBS verification jobs surface chunk-level corruption. Monitor the verification output for any chunk errors — a single corrupted chunk can make an entire backup snapshot unrestorable.

Click Add Monitor → Heartbeat / Cron.
Name: PBS Chunk Store Integrity.
Heartbeat interval: 24 hours (run after each scheduled verify job).

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-chunks-heartbeat-id"

# Check verify task output for chunk errors
chunk_errors=$(proxmox-backup-manager task list --limit 200 2>/dev/null \
  | grep -c "verify.*chunk.*error" || echo 0)

if [ "$chunk_errors" -gt 0 ]; then
  echo "${chunk_errors} chunk error(s) detected in PBS verification — backup data may be corrupt"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 8: Monitor the PBS Web UI

The PBS web UI (port 8007) provides access to datastore management, task logs, and backup browsing. An unavailable web UI doesn't stop backups from running, but it prevents operators from reviewing backup status or initiating restores.

Click Add Monitor → HTTP / HTTPS.
URL: https://your-pbs-host:8007/
Check interval: 2 minutes.
Expected HTTP status: 200.
Under SSL/TLS, if using a self-signed certificate, disable certificate verification or upload the PBS CA certificate.
Click Save.

Step 9: Monitor Deduplication Ratio

PBS's content-defined chunking provides significant storage savings through deduplication. A declining dedup ratio may indicate backup data is changing significantly (e.g., large encrypted files that don't deduplicate) or that the chunk index is fragmented.

Click Add Monitor → Heartbeat / Cron.
Name: PBS Deduplication Ratio.
Heartbeat interval: 1 hour.

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-dedup-heartbeat-id"
DATASTORE="main"
MIN_DEDUP_RATIO=1.5   # alert if dedup ratio falls below 1.5x

dedup=$(proxmox-backup-manager datastore info "$DATASTORE" 2>/dev/null \
  | grep -i "dedup" | awk '{print $NF}')

# dedup ratio is reported as e.g. "2.47"
if (( $(echo "${dedup:-0} < $MIN_DEDUP_RATIO" | bc -l) )); then
  echo "PBS datastore $DATASTORE dedup ratio ${dedup} below threshold ${MIN_DEDUP_RATIO}"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 10: Monitor PBS Sync Job Health

If you use PBS sync jobs to replicate backups to an off-site PBS instance, a failing sync job silently breaks your off-site recovery capability. Monitor sync task success and sync lag:

Click Add Monitor → Heartbeat / Cron.
Name: PBS Sync Job Health.
Heartbeat interval: 1 hour (adjust to match your sync job schedule).

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-sync-heartbeat-id"
MAX_SYNC_AGE_HOURS=25   # alert if sync hasn't succeeded in 25h

sync_ok=$(proxmox-backup-manager task list --limit 200 2>/dev/null \
  | awk -v cutoff="$(date -d "${MAX_SYNC_AGE_HOURS} hours ago" '+%Y-%m-%dT%H:%M:%S')" \
    '$0 ~ /sync/ && $0 ~ /OK/ && $4 > cutoff {found=1} END {print found+0}')

if [ "$sync_ok" -eq 0 ]; then
  echo "PBS sync job has not completed successfully in ${MAX_SYNC_AGE_HOURS} hours — off-site backup at risk"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Configuring Alerts

Set up alert channels in Vigilmon to route PBS failures to the right people:

Go to Alert Channels and add email, Slack, PagerDuty, or webhook destinations.
For each monitor, open Settings → Alerting and assign a channel.

Recommended thresholds:

| Monitor | Alert Condition | Severity | |---|---|---| | PBS daemon health | API not 200 | Critical | | Backup task success | Any task failure | Critical | | Last backup age | Older than RPO | Critical | | Chunk integrity | Any chunk error | Critical | | Sync job health | No success in 25h | Critical | | Datastore usage | > 80% full | High | | Verification health | Any verify failure | High | | GC health | No success in 3 days | Warning | | Deduplication ratio | Below 1.5x | Warning | | Web UI | HTTP not 200 | Low |

For the deduplication ratio alert, allow a 1-hour delay to avoid alerting on temporary measurements taken during active backup tasks.

Conclusion

Proxmox Backup Server is designed to be reliable — but silent failures in verification, garbage collection, sync, and disk capacity can leave you without a working backup when you need one most. With Vigilmon continuously monitoring PBS daemon health, datastore capacity, backup task outcomes, backup age against your RPO, verification integrity, and off-site sync status, you'll catch every gap before a restore event makes it critical.

Get started at vigilmon.online.