Monitoring Amanda Backup with Vigilmon

Amanda (Advanced Maryland Automatic Network Disk Archiver) has been protecting data since 1991 — it pioneered tape rotation schemes, client compression, and multi-host backup coordination that enterprise products later adopted. But Amanda fails quietly: a full holding disk, a stale tape, or a failed amcheck run produces log entries, not phone calls. Vigilmon bridges that gap, turning Amanda's log-based output into real-time alerts before your backup window closes and you find out the hard way that last night's job never completed.

What You'll Set Up

• Amanda server process health monitoring
• Backup job success/failure alerting via heartbeat
• Holding disk usage threshold alerts
• Tape/vtape media availability checks
• Daily amcheck verification alerts
• Last full backup age monitoring
• Amanda catalog integrity checks

Prerequisites

• Amanda 3.5+ (Community Edition or Zmanda) installed and running nightly backups
• Amanda server with /var/log/amanda/ log directory accessible
• A free Vigilmon account

Step 1: Monitor the Amanda Server Process

The Amanda server daemon (amandad) coordinates all backup jobs. If the process is unhealthy, no clients can be backed up. Use Vigilmon's TCP port check to verify the Amanda server is accepting connections.

Amanda clients connect to the server on TCP port 10080 (default):

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP Port.
3. Enter your Amanda server's hostname or IP and port 10080.
4. Set Check interval to 5 minutes.
5. Click Save.

For a richer check, wrap amservice availability in a small health script exposed via HTTP:

#!/bin/bash
# /usr/local/bin/amanda-health.sh
# Returns 0 if Amanda server is operational, 1 if not

if systemctl is-active --quiet amanda || pgrep -x amandad > /dev/null; then
    echo "Content-Type: text/plain"
    echo ""
    echo "OK"
    exit 0
else
    echo "Status: 503 Service Unavailable"
    echo "Content-Type: text/plain"
    echo ""
    echo "Amanda server process not running"
    exit 1
fi

Expose it via a lightweight HTTP server (e.g. python3 -m http.server --cgi) and add an HTTP monitor in Vigilmon pointing to that endpoint.

Step 2: Alert on Backup Job Failures with Heartbeats

Amanda runs nightly backup jobs and logs the result to /var/log/amanda/. The cleanest way to monitor job completion is a Vigilmon cron heartbeat: the backup script pings Vigilmon only when the job succeeds.

Set up the heartbeat monitor:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 25 hours (gives a 1-hour grace window for nightly jobs that run at midnight).
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Add the ping to your Amanda post-backup script:

Create /etc/amanda/DailySet1/amdump-post.sh (or add to an existing post-hook):

#!/bin/bash
# Called by Amanda after the nightly amdump run
CONFIG=$1

# Parse the log to determine success
LOG_FILE="/var/log/amanda/${CONFIG}/log"
FAILURES=$(grep -c "^FAIL" "${LOG_FILE}" 2>/dev/null || echo "0")

if [ "${FAILURES}" -eq 0 ]; then
    # All clients backed up successfully — ping Vigilmon
    curl -s --max-time 10 "https://vigilmon.online/heartbeat/abc123"
    echo "Amanda heartbeat sent: backup succeeded"
else
    echo "Amanda backup had ${FAILURES} failures — heartbeat NOT sent"
fi

Make it executable: chmod +x /etc/amanda/DailySet1/amdump-post.sh

Reference it in your amanda.conf:

# amanda.conf
postscript "/etc/amanda/DailySet1/amdump-post.sh DailySet1"

If the job fails or the server crashes mid-backup, the heartbeat is never sent and Vigilmon alerts after the 25-hour window expires.

Step 3: Monitor Holding Disk Usage

Amanda's holding disk is the staging area where client data is buffered before being written to tape or virtual tape. When the holding disk fills above 80%, Amanda starts failing client backup dumps. Monitor it with a cron job that exports disk usage to a Vigilmon webhook.

Create the disk check script:

#!/bin/bash
# /usr/local/bin/amanda-disk-check.sh
# Reads holding disk path from amanda.conf and alerts Vigilmon if >80%

HOLDINGDISK=$(grep -i "^holdingdisk" /etc/amanda/DailySet1/amanda.conf | awk '{print $2}')
HOLDINGDIR=$(grep -A5 "holdingdisk ${HOLDINGDISK}" /etc/amanda/DailySet1/amanda.conf | grep "directory" | awk '{print $2}')

# Default path if parsing fails
HOLDINGDIR=${HOLDINGDIR:-/var/amanda/holding}

USAGE=$(df -h "${HOLDINGDIR}" | awk 'NR==2{print $5}' | tr -d '%')

VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_WEBHOOK_ID"

if [ "${USAGE}" -gt 80 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"Holding disk at ${USAGE}% — backup may fail\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"Holding disk at ${USAGE}%\"}"
fi

Schedule it every 15 minutes in cron:

*/15 * * * * root /usr/local/bin/amanda-disk-check.sh

In Vigilmon, create a Webhook (Push) monitor and set alerts when the status is down.

Step 4: Monitor Tape and Virtual Tape Media Availability

Amanda manages backup volumes in pools. When all tapes in a pool are full or out-of-rotation, the next backup run fails immediately. Monitor tape availability by parsing amadmin output.

Create the media check script:

#!/bin/bash
# /usr/local/bin/amanda-media-check.sh
CONFIG="DailySet1"
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_MEDIA_WEBHOOK_ID"

# Count tapes in each state
USABLE=$(amadmin ${CONFIG} balance 2>/dev/null | grep -c "^[0-9]")
FULL=$(amadmin ${CONFIG} find 2>/dev/null | grep -i "FULL" | wc -l)

if [ "${USABLE}" -eq 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"No usable tapes available in pool ${CONFIG}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"${USABLE} tapes available, ${FULL} full\"}"
fi

Schedule daily before the backup window:

0 20 * * * root /usr/local/bin/amanda-media-check.sh

Step 5: Alert on amcheck Failures

Amanda's amcheck tool verifies that all clients are reachable, the Amanda server can communicate with them, and backup media is ready. Running amcheck each morning before the nightly backup is a best practice — failures mean that night's backup will likely fail.

Create the amcheck wrapper:

#!/bin/bash
# /usr/local/bin/amanda-amcheck.sh
CONFIG="DailySet1"
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_AMCHECK_WEBHOOK_ID"

# Run amcheck and capture exit code
amcheck -M root@localhost ${CONFIG} > /tmp/amcheck-output.txt 2>&1
EXIT_CODE=$?

if [ "${EXIT_CODE}" -eq 0 ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"amcheck passed — all clients and media ready\"}"
else
    ERRORS=$(cat /tmp/amcheck-output.txt | tail -20 | tr '"' "'")
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"amcheck FAILED: ${ERRORS}\"}"
fi

Schedule it to run each morning at 8 AM (well before the evening backup window):

0 8 * * * amanda /usr/local/bin/amanda-amcheck.sh

Step 6: Monitor Last Full Backup Age

Amanda uses incremental levels (0–9), where a level-0 dump is a full backup. Regular full backups are required for recovery. If the last level-0 dump for a client is older than your retention policy, alert immediately.

Create the full backup age check:

#!/bin/bash
# /usr/local/bin/amanda-fullbackup-age.sh
CONFIG="DailySet1"
MAX_DAYS=30  # Alert if full backup older than 30 days
VIGILMON_WEBHOOK="https://vigilmon.online/api/webhook/YOUR_FULLAGE_WEBHOOK_ID"

ALERTS=""
while IFS= read -r client; do
    LAST_FULL=$(amadmin ${CONFIG} find 2>/dev/null | grep "${client}" | grep "level 0" | tail -1 | awk '{print $1}')
    if [ -n "${LAST_FULL}" ]; then
        DAYS_AGO=$(( ($(date +%s) - $(date -d "${LAST_FULL}" +%s 2>/dev/null || echo 0)) / 86400 ))
        if [ "${DAYS_AGO}" -gt "${MAX_DAYS}" ]; then
            ALERTS="${ALERTS}${client}: full backup ${DAYS_AGO} days ago; "
        fi
    fi
done < <(amhost ${CONFIG} 2>/dev/null | awk '{print $1}' | sort -u)

if [ -n "${ALERTS}" ]; then
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"down\", \"message\": \"Stale full backups: ${ALERTS}\"}"
else
    curl -s -X POST "${VIGILMON_WEBHOOK}" \
        -H "Content-Type: application/json" \
        -d "{\"status\": \"up\", \"message\": \"All clients have recent full backups\"}"
fi

Step 7: Configure Alert Channels

1. In Vigilmon, go to Alert Channels and add your preferred channel: email, Slack, PagerDuty, or a webhook.
2. For backup failure and amcheck monitors, set Alert immediately on first failure — backup issues are always urgent.
3. For the Amanda server TCP monitor, set Consecutive failures before alert to 2 to avoid false positives from brief network blips.
4. For the holding disk monitor, set thresholds: warn at 80%, critical at 90%.

Summary

| Monitor | Type | What It Catches | |---|---|---| | Amanda server process | TCP port 10080 | amandad crash | | Nightly backup job | Cron heartbeat (25h) | Backup run failure or skip | | Holding disk usage | Webhook push | Disk full — backup staging area | | Tape/vtape availability | Webhook push | No writable media for next run | | amcheck morning check | Webhook push | Client or media not ready | | Last full backup age | Webhook push | Overdue level-0 dump |

Amanda has protected data reliably for over 30 years, but it was designed in an era when a sysadmin read every log file every morning. With Vigilmon watching your holding disk, tape media, job success, and amcheck status, you get the same visibility without the daily log archaeology — and alerts before your next backup window opens, not after it closes.