tutorial

How to Monitor Peering Manager Health with Vigilmon

Peering Manager database failures block all peering session management, Celery worker failures cause BGP polling to go stale while sessions silently fail, and NAPALM connection errors leave your IXP session dashboard showing outdated Established states for hours after tunnels have dropped. Learn how to monitor Peering Manager web app health, PostgreSQL, Redis, Celery workers, and BGP session accuracy with Vigilmon HTTP probes and heartbeat monitors.

Peering Manager is the BGP session and IXP peering management platform used by ISPs, CDNs, and content networks to track and configure hundreds of BGP peering sessions — each representing a traffic exchange agreement with a peer AS. When the PostgreSQL database that stores all peering session records, router configurations, and IXP membership data becomes unreachable due to a failed database host migration, the Peering Manager web application returns 500 errors for every page load and every API call — operators cannot view peering session status, cannot push configuration changes to routers, and cannot identify which sessions are failing during a network incident; when the Celery worker that periodically polls routers via NAPALM for actual BGP session state crashes due to an uncaught exception in a NAPALM driver update, the BGP session status displayed in Peering Manager's UI continues showing the last-known state (often all Established) while actual BGP sessions fail and recover without the database ever being updated — operators see a green dashboard while the network is flapping; when a NAPALM connection to a router fails silently because the router's management IP changed after a hardware replacement and Peering Manager's router inventory was not updated, every BGP session on that router appears permanently in its last-polled state — an IXP route server's 200 peering sessions appear Established in Peering Manager while the actual sessions may be in any state. These are peering management failures that create a dangerous gap between the displayed state of your BGP peering infrastructure and the actual operational state.

Vigilmon gives you external visibility into Peering Manager's operational health through HTTP probe monitoring and heartbeat monitors for Celery worker status, PeeringDB sync freshness, and NAPALM router reachability. This tutorial covers both.

Why Peering Manager Needs External Monitoring

Peering Manager failure modes create a gap between displayed and actual BGP session state:

  • PostgreSQL database failure: Peering Manager stores all BGP peering sessions, AS information, IX memberships, router configurations, and synchronization history in PostgreSQL; a database failure causes all Peering Manager web requests to fail with 500 errors — operators lose the ability to view peering session status, create new sessions, push configuration updates to routers, or investigate incident impact during a BGP event; the application cannot serve any useful information
  • Redis failure causing Celery task queue failure: Celery workers use Redis as the task queue broker; when Redis becomes unreachable, Celery workers cannot receive new task messages — scheduled tasks like PeeringDB synchronization, BGP session polling, and router configuration push queue up without executing; the displayed peering state in the UI freezes at the last time polling ran successfully, which may have been minutes or hours ago
  • Celery worker crash or stuck tasks: Even when Redis is healthy, Celery workers can crash due to NAPALM driver bugs, memory leaks from large router configurations, or unhandled exceptions in task code; when all Celery workers are down, scheduled BGP polling stops, PeeringDB sync stops, and configuration push tasks silently fail — operators submit configuration pushes through the UI and receive no feedback that the task never executed
  • NAPALM router connection failures: Peering Manager uses NAPALM to connect to each managed router and query BGP session state; when a router's management IP changes, SSH credentials rotate, or a firewall rule blocks NAPALM's management access, the connection to that router fails silently; Peering Manager continues displaying the last successfully polled BGP session state for all sessions on that router, which may be hours or days old
  • PeeringDB synchronization staleness: Peering Manager pulls AS information, IX presence, and peering policies from PeeringDB; when the PeeringDB sync fails (API rate limit, PeeringDB downtime, network issue), Peering Manager's AS database becomes stale; operators attempting to configure new peering sessions may see outdated NOC contacts, incorrect IX memberships, or missing peering policy information
  • BGP session count divergence: The expected count of Established BGP sessions in Peering Manager should match the count configured; when NAPALM polling shows sessions that are not Established (the actual state), but the last-polled UI display hasn't been refreshed, operators operate with a false picture of network health

External monitoring with Vigilmon adds:

  • Proactive alerting when the Peering Manager web application returns errors (DB or app failure)
  • Celery worker health tracking through heartbeat monitors on scheduled task execution
  • PeeringDB sync freshness alerts through heartbeat monitors on the sync schedule
  • BGP session accuracy alerts when polled state diverges from expected state

Step 1: Build a Peering Manager Health Endpoint

Peering Manager does not expose a dedicated health endpoint. Build a sidecar that checks application, database, Redis, and Celery health.

Node.js Health Sidecar

// health/peering_manager.js
const express = require('express');
const http = require('http');
const { Client } = require('pg');
const { createClient } = require('redis');
const { execSync } = require('child_process');

const app = express();

const PM_URL = process.env.PEERING_MANAGER_URL || 'http://localhost:8000';
const PM_API_TOKEN = process.env.PEERING_MANAGER_API_TOKEN || '';
const DB_URL = process.env.DATABASE_URL || 'postgresql://peering:password@localhost:5432/peering_manager';
const REDIS_URL = process.env.REDIS_URL || 'redis://localhost:6379';
const EXPECTED_SESSIONS = parseInt(process.env.PM_EXPECTED_BGP_SESSIONS || '0');

function fetchPm(path) {
  return new Promise((resolve, reject) => {
    const url = new URL(path, PM_URL);
    const req = http.get(url.href, {
      headers: PM_API_TOKEN ? { Authorization: `Token ${PM_API_TOKEN}` } : {},
      timeout: 10000,
    }, res => {
      let body = '';
      res.on('data', c => body += c);
      res.on('end', () => {
        try {
          resolve({ status: res.statusCode, body: JSON.parse(body || '{}') });
        } catch {
          resolve({ status: res.statusCode, body: {} });
        }
      });
    });
    req.on('error', reject);
    req.on('timeout', () => { req.destroy(); reject(new Error('timeout')); });
  });
}

async function checkDatabase() {
  const client = new Client({ connectionString: DB_URL, connectionTimeoutMillis: 5000 });
  await client.connect();
  const start = Date.now();
  await client.query('SELECT 1');
  const latencyMs = Date.now() - start;
  await client.end();
  return latencyMs;
}

async function checkRedis() {
  const client = createClient({ url: REDIS_URL, socket: { connectTimeout: 3000 } });
  client.on('error', () => {});
  const start = Date.now();
  await client.connect();
  await client.ping();
  const latencyMs = Date.now() - start;
  await client.disconnect();
  return latencyMs;
}

async function getCeleryWorkerCount() {
  // Try celery inspect active workers via CLI (requires same virtualenv as app)
  try {
    const output = execSync(
      'celery -A peering_manager inspect ping --timeout 5 2>/dev/null | grep -c "pong"',
      { timeout: 10000 }
    ).toString().trim();
    return parseInt(output) || 0;
  } catch {
    return -1;
  }
}

async function getBgpSessionStats() {
  if (!PM_API_TOKEN) return {};
  try {
    const { status, body } = await fetchPm('/api/peering/bgp-groups/?format=json&limit=1');
    if (status !== 200) return { api_error: status };
    // Query sessions in non-Established state
    const sessResp = await fetchPm('/api/peering/direct-peering-sessions/?status__n=established&limit=1');
    const failedCount = sessResp.body.count || 0;
    const totalResp = await fetchPm('/api/peering/direct-peering-sessions/?limit=1');
    const totalCount = totalResp.body.count || 0;
    return {
      bgp_sessions_total: totalCount,
      bgp_sessions_not_established: failedCount,
    };
  } catch (err) {
    return { bgp_session_error: err.message };
  }
}

app.get('/health/peering-manager', async (req, res) => {
  const checks = {};
  let healthy = true;

  // Web application health
  try {
    const start = Date.now();
    const { status } = await fetchPm('/');
    checks.webapp_status = status;
    checks.webapp_ms = Date.now() - start;
    if (status >= 500) {
      healthy = false;
      checks.webapp = `error: HTTP ${status}`;
    } else {
      checks.webapp = 'ok';
    }
  } catch (err) {
    checks.webapp = `down: ${err.message}`;
    healthy = false;
  }

  // PostgreSQL check
  try {
    const latencyMs = await checkDatabase();
    checks.postgresql_ms = latencyMs;
    checks.postgresql = 'ok';
    if (latencyMs > 1000) {
      checks.postgresql_warning = `DB query ${latencyMs}ms (>1000ms)`;
    }
  } catch (err) {
    checks.postgresql = `down: ${err.message}`;
    healthy = false;
  }

  // Redis check
  try {
    const latencyMs = await checkRedis();
    checks.redis_ping_ms = latencyMs;
    checks.redis = 'ok';
  } catch (err) {
    checks.redis = `down: ${err.message}`;
    healthy = false;
  }

  // Celery worker check
  try {
    const workerCount = await getCeleryWorkerCount();
    checks.celery_workers = workerCount;
    if (workerCount === 0) {
      checks.celery_warning = 'No active Celery workers detected';
      healthy = false;
    }
  } catch (err) {
    checks.celery = `error: ${err.message}`;
  }

  // BGP session accuracy
  const bgpStats = await getBgpSessionStats();
  Object.assign(checks, bgpStats);
  if (bgpStats.bgp_sessions_not_established > 0) {
    checks.bgp_warning = `${bgpStats.bgp_sessions_not_established} BGP sessions not Established`;
    healthy = false;
  }

  return res.status(healthy ? 200 : 503).json({
    status: healthy ? 'ok' : 'degraded',
    checks,
  });
});

app.listen(3023, () => console.log('Peering Manager health sidecar on :3023'));

Python (FastAPI) Alternative

# health/peering_manager_health.py
import os
import time
import json
import subprocess
import httpx
import psycopg2
import redis as redislib
from fastapi import FastAPI, Response

app = FastAPI()

PM_URL = os.environ.get("PEERING_MANAGER_URL", "http://localhost:8000")
PM_API_TOKEN = os.environ.get("PEERING_MANAGER_API_TOKEN", "")
DB_URL = os.environ.get("DATABASE_URL", "postgresql://peering:password@localhost:5432/peering_manager")
REDIS_HOST = os.environ.get("REDIS_HOST", "localhost")
REDIS_PORT = int(os.environ.get("REDIS_PORT", "6379"))

def pm_headers():
    return {"Authorization": f"Token {PM_API_TOKEN}"} if PM_API_TOKEN else {}

async def check_webapp() -> dict:
    async with httpx.AsyncClient(timeout=10.0) as client:
        start = time.monotonic()
        resp = await client.get(PM_URL + "/", headers=pm_headers(), follow_redirects=True)
        ms = int((time.monotonic() - start) * 1000)
        return {"status": resp.status_code, "ms": ms}

def check_postgresql() -> int:
    conn = psycopg2.connect(DB_URL, connect_timeout=5)
    start = time.monotonic()
    with conn.cursor() as cur:
        cur.execute("SELECT 1")
    ms = int((time.monotonic() - start) * 1000)
    conn.close()
    return ms

def check_redis() -> int:
    r = redislib.Redis(host=REDIS_HOST, port=REDIS_PORT, socket_connect_timeout=3)
    start = time.monotonic()
    r.ping()
    return int((time.monotonic() - start) * 1000)

def check_celery_workers() -> int:
    try:
        result = subprocess.run(
            "celery -A peering_manager inspect ping --timeout 5 2>/dev/null | grep -c pong",
            shell=True, capture_output=True, text=True, timeout=12
        )
        return int(result.stdout.strip() or 0)
    except Exception:
        return -1

async def get_bgp_session_stats() -> dict:
    if not PM_API_TOKEN:
        return {}
    try:
        async with httpx.AsyncClient(timeout=10.0) as client:
            total_resp = await client.get(
                PM_URL + "/api/peering/direct-peering-sessions/?limit=1",
                headers=pm_headers()
            )
            not_est_resp = await client.get(
                PM_URL + "/api/peering/direct-peering-sessions/?status__n=established&limit=1",
                headers=pm_headers()
            )
            return {
                "bgp_sessions_total": total_resp.json().get("count", 0),
                "bgp_sessions_not_established": not_est_resp.json().get("count", 0),
            }
    except Exception as e:
        return {"bgp_session_error": str(e)}

@app.get("/health/peering-manager")
async def peering_manager_health():
    checks = {}
    healthy = True

    try:
        result = await check_webapp()
        checks["webapp_status"] = result["status"]
        checks["webapp_ms"] = result["ms"]
        checks["webapp"] = "ok" if result["status"] < 500 else f"error: HTTP {result['status']}"
        if result["status"] >= 500:
            healthy = False
    except Exception as e:
        checks["webapp"] = f"down: {str(e)}"
        healthy = False

    try:
        ms = check_postgresql()
        checks["postgresql_ms"] = ms
        checks["postgresql"] = "ok"
        if ms > 1000:
            checks["postgresql_warning"] = f"DB query {ms}ms"
    except Exception as e:
        checks["postgresql"] = f"down: {str(e)}"
        healthy = False

    try:
        ms = check_redis()
        checks["redis_ping_ms"] = ms
        checks["redis"] = "ok"
    except Exception as e:
        checks["redis"] = f"down: {str(e)}"
        healthy = False

    try:
        workers = check_celery_workers()
        checks["celery_workers"] = workers
        if workers == 0:
            checks["celery_warning"] = "No active Celery workers"
            healthy = False
    except Exception as e:
        checks["celery"] = f"error: {str(e)}"

    bgp_stats = await get_bgp_session_stats()
    checks.update(bgp_stats)
    if bgp_stats.get("bgp_sessions_not_established", 0) > 0:
        checks["bgp_warning"] = f"{bgp_stats['bgp_sessions_not_established']} sessions not Established"
        healthy = False

    status_code = 200 if healthy else 503
    return Response(
        content=json.dumps({"status": "ok" if healthy else "degraded", "checks": checks}),
        status_code=status_code, media_type="application/json"
    )

Step 2: Configure Vigilmon Monitoring

HTTP Monitor — Peering Manager Web Application

In your Vigilmon dashboard, create an HTTP monitor for the Peering Manager sidecar:

| Field | Value | |-------|-------| | Monitor name | Peering Manager Web App | | URL | http://peering-manager.internal:3023/health/peering-manager | | Method | GET | | Check interval | Every 1 minute | | Expected status | 200 | | Alert threshold | 2 consecutive failures | | Regions | Select 2+ regions for consensus |

Also create a direct probe on the Peering Manager web interface:

| Field | Value | |-------|-------| | Monitor name | Peering Manager UI | | URL | http://peering.your-org.example.com/ | | Method | GET | | Check interval | Every 2 minutes | | Expected status | 200 |

Heartbeat Monitor — PeeringDB Synchronization Freshness

Monitor PeeringDB sync and alert if the sync hasn't run in 24 hours. Configure a 25-hour timeout heartbeat:

#!/bin/bash
# /etc/cron.d/peering-manager-peeringdb-sync — runs every 6 hours
HEARTBEAT_URL="https://vigilmon.online/hb/your-pm-peeringdb-sync-heartbeat"
PM_MANAGE="${PM_MANAGE:-/opt/peering-manager/venv/bin/python /opt/peering-manager/manage.py}"
MAX_STALE_HOURS=24

# Check when PeeringDB was last synchronized successfully
LAST_SYNC=$(${PM_MANAGE} shell -c "
from peering.models import AutonomousSystem
from django.utils import timezone
import datetime
last = AutonomousSystem.objects.order_by('-updated').first()
if last:
    age = (timezone.now() - last.updated).total_seconds() / 3600
    print(f'{age:.1f}')
else:
    print('999')
" 2>/dev/null || echo "999")

STALE=$(echo "$LAST_SYNC" | python3 -c "
import sys
age = float(sys.stdin.read().strip())
print('stale' if age > $MAX_STALE_HOURS else 'ok')
" 2>/dev/null || echo "stale")

if [ "$STALE" = "ok" ]; then
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
else
  echo "Peering Manager: PeeringDB sync is stale (last sync ${LAST_SYNC}h ago)" >&2
fi

Configure with a 25-hour timeout (runs every 6 hours with ample slack).

Heartbeat Monitor — Celery Worker BGP Polling

Verify Celery workers are executing BGP session polling tasks:

#!/bin/bash
# /etc/cron.d/peering-manager-celery — runs every 10 minutes
HEARTBEAT_URL="https://vigilmon.online/hb/your-pm-celery-heartbeat"
PM_DIR="${PM_DIR:-/opt/peering-manager}"

# Check active Celery workers
WORKER_COUNT=$(cd "$PM_DIR" && \
  ./venv/bin/celery -A peering_manager inspect ping \
  --timeout 5 2>/dev/null | grep -c "pong" || echo 0)

if [ "$WORKER_COUNT" -gt 0 ]; then
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
else
  echo "Peering Manager: No active Celery workers detected (count: ${WORKER_COUNT})" >&2
fi

Configure with a 15-minute timeout (runs every 10 minutes with 5-minute slack).

Heartbeat Monitor — Router Inventory Reachability

Verify all managed routers are reachable via NAPALM:

#!/bin/bash
# /etc/cron.d/peering-manager-routers — runs every 15 minutes
HEARTBEAT_URL="https://vigilmon.online/hb/your-pm-routers-heartbeat"
PM_MANAGE="${PM_MANAGE:-/opt/peering-manager/venv/bin/python /opt/peering-manager/manage.py}"

# Check for routers that failed their last NAPALM connection attempt
UNREACHABLE=$(${PM_MANAGE} shell -c "
from devices.models import Router
unreachable = Router.objects.filter(connection_status__icontains='error').count()
print(unreachable)
" 2>/dev/null || echo "0")

if [ "$UNREACHABLE" -eq 0 ]; then
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
else
  echo "Peering Manager: ${UNREACHABLE} router(s) unreachable via NAPALM" >&2
fi

Configure with a 20-minute timeout (runs every 15 minutes with 5-minute slack).

Heartbeat Monitor — Configuration Push Success

Track configuration push failures to routers:

#!/bin/bash
# /etc/cron.d/peering-manager-config-push — runs every 30 minutes
HEARTBEAT_URL="https://vigilmon.online/hb/your-pm-config-push-heartbeat"
PM_MANAGE="${PM_MANAGE:-/opt/peering-manager/venv/bin/python /opt/peering-manager/manage.py}"
MAX_PUSH_FAILURES=0

# Count failed configuration push tasks in last 30 minutes
FAILURES=$(${PM_MANAGE} shell -c "
from django.utils import timezone
from datetime import timedelta
# Check Celery task results for failed config push tasks
try:
    from django_celery_results.models import TaskResult
    cutoff = timezone.now() - timedelta(minutes=30)
    failed = TaskResult.objects.filter(
        task_name__contains='push_configuration',
        status='FAILURE',
        date_done__gte=cutoff
    ).count()
    print(failed)
except Exception:
    print(0)
" 2>/dev/null || echo "0")

if [ "$FAILURES" -le "$MAX_PUSH_FAILURES" ]; then
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
else
  echo "Peering Manager: ${FAILURES} configuration push failures in last 30 minutes" >&2
fi

Step 3: Configure Alerting

Alert Policies

Peering Manager App Down Alert

  • Trigger: Health endpoint returns 503 with webapp showing error or down
  • Severity: Critical — P1
  • Channels: PagerDuty (immediate) + Slack #network-ops-critical
  • Message: "Peering Manager web application is down. Operators cannot view BGP session status, create new peering sessions, or push router configuration changes. Check application logs and database connectivity."

PostgreSQL Database Down Alert

  • Trigger: Health endpoint returns 503 with postgresql: down
  • Severity: Critical — P1
  • Channels: PagerDuty (immediate) + Slack #network-ops-critical
  • Message: "Peering Manager database (PostgreSQL) is unreachable. The application cannot serve any requests and all BGP session management is offline. Restore database connectivity immediately."

Celery Workers Down Alert

  • Trigger: Celery worker heartbeat not received for 15 minutes
  • Severity: High
  • Channels: PagerDuty + Slack #network-ops-alerts
  • Message: "Peering Manager has no active Celery workers. BGP session polling, PeeringDB synchronization, and router configuration pushes are not executing. Dashboard BGP session states may be stale. Check Celery worker processes and Redis task queue."

PeeringDB Sync Stale Alert

  • Trigger: PeeringDB sync heartbeat not received for 25 hours
  • Severity: Medium
  • Channels: Slack #network-ops-alerts + email to peering team
  • Message: "Peering Manager's PeeringDB synchronization has not completed in >24 hours. AS contact information, IX memberships, and peering policy data may be outdated. New peering session configuration may reference stale data."

Router Unreachable Alert

  • Trigger: Router inventory heartbeat missed (unreachable router count >0)
  • Severity: High
  • Channels: Slack #network-ops-alerts + email to network team
  • Message: "Peering Manager cannot reach one or more managed routers via NAPALM. BGP session status for sessions on affected routers is stale. Configuration pushes to affected routers will fail. Verify router management IP and credential configuration."

BGP Session Not Established Alert

  • Trigger: Health endpoint shows bgp_sessions_not_established > 0
  • Severity: High
  • Channels: PagerDuty + Slack #peering-ops
  • Message: "Peering Manager reports BGP sessions in non-Established state. Verify whether this reflects actual BGP session state or NAPALM polling staleness. Check router connectivity and BGP session configuration."

Key Metrics Summary

| Metric | Alert Threshold | Impact | |--------|----------------|--------| | Peering Manager web app | HTTP 500 or down | All peering management operations blocked | | PostgreSQL connectivity | Any failure | Complete application failure | | PostgreSQL query latency | >1 second | UI slowdown; API timeout | | Redis connectivity | Any failure | Celery tasks stop; background jobs queue | | Celery worker count | 0 active workers | BGP polling, PeeringDB sync, config push stop | | PeeringDB sync freshness | >24 hours stale | Outdated AS/IX data for new session config | | NAPALM router reachability | Any unreachable | Stale BGP session state data | | BGP sessions not Established | >0 | Active BGP sessions failing | | Config push success rate | Any failure | Router BGP config not updated | | IXP route server sessions | Not Established | IXP traffic exchange failing |

Conclusion

Peering Manager is the single source of truth for your BGP peering topology — when its database is unavailable, Celery workers stop polling, or NAPALM cannot reach routers, the gap between displayed and actual BGP session state widens until operators are making network decisions based on hours-old data. Vigilmon HTTP probes on the Peering Manager health sidecar catch application and database failures immediately, while heartbeat monitors on Celery worker health, PeeringDB sync freshness, and router reachability give you coverage of the background job failures that create the dangerous displayed-vs-actual state divergence. Configure the monitors in this tutorial and your peering operations team will receive alerts about Peering Manager infrastructure problems before operators discover that the BGP session dashboard no longer reflects real network state.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles