Monitoring NOC Project with Vigilmon

NOC Project is a comprehensive open source Network Management System (NMS) and Operations Support System (OSS) designed for network operations centers. It provides fault management, performance management, configuration management, IP address management, and service activation for networks of any scale. When you self-host NOC Project, you're running a Python microservices platform that spans an activator polling engine, an event classifier, a correlator for root cause analysis, network topology discovery, and a data pipeline built on MongoDB, ClickHouse, Redis, and Apache Kafka. Any of these services failing silently can create polling gaps, delay alarm escalation, or lose telemetry data — all invisible unless you're actively monitoring the platform itself. Vigilmon gives you continuous health coverage across every NOC Project service tier.

What You'll Set Up

• NOC activator process health monitor
• Device availability rate monitoring
• Event processing rate and classifier lag heartbeat
• Kafka consumer group lag per NOC service
• MongoDB primary reachability and replication lag monitor
• ClickHouse insert health monitor
• Redis health and memory monitor
• Discovery job success rate heartbeat
• Alarm escalation pipeline health check
• NOC web interface HTTP monitor

Prerequisites

• NOC Project deployed (supervisor or systemd managed microservices)
• MongoDB, ClickHouse, Redis, and Kafka accessible on their standard ports
• NOC web interface accessible via HTTP/HTTPS
• A free Vigilmon account

Step 1: Monitor the NOC Activator Health

The activator is the polling engine that queries managed network devices via SNMP, NETCONF, SSH, and CLI. An activator crash creates silent polling gaps — devices appear available in NOC's database but are no longer being polled.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to Heartbeat / Cron.
3. Name: NOC Activator Health.
4. Heartbeat interval: 5 minutes (alert if no heartbeat for 10 minutes).
5. Copy the heartbeat URL.

Activator health check script:

#!/bin/bash
# /usr/local/bin/check-noc-activator.sh
HEARTBEAT_URL="https://hb.vigilmon.online/your-activator-heartbeat-id"
NOC_DIR="/opt/noc"

# Check that at least one activator process is running
activator_count=$(pgrep -f "noc.services.activator" | wc -l)

if [ "$activator_count" -eq 0 ]; then
  echo "No NOC activator processes running"
  exit 1
fi

# Optionally check via NOC's supervisor/systemd status
# supervisorctl status noc-activator | grep -q RUNNING

curl -s "$HEARTBEAT_URL" > /dev/null

Add to cron:

*/5 * * * * /usr/local/bin/check-noc-activator.sh

For multi-activator deployments, create one heartbeat monitor per activator instance and adjust the script to target the specific process or systemd service name.

Step 2: Monitor Device Availability Rate

NOC's discovery and activator services track which managed devices are reachable. A spike in unreachable devices could indicate a network partition, a mass equipment failure, or (critically) a NOC polling engine failure that's reporting stale data.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: NOC Device Availability Rate.
3. Heartbeat interval: 10 minutes.

Availability rate script:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-availability-heartbeat-id"
MAX_UNREACHABLE_PCT=10   # alert if >10% of devices unreachable
NOC_DB_HOST="localhost"
NOC_DB_PORT="5432"
NOC_PG_DB="noc"

# Query NOC PostgreSQL for managed object availability
total=$(psql -h "$NOC_DB_HOST" -p "$NOC_DB_PORT" -U noc -d "$NOC_PG_DB" -t -c \
  "SELECT COUNT(*) FROM sa_managedobject WHERE is_managed=true;" | tr -d ' ')

unreachable=$(psql -h "$NOC_DB_HOST" -p "$NOC_DB_PORT" -U noc -d "$NOC_PG_DB" -t -c \
  "SELECT COUNT(*) FROM sa_managedobject WHERE is_managed=true AND availability_state='u';" | tr -d ' ')

if [ "$total" -gt 0 ]; then
  pct=$(echo "scale=0; $unreachable * 100 / $total" | bc)
  if [ "$pct" -gt "$MAX_UNREACHABLE_PCT" ]; then
    echo "Device availability: ${pct}% unreachable (${unreachable}/${total} devices)"
    exit 1
  fi
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 3: Monitor Event Processing Rate and Classifier Lag

NOC receives SNMP traps and syslog events from managed devices and passes them through the classifier service, which maps raw events to managed object event classes. If the classifier falls behind its input queue, alarms are delayed.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: NOC Classifier Processing Lag.
3. Heartbeat interval: 5 minutes.

Classifier lag check via Kafka consumer offset:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-classifier-heartbeat-id"
KAFKA_BOOTSTRAP="localhost:9092"
CONSUMER_GROUP="noc-classifier"
MAX_LAG=1000   # alert if classifier is >1000 events behind

lag=$(kafka-consumer-groups.sh --bootstrap-server "$KAFKA_BOOTSTRAP" \
  --describe --group "$CONSUMER_GROUP" 2>/dev/null \
  | awk 'NR>1 {sum+=$5} END {print sum}')

if [ -z "$lag" ]; then
  echo "Could not read Kafka consumer lag for group $CONSUMER_GROUP"
  exit 1
fi

if [ "$lag" -gt "$MAX_LAG" ]; then
  echo "NOC classifier Kafka lag: $lag messages (threshold: $MAX_LAG)"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 4: Monitor Kafka Consumer Lag per NOC Service

NOC's microservices communicate through Kafka topics. Each service (activator, classifier, correlator, discovery) has its own consumer group. Growing lag on any consumer group indicates that service is falling behind.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: NOC Kafka Consumer Lag.
3. Heartbeat interval: 5 minutes.

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-kafka-lag-heartbeat-id"
KAFKA_BOOTSTRAP="localhost:9092"
MAX_LAG=5000

# Check all NOC consumer groups
for group in noc-classifier noc-correlator noc-discovery noc-activator; do
  lag=$(kafka-consumer-groups.sh --bootstrap-server "$KAFKA_BOOTSTRAP" \
    --describe --group "$group" 2>/dev/null \
    | awk 'NR>1 && $5~/^[0-9]+$/ {sum+=$5} END {print sum+0}')

  if [ "$lag" -gt "$MAX_LAG" ]; then
    echo "NOC service $group Kafka lag: $lag (threshold: $MAX_LAG)"
    exit 1
  fi
done

curl -s "$HEARTBEAT_URL" > /dev/null

Step 5: Monitor MongoDB Health

MongoDB is NOC's primary datastore for configuration data, managed objects, and event history. A MongoDB primary failure or high replication lag affects all NOC services that read or write object state.

Primary reachability — direct HTTP monitor:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-mongodb-host:28017/ (MongoDB HTTP status interface, if enabled) or use a heartbeat with mongosh.
3. Check interval: 1 minute.

Replication lag heartbeat:

1. Click Add Monitor → Heartbeat / Cron.
2. Name: NOC MongoDB Health.
3. Heartbeat interval: 2 minutes.

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-mongo-heartbeat-id"
MAX_REPL_LAG_SECS=30

# Check primary is reachable
mongo_status=$(mongosh --quiet --eval \
  "JSON.stringify(rs.status())" 2>/dev/null)

if [ -z "$mongo_status" ]; then
  echo "MongoDB not responding"
  exit 1
fi

# Check replication lag on secondaries
max_lag=$(echo "$mongo_status" | python3 -c "
import sys, json
d = json.load(sys.stdin)
members = d.get('members', [])
lags = [m.get('optimeDate', 0) for m in members if m.get('stateStr') == 'SECONDARY']
print(0)  # simplified; implement lag calculation from optimeDate diff
")

curl -s "$HEARTBEAT_URL" > /dev/null

Step 6: Monitor ClickHouse Health

NOC uses ClickHouse to store high-throughput telemetry: performance metrics and flow data from polled devices. ClickHouse insert failures cause metric data to be lost permanently — there is no replay from Kafka once the retention window expires.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-clickhouse-host:8123/ping
3. Check interval: 1 minute.
4. Expected HTTP status: 200.
5. Keyword check: Ok.
6. Click Save.

For insert rate monitoring, add a heartbeat:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-clickhouse-heartbeat-id"

# Check that recent inserts are succeeding
insert_errors=$(curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.errors+WHERE+event_time+>+now()-60" 2>/dev/null)

if [ "$insert_errors" -gt 0 ]; then
  echo "ClickHouse reported $insert_errors errors in last 60 seconds"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 7: Monitor Redis Health

NOC uses Redis for inter-service messaging and caching. A Redis failure disrupts communication between NOC microservices, causing services to operate on stale cache data or fail to pass messages.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: NOC Redis Health.
3. Heartbeat interval: 2 minutes.

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-redis-heartbeat-id"
MAX_MEMORY_PCT=85

# Check Redis is responding
pong=$(redis-cli ping 2>/dev/null)
if [ "$pong" != "PONG" ]; then
  echo "Redis not responding to PING"
  exit 1
fi

# Check memory usage
used=$(redis-cli info memory | grep "used_memory:" | cut -d: -f2 | tr -d '\r')
max=$(redis-cli info memory | grep "maxmemory:" | cut -d: -f2 | tr -d '\r')

if [ "$max" -gt 0 ]; then
  pct=$(echo "scale=0; $used * 100 / $max" | bc)
  if [ "$pct" -gt "$MAX_MEMORY_PCT" ]; then
    echo "Redis memory at ${pct}% (used: ${used}, max: ${max})"
    exit 1
  fi
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 8: Monitor Discovery Job Health

NOC's discovery service automatically discovers network topology (neighbors, interfaces, VLANs) by querying devices. Discovery errors result in stale or incorrect topology data, affecting IP address management and configuration management workflows.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: NOC Discovery Job Health.
3. Heartbeat interval: 30 minutes.

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-discovery-heartbeat-id"
MAX_ERROR_RATE=10   # alert if >10% of discovery jobs erroring

# Query NOC discovery job logs in MongoDB
error_count=$(mongosh --quiet noc --eval \
  'db.noc.discovery.log.countDocuments({status:"F", ts:{$gt:new Date(Date.now()-3600000)}})' 2>/dev/null)

total_count=$(mongosh --quiet noc --eval \
  'db.noc.discovery.log.countDocuments({ts:{$gt:new Date(Date.now()-3600000)}})' 2>/dev/null)

if [ "$total_count" -gt 0 ]; then
  pct=$(echo "scale=0; ${error_count:-0} * 100 / $total_count" | bc)
  if [ "$pct" -gt "$MAX_ERROR_RATE" ]; then
    echo "Discovery error rate: ${pct}% (${error_count}/${total_count} jobs in last hour)"
    exit 1
  fi
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 9: Monitor Alarm Escalation Health

NOC's correlator service performs root cause analysis and raises alarms when events exceed thresholds. A correlator failure means incoming events are not being processed into alarms, leaving operators blind to ongoing network issues.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: NOC Alarm Escalation Health.
3. Heartbeat interval: 5 minutes.

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-alarm-heartbeat-id"

# Check correlator process is running
if ! pgrep -f "noc.services.correlator" > /dev/null; then
  echo "NOC correlator process not running — alarm escalation stopped"
  exit 1
fi

# Check Kafka consumer lag for correlator
lag=$(kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group noc-correlator 2>/dev/null \
  | awk 'NR>1 && $5~/^[0-9]+$/ {sum+=$5} END {print sum+0}')

if [ "$lag" -gt 10000 ]; then
  echo "NOC correlator Kafka lag ${lag} — alarm escalation is behind"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 10: Monitor the NOC Web Interface

The NOC web interface is the operational console for NOC operators. An unavailable web UI means operators cannot acknowledge alarms, view device status, or run manual checks — even if the backend services are healthy.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://noc.yourdomain.com/ (or your NOC web interface URL).
3. Check interval: 1 minute.
4. Expected HTTP status: 200.
5. Click Save.

Configuring Alerts

Configure alert channels in Vigilmon to route NOC Project failures to the right responders:

1. Go to Alert Channels and add email, Slack, PagerDuty, or webhook destinations.
2. For each monitor, open Settings → Alerting and assign a channel.

Recommended thresholds:

| Monitor | Alert Condition | Severity | |---|---|---| | Activator health | Process not running | Critical | | MongoDB health | Primary unreachable | Critical | | ClickHouse insert | HTTP not 200 | Critical | | Redis health | PING failure | Critical | | Correlator health | Process not running | Critical | | Device availability | > 10% unreachable | High | | Classifier lag | > 1,000 messages | Warning | | Kafka consumer lag | > 5,000 messages | Warning | | Discovery error rate | > 10% | Warning | | Web interface | HTTP not 200 | Low |

Conclusion

NOC Project is a powerful but operationally complex platform — its microservices architecture means a single failing component (a crashed correlator, a full Redis memory, a ClickHouse insert error) can silently degrade alarm management or telemetry collection without any obvious UI error. With Vigilmon monitoring every service tier, you'll catch issues in the polling engine, event pipeline, datastore, and web interface before they affect your network operations team's ability to respond to real events.

Get started at vigilmon.online.