Monitoring DRBD with Vigilmon

DRBD (Distributed Replicated Block Device) is the technology that makes two Linux servers share a disk over the network — no SAN required. It's the backbone of HA clusters built with Pacemaker/Corosync, powering databases, file servers, and any workload that needs a fast failover. The problem: DRBD replication failures are invisible until you actually try to fail over and discover the secondary is Outdated or split-brained. Vigilmon gives you continuous visibility into connection state, disk sync progress, out-of-sync sectors, and replication throughput before a failure turns into an incident.

What You'll Set Up

• DRBD connection state monitor (Connected/Disconnected per resource)
• Disk state monitor per node (UpToDate/Outdated/Failed)
• Out-of-sync sector alert for steady-state data drift
• Resync progress and speed monitor after reconnection
• Split-brain detection via kernel log scraping
• Replication network throughput monitor
• Cron heartbeat confirming the DRBD status script runs

Prerequisites

• DRBD 9.x installed and at least one resource configured in /etc/drbd.d/
• Both nodes accessible (SSH or direct)
• drbdsetup, drbdadm, and /proc/drbd available
• A free Vigilmon account

Step 1: Expose DRBD Metrics via a Status Script

DRBD exposes its state through /proc/drbd and the drbdsetup status command. Create a lightweight HTTP endpoint that Vigilmon can scrape on each node.

Install a minimal HTTP wrapper

# On each DRBD node
cat > /usr/local/bin/drbd-status-http.sh << 'EOF'
#!/bin/bash
# Parse /proc/drbd and return JSON status
RESOURCE=${1:-r0}
STATE=$(drbdsetup status "$RESOURCE" --statistics 2>/dev/null)
CONN=$(echo "$STATE" | grep -oP 'peer-node-id:\d+ conn-name:\S+ connection:\K\S+' | head -1)
DISK=$(echo "$STATE" | grep -oP 'disk:\K\S+' | head -1)
OOS=$(cat /sys/kernel/debug/drbd/$RESOURCE/*/out_of_sync 2>/dev/null | awk '{s+=$1} END {print s+0}')
echo "Content-Type: application/json"
echo ""
echo "{\"connection\":\"${CONN:-unknown}\",\"disk\":\"${DISK:-unknown}\",\"out_of_sync_sectors\":${OOS:-0}}"
EOF
chmod +x /usr/local/bin/drbd-status-http.sh

Serve it with ncat or a small Python server for Vigilmon to probe:

# Run as a systemd service or with socat
socat TCP-LISTEN:9955,fork,reuseaddr EXEC:"/usr/local/bin/drbd-status-http.sh r0"

Create /etc/systemd/system/drbd-exporter.service:

[Unit]
Description=DRBD status HTTP exporter
After=network.target

[Service]
ExecStart=/usr/bin/socat TCP-LISTEN:9955,fork,reuseaddr EXEC:"/usr/local/bin/drbd-status-http.sh r0"
Restart=always

[Install]
WantedBy=multi-user.target

systemctl enable --now drbd-exporter

Step 2: Monitor DRBD Connection State

The most critical DRBD metric is whether the two nodes are talking to each other. Any state other than Connected means replication has stopped.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the exporter URL: http://node1.internal:9955/.
4. Under Keyword check, add: "connection":"Connected".
5. Set Check interval to 1 minute.
6. Set Alert after 2 consecutive failures (to avoid transient network blips).
7. Click Save.

Repeat for the secondary node. If either node loses the connection, Vigilmon alerts before replication falls further behind.

Step 3: Monitor Disk State Per Node

A node's disk state tells you whether its copy of the data is current. UpToDate is the only healthy state; Outdated, Consistent, or Failed all require immediate investigation.

1. Add a second HTTP / HTTPS monitor pointing to the same exporter endpoint.
2. Under Keyword check, set the expected string to: "disk":"UpToDate".
3. Set the description to DRBD disk state — node1.
4. Save.

You can also check from the command line at any time:

drbdadm status r0
# r0 role:Primary
#   disk:UpToDate
#   peer connection:Connected role:Secondary
#     replication:Established peer-disk:UpToDate

Both disk:UpToDate and peer-disk:UpToDate should appear. If either is missing, alert immediately.

Step 4: Alert on Out-of-Sync Sectors

Out-of-sync sectors in steady state (when the nodes are connected) indicate a replication problem. The count should be zero when connection:Connected.

Create a monitoring script that fails if out-of-sync sectors exceed zero:

cat > /usr/local/bin/check-drbd-oos.sh << 'EOF'
#!/bin/bash
RESOURCE=${1:-r0}
OOS=$(drbdsetup status "$RESOURCE" --statistics 2>/dev/null \
  | grep -oP 'out-of-sync:\K\d+' | head -1)
OOS=${OOS:-0}
if [ "$OOS" -gt 0 ]; then
  echo "CRITICAL: $OOS out-of-sync sectors on $RESOURCE"
  exit 1
fi
echo "OK: 0 out-of-sync sectors"
exit 0
EOF
chmod +x /usr/local/bin/check-drbd-oos.sh

Wrap it in the HTTP exporter response, or add a dedicated TCP check endpoint. In Vigilmon, add a Keyword check that looks for OK: 0 out-of-sync.

Step 5: Monitor Resync Progress and Speed

After a node reconnects (following a crash or maintenance), DRBD resyncs the out-of-sync bitmap. Slow resync (below 10 MB/s) means a long window of vulnerability.

Check resync state:

cat /proc/drbd
# version: 9.1.x ...
# 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----
#    ...
#    sync'ed: 42.3% (23456/40960)M
#    finish: 0:12:34 speed: 32,456 (25,100) K/sec

Extract the speed and alert when below threshold:

cat > /usr/local/bin/check-drbd-resync.sh << 'EOF'
#!/bin/bash
SPEED=$(grep -oP 'speed: \K[\d,]+' /proc/drbd | tr -d ',' | head -1)
if [ -z "$SPEED" ]; then
  echo "OK: no resync in progress"
  exit 0
fi
# Convert K/sec to MB/s
SPEED_MB=$(( SPEED / 1024 ))
if [ "$SPEED_MB" -lt 10 ]; then
  echo "WARNING: resync speed ${SPEED_MB} MB/s below 10 MB/s threshold"
  exit 1
fi
echo "OK: resync speed ${SPEED_MB} MB/s"
exit 0
EOF
chmod +x /usr/local/bin/check-drbd-resync.sh

Add a Vigilmon Keyword check monitor that runs via the HTTP exporter and looks for OK:.

Step 6: Detect Split-Brain Events

Split-brain — both nodes simultaneously becoming Primary — is the most dangerous DRBD failure mode. DRBD logs it to the kernel ring buffer immediately.

Set up a watcher that flags split-brain events:

cat > /usr/local/bin/check-drbd-splitbrain.sh << 'EOF'
#!/bin/bash
# Check dmesg for split-brain events in the last 5 minutes
SPLITBRAIN=$(dmesg --since "5 minutes ago" 2>/dev/null | grep -c "Split-Brain")
if [ "$SPLITBRAIN" -gt 0 ]; then
  echo "CRITICAL: $SPLITBRAIN split-brain event(s) detected"
  exit 1
fi
echo "OK: no split-brain events"
exit 0
EOF
chmod +x /usr/local/bin/check-drbd-splitbrain.sh

Run this via a cron every minute and have it ping a Vigilmon heartbeat — or expose it via the HTTP exporter with a Keyword check for OK:. A split-brain event means both nodes may have divergent writes; always alert immediately regardless of other thresholds.

Step 7: Monitor Replication Network Throughput

DRBD uses a dedicated replication link. A throughput drop indicates a network issue before the connection fully drops.

# Check DRBD replication traffic using drbdsetup statistics
drbdsetup status r0 --statistics | grep -E 'send|receive'
# send: 1234567890
# receive: 987654321

For ongoing monitoring, track bytes/second using a script that takes a delta between two readings:

cat > /usr/local/bin/check-drbd-throughput.sh << 'EOF'
#!/bin/bash
RESOURCE=${1:-r0}
STATFILE=/tmp/drbd-bytes-${RESOURCE}
NOW_BYTES=$(drbdsetup status "$RESOURCE" --statistics 2>/dev/null \
  | grep -oP 'send:\K\d+' | head -1)
NOW_BYTES=${NOW_BYTES:-0}
NOW_TIME=$(date +%s)

if [ -f "$STATFILE" ]; then
  read PREV_BYTES PREV_TIME < "$STATFILE"
  DELTA_BYTES=$(( NOW_BYTES - PREV_BYTES ))
  DELTA_TIME=$(( NOW_TIME - PREV_TIME ))
  if [ "$DELTA_TIME" -gt 0 ]; then
    BPS=$(( DELTA_BYTES / DELTA_TIME ))
    echo "OK: replication throughput ${BPS} B/s"
  fi
fi

echo "$NOW_BYTES $NOW_TIME" > "$STATFILE"
echo "OK: baseline recorded"
exit 0
EOF
chmod +x /usr/local/bin/check-drbd-throughput.sh

Step 8: Heartbeat for the DRBD Monitoring Cron

Use a Vigilmon cron heartbeat to confirm your DRBD status collection scripts are running:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set expected ping interval to 2 minutes.
3. Copy the heartbeat URL.
4. Add to your monitoring cron:

# /etc/cron.d/drbd-monitor
* * * * * root /usr/local/bin/check-drbd-oos.sh r0 \
  && /usr/local/bin/check-drbd-splitbrain.sh \
  && curl -s https://vigilmon.online/heartbeat/YOUR_TOKEN

If the cron script itself crashes (missing drbdsetup binary, wrong kernel version, permission error), Vigilmon alerts within 2 minutes.

Step 9: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your preferred channel (email, Slack, PagerDuty webhook).
2. For split-brain and out-of-sync alerts, set Consecutive failures before alert to 1 — these are always urgent.
3. For connection state and disk state, use 2 consecutive failures to filter transient network blips during failover transitions.
4. Configure a maintenance window during planned failover tests so alert fatigue doesn't dull your response to real incidents.

Summary

| Monitor | Target | Alert Condition | |---|---|---| | Connection state | HTTP exporter keyword "connection":"Connected" | Any non-Connected state | | Disk state | HTTP exporter keyword "disk":"UpToDate" | Any non-UpToDate state | | Out-of-sync sectors | Script keyword OK: 0 | OOS sectors > 0 in steady state | | Resync speed | Script keyword OK: | Speed < 10 MB/s during resync | | Split-brain | Script keyword OK: no split-brain | Any split-brain event | | Replication throughput | HTTP exporter throughput check | Throughput drop > 50% from baseline | | Monitoring cron | Heartbeat every 2 min | Script fails to run |

DRBD's strength is silent, continuous replication — its weakness is that failures are equally silent. With Vigilmon watching every DRBD resource for connection drops, disk state degradation, out-of-sync sectors, and split-brain events, you catch replication problems in minutes, not during the moment of failover when it's too late.