Monitoring Balena (openBalena) with Vigilmon

Balena lets you deploy and update containerised applications to thousands of IoT edge devices with a single git push — but when the openBalena server goes down, every device in your fleet loses its update channel and OTA deployments silently halt. Vigilmon fills that gap, watching the openBalena API, VPN server, Docker registry, and fleet-level health metrics so you know about problems before your devices do.

What You'll Set Up

• openBalena API server uptime monitor
• VPN server health check (the backbone of device connectivity)
• Docker registry availability monitor
• Fleet device online percentage via heartbeat cron
• OTA update success rate alerting
• PostgreSQL database connectivity check

Prerequisites

• openBalena deployed (self-hosted) or balenaCloud account
• openBalena API accessible on port 443
• A free Vigilmon account

Step 1: Monitor the openBalena API Server

The openBalena API is the single point through which devices register, report telemetry, and receive OTA update commands. If it goes down, devices queue updates locally but can't receive new releases.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your openBalena API URL: https://api.youropenbalena.example.com/ping
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

The /ping endpoint is a built-in health probe in openBalena that returns OK when the API process is healthy. If this monitor fires, devices are immediately cut off from the fleet management server — treat this as a P1 alert.

Step 2: Monitor the openBalena VPN Server

openBalena uses an OpenVPN-based overlay to maintain persistent tunnels to every online device, enabling SSH access, log streaming, and remote debugging even behind NAT. If the VPN server crashes, all device tunnels drop simultaneously and operators lose the ability to SSH into any device.

1. Add a new monitor → TCP Port.
2. Set Host to vpn.youropenbalena.example.com.
3. Set Port to 443 (openBalena VPN uses TLS on port 443 by default).
4. Set Check interval to 1 minute.
5. Click Save.

Separately, add an HTTP monitor for the VPN management API if your openBalena version exposes one:

https://vpn.youropenbalena.example.com/ping

A VPN server outage causes mass device disconnection — all devices will show as "offline" in the openBalena dashboard even if the devices themselves are running fine.

Step 3: Monitor the Docker Registry

openBalena runs a private Docker registry that stores container images for all your applications. Devices pull images from this registry during OTA updates. Registry downtime doesn't immediately affect running containers, but it blocks all new deployments and OTA updates.

1. Add a new monitor → HTTP / HTTPS.
2. Enter: https://registry.youropenbalena.example.com/v2/
3. Set Expected HTTP status to 401 (the registry returns 401 for unauthenticated requests — this is the correct health signal, confirming the registry is alive and requiring authentication).
4. Set Check interval to 2 minutes.
5. Click Save.

Alternatively, if your registry is configured to allow anonymous health checks:

# Test locally
curl -s https://registry.youropenbalena.example.com/v2/ -o /dev/null -w "%{http_code}"
# Returns 200 or 401 when healthy, 000 or 5xx when down

Step 4: Fleet Device Online Percentage via Cron Heartbeat

The most important fleet-level metric is what percentage of your devices are currently online. A sudden drop below your baseline (say, from 95% to 70%) indicates a network event, power outage, or ISP issue affecting a device cluster.

Create a script that queries the openBalena API for fleet status and sends a heartbeat to Vigilmon only when the online percentage is above your threshold:

1. In Vigilmon, click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to 5 minutes.
3. Copy the heartbeat URL (e.g. https://vigilmon.online/heartbeat/abc123).

Deploy this script on the openBalena server or a management host:

#!/bin/bash
# fleet-health-check.sh

BALENA_API="https://api.youropenbalena.example.com"
BALENA_TOKEN="your-api-token"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/abc123"
THRESHOLD=90  # alert if online % drops below 90%

# Fetch device list
RESPONSE=$(curl -s -H "Authorization: Bearer $BALENA_TOKEN" \
  "$BALENA_API/v6/device?\$select=is_online,uuid&\$top=2000")

TOTAL=$(echo "$RESPONSE" | jq '.d | length')
ONLINE=$(echo "$RESPONSE" | jq '[.d[] | select(.is_online == true)] | length')

if [ "$TOTAL" -gt 0 ]; then
  PCT=$(( ONLINE * 100 / TOTAL ))
  echo "Fleet online: $ONLINE/$TOTAL ($PCT%)"
  if [ "$PCT" -ge "$THRESHOLD" ]; then
    curl -s "$HEARTBEAT_URL"
  fi
fi

Add to crontab:

*/5 * * * * /opt/balena/fleet-health-check.sh >> /var/log/balena-health.log 2>&1

If the online percentage drops below 90%, the heartbeat stops and Vigilmon alerts after the 5-minute window passes. Tune the threshold to match your fleet's normal variance.

Step 5: OTA Update Success Rate Monitoring

Stuck OTA updates are silent failures — devices stop at a partial state, new releases don't propagate, and nothing alerts you unless you're actively checking the dashboard. Monitor update success rate with a cron heartbeat that only pings when updates are proceeding normally:

#!/bin/bash
# ota-health-check.sh

BALENA_API="https://api.youropenbalena.example.com"
BALENA_TOKEN="your-api-token"
OTA_HEARTBEAT="https://vigilmon.online/heartbeat/def456"
MAX_STUCK=10  # alert if more than 10% of devices are stuck

# Count devices in "updating" state for more than 30 minutes
STUCK=$(curl -s -H "Authorization: Bearer $BALENA_TOKEN" \
  "$BALENA_API/v6/device?\$filter=status eq 'Updating'" | jq '.d | length')

TOTAL=$(curl -s -H "Authorization: Bearer $BALENA_TOKEN" \
  "$BALENA_API/v6/device?\$select=uuid" | jq '.d | length')

if [ "$TOTAL" -gt 0 ]; then
  STUCK_PCT=$(( STUCK * 100 / TOTAL ))
  if [ "$STUCK_PCT" -le "$MAX_STUCK" ]; then
    curl -s "$OTA_HEARTBEAT"
  fi
fi

Set the Vigilmon heartbeat interval to 10 minutes. If the stuck-device percentage spikes above your threshold, the heartbeat stops and Vigilmon alerts.

Step 6: PostgreSQL Database Connectivity

openBalena stores all device state, fleet configuration, and release metadata in PostgreSQL. Database connectivity loss causes device state to become stale and OTA update tracking to stop working. Monitor the database port from the openBalena server:

1. Add a new monitor → TCP Port.
2. Set Host to your PostgreSQL host (or localhost if running on the same server).
3. Set Port to 5432.
4. Set Check interval to 1 minute.
5. Click Save.

If your PostgreSQL port isn't directly accessible, add a lightweight health endpoint to your openBalena deployment that checks DB connectivity:

// health.js — add to openBalena's internal health check
app.get('/healthz/db', async (req, res) => {
  try {
    await db.query('SELECT 1');
    res.json({ db: 'ok' });
  } catch (err) {
    res.status(503).json({ db: 'error', message: err.message });
  }
});

Then monitor https://api.youropenbalena.example.com/healthz/db with an expected status of 200.

Step 7: Device Last-Seen Timestamp Alerts

For latency-sensitive IoT applications (industrial sensors, medical devices, security cameras), any device that hasn't been seen in the last 6 hours needs immediate attention. This script generates an alert for offline devices:

#!/bin/bash
# last-seen-check.sh

BALENA_API="https://api.youropenbalena.example.com"
BALENA_TOKEN="your-api-token"
LAST_SEEN_HEARTBEAT="https://vigilmon.online/heartbeat/ghi789"
MAX_OFFLINE_HOURS=6

# Find devices not seen in the last N hours
CUTOFF=$(date -u -d "$MAX_OFFLINE_HOURS hours ago" +%Y-%m-%dT%H:%M:%SZ)

OVERDUE=$(curl -s -H "Authorization: Bearer $BALENA_TOKEN" \
  "$BALENA_API/v6/device?\$filter=last_connectivity_event lt '$CUTOFF' and is_online eq false" \
  | jq '.d | length')

if [ "$OVERDUE" -eq 0 ]; then
  curl -s "$LAST_SEEN_HEARTBEAT"
fi

Run every 30 minutes. If any device goes quiet for 6 hours, the heartbeat stops and Vigilmon alerts.

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or webhook.
2. Set Consecutive failures before alert to 2 for the API and VPN monitors — brief restarts are normal during openBalena upgrades.
3. Set Consecutive failures before alert to 1 for the PostgreSQL monitor — database downtime should alert immediately.
4. Use Maintenance windows in Vigilmon when deploying openBalena server updates:

# Before updating openBalena server
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "api-monitor-id", "duration_minutes": 15}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | API server | https://api.openbalena/ping | Device connectivity loss, OTA update halt | | VPN server | TCP :443 on VPN host | Mass device disconnection | | Docker registry | https://registry.openbalena/v2/ | Deployment blockage | | Fleet online % | Cron heartbeat every 5 min | Network outage, mass device failure | | OTA update rate | Cron heartbeat every 10 min | Stuck updates, image incompatibility | | PostgreSQL | TCP :5432 | Device state staleness | | Device last-seen | Cron heartbeat every 30 min | Individual device offline >6 hours |

openBalena gives you full control over your IoT fleet — but that control means you own the monitoring too. With Vigilmon watching the server stack and fleet health metrics, you'll know about connectivity issues and stuck OTA updates before they become fleet-wide outages.