Monitoring Open Horizon with Vigilmon

Open Horizon is an IBM-originated LF Edge platform that manages the deployment, execution, and lifecycle of containerized applications (edge services) across distributed edge nodes — from IoT gateways and single-board computers to cloud edge servers — all without requiring persistent cloud connectivity for ongoing operation. When the Management Hub is degraded, Agbot agreement processing stalls, edge agents stop heartbeating, or service deployments fail, your edge fleet drifts silently out of policy compliance. Vigilmon monitors the full Open Horizon stack — Management Hub APIs, Agbot health, per-node agent liveness, service deployment success, and Model Management System (MMS) availability — giving you complete fleet visibility from a single dashboard.

What You'll Set Up

• Management Hub health monitor (Exchange API + Agbot)
• Exchange REST API availability monitor
• Agbot agreement processing heartbeat
• Edge agent heartbeat health monitor
• Edge service deployment success rate heartbeat
• Edge node registration health heartbeat
• Service policy compliance heartbeat
• Secrets Manager health monitor
• MMS (Model Management System) health heartbeat

Prerequisites

• Open Horizon Management Hub deployed (Docker Compose or Kubernetes)
• Exchange API accessible (typically port 8080 or via HTTPS)
• Agbot service running and connected to the Exchange
• hzn CLI available on a host with access to the Management Hub
• A free Vigilmon account

Step 1: Monitor Management Hub Health

The Open Horizon Management Hub comprises the Exchange registry and the Agbot. When either is down, policy deployment to edge nodes stops and existing nodes lose their agreement negotiation path.

Monitor the Exchange API with an HTTP health check:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: http://your-exchange-host:8080/v1/admin/version.
4. Check interval: 1 minute.
5. Expected HTTP status: 200.
6. Under Keyword match, add "version" to verify a real Exchange response.
7. Click Save.

For HTTPS deployments with a certificate:

1. Enable Monitor SSL certificate → set expiry alert to 21 days.
2. Click Save.

Step 2: Monitor Exchange API Detailed Health

Edge agents and operators interact with the Exchange API to register services, update policies, and query deployment agreements. Exchange API failures block all service registration and policy updates across the fleet.

#!/bin/bash
# /opt/monitoring/check-horizon-exchange.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXCHANGE_URL="http://your-exchange-host:8080"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"

# Query the orgs endpoint as a real API health probe
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs?limit=1" \
  --max-time 10)

if [ "$HTTP_STATUS" = "200" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Name: Open Horizon Exchange API.
3. Expected interval: 2 minutes.
4. Grace period: 5 minutes.
5. Copy the heartbeat URL into the script.
6. Schedule: */2 * * * * /opt/monitoring/check-horizon-exchange.sh.
7. Click Save.

Step 3: Monitor Agbot Agreement Processing

The Agreement Bot (Agbot) matches service deployment policies with edge nodes and negotiates deployment agreements. Agbot failures cause new matching nodes to never receive their edge services, even when policies and nodes are correctly configured.

#!/bin/bash
# /opt/monitoring/check-horizon-agbot.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXCHANGE_URL="http://your-exchange-host:8080"
HZN_ORG_ID="your-org-id"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"

# Count active agreements via the Exchange API
RESPONSE=$(curl -s \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/agreements?limit=1" \
  --max-time 15)

HTTP_STATUS=$(echo "$RESPONSE" | grep -o '"code":[0-9]*' | head -1 | cut -d: -f2)

# Alternative: check Agbot health endpoint directly
AGBOT_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/agbots" \
  --max-time 10)

if [ "$AGBOT_STATUS" = "200" ]; then
  AGBOT_COUNT=$(curl -s \
    -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
    "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/agbots" \
    --max-time 10 | grep -o '"agbots":{[^}]*}' | grep -c '"lastHeartbeat"')

  if [ "$AGBOT_COUNT" -gt 0 ]; then
    curl -s -X POST "$HEARTBEAT_URL" > /dev/null
  fi
fi

Set the expected interval to 2 minutes with a 5-minute grace period.

Step 4: Monitor Edge Agent Heartbeat Health

Open Horizon agents on edge nodes heartbeat to the Management Hub at regular intervals (typically every 30 seconds). A node that stops heartbeating for more than 10 minutes indicates the node is offline, has lost network connectivity, or the Open Horizon agent service has crashed.

#!/bin/bash
# /opt/monitoring/check-horizon-agents.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXCHANGE_URL="http://your-exchange-host:8080"
HZN_ORG_ID="your-org-id"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"
MAX_STALE_MINUTES=10

# Fetch all nodes and check last heartbeat time
NODES=$(curl -s \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/nodes" \
  --max-time 30)

# Count nodes with stale heartbeats
NOW=$(date +%s)
STALE_COUNT=0

while IFS= read -r LAST_HB; do
  if [ -n "$LAST_HB" ]; then
    # Parse ISO8601 timestamp
    HB_TS=$(date -d "$LAST_HB" +%s 2>/dev/null || echo 0)
    AGE_MINUTES=$(( (NOW - HB_TS) / 60 ))
    if [ "$AGE_MINUTES" -gt "$MAX_STALE_MINUTES" ]; then
      STALE_COUNT=$((STALE_COUNT + 1))
    fi
  fi
done < <(echo "$NODES" | grep -o '"lastHeartbeat":"[^"]*"' | cut -d'"' -f4)

if [ "$STALE_COUNT" = "0" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 15-minute grace period. Stale agent heartbeats are a lagging indicator — give nodes time to reconnect before alerting.

Alternatively, monitor individual critical nodes with per-node Vigilmon heartbeats pushed from the hzn agent on each node:

# On each edge node — add to crontab
*/5 * * * * hzn node list 2>/dev/null | grep '"configstate":"configured"' \
  && curl -s -X POST "https://vigilmon.online/api/v1/heartbeat/YOUR_NODE_SPECIFIC_ID"

Step 5: Monitor Edge Service Deployment Success

The Open Horizon agent on each edge node pulls and runs containerized edge services based on deployment policies. Deployment failures — caused by Docker pull errors, image digest mismatches, or policy conflicts — leave nodes running outdated or no services.

#!/bin/bash
# /opt/monitoring/check-horizon-deployments.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXCHANGE_URL="http://your-exchange-host:8080"
HZN_ORG_ID="your-org-id"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"

# Count agreements in "terminated" state (deployment failed)
TERMINATED=$(curl -s \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/agreements" \
  --max-time 30 | grep -c '"terminated"')

if [ "$TERMINATED" = "0" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

For per-node deployment checking from the edge node itself:

#!/bin/bash
# Run on the edge node
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Check all expected services are running via hzn
FAILED_SERVICES=$(hzn agreement list 2>/dev/null \
  | grep -c '"terminated_description"')

if [ "$FAILED_SERVICES" = "0" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 10-minute grace period.

Step 6: Monitor Edge Node Registration Health

Edge nodes must register with the Management Hub on initial setup. Failed registrations prevent nodes from ever receiving edge services. Track the active registered node count to catch registration failures and unexpected deregistrations.

#!/bin/bash
# /opt/monitoring/check-horizon-nodes.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXCHANGE_URL="http://your-exchange-host:8080"
HZN_ORG_ID="your-org-id"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"
EXPECTED_MIN_NODES=3  # set to minimum expected registered node count

NODE_COUNT=$(curl -s \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/nodes" \
  --max-time 15 | grep -o '"id"' | wc -l)

if [ "$NODE_COUNT" -ge "$EXPECTED_MIN_NODES" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 15-minute grace period.

Step 7: Monitor Service Policy Compliance

Each edge node should be running the services specified by its deployment policy. Nodes with policy violations — where the running services don't match the expected deployment policy — indicate agent failures or agreement negotiation problems.

#!/bin/bash
# /opt/monitoring/check-horizon-compliance.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXCHANGE_URL="http://your-exchange-host:8080"
HZN_ORG_ID="your-org-id"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"

# Count nodes with active agreements (policy-compliant) vs total nodes
TOTAL_NODES=$(curl -s \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/nodes" \
  --max-time 15 | grep -o '"id"' | wc -l)

NODES_WITH_AGREEMENTS=$(curl -s \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/agreements" \
  --max-time 15 | grep -o '"deviceid"[^,]*' | sort -u | wc -l)

# Alert if more than 20% of nodes have no agreements
if [ "$TOTAL_NODES" -gt 0 ]; then
  COMPLIANCE_PCT=$(( (NODES_WITH_AGREEMENTS * 100) / TOTAL_NODES ))
  if [ "$COMPLIANCE_PCT" -ge 80 ]; then
    curl -s -X POST "$HEARTBEAT_URL" > /dev/null
  fi
fi

Set the expected interval to 5 minutes with a 15-minute grace period.

Step 8: Monitor Secrets Manager Health

The Open Horizon Secrets Manager stores credentials used by edge services at runtime. When the Secrets Manager is unavailable, edge services that retrieve credentials at startup fail to launch — causing deployment failures without any obvious error in the agreement log.

#!/bin/bash
# /opt/monitoring/check-horizon-secrets.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXCHANGE_URL="http://your-exchange-host:8080"
HZN_ORG_ID="your-org-id"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"

# Check secrets API endpoint via Exchange
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${EXCHANGE_URL}/v1/orgs/${HZN_ORG_ID}/secrets" \
  --max-time 10)

if [ "$HTTP_STATUS" = "200" ] || [ "$HTTP_STATUS" = "404" ]; then
  # 404 = no secrets configured yet, but API is healthy
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 2 minutes with a 5-minute grace period.

Step 9: Monitor Model Management System (MMS) Health

The Open Horizon MMS distributes ML models and configuration objects to edge nodes. When MMS fails, edge AI services stop receiving model updates — nodes continue running with stale models until MMS recovers and re-syncs.

#!/bin/bash
# /opt/monitoring/check-horizon-mms.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
CSS_URL="http://your-css-host:9443"  # Cloud Sync Service (MMS backend)
HZN_ORG_ID="your-org-id"
EXCHANGE_USER="root/root"
EXCHANGE_PASS="your-exchange-root-password"

# Check CSS (Cloud Sync Service) health — the MMS backend
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  -u "${EXCHANGE_USER}:${EXCHANGE_PASS}" \
  "${CSS_URL}/api/v1/health" \
  --max-time 10)

if [ "$HTTP_STATUS" = "200" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 10-minute grace period.

Step 10: Configure Alerting

Apply alert channels to all Open Horizon monitors:

1. Go to Alerts → Add Alert Channel → choose Email, Slack, PagerDuty, or Webhook.
2. Apply the channel to all Open Horizon monitors.

Recommended thresholds:

| Monitor | Alert After | Severity | |---|---|---| | Exchange API HTTP health | 2 missed checks | Critical | | Exchange API detailed health | 1 missed check | Critical | | Agbot agreement processing | 1 missed check | Critical | | Edge agent heartbeat (fleet-wide) | 1 missed check | High | | Edge service deployment success | 1 missed check | High | | Edge node registration count | 1 missed check | High | | Service policy compliance | 2 missed checks | High | | Secrets Manager health | 1 missed check | High | | MMS health | 2 missed checks | Medium |

Conclusion

Open Horizon manages containerized edge services across a distributed fleet that may span thousands of nodes — and those nodes are designed to run without persistent cloud connectivity. That autonomy makes hub-side monitoring even more important: when the Exchange API, Agbot, or Secrets Manager degrades, the impact propagates silently to every node that tries to negotiate a new agreement or fetch updated credentials. With Vigilmon covering the Management Hub APIs, per-node agent heartbeats, agreement health, policy compliance, and the MMS, you get complete fleet observability without deploying a dedicated edge monitoring stack.

Get started at vigilmon.online — free for up to 5 monitors.