tutorial

Monitoring Elemental OS Nodes with Vigilmon

Elemental is SUSE/Rancher's Kubernetes-native OS lifecycle management platform for immutable edge nodes — but when the operator crashes or an OS upgrade fails silently, you lose visibility into your entire fleet. Here's how to monitor Elemental with Vigilmon.

Elemental is an open source Kubernetes-native OS management platform developed by SUSE/Rancher that provides lifecycle management for immutable Linux OS instances running on edge nodes, bare-metal servers, and VMs. It is tightly integrated with Rancher and uses Elemental OS (built from SLE Micro or openSUSE MicroOS) as the managed OS image.

The Elemental operator in a Rancher management cluster handles everything: initial OS installation via iPXE, OS upgrades via A/B partition switching, and a node inventory of MachineInventory and MachineRegistration CRDs. When the operator crashes, OS upgrades stop. When a node disappears from inventory, it's silently lost. Vigilmon gives you the monitoring layer to catch these failures across the entire Elemental stack — from the operator to individual node health.

What You'll Set Up

  • Elemental operator health monitoring
  • Channel server availability checks
  • Node registration and MachineInventory health
  • OS upgrade success rate and stuck upgrade alerts
  • A/B partition rollback detection
  • Fleet online percentage monitoring
  • Rancher cluster registration health
  • iPXE/network boot server health

Prerequisites

  • Elemental operator installed in a Rancher management cluster
  • At least one managed node registered via MachineInventory
  • Access to the Rancher API (or kubectl access to the management cluster)
  • A free Vigilmon account

Step 1: Monitor the Elemental Operator

The Elemental operator is a Kubernetes Deployment in the Rancher management cluster. If it crashes, no OS upgrades are processed, new node registrations may stall, and the MachineInventory CRDs stop being updated.

Deploy a health checker as a Kubernetes CronJob in the management cluster:

#!/bin/bash
# Check elemental-operator deployment
READY=$(kubectl get deployment elemental-operator \
  -n cattle-elemental-system \
  -o jsonpath='{.status.readyReplicas}')
DESIRED=$(kubectl get deployment elemental-operator \
  -n cattle-elemental-system \
  -o jsonpath='{.spec.replicas}')

if [ "$READY" != "$DESIRED" ] || [ -z "$READY" ]; then
  echo "Elemental operator unhealthy: $READY/$DESIRED ready"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_OPERATOR_HB_TOKEN

In Vigilmon:

  1. Log in to vigilmon.online and click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 5 minutes.
  3. Copy the heartbeat URL and paste it into the script above.
  4. Click Save.

If the operator pod goes missing or enters CrashLoopBackOff, the heartbeat stops and Vigilmon alerts within 5 minutes.

Step 2: Monitor the Elemental Channel Server

The Elemental channel server distributes OS images to managed nodes. Nodes fetch the OS image from the channel server during upgrades. If it goes down, all OS upgrades fail silently.

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter the channel server URL: https://channel.elemental.yourdomain.com/v1/channels (or the internal service URL if monitoring from within the cluster).
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 2 minutes.
  5. Enable Monitor SSL certificate, set expiry alert to 21 days.
  6. Click Save.

If the channel server is an internal Kubernetes Service, expose a health endpoint via a Kubernetes Service of type LoadBalancer or use an ingress, then add the HTTP monitor against the ingress URL.

Step 3: Monitor Node Registration Health

New Elemental nodes register with the management cluster via MachineRegistration CRDs. Registration failures mean new nodes cannot be provisioned. Monitor registration success with a heartbeat that fires when a new node successfully registers:

Add a registration hook to your Elemental MachineRegistration cloud-init config:

# In your MachineRegistration spec, under config.cloud-config
write_files:
  - path: /oem/registration-notify.sh
    permissions: "0755"
    content: |
      #!/bin/bash
      # Fires after successful registration
      curl -sf https://vigilmon.online/heartbeat/YOUR_REGISTRATION_HB_TOKEN || true
runcmd:
  - /oem/registration-notify.sh

For ongoing monitoring, check the MachineRegistration CRD status from the management cluster:

#!/bin/bash
# Check for MachineRegistrations in error state
FAILED=$(kubectl get machineregistrations -A \
  -o jsonpath='{range .items[?(@.status.conditions[-1].type=="Ready")]}{"\n"}{.metadata.namespace}/{.metadata.name}: {.status.conditions[-1].status}{end}' \
  | grep -v ": True")

if [ -n "$FAILED" ]; then
  echo "Failed MachineRegistrations: $FAILED"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_REG_HEALTH_HB_TOKEN

Step 4: Monitor MachineInventory Count

The Elemental operator maintains a MachineInventory CRD for every managed node. If nodes disappear from inventory — due to operator failure, etcd issues, or accidental CRD deletion — they are silently lost from the fleet.

Monitor inventory count against your expected fleet size:

#!/bin/bash
EXPECTED_NODES=10  # Set to your expected fleet size
INVENTORY_COUNT=$(kubectl get machineinventory -A --no-headers | wc -l)

echo "MachineInventory count: $INVENTORY_COUNT (expected: $EXPECTED_NODES)"

if [ "$INVENTORY_COUNT" -lt "$EXPECTED_NODES" ]; then
  echo "ALERT: Node inventory below expected count"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_INVENTORY_HB_TOKEN

Set the Vigilmon heartbeat interval to 10 minutes. Adjust EXPECTED_NODES when you add or remove nodes from the fleet.

Step 5: Monitor OS Upgrade Health

Elemental manages OS upgrades via ManagedOSImage CRDs. Failed upgrades or nodes stuck in the upgrading state indicate a problem with the OS image, the channel server, or individual nodes.

Deploy an upgrade health checker:

#!/bin/bash
# Check for failed upgrades
FAILED=$(kubectl get managedosimages -A -o json | python3 -c "
import sys, json
data = json.load(sys.stdin)
for item in data['items']:
    for cond in item.get('status', {}).get('conditions', []):
        if cond.get('type') == 'Ready' and cond.get('status') == 'False':
            print(item['metadata']['namespace'] + '/' + item['metadata']['name'])
")

# Check for nodes stuck in upgrading > 30 minutes
STUCK=$(kubectl get managedosimages -A -o json | python3 -c "
import sys, json
from datetime import datetime, timezone, timedelta
data = json.load(sys.stdin)
now = datetime.now(timezone.utc)
for item in data['items']:
    phase = item.get('status', {}).get('upgradeStatus', '')
    if phase != 'Upgrading':
        continue
    ts = item.get('status', {}).get('upgradeStartTime', '')
    if ts:
        started = datetime.fromisoformat(ts.replace('Z', '+00:00'))
        if now - started > timedelta(minutes=30):
            print(item['metadata']['namespace'] + '/' + item['metadata']['name'])
")

if [ -n "$FAILED" ] || [ -n "$STUCK" ]; then
  echo "Upgrade issues - Failed: $FAILED Stuck: $STUCK"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_UPGRADE_HB_TOKEN

Set the heartbeat interval to 15 minutes.

Step 6: Monitor A/B Partition Rollback Events

Elemental uses A/B partition switching for safe OS updates. An automatic rollback means the new OS image failed to boot, and the node fell back to the previous version. This requires immediate investigation.

Configure a post-boot cloud-init hook on each managed node to report the active partition version:

# /oem/partition-report.yaml — deployed via MachineRegistration
stages:
  boot:
    - name: "Report active partition version to Vigilmon"
      commands:
        - |
          EXPECTED_VERSION="v1.2.3"  # Set to your current deployed version
          CURRENT_VERSION=$(kairos-agent state 2>/dev/null | grep version | awk '{print $2}' \
            || cat /etc/elemental-version 2>/dev/null || echo "unknown")
          if [ "$CURRENT_VERSION" = "$EXPECTED_VERSION" ]; then
            curl -sf https://vigilmon.online/heartbeat/YOUR_PARTITION_HB_TOKEN || true
          fi
          # If versions don't match: rollback occurred, heartbeat NOT sent → Vigilmon alerts

Set the heartbeat interval to 48 hours (or your OS upgrade cadence). If a rollback occurs after an upgrade, the node won't ping the expected-version heartbeat, and Vigilmon alerts.

Step 7: Monitor Fleet Online Percentage

Track the percentage of Elemental-managed nodes that are online. A sudden drop below 90% indicates a widespread failure — bad OS upgrade, network outage, or management cluster etcd issues.

#!/bin/bash
# Count online vs total from MachineInventory heartbeat timestamps
TOTAL=$(kubectl get machineinventory -A --no-headers | wc -l)
# Nodes that have checked in within the last 6 hours are "online"
ONLINE=$(kubectl get machineinventory -A -o json | python3 -c "
import sys, json
from datetime import datetime, timezone, timedelta
data = json.load(sys.stdin)
count = 0
threshold = datetime.now(timezone.utc) - timedelta(hours=6)
for item in data['items']:
    last_seen = item.get('status', {}).get('lastContactTime', '')
    if last_seen:
        ts = datetime.fromisoformat(last_seen.replace('Z', '+00:00'))
        if ts > threshold:
            count += 1
print(count)
")

ONLINE_PCT=$((ONLINE * 100 / TOTAL))
echo "Fleet: $ONLINE/$TOTAL online ($ONLINE_PCT%)"

if [ "$ONLINE_PCT" -lt 90 ]; then
  echo "Fleet online percentage below 90%"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_FLEET_HB_TOKEN

Set the heartbeat interval to 15 minutes.

Step 8: Monitor Rancher Cluster Registration

After OS provisioning, Elemental nodes enroll into Rancher-managed Kubernetes clusters. Failure to join Rancher means the node is running but unmanaged and not contributing to any cluster.

Monitor cluster membership from the Rancher management cluster:

#!/bin/bash
# Check that all nodes in elemental-managed clusters are in Ready state
NOTREADY=$(kubectl get nodes --all-namespaces \
  -l node.elemental.cattle.io/managed=true \
  --no-headers \
  | grep -v " Ready " \
  | awk '{print $1"/"$2}')

if [ -n "$NOTREADY" ]; then
  echo "Elemental nodes not Ready in Rancher: $NOTREADY"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_RANCHER_REG_HB_TOKEN

Set the heartbeat interval to 10 minutes.

Step 9: Monitor the iPXE/Network Boot Server

If you use network boot (iPXE) for initial OS installation on bare-metal nodes, monitor the iPXE server. An outage blocks all new bare-metal node provisioning:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter the iPXE server URL: http://ipxe.yourdomain.com/ipxe/elemental
  3. Set Expected HTTP status to 200.
  4. Set Check interval to 5 minutes.
  5. Click Save.

Also add a TCP monitor for the TFTP port (69) used in the PXE boot sequence:

  1. Click Add Monitor → TCP Port.
  2. Enter: ipxe.yourdomain.com:69
  3. Set Check interval to 5 minutes.
  4. Click Save.

Step 10: Configure Alert Channels and Thresholds

  1. Go to Alert Channels in Vigilmon and add Slack, email, or PagerDuty.
  2. Configure per-monitor alert sensitivity:
    • Elemental operator: alert immediately on heartbeat expiry.
    • Channel server: alert after 2 failures.
    • MachineInventory count: alert immediately on count drop.
    • OS upgrade health: alert on heartbeat expiry.
    • Rollback detection: alert on heartbeat expiry.
    • Fleet online %: alert immediately when below 90%.
    • Rancher registration: alert after 2 heartbeat expiry cycles.
    • iPXE server: alert after 3 failures (less urgent than live nodes).
  3. Use Maintenance Windows before planned fleet-wide OS upgrades:
curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "FLEET_MONITOR_ID",
    "duration_minutes": 120,
    "comment": "Elemental OS upgrade to v1.3.0"
  }'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Elemental operator | CronJob → k8s API | Operator crash, upgrades stop | | Channel server | HTTP health endpoint | OS image downloads fail | | Registration health | CronJob → k8s API | New node provisioning blocked | | MachineInventory count | CronJob → k8s API | Nodes disappear from fleet | | OS upgrade health | CronJob → ManagedOSImage | Failed or stuck upgrades | | A/B rollback detection | Per-node heartbeat | Bad OS update auto-rolled back | | Fleet online % | CronJob → MachineInventory | Widespread node outage | | Rancher registration | CronJob → k8s node API | Node not joining managed cluster | | iPXE server | HTTP + TCP port 69 | Bare-metal provisioning blocked |

Elemental gives you a complete, declarative lifecycle for immutable edge OS nodes — but the lifecycle itself depends on the operator, channel server, and network boot infrastructure staying healthy. With Vigilmon monitoring each layer, from the management cluster operator to individual node heartbeats and fleet-wide online percentages, you maintain the operational visibility needed to confidently manage an immutable edge fleet at scale.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles