Monitoring Shifu with Vigilmon

Shifu is an Apache 2.0-licensed Kubernetes-native IoT device virtualization platform developed by Edgenesis Inc. Every physical device connected to a Shifu cluster gets a corresponding deviceShifu Pod — a virtual twin that speaks REST, regardless of whether the actual device uses HTTP, MQTT, OPC-UA, TCP socket, RS232, or PLC protocols. When the Shifu controller crashes, a deviceShifu Pod enters CrashLoopBackOff, or device protocol connectivity drops, your automation workflows silently receive no device data. Vigilmon monitors the Shifu controller, individual deviceShifu REST APIs, telemetry collection health, and cluster resource usage — giving you per-device visibility across your entire connected fleet.

What You'll Set Up

• Shifu controller (Kubernetes operator) health heartbeat
• deviceShifu Pod liveness heartbeat
• Device protocol connectivity heartbeat
• deviceShifu REST API availability monitor
• EdgeDevice CRD count drift heartbeat
• Telemetry Service collection health heartbeat
• Kubernetes node affinity scheduling health heartbeat
• Device command execution rate heartbeat
• Shifu API server health monitor
• Per-deviceShifu resource usage heartbeat

Prerequisites

• Shifu deployed on a Kubernetes cluster (v1.20+)
• kubectl configured with access to the Shifu namespace
• Shifu Telemetry Service configured (optional but recommended)
• A monitoring host or Kubernetes CronJob with outbound HTTPS access to vigilmon.online
• A free Vigilmon account

Step 1: Monitor the Shifu Controller

The Shifu controller is a Kubernetes operator that watches EdgeDevice custom resources and creates deviceShifu Pods. If the controller crashes, new EdgeDevice CRDs are ignored — physical devices get no virtual twin and no REST API.

Deploy a heartbeat script as a Kubernetes CronJob in the Shifu namespace:

#!/bin/bash
# check-shifu-controller.sh (runs inside the cluster)
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Check the Shifu controller deployment is fully available
DESIRED=$(kubectl get deployment shifu-crd-controller-manager \
  -n shifu-crd-system \
  -o jsonpath='{.spec.replicas}' 2>/dev/null)

AVAILABLE=$(kubectl get deployment shifu-crd-controller-manager \
  -n shifu-crd-system \
  -o jsonpath='{.status.availableReplicas}' 2>/dev/null)

if [ -n "$DESIRED" ] && [ "$DESIRED" = "$AVAILABLE" ] && [ "$AVAILABLE" -gt 0 ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Apply as a Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: shifu-controller-health
  namespace: shifu-crd-system
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: shifu-monitor-sa
          restartPolicy: Never
          containers:
          - name: monitor
            image: bitnami/kubectl:latest
            command: ["/bin/bash", "-c"]
            args:
            - |
              DESIRED=$(kubectl get deployment shifu-crd-controller-manager \
                -n shifu-crd-system -o jsonpath='{.spec.replicas}')
              AVAILABLE=$(kubectl get deployment shifu-crd-controller-manager \
                -n shifu-crd-system -o jsonpath='{.status.availableReplicas}')
              if [ "$DESIRED" = "$AVAILABLE" ] && [ "$AVAILABLE" -gt 0 ]; then
                curl -s -X POST "https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
              fi

In Vigilmon:

1. Click Add Monitor → Cron Heartbeat.
2. Name: Shifu Controller Health.
3. Expected interval: 2 minutes.
4. Grace period: 5 minutes.
5. Click Save.

Step 2: Monitor deviceShifu Pod Health

Each physical device gets a corresponding deviceShifu Pod. A Pod entering CrashLoopBackOff means the protocol driver cannot connect to the physical device — the device's virtual twin is unreachable.

#!/bin/bash
# check-deviceshifu-pods.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
DEVICESHIFU_NAMESPACE="deviceshifu"

# Count pods not in Running state
NOT_RUNNING=$(kubectl get pods -n "$DEVICESHIFU_NAMESPACE" \
  --no-headers 2>/dev/null \
  | grep -v "Running" \
  | grep -v "Completed" \
  | wc -l)

TOTAL=$(kubectl get pods -n "$DEVICESHIFU_NAMESPACE" \
  --no-headers 2>/dev/null \
  | wc -l)

# Only send heartbeat when all deviceShifu pods are Running
if [ "$NOT_RUNNING" = "0" ] && [ "$TOTAL" -gt 0 ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 2 minutes with a 5-minute grace period.

Alert recommendation: Any deviceShifu Pod entering CrashLoopBackOff means a physical device lost its virtual twin. Alert immediately — this breaks all automation workflows interacting with that device.

Step 3: Monitor Device Protocol Connectivity

deviceShifu Pods connect to physical devices via their native protocol (HTTP, MQTT, OPC-UA, Modbus, TCP socket). Protocol connectivity failures are distinct from Pod health — the Pod may be Running but silently failing to read device data if the physical device is offline.

#!/bin/bash
# check-shifu-device-connectivity.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
DEVICESHIFU_NAMESPACE="deviceshifu"
MAX_STALE_SECONDS=120

# Check each deviceShifu's last-data-fetch time via its REST API
# deviceShifu exposes a /health endpoint showing last successful device read
ALL_HEALTHY=true

for POD_IP in $(kubectl get pods -n "$DEVICESHIFU_NAMESPACE" \
  -o jsonpath='{.items[*].status.podIP}' 2>/dev/null); do

  LAST_READ=$(curl -s "http://${POD_IP}:8080/health" --max-time 5 \
    2>/dev/null | grep -o '"lastDeviceReadTimestamp":[0-9]*' \
    | cut -d: -f2)

  if [ -n "$LAST_READ" ]; then
    NOW=$(date +%s)
    AGE=$((NOW - LAST_READ / 1000))
    if [ "$AGE" -gt "$MAX_STALE_SECONDS" ]; then
      ALL_HEALTHY=false
      break
    fi
  fi
done

if [ "$ALL_HEALTHY" = true ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 2 minutes with a 5-minute grace period.

Step 4: Monitor deviceShifu REST API Availability

deviceShifu Pods expose a REST API that operators and automation workflows use to read device data and send commands. An unreachable REST API breaks all integrations for that device.

For each critical device, create a dedicated Vigilmon HTTP monitor:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: http://deviceshifu-DEVICE_NAME.deviceshifu.svc.cluster.local:8080/health (use the Kubernetes service DNS name, or the external IP if exposed via LoadBalancer/NodePort).
4. Check interval: 1 minute.
5. Expected HTTP status: 200.
6. Click Save.

For fleets with many devices, automate monitor creation using the Vigilmon API and a script that iterates over all EdgeDevice resources.

Step 5: Monitor EdgeDevice CRD Count

The EdgeDevice CRD count is the authoritative record of how many physical devices are registered in Shifu. An unexpected drop indicates EdgeDevice deletion, controller sync failure, or etcd state corruption.

#!/bin/bash
# check-shifu-edgedevice-count.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
EXPECTED_MIN_DEVICES=5  # set to the minimum expected registered device count

DEVICE_COUNT=$(kubectl get edgedevices --all-namespaces --no-headers \
  2>/dev/null | wc -l)

if [ "$DEVICE_COUNT" -ge "$EXPECTED_MIN_DEVICES" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 15-minute grace period.

Step 6: Monitor Telemetry Collection Health

Shifu's Telemetry Service collects device metrics from deviceShifu Pods and exports them to InfluxDB or Prometheus. Telemetry failures create blind spots in device monitoring dashboards without any direct device impact.

#!/bin/bash
# check-shifu-telemetry.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Check Shifu Telemetry Service Pod is Running
TELEMETRY_STATUS=$(kubectl get pods \
  -n shifu-crd-system \
  -l app=shifu-telemetry-service \
  --no-headers 2>/dev/null \
  | awk '{print $3}' | head -1)

if [ "$TELEMETRY_STATUS" = "Running" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

For InfluxDB-backed telemetry, also check write success by querying the latest metric timestamp:

#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
INFLUXDB_HOST="your-influxdb-host"
INFLUXDB_TOKEN="your-token"
INFLUXDB_ORG="your-org"
MAX_STALE_SECONDS=300

LAST_WRITE=$(curl -s \
  -H "Authorization: Token ${INFLUXDB_TOKEN}" \
  -H "Content-Type: application/vnd.flux" \
  --data 'from(bucket:"shifu") |> range(start:-5m) |> last()' \
  "http://${INFLUXDB_HOST}:8086/api/v2/query?org=${INFLUXDB_ORG}" \
  --max-time 10 | grep -c "_value")

if [ "$LAST_WRITE" -gt 0 ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 10-minute grace period.

Step 7: Monitor Node Affinity Scheduling Health

deviceShifu Pods often require node affinity constraints — they must run on the specific Kubernetes node that is physically connected to their device (e.g., via USB, RS232, or local network). Pods stuck in Pending due to unsatisfied node affinity mean the physical device has no virtual twin.

#!/bin/bash
# check-shifu-scheduling.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"

# Count deviceShifu pods stuck in Pending state
PENDING_COUNT=$(kubectl get pods \
  -n deviceshifu \
  --field-selector=status.phase=Pending \
  --no-headers 2>/dev/null \
  | wc -l)

if [ "$PENDING_COUNT" = "0" ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 10-minute grace period.

Step 8: Monitor Per-deviceShifu Resource Usage

deviceShifu Pods can enter a CPU or memory spike if the underlying protocol driver enters a retry loop against an unresponsive physical device. Catching resource anomalies early prevents node resource exhaustion.

#!/bin/bash
# check-shifu-resources.sh
HEARTBEAT_URL="https://vigilmon.online/api/v1/heartbeat/YOUR_HEARTBEAT_ID"
MAX_CPU_MILLICORES=500    # alert if any deviceShifu pod uses >500m CPU
MAX_MEMORY_MB=512         # alert if any deviceShifu pod uses >512Mi RAM

ALL_OK=true

while IFS= read -r line; do
  POD_NAME=$(echo "$line" | awk '{print $1}')
  CPU_M=$(echo "$line" | awk '{print $2}' | sed 's/m//')
  MEM_MI=$(echo "$line" | awk '{print $3}' | sed 's/Mi//')

  if [ -n "$CPU_M" ] && [ "$CPU_M" -gt "$MAX_CPU_MILLICORES" ]; then
    ALL_OK=false; break
  fi
  if [ -n "$MEM_MI" ] && [ "$MEM_MI" -gt "$MAX_MEMORY_MB" ]; then
    ALL_OK=false; break
  fi
done < <(kubectl top pods -n deviceshifu --no-headers 2>/dev/null)

if [ "$ALL_OK" = true ]; then
  curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Set the expected interval to 5 minutes with a 10-minute grace period.

Step 9: Configure Alerting

Apply alert channels to all Shifu monitors:

1. Go to Alerts → Add Alert Channel → choose Email, Slack, PagerDuty, or Webhook.
2. Apply the channel to all Shifu monitors.

Recommended thresholds:

| Monitor | Alert After | Severity | |---|---|---| | Shifu controller deployment | 1 missed check | Critical | | deviceShifu Pod health (any CrashLoop) | 1 missed check | Critical | | Device protocol connectivity | 1 missed check | High | | deviceShifu REST API | 2 missed checks | High | | EdgeDevice CRD count drop | 1 missed check | High | | Telemetry Service health | 2 missed checks | Medium | | Node affinity pending Pods | 1 missed check | High | | Per-Pod resource usage | 1 missed check | High |

Conclusion

Shifu creates a Kubernetes-native virtual twin for every physical device — but when the controller crashes, deviceShifu Pods enter crash loops, or protocol drivers stall, the fleet goes blind with no automatic recovery signal. With Vigilmon heartbeats monitoring the Shifu controller, per-device Pod health, protocol connectivity, telemetry pipelines, and resource usage, you get per-device observability without needing an external APM platform. The heartbeat-driven approach works from inside the cluster via CronJobs, keeping monitoring logic close to the system being observed.

Get started at vigilmon.online — free for up to 5 monitors.