How to Monitor k0smotron Managed Kubernetes Control Planes with Vigilmon

k0smotron packs managed Kubernetes control planes as StatefulSets into a host cluster, making it possible to run dozens of isolated k0s clusters from a single management plane. But when a managed cluster's etcd pod loses quorum, that cluster's workloads become unmanageable — and the failure appears as a scheduling backoff in the host cluster's event log, not as a page to your on-call team. When a PersistentVolumeClaim for a control plane pod becomes unbound, the entire control plane for that managed cluster stops starting.

Vigilmon gives you external uptime monitoring for the k0smotron operator, each managed cluster's API server endpoint, and host cluster capacity — backed by HTTP probes, TCP checks, and heartbeat monitors. This tutorial covers the full setup.

Why k0smotron Needs External Monitoring

The host Kubernetes cluster has kubectl get pods and Prometheus metrics. But these tell you what the host cluster can see. External monitoring with Vigilmon adds:

• Managed cluster API server reachability — from outside the host cluster, verifying that the endpoint returned by k0smotron is actually serving API requests
• k0smotron operator crash detection — if the k0smotron controller manager pod crashes, no new Cluster CRDs are reconciled; new managed clusters silently fail to provision
• Per-cluster alerting — each managed cluster's API server is a separate Vigilmon monitor, so you know exactly which tenant's cluster is down
• Host cluster capacity headroom — when the host cluster runs out of schedulable CPU/memory, new managed cluster control planes fail to start; Vigilmon's heartbeat monitors catch this before it blocks provisioning

What you'll need

• A host Kubernetes cluster running k0smotron (version 1.x or later)
• kubectl configured with cluster-admin access to the host cluster
• A lightweight health sidecar with access to the Kubernetes API
• A free Vigilmon account

Step 1: Build a k0smotron health endpoint

The sidecar queries the Kubernetes API for k0smotron operator pod health, managed cluster StatefulSet status, and host cluster capacity.

# k0smotron_health.py
import os
import json
from flask import Flask, jsonify
from kubernetes import client, config

app = Flask(__name__)

try:
    config.load_incluster_config()
except config.ConfigException:
    config.load_kube_config()

v1 = client.CoreV1Api()
apps_v1 = client.AppsV1Api()
custom_api = client.CustomObjectsApi()


@app.route('/health/k0smotron/operator')
def operator_health():
    try:
        pods = v1.list_namespaced_pod(
            namespace='k0smotron',
            label_selector='app.kubernetes.io/name=k0smotron'
        )
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    if not pods.items:
        return jsonify(status='down', error='k0smotron operator pod not found'), 503

    not_running = [
        p.metadata.name for p in pods.items
        if p.status.phase != 'Running'
    ]

    if not_running:
        return jsonify(
            status='down',
            reason='operator_pod_not_running',
            pods=not_running,
        ), 503

    return jsonify(status='ok', operator_pods=len(pods.items))


@app.route('/health/k0smotron/clusters')
def cluster_health():
    try:
        clusters = custom_api.list_cluster_custom_object(
            group='k0smotron.io',
            version='v1beta1',
            plural='clusters',
        )
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    degraded = []
    healthy = []

    for cluster in clusters.get('items', []):
        name = cluster['metadata']['name']
        namespace = cluster['metadata'].get('namespace', 'default')
        conditions = cluster.get('status', {}).get('conditions', [])

        ready = any(
            c.get('type') == 'Ready' and c.get('status') == 'True'
            for c in conditions
        )

        try:
            sts = apps_v1.read_namespaced_stateful_set(
                name=name,
                namespace=namespace,
            )
            desired = sts.spec.replicas or 1
            ready_replicas = sts.status.ready_replicas or 0
        except Exception:
            desired, ready_replicas = 1, 0

        entry = {
            'cluster': name,
            'namespace': namespace,
            'ready': ready,
            'desired_replicas': desired,
            'ready_replicas': ready_replicas,
        }

        if not ready or ready_replicas < desired:
            degraded.append(entry)
        else:
            healthy.append(entry)

    if degraded:
        return jsonify(
            status='degraded',
            degraded_clusters=degraded,
            healthy_clusters=len(healthy),
        ), 503

    return jsonify(
        status='ok',
        total_clusters=len(healthy),
        clusters=healthy,
    )


@app.route('/health/k0smotron/pvcs')
def pvc_health():
    try:
        # Find PVCs used by k0smotron control plane StatefulSets
        pvcs = v1.list_persistent_volume_claim_for_all_namespaces(
            label_selector='app.kubernetes.io/managed-by=k0smotron'
        )
    except Exception as e:
        # Fallback: list all PVCs with 'k0smotron' in name
        try:
            pvcs = v1.list_persistent_volume_claim_for_all_namespaces()
        except Exception as e2:
            return jsonify(status='down', error=str(e2)), 503

    unbound = [
        {
            'name': pvc.metadata.name,
            'namespace': pvc.metadata.namespace,
            'phase': pvc.status.phase,
        }
        for pvc in pvcs.items
        if pvc.status.phase != 'Bound'
        and 'k0smotron' in (pvc.metadata.name or '')
    ]

    if unbound:
        return jsonify(
            status='critical',
            reason='pvc_unbound',
            unbound_pvcs=unbound,
        ), 503

    return jsonify(status='ok', pvc_count=len(pvcs.items))


@app.route('/health/k0smotron/capacity')
def capacity_health():
    try:
        nodes = v1.list_node()
        pods = v1.list_pod_for_all_namespaces()
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    total_cpu_millicores = 0
    total_mem_bytes = 0

    for node in nodes.items:
        if any(
            t.effect == 'NoSchedule' for t in (node.spec.taints or [])
        ):
            continue
        cap = node.status.allocatable or {}
        cpu_str = cap.get('cpu', '0')
        mem_str = cap.get('memory', '0Ki')
        if cpu_str.endswith('m'):
            total_cpu_millicores += int(cpu_str[:-1])
        else:
            total_cpu_millicores += int(float(cpu_str) * 1000)
        mem_val = mem_str.rstrip('Ki')
        total_mem_bytes += int(mem_val) * 1024

    requested_cpu = 0
    requested_mem = 0

    for pod in pods.items:
        if pod.status.phase not in ('Running', 'Pending'):
            continue
        for container in (pod.spec.containers or []):
            req = (container.resources.requests or {}) if container.resources else {}
            c = req.get('cpu', '0')
            m = req.get('memory', '0Ki')
            if c.endswith('m'):
                requested_cpu += int(c[:-1])
            else:
                requested_cpu += int(float(c) * 1000)
            if m.endswith('Ki'):
                requested_mem += int(m[:-2]) * 1024
            elif m.endswith('Mi'):
                requested_mem += int(m[:-2]) * 1024 * 1024

    cpu_headroom_pct = round((1 - requested_cpu / max(total_cpu_millicores, 1)) * 100, 1)
    mem_headroom_pct = round((1 - requested_mem / max(total_mem_bytes, 1)) * 100, 1)

    if cpu_headroom_pct < 20 or mem_headroom_pct < 20:
        return jsonify(
            status='critical',
            reason='low_capacity_headroom',
            cpu_headroom_pct=cpu_headroom_pct,
            mem_headroom_pct=mem_headroom_pct,
        ), 503

    return jsonify(
        status='ok',
        cpu_headroom_pct=cpu_headroom_pct,
        mem_headroom_pct=mem_headroom_pct,
    )


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=9082)

Deploy to the host cluster as a Deployment:

# k0smotron-health-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k0smotron-health
  namespace: k0smotron
spec:
  replicas: 1
  selector:
    matchLabels:
      app: k0smotron-health
  template:
    metadata:
      labels:
        app: k0smotron-health
    spec:
      serviceAccountName: k0smotron-health-reader
      containers:
        - name: health
          image: python:3.12-slim
          command: [sh, -c, "pip install flask kubernetes && python /app/k0smotron_health.py"]
          ports:
            - containerPort: 9082
          volumeMounts:
            - name: app
              mountPath: /app
      volumes:
        - name: app
          configMap:
            name: k0smotron-health-script
---
apiVersion: v1
kind: Service
metadata:
  name: k0smotron-health
  namespace: k0smotron
spec:
  type: LoadBalancer
  selector:
    app: k0smotron-health
  ports:
    - port: 9082
      targetPort: 9082

# RBAC for the health sidecar
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k0smotron-health-reader
  namespace: k0smotron
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k0smotron-health-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "persistentvolumeclaims"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    verbs: ["get", "list"]
  - apiGroups: ["k0smotron.io"]
    resources: ["clusters"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k0smotron-health-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: k0smotron-health-reader
subjects:
  - kind: ServiceAccount
    name: k0smotron-health-reader
    namespace: k0smotron

Apply and verify:

kubectl apply -f k0smotron-health-deploy.yaml
kubectl apply -f rbac.yaml

HEALTH_IP=$(kubectl get svc k0smotron-health -n k0smotron -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl http://$HEALTH_IP:9082/health/k0smotron/operator
# {"status": "ok", "operator_pods": 1}

curl http://$HEALTH_IP:9082/health/k0smotron/clusters
# {"status": "ok", "total_clusters": 5, ...}

Step 2: Monitor the k0smotron operator

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: http://<HEALTH_IP>:9082/health/k0smotron/operator
4. Expected status: 200
5. Check interval: 1 minute
6. Save as "k0smotron Operator Health"

Step 3: Monitor managed cluster API servers

Each managed k0s cluster's API server is exposed as a Service in the host cluster. Monitor each one directly.

Get the API server endpoint for each managed cluster:

kubectl get clusters -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,APIENDPOINT:.status.externalAddress'

For each cluster, add an HTTP monitor in Vigilmon:

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: https://<cluster-api-server-ip>:6443/healthz (or /readyz)
3. Expected status: 200
4. Expected body contains: ok
5. Save as "k0smotron Cluster: <cluster-name> API Server"

Also add TCP monitors for each API server port:

1. Monitors → New Monitor → TCP Port
2. Host: <cluster-api-server-ip>
3. Port: 6443
4. Save as "k0smotron Cluster: <cluster-name> API Port"

Automate this with a script that registers monitors for all clusters:

#!/bin/bash
# register-cluster-monitors.sh
kubectl get clusters -A -o json | jq -r '.items[] | "\(.metadata.name) \(.status.externalAddress)"' | while read name addr; do
    echo "Register monitor for cluster $name at $addr:6443"
    # POST to Vigilmon API to create monitor (use your API key)
done

Step 4: Monitor managed cluster health via sidecar

Add the aggregate cluster health check as a monitor:

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://<HEALTH_IP>:9082/health/k0smotron/clusters
3. Expected status: 200
4. Check interval: 2 minutes
5. Save as "k0smotron Managed Cluster Control Planes"

PVC health (control plane storage)

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://<HEALTH_IP>:9082/health/k0smotron/pvcs
3. Expected status: 200
4. Check interval: 5 minutes
5. Save as "k0smotron Control Plane PVC Health"

Host cluster capacity

1. Monitors → New Monitor → HTTP / HTTPS
2. URL: http://<HEALTH_IP>:9082/health/k0smotron/capacity
3. Expected status: 200
4. Check interval: 10 minutes
5. Save as "k0smotron Host Cluster Capacity"

Step 5: Add heartbeat monitors for cluster reconciliation

k0smotron continuously reconciles Cluster CRDs. Add a heartbeat that verifies reconciliation is happening:

#!/bin/bash
# k0smotron-reconcile-check.sh — run every 5 minutes via CronJob

HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_TOKEN"

# Check last reconcile time on all clusters
kubectl get clusters -A -o json | \
  jq '[.items[] | .status.conditions[] | select(.type=="Ready") | .lastTransitionTime] | max' | \
  xargs -I{} bash -c 'echo "Last reconcile: {}"'

# If any cluster has not been reconciled in 15 minutes, don't ping heartbeat
STALE=$(kubectl get clusters -A -o json | jq '[.items[] | .status.conditions[] | select(.type=="Ready" and (.lastTransitionTime | fromdateiso8601) < (now - 900))] | length')

if [ "$STALE" = "0" ]; then
    curl -s -X POST "$HEARTBEAT_URL" > /dev/null
fi

Deploy as a Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: k0smotron-reconcile-heartbeat
  namespace: k0smotron
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: k0smotron-health-reader
          containers:
            - name: heartbeat
              image: bitnami/kubectl:latest
              command: [/bin/sh, -c]
              args:
                - |
                  HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_HEARTBEAT_TOKEN"
                  kubectl get clusters -A > /dev/null 2>&1 && curl -s -X POST "$HEARTBEAT_URL" > /dev/null
          restartPolicy: OnFailure

Step 6: Configure alert channels

Email alerts

1. Alert Channels → Add Channel → Email
2. Enter your platform engineering or SRE on-call email
3. Assign to all k0smotron monitors

PagerDuty/Slack webhook

Vigilmon sends:

{
  "monitor_name": "k0smotron Cluster: prod-eu-1 API Server",
  "status": "down",
  "url": "https://10.100.5.22:6443/healthz",
  "started_at": "2026-05-12T08:15:00Z",
  "duration_seconds": 90
}

Use this to route alerts to PagerDuty with cluster name in the title so the right team gets paged.

Recommended alert configuration

| Monitor | Urgency | First response | |---------|---------|----------------| | Operator Health | Critical | Check k0smotron pod logs: kubectl logs -n k0smotron deploy/k0smotron | | Cluster API Server (per cluster) | Critical | Check StatefulSet: kubectl get sts -n <ns> | | Cluster Control Planes | High | kubectl describe cluster <name> for conditions | | PVC Health | High | kubectl get pvc -A — check storage class provisioner | | Host Capacity | Medium | Scale host cluster nodes before headroom < 10% | | Reconcile Heartbeat | Medium | kubectl get events -n k0smotron |

Full monitor summary

| Monitor | Type | Endpoint | What it catches | |---------|------|----------|-----------------| | k0smotron Operator | HTTP | :9082/health/k0smotron/operator | Operator pod crash | | Cluster API: <name> | HTTP | <api-server-ip>:6443/healthz | Managed cluster unreachable | | API Port: <name> | TCP | <api-server-ip>:6443 | Port-level unreachability | | Cluster Control Planes | HTTP | :9082/health/k0smotron/clusters | StatefulSet degraded | | PVC Health | HTTP | :9082/health/k0smotron/pvcs | Unbound PVCs | | Host Capacity | HTTP | :9082/health/k0smotron/capacity | <20% CPU/memory headroom | | Reconcile Heartbeat | Heartbeat | heartbeat URL | Reconciliation stopped |

What's next

• Cluster API integration — if you use k0smotron as a Cluster API provider, add monitors for MachineDeployment readiness via the health sidecar's Kubernetes API
• Status page per tenant — create a Vigilmon status page for each managed cluster, giving tenants visibility into their control plane health without host cluster access
• Certificate expiry monitoring — k0s-generated API server certificates have fixed lifetimes; Vigilmon will alert you before they expire when monitoring the HTTPS endpoint

Get started free at vigilmon.online — no credit card required, monitors start running in under a minute.