tutorial

How to Monitor RBAC Manager with Vigilmon

RBAC Manager is a Kubernetes operator from FairwindsOps that simplifies the management of Role-Based Access Control (RBAC) by introducing higher-level CRDs —...

RBAC Manager is a Kubernetes operator from FairwindsOps that simplifies the management of Role-Based Access Control (RBAC) by introducing higher-level CRDs — RBACDefinition and RBACBinding — that abstract the complexity of creating and maintaining ClusterRoles, ClusterRoleBindings, Roles, and RoleBindings across many namespaces. In a large cluster, RBAC Manager is the single point of truth for who can access what. If it fails silently, your RBAC definitions drift out of sync with reality and security guarantees break down — often without any obvious error surfaced to operators.

This tutorial walks you through monitoring RBAC Manager's health surfaces with Vigilmon so you catch operator failures, reconciliation drift, and missing bindings before they become a security or access incident.

Why RBAC Manager needs external monitoring

RBAC Manager is an always-on reconciliation loop. It watches Kubernetes API events and continuously ensures that actual RoleBindings match the desired state expressed in RBACDefinitions. When the operator goes wrong, the failure mode is silent drift — Kubernetes keeps running, pods keep scheduling, but the access control layer is no longer what you think it is.

External monitoring catches what internal Kubernetes probes cannot:

  • Operator pod crash — the rbac-manager Deployment exits; no further reconciliation happens; RBACDefinitions accumulate changes that are never applied to actual RoleBindings
  • Reconciliation failure — the operator is running but emitting reconciliation errors; new namespaces get created without their RBAC policies applied
  • RoleBinding count dropping — automated drift detection catches manual kubectl delete rolebinding commands or Helm upgrades that accidentally delete generated bindings
  • Kubernetes API errors — if the RBAC Manager service account itself is missing permissions, every API call fails silently; the operator looks healthy but does nothing
  • Leader election failure — in HA deployments, leader election failure means no operator is actively reconciling
  • Webhook timeouts — the admission webhook that validates RBACDefinition CRDs starts timing out, blocking all RBAC policy changes

What you'll need

  • A Kubernetes cluster with RBAC Manager installed (namespace: rbac-manager)
  • kubectl access with permission to read RBAC Manager resources
  • A free Vigilmon account — no credit card required

Step 1: Expose the RBAC Manager health endpoint

RBAC Manager exposes a health endpoint on port 8080. Expose it via a Service or Ingress so Vigilmon can probe it from outside the cluster:

# Verify RBAC Manager is running
kubectl get deployment rbac-manager -n rbac-manager

# Check the health endpoint port
kubectl describe deployment rbac-manager -n rbac-manager | grep -A5 "Ports:"

# Expose via NodePort for external monitoring
kubectl expose deployment rbac-manager \
  --name=rbac-manager-health \
  --type=NodePort \
  --port=8080 \
  --target-port=8080 \
  -n rbac-manager

Or add an Ingress rule specifically for the health path:

# rbac-manager-ingress-health.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rbac-manager-health
  namespace: rbac-manager
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /healthz
spec:
  rules:
    - host: rbac-manager.internal.example.com
      http:
        paths:
          - path: /healthz
            pathType: Prefix
            backend:
              service:
                name: rbac-manager
                port:
                  number: 8080

Verify the health endpoint responds:

curl http://rbac-manager.internal.example.com/healthz
# {"status":"ok"}

Step 2: Monitor the RBAC Manager operator pod health

The operator pod is the most critical health surface. Add it to Vigilmon as an HTTP monitor:

  1. Log in to vigilmon.online and go to Monitors → New Monitor
  2. Choose HTTP / HTTPS
  3. Set the URL to http://rbac-manager.internal.example.com/healthz
  4. Set the check interval to 1 minute
  5. Under Expected response, set status code: 200
  6. Save the monitor with the name rbac-manager operator health

When the rbac-manager pod crashes or enters CrashLoopBackOff, the health endpoint stops responding and Vigilmon fires an alert within minutes. This is far faster than waiting for someone to notice that newly created namespaces are missing their expected RoleBindings.

Correlating operator crashes with RBAC drift

When you receive a Vigilmon alert for this monitor, run:

# Check operator pod status
kubectl get pods -n rbac-manager

# Check recent events
kubectl get events -n rbac-manager --sort-by='.lastTimestamp' | tail -20

# List all RBACDefinitions and their conditions
kubectl get rbacdefinitons -A
kubectl get rbacdefinitions -A -o json | \
  jq '.items[] | {name: .metadata.name, status: .status}'

Step 3: Monitor RBACDefinition reconciliation health

Reconciliation errors are the subtler failure mode. The operator pod is running, health checks pass, but individual RBACDefinitions are failing to reconcile because of API permission errors or invalid CRD fields.

Set up a scripted health check using a Kubernetes CronJob that probes reconciliation status and pings a Vigilmon heartbeat:

# rbac-reconciliation-check.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: rbac-reconciliation-check
  namespace: rbac-manager
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: rbac-health-checker
          restartPolicy: OnFailure
          containers:
            - name: checker
              image: bitnami/kubectl:latest
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: rbac-reconciliation-heartbeat
              command:
                - /bin/sh
                - -c
                - |
                  set -e
                  # Check for RBACDefinitions with failed conditions
                  FAILED=$(kubectl get rbacdefinitions -A -o json | \
                    jq '[.items[] | select(.status.conditions[]?.type == "Ready" and .status.conditions[]?.status == "False")] | length')
                  if [ "$FAILED" -gt 0 ]; then
                    echo "ERROR: $FAILED RBACDefinitions not reconciled"
                    exit 1
                  fi
                  # All healthy — ping heartbeat
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" > /dev/null
                  echo "All RBACDefinitions reconciled. Heartbeat sent."

Set up the Vigilmon heartbeat monitor:

  1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
  2. Name: rbac-reconciliation-check
  3. Expected interval: 5 minutes
  4. Grace period: 3 minutes
  5. Save to get the heartbeat URL

Store the heartbeat URL as a Kubernetes Secret:

kubectl create secret generic vigilmon-secrets \
  --from-literal=rbac-reconciliation-heartbeat='https://vigilmon.online/heartbeat/YOUR_ID' \
  -n rbac-manager

Step 4: Monitor RoleBinding count drift

RBAC Manager manages a fleet of RoleBindings generated from RBACDefinitions. An unexpected drop in the count — caused by a Helm upgrade, an accidental kubectl delete, or namespace deletion — is a security signal.

Add a drift detection check to the same CronJob or a dedicated one:

# Count RoleBindings managed by RBAC Manager (they have the owner annotation)
kubectl get rolebindings -A -o json | \
  jq '[.items[] | select(.metadata.annotations."rbac-manager" != null)] | length'

# Count ClusterRoleBindings managed by RBAC Manager
kubectl get clusterrolebindings -o json | \
  jq '[.items[] | select(.metadata.annotations."rbac-manager" != null)] | length'

Store baseline counts as ConfigMap values and alert if the actual count drops below the expected floor:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rbac-baseline
  namespace: rbac-manager
data:
  min_rolebindings: "42"      # Set this to your expected floor
  min_clusterrolebindings: "8"

The heartbeat skips the ping if counts fall below the baseline, which fires a Vigilmon alert.

Step 5: Monitor the admission webhook

RBAC Manager installs a validating webhook for RBACDefinition CRDs. If the webhook pod or service becomes unhealthy, kubectl apply -f rbacdefiniton.yaml starts timing out or returning 500 errors — blocking all RBAC policy changes.

Add a TCP monitor to verify the webhook service port is reachable:

  1. In Vigilmon, go to Monitors → New Monitor → TCP Port
  2. Host: your node hostname or webhook service IP
  3. Port: 9443 (default webhook port for RBAC Manager)
  4. Name: rbac-manager webhook
  5. Check interval: 1 minute

Verify the webhook is registered:

kubectl get validatingwebhookconfigurations | grep rbac
kubectl describe validatingwebhookconfiguration rbac-manager

Step 6: Monitor leader election health

In HA RBAC Manager deployments (multiple replicas), only the leader actively reconciles. A leader election failure leaves all replicas in standby with no active reconciler.

Check the lease object to verify leader election is functioning:

# Check the leader lease
kubectl get lease rbac-manager -n rbac-manager -o yaml

# The holderIdentity field should contain an active pod name
kubectl get lease rbac-manager -n rbac-manager -o \
  jsonpath='{.spec.holderIdentity}'

Add a heartbeat CronJob that verifies the lease holder is a running pod:

LEADER=$(kubectl get lease rbac-manager -n rbac-manager \
  -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
kubectl get pod "$LEADER" -n rbac-manager --no-headers | grep -q Running

Step 7: Configure alert channels

When RBAC Manager fails, the impact is diffuse — new users don't get access, new namespaces miss policies, service accounts lack permissions. Route alerts to your security and platform engineering on-call channels, not just generic ops alerts.

Slack alert for operator health

  1. In Vigilmon, go to Alert Channels → New Channel → Webhook
  2. Paste your Slack #platform-security webhook URL
  3. Assign this channel to:
    • rbac-manager operator health (HTTP monitor)
    • rbac-manager webhook (TCP monitor)
    • rbac-reconciliation-check (heartbeat monitor)

PagerDuty escalation

For production clusters where RBAC drift is a compliance incident, add a PagerDuty channel:

  1. In Vigilmon, go to Alert Channels → New Channel → Webhook
  2. Enter your PagerDuty Events API v2 endpoint
  3. Assign to the rbac-manager operator health monitor only — this is your highest-severity signal

Step 8: Create a status page for RBAC infrastructure

  1. In Vigilmon, go to Status Pages → New Status Page
  2. Name: "RBAC Infrastructure"
  3. Add monitors:
    • rbac-manager operator health
    • rbac-manager webhook
    • rbac-reconciliation-check (heartbeat)
  4. Share with your security team

Summary

| Monitor | Type | What it catches | |---|---|---| | rbac-manager /healthz | HTTP | Operator pod crash, CrashLoopBackOff | | rbac-manager webhook :9443 | TCP | Admission webhook unreachable | | rbac-reconciliation-check | Heartbeat | Reconciliation errors, stuck definitions | | rbac-rolebinding-drift | Heartbeat | RoleBinding count drops, drift from definitions | | rbac-leader-election | Heartbeat | No active leader in HA deployment |

RBAC Manager is a security-critical component. Silent failures here don't cause pod crashes — they cause access policy drift that may go undetected for hours or days. Get Vigilmon watching the operator health surfaces and you'll catch problems at the control plane level before they surface as access denied errors or, worse, unexpected access grants.

Get started free at vigilmon.online — no credit card, first monitor running in under two minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles