Monitoring Karma Alertmanager Dashboard with Vigilmon

Karma is the multi-cluster Alertmanager dashboard that gives platform teams a single pane of glass across every Prometheus/Alertmanager instance in their environment. It's the tool that tells you your production cluster is on fire — which creates an obvious problem: if Karma itself goes down, or if it silently loses connectivity to one of your Alertmanager instances, your on-call team stops seeing alerts from that cluster entirely. Vigilmon monitors Karma's own availability, Alertmanager connectivity per cluster, alert refresh lag, and memory pressure, so Karma's health is never a blind spot.

What You'll Set Up

• Karma server health endpoint monitor
• Alertmanager connectivity check per configured cluster
• Alert refresh lag monitor (polling freshness)
• Total firing alert count trend monitor
• Critical alert count with acknowledgment window
• Karma memory usage monitor
• Connected cluster count vs. configured cluster count

Prerequisites

• Karma running as a Docker container or binary (port 8080 by default)
• At least one Alertmanager instance configured in Karma's config
• Access to Karma's /health endpoint and Prometheus metrics endpoint
• A free Vigilmon account

Step 1: Monitor Karma Server Health

Karma exposes a /health endpoint that returns Healthy when the server is up and connected to at least one Alertmanager.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Karma URL: http://karma.internal:8080/health.
4. Under Keyword check, add: Healthy.
5. Set Check interval to 1 minute.
6. Set Alert after 2 consecutive failures.
7. Click Save.

If Karma is behind an auth proxy (OAuth2, basic auth), probe the /health endpoint directly by IP if possible, or configure a service account token. The health endpoint is intentionally unauthenticated in most Karma deployments for this purpose.

Step 2: Monitor Alertmanager Connectivity Per Cluster

Karma polls each configured Alertmanager on a schedule. If a cluster's Alertmanager becomes unreachable, Karma silently stops showing alerts from that cluster — exactly when you need them most.

Karma exposes per-upstream status in its /api/v1/status endpoint:

curl -s http://karma.internal:8080/api/v1/status | jq '.clusters'
# {
#   "production": {"name": "production", "members": ["https://alertmanager-prod/"]},
#   "staging":    {"name": "staging",    "members": ["https://alertmanager-staging/"]}
# }

Create a connectivity check script:

cat > /usr/local/bin/check-karma-clusters.sh << 'EOF'
#!/bin/bash
KARMA_URL=${KARMA_URL:-http://localhost:8080}
EXPECTED_CLUSTERS=${EXPECTED_CLUSTERS:-2}

# Get current cluster count from Karma API
CLUSTER_COUNT=$(curl -s "${KARMA_URL}/api/v1/status" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('clusters',{})))" 2>/dev/null)

CLUSTER_COUNT=${CLUSTER_COUNT:-0}

if [ "$CLUSTER_COUNT" -lt "$EXPECTED_CLUSTERS" ]; then
  echo "CRITICAL: only ${CLUSTER_COUNT}/${EXPECTED_CLUSTERS} clusters connected"
  exit 1
fi
echo "OK: ${CLUSTER_COUNT} clusters connected"
exit 0
EOF
chmod +x /usr/local/bin/check-karma-clusters.sh

Export this as an HTTP endpoint and add a Vigilmon Keyword check for OK:.

Step 3: Monitor Alert Refresh Lag

Karma polls each Alertmanager instance on a configurable interval (default 60 seconds). If polling falls behind — due to network issues, Alertmanager slowness, or Karma resource pressure — the dashboard goes stale.

Check the last successful poll timestamp from Karma's metrics:

# Karma exposes Prometheus metrics at /metrics
curl -s http://karma.internal:8080/metrics | grep karma_collect_duration_seconds
# karma_collect_duration_seconds_bucket{source="https://alertmanager-prod/",le="0.1"} 42
# karma_collect_duration_seconds_count{source="https://alertmanager-prod/"} 50

For lag detection, check the age of the most recent successful collection:

cat > /usr/local/bin/check-karma-lag.sh << 'EOF'
#!/bin/bash
KARMA_URL=${KARMA_URL:-http://localhost:8080}
LAG_THRESHOLD_SECONDS=${LAG_THRESHOLD:-300}  # 5 minutes

# Check karma_collect_errors_total - if it's increasing, polling is failing
ERRORS=$(curl -s "${KARMA_URL}/metrics" 2>/dev/null \
  | grep 'karma_collect_errors_total' | awk '{print $2}' | head -1)

ERRORS=${ERRORS:-0}
# Compare against a baseline stored from the previous run
ERRFILE=/tmp/karma-errors-baseline
if [ -f "$ERRFILE" ]; then
  PREV=$(cat "$ERRFILE")
  if [ "$ERRORS" -gt "$PREV" ]; then
    echo "WARNING: Alertmanager poll error count increased (${PREV} -> ${ERRORS})"
    echo "$ERRORS" > "$ERRFILE"
    exit 1
  fi
fi
echo "$ERRORS" > "$ERRFILE"
echo "OK: poll error count stable at ${ERRORS}"
exit 0
EOF
chmod +x /usr/local/bin/check-karma-lag.sh

Step 4: Monitor Total Firing Alert Count

A sudden spike in firing alerts across all clusters indicates a systemic incident. Karma's API provides the current alert counts.

cat > /usr/local/bin/check-karma-alertcount.sh << 'EOF'
#!/bin/bash
KARMA_URL=${KARMA_URL:-http://localhost:8080}
SPIKE_THRESHOLD=${SPIKE_THRESHOLD:-50}

ALERT_COUNT=$(curl -s "${KARMA_URL}/api/v1/alerts.json" 2>/dev/null \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
total = sum(len(g.get('alerts', [])) for g in data.get('groups', []))
print(total)
" 2>/dev/null)

ALERT_COUNT=${ALERT_COUNT:-0}

if [ "$ALERT_COUNT" -gt "$SPIKE_THRESHOLD" ]; then
  echo "WARNING: ${ALERT_COUNT} active alerts exceeds threshold ${SPIKE_THRESHOLD}"
  exit 1
fi
echo "OK: ${ALERT_COUNT} active alerts"
exit 0
EOF
chmod +x /usr/local/bin/check-karma-alertcount.sh

Tune SPIKE_THRESHOLD to 3x your typical baseline alert count. This catches alert storms, not normal steady-state firing.

Step 5: Monitor Critical Alerts Requiring Acknowledgment

Critical-severity alerts that fire for more than 15 minutes without a silence or acknowledgment need immediate escalation.

cat > /usr/local/bin/check-karma-critical.sh << 'EOF'
#!/bin/bash
KARMA_URL=${KARMA_URL:-http://localhost:8080}
AGE_THRESHOLD_MINUTES=${AGE_THRESHOLD:-15}

# Find critical alerts older than threshold with no silence
UNACKED=$(curl -s "${KARMA_URL}/api/v1/alerts.json" 2>/dev/null \
  | python3 -c "
import sys, json
from datetime import datetime, timezone, timedelta
data = json.load(sys.stdin)
threshold = datetime.now(timezone.utc) - timedelta(minutes=${AGE_THRESHOLD_MINUTES})
count = 0
for group in data.get('groups', []):
    for alert in group.get('alerts', []):
        if alert.get('labels', {}).get('severity') == 'critical':
            if not alert.get('silences'):
                start_str = alert.get('startsAt', '')
                try:
                    start = datetime.fromisoformat(start_str.replace('Z', '+00:00'))
                    if start < threshold:
                        count += 1
                except Exception:
                    pass
print(count)
" 2>/dev/null)

UNACKED=${UNACKED:-0}
if [ "$UNACKED" -gt 0 ]; then
  echo "WARNING: ${UNACKED} unacknowledged critical alert(s) >15 minutes old"
  exit 1
fi
echo "OK: no unacknowledged critical alerts"
exit 0
EOF
chmod +x /usr/local/bin/check-karma-critical.sh

Step 6: Monitor Karma Memory Usage

Alert storms can cause Karma's in-memory alert aggregation to consume significant RAM, eventually causing OOM kills.

cat > /usr/local/bin/check-karma-memory.sh << 'EOF'
#!/bin/bash
THRESHOLD_MB=${MEMORY_THRESHOLD_MB:-512}

KARMA_PID=$(pgrep -f karma 2>/dev/null | head -1)
if [ -z "$KARMA_PID" ]; then
  echo "CRITICAL: karma process not found"
  exit 1
fi

RSS_KB=$(awk '/VmRSS/{print $2}' /proc/$KARMA_PID/status 2>/dev/null)
RSS_MB=$(( ${RSS_KB:-0} / 1024 ))

if [ "$RSS_MB" -gt "$THRESHOLD_MB" ]; then
  echo "WARNING: Karma memory ${RSS_MB}MB exceeds ${THRESHOLD_MB}MB threshold"
  exit 1
fi
echo "OK: Karma memory ${RSS_MB}MB"
exit 0
EOF
chmod +x /usr/local/bin/check-karma-memory.sh

Step 7: Set Up the Monitoring HTTP Exporter

Expose all check scripts through a unified HTTP exporter so Vigilmon can probe them:

cat > /usr/local/bin/karma-health-server.sh << 'EOF'
#!/bin/bash
while true; do
  {
    # Run all checks and aggregate
    CLUSTER=$(/usr/local/bin/check-karma-clusters.sh 2>&1)
    LAG=$(/usr/local/bin/check-karma-lag.sh 2>&1)
    MEM=$(/usr/local/bin/check-karma-memory.sh 2>&1)
    CRITICAL=$(/usr/local/bin/check-karma-critical.sh 2>&1)

    if echo "$CLUSTER$LAG$MEM$CRITICAL" | grep -q "CRITICAL\|WARNING"; then
      STATUS="DEGRADED"
    else
      STATUS="OK"
    fi

    BODY=$(printf '{"status":"%s","clusters":"%s","lag":"%s","memory":"%s","critical_alerts":"%s"}' \
      "$STATUS" "$CLUSTER" "$LAG" "$MEM" "$CRITICAL")
    printf "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n\r\n%s" "$BODY"
  } | nc -l -p 9960 -q 1
done
EOF
chmod +x /usr/local/bin/karma-health-server.sh

Add a Vigilmon monitor with a Keyword check for "status":"OK".

Step 8: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your team's notification channel.
2. Set alert thresholds:
   • Karma /health failure: alert after 2 failures — a brief restart shouldn't page.
   • Cluster count drop: alert after 1 failure — losing a cluster is always urgent.
   • Critical unacked alerts: alert after 1 check — these need immediate human attention.
   • Memory spike: alert after 3 checks — gives garbage collection time to run.
3. Create a maintenance window during Karma upgrades or Alertmanager migrations to suppress alert storms during intentional changes.

Summary

| Monitor | Target | Alert Condition | |---|---|---| | Karma health | /health keyword Healthy | Server unavailable | | Cluster connectivity | API cluster count vs. expected | Any cluster unreachable | | Poll lag | Error count delta | Poll errors increasing | | Alert count spike | Total alerts vs. baseline | > 3x baseline active alerts | | Unacked critical alerts | Alerts age > 15 min | Any unacked critical | | Karma memory | Process RSS vs. threshold | Memory > 512 MB |

Karma's value proposition is that nothing slips through unnoticed across your multi-cluster Prometheus estate. The irony is that Karma itself can fail silently. With Vigilmon watching Karma's health endpoint, per-cluster connectivity, and alert freshness, you close the blind spot in your monitoring platform's own observability.