Monitoring Steadybit Chaos Engineering Platform with Vigilmon

Steadybit is an open source chaos engineering platform that helps engineering teams discover system weaknesses before production incidents do. By deliberately injecting failures — network latency, container kills, CPU stress, disk fills — and observing system behavior, teams build confidence in their resilience architecture. But the chaos engineering toolchain itself needs monitoring: if the Steadybit hub goes down, experiments cannot be managed; if chaos agents lose connectivity, experiments cannot execute; if target discovery fails, the wrong (or zero) targets get attacked. Vigilmon gives you external monitoring of the Steadybit platform so your chaos engineering practice stays operational — and you know immediately when the tool that validates your system's resilience is itself failing.

What You'll Set Up

HTTP uptime monitor for the Steadybit hub
Agent connectivity heartbeat
Experiment execution success rate heartbeat
Attack extension registration heartbeat
Target discovery health heartbeat
Hub database connectivity monitor
Alert channels for the chaos engineering management plane

Prerequisites

Steadybit platform (hub) deployed on Kubernetes or a Linux host
Steadybit agents deployed on target infrastructure (Kubernetes nodes, Linux hosts, Docker environments)
Hub web interface accessible over HTTP/HTTPS
A free Vigilmon account

Step 1: Monitor the Steadybit Hub

The Steadybit hub is the central management plane for all chaos experiments. If it goes down, your team cannot create, schedule, or monitor experiments — and any running experiments may behave unexpectedly without hub coordination.

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
URL: https://steadybit.yourdomain.com/ (your Steadybit hub URL).
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Under Keyword check, enter Steadybit to verify the hub UI loads, not just that the reverse proxy responds.
Enable Monitor SSL certificate and set the expiry alert threshold to 21 days.
Click Save.

Also monitor the Steadybit health API endpoint:

Click Add Monitor → HTTP / HTTPS.
URL: https://steadybit.yourdomain.com/api/health (or /health depending on your deployment).
Check interval: 1 minute
Expected HTTP status: 200
Click Save.

Step 2: Monitor Agent Connectivity via Heartbeat

Steadybit agents are the execution layer — they run on target hosts and carry out chaos attacks. If agents disconnect from the hub, experiments cannot execute even though the hub appears healthy. Monitor agent connectivity by querying the Steadybit API and emitting a heartbeat when the expected agent count is online.

Click Add Monitor → Cron Heartbeat in Vigilmon.
Set the expected interval to 5 minutes.
Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
Create a script that checks agent count via the Steadybit API:

#!/bin/bash
# /usr/local/bin/check-steadybit-agents.sh
STEADYBIT_URL="https://steadybit.yourdomain.com"
API_TOKEN="your-steadybit-api-token"
EXPECTED_AGENTS=3  # adjust to match your deployment

AGENTS=$(curl -sf \
  -H "Authorization: accessToken $API_TOKEN" \
  "$STEADYBIT_URL/api/agents" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(len([a for a in d.get('content',[]) if a.get('connected')]))" 2>/dev/null)

if [ -n "$AGENTS" ] && [ "$AGENTS" -ge "$EXPECTED_AGENTS" ]; then
  curl -sf "https://vigilmon.online/heartbeat/abc123" > /dev/null
fi

Add to crontab:

*/5 * * * * /usr/local/bin/check-steadybit-agents.sh

When any agent disconnects and the connected count drops below your expected minimum, the heartbeat stops and Vigilmon alerts you — before a scheduled experiment silently fails to execute because no agent is available to carry out the attacks.

Step 3: Monitor Experiment Execution Success Rate

Chaos experiments that fail to execute (as opposed to deliberately finding weaknesses) indicate platform problems — not system weaknesses. Monitor experiment execution success by checking recent experiment outcomes via the Steadybit API.

Click Add Monitor → Cron Heartbeat.
Set the expected interval to 30 minutes.
Copy the heartbeat URL.
Create a script that checks recent experiment results:

#!/bin/bash
# /usr/local/bin/check-steadybit-experiments.sh
STEADYBIT_URL="https://steadybit.yourdomain.com"
API_TOKEN="your-steadybit-api-token"

# Get experiments run in the last hour
RESULT=$(curl -sf \
  -H "Authorization: accessToken $API_TOKEN" \
  "$STEADYBIT_URL/api/experiment-runs?pageSize=20&sort=created,desc" \
  2>/dev/null)

if [ -z "$RESULT" ]; then
  echo "API unavailable" >&2
  exit 1
fi

# Count failed vs total runs (FAILED = platform/execution failure, not experiment finding)
TOTAL=$(echo "$RESULT" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('content',[])))")
FAILED=$(echo "$RESULT" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len([r for r in d.get('content',[]) if r.get('state')=='FAILED']))")

# Alert threshold: >10% failure rate
if [ "$TOTAL" -gt 0 ]; then
  FAIL_PCT=$((FAILED * 100 / TOTAL))
  if [ "$FAIL_PCT" -le 10 ]; then
    curl -sf "https://vigilmon.online/heartbeat/your-experiments-heartbeat-id" > /dev/null
  fi
else
  # No recent experiments — still healthy, ping to confirm API is accessible
  curl -sf "https://vigilmon.online/heartbeat/your-experiments-heartbeat-id" > /dev/null
fi

*/30 * * * * /usr/local/bin/check-steadybit-experiments.sh

Step 4: Monitor Attack Extension Registration

Steadybit uses an extension model where attack types (network chaos, container attacks, AWS API chaos) are implemented as plugins that register with the hub. If an extension fails to register, that category of attack becomes silently unavailable.

Click Add Monitor → Cron Heartbeat.
Set the expected interval to 10 minutes.
Copy the heartbeat URL.
Create a script that checks extension registration:

#!/bin/bash
# /usr/local/bin/check-steadybit-extensions.sh
STEADYBIT_URL="https://steadybit.yourdomain.com"
API_TOKEN="your-steadybit-api-token"
EXPECTED_EXTENSIONS=5  # adjust for your installed extensions

EXTENSIONS=$(curl -sf \
  -H "Authorization: accessToken $API_TOKEN" \
  "$STEADYBIT_URL/api/extensions" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(len([e for e in d.get('content',[]) if e.get('state')=='REGISTERED']))" 2>/dev/null)

if [ -n "$EXTENSIONS" ] && [ "$EXTENSIONS" -ge "$EXPECTED_EXTENSIONS" ]; then
  curl -sf "https://vigilmon.online/heartbeat/your-extensions-heartbeat-id" > /dev/null
fi

*/10 * * * * /usr/local/bin/check-steadybit-extensions.sh

Step 5: Monitor Target Discovery Health

Steadybit continuously discovers targets from your infrastructure — Kubernetes pods, Linux hosts, Docker containers, AWS resources. If target discovery fails, experiments may run against stale target lists (attacking decommissioned pods) or zero targets (running no attacks while reporting success).

Click Add Monitor → Cron Heartbeat.
Set the expected interval to 15 minutes.
Copy the heartbeat URL.
Create a script that checks target counts per type:

#!/bin/bash
# /usr/local/bin/check-steadybit-targets.sh
STEADYBIT_URL="https://steadybit.yourdomain.com"
API_TOKEN="your-steadybit-api-token"
MIN_CONTAINERS=5  # set to your minimum expected container count

CONTAINERS=$(curl -sf \
  -H "Authorization: accessToken $API_TOKEN" \
  "$STEADYBIT_URL/api/targets?targetType=com.steadybit.extension_container.container&pageSize=1" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('totalElements', 0))" 2>/dev/null)

if [ -n "$CONTAINERS" ] && [ "$CONTAINERS" -ge "$MIN_CONTAINERS" ]; then
  curl -sf "https://vigilmon.online/heartbeat/your-targets-heartbeat-id" > /dev/null
fi

*/15 * * * * /usr/local/bin/check-steadybit-targets.sh

A target count that drops to zero for any critical target type almost always indicates a discovery failure — not that your entire fleet has vanished.

Step 6: Monitor Hub Database Health

Steadybit stores all experiment definitions, execution history, and scheduling data in a relational database. Database loss prevents experiment scheduling, blocks results retrieval, and can cause data loss for in-progress experiments.

For PostgreSQL (default Steadybit configuration):

Click Add Monitor → TCP Port.
Host: localhost (or your DB host).
Port: 5432
Check interval: 1 minute
Click Save.

Combine this with the hub health monitor from Step 1 — if the hub health endpoint fails at the same time the DB TCP port goes down, database failure is the root cause.

Step 7: Monitor Scheduled Experiment Adherence

Scheduled chaos experiments are a core part of a mature chaos engineering practice — running the same failure scenarios repeatedly builds confidence that resilience controls hold over time. Monitor whether scheduled experiments are actually executing on schedule.

Click Add Monitor → Cron Heartbeat.
Set the expected interval to match your slowest recurring experiment schedule (e.g., 1440 minutes for daily experiments, 60 minutes for hourly).
Copy the heartbeat URL.
Trigger the heartbeat from your experiment's post-run webhook or via the Steadybit scheduling notification API:

# In a post-experiment webhook handler (your webhook receiver script):
if [ "$EXPERIMENT_STATE" = "COMPLETED" ] || [ "$EXPERIMENT_STATE" = "COMPLETED_WITH_ERRORS" ]; then
  curl -sf "https://vigilmon.online/heartbeat/your-schedule-heartbeat-id" > /dev/null
fi

Alternatively, query for the most recent run of a specific scheduled experiment:

#!/bin/bash
# /usr/local/bin/check-scheduled-experiments.sh
STEADYBIT_URL="https://steadybit.yourdomain.com"
API_TOKEN="your-steadybit-api-token"
EXPERIMENT_KEY="ADV-1"  # your scheduled experiment key

LAST_RUN=$(curl -sf \
  -H "Authorization: accessToken $API_TOKEN" \
  "$STEADYBIT_URL/api/experiment-runs?experimentKey=$EXPERIMENT_KEY&pageSize=1&sort=created,desc" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); runs=d.get('content',[]); print(runs[0]['created'] if runs else '')" 2>/dev/null)

if [ -n "$LAST_RUN" ]; then
  curl -sf "https://vigilmon.online/heartbeat/your-schedule-heartbeat-id" > /dev/null
fi

Step 8: Configure Alert Channels and Thresholds

Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
For the Steadybit hub HTTP monitor, set Consecutive failures before alert to 2 — hub restarts cause brief probe gaps.
For the hub health API monitor, set to 2 for the same reason.
For the database TCP monitor, set to 1 — database loss is an immediate, complete failure of the scheduling and history layer.
For all cron heartbeats (agent connectivity, experiment execution, target discovery, extensions), leave at the default heartbeat expiry window — Vigilmon alerts when the expected ping window expires.
Route agent connectivity alerts to the infrastructure team that manages agent deployments — these indicate host-level or network connectivity failures, not chaos engineering issues.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Hub UI | https://steadybit.domain.com/ | Hub application failure | | Hub health API | /api/health | Hub internal health failure | | Agent connectivity heartbeat | Steadybit agents API | Agent disconnection from hub | | Experiment success heartbeat | Experiment runs API | Platform-level execution failures | | Extension registration heartbeat | Extensions API | Attack type becoming unavailable | | Target discovery heartbeat | Targets API | Discovery failure, stale target list | | PostgreSQL TCP | :5432 | Hub database connectivity loss | | Schedule adherence heartbeat | Scheduled experiment API | Missed scheduled chaos runs |

Steadybit is the tool you rely on to find weaknesses before they become incidents — which means Steadybit itself needs to be monitored. With Vigilmon covering the hub, agent connectivity, experiment execution, attack extensions, target discovery, and the database, you know immediately when your chaos engineering platform is compromised. A chaos tool that fails silently is worse than no chaos tool at all: it creates false confidence that experiments are running when nothing is actually being tested.