Monitoring WarpStream with Vigilmon

WarpStream is an open source, Kafka-compatible message streaming system with a fundamental architectural twist: there is no local broker disk. Instead of persisting messages to SSD like Apache Kafka, WarpStream agents are completely stateless and write all message data directly to S3-compatible object storage in real time. This eliminates inter-broker replication, makes agents instantly restartable, and cuts storage costs by 80%+ compared to EBS-backed Kafka brokers. But the architecture also introduces new failure modes: S3 connectivity is now the critical path for both producers and consumers, and end-to-end latency includes S3 round-trip overhead. Vigilmon gives you visibility into every layer of the WarpStream stack — agent health, S3 write and read paths, consumer lag, and end-to-end latency — so you catch failures before they interrupt your streaming pipelines.

What You'll Set Up

• WarpStream agent pod health and Kafka API port monitor
• S3 write health via agent metrics or heartbeat
• S3 read health and consumer fetch latency monitor
• Producer throughput monitor
• Consumer group lag monitor
• WarpStream BYOC cluster control plane connectivity monitor
• Topic metadata health check
• Agent scaling health monitor
• End-to-end produce-to-consume latency heartbeat
• Alert channels with appropriate thresholds

Prerequisites

• WarpStream agents deployed (Docker, Kubernetes, or bare metal)
• WarpStream agent admin HTTP port accessible (default 8080)
• S3-compatible object storage configured (AWS S3, MinIO, GCS)
• Kafka client connectivity to WarpStream agents on port 9092
• A free Vigilmon account

Step 1: Monitor WarpStream Agent Health

WarpStream agents are stateless — they can restart instantly without data loss because all data is in S3. But an agent pod crash still means Kafka API requests to that agent fail until it restarts. Monitor the agent HTTP admin port to catch crashes early.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: http://warpstream-agent.yourdomain.com:8080/health (or the agent's load-balancer address).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

For Kubernetes deployments, monitor the service endpoint:

http://warpstream-agent.warpstream.svc.cluster.local:8080/health

Because agents are stateless, a brief restart is self-healing — set Consecutive failures before alert to 2 to absorb the restart window before paging on-call.

Step 2: Monitor the Kafka API Port

WarpStream is Kafka protocol-compatible. Producers and consumers connect on the standard Kafka port. Monitor TCP reachability to confirm the Kafka listener is up, independent of the admin HTTP port.

1. Click Add Monitor → TCP Port.
2. Host: your WarpStream agent host or load balancer.
3. Port: 9092 (standard Kafka port; adjust if you've configured a different listener port).
4. Check interval: 1 minute.
5. Click Save.

A Kafka API port failure with a healthy admin port (/health) suggests a listener configuration issue rather than an agent crash.

Step 3: Monitor S3 Write Health

S3 write health is the most critical path in WarpStream. If agents cannot write to S3, producer produce() calls block and data is lost. Monitor S3 write success directly.

Option A — Monitor MinIO (self-hosted S3-compatible):

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://minio.yourdomain.com:9000/minio/health/live.
3. Expected HTTP status: 200.
4. Check interval: 1 minute.
5. Click Save.

Option B — Monitor via WarpStream agent metrics:

WarpStream agents expose Prometheus metrics. Add a heartbeat that verifies S3 writes are succeeding:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. Deploy a sidecar checker:

#!/bin/bash
# s3-write-check.sh — verify S3 write success rate via Prometheus metrics
METRICS=$(curl -sf http://warpstream-agent:8080/metrics)
# Check that s3_put_object_errors_total is not increasing
ERRORS=$(echo "$METRICS" | grep 'warpstream_s3_put_errors_total' | awk '{print $2}')
TOTAL=$(echo "$METRICS" | grep 'warpstream_s3_put_total' | awk '{print $2}')

if [ -n "$TOTAL" ] && [ "$TOTAL" -gt 0 ]; then
  ERROR_RATE=$(echo "scale=4; $ERRORS / $TOTAL" | bc)
  # Only ping heartbeat if error rate is below 1%
  THRESHOLD=$(echo "$ERROR_RATE < 0.01" | bc)
  if [ "$THRESHOLD" -eq 1 ]; then
    curl -sf https://vigilmon.online/heartbeat/abc123
  fi
fi

If the heartbeat stops arriving, S3 write health has degraded below threshold.

Step 4: Monitor S3 Read Health and Consumer Fetch Latency

Consumers read messages from S3 via WarpStream agents. High S3 read latency directly increases the time consumers wait for messages. Monitor the agent's fetch path health.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://warpstream-agent.yourdomain.com:8080/metrics.
3. Expected HTTP status: 200.
4. Under Keyword check, enter warpstream_s3_get to verify the agent is reporting S3 read metrics (confirming the metrics pipeline is healthy).
5. Check interval: 2 minutes.
6. Click Save.

For a deeper latency check, add a Prometheus alert on P99 fetch latency:

groups:
  - name: warpstream
    rules:
      - alert: WarpStreamHighConsumerFetchLatency
        expr: histogram_quantile(0.99, rate(warpstream_s3_get_duration_seconds_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "WarpStream S3 read P99 latency >500ms"

Step 5: Monitor Producer Throughput via Heartbeat

A sustained drop in producer message rate — bytes or messages written per second — is an early signal of S3 throughput saturation or agent capacity issues.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. Deploy a producer throughput checker that pings the heartbeat only when throughput is above your baseline:

#!/usr/bin/env python3
# producer-throughput-check.py
import requests

METRICS_URL = "http://warpstream-agent:8080/metrics"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/abc123"
MIN_MSGS_PER_SEC = 500  # your baseline

resp = requests.get(METRICS_URL, timeout=10)
lines = resp.text.splitlines()

for line in lines:
    if line.startswith("warpstream_produce_records_total"):
        # Compare with last-known value via a state file
        # Simplified: just verify metric exists and is non-zero
        value = float(line.split()[-1])
        if value > 0:
            requests.get(HEARTBEAT_URL, timeout=5)
        break

Step 6: Monitor Consumer Group Lag

Consumer lag is the number of unread messages per consumer group per topic. Lag growing beyond your SLA means consumers are falling behind production. Monitor Kafka consumer lag via a lag exporter.

1. Deploy kminion or kafka-lag-exporter pointed at your WarpStream cluster's Kafka endpoint.
2. Click Add Monitor → HTTP / HTTPS.
3. URL: http://kminion.monitoring.svc.cluster.local:8080/metrics.
4. Expected HTTP status: 200.
5. Under Keyword check, enter kminion_kafka_consumer_group_topic_partition_lag to verify lag metrics are being reported.
6. Check interval: 2 minutes.
7. Click Save.

Add a Prometheus alerting rule for lag threshold breaches:

- alert: WarpStreamConsumerLagHigh
  expr: kminion_kafka_consumer_group_topic_partition_lag > 10000
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "WarpStream consumer lag >10k messages — consumers falling behind"

Step 7: Monitor BYOC Cluster Control Plane Connectivity

If you're deploying WarpStream BYOC (Bring Your Own Cloud), your agents must maintain connectivity to the WarpStream control plane for cluster registration and metadata. A connectivity loss means agents can't register new partitions or serve metadata requests.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://warpstream-agent.yourdomain.com:8080/health — in BYOC mode, the health endpoint also reflects control plane connectivity status.
3. Under Keyword check, enter ok or healthy to verify the full health response.
4. Check interval: 1 minute.
5. Click Save.

For a direct control plane ping, if your WarpStream version exposes a connectivity status endpoint:

http://warpstream-agent:8080/api/v1/cluster/status

Monitor this endpoint and alert on any non-200 response or missing connected keyword.

Step 8: Monitor Topic Metadata Health

WarpStream stores topic and partition metadata. Metadata fetch failures cause producers and consumers to fail at connection time — before any messages are produced or consumed.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://warpstream-agent.yourdomain.com:8080/metrics.
3. Under Keyword check, enter warpstream_metadata to verify metadata metrics are being tracked.
4. Check interval: 2 minutes.
5. Click Save.

Metadata failures combined with healthy S3 and agent metrics typically indicate a control plane or registry issue rather than a storage problem.

Step 9: Monitor End-to-End Produce-to-Consume Latency

WarpStream's S3-backed architecture introduces additional latency compared to local-disk Kafka. Monitor end-to-end latency to catch S3 throughput saturation before it breaches your SLA.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. Deploy an E2E latency probe that produces a test message and measures how long it takes to appear on the consumer side:

#!/usr/bin/env python3
# e2e-latency-probe.py — run every 5 minutes
import time
import requests
from kafka import KafkaProducer, KafkaConsumer

WARPSTREAM_BOOTSTRAP = "warpstream-agent:9092"
TEST_TOPIC = "_vigilmon_e2e_probe"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/abc123"
P99_THRESHOLD_MS = 500

producer = KafkaProducer(bootstrap_servers=WARPSTREAM_BOOTSTRAP)
consumer = KafkaConsumer(TEST_TOPIC, bootstrap_servers=WARPSTREAM_BOOTSTRAP,
                         auto_offset_reset='latest', consumer_timeout_ms=5000)

send_time = time.time()
producer.send(TEST_TOPIC, b"probe")
producer.flush()

for msg in consumer:
    latency_ms = (time.time() - send_time) * 1000
    if latency_ms < P99_THRESHOLD_MS:
        requests.get(HEARTBEAT_URL, timeout=5)
    break

producer.close()
consumer.close()

If the heartbeat stops arriving, E2E latency has exceeded 500ms — or the S3 path is fully broken.

Step 10: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. For agent health and Kafka API port, set Consecutive failures before alert to 2 — stateless restarts are fast.
3. For S3 write health heartbeat, set to 1 — S3 write failure is immediately critical (data loss risk).
4. For consumer lag and throughput checks, set to 1 — lag compounds exponentially.
5. For BYOC control plane and metadata health, set to 2.
6. For E2E latency heartbeat, Vigilmon alerts automatically after the interval window passes.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Agent health | :8080/health | Agent crash, pod restart loop | | Kafka API port | :9092 TCP | Kafka listener down | | S3 write health | MinIO health or heartbeat | S3 connectivity loss, produce blocked | | S3 read metrics | :8080/metrics keyword | Consumer fetch degradation | | Producer throughput | Heartbeat URL | Throughput drop from baseline | | Consumer group lag | kminion metrics | Consumers falling behind SLA | | BYOC connectivity | :8080/health keyword ok | Control plane disconnect | | Topic metadata | :8080/metrics keyword | Metadata fetch failures | | E2E latency | Heartbeat URL | P99 >500ms, S3 round-trip bottleneck |

WarpStream's stateless, S3-native architecture makes broker nodes trivially restartable — but it moves the critical dependency from local disk to object storage. With Vigilmon covering agent health, the Kafka API port, S3 write and read paths, consumer lag, and end-to-end latency, you maintain full observability over the components that actually matter in a WarpStream deployment.