tutorial

Monitoring Feldera with Vigilmon

Feldera is an incremental view maintenance engine built on DBSP theory — here's how to monitor its pipelines, Kafka connectors, output sinks, throughput, view freshness, checkpoints, and REST API with Vigilmon.

Feldera is an open source continuous analytics engine built on DBSP (Database Stream Processing) theory — a mathematical framework that enables any SQL query to be computed incrementally as data changes, including deeply nested aggregations and joins that other streaming SQL engines must materialize as batch. Feldera maintains near-real-time materialized views on event streams, making it valuable for fraud detection analytics, ML feature pipelines, and live customer-facing dashboards. When you self-host Feldera, you're running the Feldera pipeline process (executing SQL programs), Kafka connectors (reading input from Kafka topics), output sinks (writing computed views to Kafka, HTTP endpoints, or Delta Lake), and the REST API and web console. A failed pipeline or stale output view can silently break downstream dashboards or feature pipelines without any visible error. Vigilmon gives you continuous monitoring across every layer of the Feldera stack.

What You'll Set Up

  • Feldera pipeline status health monitor (Running/Failed transitions)
  • Kafka input connector health and consumer lag monitor
  • Output connector write health via heartbeat
  • Incremental computation throughput monitor
  • Output view freshness (staleness) heartbeat
  • Feldera REST API health monitor
  • Checkpoint health via heartbeat
  • Worker thread health monitor
  • Memory usage alerting
  • Feldera web console availability monitor
  • Alert channels with appropriate thresholds

Prerequisites

  • Feldera deployed (Docker Compose or Kubernetes)
  • Feldera REST API accessible (default port 8080)
  • Feldera web console accessible (default port 8080 under the / path)
  • Kafka broker accessible (if using Kafka input connectors)
  • A free Vigilmon account

Step 1: Monitor the Feldera REST API

The Feldera REST API is the control plane for all pipeline operations — starting, stopping, and monitoring SQL programs. A REST API failure blocks pipeline management and output polling for HTTP-based sinks.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. URL: http://feldera.yourdomain.com:8080/healthz (or the Feldera health endpoint in your deployment).
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Click Save.

If Feldera doesn't expose a dedicated /healthz route, use the pipelines list endpoint:

http://feldera.yourdomain.com:8080/v0/pipelines

A 200 response confirms the API is up. A timeout or 5xx means pipeline management and output polling are unavailable.

Step 2: Monitor Pipeline Status

Feldera pipelines transition between Running, Paused, and Failed states. A Failed pipeline is the most critical alert — it means incremental computation has halted and your materialized views are no longer being updated.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://feldera.yourdomain.com:8080/v0/pipelines.
  3. Expected HTTP status: 200.
  4. Under Keyword check, enter Running to verify at least one pipeline is in the running state.
  5. Check interval: 1 minute.
  6. Click Save.

For per-pipeline monitoring, use the individual pipeline status endpoint:

http://feldera.yourdomain.com:8080/v0/pipelines/{pipeline-name}

Monitor this endpoint for each production pipeline and keyword-check for "state":"Running" to confirm the pipeline has not transitioned to Failed or Paused.

Step 3: Monitor Kafka Input Connector Health and Consumer Lag

Feldera reads input data from Kafka topics via its Kafka connector. Consumer lag growing beyond your threshold means Feldera is falling behind the event stream — your materialized views will become stale.

Monitor Kafka connectivity:

  1. Click Add Monitor → TCP Port.
  2. Host: your Kafka broker host.
  3. Port: 9092 (standard Kafka port).
  4. Check interval: 1 minute.
  5. Click Save.

Monitor consumer lag via a lag exporter:

  1. Deploy kminion or kafka-lag-exporter pointed at your Kafka cluster, configured with the consumer group Feldera uses (typically the pipeline name).
  2. Click Add Monitor → HTTP / HTTPS.
  3. URL: http://kminion.monitoring.svc.cluster.local:8080/metrics.
  4. Expected HTTP status: 200.
  5. Under Keyword check, enter consumer_group_topic_partition_lag to verify lag metrics are being reported.
  6. Check interval: 2 minutes.
  7. Click Save.

Add a Prometheus alert for lag threshold breaches:

groups:
  - name: feldera
    rules:
      - alert: FelderaKafkaConsumerLagHigh
        expr: kminion_kafka_consumer_group_topic_partition_lag{group=~"feldera-.*"} > 50000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Feldera Kafka consumer lag >50k messages — views becoming stale"

Step 4: Monitor Output Connector Write Health

Feldera writes computed view results to output sinks — Kafka topics, HTTP endpoints, or Delta Lake. A write failure means your materialized view results are being computed but not propagated to downstream consumers.

For Kafka output sinks:

  1. Click Add Monitor → TCP Port.
  2. Host: your Kafka output broker host.
  3. Port: 9092.
  4. Check interval: 1 minute.
  5. Click Save.

For HTTP output endpoint sinks:

If Feldera is pushing results to an HTTP endpoint, monitor that endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: your HTTP output sink endpoint.
  3. Expected HTTP status: 200 (or 202 Accepted depending on your endpoint).
  4. Check interval: 2 minutes.
  5. Click Save.

For a generic output health heartbeat:

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 5 minutes.
  3. Copy the heartbeat URL.
  4. Configure your output sink (or a downstream consumer) to ping the heartbeat each time it receives a batch of view results:
# In your output sink consumer (e.g., reading from Kafka output topic)
import requests

def on_batch_received(records):
    if len(records) > 0:
        requests.get("https://vigilmon.online/heartbeat/abc123", timeout=5)

Step 5: Monitor Incremental Computation Throughput via Heartbeat

Feldera's throughput — records processed per second — is a leading indicator of computation health. A drop below baseline means input data is queuing faster than Feldera is processing it, causing view staleness to grow.

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 5 minutes.
  3. Copy the heartbeat URL.
  4. Monitor throughput via the Feldera pipeline stats endpoint and ping the heartbeat only when throughput is healthy:
#!/usr/bin/env python3
# throughput-check.py — run every 5 minutes
import requests

FELDERA_URL = "http://feldera.yourdomain.com:8080"
PIPELINE_NAME = "your-pipeline"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/abc123"
MIN_RECORDS_PER_SEC = 100  # your baseline

resp = requests.get(
    f"{FELDERA_URL}/v0/pipelines/{PIPELINE_NAME}/stats",
    timeout=10
)
stats = resp.json()

# Check throughput from pipeline stats
throughput = stats.get("metrics", {}).get("input_records_per_second", 0)
if throughput >= MIN_RECORDS_PER_SEC:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print(f"Throughput below threshold: {throughput:.1f} rec/s")

Step 6: Monitor Output View Freshness

View freshness — the age of the most recently computed output view result — is the primary SLA metric for Feldera. If your pipeline is supposed to produce near-real-time views but output lag exceeds 30 seconds, downstream consumers are reading stale data.

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to 30 seconds (or your view freshness SLA).
  3. Copy the heartbeat URL.
  4. Configure your Feldera pipeline or an output consumer to ping the heartbeat each time fresh output is produced:
# Option A: In a downstream consumer that reads Feldera output
from kafka import KafkaConsumer
import requests, time

consumer = KafkaConsumer(
    "feldera-output-topic",
    bootstrap_servers="kafka:9092",
    group_id="freshness-monitor"
)

for message in consumer:
    # Received fresh output — ping the heartbeat
    requests.get("https://vigilmon.online/heartbeat/abc123", timeout=5)

# Option B: HTTP output polling — run every 30 seconds via cron
#!/bin/bash
RESULT=$(curl -sf \
  "http://feldera.yourdomain.com:8080/v0/pipelines/your-pipeline/egress/your-view?format=json&mode=snapshot" \
  | jq '.total_records // 0')
if [ "$RESULT" -gt 0 ]; then
  curl -sf https://vigilmon.online/heartbeat/abc123
fi

If Vigilmon stops receiving heartbeats within the interval, it means view freshness has exceeded your SLA — alert immediately.

Step 7: Monitor Checkpoint Health via Heartbeat

Feldera supports fault tolerance via checkpoints — periodic snapshots of the incremental DBSP computation state. A checkpoint failure means that if the pipeline crashes, it will need to replay input data from further back, delaying recovery.

  1. Click Add Monitor → Cron Heartbeat.
  2. Set the expected interval to match your checkpoint frequency (e.g., 10 minutes).
  3. Copy the heartbeat URL.
  4. Add a checkpoint monitor that pings after each successful checkpoint:
#!/usr/bin/env python3
# checkpoint-monitor.py — run every 10 minutes
import requests

FELDERA_URL = "http://feldera.yourdomain.com:8080"
PIPELINE_NAME = "your-pipeline"
HEARTBEAT_URL = "https://vigilmon.online/heartbeat/abc123"

# Check for checkpoint completion via pipeline metrics
resp = requests.get(
    f"{FELDERA_URL}/v0/pipelines/{PIPELINE_NAME}/stats",
    timeout=10
)
stats = resp.json()
checkpoint_ok = stats.get("metrics", {}).get("last_checkpoint_ok", False)

if checkpoint_ok:
    requests.get(HEARTBEAT_URL, timeout=5)
else:
    print("Last checkpoint was not successful")

Step 8: Monitor Worker Thread Health

Feldera uses a multi-threaded DBSP execution engine. A worker thread crash causes the pipeline to stop computing, even if the pipeline process itself remains running.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://feldera.yourdomain.com:8080/v0/pipelines/{your-pipeline}/stats.
  3. Expected HTTP status: 200.
  4. Under Keyword check, enter worker_threads to verify worker thread metrics are being reported.
  5. Check interval: 2 minutes.
  6. Click Save.

Add a Prometheus alert on worker thread count:

- alert: FelderaWorkerThreadDrop
  expr: feldera_worker_threads_active < feldera_worker_threads_configured
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Feldera worker thread count below configured value — computation degraded"

Step 9: Monitor Memory Usage

DBSP incremental state is stored in memory. As Feldera processes more distinct key values (joins, aggregations over high-cardinality keys), memory usage grows. Exceeding available RAM causes OOM kills and pipeline restarts.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://feldera.yourdomain.com:8080/v0/pipelines/{your-pipeline}/stats.
  3. Expected HTTP status: 200.
  4. Under Keyword check, enter memory to verify memory metrics are present.
  5. Check interval: 2 minutes.
  6. Click Save.

Add a Prometheus alert for high memory usage:

- alert: FelderaHighMemoryUsage
  expr: |
    feldera_process_memory_bytes /
    feldera_process_memory_limit_bytes > 0.85
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Feldera memory usage >85% — OOM risk, pipeline may restart"

For Docker deployments, also monitor container memory via cAdvisor:

- alert: FelderaContainerOOMRisk
  expr: container_memory_usage_bytes{container="feldera"} / container_spec_memory_limit_bytes{container="feldera"} > 0.85
  for: 5m
  labels:
    severity: warning

Step 10: Monitor the Feldera Web Console

The Feldera web console is the primary interface for SQL pipeline development, pipeline management, and output inspection. If the console is unavailable, engineers can't develop or debug SQL programs.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: https://feldera.yourdomain.com (or the port-forwarded local URL if not publicly exposed).
  3. Expected HTTP status: 200.
  4. Under Keyword check, enter Feldera to verify the console content loads.
  5. Check interval: 3 minutes.
  6. Enable Monitor SSL certificate with 21 days expiry alert.
  7. Click Save.

Step 11: Configure Alert Channels

  1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
  2. For the REST API and pipeline status, set Consecutive failures before alert to 1 — pipeline failure is immediately actionable.
  3. For Kafka consumer lag and throughput heartbeats, set to 1 — lag compounds quickly in streaming systems.
  4. For view freshness heartbeat, Vigilmon alerts automatically after the interval passes — this is your primary SLA monitor.
  5. For checkpoint and output health heartbeats, set Vigilmon to alert after 1 missed interval.
  6. For memory alerts, set to 2 — brief memory spikes during large batch catch-up are expected.
  7. For the web console, set to 3 — console unavailability is less critical than view freshness failures.

Summary

| Monitor | Target | What It Catches | |---|---|---| | REST API health | :8080/healthz | API unavailability, pipeline mgmt blocked | | Pipeline status | /v0/pipelines keyword Running | Pipeline failure, views no longer updating | | Kafka TCP | :9092 TCP | Kafka broker connectivity loss | | Kafka consumer lag | kminion metrics | Feldera falling behind input stream | | Output sink health | HTTP endpoint or heartbeat | View results not propagating downstream | | Throughput heartbeat | Heartbeat URL | Processing throughput drop from baseline | | View freshness heartbeat | Heartbeat URL (30s interval) | Output lag exceeding SLA | | Checkpoint heartbeat | Heartbeat URL (10m interval) | Checkpoint failure, recovery gap | | Worker thread health | /v0/pipelines/.../stats keyword | Worker thread crash | | Memory usage | Prometheus alert | Memory >85%, OOM risk | | Web console | https://feldera.yourdomain.com | SQL development console unavailable |

Feldera's DBSP-based incremental computation delivers correctness guarantees that other streaming SQL engines can't match — but self-hosting means owning the observability. With Vigilmon covering the REST API, pipeline state, Kafka connectivity, consumer lag, output sink health, view freshness, checkpoint integrity, and memory usage, your data engineering team gets early warning on every failure mode before stale views reach your dashboards or downstream ML features.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles