Monitoring Arroyo with Vigilmon

Arroyo is an open source distributed stream processing engine written in Rust, designed as a developer-friendly, high-performance alternative to Apache Flink. Built on Apache DataFusion with streaming SQL extensions, Arroyo lets you write complex windowed queries, joins, and aggregations in familiar SQL — without the JVM overhead or operational complexity of a Flink cluster. When you self-host Arroyo, you're running a controller pod (the API and scheduler) and a set of worker pods (executing pipeline tasks), plus the object storage backend for checkpoints and the optional web UI. Any of these components failing silently can halt your streaming pipelines, cause data loss, or break exactly-once guarantees. Vigilmon gives you visibility across every layer of the Arroyo stack.

What You'll Set Up

• Arroyo controller pod health and REST API monitor
• Pipeline status health check (Running/Failed/Stopped transitions)
• Worker pod health per pipeline
• Checkpoint health via heartbeat from pipeline code
• Kafka source consumer lag monitor
• Event throughput drop alert
• Object storage checkpoint upload health
• Arroyo web UI availability monitor
• Alert channels with appropriate thresholds

Prerequisites

• Arroyo deployed on Kubernetes (controller + worker pods)
• Arroyo REST API accessible (default port 8000 on the controller service)
• Arroyo web UI accessible (default port 8080)
• Kafka broker accessible (if using Kafka sources)
• A free Vigilmon account

Step 1: Monitor the Arroyo Controller REST API

The Arroyo controller is the single point of pipeline management — it schedules workers, accepts SQL queries, and tracks pipeline state. A monitor on its health endpoint catches controller crashes before your team notices that new pipelines can't be submitted.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: http://arroyo-controller.your-namespace.svc.cluster.local:8000/api/v1/ping (or the external load-balancer URL if you expose the API externally).
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

If you expose the Arroyo API behind an ingress, use the public URL instead:

https://arroyo.yourdomain.com/api/v1/ping

A 200 response confirms the controller is up and its REST layer is serving requests. A timeout or 5xx here means no new pipelines can be submitted and existing ones may be orphaned.

Step 2: Monitor Pipeline Status via the Pipelines API

Arroyo pipelines transition between Running, Stopped, and Failed states. A Failed pipeline is the most critical alert — it means the streaming job has halted and data is no longer being processed.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://arroyo-controller.your-namespace.svc.cluster.local:8000/api/v1/pipelines.
3. Check interval: 2 minutes.
4. Expected HTTP status: 200.
5. Under Keyword check, enter Running to verify at least one pipeline is in the running state.
6. Click Save.

For more precise per-pipeline monitoring, expose a lightweight health sidecar from your pipeline that returns 200 when the pipeline is running and 500 when it is in a failed state, then monitor that sidecar endpoint instead.

Step 3: Monitor Worker Pod Health

Arroyo worker pods execute the actual computation for each pipeline. A worker pod crash causes the pipeline to enter a recovery phase — Arroyo will attempt to reschedule workers, but repeated crashes indicate a resource or code problem.

1. Click Add Monitor → TCP Port.
2. Host: the headless service for your Arroyo workers (e.g., arroyo-worker.your-namespace.svc.cluster.local).
3. Port: 6900 (Arroyo worker gRPC port, or adjust to your deployment's configured port).
4. Check interval: 1 minute.
5. Click Save.

Combine this with the controller API monitor — if the controller is healthy but worker TCP connections time out, the workers are crashing independent of the controller.

Step 4: Monitor Checkpoint Health via Heartbeat

Arroyo uses periodic state checkpointing to object storage (S3, GCS, or local) for exactly-once semantics. A checkpoint failure means that if the pipeline crashes and recovers, it will reprocess data from the last successful checkpoint — which may be stale.

Add a heartbeat from your Arroyo pipeline's checkpoint callback or a sidecar job that runs after each successful checkpoint:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to match your checkpoint interval (e.g., 10 minutes if checkpoints run every 10 minutes).
3. Copy the heartbeat URL (e.g., https://vigilmon.online/heartbeat/abc123).
4. Add a checkpoint notification hook to your pipeline or a Kubernetes CronJob that pings the URL after verifying the latest checkpoint timestamp in S3:

#!/bin/bash
# checkpoint-heartbeat.sh — run as a Kubernetes CronJob every 10 minutes
LATEST=$(aws s3 ls s3://your-arroyo-checkpoints/pipeline-id/ \
  --recursive | sort | tail -n 1)
if [ -n "$LATEST" ]; then
  curl -sf https://vigilmon.online/heartbeat/abc123
fi

# kubernetes/checkpoint-heartbeat-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: arroyo-checkpoint-heartbeat
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: heartbeat
            image: amazon/aws-cli
            command: ["/bin/bash", "/scripts/checkpoint-heartbeat.sh"]
          restartPolicy: OnFailure

If no heartbeat arrives within the expected window, Vigilmon alerts you that checkpoint uploads have stopped — indicating a fault tolerance gap before the next pipeline failure.

Step 5: Monitor Kafka Source Consumer Lag

For Arroyo pipelines reading from Kafka, consumer lag is the leading indicator of a processing bottleneck. Lag growing beyond the acceptable threshold means your pipeline is falling behind real-time event ingestion.

1. Deploy a Kafka consumer lag exporter (e.g., Kafka Exporter, Burrow, or kminion) that exposes an HTTP metrics endpoint.
2. Click Add Monitor → HTTP / HTTPS.
3. URL: http://kafka-exporter.your-namespace.svc.cluster.local:9308/metrics.
4. Expected HTTP status: 200.
5. Under Keyword check, enter kafka_consumergroup_lag to verify the exporter is reporting lag metrics.
6. Check interval: 2 minutes.
7. Click Save.

For alerting on lag thresholds, configure your Kafka exporter to expose an alert endpoint, or use Vigilmon's keyword check to scan for a zero-lag line in the metrics output and alert when that keyword disappears.

Step 6: Monitor Event Processing Throughput via Heartbeat

A sudden drop in event throughput — records processed per second — is an early warning that the pipeline is stalling, even if it hasn't transitioned to Failed yet.

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected interval to 5 minutes.
3. Copy the heartbeat URL.
4. Add a pipeline-side metric check that pings the heartbeat only when throughput is above your baseline threshold. If Arroyo exposes Prometheus metrics, use a sidecar:

#!/usr/bin/env python3
# throughput-check.py — run every 5 minutes via CronJob
import requests

PROMETHEUS_URL = "http://prometheus.monitoring.svc.cluster.local:9090"
PIPELINE_ID = "your-pipeline-id"
THROUGHPUT_THRESHOLD = 1000  # records/sec

query = f'rate(arroyo_records_out_total{{pipeline="{PIPELINE_ID}"}}[5m])'
resp = requests.get(f"{PROMETHEUS_URL}/api/v1/query", params={"query": query})
data = resp.json()

if data["data"]["result"]:
    throughput = float(data["data"]["result"][0]["value"][1])
    if throughput >= THROUGHPUT_THRESHOLD:
        requests.get("https://vigilmon.online/heartbeat/abc123", timeout=5)

If the heartbeat stops arriving, it means throughput has dropped below threshold — alert before the pipeline fully stalls.

Step 7: Monitor Object Storage for Checkpoint Uploads

Arroyo stores checkpoints in S3 or GCS. Monitor S3 directly to catch connectivity issues between the Arroyo workers and object storage.

For MinIO (self-hosted S3-compatible):

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://minio.your-namespace.svc.cluster.local:9000/minio/health/live.
3. Expected HTTP status: 200.
4. Check interval: 2 minutes.
5. Click Save.

For AWS S3 or GCS:

Monitor the IAM or service account permissions indirectly by adding a Kubernetes CronJob that performs a test PutObject and GetObject every 5 minutes and pings a Vigilmon heartbeat on success. A heartbeat miss indicates permission or connectivity failures before Arroyo itself surfaces a checkpoint error.

Step 8: Monitor the Arroyo Web UI

The Arroyo web console is the primary interface for submitting SQL queries, managing pipelines, and viewing pipeline metrics. If the UI pod is down, your team can't inspect pipeline state or submit new queries.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://arroyo.yourdomain.com (or http://arroyo-ui.your-namespace.svc.cluster.local:8080).
3. Expected HTTP status: 200.
4. Under Keyword check, enter Arroyo to verify the UI content loads.
5. Check interval: 2 minutes.
6. Enable Monitor SSL certificate with a 21 days expiry alert.
7. Click Save.

Step 9: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add Slack, email, or a webhook.
2. For the controller REST API, set Consecutive failures before alert to 2 — brief pod restarts cause transient failures.
3. For pipeline status (keyword check on Running), set to 1 — a failed pipeline is always actionable immediately.
4. For Kafka consumer lag and throughput heartbeats, set to 1 — lag growth compounds quickly.
5. For checkpoint heartbeats, Vigilmon alerts automatically after the expected interval passes with no ping.
6. For the web UI, set to 3 — UI pod restarts are less critical than pipeline failures.

SQL Query Error Rate Monitoring

Arroyo SQL UDFs and runtime expressions can fail at runtime. Monitor query error rate by checking Arroyo's Prometheus metrics for the arroyo_query_errors_total counter:

1. Deploy Prometheus in your cluster and scrape Arroyo's metrics endpoint.
2. Add a Prometheus alerting rule:

groups:
  - name: arroyo
    rules:
      - alert: ArroyoHighQueryErrorRate
        expr: rate(arroyo_query_errors_total[5m]) > 0.001
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Arroyo SQL query error rate above 0.1%"

3. Configure Prometheus Alertmanager to send alerts to your Vigilmon webhook channel for unified alert management.

Summary

| Monitor | Target | What It Catches | |---|---|---| | Controller REST API | /api/v1/ping | Controller crash, scheduling stopped | | Pipeline status | /api/v1/pipelines keyword Running | Pipeline failure, no active jobs | | Worker pod TCP | :6900 | Worker crash, computation stopped | | Checkpoint heartbeat | Heartbeat URL | Checkpoint upload failure, fault tolerance gap | | Kafka consumer lag | Kafka exporter metrics | Processing falling behind ingestion | | Throughput heartbeat | Heartbeat URL | Throughput drop >30% below baseline | | MinIO health | :9000/minio/health/live | Object storage unavailable | | Web UI | https://arroyo.yourdomain.com | Console unavailable, SQL dev blocked |

Arroyo's Rust-based architecture delivers exceptional throughput and low latency, but self-hosting means you own the observability. With Vigilmon covering the controller API, pipeline status, worker health, checkpoint integrity, Kafka lag, and throughput, you get early warning on every failure mode — from a single worker pod crash to a checkpoint upload gap that would otherwise only surface after the next pipeline failure event.