tutorial

Monitoring Proton (Timeplus) with Vigilmon

Proton is the open source streaming SQL engine built on ClickHouse — but a crashed Proton server silently terminates all streaming queries and materialized views. Here's how to monitor Proton end-to-end with Vigilmon.

Proton is the open source streaming SQL engine from Timeplus — built on ClickHouse's columnar storage and vectorized query engine, with a native streaming layer added on top. It lets you write a single SQL query that joins real-time Kafka data with historical ClickHouse-compatible tables, without the JVM overhead of Apache Flink or the limited historical query capabilities of ksqlDB. When Proton is healthy, your streaming queries run continuously, materialized views stay fresh, and Kafka external streams process data at ClickHouse-level throughput. When the Proton server crashes, every streaming query terminates simultaneously — and there is no external alerting built into the open source distribution. Vigilmon provides the uptime monitoring and heartbeat checks that Proton needs in production: server HTTP health, streaming query liveness, Kafka consumer lag, and storage capacity alerts.

What You'll Set Up

  • HTTP uptime monitor for the Proton HTTP API (port 3218)
  • TCP port monitor for the Proton native protocol port (9000)
  • Cron heartbeats for each production streaming query and materialized view
  • Kafka consumer lag monitoring for external streams
  • Storage capacity monitor for Proton's ClickHouse-compatible storage

Prerequisites

  • Proton 1.5+ deployed via Docker or as a binary on Linux
  • Proton HTTP API accessible on port 3218 (configurable)
  • Kafka accessible for external stream lag monitoring
  • A free Vigilmon account

Why Monitoring Proton Matters

Proton serves as both a streaming compute engine and a historical data store. Failures affect two very different use cases simultaneously:

  • Streaming queries — SQL queries with FROM stream(your_stream) run indefinitely. A Proton crash terminates them all, including materialized views that downstream dashboards depend on
  • Historical queries — Analytical queries over historical data fail if Proton storage is corrupted or disk is full
  • Kafka external streams — If Proton's consumer falls behind on Kafka partitions, streaming queries return stale results without raising errors
  • Materialized views — MV refresh failures cause derived tables to accumulate lag; queries against the MV return progressively staler data

External monitoring is essential because Proton has no built-in dead-man's-switch to alert on streaming query termination.

Step 1: Monitor the Proton HTTP API

Proton exposes an HTTP API on port 3218 (default) for query execution, cluster health, and management. This is the primary health signal for the Proton server process.

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter the Proton HTTP API ping endpoint:
http://your-proton-host:3218/ping
  1. Set Check interval to 1 minute.
  2. Set Expected HTTP status to 200.
  3. Set Expected response body contains to OK.
  4. Click Save.

The /ping endpoint returns OK when the Proton server is healthy and accepting queries. This is a lightweight check that does not execute any SQL.

For a deeper health check, use the Proton query endpoint to run a trivial query:

http://your-proton-host:3218/?query=SELECT%201

Set Expected HTTP status to 200 and Expected response body contains to 1. This confirms the query engine itself is functional, not just the HTTP listener.

Step 2: Monitor the Proton TCP Port

Client applications connect to Proton via the ClickHouse native protocol on TCP port 9000. This is the port used by ClickHouse client libraries (clickhouse-driver in Python, clickhouse-go, clickhouse-client CLI). Add a TCP port monitor:

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Enter your-proton-host:9000.
  3. Set Check interval to 1 minute.
  4. Click Save.

If you changed the TCP port in proton-server.xml (the ClickHouse-compatible configuration), use the configured port instead.

Step 3: Monitor the Proton Server Process with a Heartbeat

For bare-metal or VM deployments where Proton runs as a systemd service, add a process heartbeat:

  1. In Vigilmon, click Add Monitor → Cron Heartbeat.
  2. Name it Proton server process.
  3. Set the expected interval to 5 minutes.
  4. Copy the heartbeat URL.

Add a cron job on the Proton host:

# /etc/cron.d/proton-process-heartbeat
*/3 * * * * proton systemctl is-active proton-server > /dev/null 2>&1 && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_PROCESS_KEY" > /dev/null 2>&1

For Docker deployments, check the container health:

# /etc/cron.d/proton-docker-heartbeat
*/3 * * * * root docker inspect --format='{{.State.Health.Status}}' proton | \
  grep -q healthy && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_PROCESS_KEY" > /dev/null 2>&1

Add a Docker health check to your Proton container if you haven't already:

# docker-compose.yml
services:
  proton:
    image: ghcr.io/timeplus-io/proton:latest
    ports:
      - "3218:3218"
      - "9000:9000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3218/ping"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

Step 4: Monitor Active Streaming Queries

Proton streaming queries (SELECT ... FROM stream(your_stream)) run indefinitely. Use the Proton HTTP API to count active streaming queries and send a heartbeat when they are running:

#!/bin/bash
# /opt/monitoring/check-proton-streaming.sh
PROTON_HTTP="http://localhost:3218"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_STREAMING_KEY"

# Query the system.processes table to count active streaming queries
ACTIVE=$(curl -sf "$PROTON_HTTP/?query=SELECT+count()+FROM+system.processes+WHERE+query+LIKE+'%25stream(%25'")

if [ "$ACTIVE" -gt "0" ] 2>/dev/null; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 2 minutes. Set the heartbeat interval to 10 minutes. If all streaming queries terminate (due to a Proton restart or query error), the heartbeat expires and you are alerted.

For critical named streaming queries, monitor them individually. Proton assigns a query ID that you can set explicitly:

-- Set a stable query_id for your production streaming query
SELECT * FROM stream(orders)
WHERE status = 'PAID'
SETTINGS query_id = 'prod-orders-stream'

Then check for this specific query ID:

curl -sf "http://localhost:3218/?query=SELECT+count()+FROM+system.processes+WHERE+query_id%3D'prod-orders-stream'"

Step 5: Monitor Materialized View Refresh Health

Proton materialized views incrementally compute over streams — they process each new batch of streaming data and write results to a target table. If the MV refresh fails or falls behind, downstream queries return stale data.

Check MV health via the system tables:

#!/bin/bash
# /opt/monitoring/check-proton-mvs.sh
PROTON_HTTP="http://localhost:3218"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_MV_KEY"

# Check if materialized views are listed (indicates the MV subsystem is healthy)
MV_COUNT=$(curl -sf "$PROTON_HTTP/?query=SELECT+count()+FROM+system.tables+WHERE+engine%3D'MaterializedView'")

if [ "$MV_COUNT" -gt "0" ] 2>/dev/null; then
  curl -fsS "$HEARTBEAT_URL"
fi

For freshness monitoring, check when the MV target table was last updated:

#!/bin/bash
# Check MV target table freshness
MAX_LAG_SECONDS=300
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_MV_FRESHNESS_KEY"

LAST_UPDATE=$(curl -sf "http://localhost:3218/?query=SELECT+max(toUnixTimestamp(_tp_time))+FROM+mv_orders_summary")
NOW=$(date +%s)
LAG=$((NOW - LAST_UPDATE))

if [ "$LAG" -lt "$MAX_LAG_SECONDS" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 2 minutes. Set the Vigilmon heartbeat interval to 10 minutes. If the MV stops updating (due to a streaming query failure or ingestion stall), the heartbeat expires.

Step 6: Monitor Kafka External Stream Lag

Proton reads from Kafka via external streams defined with CREATE EXTERNAL STREAM. Consumer lag in these streams means streaming queries receive delayed data. Monitor lag using Kafka's built-in consumer group tools:

#!/bin/bash
# /etc/cron.d/proton-kafka-lag
*/5 * * * * monitoring /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --describe \
  --all-groups 2>/dev/null | \
  grep "proton" | \
  awk '{if ($6 > 10000) exit 1}' && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_KAFKA_LAG_KEY" > /dev/null 2>&1

This pings Vigilmon only when Kafka lag is below 10,000 messages. Set the threshold based on your throughput and acceptable data freshness. Set the heartbeat interval to 15 minutes.

Proton's consumer group name is typically prefixed with proton_ followed by the external stream name. Find the exact consumer group:

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka:9092 \
  --list | grep proton

Step 7: Monitor Proton Storage Health

Proton stores historical data using ClickHouse-compatible storage on local disk. Disk capacity and I/O health are critical — a full disk causes Proton to reject writes and may corrupt ongoing transactions.

Add a Vigilmon cron heartbeat that checks disk usage:

#!/bin/bash
# /etc/cron.d/proton-storage-health
*/10 * * * * proton DISK_USAGE=$(df /var/lib/proton | awk 'NR==2 {print $5}' | tr -d '%') && \
  [ "$DISK_USAGE" -lt "85" ] && \
  curl -fsS "https://vigilmon.online/heartbeat/YOUR_STORAGE_KEY" > /dev/null 2>&1

Set the heartbeat interval to 30 minutes. Alert before reaching 85% capacity — Proton (like ClickHouse) can become unpredictable at very high disk utilization.

You can also query storage usage directly via the Proton HTTP API:

# Query disk usage from Proton system tables
curl -sf "http://localhost:3218/?query=SELECT+free_space,total_space+FROM+system.disks"

Step 8: Monitor Query Latency

For interactive analytics workloads where Proton serves as the backend for real-time dashboards, P99 query latency is a key SLA metric. Monitor query latency via a synthetic probe:

#!/bin/bash
# /opt/monitoring/probe-proton-latency.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_LATENCY_KEY"
MAX_LATENCY_MS=10000  # 10 second P99 threshold

START=$(date +%s%3N)

# Run a representative analytical query
RESULT=$(curl -sf "http://localhost:3218/?query=SELECT+count(),+avg(amount)+FROM+orders+WHERE+_tp_time+>+now()-interval+1+hour")

END=$(date +%s%3N)
DURATION=$((END - START))

if [ $? -eq 0 ] && [ "$DURATION" -lt "$MAX_LATENCY_MS" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 5 minutes. If the probe query takes longer than 10 seconds or returns an error (indicating insufficient compaction, storage I/O contention, or query planner regression), the heartbeat expires.

Step 9: Monitor the Proton Error Log Rate

Proton logs server-side errors to the system log (accessible via system.text_log). A sustained error rate indicates systematic query failures, ingestion issues, or storage problems:

#!/bin/bash
# /opt/monitoring/check-proton-errors.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_ERRORLOG_KEY"
MAX_ERRORS_PER_5MIN=10

# Count ERROR-level log entries in the last 5 minutes
ERROR_COUNT=$(curl -sf "http://localhost:3218/?query=SELECT+count()+FROM+system.text_log+WHERE+level%3D'Error'+AND+event_time+>+now()-interval+5+minute")

if [ "$ERROR_COUNT" -lt "$MAX_ERRORS_PER_5MIN" ] 2>/dev/null; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run every 5 minutes. Set the heartbeat interval to 15 minutes. A sustained error rate above your threshold stops the heartbeat and triggers an alert.

Step 10: Configure Alerting

Set up alert channels for all Proton monitors:

  1. In Vigilmon, go to Alert Channels → Add Channel.
  2. Configure Slack, PagerDuty, or email.
  3. For the Proton HTTP API and TCP port monitors (uptime signals), use PagerDuty or high-priority Slack.
  4. For heartbeat-based monitors (query health, lag, latency), use a lower-priority channel for initial investigation.

Recommended alert thresholds:

| Monitor | Alert Condition | Severity | |---|---|---| | Proton HTTP API (/ping) | Down for 2 checks (2 min) | Critical | | Proton TCP port 9000 | Down for 1 check | Critical | | Proton process heartbeat | Missed for 5 min | Critical | | Active streaming queries heartbeat | Missed for 10 min | High | | MV freshness heartbeat | Missed for 10 min | High | | Kafka external stream lag heartbeat | Missed for 15 min | High | | Storage capacity heartbeat | Missed for 30 min | Medium | | Query latency heartbeat | Missed for 15 min | Medium | | Error log rate heartbeat | Missed for 15 min | Medium |

Conclusion

Proton's unified streaming and historical SQL capabilities make it a powerful replacement for multiple specialized systems — but they also mean a single Proton server failure simultaneously breaks streaming queries, materialized views, historical analytics, and Kafka-connected pipelines. With Vigilmon monitoring the HTTP API, TCP port, streaming query health, MV freshness, Kafka lag, and storage capacity, you have comprehensive coverage across the full Proton stack. A Proton server crash that would previously go undetected until a user reports stale dashboard data now triggers a PagerDuty alert within two minutes.

Start with the HTTP ping monitor and the process heartbeat, then add streaming query health and Kafka lag monitoring for your production streams. Sign up for a free Vigilmon account to get started.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles