tutorial

Monitoring Apache Kyuubi with Vigilmon

Apache Kyuubi is the multi-tenant SQL gateway that makes Spark SQL production-ready — here's how to monitor server health, engine startup latency, session counts, query success rates, YARN queue utilization, and ZooKeeper HA with Vigilmon.

Apache Kyuubi is the SQL gateway that tames Apache Spark for multi-tenant production use. Without Kyuubi, every data analyst spinning up a Spark session competes for YARN resources, bypasses security policies, and keeps a Spark application alive indefinitely. Kyuubi adds session management, per-user engine isolation, and a JDBC/ODBC interface that lets Tableau, DBeaver, and any Hive-compatible client run Spark SQL without knowing Spark exists. When Kyuubi is healthy, your entire data warehouse query tier works. When it is not, silent failures — stalled engine startups, exhausted session pools, and YARN queue saturation — block every analyst and BI tool simultaneously. Vigilmon gives you continuous visibility into Kyuubi server health, engine lifecycle, query reliability, and infrastructure dependencies before a session pool exhaustion takes your SQL tier offline.

What You'll Set Up

  • Kyuubi REST API and Thrift service health checks
  • Engine startup latency monitoring
  • Active session count tracking
  • Query success rate alerting
  • Engine pool utilization monitoring
  • ZooKeeper HA health for Kyuubi clusters
  • YARN queue capacity alerting
  • Thrift connection count monitoring
  • Engine OOM event detection

Prerequisites

  • Apache Kyuubi 1.7+ deployed (standalone or on Kubernetes/YARN)
  • Kyuubi REST API enabled (default port 10099)
  • Kyuubi Thrift service running (default port 10009)
  • A free Vigilmon account

Why Monitoring Apache Kyuubi Matters

Kyuubi introduces an abstraction layer between JDBC clients and Spark, which creates failure modes at multiple levels:

  • Engine startup is slow and can fail silently. Kyuubi starts a Spark application on demand when a new session opens. Startup takes 2–5 minutes under normal conditions. If YARN is resource-constrained, a startup can hang indefinitely — the client's JDBC connection is stuck, but no error is surfaced until a timeout fires.
  • Session pool exhaustion blocks all new connections. Kyuubi enforces a maximum concurrent session count. When the pool is full, new JDBC connections queue and eventually time out. BI tools like Tableau show a spinner, not an error, while the session pool is exhausted.
  • ZooKeeper is the HA bus. In high-availability mode, Kyuubi nodes register themselves in ZooKeeper and clients discover them there. If ZooKeeper loses quorum, Kyuubi clients cannot establish new sessions even if all Kyuubi server processes are alive.
  • Engine OOMs cascade. When a Spark executor runs out of memory, the executor dies, Kyuubi's engine session terminates, and any query running on that session fails. If OOMs are occurring at high frequency, queries fail faster than sessions can be restarted.

Step 1: Monitor the Kyuubi REST API

Kyuubi exposes a REST API with a dedicated ping endpoint. This is the fastest and most reliable liveness check for the Kyuubi server process:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to HTTP / HTTPS.
  3. Enter URL: http://your-kyuubi-host:10099/api/v1/ping
  4. Set Check interval to 1 minute.
  5. Set Expected HTTP status to 200.
  6. Under Alert conditions, set Alert after to 1 failure.
  7. Click Save.

A 200 response with body pong confirms the Kyuubi server is alive and its REST service is accepting connections.

Step 2: Monitor the Kyuubi Thrift Service

The Thrift service is the HiveServer2-compatible endpoint that JDBC/ODBC clients connect to. It runs on a different port from the REST API and can fail independently (e.g., if the Thrift thread pool is exhausted while the REST API is healthy).

  1. Click Add Monitor → set Type to TCP Port.
  2. Enter Host: your-kyuubi-host, Port: 10009.
  3. Set Check interval to 1 minute.
  4. Enable alerts on connection failure.
  5. Click Save.

A TCP connection failure on port 10009 means no JDBC or ODBC client can establish a session.

Step 3: Monitor Engine Startup Latency

Engine startup time is the most common performance problem in Kyuubi deployments. When YARN is under load, Spark driver startup can take 5–10 minutes — or hang indefinitely. Users experience this as a very slow JDBC connection that eventually times out.

Create a Vigilmon heartbeat that measures startup time end-to-end:

#!/bin/bash
START=$(date +%s)

# Submit a minimal Spark session and run a trivial query
beeline -u "jdbc:hive2://your-kyuubi-host:10009/default" \
  -e "SELECT 1" \
  --timeout=300 \
  --silent=true 2>/dev/null

EXIT_CODE=$?
END=$(date +%s)
ELAPSED=$((END - START))

if [ $EXIT_CODE -eq 0 ] && [ $ELAPSED -lt 300 ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_ENGINE_STARTUP_HEARTBEAT"
else
  echo "Engine startup failed or timed out after ${ELAPSED}s (exit: $EXIT_CODE)"
fi

Schedule this every 15 minutes via cron. Configure the Vigilmon heartbeat with a 20-minute expected interval. If startup exceeds 5 minutes or fails, no heartbeat is sent and Vigilmon alerts.

Step 4: Monitor Active Session Count

The active session count is a direct indicator of whether the session pool is approaching exhaustion. Kyuubi exposes session counts via its REST API:

#!/bin/bash
SESSIONS=$(curl -s "http://your-kyuubi-host:10099/api/v1/sessions" | jq 'length')
MAX_SESSIONS=100  # Set to your kyuubi.session.engine.share.level.subdomain.max value

echo "Active sessions: $SESSIONS / $MAX_SESSIONS"

# Send heartbeat only when below 90% capacity
THRESHOLD=$((MAX_SESSIONS * 90 / 100))
if [ "$SESSIONS" -lt "$THRESHOLD" ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_SESSIONS_HEARTBEAT"
else
  echo "Session pool at ${SESSIONS}/${MAX_SESSIONS} — approaching limit, not sending heartbeat"
fi

Schedule every 5 minutes. Set the Vigilmon heartbeat expected interval to 10 minutes. When the session pool exceeds 90% capacity, the heartbeat lapses and Vigilmon alerts before new connections start queuing.

Step 5: Monitor Query Success Rate

Query failures at scale indicate engine errors, resource exhaustion, or bad SQL being submitted. Kyuubi's REST API exposes operation (query) statistics per session. Use the metrics endpoint for aggregate success/failure counts:

If Kyuubi is configured with Prometheus metrics (via kyuubi.metrics.reporters=PROMETHEUS), alert on the query failure rate:

- alert: KyuubiHighQueryFailureRate
  expr: rate(kyuubi_operation_failed_total[5m]) / (rate(kyuubi_operation_total[5m]) + 0.001) > 0.05
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Kyuubi query failure rate > 5%"
    description: "{{ $value | humanizePercentage }} of queries are failing. Check engine OOM events and YARN queue capacity."

For deployments without Prometheus, aggregate success/failure from Kyuubi's audit log or the operations API:

#!/bin/bash
TOTAL=$(curl -s "http://your-kyuubi-host:10099/api/v1/operations" | jq 'length')
FAILED=$(curl -s "http://your-kyuubi-host:10099/api/v1/operations" | jq '[.[] | select(.state == "ERROR")] | length')

if [ "$TOTAL" -gt 0 ]; then
  FAIL_PCT=$((FAILED * 100 / TOTAL))
  if [ "$FAIL_PCT" -lt 5 ]; then
    curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_QUERY_SUCCESS_HEARTBEAT"
  fi
fi

Step 6: Monitor Engine Pool Utilization

Kyuubi manages a pool of Spark engine sessions. When all engine slots are occupied, new JDBC sessions queue. Monitor engine utilization via the Kyuubi engines REST endpoint:

#!/bin/bash
RUNNING=$(curl -s "http://your-kyuubi-host:10099/api/v1/engines" \
  | jq '[.[] | select(.state == "STARTED")] | length')
MAX_ENGINES=20  # Set to your configured maximum

echo "Running engines: $RUNNING / $MAX_ENGINES"

THRESHOLD=$((MAX_ENGINES * 85 / 100))
if [ "$RUNNING" -lt "$THRESHOLD" ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_ENGINES_HEARTBEAT"
fi

Schedule every 5 minutes with a 10-minute heartbeat expected interval. Alerts fire when engine utilization exceeds 85% — early enough to warn on-call before the pool is full.

Step 7: Monitor ZooKeeper HA Health

For Kyuubi HA deployments, ZooKeeper is the service registry that clients use to discover Kyuubi nodes. Monitor each ZooKeeper node's TCP port:

  1. Click Add Monitor → set Type to TCP Port.
  2. Enter Host: your-zk-node-1, Port: 2181.
  3. Set Check interval to 1 minute.
  4. Enable alerts on connection failure.
  5. Click Save.

Repeat for each ZooKeeper node in the ensemble. Also verify Kyuubi's own registration in ZooKeeper using a script:

#!/bin/bash
ZK_PATH="/kyuubi/serviceUri"
REGISTERED=$(zkCli.sh -server your-zk-host:2181 ls "$ZK_PATH" 2>/dev/null | grep -c "kyuubi")

if [ "$REGISTERED" -gt 0 ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_ZK_REGISTRATION_HEARTBEAT"
else
  echo "Kyuubi not registered in ZooKeeper at $ZK_PATH"
fi

Schedule every 5 minutes. If Kyuubi drops its ZooKeeper registration (network partition, ZooKeeper session expiry), no new clients can discover the server.

Step 8: Monitor YARN Queue Utilization

Kyuubi submits Spark applications to YARN. If the YARN queue allocated to Kyuubi engines is at 100% capacity, new engine startup requests queue indefinitely. Monitor YARN queue utilization via the YARN ResourceManager REST API:

#!/bin/bash
QUEUE_NAME="kyuubi"  # Your configured YARN queue name
RESPONSE=$(curl -s "http://your-yarn-rm-host:8088/ws/v1/cluster/scheduler")
USED=$(echo "$RESPONSE" | jq --arg q "$QUEUE_NAME" \
  '.scheduler.schedulerInfo.queues.queue[] | select(.queueName==$q) | .usedCapacity')

echo "YARN queue $QUEUE_NAME utilization: ${USED}%"

# Alert when queue is over 90%
if (( $(echo "$USED < 90" | bc -l) )); then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_YARN_HEARTBEAT"
fi

Schedule every 2 minutes. Set the Vigilmon heartbeat expected interval to 5 minutes.

Step 9: Monitor Thrift Connection Count

The active Thrift connection count shows how many JDBC/ODBC clients are currently connected. A spike may indicate a connection leak in a BI tool or a stuck batch job. A sudden drop to zero may indicate the Thrift service has crashed.

#!/bin/bash
CONNECTIONS=$(curl -s "http://your-kyuubi-host:10099/api/v1/sessions" \
  | jq 'length')

echo "Active Thrift connections: $CONNECTIONS"
curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_THRIFT_CONNECTIONS_HEARTBEAT"

Schedule every 5 minutes. Review connection counts in Vigilmon's heartbeat history over time to establish a normal range and identify spikes or drops.

Step 10: Monitor Engine OOM Events

Spark executor OOM events cause engine sessions to terminate, which fails any query that was running and forces a new engine startup for the next user request. OOM events typically indicate undersized executor memory or data skew in the query workload.

Parse OOM events from Kyuubi's engine logs or Spark driver logs:

#!/bin/bash
LOG_FILE="/var/log/kyuubi/kyuubi-engine.log"
OOM_COUNT=$(grep -c "OutOfMemoryError" "$LOG_FILE" 2>/dev/null || echo 0)

# Alert on OOM events in the last 10 minutes
RECENT_OOMS=$(grep "OutOfMemoryError" "$LOG_FILE" 2>/dev/null \
  | awk -v cutoff="$(date -d '10 minutes ago' '+%Y-%m-%d %H:%M')" '$0 >= cutoff' \
  | wc -l)

echo "Recent OOM events (last 10m): $RECENT_OOMS"

if [ "$RECENT_OOMS" -eq 0 ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_OOM_HEARTBEAT"
fi

Schedule every 10 minutes. Set the heartbeat expected interval to 15 minutes. OOM events cause the heartbeat to lapse, triggering a Vigilmon alert.

Step 11: Set Up Alerting

  1. Go to Alert Policies → click New Policy.
  2. Name it Kyuubi SQL Gateway Alerts.
  3. Add your Slack webhook (for engineering) and email (for on-call).
  4. Assign this policy to all Kyuubi monitors.
  5. For the REST API ping and Thrift TCP monitors, set Alert after to 1 failure — these are critical.
  6. For heartbeat monitors (engine startup, session count, YARN queue), use 1 missed ping.
  7. Click Save.

Alert Reference

| Monitor | Alert Threshold | Severity | Impact | |---|---|---|---| | REST API /api/v1/ping | Non-200 | Critical | All SQL access blocked | | Thrift TCP :10009 | Connection refused | Critical | All JDBC/ODBC connections blocked | | ZooKeeper TCP :2181 | Connection refused | High | HA discovery failing | | Kyuubi ZooKeeper registration | Not registered | High | Clients cannot discover Kyuubi | | Engine startup heartbeat | Missed (>5m startup) | High | New sessions hanging | | Session pool utilization | > 90% capacity | High | New connections queuing | | Engine pool utilization | > 85% capacity | Warning | Engine startup requests queuing | | Query success rate | < 95% | High | Engine errors or resource exhaustion | | YARN queue utilization | > 90% | High | Engine startup will block | | Engine OOM events | Any in 10m | Warning | Queries failing and sessions terminating |

Conclusion

Apache Kyuubi's multi-tenant session management makes it a powerful production SQL gateway, but it introduces failure modes at every layer of the stack: the Kyuubi server process, the Thrift service, the engine pool, the YARN queue, ZooKeeper HA, and individual Spark executors. The monitoring setup in this guide catches failures at each layer before they cascade to your data consumers.

Start with the REST API ping and Thrift TCP monitors — those two give you liveness coverage in under 5 minutes. Then add engine startup latency monitoring via the beeline heartbeat script, which catches the most common production problem (YARN queue starvation causing slow session startup) before users file tickets.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles