tutorial

Monitoring Apache Impala with Vigilmon

Apache Impala powers sub-second SQL queries on your Hadoop cluster — here's how to monitor impalad health across every data node, statestored, catalogd, admission control pools, query latency, and memory usage with Vigilmon.

Apache Impala is a massively parallel processing (MPP) SQL query engine that runs directly on your Hadoop cluster, delivering sub-second interactive queries on HDFS, HBase, Kudu, and object storage. Unlike batch-oriented Hive, Impala keeps long-running daemons resident on every data node — which means a silent failure on any node silently reduces your query execution capacity without obvious errors. Vigilmon gives you visibility across every Impala daemon, the central statestored and catalogd services, admission control pools, and query execution health.

What You'll Set Up

  • Impalad process health checks on every data node (Beeswax port 21000, HS2 port 21050)
  • Statestored health monitor with subscriber count tracking
  • Catalogd health monitor and metadata broadcast alerting
  • Admission control pool saturation alerting
  • Query success rate and latency monitoring via the Impala web UI metrics endpoint
  • Memory usage alerts per impalad daemon
  • HDFS/object storage read throughput monitoring
  • Hive Metastore connectivity alerts

Prerequisites

  • Apache Impala deployed on a Hadoop cluster (Cloudera CDH, CDP, or bare Apache)
  • Impala Web UI enabled (default port 25000 on impalad, 25010 on statestored, 25020 on catalogd)
  • Impala metrics endpoint enabled (enabled by default)
  • A free Vigilmon account

Step 1: Monitor Impalad Health on Every Data Node

Every data node in your Hadoop cluster runs an impalad — it acts as both a query coordinator (accepting queries from clients) and an executor (running query fragments on local data). If any impalad goes down, you silently lose execution capacity for queries that need data on that node.

Monitor the Beeswax port (21000) on each data node — this is the Impala client protocol port that impalad listens on:

  1. Log in to vigilmon.online and click Add Monitor.
  2. Set Type to TCP Port.
  3. Enter Host: datanode1.yourdomain.com and Port: 21000.
  4. Set Check interval to 1 minute.
  5. Set Alert after: 2 consecutive failures to avoid false alarms from brief network blips.
  6. Click Save.

Repeat for every data node in your cluster, and also add a TCP monitor on port 21050 (the HiveServer2 / JDBC port used by BI tools):

  1. Click Add Monitor → TCP Port.
  2. Host: datanode1.yourdomain.com, Port: 21050.
  3. Check interval: 1 minute.
  4. Click Save.

If you have 10 data nodes, you'll have 20 TCP monitors (one pair per node). Use Vigilmon's tagging feature to group them under an impala-daemons tag for easy filtering.

To also verify the Impala Web UI (which exposes Impala metrics), add an HTTP monitor:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://datanode1.yourdomain.com:25000/healthz.
  3. Expected HTTP status: 200.
  4. Check interval: 1 minute.
  5. Click Save.

Step 2: Monitor Statestored Health

The statestored is the pub-sub membership service that tracks which impalad processes are alive and distributes this cluster membership information to all other Impala daemons. If statestored crashes, impalad coordinators can no longer verify which executors are healthy, and query planning degrades.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://statestored-host.yourdomain.com:25010/healthz.
  3. Expected HTTP status: 200.
  4. Check interval: 1 minute.
  5. Set Alert after: 1 failure — statestored is a singleton; a single failure is significant.
  6. Click Save.

Also monitor the statestored subscriber count by checking its metrics page. The statestored exposes a /metrics endpoint. Add a keyword check to verify all impalads are subscribed:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://statestored-host.yourdomain.com:25010/metrics?json.
  3. Keyword check: enter statestore.live-backends.
  4. Check interval: 2 minutes.
  5. Click Save.

Alert in Vigilmon with a threshold: if the page stops returning the keyword (meaning the statestored metrics endpoint is unhealthy), you'll be notified immediately.

Step 3: Monitor Catalogd Health

The catalogd caches Hive Metastore metadata (table schemas, partition lists) and broadcasts changes to all impalad processes. If catalogd crashes, table metadata on all impalads becomes stale — DDL operations like ALTER TABLE and REFRESH fail, and newly created tables may be invisible to queries.

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://catalogd-host.yourdomain.com:25020/healthz.
  3. Expected HTTP status: 200.
  4. Check interval: 1 minute.
  5. Set Alert after: 1 failure.
  6. Click Save.

Add a metrics check to detect catalogd metadata broadcast staleness:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://catalogd-host.yourdomain.com:25020/metrics?json.
  3. Keyword check: enter catalog.num-databases.
  4. Check interval: 2 minutes.
  5. Click Save.

If catalogd stops responding to the metrics endpoint, the keyword check fails and Vigilmon alerts you.

Step 4: Monitor Admission Control Pool Saturation

Impala's admission control system enforces per-resource-pool concurrency and memory limits, preventing query storms from overwhelming the cluster. When a pool is saturated, incoming queries queue up — and if the queue fills, queries are rejected.

The impalad Web UI exposes admission control pool status. Add monitors that check for queue build-up:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://coordinator-node.yourdomain.com:25000/admission.
  3. Keyword check: enter queued — the absence of queued queries is healthy; the presence indicates saturation.
  4. Check interval: 1 minute.
  5. Click Save.

For more detailed pool tracking, configure an external monitoring script that scrapes the Impala metrics JSON and posts the result to Vigilmon via a heartbeat. Send a heartbeat every minute from a cron job that checks pool utilization:

#!/bin/bash
# Check Impala admission pool - alert if queued queries > 0
QUEUED=$(curl -s http://coordinator-node:25000/metrics?json | \
  python3 -c "import sys,json; m=json.load(sys.stdin); \
  print(sum(v for k,v in m.items() if 'admission-controller.queued-queries' in k))")

if [ "$QUEUED" -eq "0" ]; then
  curl -fsS -m 10 "https://vigilmon.online/api/push/YOUR_HEARTBEAT_KEY" > /dev/null
fi

Add a cron heartbeat monitor in Vigilmon (set interval to 2 minutes) — if the heartbeat stops arriving, it means either the cron failed or there are queued queries blocking the check.

Step 5: Monitor Query Success Rate and Latency

Impala tracks query execution statistics on the impalad Web UI. The key metrics are total queries completed, queries failed, and latency percentiles. Scrape these from the metrics endpoint to drive your alerting.

Add a heartbeat monitor in Vigilmon for your query health checker:

  1. Click Add Monitor → Heartbeat / Cron.
  2. Set Expected interval: 5 minutes.
  3. Copy the heartbeat URL (e.g., https://vigilmon.online/api/push/YOUR_KEY).
  4. Click Save.

Create a query health check script:

#!/bin/bash
# Scrape Impala query metrics and alert on failure rate
METRICS=$(curl -s http://coordinator-node:25000/metrics?json)

COMPLETED=$(echo "$METRICS" | python3 -c \
  "import sys,json; m=json.load(sys.stdin); print(m.get('impala.queries.num-queries-executed', 0))")
FAILED=$(echo "$METRICS" | python3 -c \
  "import sys,json; m=json.load(sys.stdin); print(m.get('impala.queries.num-queries-failed', 0))")

# Calculate failure rate (requires tracking delta between runs)
# For a simple check: if no failures, send heartbeat
if [ "$FAILED" -eq "0" ] || [ "$COMPLETED" -gt "100" ]; then
  curl -fsS -m 10 "https://vigilmon.online/api/push/YOUR_KEY" > /dev/null
fi

Schedule this in crontab:

*/5 * * * * /opt/monitoring/impala-query-health.sh

For latency monitoring, add a Vigilmon HTTP monitor against your Impala coordinator using the HiveServer2 HTTP endpoint if you have it exposed, or add a dedicated latency probe query script as a separate heartbeat.

Step 6: Monitor Memory Usage Per Impalad

Each impalad operates within a configured mem_limit. If an impalad's memory usage approaches this limit, queries spill to disk (degrading performance) or are rejected entirely. Monitor memory headroom per node.

Add a heartbeat monitor for your Impala memory checker:

#!/bin/bash
# Alert if any impalad memory > 85% of mem_limit
NODES=("datanode1" "datanode2" "datanode3")
HEALTHY=true

for NODE in "${NODES[@]}"; do
  MEM_USED=$(curl -s "http://${NODE}:25000/metrics?json" | python3 -c \
    "import sys,json; m=json.load(sys.stdin); print(m.get('mem-tracker.Process.current-usage-bytes', 0))")
  MEM_LIMIT=$(curl -s "http://${NODE}:25000/metrics?json" | python3 -c \
    "import sys,json; m=json.load(sys.stdin); print(m.get('mem-tracker.Process.limit-bytes', 1))")
  
  PCTUSED=$(echo "scale=2; $MEM_USED * 100 / $MEM_LIMIT" | bc)
  if (( $(echo "$PCTUSED > 85" | bc -l) )); then
    HEALTHY=false
    echo "WARNING: ${NODE} memory at ${PCTUSED}%"
  fi
done

if [ "$HEALTHY" = true ]; then
  curl -fsS -m 10 "https://vigilmon.online/api/push/YOUR_MEMORY_KEY" > /dev/null
fi

Set the Vigilmon heartbeat interval to 5 minutes. If memory exceeds 85% on any node, the heartbeat stops and Vigilmon alerts you.

Step 7: Monitor HDFS Read Throughput

Impala's query performance is tightly coupled to HDFS (or object storage) read throughput. A drop in storage read throughput — caused by a slow HDFS NameNode, DataNode failures, or network congestion — causes query slowdowns that look like Impala problems but are actually storage issues.

Monitor the HDFS NameNode health as a proxy for storage cluster health:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://namenode.yourdomain.com:9870/dfshealth.html.
  3. Expected HTTP status: 200.
  4. Keyword check: enter Live Nodes to verify the HDFS cluster has live DataNodes.
  5. Check interval: 2 minutes.
  6. Click Save.

Also monitor the HDFS DataNode health (run on each DataNode):

  1. Click Add Monitor → TCP Port.
  2. Host: datanode1.yourdomain.com, Port: 9866 (HDFS DataNode transfer port).
  3. Check interval: 2 minutes.
  4. Click Save.

Step 8: Monitor Hive Metastore Connectivity

Impala's catalogd reads table schemas and partition metadata from the Hive Metastore. If the Metastore becomes unavailable, REFRESH and INVALIDATE METADATA operations fail, and new tables created outside Impala (e.g., by Spark or Hive) won't be visible.

Monitor the Hive Metastore Thrift port (default 9083):

  1. Click Add Monitor → TCP Port.
  2. Host: metastore-host.yourdomain.com, Port: 9083.
  3. Check interval: 1 minute.
  4. Set Alert after: 2 consecutive failures.
  5. Click Save.

Step 9: Configure Alerting

Open Alert Channels in Vigilmon and configure your notification preferences:

Critical alerts (immediate, any time):

  • Any impalad TCP port failure (reduces executor capacity)
  • Statestored down (cluster loses membership awareness)
  • Catalogd down (metadata distribution stops)
  • Hive Metastore TCP failure

Warning alerts (business hours):

  • Admission control heartbeat stops (queries queuing)
  • Memory usage heartbeat stops (node approaching limit)
  • HDFS NameNode Web UI down

Recommended alert thresholds:

  • Statestored: alert after 1 failure (singleton — no redundancy)
  • Catalogd: alert after 1 failure (singleton — no redundancy)
  • Impalad: alert after 2 consecutive failures (brief restarts are normal)
  • Heartbeat monitors: alert after 1 missed expected ping

In each monitor's Alert Settings, set an Alert cooldown of 10 minutes to prevent notification floods if a whole rack of nodes goes down simultaneously.

Summary: Your Impala Monitoring Stack

| Monitor | Type | What It Catches | |---|---|---| | Impalad TCP :21000 (per node) | TCP Port | Executor daemon crash | | Impalad TCP :21050 (per node) | TCP Port | JDBC/BI tool connectivity loss | | Impalad Web UI :25000/healthz | HTTP | Daemon web interface failure | | Statestored :25010/healthz | HTTP | Cluster membership service crash | | Catalogd :25020/healthz | HTTP | Metadata distribution crash | | Admission control heartbeat | Heartbeat | Query queue saturation | | Query success rate heartbeat | Heartbeat | Elevated query failure rate | | Memory usage heartbeat | Heartbeat | Impalad memory exhaustion | | HDFS NameNode :9870 | HTTP | Storage cluster degradation | | Hive Metastore TCP :9083 | TCP Port | Metadata service unavailability |

Impala is a distributed system where silent partial failures — a single impalad down, catalogd stale, admission pool saturated — degrade performance without obvious errors. Vigilmon makes every daemon and service boundary observable, so you catch issues before your data analysts file tickets about slow queries.

Get started free at vigilmon.online.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles