Monitoring Apache Geode with Vigilmon

Apache Geode is the in-memory data platform behind some of the highest-throughput distributed systems in production — and the open-source core of VMware Tanzu GemFire. Applications that depend on Geode for real-time key-value access, distributed caching, and co-located function execution need Geode to be fast, balanced, and available at all times. Vigilmon gives you continuous coverage of locator and server health, Region data load, cache hit rates, JVM heap pressure, WAN gateway replication lag, and disk store integrity — so a JVM GC storm or a failed locator doesn't take your application by surprise.

What You'll Set Up

Locator process health monitoring
Cache server member health checks
Region data load and entry count monitoring
Cache hit rate alerting
JVM heap and GC pressure monitoring
Partition Region balance checks
WAN gateway sender queue monitoring
Disk store write health alerts
Client connection count tracking

Prerequisites

Apache Geode 1.14+ or VMware Tanzu GemFire 9.x+ deployed
Geode management REST API enabled (default port 7070 or your configured management port)
At least one locator and one server running
A free Vigilmon account

Why Monitoring Apache Geode Matters

Geode is designed to be fast and distributed, but those properties create specific failure modes that passive monitoring misses:

Locator loss is catastrophic. Locators are the membership coordinators of the Geode cluster. Without a locator, new clients cannot connect and servers cannot discover each other. A single-locator deployment has no HA — and even a two-locator deployment where one fails silently creates a single point of failure.
JVM heap eviction is silent data loss. When a Geode Region's JVM heap exceeds the eviction threshold, Geode begins evicting entries. For non-persistent Regions, evicted entries are gone. An application reading an evicted entry gets a cache miss, falls back to the database (or fails), and never knows data was silently dropped.
Partition Region imbalance causes hot spots. When servers are added to or removed from a Geode cluster, Regions need to rebalance. If rebalancing is not triggered, some servers hold more buckets than others, creating GC pressure hot spots and uneven memory usage.
WAN gateway queue growth means replication lag. For multi-datacenter deployments, a WAN gateway sender queue that keeps growing means the remote site is not receiving updates. During a network partition, this queue grows unbounded and can cause out-of-memory failures.

Step 1: Monitor Locator Health

Locators are the most critical Geode processes. Geode exposes a management REST API on each locator. Monitor it as your primary cluster health signal:

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
Enter URL: http://your-locator-host:7070/management/v1/ping
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Under Alert conditions, set Alert after to 1 failure.
Click Save.

If you run multiple locators (recommended for production), add a separate monitor for each locator's management API. A locator failure alerts within 1 minute — well before clients start failing to connect.

For TCP-level locator monitoring (faster than HTTP):

Click Add Monitor → set Type to TCP Port.
Enter Host: your-locator-host, Port: 10334 (default locator port).
Set Check interval to 30 seconds.
Enable alerts on connection failure.
Click Save.

Step 2: Monitor Cache Server Member Health

Monitor the Geode management REST API cluster members endpoint to detect server failures:

Click Add Monitor → set Type to HTTP / HTTPS.
Enter URL: http://your-locator-host:7070/management/v1/members
Set Expected HTTP status to 200.
Set Check interval to 1 minute.
Under Keyword check, add your expected server name (e.g., server1) to verify it is present in the member list response.
Click Save.

Create one keyword-check monitor per critical server member. A missing server name in the response means that member has left the cluster.

Step 3: Monitor Region Data Load and Entry Count

Geode's management REST API exposes per-Region statistics. Monitor Region entry count and memory usage to detect unexpected data loss or approaching eviction thresholds:

Use a cron-driven heartbeat script:

#!/bin/bash
# Check Region size and send heartbeat if within expected range
RESPONSE=$(curl -s "http://your-locator-host:7070/management/v1/regions/YOUR-REGION-NAME")
ENTRY_COUNT=$(echo "$RESPONSE" | jq '.entryCount')

if [ -n "$ENTRY_COUNT" ] && [ "$ENTRY_COUNT" -gt 0 ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_REGION_HEARTBEAT_SLUG"
fi

Schedule with cron every 5 minutes. If entry count drops to zero (Region wipe) or the management API becomes unresponsive, the heartbeat is missed and Vigilmon alerts.

For heap usage monitoring, expose JVM heap metrics via JMX or Prometheus and alert on heap usage above 80%:

- alert: GeodeHighHeapUsage
  expr: jvm_memory_used_bytes{area="heap", job="geode"} / jvm_memory_max_bytes{area="heap", job="geode"} > 0.8
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Geode JVM heap > 80%"
    description: "Server {{ $labels.instance }} heap is at {{ $value | humanizePercentage }}. Eviction may begin soon."

Step 4: Monitor Cache Hit Rate

Cache hit rate is the primary indicator of whether your Geode deployment is doing its job. A hit rate below 80% means too many requests are falling through to your backing store (database). This can be caused by cache misses from a misconfigured TTL, a Region wipe, or partition imbalance causing some keys to be unreachable.

Expose cache hit rate through a synthetic health endpoint in your application:

#!/bin/bash
# Query Geode management API for cache statistics
HIT_RATE=$(curl -s "http://your-locator-host:7070/management/v1/members/your-server/caches" \
  | jq '.hitRatio')

# Alert if hit rate drops below 0.8
if (( $(echo "$HIT_RATE < 0.8" | bc -l) )); then
  echo "Cache hit rate critical: $HIT_RATE"
  # Do NOT send heartbeat — alert fires on missed ping
else
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_HIT_RATE_HEARTBEAT"
fi

Configure the Vigilmon heartbeat with a 10-minute expected interval and a 1-missed-ping alert threshold.

Step 5: Monitor GC Pressure Per Server

Geode is JVM-based and GC pauses directly affect read/write latency. Long GC pauses (>200ms) can trigger cluster membership suspects and potentially cause servers to be removed from the cluster by the membership protocol.

Export JVM GC metrics via Prometheus JMX exporter and alert on GC pause duration:

- alert: GeodeHighGCPause
  expr: rate(jvm_gc_collection_seconds_sum{job="geode"}[5m]) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Geode server {{ $labels.instance }} GC pause rate is high"
    description: "GC is consuming {{ $value | humanizePercentage }} of CPU time. Consider increasing heap or reviewing Region eviction policies."

Step 6: Monitor Partition Region Balance

Partitioned Regions shard data across servers by key hash. After a server join or leave event, buckets are not automatically rebalanced unless triggered. Imbalance causes some servers to hold more data and serve more requests than others.

Check bucket distribution via the management API and alert on imbalance:

#!/bin/bash
# Get bucket counts per member for a partitioned Region
MEMBERS=$(curl -s "http://your-locator-host:7070/management/v1/regions/YOUR-PARTITIONED-REGION/members")

MAX=$(echo "$MEMBERS" | jq '[.[].bucketCount] | max')
MIN=$(echo "$MEMBERS" | jq '[.[].bucketCount] | min')
AVG=$(echo "$MEMBERS" | jq '[.[].bucketCount] | add / length')

# Alert if max bucket count is >20% above average
THRESHOLD=$(echo "$AVG * 1.2" | bc -l)
if (( $(echo "$MAX > $THRESHOLD" | bc -l) )); then
  echo "Partition imbalance detected: max=$MAX avg=$AVG"
  # Trigger rebalance via Geode gfsh: rebalance --include-region=/YOUR-REGION
fi

Run this check every 30 minutes via cron. Send a Vigilmon heartbeat when balance is healthy; let it lapse when imbalance is detected.

Step 7: Monitor WAN Gateway Sender Queue

For multi-datacenter Geode deployments, WAN gateway senders replicate Region events to remote Geode clusters. A growing sender queue means the remote site is not receiving updates and you are accumulating replication lag that will cause data divergence between sites.

#!/bin/bash
# Check WAN gateway sender queue size
QUEUE_SIZE=$(curl -s "http://your-locator-host:7070/management/v1/gateways/senders/YOUR-SENDER-ID" \
  | jq '.eventQueueSize')

if [ -n "$QUEUE_SIZE" ] && [ "$QUEUE_SIZE" -lt 1000 ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_WAN_HEARTBEAT"
else
  echo "WAN gateway queue size: $QUEUE_SIZE — not sending heartbeat"
fi

Set the heartbeat expected interval to 5 minutes. If queue size exceeds 1000 events, the heartbeat is withheld and Vigilmon alerts.

Step 8: Monitor Disk Store Health

For persistent Regions backed by disk stores, disk write failures cause server crashes. Monitor disk store health by checking available disk space on the data directory mount point:

Click Add Monitor → set Type to HTTP / HTTPS.
Enter the URL of a custom disk health endpoint your application exposes (e.g., a simple HTTP handler that checks available disk space and returns 200/503).
Set Check interval to 5 minutes.
Enable alerts on non-200 response.
Click Save.

For disk latency, export JMX disk store metrics via Prometheus:

- alert: GeodeDiskStoreHighLatency
  expr: geode_disk_store_writes_latency_ms{job="geode"} > 100
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Geode disk store write latency > 100ms"
    description: "Disk store on {{ $labels.instance }} is slow. Check I/O saturation."

Step 9: Monitor Client Connection Count

The number of active client connections is a proxy for application health. A sudden spike may indicate a connection leak; a sudden drop may indicate a mass client failure.

#!/bin/bash
CONNECTIONS=$(curl -s "http://your-locator-host:7070/management/v1/members/your-server" \
  | jq '.clientConnectionCount')

echo "Active client connections: $CONNECTIONS"
curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_CONNECTIONS_HEARTBEAT"

Schedule every 5 minutes. Set the Vigilmon heartbeat expected interval to 10 minutes so a single script failure does not immediately alert. Review connection counts in Vigilmon's heartbeat history to spot trends.

Step 10: Configure Alerting

Go to Alert Policies → click New Policy.
Name it Geode Cluster Alerts.
Add your email, Slack webhook, or PagerDuty integration.
Assign this policy to all Geode monitors (locator health, server health, WAN gateway, and disk store).
For the locator and server monitors, set Alert after to 1 failure — these are critical.
For heartbeat monitors (hit rate, balance, WAN queue), use the default 1-missed-ping threshold.
Click Save.

Alert Reference

| Monitor | Alert Threshold | Severity | Impact | |---|---|---|---| | Locator TCP :10334 | Connection refused | Critical | No new client connections or server discovery | | Locator management API /ping | Non-200 | Critical | Management API unavailable | | Server member in /members | Name missing | Critical | Server left cluster | | JVM heap usage | > 80% | High | Region eviction starting | | Cache hit rate | < 80% | High | Backing store under load | | GC pause rate | > 10% CPU | Warning | Read/write latency degraded | | Partition bucket imbalance | Max > 120% avg | Warning | GC hot spots forming | | WAN gateway queue | > 1000 events | High | Remote site replication lagging | | Disk store latency | > 100ms | Warning | I/O bottleneck on persistence | | Client connection count | Spike/drop > 50% | Warning | Connection leak or mass client failure |

Conclusion

Apache Geode's in-memory design makes it fast under normal conditions but sensitive to JVM heap pressure, partition imbalance, and locator availability. The monitoring setup in this guide covers the full failure surface: locator death, server eviction from the cluster, Region data loss, cache performance degradation, WAN replication lag, and disk store health.

Start with the locator TCP monitor and the server member keyword check — those two monitors cover the most severe failure modes with zero infrastructure changes. Then add JVM heap alerting and the cache hit rate heartbeat as you tune your Region configurations for production load.