tutorial

Monitoring Grafana Alloy with Vigilmon

Grafana Alloy is your OpenTelemetry pipeline — but when the collector goes down, your entire observability stack goes dark. Here's how to monitor Alloy process health, component state, scrape success, and export throughput with Vigilmon.

Grafana Alloy is the OpenTelemetry-native observability collector that pipelines metrics, traces, and logs to your backends. When Alloy is healthy, it's invisible — data flows. When Alloy fails, the damage is invisible too: your dashboards go stale, your traces stop, your logs stop arriving, and your alerts stop firing. You need a monitoring layer outside your Alloy pipeline to watch the pipeline itself. Vigilmon is that external layer — HTTP health checks and heartbeats that tell you when Alloy goes dark before your dashboards do.

What You'll Set Up

  • Alloy process health and readiness monitoring
  • Component health status per pipeline stage
  • Scrape success rate heartbeats
  • OTLP ingestion throughput tracking
  • Data export success rate monitoring
  • WAL queue depth alerting
  • Cluster health monitoring (if clustering enabled)
  • Configuration reload success tracking
  • Memory usage monitoring

Prerequisites

  • Grafana Alloy installed (standalone, Docker, or Kubernetes)
  • Alloy UI accessible (default port 12345)
  • A free Vigilmon account

Why Monitoring Alloy Matters

Alloy occupies a uniquely dangerous position in your stack: it's the component responsible for all observability data collection. A crashed Alloy process creates a silent observability blind spot — your metrics dashboards go stale, your trace backends stop receiving spans, and your log aggregation platform stops receiving logs. Meanwhile, every downstream alert that depends on those signals stops firing. The failure is self-concealing.

The only reliable way to monitor an observability pipeline is with a monitoring layer that operates outside that pipeline. Vigilmon's HTTP checks and heartbeats are independent of Alloy — they'll alert you even when Alloy is completely down and your internal observability is dark.

Key Metrics to Monitor

| Metric | Why It Matters | Alert Threshold | |--------|---------------|-----------------| | Alloy HTTP liveness (/readyz) | Process is alive | Any failure | | Alloy readiness (/-/healthy) | Pipeline is ready | Any failure | | Component health per stage | Individual component failures | Any unhealthy | | Scrape success rate | Metrics collection coverage | < 95% | | OTLP received spans/min | Trace ingestion | Drop to zero | | OTLP received metrics/min | Metrics ingestion | Drop to zero | | Export success rate per exporter | Data reaching backends | < 99% | | WAL queue depth | Export backend backpressure | Growing trend | | Cluster member count | Horizontal scale coverage | Below expected | | Configuration reload success | Hot-reload failures | Any failure | | Process RSS memory | Cardinality explosion | Growing unboundedly |

Step 1: Monitor Alloy Liveness and Readiness

Alloy exposes two built-in HTTP endpoints on its UI port (default 12345):

  • /:12345/-/healthy — returns 200 when the Alloy process is healthy
  • /:12345/readyz — returns 200 when Alloy is ready to serve traffic

Add two Vigilmon HTTP monitors:

Monitor 1 — Liveness:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-alloy-host:12345/-/healthy
  3. Check interval: 1 minute
  4. Expected status: 200
  5. Click Save.

Monitor 2 — Readiness:

  1. Click Add Monitor → HTTP / HTTPS.
  2. URL: http://your-alloy-host:12345/readyz
  3. Check interval: 1 minute
  4. Expected status: 200
  5. Click Save.

If Alloy runs on a private network (not publicly accessible), use a Vigilmon heartbeat instead: a probe script running on the same host sends periodic pings. See Step 2 for the heartbeat pattern.

Alert condition: immediate notification on any failure — an Alloy liveness failure means your entire observability pipeline is dark.

Step 2: Monitor Component Health Per Pipeline Stage

Alloy exposes component status via its API. Each component in your River/Alloy DSL pipeline has an individual health state. A single failed component (e.g., a Prometheus scrape job or an OTLP exporter) can silently drop data for that pipeline branch.

Query component health via Alloy's API and send a heartbeat to Vigilmon:

#!/bin/bash
# /usr/local/bin/alloy-component-check.sh
ALLOY_HOST="http://localhost:12345"
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_COMPONENT_HEARTBEAT_ID"

# Fetch component health from Alloy API
COMPONENTS=$(curl -s "$ALLOY_HOST/api/v0/web/components" 2>/dev/null)

if [ -z "$COMPONENTS" ]; then
    echo "Failed to fetch component status — Alloy may be down"
    exit 1
fi

# Check for any unhealthy components
UNHEALTHY=$(echo "$COMPONENTS" | python3 -c "
import sys, json
data = json.load(sys.stdin)
unhealthy = [c['id'] for c in data.get('components', []) if c.get('health', {}).get('state') not in ['healthy', 'exited']]
print(len(unhealthy))
if unhealthy:
    print('Unhealthy:', unhealthy, file=sys.stderr)
" 2>&1)

UNHEALTHY_COUNT=$(echo "$UNHEALTHY" | head -1)

if [ "$UNHEALTHY_COUNT" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
    echo "All components healthy"
else
    echo "ALERT: $UNHEALTHY_COUNT unhealthy components"
    # Heartbeat NOT sent — Vigilmon alerts
fi

Add to cron (every 2 minutes):

*/2 * * * * /usr/local/bin/alloy-component-check.sh

Configure the Vigilmon heartbeat with a 5-minute grace period — if not received within 5 minutes, alert immediately.

Step 3: Monitor Scrape Success Rate

If Alloy is collecting Prometheus metrics via prometheus.scrape components, scrape failures silently drop data for affected targets. Monitor scrape success rate:

#!/bin/bash
# /usr/local/bin/alloy-scrape-check.sh
ALLOY_HOST="http://localhost:12345"
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_SCRAPE_HEARTBEAT_ID"

# Alloy exposes Prometheus metrics including scrape stats
# Query the self-metrics endpoint
METRICS=$(curl -s "$ALLOY_HOST/metrics" 2>/dev/null)

# Extract scrape success/failure counts
SUCCESS=$(echo "$METRICS" | grep 'prometheus_target_scrape_pool_reloads_total' | tail -1 | awk '{print $2}' || echo "0")
FAILED=$(echo "$METRICS" | grep 'prometheus_target_scrape_pool_reloads_failed_total' | tail -1 | awk '{print $2}' || echo "0")

# Calculate failure rate (simplified — check last-scrape success)
LAST_SCRAPE_ERRORS=$(echo "$METRICS" | grep 'up{' | grep ' 0$' | wc -l)
LAST_SCRAPE_TOTAL=$(echo "$METRICS" | grep 'up{' | wc -l)

if [ "$LAST_SCRAPE_TOTAL" -gt 0 ]; then
    FAILURE_RATE=$(( LAST_SCRAPE_ERRORS * 100 / LAST_SCRAPE_TOTAL ))
    if [ "$FAILURE_RATE" -lt 5 ]; then
        curl -s "$HEARTBEAT_URL" > /dev/null
        echo "Scrape success rate OK: ${FAILURE_RATE}% failure"
    else
        echo "ALERT: Scrape failure rate ${FAILURE_RATE}% (threshold: 5%)"
    fi
else
    # No scrape targets configured — send heartbeat (not an error state)
    curl -s "$HEARTBEAT_URL" > /dev/null
fi

Alert threshold: scrape failure rate > 5% — some targets are consistently unreachable.

Step 4: Monitor OTLP Ingestion Throughput

If Alloy receives OTLP data (spans, metrics, logs) from your services, a drop in ingestion throughput is a critical signal: your applications may have stopped exporting data, or Alloy's OTLP receiver is failing.

#!/bin/bash
# /usr/local/bin/alloy-otlp-check.sh
ALLOY_HOST="http://localhost:12345"
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_OTLP_HEARTBEAT_ID"
STATE_FILE="/tmp/alloy-otlp-last-count"

# Read current OTLP received count from Alloy self-metrics
METRICS=$(curl -s "$ALLOY_HOST/metrics" 2>/dev/null)

# Extract received spans count (otelcol metrics)
CURRENT_SPANS=$(echo "$METRICS" | grep 'otelcol_receiver_accepted_spans_total' | \
  awk '{sum += $2} END {print sum}')
CURRENT_SPANS=${CURRENT_SPANS:-0}

# Compare with last reading
LAST_SPANS=$(cat "$STATE_FILE" 2>/dev/null || echo "0")
echo "$CURRENT_SPANS" > "$STATE_FILE"

DELTA=$(echo "$CURRENT_SPANS - $LAST_SPANS" | bc)

# If we received any new spans in the last check period, all is well
# Adjust threshold based on your expected OTLP ingestion rate
if [ "$DELTA" -gt 0 ] || [ "$LAST_SPANS" -eq 0 ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
    echo "OTLP ingestion OK: +${DELTA} spans"
else
    echo "ALERT: No new OTLP spans received (throughput dropped to zero)"
    # Heartbeat NOT sent
fi

Note: Adjust this check based on your actual traffic pattern. If you have expected quiet periods (e.g., nightly maintenance windows), exclude them from the alerting window.

Step 5: Monitor Data Export Success Rate

Alloy's export components (prometheus.remote_write, loki.write, otelcol.exporter.otlp) can fail independently of the ingestion side. A failed exporter means data is being received but not delivered to your backends. Alloy buffers in its WAL (Write-Ahead Log) but eventually drops data if the backend is unavailable for too long.

#!/bin/bash
# /usr/local/bin/alloy-export-check.sh
ALLOY_HOST="http://localhost:12345"
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_EXPORT_HEARTBEAT_ID"

METRICS=$(curl -s "$ALLOY_HOST/metrics" 2>/dev/null)

# Check Prometheus remote_write failures
RW_FAILED=$(echo "$METRICS" | grep 'prometheus_remote_storage_samples_failed_total' | \
  awk '{sum += $2} END {print sum}')
RW_SENT=$(echo "$METRICS" | grep 'prometheus_remote_storage_samples_total' | \
  awk '{sum += $2} END {print sum}')

# Check OTLP export failures
OTLP_FAILED=$(echo "$METRICS" | grep 'otelcol_exporter_send_failed_metric_points_total' | \
  awk '{sum += $2} END {print sum}')
OTLP_SENT=$(echo "$METRICS" | grep 'otelcol_exporter_sent_metric_points_total' | \
  awk '{sum += $2} END {print sum}')

RW_FAILED=${RW_FAILED:-0}
RW_SENT=${RW_SENT:-1}
OTLP_FAILED=${OTLP_FAILED:-0}
OTLP_SENT=${OTLP_SENT:-1}

RW_FAIL_RATE=$(echo "scale=2; $RW_FAILED * 100 / ($RW_SENT + 1)" | bc)
OTLP_FAIL_RATE=$(echo "scale=2; $OTLP_FAILED * 100 / ($OTLP_SENT + 1)" | bc)

# Alert if either export failure rate exceeds 1%
if (( $(echo "$RW_FAIL_RATE < 1" | bc -l) )) && (( $(echo "$OTLP_FAIL_RATE < 1" | bc -l) )); then
    curl -s "$HEARTBEAT_URL" > /dev/null
    echo "Export OK — RW fail: ${RW_FAIL_RATE}%, OTLP fail: ${OTLP_FAIL_RATE}%"
else
    echo "ALERT: Export failures — RW: ${RW_FAIL_RATE}%, OTLP: ${OTLP_FAIL_RATE}%"
fi

Step 6: Monitor WAL Queue Depth

Alloy uses a Write-Ahead Log (WAL) to buffer data when export backends are temporarily unavailable. A growing WAL queue depth indicates that the export backend is unavailable and Alloy is accumulating data it cannot deliver.

#!/bin/bash
# /usr/local/bin/alloy-wal-check.sh
ALLOY_HOST="http://localhost:12345"
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_WAL_HEARTBEAT_ID"
WAL_WARN_THRESHOLD=50000  # samples

METRICS=$(curl -s "$ALLOY_HOST/metrics" 2>/dev/null)

# WAL queue depth
WAL_QUEUE=$(echo "$METRICS" | grep 'prometheus_remote_storage_queue_highest_sent_timestamp_seconds' | \
  awk '{print $2}')
WAL_PENDING=$(echo "$METRICS" | grep 'prometheus_remote_storage_pending_samples' | \
  awk '{sum += $2} END {print sum}')
WAL_PENDING=${WAL_PENDING:-0}

if [ "$WAL_PENDING" -lt "$WAL_WARN_THRESHOLD" ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
    echo "WAL queue OK: $WAL_PENDING pending samples"
else
    echo "ALERT: WAL queue depth high: $WAL_PENDING pending samples (threshold: $WAL_WARN_THRESHOLD)"
fi

A growing WAL is a leading indicator of data loss — Alloy will eventually drop the oldest WAL segments to bound disk usage.

Step 7: Monitor Alloy Cluster Health

If you run Alloy in clustering mode (multiple Alloy instances sharing workload), cluster membership changes indicate coverage gaps. A leaving member means that member's scrape targets are unmonitored until another member picks them up.

#!/bin/bash
# /usr/local/bin/alloy-cluster-check.sh
ALLOY_HOST="http://localhost:12345"
EXPECTED_MEMBERS=3  # adjust to your cluster size
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_CLUSTER_HEARTBEAT_ID"

# Fetch cluster peers from Alloy API
PEERS=$(curl -s "$ALLOY_HOST/api/v0/web/peers" 2>/dev/null)
MEMBER_COUNT=$(echo "$PEERS" | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(len(data.get('peers', [])) + 1)  # +1 for self
" 2>/dev/null || echo "0")

if [ "$MEMBER_COUNT" -ge "$EXPECTED_MEMBERS" ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
    echo "Cluster OK: $MEMBER_COUNT members"
else
    echo "ALERT: Cluster degraded — expected $EXPECTED_MEMBERS, got $MEMBER_COUNT"
fi

Skip this check if you run Alloy in standalone (non-clustered) mode.

Step 8: Monitor Configuration Reload Success

Alloy supports hot-reloading configuration without restart. A failed configuration reload leaves Alloy running on the previous (potentially stale) configuration. Monitor reload success:

#!/bin/bash
# /usr/local/bin/alloy-reload-check.sh
ALLOY_HOST="http://localhost:12345"
HEARTBEAT_URL="https://vigilmon.online/api/heartbeat/YOUR_RELOAD_HEARTBEAT_ID"

METRICS=$(curl -s "$ALLOY_HOST/metrics" 2>/dev/null)

RELOAD_SUCCESS=$(echo "$METRICS" | grep 'alloy_config_last_reload_successful' | awk '{print $2}')
RELOAD_SUCCESS=${RELOAD_SUCCESS:-1}  # Default to 1 if metric not present (no reload attempted)

if [ "$RELOAD_SUCCESS" = "1" ] || [ -z "$RELOAD_SUCCESS" ]; then
    curl -s "$HEARTBEAT_URL" > /dev/null
    echo "Config reload status: OK"
else
    echo "ALERT: Last configuration reload failed"
fi

Run every 5 minutes. Alert on any reload failure — Alloy may be operating on outdated pipeline configuration.

Step 9: Configure Alerting

Set up your Vigilmon notification channels:

  1. Go to Settings → Notifications.
  2. Add Slack, email, PagerDuty, or OpsGenie.
  3. Route by severity.

Recommended routing for Alloy:

| Monitor | Severity | Notification | |---------|----------|-------------| | Alloy process down | Critical | Immediate: Slack + PagerDuty | | Any component unhealthy | High | Immediate: Slack | | OTLP throughput → zero | Critical | Immediate: Slack + PagerDuty | | Export failure > 1% | High | Slack | | WAL queue growing | High | Slack | | Cluster member missing | High | Slack | | Scrape failure > 5% | Medium | Email | | Config reload failure | Medium | Slack | | Memory growing | Medium | Email |

Step 10: Test Your Monitors

Before relying on these monitors:

# Test Alloy liveness alert
sudo systemctl stop alloy
# → Vigilmon HTTP monitor alerts within 1-2 minutes
sudo systemctl start alloy

# Test component failure detection
# Edit alloy config to add a bad scrape target, then reload
# → Component check script should detect unhealthy component

# Test WAL alert
# Stop your Prometheus/Mimir remote_write target temporarily
# → WAL pending samples will grow; heartbeat withheld

# Verify scrape check
# Take a target offline temporarily
# → up{} metric becomes 0 for that target; failure rate increases

Conclusion

Grafana Alloy is the foundation of your observability stack — which makes it uniquely important to monitor with an external tool. When Alloy fails, your internal dashboards, alerts, and traces all fail with it. Vigilmon's external HTTP checks and heartbeat monitors give you visibility into Alloy's health that doesn't depend on Alloy being healthy to work.

Start with the liveness and readiness checks (Step 1) — they catch the most impactful failure (complete Alloy crash) immediately. Then add component health monitoring (Step 2) to catch per-pipeline-branch failures. Export success rate and WAL depth monitoring (Steps 5-6) catch the silent data loss scenarios that uptime checks miss.

Start monitoring Grafana Alloy with Vigilmon →

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles