tutorial

How to Monitor Arkime (Moloch) Health with Vigilmon

Arkime Capture process failures, PCAP disk saturation, and Elasticsearch indexing lag silently stop full-packet capture and make forensic investigation impossible during active incidents. Learn how to monitor Arkime Capture health, PCAP ingestion rates, Elasticsearch indexing latency, and Viewer availability with Vigilmon HTTP probes and heartbeat monitors.

Arkime (formerly Moloch) is the open source large-scale full-PCAP capture, indexing, and analysis system used in network security operations centers for forensic investigation. When the Arkime Capture daemon crashes on a high-speed sensor node because the libpcap ring buffer overflows at sustained 40 Gbps traffic and the process does not restart automatically, raw PCAP files stop being written to disk and session metadata stops flowing to Elasticsearch; the Arkime Viewer continues serving the web UI and returning results for historical sessions, making the failure invisible to analysts who are not actively looking at the current-time session window; when Elasticsearch indexing latency climbs from 200ms to 12 seconds because a data node is overloaded by a traffic burst writing 500,000 sessions per minute, new sessions are not searchable in near-real-time; during an active incident, an analyst searching for sessions from 3 minutes ago finds nothing and concludes the threat actor has stopped activity — when in reality the sessions exist but have not yet been indexed; when the PCAP storage disk on a capture node hits 90% utilization, Arkime's rotation policy begins deleting the oldest PCAP at an accelerating rate to reclaim space — in a busy environment, the retention window can collapse from 30 days to 6 hours within a single work shift, making the PCAP unavailable exactly when forensic investigators need to retrieve a session from last week's incident.

Vigilmon gives you external visibility into Arkime's multi-component health through HTTP probe monitoring and heartbeat monitors for Capture process health, PCAP ingestion rates, Elasticsearch indexing performance, and Viewer availability. This tutorial covers the complete Arkime monitoring stack.

Why Arkime Needs External Monitoring

Arkime failure modes are particularly treacherous because they produce false confidence — the Viewer continues working with historical data even when live capture has stopped:

  • Arkime Capture crash with no restart: Arkime Capture is a single C process (moloch-capture or arkime-capture) that runs on sensor nodes; when it crashes due to buffer overflow, OOM kill, or a network interface error, PCAP capture stops; because the Viewer continues to serve historical data from Elasticsearch, the platform appears healthy to analysts browsing sessions from earlier in the day; the gap in capture coverage is only visible when searching for sessions in the current time window
  • libpcap packet drops at high throughput: At high traffic rates (10–100 Gbps), libpcap's ring buffer can overflow if the Capture process cannot write to disk fast enough; libpcap drop counters increment silently; the Capture process continues running and appears healthy, but a percentage of network traffic is not captured; forensic investigators may search for sessions that existed on the wire but were never written to PCAP
  • Elasticsearch indexing lag: Arkime Capture sends session metadata to Elasticsearch via bulk indexing; when Elasticsearch is overloaded (large shard count, JVM GC pressure, a disk-bound data node), bulk indexing latency climbs; sessions that have been captured to PCAP on disk are not yet indexed and therefore not searchable via the Viewer; during an active incident, this lag appears to analysts as missing traffic
  • PCAP disk saturation and aggressive rotation: Arkime manages PCAP rotation based on disk usage thresholds; when disk usage approaches 100%, Arkime begins deleting older PCAP files to stay below the threshold; in environments with high traffic volume, this rotation can reduce retention from days to hours without any notification; the PCAP exists on disk (as indexed sessions) but the files are gone when retrieved
  • Viewer unavailability during database failover: The Arkime Viewer is a Node.js application that queries Elasticsearch; when Elasticsearch is undergoing a rolling restart or a node is being replaced, the Viewer may return 500 errors or empty result sets for sessions that exist in the index; analysts investigating an incident see no results and may incorrectly conclude the session was not captured
  • NIC ring buffer drops at high speed: High-speed PCAP capture requires NIC ring buffer tuning (ethtool); when the NIC ring buffer is undersized for the traffic rate, the NIC drops packets at the hardware layer before libpcap can read them; NIC drops appear as RX missed in ethtool -S statistics; these are distinct from libpcap drops and require different tuning

External monitoring with Vigilmon adds:

  • Capture process liveness checks that detect crashes within minutes rather than after an analyst notices the time gap in sessions
  • PCAP ingestion rate heartbeats that fire when Gbps throughput drops to zero
  • Elasticsearch indexing latency monitoring that catches lag before it affects investigation timeliness
  • PCAP disk usage alerts before rotation aggressively prunes the forensic window

Step 1: Build an Arkime Health Endpoint

Arkime Viewer exposes a built-in stats API at /_arkime_/api/stats that provides Capture node statistics. Use it directly or build a sidecar.

Using the Arkime Viewer Stats API Directly

# Arkime Viewer stats endpoint (no auth required in default community config)
curl -s 'http://localhost:8005/_arkime_/api/stats?flatten=1' | python3 -m json.tool

The stats API returns per-node Capture statistics including:

  • deltaPackets — packets processed since last stats update
  • deltaBytes — bytes processed
  • deltaDropped — libpcap drops since last update
  • deltaTotalDropped — cumulative drops
  • deltaMS — milliseconds since last update
  • diskQueue — sessions queued for indexing

Python Health Sidecar

# health/arkime_health.py
import os
import requests
from flask import Flask, jsonify
from elasticsearch import Elasticsearch

app = Flask(__name__)

ARKIME_VIEWER   = os.environ.get('ARKIME_VIEWER', 'http://localhost:8005')
ARKIME_USER     = os.environ.get('ARKIME_USER', 'admin')
ARKIME_PASS     = os.environ.get('ARKIME_PASS', '')
ES_HOST         = os.environ.get('ARKIME_ES_HOST', 'localhost')
ES_PORT         = int(os.environ.get('ARKIME_ES_PORT', 9200))
ES_PREFIX       = os.environ.get('ARKIME_ES_PREFIX', 'arkime')

es = Elasticsearch(
    [{'host': ES_HOST, 'port': ES_PORT, 'scheme': 'http'}],
    timeout=10,
)

def get_capture_stats():
    try:
        r = requests.get(
            f'{ARKIME_VIEWER}/_arkime_/api/stats',
            auth=(ARKIME_USER, ARKIME_PASS),
            timeout=10,
        )
        if not r.ok:
            return {'error': f'HTTP {r.status_code}'}
        data = r.json()
        nodes = data.get('data', [])
        total_packets = sum(n.get('deltaPackets', 0) for n in nodes)
        total_dropped = sum(n.get('deltaDropped', 0) for n in nodes)
        total_bytes   = sum(n.get('deltaBytes', 0) for n in nodes)
        drop_rate_pct = (total_dropped / max(total_packets + total_dropped, 1)) * 100
        return {
            'nodes': len(nodes),
            'delta_packets': total_packets,
            'delta_bytes': total_bytes,
            'delta_dropped': total_dropped,
            'drop_rate_percent': round(drop_rate_pct, 3),
            'node_names': [n.get('nodeName') for n in nodes],
        }
    except Exception as e:
        return {'error': str(e)}

def get_es_indexing_lag():
    try:
        # Check how fresh the most recent indexed session is
        result = es.search(
            index=f'{ES_PREFIX}_sessions3-*',
            body={
                'query': {'match_all': {}},
                'sort': [{'lastPacket': {'order': 'desc'}}],
                'size': 1,
                '_source': ['lastPacket'],
            },
        )
        hits = result.get('hits', {}).get('hits', [])
        if not hits:
            return {'lag_seconds': -1, 'error': 'no recent sessions'}
        import time
        last_packet_ms = hits[0]['_source'].get('lastPacket', 0)
        now_ms = int(time.time() * 1000)
        lag_s = (now_ms - last_packet_ms) / 1000.0
        return {'lag_seconds': round(lag_s, 1)}
    except Exception as e:
        return {'error': str(e)}

def get_viewer_health():
    try:
        r = requests.get(
            f'{ARKIME_VIEWER}/',
            auth=(ARKIME_USER, ARKIME_PASS),
            timeout=10,
            allow_redirects=True,
        )
        return {'reachable': r.ok, 'status_code': r.status_code}
    except Exception as e:
        return {'reachable': False, 'error': str(e)}

def get_es_disk_usage():
    try:
        stats = es.cluster.stats()
        fs = stats['nodes']['fs']
        total = fs['total_in_bytes']
        avail = fs['available_in_bytes']
        used  = total - avail
        pct   = (used / total * 100) if total > 0 else 0
        return {'usage_percent': round(pct, 1), 'used_gb': round(used / 1e9, 1), 'total_gb': round(total / 1e9, 1)}
    except Exception as e:
        return {'error': str(e)}

@app.route('/health')
def health():
    capture = get_capture_stats()
    lag     = get_es_indexing_lag()
    viewer  = get_viewer_health()
    disk    = get_es_disk_usage()

    capturing = 'error' not in capture and capture.get('delta_packets', 0) > 0
    low_drop  = capture.get('drop_rate_percent', 100) < 1.0
    low_lag   = 'error' not in lag and 0 < lag.get('lag_seconds', 999) < 30
    viewer_ok = viewer.get('reachable', False)
    disk_ok   = disk.get('usage_percent', 100) < 80

    healthy = capturing and low_drop and low_lag and viewer_ok and disk_ok
    return jsonify({
        'healthy': healthy,
        'capture': capture,
        'indexing_lag': lag,
        'viewer': viewer,
        'es_disk': disk,
    }), (200 if healthy else 503)

@app.route('/health/capture')
def health_capture():
    data = get_capture_stats()
    healthy = 'error' not in data and data.get('delta_packets', 0) > 0 and data.get('drop_rate_percent', 100) < 1.0
    return jsonify({'healthy': healthy, **data}), (200 if healthy else 503)

@app.route('/health/indexing')
def health_indexing():
    data = get_es_indexing_lag()
    healthy = 'error' not in data and 0 < data.get('lag_seconds', 999) < 30
    return jsonify({'healthy': healthy, **data}), (200 if healthy else 503)

@app.route('/health/viewer')
def health_viewer():
    data = get_viewer_health()
    return jsonify(data), (200 if data.get('reachable') else 503)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8767)

pip install flask requests elasticsearch
ARKIME_VIEWER=http://localhost:8005 ARKIME_USER=admin ARKIME_PASS=yourpass \
  python health/arkime_health.py &

Step 2: Monitor Arkime Capture Health

The Capture daemon is the most critical component — when it stops, no new PCAP is written.

HTTP Monitor for Capture Process Health

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Arkime — Capture Health
  3. Set URL: http://your-arkime-host:8767/health/capture
  4. Set Method: GET
  5. Set Expected status: 200
  6. Set Check interval: 60 seconds
  7. Set Regions: at least two probe regions

Alternatively, probe the Arkime Viewer stats API directly:

  • URL: http://your-arkime-host:8005/_arkime_/api/stats
  • Expected status: 200
  • Response assertion: body must not contain "deltaPackets":0 (needs scripted assertion)

Alert Configuration

Alert message: Arkime Capture daemon has stopped or dropped >1% of packets — PCAP capture may have a gap. Check: systemctl status arkime-capture; check /etc/arkime/config.ini for capture interface config

Step 3: Monitor PCAP Ingestion Rate

A drop to zero Gbps ingested indicates Capture has crashed or the monitored network interface has gone down.

Heartbeat Monitor for PCAP Ingestion

# /opt/arkime/scripts/check_pcap_rate.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PCAP_RATE_KEY"
ARKIME_URL="http://localhost:8005"
ARKIME_USER="admin"
ARKIME_PASS="yourpass"

PACKETS=$(curl -sf -u "$ARKIME_USER:$ARKIME_PASS" \
  "$ARKIME_URL/_arkime_/api/stats?flatten=1" \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
total = sum(n.get('deltaPackets', 0) for n in data.get('data', []))
print(total)
" 2>/dev/null || echo 0)

if [ "$PACKETS" -gt "0" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

# /etc/cron.d/arkime-ingestion-heartbeat
* * * * * root /opt/arkime/scripts/check_pcap_rate.sh
chmod +x /opt/arkime/scripts/check_pcap_rate.sh

Configure the Heartbeat in Vigilmon

  1. Open Vigilmon → Heartbeats → New Heartbeat
  2. Set Name: Arkime — PCAP Ingestion Rate
  3. Set Expected interval: 2 minutes
  4. Set Grace period: 3 minutes
  5. Copy the heartbeat URL into the script

Alert message: Arkime PCAP ingestion rate has dropped to zero — Capture may have crashed or the monitored NIC is down. PCAP capture has a gap starting now.

Step 4: Monitor Elasticsearch Indexing Latency

High indexing latency makes sessions unsearchable in near-real-time, causing false gaps during active investigations.

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Arkime — Elasticsearch Indexing Latency
  3. Set URL: http://your-arkime-host:8767/health/indexing
  4. Set Expected status: 200
  5. Set Check interval: 120 seconds

Alert message: Arkime Elasticsearch indexing lag exceeds 30 seconds — sessions are captured to PCAP but not yet searchable. Analysts will see missing data in the current time window. Check ES node health and JVM heap usage.

Step 5: Monitor PCAP Disk Usage

PCAP disk saturation causes aggressive rotation that shrinks the forensic retention window.

Heartbeat for PCAP Disk Usage

# /opt/arkime/scripts/check_pcap_disk.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PCAP_DISK_KEY"
PCAP_DIR="${ARKIME_PCAP_DIR:-/opt/arkime/raw}"

USAGE=$(df "$PCAP_DIR" | awk 'NR==2 {gsub(/%/,""); print $5}')
if [ "$USAGE" -lt "90" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Run every 5 minutes. Configure Vigilmon heartbeat with a 10-minute interval and 12-minute grace period (fires when >90% disk is used for more than 10 minutes).

Alert message: Arkime PCAP storage disk is above 90% — Capture is aggressively rotating old PCAP files. Forensic retention window is shrinking. Add disk capacity or reduce PCAP retention threshold.

Step 6: Monitor Arkime Viewer Availability

Viewer unavailability blocks forensic investigation of active incidents.

  1. Open Vigilmon → Monitors → New Monitor
  2. Set Name: Arkime — Viewer Web Interface
  3. Set URL: http://your-arkime-host:8005/
  4. Set Method: GET
  5. Set Expected status: 200 or 302
  6. Set Check interval: 60 seconds

Alert message: Arkime Viewer web interface is unreachable — forensic analysts cannot search sessions or retrieve PCAP. Check: systemctl status arkime-viewer; check Elasticsearch connectivity

Step 7: Monitor Arkime Parliament Multi-Cluster Health

If running Parliament for multi-cluster management, monitor each cluster's status.

Heartbeat for Parliament Cluster Status

# /opt/arkime/scripts/check_parliament.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_PARLIAMENT_KEY"
PARLIAMENT_URL="http://localhost:8008"

STATUS=$(curl -sf "$PARLIAMENT_URL/api/parliament" \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
groups = data.get('groups', [])
for g in groups:
    for cluster in g.get('clusters', []):
        if cluster.get('status') != 'green':
            sys.exit(1)
sys.exit(0)
" 2>/dev/null; echo $?)

if [ "$STATUS" = "0" ]; then
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Configure with a 2-minute interval and 4-minute grace period.

Alert message: Arkime Parliament reports one or more clusters offline — check Parliament at http://your-parliament-host:8008 for cluster status details

Step 8: Monitor NIC Queue Drops (Per Sensor)

NIC hardware drops are distinct from libpcap drops and indicate the capture NIC is undersized for the traffic rate.

# /opt/arkime/scripts/check_nic_drops.sh
#!/bin/bash
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_NIC_KEY"
CAPTURE_INTERFACE="${ARKIME_IFACE:-eth0}"

# Get RX missed count (NIC drops at hardware ring buffer)
RX_MISSED=$(ethtool -S "$CAPTURE_INTERFACE" 2>/dev/null \
  | awk '/rx_missed|rx_dropped|RX missed/{sum += $NF} END{print sum+0}')

# Get RX packet count for rate calculation
RX_PACKETS=$(cat /proc/net/dev | awk -v iface="$CAPTURE_INTERFACE:" '$1==iface{print $2}')

# Alert if RX missed > 0.1% of traffic in the last check window
# Simple check: only ping if RX_MISSED hasn't changed since last check
PREV_FILE="/tmp/arkime_nic_drops_prev"
PREV=$(cat "$PREV_FILE" 2>/dev/null || echo 0)
echo "$RX_MISSED" > "$PREV_FILE"

NEW_DROPS=$(( RX_MISSED - PREV ))
if [ "$NEW_DROPS" -lt "1000" ]; then  # fewer than 1000 new drops in last minute
  curl -sf "$HEARTBEAT_URL" > /dev/null
fi

Run every minute. Configure with a 3-minute interval.

Alert message: Arkime sensor NIC is dropping packets at the hardware ring buffer — captured PCAP has gaps. Run: ethtool -G $IFACE rx 4096 to increase ring buffer, or reduce traffic rate on this tap.

Step 9: Alerting Configuration Summary

| Monitor | Type | Interval | Alert Threshold | |---------|------|----------|-----------------| | Arkime Capture health | HTTP probe | 60s | Non-200 or zero packets | | Arkime Viewer availability | HTTP probe | 60s | Non-200 | | ES indexing latency | HTTP assertion | 120s | Lag >30 seconds | | PCAP ingestion rate | Heartbeat | 2 min | Missing for 3+ min | | PCAP disk usage (per sensor) | Heartbeat | 5 min | Missing when >90% used | | ES disk usage | HTTP assertion | 300s | >80% | | Parliament cluster health | Heartbeat | 2 min | Missing for 4+ min | | NIC ring buffer drops | Heartbeat | 2 min | Missing for 3+ min |

Conclusion

Arkime's failure modes are uniquely dangerous because the platform presents false confidence — the Viewer continues to serve historical sessions even when live capture has stopped, and the gap in coverage only becomes visible when an analyst searches for sessions in the current time window during an active incident. Vigilmon's external monitoring eliminates this false confidence: Capture process health is checked every minute, PCAP ingestion rate heartbeats detect crashes within three minutes, Elasticsearch indexing lag is monitored so analysts can trust that new sessions are searchable, and PCAP disk saturation is caught before it silently shrinks the forensic retention window.

Combined, these monitors give your forensic team confidence that Arkime is actively capturing traffic and that PCAP will be retrievable when you need it for post-incident investigation.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles