How to Monitor 389 Directory Server (LDAP) Health with Vigilmon

389 Directory Server is Red Hat's upstream enterprise LDAP server, the foundation of FreeIPA and Red Hat Identity Management (IdM), and the authentication backbone for a wide range of applications that rely on LDAP bind operations for user login. When the ns-slapd process crashes due to a BDB environment corruption after an unclean shutdown — a documented failure mode when ns-slapd is OOM-killed without completing write operations — the LDAP port on 389 goes unreachable and every application using LDAP authentication across your infrastructure fails simultaneously; when multi-supplier replication between two 389 DS instances develops a replication conflict that the tombstone resolution algorithm cannot automatically resolve, replicated entries diverge silently — some users can authenticate against replica A but not replica B depending on which instance their application is load-balanced to; when the BDB cache hit ratio drops from 99% to 70% because a directory expansion pushed the working set above available RAM, LDAP bind operations that previously completed in 5ms now take 80ms as every cache miss requires a disk read — application login pages slow dramatically before operators realize the directory performance has degraded. These are infrastructure-wide authentication failures that present as application errors and require external directory monitoring to catch early.

Vigilmon gives you external visibility into 389 Directory Server's LDAP health through HTTP probe monitoring and heartbeat monitors for replication health and certificate expiry checks. This tutorial covers both.

Why 389 Directory Server Needs External Monitoring

389 DS failure modes have infrastructure-wide authentication blast radius:

• ns-slapd process crash after BDB corruption: 389 DS uses Berkeley DB (BDB) or LMDB as its database backend; when ns-slapd is killed without completing a write (OOM kill, forced shutdown during heavy write load), the BDB transaction log may become inconsistent; on next startup, 389 DS may attempt a recovery that fails, leaving the process unable to start and the LDAP port unreachable — every application in the infrastructure that uses LDAP bind for authentication fails simultaneously
• Replication divergence: 389 DS multi-supplier replication propagates changes between directory replicas; when a replication conflict occurs (the same entry modified on two suppliers before the changes replicate), the conflict resolution algorithm generates a "glue entry" or tombstone; if the conflict cannot be automatically resolved, the two replicas contain different data — users who changed their password on supplier A may not be able to authenticate against supplier B because the password change has not propagated
• BDB/LMDB cache pressure: 389 DS performs best when the entire working set of directory entries fits in the BDB/LMDB cache (configured via nsslapd-dbcachesize); when a directory expansion (merger, large batch import) pushes the working set above cache, read operations begin hitting disk; LDAP bind latency climbs from <10ms to 50–200ms; at scale, this causes visible slowdowns in every application using LDAP authentication
• Certificate expiry on LDAPS: 389 DS serves LDAPS (encrypted LDAP) on port 636; when the TLS certificate expires, new LDAPS connections fail with SSL_ERROR_EXPIRED_CERT_ALERT; existing long-lived connections continue working until they are recycled, masking the expiry until the next connection storm; monitoring that only checks port 389 misses the LDAPS failure entirely
• Connection limit exhaustion: 389 DS has a configurable nsslapd-maxdescriptors and nsslapd-maxbersize limit; during authentication storms (all users logging in simultaneously at shift start), the connection count can hit the limit; new connection attempts receive LDAP_UNAVAILABLE (resultCode: 52) while existing connections continue working — the failure is partial and load-dependent
• Failed bind spike (password spray attack): 389 DS logs failed bind operations (invalid password) in the errors log; a sudden spike in failed binds — from an attacker testing a credential list — is invisible unless monitoring is watching the failed bind rate

External monitoring with Vigilmon adds:

• Proactive alerting when ns-slapd goes unreachable on either LDAP (389) or LDAPS (636)
• Replication health liveness through heartbeat monitors on scheduled replication status checks
• Certificate expiry pre-alerting through heartbeat monitors that check certificate days-to-expiry and only ping if above threshold
• Multi-region probe consensus that filters transient ns-slapd GC-equivalent BDB checkpoint pauses from genuine process crashes

Step 1: Build a 389 DS Health Endpoint

389 Directory Server does not expose an HTTP health endpoint natively. Build a sidecar that performs LDAP bind probes and exposes health over HTTP.

Node.js LDAP Health Sidecar

// health/389ds.js
const express = require('express');
const ldap = require('ldapjs');
const { execSync } = require('child_process');

const app = express();

const LDAP_URL    = process.env.LDAP_URL    || 'ldap://localhost:389';
const LDAPS_URL   = process.env.LDAPS_URL   || 'ldaps://localhost:636';
const BIND_DN     = process.env.LDAP_BIND_DN  || 'cn=Directory Manager';
const BIND_PW     = process.env.LDAP_BIND_PW  || '';
const BASE_DN     = process.env.LDAP_BASE_DN  || 'dc=example,dc=com';
const MONITOR_DN  = 'cn=monitor';  // 389 DS built-in monitor entry

function ldapBind(url) {
  return new Promise((resolve, reject) => {
    const client = ldap.createClient({
      url,
      connectTimeout: 5000,
      tlsOptions: { rejectUnauthorized: false },
    });
    client.on('error', err => { client.destroy(); reject(err); });
    client.bind(BIND_DN, BIND_PW, err => {
      if (err) { client.destroy(); return reject(err); }
      client.unbind();
      resolve(true);
    });
  });
}

function ldapSearch(url, base, filter, attrs) {
  return new Promise((resolve, reject) => {
    const client = ldap.createClient({
      url,
      connectTimeout: 5000,
      tlsOptions: { rejectUnauthorized: false },
    });
    const entries = [];
    client.on('error', err => { client.destroy(); reject(err); });
    client.bind(BIND_DN, BIND_PW, err => {
      if (err) { client.destroy(); return reject(err); }
      client.search(base, { scope: 'base', filter, attributes: attrs }, (serr, res) => {
        if (serr) { client.destroy(); return reject(serr); }
        res.on('searchEntry', entry => entries.push(entry.object));
        res.on('error', e => { client.destroy(); reject(e); });
        res.on('end', result => {
          client.unbind();
          if (result.status !== 0) return reject(new Error(`Search status ${result.status}`));
          resolve(entries);
        });
      });
    });
  });
}

async function check389DSMonitor(url) {
  // Query cn=monitor for operational statistics
  const entries = await ldapSearch(
    url,
    MONITOR_DN,
    '(objectClass=*)',
    ['currentconnections', 'totalconnections', 'opsinitiated', 'opscompleted', 'bytesrecv', 'bytessent']
  );
  if (entries.length === 0) throw new Error('cn=monitor returned no entries');
  return entries[0];
}

app.get('/health/389ds', async (req, res) => {
  const checks = {};
  let healthy = true;
  const start = Date.now();

  // LDAP port 389 bind probe
  try {
    await ldapBind(LDAP_URL);
    checks.ldap_389 = 'ok';
    checks.ldap_bind_ms = Date.now() - start;
  } catch (err) {
    checks.ldap_389 = `down: ${err.message}`;
    healthy = false;
  }

  // LDAPS port 636 bind probe
  const tlsStart = Date.now();
  try {
    await ldapBind(LDAPS_URL);
    checks.ldaps_636 = 'ok';
    checks.ldaps_bind_ms = Date.now() - tlsStart;
  } catch (err) {
    checks.ldaps_636 = `down: ${err.message}`;
    // LDAPS failure is critical — mark unhealthy
    healthy = false;
  }

  // cn=monitor operational stats (only if LDAP is up)
  if (checks.ldap_389 === 'ok') {
    try {
      const monitor = await check389DSMonitor(LDAP_URL);
      checks.current_connections = parseInt(monitor.currentconnections || 0);
      checks.ops_initiated = parseInt(monitor.opsinitiated || 0);
    } catch (err) {
      checks.monitor = `error: ${err.message}`;
    }
  }

  // Certificate expiry check via openssl
  try {
    const host = LDAPS_URL.replace('ldaps://', '').split(':')[0];
    const port = LDAPS_URL.includes(':636') ? 636 : 636;
    const certInfo = execSync(
      `echo Q | openssl s_client -connect ${host}:${port} -servername ${host} 2>/dev/null | openssl x509 -noout -enddate`,
      { timeout: 10000 }
    ).toString().trim();
    // Format: notAfter=Jun 15 12:00:00 2026 GMT
    const dateStr = certInfo.replace('notAfter=', '');
    const expiry = new Date(dateStr);
    const daysLeft = Math.floor((expiry - Date.now()) / 86400000);
    checks.cert_days_to_expiry = daysLeft;
    if (daysLeft < 30) {
      checks.cert_warning = `TLS cert expires in ${daysLeft} days`;
      healthy = false;
    }
  } catch (err) {
    checks.cert_check = `error: ${err.message}`;
  }

  return res.status(healthy ? 200 : 503).json({
    status: healthy ? 'ok' : 'degraded',
    checks,
  });
});

app.listen(3013, () => console.log('389 DS health sidecar on :3013'));

Python (FastAPI) Alternative

# health/389ds_health.py
import os
import ssl
import time
import socket
import subprocess
import ldap3
from fastapi import FastAPI, Response

app = FastAPI()

LDAP_HOST    = os.environ.get("LDAP_HOST", "localhost")
BIND_DN      = os.environ.get("LDAP_BIND_DN", "cn=Directory Manager")
BIND_PW      = os.environ.get("LDAP_BIND_PW", "")
BASE_DN      = os.environ.get("LDAP_BASE_DN", "dc=example,dc=com")

def ldap_bind_probe(use_tls: bool = False):
    port = 636 if use_tls else 389
    server = ldap3.Server(LDAP_HOST, port=port, use_ssl=use_tls, connect_timeout=5,
                          get_info=ldap3.NONE)
    conn = ldap3.Connection(server, user=BIND_DN, password=BIND_PW,
                            authentication=ldap3.SIMPLE, auto_bind=False)
    start = time.monotonic()
    if not conn.bind():
        raise Exception(f"Bind failed: {conn.result}")
    elapsed_ms = int((time.monotonic() - start) * 1000)
    conn.unbind()
    return elapsed_ms

def get_monitor_stats():
    server = ldap3.Server(LDAP_HOST, port=389, connect_timeout=5)
    conn = ldap3.Connection(server, user=BIND_DN, password=BIND_PW,
                            authentication=ldap3.SIMPLE, auto_bind=True)
    conn.search("cn=monitor", "(objectClass=*)", attributes=[
        "currentconnections", "totalconnections", "opsinitiated", "opscompleted"
    ])
    if conn.entries:
        entry = conn.entries[0]
        return {
            "current_connections": int(entry.currentconnections.value or 0),
            "ops_initiated": int(entry.opsinitiated.value or 0),
        }
    return {}

def check_cert_expiry():
    result = subprocess.run(
        ["openssl", "s_client", f"-connect", f"{LDAP_HOST}:636",
         "-servername", LDAP_HOST],
        input=b"Q", capture_output=True, timeout=10
    )
    cert_result = subprocess.run(
        ["openssl", "x509", "-noout", "-enddate"],
        input=result.stdout, capture_output=True, timeout=5
    )
    output = cert_result.stdout.decode().strip()
    # notAfter=Jun 15 12:00:00 2026 GMT
    date_str = output.replace("notAfter=", "")
    from datetime import datetime
    expiry = datetime.strptime(date_str, "%b %d %H:%M:%S %Y %Z")
    days_left = (expiry - datetime.utcnow()).days
    return days_left

@app.get("/health/389ds")
async def ds389_health():
    checks = {}
    healthy = True

    try:
        ms = ldap_bind_probe(use_tls=False)
        checks["ldap_389"] = "ok"
        checks["ldap_bind_ms"] = ms
    except Exception as e:
        checks["ldap_389"] = f"down: {str(e)}"
        healthy = False

    try:
        ms = ldap_bind_probe(use_tls=True)
        checks["ldaps_636"] = "ok"
        checks["ldaps_bind_ms"] = ms
    except Exception as e:
        checks["ldaps_636"] = f"down: {str(e)}"
        healthy = False

    if checks.get("ldap_389") == "ok":
        try:
            stats = get_monitor_stats()
            checks.update(stats)
        except Exception as e:
            checks["monitor"] = f"error: {str(e)}"

    try:
        days = check_cert_expiry()
        checks["cert_days_to_expiry"] = days
        if days < 30:
            checks["cert_warning"] = f"TLS cert expires in {days} days"
            healthy = False
    except Exception as e:
        checks["cert_check"] = f"error: {str(e)}"

    status_code = 200 if healthy else 503
    import json
    return Response(
        content=json.dumps({"status": "ok" if healthy else "degraded", "checks": checks}),
        status_code=status_code,
        media_type="application/json",
    )

Step 2: Configure Vigilmon Monitoring

HTTP Monitor — 389 DS Health Endpoint

In your Vigilmon dashboard, create an HTTP monitor for the 389 DS sidecar:

| Field | Value | |-------|-------| | Monitor name | 389 Directory Server | | URL | http://ldap-server.internal:3013/health/389ds | | Method | GET | | Check interval | Every 1 minute | | Expected status | 200 | | Alert threshold | 2 consecutive failures | | Regions | Select 2+ regions for consensus |

HTTP Monitor — LDAP Port TCP Probe

Also create a TCP port probe directly on port 389 to catch cases where the sidecar is down but ns-slapd is also down:

| Field | Value | |-------|-------| | Monitor name | 389 DS LDAP Port 389 | | URL | ldap://ldap-server.your-org.example.com:389 | | Method | TCP | | Check interval | Every 1 minute | | Alert threshold | 1 failure |

Heartbeat Monitor — Replication Health

Monitor replication agreement status between 389 DS suppliers. Configure a 15-minute timeout heartbeat:

#!/bin/bash
# /etc/cron.d/389ds-replication-health — runs every 10 minutes
# Check all replication agreements; ping heartbeat only if all are in sync

LDAP_HOST="localhost"
BIND_DN="cn=Directory Manager"
BIND_PW="${LDAP_BIND_PW}"
SUFFIX="dc=example,dc=com"
HEARTBEAT_URL="https://vigilmon.online/hb/your-replication-heartbeat"

# Query all replication manager entries under cn=config
STATUS=$(ldapsearch -x -H "ldap://${LDAP_HOST}:389" \
  -D "$BIND_DN" -w "$BIND_PW" \
  -b "cn=replica,cn=\"${SUFFIX}\",cn=mapping tree,cn=config" \
  -s sub '(objectClass=nsDS5ReplicationAgreement)' \
  nsDS5ReplicaLastUpdateStatus nsDS5ReplicaLastUpdateStatusJSON 2>/dev/null)

# Check for replication errors (status code != 0)
ERRORS=$(echo "$STATUS" | grep -i "nsDS5ReplicaLastUpdateStatus:" | grep -v "Error (0)")

if [ -z "$ERRORS" ]; then
  # All agreements healthy
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
else
  echo "389 DS replication errors detected:"
  echo "$ERRORS"
  # Optionally: send alert email or Slack webhook here
fi

Heartbeat Monitor — Failed Bind Rate

Watch for password spray attacks via a spike in failed LDAP binds:

#!/bin/bash
# /etc/cron.d/389ds-failed-bind-check — runs every 5 minutes
ERRORS_LOG="/var/log/dirsrv/slapd-YOUR_INSTANCE/errors"
HEARTBEAT_URL="https://vigilmon.online/hb/your-failed-bind-heartbeat"

# Count failed bind operations in the last 5 minutes
FAILED_BINDS=$(grep "Invalid credentials" "$ERRORS_LOG" | \
  awk -v cutoff="$(date -d '5 minutes ago' '+%Y%m%d%H%M%S')" \
  'substr($1,1,15) > cutoff' 2>/dev/null | wc -l)

# Alert if more than 50 failed binds in 5 minutes (10/minute threshold)
if [ "$FAILED_BINDS" -lt 50 ]; then
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
else
  echo "389 DS failed bind spike: $FAILED_BINDS in last 5 minutes" | \
    mail -s "SECURITY ALERT: LDAP password spray detected" security@your-org.example.com
fi

Configure the heartbeat with a 10-minute timeout so any missed check (or spike) triggers an alert.

Heartbeat Monitor — Certificate Expiry Pre-Alert

#!/bin/bash
# /etc/cron.d/389ds-cert-check — runs daily
LDAP_HOST="ldap.your-org.example.com"
HEARTBEAT_URL="https://vigilmon.online/hb/your-cert-expiry-heartbeat"

DAYS_LEFT=$(echo Q | openssl s_client -connect "${LDAP_HOST}:636" \
  -servername "${LDAP_HOST}" 2>/dev/null | \
  openssl x509 -noout -enddate 2>/dev/null | \
  sed 's/notAfter=//' | \
  python3 -c "
import sys
from datetime import datetime
expiry = datetime.strptime(sys.stdin.read().strip(), '%b %d %H:%M:%S %Y %Z')
print((expiry - datetime.utcnow()).days)
" 2>/dev/null)

if [ -n "$DAYS_LEFT" ] && [ "$DAYS_LEFT" -gt 30 ]; then
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
else
  echo "LDAPS certificate expires in ${DAYS_LEFT:-unknown} days" | \
    mail -s "389 DS: TLS Certificate Expiry Warning" ops@your-org.example.com
fi

Configure the heartbeat with a 25-hour timeout (daily check with slack) so any missed check or certificate within 30 days of expiry triggers an alert.

Step 3: Configure Alerting

Alert Policies

389 DS LDAP Port Alert

• Trigger: 1 failed TCP probe on port 389 from 2+ regions
• Severity: Critical — P1
• Channels: PagerDuty (immediate) + Slack #infrastructure-critical
• Message: "389 Directory Server LDAP port 389 is unreachable. All LDAP-authenticated applications are failing. Check ns-slapd process and BDB environment health immediately."

LDAPS Port Alert

• Trigger: Health endpoint returns 503 (cert expired or LDAPS bind failure)
• Severity: Critical
• Channels: PagerDuty + Slack #infrastructure-critical
• Message: "389 Directory Server LDAPS port 636 is unreachable or TLS certificate has expired. Applications using LDAPS for authentication are failing."

Replication Heartbeat Alert

• Trigger: Heartbeat not received for 15 minutes
• Severity: High
• Channels: Slack #infrastructure-alerts + email to directory admins
• Message: "389 DS replication health check missed. One or more replication agreements may have errors. Users on failed replicas may have stale directory data."

Certificate Expiry Alert

• Trigger: Heartbeat not received for 25 hours (cert <30 days from expiry or check failed)
• Severity: High
• Channels: Slack #infrastructure-alerts + email to ops
• Message: "389 DS LDAPS TLS certificate expires within 30 days. Renew via certutil or the FreeIPA certificate lifecycle tools before expiry causes LDAPS authentication failure."

Failed Bind Spike Alert (Security)

• Trigger: Heartbeat not received for 10 minutes
• Severity: High (Security)
• Channels: Slack #security-alerts + PagerDuty security rotation
• Message: "389 DS failed bind rate spike detected. Possible password spray attack in progress. Review /var/log/dirsrv/errors for source IP patterns."

Key Metrics Summary

| Metric | Alert Threshold | Impact | |--------|----------------|--------| | ns-slapd process / LDAP port 389 | Any failure | Infrastructure-wide authentication failure | | LDAPS port 636 | Any failure | TLS-encrypted auth failures across all applications | | LDAP bind latency | p99 >100ms | Visible login slowdowns across all LDAP-dependent apps | | Replication agreement status | Any non-zero status | Directory divergence; inconsistent auth across replicas | | BDB/LMDB cache hit ratio | <95% | Disk reads causing 10–20× bind latency increase | | Active connection count | >80% of nsslapd-maxdescriptors | Imminent connection refusal under load | | Failed bind rate | >10/minute sustained | Password spray or misconfigured application retry loop | | TLS certificate days to expiry | <30 days | Scheduled LDAPS authentication failure at expiry | | Database directory disk usage | >80% | Imminent BDB environment corruption from disk full | | Error log rate | Spike vs baseline | Crash loops, schema violations, or replication errors |

Conclusion

389 Directory Server is the authentication foundation for FreeIPA, Red Hat IdM, and every application in your infrastructure that uses LDAP bind — which makes it one of the highest-impact services to monitor. A process crash, replication failure, or certificate expiry can disable authentication across your entire application stack within seconds. Vigilmon HTTP probes on the 389 DS health sidecar catch process failures the moment LDAP ports go dark, while heartbeat monitors on replication health checks and certificate expiry pre-alerts give you coverage of the slow-burn failures that process monitors miss entirely.

Configure the monitors in this tutorial and your infrastructure team will receive alerts about directory server problems before users start filing tickets about being unable to log in.