How to Monitor HAProxy Health and Backend Servers with Vigilmon

HAProxy is the workhorse of many production stacks — absorbing millions of requests per second and routing them to backend pools. But a load balancer that appears healthy can be hiding serious problems: backends taken out of rotation by health checks, high backend response times that the balancer absorbs but never escalates, or the HAProxy process itself running but no longer forwarding traffic.

Vigilmon gives you external uptime monitoring for HAProxy via the stats API and HTTP probe monitoring that tests end-to-end load balancer function. This tutorial walks through both.

Why HAProxy Monitoring Matters

HAProxy has built-in health checks that remove unhealthy backends from rotation automatically. That's valuable — but it creates a dangerous blind spot. When backends fail and are removed:

• HAProxy continues accepting connections and routing to remaining backends
• Traffic is silently redistributed without any alert to your team
• If all backends in a pool fail, HAProxy returns a 503 to clients — but only then
• The stat that a backend was taken down exists only in the HAProxy stats page

Without external monitoring, your team finds out about backend failures from:

• End-user complaints about increased latency (fewer backends = more load on survivors)
• A sudden spike in 503 errors when the last backend fails
• Log analysis hours later

Vigilmon catches degradation before it becomes outage by probing both the load balancer and individual backend health in real time.

Step 1: Enable the HAProxy Stats Page

HAProxy exposes a stats page via HTTP. Enable it in your haproxy.cfg:

global
  log /dev/log local0
  maxconn 50000

defaults
  mode http
  timeout connect 5s
  timeout client 30s
  timeout server 30s

# Stats socket for runtime API
stats socket /run/haproxy/admin.sock mode 660 level admin

# Stats HTTP page
frontend stats
  bind *:8404
  stats enable
  stats uri /stats
  stats refresh 10s
  stats auth admin:your-secret-password
  no log

The stats page is available at http://haproxy:8404/stats. The stats API endpoint (which returns CSV data for programmatic consumption) is at:

http://haproxy:8404/stats;csv

Step 2: Build an HAProxy Health Endpoint

Create a /health/haproxy endpoint in your application that parses the stats CSV and returns 200/503 based on backend health:

# healthcheck.py — FastAPI HAProxy health proxy
import os, httpx, csv, io
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()
HAPROXY_STATS = os.environ.get("HAPROXY_STATS_URL", "http://haproxy:8404/stats")
HAPROXY_AUTH = (
    os.environ.get("HAPROXY_USER", "admin"),
    os.environ.get("HAPROXY_PASS", "secret"),
)

@app.get("/health/haproxy")
async def haproxy_health():
    try:
        async with httpx.AsyncClient() as client:
            resp = await client.get(
                f"{HAPROXY_STATS};csv",
                auth=HAPROXY_AUTH,
                timeout=5,
            )
        
        reader = csv.DictReader(io.StringIO(resp.text.lstrip("# ")))
        backends = []
        down_count = 0

        for row in reader:
            if row.get("svname") in ("FRONTEND", "BACKEND"):
                continue
            status = row.get("status", "")
            backends.append({
                "backend": row.get("pxname"),
                "server": row.get("svname"),
                "status": status,
            })
            if status != "UP":
                down_count += 1

        if down_count > 0:
            return JSONResponse(status_code=503, content={
                "status": "degraded",
                "down_backends": down_count,
                "backends": backends,
            })
        return JSONResponse(status_code=200, content={
            "status": "ok",
            "backends": backends,
        })
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

// healthcheck.js — Express HAProxy health proxy
const express = require('express');
const axios = require('axios');
const { parse } = require('csv-parse/sync');

const app = express();
const HAPROXY_STATS = process.env.HAPROXY_STATS_URL || 'http://haproxy:8404/stats';
const auth = {
  username: process.env.HAPROXY_USER || 'admin',
  password: process.env.HAPROXY_PASS || 'secret',
};

app.get('/health/haproxy', async (req, res) => {
  try {
    const { data: csv } = await axios.get(`${HAPROXY_STATS};csv`, { auth, timeout: 5000 });

    const rows = parse(csv.replace(/^# /, ''), { columns: true, skip_empty_lines: true });
    const servers = rows.filter(r => !['FRONTEND', 'BACKEND'].includes(r.svname));
    const downServers = servers.filter(r => r.status !== 'UP');

    if (downServers.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        down_count: downServers.length,
        down: downServers.map(r => ({ backend: r.pxname, server: r.svname, status: r.status })),
      });
    }

    return res.status(200).json({ status: 'ok', server_count: servers.length });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3001);

Verify before adding to Vigilmon:

curl -i https://your-app.example.com/health/haproxy
# HTTP/1.1 200 OK
# {"status":"ok","server_count":4}

Step 3: Configure a Vigilmon HTTP Monitor for HAProxy

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. Set the URL to your HAProxy health endpoint
4. Set the check interval to 1 minute
5. Under Expected response, configure:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 2000ms
6. Under Alert channels, assign your Slack or PagerDuty channel
7. Save the monitor

What This Catches

| Failure | HAProxy internal | Vigilmon | |---|---|---| | HAProxy process crash | ✗ | ✓ | | All backends down (503 to clients) | ✗ | ✓ | | One or more backends removed from rotation | ✗ | ✓ | | Network partition from clients to HAProxy | ✗ | ✓ | | HAProxy misconfiguration after reload | ✗ | ✓ |

Step 4: Monitor Individual Backend Pools

For multi-tier applications, each backend pool deserves its own monitor. Add pool-specific health endpoints:

@app.get("/health/haproxy/backend/{backend_name}")
async def backend_health(backend_name: str):
    try:
        async with httpx.AsyncClient() as client:
            resp = await client.get(
                f"{HAPROXY_STATS};csv",
                auth=HAPROXY_AUTH,
                timeout=5,
            )
        
        reader = csv.DictReader(io.StringIO(resp.text.lstrip("# ")))
        pool_servers = [
            row for row in reader
            if row.get("pxname") == backend_name
            and row.get("svname") not in ("FRONTEND", "BACKEND")
        ]

        if not pool_servers:
            return JSONResponse(status_code=404, content={
                "status": "not_found",
                "backend": backend_name,
            })

        down = [s for s in pool_servers if s.get("status") != "UP"]
        total = len(pool_servers)

        if len(down) == total:
            return JSONResponse(status_code=503, content={
                "status": "all_down",
                "backend": backend_name,
                "down": len(down),
                "total": total,
            })
        if down:
            return JSONResponse(status_code=200, content={
                "status": "degraded",
                "backend": backend_name,
                "down": len(down),
                "total": total,
            })
        return JSONResponse(status_code=200, content={
            "status": "ok",
            "backend": backend_name,
            "total": total,
        })
    except Exception as e:
        return JSONResponse(status_code=503, content={"status": "down", "error": str(e)})

Create Vigilmon monitors per backend pool:

• https://your-app.example.com/health/haproxy/backend/api-servers
• https://your-app.example.com/health/haproxy/backend/web-servers
• https://your-app.example.com/health/haproxy/backend/database-proxies

Step 5: End-to-End Load Balancer Uptime Monitoring

Beyond monitoring the stats API, configure Vigilmon to probe through the load balancer to a known-healthy endpoint. This validates the full traffic path:

1. Go to Monitors → New Monitor → HTTP / HTTPS
2. Set the URL to your application endpoint served through HAProxy: https://api.example.com/health
3. Set the check interval to 1 minute
4. Set the expected response: status 200, body contains "status":"ok"
5. Save the monitor

This end-to-end probe catches failures that the stats API alone cannot:

• TLS termination failure at HAProxy (SSL certificate expiry)
• Firewall rule changes blocking traffic to HAProxy
• DNS misconfiguration pointing to the wrong IP
• ACL rule changes breaking routing for specific paths

Step 6: Alert Routing for Backend Server Failures

In Vigilmon, configure alert priority by severity:

1. HAProxy overall health (all backends down) → immediate Slack + PagerDuty (P1)
2. Backend pool health (some servers down) → Slack + email (P2 — degraded, not failed)
3. End-to-end probe → immediate Slack + PagerDuty (P1 — traffic not flowing)

For a multi-region HAProxy setup, create separate monitors per region and group them on a Status Page for your infrastructure team.

Summary

HAProxy's internal health checks are good at removing failed backends silently — too silently. Vigilmon adds the external visibility your team needs:

| Monitor Type | What It Covers | |---|---| | HTTP monitor on HAProxy health endpoint | Overall backend pool health | | HTTP monitors per backend pool | Per-pool degradation and failover | | End-to-end HTTP probe through HAProxy | Full traffic path including TLS, DNS, ACLs |

Get started free at vigilmon.online — your first HAProxy monitor is running in under two minutes.