tutorial

How to Monitor Request Tracker (RT) Help Desk Health with Vigilmon

RT email gateway failures, database query slowdowns, and SLA breaches go undetected until users report tickets are not being created or responders stop receiving notifications. Learn how to monitor RT application health, email processing, and ticket throughput with Vigilmon HTTP probes and heartbeat monitors.

Request Tracker (RT) is an enterprise-grade Perl-based help desk platform used by IT organizations, universities, and government agencies to manage tickets, change requests, and customer support workflows. When RT's mod_perl or FastCGI process pool exhausts under heavy load — common during an IT incident when every user submits a ticket simultaneously — the Apache server returns 503 errors and new ticket submissions fail; when RT's email gateway (rt-mailgate) cannot reach the RT application due to a misconfigured rtserver URL, inbound email addressed to support@your-org.example.com continues to be accepted by the mail server but silently discarded without creating tickets — users believe their request was submitted while the queue stalls; when RT's database (PostgreSQL or MySQL) develops query latency above a few hundred milliseconds, the session-heavy Mason template rendering causes cascading slowdowns that make RT appear hung, and active users are logged out as sessions time out. These failures disconnect your support operations from user requests without any automatic alerting unless external monitoring is in place.

Vigilmon gives you external visibility into RT's help desk health through HTTP probe monitoring and heartbeat monitors for email processing and SLA jobs. This tutorial covers both.

Why Request Tracker Needs External Monitoring

RT's failure modes span the entire support ticket lifecycle:

  • Process pool exhaustion: RT's Perl-based application runs under Apache mod_perl or FastCGI; during incident spikes, concurrent ticket submissions from hundreds of users simultaneously exhaust the process pool — new requests queue and then time out with 503 errors, and users assume their support requests were submitted when they were not; RT's own dashboards are also affected because the same process pool serves the UI
  • Email gateway silent failure: RT processes inbound email via rt-mailgate, a CGI script that posts incoming email to the RT application; when the rtserver URL in rt-mailgate's configuration points to a stale hostname after a server migration, rt-mailgate receives a connection refused error and exits silently — the mail server delivers the message to rt-mailgate, rt-mailgate returns a non-zero exit code, and the mail server bounces the email or discards it depending on MTA configuration; users see no ticket created for their emailed request
  • Database performance degradation: RT is extremely database-intensive; every ticket view, queue listing, and saved search executes multiple SQL queries; when the database develops query latency (connection pool exhaustion, lock contention from a long-running report query, or bloated attachment storage) above ~300ms per query, Mason template rendering stacks these delays, causing page load times of 10–30 seconds; users report RT as "slow" or "down" when it is technically available but functionally unusable
  • Attachment storage disk full: RT stores email attachments in the database by default (as BLOBs) or on disk depending on configuration; when the attachment storage fills — a single large incident thread with many attachments can consume gigabytes — new ticket updates that include attachments fail with internal server errors while text-only replies continue to work, creating an asymmetric failure that is difficult to diagnose without monitoring
  • ScripsScripts failure: RT's Scrips automation engine triggers on ticket events to send notifications, enforce escalation policies, and update custom fields; when a Scrips subroutine throws a Perl exception due to a bug introduced in a custom Scrips update, the Scrip itself fails silently — notifications stop going out, but no error is surfaced in the UI and RT continues accepting tickets
  • Outbound SMTP failure: RT sends ticket notification emails via SMTP; when the SMTP relay becomes unavailable (authentication failure after a password rotation, TLS certificate mismatch, or relay server restart), all outbound notifications from RT — new ticket confirmations, reply notifications, SLA escalations — fail to deliver; responders stop receiving ticket updates and believe queues are empty

External monitoring with Vigilmon adds:

  • Proactive alerting when the RT application stops accepting HTTP connections
  • Email gateway liveness through heartbeat monitors attached to email processing health checks
  • Outbound SMTP validation via periodic test messages that ping a heartbeat on successful delivery
  • Multi-region probe consensus that filters transient mod_perl GC pauses from genuine RT application crashes

Step 1: Build an RT Health Endpoint

RT does not expose a native /health endpoint. Build a thin health check using RT's REST API v2 (available in RT 5.x) or a lightweight sidecar that validates RT application and database health.

RT REST API Health Probe

RT 5.x provides a REST API v2 at /REST/2.0/. Probe it to verify the application is accepting connections:

# Test RT REST API liveness
curl -u 'rt-health-user:password' \
  https://rt.your-org.example.com/REST/2.0/ \
  -H 'Content-Type: application/json'

A healthy response returns {"_url":"https://rt.your-org.example.com/REST/2.0/","version":"5.0.x"}.

Node.js Health Sidecar

// health/request-tracker.js
const express = require('express');
const https = require('https');

const app = express();
const RT_HOST = process.env.RT_HOST || 'rt.your-org.example.com';
const RT_USER = process.env.RT_API_USER || 'rt-health';
const RT_PASS = process.env.RT_API_PASS || '';

function rtApiProbe() {
  return new Promise((resolve, reject) => {
    const auth = Buffer.from(`${RT_USER}:${RT_PASS}`).toString('base64');
    const options = {
      hostname: RT_HOST,
      port: 443,
      path: '/REST/2.0/',
      method: 'GET',
      headers: {
        'Authorization': `Basic ${auth}`,
        'Content-Type': 'application/json',
      },
      rejectUnauthorized: false,
    };
    const req = https.request(options, res => {
      let body = '';
      res.on('data', chunk => body += chunk);
      res.on('end', () => {
        if (res.statusCode === 200) {
          try {
            const data = JSON.parse(body);
            return resolve({ version: data.version || 'unknown' });
          } catch {
            return reject(new Error('Invalid JSON from RT API'));
          }
        }
        reject(new Error(`HTTP ${res.statusCode} from RT`));
      });
    });
    req.on('error', reject);
    req.setTimeout(8000, () => { req.destroy(); reject(new Error('RT API timeout')); });
    req.end();
  });
}

async function checkQueueDepth() {
  // Query RT REST API v2 for open ticket count in critical queues
  return new Promise((resolve, reject) => {
    const auth = Buffer.from(`${RT_USER}:${RT_PASS}`).toString('base64');
    const query = encodeURIComponent("Status='open' OR Status='new'");
    const options = {
      hostname: RT_HOST,
      port: 443,
      path: `/REST/2.0/tickets?query=${query}&fields=id,Subject,Status&per_page=1`,
      method: 'GET',
      headers: {
        'Authorization': `Basic ${auth}`,
        'Content-Type': 'application/json',
      },
      rejectUnauthorized: false,
    };
    const req = https.request(options, res => {
      let body = '';
      res.on('data', chunk => body += chunk);
      res.on('end', () => {
        try {
          const data = JSON.parse(body);
          resolve({ open_ticket_count: data.total || 0 });
        } catch {
          reject(new Error('Invalid JSON from RT ticket query'));
        }
      });
    });
    req.on('error', reject);
    req.setTimeout(8000, () => { req.destroy(); reject(new Error('Timeout')); });
    req.end();
  });
}

app.get('/health/rt', async (req, res) => {
  const checks = {};
  let healthy = true;

  try {
    const result = await rtApiProbe();
    checks.rt_api = 'ok';
    checks.rt_version = result.version;
  } catch (err) {
    checks.rt_api = `down: ${err.message}`;
    healthy = false;
  }

  try {
    const { open_ticket_count } = await checkQueueDepth();
    checks.open_ticket_count = open_ticket_count;
  } catch (err) {
    checks.ticket_query = `error: ${err.message}`;
  }

  return res.status(healthy ? 200 : 503).json({
    status: healthy ? 'ok' : 'down',
    checks,
  });
});

app.listen(3012, () => console.log('RT health sidecar on :3012'));

Python (FastAPI) Alternative

# health/rt_health.py
import os
import httpx
from fastapi import FastAPI, Response

app = FastAPI()

RT_HOST = os.environ.get("RT_HOST", "rt.your-org.example.com")
RT_USER = os.environ.get("RT_API_USER", "rt-health")
RT_PASS = os.environ.get("RT_API_PASS", "")
BASE_URL = f"https://{RT_HOST}"

@app.get("/health/rt")
async def rt_health():
    checks = {}
    healthy = True

    async with httpx.AsyncClient(
        verify=False,
        timeout=8.0,
        auth=(RT_USER, RT_PASS),
    ) as client:
        try:
            r = await client.get(f"{BASE_URL}/REST/2.0/")
            r.raise_for_status()
            data = r.json()
            checks["rt_api"] = "ok"
            checks["rt_version"] = data.get("version", "unknown")
        except Exception as e:
            checks["rt_api"] = f"down: {str(e)}"
            healthy = False

        try:
            r = await client.get(
                f"{BASE_URL}/REST/2.0/tickets",
                params={"query": "Status='open' OR Status='new'", "per_page": 1},
            )
            r.raise_for_status()
            data = r.json()
            checks["open_ticket_count"] = data.get("total", 0)
        except Exception as e:
            checks["ticket_query"] = f"error: {str(e)}"

    status_code = 200 if healthy else 503
    return Response(
        content=str({"status": "ok" if healthy else "down", "checks": checks}),
        status_code=status_code,
        media_type="application/json",
    )

Step 2: Configure Vigilmon Monitoring

HTTP Monitor — RT Application

In your Vigilmon dashboard, create an HTTP monitor for the RT health endpoint:

| Field | Value | |-------|-------| | Monitor name | Request Tracker Application | | URL | http://rt-sidecar.internal:3012/health/rt | | Method | GET | | Check interval | Every 1 minute | | Expected status | 200 | | Alert threshold | 2 consecutive failures | | Regions | Select 2+ regions for consensus |

HTTP Monitor — RT Login Page

Also probe the RT web interface directly to catch cases where the application is responding to the API but the UI is broken:

| Field | Value | |-------|-------| | Monitor name | RT Web Interface | | URL | https://rt.your-org.example.com/rt/ | | Method | GET | | Expected status | 200 | | Response body must contain | Request Tracker | | Check interval | Every 2 minutes |

Heartbeat Monitor — Email Gateway Processing

RT's email gateway (rt-mailgate) is the most critical hidden component: if it fails, tickets stop being created from email without any visible error. Set up a heartbeat monitor with a 30-minute timeout and ping it from a test email processing job:

#!/bin/bash
# /etc/cron.d/rt-email-gateway-health — runs every 15 minutes
# Send a test email to RT and verify it creates a ticket via the REST API

RT_HOST="rt.your-org.example.com"
RT_USER="rt-health"
RT_PASS="${RT_API_PASS}"
QUEUE="Health-Check"  # A dedicated queue for health check tickets
HEARTBEAT_URL="https://vigilmon.online/hb/your-email-gateway-heartbeat"

# Create a test ticket via the REST API (bypassing email to isolate API health)
TICKET_ID=$(curl -su "${RT_USER}:${RT_PASS}" \
  "https://${RT_HOST}/REST/2.0/ticket" \
  -X POST \
  -H 'Content-Type: application/json' \
  -d "{\"Queue\":\"${QUEUE}\",\"Subject\":\"Health check $(date +%s)\",\"Status\":\"resolved\"}" \
  2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('id',''))" 2>/dev/null)

if [ -n "$TICKET_ID" ]; then
  # Ticket created successfully — RT API is working
  curl -fsS --max-time 10 "$HEARTBEAT_URL" > /dev/null 2>&1
fi

For testing actual email processing (catching rt-mailgate failures specifically):

#!/bin/bash
# /etc/cron.d/rt-mailgate-health — tests the actual email path every 30 minutes
SUBJECT="RT Health Check $(date +%s)"
RT_EMAIL="support@your-org.example.com"

# Send a test email through the real mail path
echo "Health check message body" | mail -s "$SUBJECT" "$RT_EMAIL"

# Wait 2 minutes for RT to process it, then check if a ticket was created
sleep 120

RT_HOST="rt.your-org.example.com"
FOUND=$(curl -su "rt-health:${RT_API_PASS}" \
  "https://${RT_HOST}/REST/2.0/tickets?query=Subject+LIKE+'RT+Health+Check'&per_page=1" \
  2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('total',0))" 2>/dev/null)

if [ "$FOUND" -gt 0 ] 2>/dev/null; then
  curl -fsS --max-time 10 \
    "https://vigilmon.online/hb/your-mailgate-heartbeat" > /dev/null 2>&1
fi

Configure this heartbeat with a 45-minute timeout to allow for the 2-minute email processing delay plus scheduling slack.

Heartbeat Monitor — Outbound SMTP

Monitor RT's outbound email notification delivery:

#!/bin/bash
# /etc/cron.d/rt-smtp-health — runs every 30 minutes
# Test SMTP delivery by sending a message and confirming receipt via IMAP or webhook

SMTP_HOST="${RT_SMTP_HOST:-localhost}"
TEST_RECIPIENT="monitoring+rt@your-org.example.com"

# Send test email via sendmail/postfix
echo "From: rt@your-org.example.com
To: ${TEST_RECIPIENT}
Subject: RT SMTP Health Check $(date +%s)

RT outbound SMTP health check." | sendmail -f rt@your-org.example.com "$TEST_RECIPIENT"

if [ $? -eq 0 ]; then
  curl -fsS --max-time 10 \
    "https://vigilmon.online/hb/your-smtp-heartbeat" > /dev/null 2>&1
fi

Step 3: Configure Alerting

Alert Policies

RT Application Alert

  • Trigger: 2 consecutive failed HTTP probes from 2+ regions
  • Severity: Critical
  • Channels: PagerDuty + Slack #helpdesk-ops
  • Message: "Request Tracker is unreachable. Ticket management is down. Users cannot submit or track support requests."

Email Gateway Alert

  • Trigger: Heartbeat not received for 30 minutes
  • Severity: Critical
  • Channels: PagerDuty + Slack #helpdesk-ops
  • Message: "RT email gateway heartbeat missed. Emails to support queues may not be creating tickets. Check rt-mailgate configuration and mail server delivery."

Outbound SMTP Alert

  • Trigger: Heartbeat not received for 45 minutes
  • Severity: High
  • Channels: Slack #helpdesk-ops + email to RT admin
  • Message: "RT outbound SMTP delivery missed. Ticket notifications may not be reaching responders. Check SMTP relay configuration."

SLA Breach Tracking

Use the RT REST API to check for tickets breaching their SLA and feed results to a heartbeat:

#!/bin/bash
# /etc/cron.d/rt-sla-check — check every 15 minutes
RT_HOST="rt.your-org.example.com"

# Query for open tickets that have breached their due date
BREACHED=$(curl -su "rt-health:${RT_API_PASS}" \
  "https://${RT_HOST}/REST/2.0/tickets?query=Status!='resolved'+AND+Due+<+'now'&per_page=1" \
  2>/dev/null | python3 -c "import json,sys; print(json.load(sys.stdin).get('total',0))" 2>/dev/null)

echo "RT SLA breached tickets: $BREACHED"

# Alert via Vigilmon only if breach count is within acceptable range (<10)
if [ "$BREACHED" -lt 10 ] 2>/dev/null; then
  curl -fsS --max-time 10 \
    "https://vigilmon.online/hb/your-sla-heartbeat" > /dev/null 2>&1
fi

Configure the SLA heartbeat with a 20-minute timeout so any missed check or breach count spike triggers an alert.

Key Metrics Summary

| Metric | Alert Threshold | Impact | |--------|----------------|--------| | RT web application HTTP | Any failure | Ticket management inaccessible to all users | | RT email gateway | Heartbeat missing >30 min | Inbound email not creating tickets; requests silently lost | | Outbound SMTP delivery | Heartbeat missing >45 min | Responders not receiving ticket notifications | | Database query latency | >300ms p95 | RT UI becomes functionally unusable (10–30s page loads) | | Open ticket queue depth | Queue-specific thresholds | SLA breach risk for high-priority queues | | SLA breached ticket count | >10 open breaches | SLA compliance failure | | Attachment storage disk | >80% | Ticket updates with attachments failing | | Active session count | Sudden drop | Session store failure logging out all users |

Conclusion

Request Tracker's email gateway is its most critical hidden component — when rt-mailgate fails silently, users believe their requests have been filed while queues remain empty and SLAs drift. Vigilmon HTTP probes on the RT application catch availability failures immediately, while heartbeat monitors on email processing and outbound SMTP delivery give you coverage of the two failure modes most likely to go undetected for hours.

Configure the monitors in this tutorial and your help desk team will never again spend an hour investigating why tickets stopped coming in.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles