Monitoring Teiid with Vigilmon

Teiid is Red Hat's open source data virtualization platform that lets applications query data from multiple heterogeneous sources — relational databases, NoSQL stores, flat files, web services, and big data systems — through a single unified SQL interface. When you self-host Teiid as a WildFly subsystem, you're running a federated query engine whose health directly affects every downstream application that depends on virtual database (VDB) queries. A crashed query engine, a failed VDB deployment, an exhausted data source connection pool, or a bloated JVM heap can silently break every analytics and reporting workflow that uses your data virtualization layer. Vigilmon gives you continuous coverage across the Teiid stack — from the WildFly management API to individual translator connection pools.

What You'll Set Up

• Teiid query engine health via WildFly management API
• VDB deployment status monitor for each virtual database
• Query throughput and latency monitoring via cron heartbeat
• Data source translator connectivity alerts
• Plan cache and result set cache hit ratio monitors
• Active session count monitor
• WildFly JVM heap health check
• Data source connection pool exhaustion alerts
• Audit log write success monitor

Prerequisites

• Teiid deployed as a WildFly subsystem (Teiid-embedded WildFly distribution)
• WildFly management HTTP API accessible (default port 9990)
• Teiid REST/OData API accessible (default port 8080)
• A free Vigilmon account

Step 1: Monitor the Teiid Query Engine Health

The WildFly management API exposes a /management endpoint that reflects the overall server state, including whether the Teiid subsystem is active. A successful HTTP response on this endpoint confirms the query engine is up.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. URL: http://your-teiid-host:9990/management?operation=attribute&name=server-state
4. Add an HTTP Basic auth header with your WildFly management user credentials, or configure an unauthenticated read-only probe account.
5. Set Check interval to 1 minute.
6. Set Expected HTTP status to 200.
7. Under Keyword check, enter running to verify the WildFly server state in the response body.
8. Click Save.

If WildFly responds but the Teiid subsystem crashed, the server state may still return running — add a second check targeting the Teiid subsystem directly:

http://your-teiid-host:9990/management/subsystem/teiid

Expected: HTTP 200 with the Teiid subsystem attributes in the response body.

Step 2: Monitor VDB Deployment Status

Each Teiid Virtual Database (VDB) must be deployed and in ACTIVE status before applications can query it. A VDB stuck in LOADING or failed with LOADING_FAILED blocks all queries against that virtual database.

Teiid exposes VDB status through its OData/REST admin API. Create a monitor per critical VDB:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-teiid-host:8080/teiid-odata/v1/admin/vdbs('YourVDBName.1') (replace YourVDBName and 1 with your VDB name and version)
3. Check interval: 2 minutes.
4. Expected HTTP status: 200.
5. Keyword check: ACTIVE — if the VDB is in any other state, the keyword check will fail and trigger an alert.
6. Click Save.

Repeat for each VDB you operate. A VDB status alert fires before any application error surfaces.

Step 3: Set Up a Query Throughput and Latency Heartbeat

Teiid exposes JMX metrics for query throughput and latency through the WildFly management API. The cleanest way to monitor these in Vigilmon is a cron-based heartbeat script on your Teiid host that reads the metrics and posts a heartbeat to Vigilmon only when they are within acceptable bounds.

Create the Vigilmon heartbeat monitor:

1. Click Add Monitor → Heartbeat / Cron.
2. Set Name: Teiid Query Metrics.
3. Set Heartbeat interval: 5 minutes (alert if no heartbeat for 10 minutes).
4. Copy the generated heartbeat URL.

Create the check script on your Teiid host:

#!/bin/bash
# /usr/local/bin/check-teiid-query-metrics.sh

MGMT_URL="http://localhost:9990/management"
MGMT_USER="mgmt-user"
MGMT_PASS="mgmt-password"
HEARTBEAT_URL="https://hb.vigilmon.online/your-heartbeat-id"
P99_THRESHOLD_MS=2000  # alert if p99 exceeds 2 seconds

response=$(curl -s -u "${MGMT_USER}:${MGMT_PASS}" \
  "${MGMT_URL}/subsystem/teiid?operation=read-resource&include-runtime=true")

p99=$(echo "$response" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('long-running-queries-count', -1))")

if [ "$p99" -lt 0 ]; then
  echo "Failed to read Teiid metrics"
  exit 1
fi

# Post heartbeat only when metrics are healthy
curl -s "$HEARTBEAT_URL" > /dev/null

Add to cron:

*/5 * * * * /usr/local/bin/check-teiid-query-metrics.sh

Step 4: Monitor Data Source Translator Connectivity

Each Teiid translator connects to a physical data source via a connection pool. A failed translator connection causes all VDB queries that reference that data source to return errors. Monitor each critical translator's connection pool via the WildFly management API:

1. Click Add Monitor → Heartbeat / Cron.
2. Name: Teiid Translator Connectivity - <DataSourceName>.
3. Heartbeat interval: 5 minutes.
4. Copy the heartbeat URL.

Check script per data source:

#!/bin/bash
# Replace 'PostgreSQLDS' with your data source JNDI name
DS_NAME="PostgreSQLDS"
MGMT_URL="http://localhost:9990/management"
HEARTBEAT_URL="https://hb.vigilmon.online/your-translator-heartbeat-id"

failed_count=$(curl -s -u "mgmt-user:mgmt-pass" \
  "${MGMT_URL}/subsystem/datasources/data-source/${DS_NAME}/statistics/pool?operation=read-resource&include-runtime=true" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('TotalCreationTime', 0))")

active=$(curl -s -u "mgmt-user:mgmt-pass" \
  "${MGMT_URL}/subsystem/datasources/data-source/${DS_NAME}/statistics/pool?operation=read-resource&include-runtime=true" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('ActiveCount', -1))")

if [ "$active" -lt 0 ]; then
  echo "Translator $DS_NAME connection pool unreachable"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 5: Monitor Plan Cache Hit Ratio

Teiid caches query execution plans to avoid recompiling the same federated query plan on every request. A low hit ratio means Teiid is spending CPU time recompiling plans, increasing query latency for all clients.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: Teiid Plan Cache Hit Ratio.
3. Heartbeat interval: 5 minutes. Alert if no heartbeat for 15 minutes.

Check script:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-plan-cache-heartbeat-id"
MIN_HIT_RATIO=0.70  # alert if hit ratio below 70%

stats=$(curl -s -u "mgmt-user:mgmt-pass" \
  "http://localhost:9990/management/subsystem/teiid?operation=read-resource&include-runtime=true")

hits=$(echo "$stats" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('cache-hit-count',0))")
total=$(echo "$stats" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('cache-request-count',1))")

ratio=$(echo "scale=2; $hits / $total" | bc)

if (( $(echo "$ratio < $MIN_HIT_RATIO" | bc -l) )); then
  echo "Plan cache hit ratio $ratio below threshold $MIN_HIT_RATIO"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 6: Monitor Result Set Cache Hit Ratio

Teiid can cache full query results to avoid re-executing identical federated queries. Monitor the result cache effectiveness:

1. Click Add Monitor → Heartbeat / Cron.
2. Name: Teiid Result Cache Hit Ratio.
3. Heartbeat interval: 5 minutes.

Add a cron script similar to Step 5 but targeting result-set-cache-hit-count and result-set-cache-request-count from the Teiid subsystem MBean attributes.

Step 7: Monitor Active Session Count

Each JDBC or ODBC client connection to Teiid occupies a session slot. When sessions approach the configured maximum (max-active-plans in teiid-jboss-beans.xml), new client connections are rejected.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: Teiid Session Count.
3. Heartbeat interval: 1 minute.

Check script:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-session-heartbeat-id"
MAX_SESSIONS=100   # set to your configured maximum
WARN_THRESHOLD=80  # alert at 80% of max

session_count=$(curl -s -u "mgmt-user:mgmt-pass" \
  "http://localhost:9990/management/subsystem/teiid?operation=list-sessions" \
  | python3 -c "import sys,json; print(len(json.load(sys.stdin)))")

if [ "$session_count" -gt "$WARN_THRESHOLD" ]; then
  echo "Teiid session count $session_count approaching maximum $MAX_SESSIONS"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 8: Monitor WildFly JVM Heap

Teiid materializes federated query results in the WildFly JVM heap — large result sets from multiple data sources can spike heap usage significantly. A sustained heap above 85% triggers aggressive GC pauses that stall all in-flight queries.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-teiid-host:9990/management/core-service/platform-mbean/type/memory?operation=read-resource&include-runtime=true
3. Auth: Basic auth with management user credentials.
4. Check interval: 1 minute.
5. Expected HTTP status: 200.

For threshold-based heap alerting, add a heartbeat cron:

#!/bin/bash
HEARTBEAT_URL="https://hb.vigilmon.online/your-heap-heartbeat-id"
HEAP_WARN_PCT=85

heap_used=$(curl -s -u "mgmt-user:mgmt-pass" \
  "http://localhost:9990/management/core-service/platform-mbean/type/memory?operation=read-resource&include-runtime=true" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); h=d['heap-memory-usage']; print(int(h['used']/h['max']*100))")

if [ "$heap_used" -gt "$HEAP_WARN_PCT" ]; then
  echo "WildFly heap at ${heap_used}% — above ${HEAP_WARN_PCT}% threshold"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 9: Monitor Data Source Connection Pool Exhaustion

When a translator's connection pool is fully exhausted, Teiid queries that reference that data source queue or fail immediately. Monitor pool active vs. available connections for each critical data source:

1. Click Add Monitor → Heartbeat / Cron.
2. Name: Teiid Pool Exhaustion - <DataSourceName>.
3. Heartbeat interval: 1 minute.

Check script:

#!/bin/bash
DS_NAME="PostgreSQLDS"
HEARTBEAT_URL="https://hb.vigilmon.online/your-pool-heartbeat-id"
MAX_POOL=20

pool_stats=$(curl -s -u "mgmt-user:mgmt-pass" \
  "http://localhost:9990/management/subsystem/datasources/data-source/${DS_NAME}/statistics/pool?operation=read-resource&include-runtime=true")

active=$(echo "$pool_stats" | python3 -c "import sys,json; print(json.load(sys.stdin).get('ActiveCount',0))")
available=$(echo "$pool_stats" | python3 -c "import sys,json; print(json.load(sys.stdin).get('AvailableCount',1))")

if [ "$available" -eq 0 ]; then
  echo "Connection pool for $DS_NAME fully exhausted (active=$active)"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Step 10: Monitor Audit Log Write Success

Data virtualization platforms serving regulated data must maintain continuous audit logs. A failed audit log write means data access is occurring without audit trail.

1. Click Add Monitor → Heartbeat / Cron.
2. Name: Teiid Audit Log Health.
3. Heartbeat interval: 5 minutes.

Check script:

#!/bin/bash
AUDIT_LOG="/var/log/teiid/audit.log"
HEARTBEAT_URL="https://hb.vigilmon.online/your-audit-heartbeat-id"
MAX_AGE_MINUTES=10

if [ ! -f "$AUDIT_LOG" ]; then
  echo "Audit log file missing: $AUDIT_LOG"
  exit 1
fi

last_modified=$(stat -c %Y "$AUDIT_LOG")
now=$(date +%s)
age_minutes=$(( (now - last_modified) / 60 ))

if [ "$age_minutes" -gt "$MAX_AGE_MINUTES" ]; then
  echo "Audit log not written in ${age_minutes} minutes (threshold: ${MAX_AGE_MINUTES})"
  exit 1
fi

curl -s "$HEARTBEAT_URL" > /dev/null

Configuring Alerts

With monitors in place, configure alert channels to route critical failures to the right people:

1. In Vigilmon, go to Alert Channels and add your preferred channel (email, Slack, PagerDuty, webhook).
2. For each monitor, open Settings → Alerting and assign the channel.

Recommended alert thresholds:

| Monitor | Alert Condition | Severity | |---|---|---| | Query engine health | Any failure | Critical | | VDB deployment status | Any VDB not ACTIVE | Critical | | Translator connectivity | Any failure | Critical | | Pool exhaustion | available = 0 | Critical | | JVM heap | > 85% | Warning | | Plan cache hit ratio | < 70% | Warning | | Session count | > 80% of max | Warning | | Audit log | Not written in 10 min | Critical |

For production environments, set a 2-minute alert delay on session count and heap monitors to avoid alerting on momentary spikes.

Conclusion

Teiid's federated query architecture introduces failure points that span the WildFly runtime, individual data source connection pools, and the virtual database deployment lifecycle. With Vigilmon monitoring the WildFly management API, VDB status, translator pools, JVM heap, cache hit ratios, and audit log writes, you'll catch issues before they cascade into application-level query failures across every service that depends on your data virtualization layer.

Get started at vigilmon.online.