Monitoring Egeria Open Metadata with Vigilmon

Egeria is the Linux Foundation's open-source metadata and governance ecosystem — a federated metadata mesh that lets data tools, catalogs, and governance platforms share information about data assets, lineage, and quality policies across an enterprise. When Egeria is healthy, your data governance platform has fresh, accurate metadata. When it is not, connectors go stale, governance actions stop firing, and your data catalog silently diverges from reality. Vigilmon gives you continuous visibility into OMAG platform health, cohort membership, JanusGraph backend availability, integration daemon connectors, and governance engine operation so you catch metadata staleness before data consumers notice.

What You'll Set Up

OMAG Server Platform health monitoring
Per-server Metadata Access Server status checks
Cohort registration monitoring
JanusGraph graph database health alerts
Kafka cohort event bus connectivity
Integration daemon connector failure detection
Governance engine and Engine Host health
View Server uptime for the Egeria React UI

Prerequisites

Egeria OMAG Server Platform 4.x or later running
At least one OMAG server configured (Metadata Access Server, Integration Daemon, or Engine Host)
Kafka running as the cohort event bus
A free Vigilmon account

Why Monitoring Egeria Matters

Egeria operates as an eventually consistent federated system: metadata flows between cohort members over Kafka, is persisted in JanusGraph (or an in-memory store), and is surfaced via the View Server to the Egeria React UI. A failure at any layer is silent by default. The OMAG platform does not expose a central dashboard — it is a runtime for multiple logical servers, each of which can fail independently.

The monitoring challenges are distinct from a typical API service:

Cohort dropout is silent. If a server drops out of the cohort (network partition, Kafka outage, misconfiguration), it stops receiving peer metadata updates. Your data catalog drifts without any error page.
Connector staleness accumulates slowly. An integration daemon connector that fails to run its sync cycle leaves metadata unchanged. Old lineage, missing column descriptions, and stale table statistics are hard to notice.
JanusGraph is a deep dependency. The Metadata Access Server cannot answer queries if JanusGraph is unavailable. Unlike a relational database, JanusGraph failures surface as long timeouts rather than fast connection errors.

Proactive monitoring catches these failure modes before they compound.

Step 1: Monitor the OMAG Server Platform

The OMAG Server Platform is the Egeria runtime process. If it goes down, all OMAG servers it hosts fail simultaneously. Egeria provides a platform origin endpoint that returns a simple version string.

Log in to vigilmon.online and click Add Monitor.
Set Type to HTTP / HTTPS.
Enter URL: https://your-egeria-host:9443/open-metadata/platform-services/server-platform/origin
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Under Alert conditions, set Alert after to 1 failure.
Click Save.

This endpoint requires no authentication and returns a string like Egeria OMAG Server Platform (version 4.3). A 200 response confirms the platform process is alive and accepting connections.

Step 2: Monitor Per-Server OMAG Server Health

Each logical OMAG server runs inside the platform and can be individually active or inactive. Monitor the operational status of each critical server using Egeria's server status API:

Metadata Access Server:

Click Add Monitor → set Type to HTTP / HTTPS.
Enter URL: https://your-egeria-host:9443/open-metadata/platform-services/servers/YOUR-SERVER-NAME/status
Set Expected HTTP status to 200.
Under Keyword check, add RUNNING to verify the server is in the active state (not STARTING or STOPPED).
Set Check interval to 2 minutes.
Click Save.

Repeat for each OMAG server (Integration Daemon, Engine Host, View Server) using the appropriate server name in the URL.

Step 3: Monitor JanusGraph Database Health

The Metadata Access Server persists metadata in JanusGraph. JanusGraph is typically deployed as an embedded instance with a BerkeleyDB backend (for single-node) or a distributed Cassandra + Elasticsearch backend (for production). Monitor JanusGraph's HTTP management interface if exposed, or its underlying storage:

JanusGraph HTTP health (if management endpoint is enabled):

Click Add Monitor → set Type to HTTP / HTTPS.
Enter URL: http://your-janusgraph-host:8182/
Set Check interval to 1 minute.
Set Expected HTTP status to 200.
Click Save.

Cassandra backend (for distributed JanusGraph):

Click Add Monitor → set Type to TCP Port.
Enter Host: your-cassandra-host, Port: 9042.
Set Check interval to 1 minute.
Enable alerts on connection failure.
Click Save.

A JanusGraph failure causes the Metadata Access Server to return errors on all metadata queries, which in turn causes the View Server to fail and the Egeria React UI to show empty results.

Step 4: Monitor Kafka Cohort Event Bus Connectivity

Egeria cohorts exchange metadata over Kafka topics. If Kafka becomes unreachable, cohort members stop receiving peer metadata updates. The failure is silent — no error, just stale data.

Click Add Monitor → set Type to TCP Port.
Enter Host: your-kafka-broker-host, Port: 9092.
Set Check interval to 1 minute.
Enable alerts on connection failure.
Click Save.

For Kafka clusters with multiple brokers, add one TCP monitor per broker. Any broker failure that removes a partition leader from the cohort topics should page immediately.

Also configure consumer group lag monitoring for the Egeria cohort consumer groups using Kafka Lag Exporter or Burrow:

- alert: EgeriaCohortConsumerLag
  expr: kafka_consumergroup_lag{consumergroup=~"egeria.*"} > 100
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Egeria cohort consumer group is lagging"
    description: "Cohort topic lag is {{ $value }}. Metadata exchange may be delayed."

Step 5: Monitor Integration Daemon Connector Health

Integration daemon connectors are the workhorses of Egeria metadata ingestion. Each connector runs on a schedule and calls an external system (a database, a BI tool, a file system) to import or export metadata. A connector failure means that system's metadata is no longer being updated in the Egeria cohort.

Egeria provides connector status via the integration daemon's REST API. Create a heartbeat monitor for each critical connector:

Script-based heartbeat approach:

Write a small polling script that calls the integration daemon status endpoint and sends a heartbeat to Vigilmon if the connector is running:

#!/bin/bash
STATUS=$(curl -s -u "$EGERIA_USER:$EGERIA_PASS" \
  "https://your-egeria-host:9443/servers/YOUR-INTEGRATION-DAEMON/open-metadata/integration-daemon/users/${EGERIA_USER}/connectors/YOUR-CONNECTOR-NAME" \
  | jq -r '.connectorStatus')

if [ "$STATUS" = "RUNNING" ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_HEARTBEAT_SLUG"
else
  echo "Connector status: $STATUS — not sending heartbeat"
fi

Schedule this script every 10 minutes with cron. If the connector is not in RUNNING status, the heartbeat is not sent and Vigilmon alerts on the missed ping after your configured grace period.

Create one Vigilmon heartbeat monitor per critical connector:

Go to Heartbeat Monitors → click Add Heartbeat.
Name it Egeria JDBC Connector (or the connector name).
Set Expected ping interval to 15 minutes.
Set Alert after to 1 missed ping.
Copy the heartbeat URL and paste it into your polling script.
Click Save.

Step 6: Monitor Governance Engine Health

The Engine Host runs governance action engines that execute automated data quality checks, classifications, and stewardship workflows. An Engine Host failure means governance actions stop firing — data quality issues go uncaught and classification policies are not applied to new assets.

Click Add Monitor → set Type to HTTP / HTTPS.
Enter URL: https://your-egeria-host:9443/servers/YOUR-ENGINE-HOST-NAME/status
Set Expected HTTP status to 200.
Under Keyword check, add RUNNING.
Set Check interval to 2 minutes.
Click Save.

For governance action completion rate, configure a Prometheus metric from Egeria's audit log or engine host metrics endpoint and alert on zero completion events over a 1-hour window when governance actions should be running.

Step 7: Monitor the View Server and Egeria React UI

The View Server provides the REST API layer for the Egeria React UI. A View Server failure makes the UI return errors even when all backend servers are healthy.

Click Add Monitor → set Type to HTTP / HTTPS.
Enter URL: https://your-egeria-host:9443/servers/YOUR-VIEW-SERVER-NAME/status
Set Expected HTTP status to 200.
Under Keyword check, add RUNNING.
Set Check interval to 2 minutes.
Click Save.

If you expose the Egeria React UI on a separate port or domain, add a second monitor for the UI itself:

Click Add Monitor → set Type to HTTP / HTTPS.
Enter URL: https://your-egeria-ui-host/.
Set Expected HTTP status to 200.
Set Check interval to 5 minutes.
Click Save.

Step 8: Monitor Metadata Asset Count for Catalog Completeness

An unexpected drop in the total asset count in your metadata store indicates bulk deletion, catalog wipe, or a JanusGraph restore from an old backup. This is rare but catastrophic. Monitor it with a cron-driven heartbeat:

#!/bin/bash
COUNT=$(curl -s -u "$EGERIA_USER:$EGERIA_PASS" \
  "https://your-egeria-host:9443/servers/YOUR-METADATA-SERVER/open-metadata/access-services/asset-catalog/users/${EGERIA_USER}/assets/count" \
  | jq '.count')

if [ -n "$COUNT" ] && [ "$COUNT" -gt 100 ]; then
  curl -s "https://vigilmon.online/api/v1/heartbeats/YOUR_ASSET_COUNT_HEARTBEAT"
fi

This sends a heartbeat only when the asset count is above your expected minimum. Adjust 100 to a sensible floor for your catalog size. If a bulk deletion drops the count below threshold, the heartbeat is not sent and Vigilmon alerts.

Step 9: Monitor OMAG REST API Response Time

Egeria REST services can slow down under JanusGraph query pressure or Kafka backpressure. Set a baseline expectation:

- alert: EgeriaHighAPILatency
  expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="egeria"}[5m])) > 2.0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Egeria REST API p95 latency > 2s"
    description: "OMAG platform API is slow. Check JanusGraph query performance and Kafka connectivity."

Alert Reference

| Monitor | Alert Threshold | Severity | Impact | |---|---|---|---| | OMAG Platform /server-platform/origin | Any failure | Critical | All metadata services unavailable | | Metadata Access Server status | Not RUNNING | Critical | Metadata queries failing | | JanusGraph TCP / HTTP | Connection failure | Critical | Metadata store inaccessible | | Kafka broker TCP | Connection failure | High | Cohort metadata exchange blocked | | Cohort consumer group lag | > 100 for 10m | Warning | Metadata exchange delayed | | Integration daemon connector | Heartbeat missed | High | External metadata going stale | | Engine Host status | Not RUNNING | High | Governance actions blocked | | View Server status | Not RUNNING | Medium | Egeria React UI broken | | Asset count | Below minimum floor | High | Possible bulk deletion | | REST API p95 latency | > 2s | Warning | Metadata query performance degraded |

Conclusion

Egeria's federated architecture makes it powerful but subtle to monitor. The most severe failure modes — cohort dropout, JanusGraph unavailability, and integration daemon failures — are all silent by default. The setup in this guide catches them proactively: platform-level uptime checks catch process failures in seconds, TCP monitors catch backend connectivity issues before Egeria surfaces them as errors, and heartbeat-based connector monitoring catches staleness that REST status endpoints would miss.

Start with the OMAG platform health check, the JanusGraph TCP monitor, and the Kafka broker TCP monitor — those three cover the critical path. Then add per-server status checks and connector heartbeats as you onboard each integration connector in your data governance stack.