tutorial

Monitoring Apache Gravitino with Vigilmon

Apache Gravitino is the unified metadata catalog connecting Hive, Iceberg, JDBC, and cloud catalogs — but catalog connector failures and DDL errors are silent without external monitoring. Here's how to monitor Gravitino end-to-end with Vigilmon.

Apache Gravitino (top-level Apache project, 2024) solves a problem every data platform team eventually hits: metadata fragmentation across incompatible catalog systems. Hive Metastore, AWS Glue, Iceberg REST catalog, Delta Lake, JDBC databases, and Elasticsearch each have their own catalog API. Gravitino provides a single unified REST API and web UI through which data engineers can discover, manage, and govern tables across all of these systems simultaneously — with unified RBAC, column-level lineage, and compatibility with Apache Spark, Flink, Trino, and Presto as catalog providers. When Gravitino is healthy, your compute engines see a consistent metadata plane across the entire data platform. When it fails — due to a database connectivity loss, a catalog connector outage, or an API crash — query planning breaks silently across every engine that depends on it. Vigilmon gives you the external monitoring layer to catch Gravitino failures before they cascade to your users.

What You'll Set Up

  • HTTP probe for Gravitino REST API server health
  • Heartbeat monitors for each registered catalog connector
  • DDL operation success rate monitoring via a health sidecar
  • Gravitino metadata database health monitoring
  • Cache health and API latency monitoring
  • Web UI health probe

Prerequisites

  • Apache Gravitino 0.5+ with at least one registered catalog (Hive Metastore, Iceberg REST, or JDBC)
  • Gravitino backing database configured (H2 for development, or MySQL/PostgreSQL for production)
  • At least one compute engine (Spark, Flink, or Trino) using Gravitino as a catalog provider
  • A free Vigilmon account

Why Monitoring Gravitino Matters

Gravitino is a control-plane component — it does not hold your data, but it controls whether your compute engines can find it. Its failure modes are subtle:

  • REST API server crash — All catalog metadata operations fail immediately. Spark, Trino, and Flink jobs that use Gravitino as a catalog provider cannot plan queries. The first symptom your users see is Table not found errors at job submission time.
  • Catalog connector failure — Gravitino proxies requests to underlying catalogs. If the Hive Metastore connector loses connectivity, all Hive-managed tables become invisible to query engines — even though the Gravitino API server itself is running and healthy.
  • Backing database loss — Gravitino stores its catalog registry in a relational database. A PostgreSQL or MySQL outage causes Gravitino to lose access to all registered catalog configurations, making recovery non-trivial.
  • DDL failure spike — A misconfigured catalog connector can cause CREATE TABLE and ALTER TABLE operations to fail for a subset of catalogs without surfacing clearly in logs. Monitoring DDL success rates gives you early visibility into connector-level issues.
  • Authorization service degradation — Gravitino enforces RBAC across all catalogs. If the authorization enforcement layer is degraded, either legitimate users get blocked (causing job failures) or access controls silently stop being enforced (a security incident).

Step 1: Monitor the Gravitino REST API

Gravitino exposes a version endpoint that confirms API server health:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://gravitino-server:8090/api/version.
  3. Set Check interval to 1 minute.
  4. Set Expected HTTP status to 200.
  5. Set Response body contains to "version".
  6. Click Save.

For a richer health check that validates the catalog registry is available, probe the metalakes list endpoint:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://gravitino-server:8090/api/metalakes.
  3. Set Check interval to 2 minutes.
  4. Set Expected HTTP status to 200.
  5. Set Response time threshold to 2000ms.
  6. Click Save.

A slow response on the metalakes endpoint (approaching 2 seconds) indicates database query pressure or cache degradation. Set a warning alert at 1500ms and a critical alert at 2000ms to catch degradation early.

Step 2: Monitor Catalog Connector Health

Each catalog registered in Gravitino has a connector that must maintain connectivity to the underlying system. Build a health sidecar that tests each catalog connector and exposes results as HTTP endpoints:

#!/usr/bin/env python3
# /opt/monitoring/gravitino_catalog_health.py
# Gravitino catalog connector health sidecar — port 9401

from http.server import HTTPServer, BaseHTTPRequestHandler
import json, urllib.request, os

GRAVITINO_URL = os.getenv("GRAVITINO_URL", "http://gravitino-server:8090")
METALAKE = os.getenv("GRAVITINO_METALAKE", "default")

def check_catalog(catalog_name):
    """Test catalog connectivity by listing schemas."""
    try:
        url = f"{GRAVITINO_URL}/api/metalakes/{METALAKE}/catalogs/{catalog_name}/schemas"
        with urllib.request.urlopen(url, timeout=10) as r:
            data = json.loads(r.read())
            return {"status": "ok", "schema_count": len(data.get("identifiers", []))}
    except Exception as e:
        return {"status": "down", "error": str(e)}

class CatalogHealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if not self.path.startswith("/health/catalog/"):
            self.send_response(404)
            self.end_headers()
            return

        catalog_name = self.path.split("/health/catalog/")[-1]
        result = check_catalog(catalog_name)

        status_code = 200 if result["status"] == "ok" else 503
        self.send_response(status_code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(result).encode())

    def log_message(self, format, *args):
        pass

if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 9401), CatalogHealthHandler)
    server.serve_forever()

Run this as a systemd service. Then add a Vigilmon HTTP monitor per catalog:

For Hive Metastore catalog:

  1. Click Add Monitor → HTTP / HTTPS.
  2. Enter: http://gravitino-server:9401/health/catalog/hive_catalog.
  3. Set Check interval to 2 minutes.
  4. Set Expected HTTP status to 200.
  5. Click Save.

Repeat for each production catalog (Iceberg REST, PostgreSQL JDBC, etc.). Name each monitor clearly: Gravitino — Hive Catalog, Gravitino — Iceberg Catalog, Gravitino — PostgreSQL Catalog.

Step 3: Monitor DDL Operation Success Rate

Gravitino handles DDL operations (CREATE TABLE, ALTER TABLE, DROP TABLE) across catalogs. A connector misconfiguration can cause DDL operations to fail silently for a specific catalog. Add a DDL health heartbeat:

#!/bin/bash
# /opt/monitoring/check-gravitino-ddl.sh
# Creates a test table in a monitoring catalog, then drops it
# Pings heartbeat only if both operations succeed

GRAVITINO_URL="http://gravitino-server:8090"
METALAKE="default"
CATALOG="monitoring_catalog"    # a dedicated low-stakes catalog for health checks
SCHEMA="health_checks"
TABLE="vigilmon_ddl_test_$(date +%s)"
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DDL_KEY"

AUTH_HEADER="Authorization: Bearer ${GRAVITINO_TOKEN}"

# Create test table
CREATE_RESPONSE=$(curl -sf -w "%{http_code}" -o /dev/null \
  -X POST \
  -H "Content-Type: application/json" \
  -H "$AUTH_HEADER" \
  -d "{
    \"name\": \"$TABLE\",
    \"columns\": [{\"name\": \"id\", \"type\": \"integer\", \"nullable\": false}],
    \"comment\": \"Vigilmon DDL health check\"
  }" \
  "$GRAVITINO_URL/api/metalakes/$METALAKE/catalogs/$CATALOG/schemas/$SCHEMA/tables")

if [ "$CREATE_RESPONSE" != "200" ]; then
  exit 1
fi

# Drop test table
DROP_RESPONSE=$(curl -sf -w "%{http_code}" -o /dev/null \
  -X DELETE \
  -H "$AUTH_HEADER" \
  "$GRAVITINO_URL/api/metalakes/$METALAKE/catalogs/$CATALOG/schemas/$SCHEMA/tables/$TABLE")

if [ "$DROP_RESPONSE" = "200" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Schedule this every 10 minutes. Set the Vigilmon heartbeat interval to 20 minutes. DDL failures for the monitoring catalog indicate connector-level issues affecting all DDL operations for that catalog type.

Step 4: Monitor the Gravitino Backing Database

Gravitino's catalog registry lives in a relational database. A database failure causes Gravitino to lose all catalog configuration state. Add connectivity monitoring:

For PostgreSQL

  1. In Vigilmon, click Add Monitor → TCP Port.
  2. Enter postgres-host:5432.
  3. Set Check interval to 1 minute.
  4. Click Save.

For a deeper check that validates Gravitino can actually reach its database:

#!/bin/bash
# /opt/monitoring/check-gravitino-db.sh
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_DB_KEY"

# The Gravitino /api/version endpoint requires a DB read for catalog count metadata
RESPONSE=$(curl -sf -w "%{http_code}" -o /tmp/gravitino_version.json \
  "http://gravitino-server:8090/api/metalakes?limit=1")

if [ "$RESPONSE" = "200" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Schedule every 2 minutes. The metalakes endpoint requires a live database read — if the database is unreachable, this endpoint will return 503, preventing the heartbeat from firing.

For H2 Embedded (Development Only)

Add a Vigilmon HTTP monitor for the Gravitino API that includes a response time threshold of 500ms — H2 degradation typically manifests as slow response times before outright failures.

Step 5: Monitor API Response Latency

Gravitino metadata queries should respond within 500ms for catalog browsing operations. Slower responses indicate database pressure or cache exhaustion. Add a Vigilmon HTTP monitor with a response time alert:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://gravitino-server:8090/api/metalakes/default/catalogs.
  3. Set Check interval to 2 minutes.
  4. Set Expected HTTP status to 200.
  5. Set Response time threshold to 500ms (alert if exceeded).
  6. Click Save.

For Spark, Trino, or Flink jobs that use Gravitino as a catalog provider, add a heartbeat to your critical ETL jobs that confirms catalog queries are succeeding:

# In your PySpark job using Gravitino as Spark catalog
import requests, os

def ping_vigilmon_catalog_health():
    """Confirm Gravitino catalog is responding before starting expensive computation."""
    try:
        resp = requests.get(
            "http://gravitino-server:8090/api/metalakes/default/catalogs",
            timeout=5
        )
        if resp.status_code == 200:
            requests.get(os.environ["VIGILMON_SPARK_CATALOG_HEARTBEAT"], timeout=5)
    except Exception:
        pass  # Let the heartbeat expire — do not swallow Spark job failures

# Call at job startup and every 10 minutes during long-running jobs
ping_vigilmon_catalog_health()

Step 6: Monitor Gravitino Web UI

The Gravitino web console allows data engineers to browse and manage catalogs. A web UI failure blocks non-programmatic catalog management:

  1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
  2. Enter: http://gravitino-server:8090/ui/ (or your Gravitino web UI URL).
  3. Set Check interval to 5 minutes.
  4. Set Expected HTTP status to 200.
  5. Click Save.

The web UI is served by the same process as the REST API, so a web UI failure that coincides with a healthy REST API check indicates a static asset serving issue — still worth knowing, but lower priority than API failures.

Step 7: Monitor Table Count per Catalog

A sudden drop in the number of tables registered in Gravitino for a given catalog indicates a catalog synchronization failure or an accidental bulk delete. Monitor table counts with a heartbeat:

#!/bin/bash
# /opt/monitoring/check-gravitino-table-count.sh
CATALOG="hive_catalog"
SCHEMA="production"
MIN_TABLE_COUNT=50   # Alert if fewer than 50 tables in production schema
HEARTBEAT_URL="https://vigilmon.online/heartbeat/YOUR_TABLE_COUNT_KEY"

TABLE_COUNT=$(curl -sf \
  "http://gravitino-server:8090/api/metalakes/default/catalogs/$CATALOG/schemas/$SCHEMA/tables" | \
  python3 -c "import sys, json; print(len(json.load(sys.stdin).get('identifiers', [])))")

if [ "${TABLE_COUNT:-0}" -ge "$MIN_TABLE_COUNT" ]; then
  curl -fsS "$HEARTBEAT_URL"
fi

Run hourly. Set the Vigilmon heartbeat interval to 2 hours. A catalog synchronization failure that drops tables will prevent the heartbeat from firing within 2 hours.

Step 8: Configure Alert Routing

| Monitor | Alert Channel | Severity | Impact | |---|---|---|---| | Gravitino REST API /api/version | PagerDuty + Slack | Critical | All catalog operations fail — query planning broken | | Catalog connector health (production catalogs) | PagerDuty + Slack | Critical | Specific catalog tables invisible to query engines | | Backing database TCP | Slack | High | Gravitino loses catalog registry | | DDL success heartbeat | Slack | High | DDL operations failing — schema management broken | | API latency HTTP | Slack | Medium | Slow catalog queries degrading job startup times | | Web UI health | Email | Low | Console unavailable — programmatic access unaffected | | Table count heartbeat | Email | Medium | Catalog sync failure or accidental bulk delete |

Set consecutive failures before alert to 1 for the REST API and catalog connector monitors — these directly block production jobs. Set it to 3 for the API latency monitor to avoid false alerts during brief load spikes.

Conclusion

Apache Gravitino is your data platform's metadata control plane — when it fails, every compute engine that depends on it for catalog discovery stops being able to find your data. The failure modes range from obvious (REST API crash) to subtle (a single catalog connector losing connectivity while the API reports healthy). With Vigilmon monitoring the API server, each catalog connector, the backing database, DDL success rates, and API response latency, you have end-to-end visibility into Gravitino's health across all of its responsibilities.

Start with the REST API health probe and per-catalog connector monitors — these cover the critical path for all catalog operations. Then add DDL health and table count monitoring for production schemas. Sign up for a free Vigilmon account to get started.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles