Monitoring GeoNetwork with Vigilmon

GeoNetwork is the open source geospatial metadata catalog used by national mapping agencies, INSPIRE directive implementations, and UN organizations like the FAO and WFP to publish and discover spatial datasets. It runs as a Java web application on Apache Tomcat backed by Elasticsearch (4.x) and PostgreSQL — a multi-process stack where a failure in any layer silently breaks catalog discovery for every GIS tool pointing at your CSW endpoint. Vigilmon gives you coverage across every tier: web app availability, search engine health, database connectivity, CSW endpoint response time, and harvesting job success — all without running your own Prometheus stack.

What You'll Set Up

• HTTP uptime monitor for the GeoNetwork web application
• Elasticsearch cluster health monitoring
• PostgreSQL database connectivity check
• CSW endpoint health and response time alerts
• Cron heartbeat for scheduled harvesting jobs
• Alerts for metadata record count anomalies and index rebuild failures

Prerequisites

• GeoNetwork 3.x or 4.x deployed (standalone or Docker)
• A free Vigilmon account
• Access to GeoNetwork's admin API and your Elasticsearch/PostgreSQL hosts

Step 1: Monitor the GeoNetwork Web Application

GeoNetwork exposes a web UI at /geonetwork. Start with a basic HTTP monitor to catch Tomcat crashes or deployment failures:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter your GeoNetwork URL: https://geonetwork.yourdomain.com/geonetwork/srv/eng/catalog.search.
4. Set Check interval to 1 minute.
5. Set Expected HTTP status to 200.
6. Click Save.

For a more precise liveness check, GeoNetwork 4.x exposes a system info endpoint you can probe instead:

https://geonetwork.yourdomain.com/geonetwork/srv/api/site

This endpoint returns JSON describing the GeoNetwork version and configuration. A 200 response confirms the Java application is up and the Spring context loaded successfully. Set Expected response contains to "system" for an extra layer of confidence.

Step 2: Monitor Elasticsearch Cluster Health

GeoNetwork 4.x indexes all metadata records in Elasticsearch. If Elasticsearch goes red, catalog search returns no results — silently. Elasticsearch exposes a health API at /_cluster/health:

curl http://localhost:9200/_cluster/health
# {"cluster_name":"geonetwork","status":"green","number_of_nodes":1,...}

Add a Vigilmon HTTP monitor for the Elasticsearch health endpoint:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-elasticsearch-host:9200/_cluster/health.
3. Set Expected response contains to "status":"green".
4. Set Check interval to 2 minutes.
5. Click Save.

If Elasticsearch is not publicly reachable, expose a thin health proxy via your nginx configuration:

location /internal/es-health {
    allow 127.0.0.1;
    deny all;
    proxy_pass http://localhost:9200/_cluster/health;
}

Then point Vigilmon at https://geonetwork.yourdomain.com/internal/es-health with IP allowlisting disabled (Vigilmon probes come from fixed IP ranges available in your account settings).

Alert condition: any response that does not contain "status":"green" should fire immediately — a yellow status means replica shards are unassigned (acceptable in single-node setups); red means primary shards are missing and search is broken.

Step 3: Monitor PostgreSQL Database Connectivity

GeoNetwork stores all metadata records, user accounts, and harvester configuration in PostgreSQL. A database outage causes 500 errors across the entire catalog API.

Add a TCP monitor to verify PostgreSQL is accepting connections:

1. Click Add Monitor → TCP Port.
2. Host: your PostgreSQL host.
3. Port: 5432.
4. Set Check interval to 1 minute.
5. Click Save.

For a deeper check, expose a lightweight database connectivity probe via GeoNetwork's admin API:

curl -u admin:yourpassword \
  https://geonetwork.yourdomain.com/geonetwork/srv/api/sources

A 200 response confirms that GeoNetwork can reach PostgreSQL and query the sources table. Add this as a second HTTP monitor with Expected HTTP status 200 and basic auth configured under Authentication in Vigilmon.

Step 4: Monitor the CSW Endpoint

The OGC Catalogue Service for the Web (CSW) endpoint is how external tools — QGIS, GeoServer, ArcGIS — discover your metadata. A broken CSW silently disconnects every downstream client:

curl "https://geonetwork.yourdomain.com/geonetwork/srv/eng/csw?\
service=CSW&version=2.0.2&request=GetCapabilities"

Add a Vigilmon HTTP monitor for CSW availability:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: https://geonetwork.yourdomain.com/geonetwork/srv/eng/csw?service=CSW&version=2.0.2&request=GetCapabilities
3. Set Expected response contains to OGC:CSW.
4. Set Check interval to 2 minutes.
5. Under Advanced, set Alert if response time exceeds to 3000 ms.
6. Click Save.

A GetCapabilities request exercises the full CSW stack — Java servlet, Elasticsearch query, and XML serialization. Latency above 3 seconds indicates indexing pressure or slow Elasticsearch queries.

Step 5: Heartbeat Monitoring for Harvesting Jobs

GeoNetwork can pull metadata from remote catalogs (OGC CSW, OAI-PMH, WMS, ArcGIS) on a schedule. Harvesting jobs run inside GeoNetwork's scheduler — if a harvester silently fails, your metadata catalog goes stale with no alert.

Use Vigilmon's cron heartbeat to verify each harvesting job completes:

1. Click Add Monitor → Cron Heartbeat.
2. Set the expected ping interval to match your harvester schedule (e.g. 1440 minutes for daily harvesting).
3. Copy the heartbeat URL: https://vigilmon.online/heartbeat/abc123.

Call the heartbeat URL from a wrapper script that runs your GeoNetwork harvester via the admin API and then signals success:

#!/bin/bash
# run-harvester.sh — trigger GeoNetwork harvesting and ping Vigilmon on success

HARVESTER_UUID="your-harvester-uuid"
GN_BASE="https://geonetwork.yourdomain.com/geonetwork"
GN_AUTH="admin:yourpassword"

# Trigger the harvesting run
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
  -u "$GN_AUTH" \
  -X PUT \
  "$GN_BASE/srv/api/harvesters/$HARVESTER_UUID/run")

if [ "$RESPONSE" = "204" ]; then
  # Signal success to Vigilmon
  curl -s "https://vigilmon.online/heartbeat/abc123"
else
  echo "Harvester trigger failed with HTTP $RESPONSE" >&2
  exit 1
fi

Schedule this wrapper with cron or your OS scheduler. If harvesting fails — network error, remote catalog unreachable, authentication problem — the heartbeat never fires and Vigilmon alerts after the expected interval passes.

Step 6: Monitor Metadata Record Count

An unexpected drop in metadata record count — caused by accidental mass deletion, a bad import, or a failed sync — is one of the hardest GeoNetwork problems to catch. GeoNetwork exposes record counts via its search API:

curl -s \
  "https://geonetwork.yourdomain.com/geonetwork/srv/api/search/records/_search" \
  -H "Content-Type: application/json" \
  -d '{"query":{"match_all":{}},"size":0}' | jq '.hits.total.value'

Build a monitoring script that compares today's count against a baseline and calls the Vigilmon push API if it drops:

#!/usr/bin/env python3
import requests, os

BASELINE = int(os.environ.get("GN_RECORD_BASELINE", "5000"))
GN_BASE = os.environ["GN_BASE_URL"]
VIGILMON_KEY = os.environ["VIGILMON_API_KEY"]
MONITOR_ID = os.environ["VIGILMON_RECORD_MONITOR_ID"]

resp = requests.post(
    f"{GN_BASE}/geonetwork/srv/api/search/records/_search",
    json={"query": {"match_all": {}}, "size": 0},
    auth=("admin", os.environ["GN_PASSWORD"]),
    timeout=10,
)
resp.raise_for_status()
count = resp.json()["hits"]["total"]["value"]

if count < BASELINE * 0.95:
    # Alert via Vigilmon API — mark monitor as down
    requests.post(
        f"https://vigilmon.online/api/monitors/{MONITOR_ID}/report",
        headers={"Authorization": f"Bearer {VIGILMON_KEY}"},
        json={"status": "down", "message": f"Record count dropped to {count} (baseline {BASELINE})"},
    )

Run this script via cron every 15 minutes. The 5% threshold (baseline × 0.95) avoids false alerts from normal minor record deletions while catching mass deletions immediately.

Step 7: Configure Alert Channels

1. Go to Alert Channels in Vigilmon and add your Slack workspace, email, or PagerDuty integration.
2. For the GeoNetwork web app monitor, set Consecutive failures before alert to 2 — Tomcat GC pauses can cause single-probe timeouts.
3. For the CSW endpoint monitor, set Consecutive failures before alert to 1 — CSW failures affect all downstream GIS clients immediately.
4. Use Vigilmon Maintenance windows during GeoNetwork upgrades or Elasticsearch re-indexing:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "monitor_id": "csw-monitor-id",
    "duration_minutes": 30,
    "reason": "GeoNetwork index rebuild"
  }'

Index rebuilds in GeoNetwork 4.x can take 5–30 minutes on large catalogs. A maintenance window prevents the rebuild latency from triggering false CSW timeout alerts.

Summary

| Monitor | Target | What It Catches | |---|---|---| | GeoNetwork web app | /geonetwork/srv/api/site | Tomcat crash, Spring startup failure | | Elasticsearch cluster | /_cluster/health (status: green) | Search index unavailable, shard failures | | PostgreSQL TCP | Port 5432 | Database server down | | CSW endpoint | GetCapabilities response | CSW broken for downstream GIS tools | | Harvester heartbeat | Cron heartbeat URL | Silent harvesting job failure | | Record count script | Vigilmon push API | Mass metadata deletion or import failure |

GeoNetwork powers the spatial data infrastructure for governments, UN agencies, and research institutions — its catalog is only as reliable as the stack it runs on. With Vigilmon watching each tier, you get immediate alerts whether the failure is a Tomcat crash, an Elasticsearch shard loss, a PostgreSQL outage, or a silent harvesting failure — before your SDI stakeholders notice empty search results in QGIS or ArcGIS.