Monitoring Lagoon with Vigilmon

Lagoon is an open source, Kubernetes-native application hosting and delivery platform developed by amazee.io. It gives development teams a GitOps-driven experience where every git branch automatically becomes a deployed environment — production, staging, or a PR preview — running on Kubernetes. When Lagoon is healthy, developers push code and their environments update automatically. When it isn't, the entire developer platform grinds to a halt: deployments stop, environments go stale, and developers lose the self-service workflow they depend on.

Vigilmon lets you monitor each Lagoon microservice — the GraphQL API, webhook handler, build pipelines, RabbitMQ broker, Keycloak, Harbor registry, and per-environment ingress — so you catch failures before developers notice them.

What You'll Set Up

• Lagoon API (GraphQL) health monitoring
• Webhook handler reachability checks
• Build pipeline success rate and queue depth alerting
• RabbitMQ broker and queue depth monitoring
• Keycloak authentication service health
• Harbor container registry health
• Per-environment pod and ingress checks
• Lagoon database connectivity monitoring
• SSL certificate alerts for all deployed environments

Prerequisites

• Lagoon deployed on a Kubernetes cluster (v1.21+)
• Access to the Lagoon API endpoint and the Lagoon admin console
• kubectl access to the Lagoon namespace
• A free Vigilmon account

Step 1: Monitor the Lagoon API (GraphQL Endpoint)

The Lagoon API is the control plane for all project and environment management. If it goes down, developers cannot create projects, trigger builds, or access environment variables through the Lagoon UI or CLI.

Add an HTTP monitor for the GraphQL introspection endpoint:

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to HTTP / HTTPS.
3. Enter the Lagoon API URL: https://api.lagoon.yourdomain.com/graphql
4. Set HTTP Method to POST and add the body:

   {"query": "{ __typename }"}
   

5. Set Expected HTTP status to 200.
6. Set Check interval to 1 minute.
7. Enable Monitor SSL certificate and set the expiry alert to 21 days.
8. Click Save.

A 200 response with {"data":{"__typename":"Query"}} confirms the API is reachable and the GraphQL layer is functional. An HTTP 502 or timeout means the API pods have crashed or the nginx ingress to the API is broken.

Step 2: Monitor the Webhook Handler

Lagoon's webhook handler receives git push events from GitHub, GitLab, and Bitbucket and queues them for build processing. If it goes down, automatic deployments triggered by git push silently stop working — developers push code and nothing happens.

Add a reachability monitor for the webhook endpoint:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter the webhook handler URL: https://webhooks.lagoon.yourdomain.com
3. Set Expected HTTP status to 200 (or 404 if the handler returns 404 on GET — check your deployment).
4. Set Check interval to 1 minute.
5. Click Save.

To also verify the webhook handler can actually process events, add a Vigilmon Cron Heartbeat that gets pinged by a test webhook. Configure your git provider to send a test webhook on a regular schedule (many platforms support scheduled webhooks or you can trigger them from CI):

# Ping Vigilmon heartbeat after a scheduled test webhook delivery succeeds
curl -s https://vigilmon.online/heartbeat/YOUR_WEBHOOK_HB_TOKEN

Step 3: Monitor Build Pipeline Health via Heartbeat

Lagoon uses Tekton pipelines to build and deploy git branches. Build failures are expected occasionally, but a high failure rate or a growing build queue indicates a systemic problem.

Heartbeat from successful builds

The most reliable signal is a heartbeat from the Lagoon build pipeline itself. Add a post-build hook to your Lagoon .lagoon.yml:

tasks:
  post-rollout:
    - run:
        name: Notify Vigilmon
        command: |
          curl -s https://vigilmon.online/heartbeat/YOUR_BUILD_HB_TOKEN || true
        service: cli

Set the Vigilmon heartbeat interval to match your expected deployment frequency. If builds run at least once per hour for any project, set the heartbeat interval to 2 hours.

Alert on RabbitMQ queue depth (see Step 4)

Build pipeline lag manifests as a growing RabbitMQ queue — monitor it in Step 4 to catch build backlogs before they become hours-long outages.

Step 4: Monitor RabbitMQ Broker Health

Lagoon uses RabbitMQ for asynchronous task queuing between the webhook handler, the Lagoon API, and the build system. A RabbitMQ crash or a spike in queue depth means builds are not being processed.

Add an HTTP monitor for the RabbitMQ management API:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter: http://lagoon-broker.lagoon.svc.cluster.local:15672/api/healthchecks/node (or the externally exposed RabbitMQ management URL if accessible).
3. Set Expected HTTP status to 200.
4. Set Expected body to contain "status":"ok".
5. Set Check interval to 2 minutes.
6. Click Save.

For queue depth monitoring, expose a custom health endpoint via a small script that queries the RabbitMQ API and returns a non-200 status when depth exceeds your threshold:

#!/bin/bash
DEPTH=$(curl -s -u guest:guest http://localhost:15672/api/queues/%2F/lagoon-tasks \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['messages'])")
if [ "$DEPTH" -gt 50 ]; then
  echo "Queue depth critical: $DEPTH"
  exit 1
fi
curl -s https://vigilmon.online/heartbeat/YOUR_RABBITMQ_HB_TOKEN

Run this script as a Kubernetes CronJob every 5 minutes and alert when the heartbeat goes silent.

Step 5: Monitor Keycloak Authentication Health

Lagoon uses Keycloak for single sign-on and RBAC. A Keycloak crash means developers cannot log in to the Lagoon UI or CLI — they lose access to all Lagoon functionality even if the API itself is running.

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter the Keycloak health endpoint: https://keycloak.lagoon.yourdomain.com/health/ready
3. Set Expected HTTP status to 200.
4. Set Check interval to 1 minute.
5. Enable Monitor SSL certificate, set expiry alert to 21 days.
6. Click Save.

If Keycloak does not expose /health/ready, use the realm endpoint instead:

https://keycloak.lagoon.yourdomain.com/realms/lagoon

A 200 response from the realm endpoint confirms Keycloak is running and the Lagoon realm is configured.

Step 6: Monitor Harbor Container Registry

Lagoon pushes built container images to Harbor and pulls them during deployments. If Harbor is unavailable, builds fail at the image push step and no new deployments can complete.

Add an HTTP monitor for the Harbor health API:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter: https://harbor.lagoon.yourdomain.com/api/v2.0/health
3. Set Expected HTTP status to 200.
4. Set Expected body to contain "status":"healthy".
5. Set Check interval to 2 minutes.
6. Enable Monitor SSL certificate, set expiry alert to 21 days.
7. Click Save.

Step 7: Monitor Per-Environment Ingress

Each Lagoon project branch gets its own environment with an ingress. Production environment ingress failures are critical — they mean the live site is down.

Add an HTTP monitor for each production environment:

1. In Vigilmon, click Add Monitor → HTTP / HTTPS.
2. Enter the production environment URL: https://main.myproject.lagoon.yourdomain.com
3. Set Expected HTTP status to 200.
4. Set Check interval to 1 minute.
5. Enable Monitor SSL certificate, set expiry alert to 14 days (tighter threshold for production).
6. Click Save.

Repeat for each production Lagoon environment. For PR preview environments, a longer check interval (5 minutes) and a higher failure threshold (3 failures before alert) reduces noise from short-lived environments.

To script adding monitors for all Lagoon production environments:

# List all Lagoon environment URLs via the Lagoon CLI
lagoon list environments --project myproject \
  | grep production \
  | awk '{print $3}' \
  | while read url; do
    echo "Add monitor for: $url"
  done

Step 8: Monitor the Lagoon Database

The Lagoon API stores all project and environment state in MariaDB. A database connectivity failure causes the Lagoon API to return errors on all project management operations.

Add a TCP port monitor for the MariaDB service:

1. In Vigilmon, click Add Monitor → TCP Port.
2. Enter the MariaDB host and port: lagoon-mariadb.lagoon.svc.cluster.local:3306 (or the external IP if monitoring from outside the cluster).
3. Set Check interval to 1 minute.
4. Click Save.

For a deeper check that validates query execution, expose a database health endpoint from the Lagoon API (or a sidecar) and monitor it:

// Express health endpoint checking DB
app.get('/db-health', async (req, res) => {
  try {
    await pool.query('SELECT 1');
    res.json({ status: 'ok' });
  } catch (e) {
    res.status(503).json({ status: 'error', message: e.message });
  }
});

Step 9: Configure Alert Channels and Thresholds

1. Go to Alert Channels in Vigilmon and add Slack and email.
2. Set priority by monitor:
   • Lagoon API, Keycloak, Harbor: alert after 1 failure — these are single points of failure.
   • Webhook handler, RabbitMQ: alert after 2 consecutive failures.
   • Production environment ingress: alert after 1 failure with immediate notification.
   • PR preview environments: alert after 3 failures to reduce noise.
3. Create a Maintenance Window in Vigilmon before planned Lagoon upgrades:

curl -X POST https://vigilmon.online/api/maintenance \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"monitor_id": "LAGOON_API_MONITOR_ID", "duration_minutes": 30}'

Summary

| Monitor | Target | What It Catches | |---|---|---| | Lagoon API (GraphQL) | https://api.lagoon.domain/graphql | API pod crash, GraphQL layer failure | | Webhook handler | https://webhooks.lagoon.domain | Deployment trigger failure | | Build pipeline heartbeat | Heartbeat URL | Build system stalled or failing | | RabbitMQ health | RabbitMQ management API | Broker crash, queue backlog | | Keycloak health | /health/ready | Developer login blocked | | Harbor registry | /api/v2.0/health | Image push/pull failure | | Production environment | Per-environment URL | Live site down | | MariaDB | TCP port 3306 | API state loss | | SSL certificates | All environment domains | Certificate expiry |

Lagoon abstracts Kubernetes complexity for your developers, but that abstraction depends on a chain of microservices all staying healthy. With Vigilmon watching the API, authentication, build pipeline, message broker, and every production environment, you get early warning on any failure in that chain — before developers start filing tickets wondering why their git push did nothing.