How to Monitor VolumeSync PVC Replication Health, Sync Status, and Restore Integrity with Vigilmon

VolumeSync (volsync) is an open-source Kubernetes operator developed by Red Hat for asynchronous replication of PersistentVolumeClaims (PVCs) between clusters or to object storage. It supports Rsync-TLS for cross-cluster replication, RClone for S3/MinIO/GCS targets, and Restic for incremental encrypted backups — all through a unified CRD-based API using ReplicationSource and ReplicationDestination objects.

The challenge with VolumeSync is that replication failures are silent. A failed Rsync handshake, an expired S3 token, or a VolumeSnapshot that times out will simply mark a sync job as failed and wait for the next scheduled run — without alerting anyone. By the time you discover a backup is stale, you may already be past your Recovery Point Objective (RPO). Vigilmon gives you external monitoring of VolumeSync health endpoints so you know the moment a replication schedule starts drifting.

Why VolumeSync Needs External Monitoring

VolumeSync manages critical data protection schedules, but it has no built-in notification system for sync failures or RPO drift. External monitoring with Vigilmon adds:

• Sync failure alerting when any ReplicationSource fails to complete within 2× its scheduled interval
• RPO drift detection — alerts when the most recent restore point is older than your recovery objective
• Rsync and RClone connectivity checks — catches cross-cluster network failures before the next scheduled sync
• Restic repository integrity monitoring — detects repository corruption that would make backups unrestorable
• VolumeSnapshot API health — catches snapshot creation failures that produce inconsistent backup sets
• Restore test validation — periodic test restores that fail silently are as dangerous as no backup at all

Step 1: Build a VolumeSync Health Sidecar

VolumeSync exposes replication status through Kubernetes CRD status fields. A health sidecar queries the Kubernetes API, aggregates ReplicationSource and ReplicationDestination status, and exposes an HTTP endpoint for Vigilmon.

Node.js Health Sidecar

// health/volsync.js
const express = require('express');
const { KubeConfig, CustomObjectsApi } = require('@kubernetes/client-node');

const app = express();

const kc = new KubeConfig();
kc.loadFromDefault(); // uses in-cluster config or ~/.kube/config
const k8s = kc.makeApiClient(CustomObjectsApi);

const NAMESPACE = process.env.VOLSYNC_NAMESPACE || 'default';
const RPO_HOURS = parseFloat(process.env.VOLSYNC_RPO_HOURS || '24');
const SYNC_OVERRUN_FACTOR = parseFloat(process.env.VOLSYNC_OVERRUN_FACTOR || '2.0');

async function getReplicationSources() {
  const res = await k8s.listNamespacedCustomObject(
    'volsync.backube', 'v1alpha1', NAMESPACE, 'replicationsources'
  );
  return res.body.items || [];
}

async function getReplicationDestinations() {
  const res = await k8s.listNamespacedCustomObject(
    'volsync.backube', 'v1alpha1', NAMESPACE, 'replicationdestinations'
  );
  return res.body.items || [];
}

function parseScheduleToSeconds(schedule) {
  // Basic cron-to-seconds heuristic: @hourly=3600, @daily=86400, @weekly=604800
  if (!schedule) return null;
  if (schedule === '@hourly') return 3600;
  if (schedule === '@daily') return 86400;
  if (schedule === '@weekly') return 604800;
  // Try to parse "0 */6 * * *" style — count minimum interval from field 1
  return null;
}

app.get('/health/volsync', async (req, res) => {
  try {
    const sources = await getReplicationSources();
    const failing = [];

    for (const src of sources) {
      const status = src.status || {};
      const spec = src.spec || {};
      const name = src.metadata.name;
      const lastSync = status.lastSyncTime ? new Date(status.lastSyncTime) : null;
      const lastResult = status.lastSyncDuration;
      const conditions = status.conditions || [];

      // Check for sync failure condition
      const failCond = conditions.find(
        c => c.type === 'Synchronizing' && c.status === 'False' && c.reason !== 'Idle'
      );
      if (failCond) {
        failing.push({
          source: name,
          reason: 'sync_condition_failed',
          message: failCond.message,
          last_sync: lastSync?.toISOString(),
        });
        continue;
      }

      // Check for schedule overrun
      const scheduleSeconds = parseScheduleToSeconds(spec.trigger?.schedule);
      if (scheduleSeconds && lastSync) {
        const secondsSinceSync = (Date.now() - lastSync.getTime()) / 1000;
        if (secondsSinceSync > scheduleSeconds * SYNC_OVERRUN_FACTOR) {
          failing.push({
            source: name,
            reason: 'sync_overdue',
            seconds_since_sync: Math.round(secondsSinceSync),
            schedule_seconds: scheduleSeconds,
            overrun_factor: SYNC_OVERRUN_FACTOR,
          });
        }
      }
    }

    if (failing.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'replication_sources_failing',
        failing,
        total_sources: sources.length,
      });
    }

    return res.status(200).json({
      status: 'ok',
      total_sources: sources.length,
      all_syncing: true,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.get('/health/volsync/destinations', async (req, res) => {
  try {
    const destinations = await getReplicationDestinations();
    const stale = [];

    for (const dest of destinations) {
      const status = dest.status || {};
      const name = dest.metadata.name;
      const lastSync = status.lastSyncTime ? new Date(status.lastSyncTime) : null;

      if (!lastSync) {
        stale.push({ destination: name, reason: 'never_synced' });
        continue;
      }

      const hoursAgo = (Date.now() - lastSync.getTime()) / (1000 * 60 * 60);
      if (hoursAgo > RPO_HOURS) {
        stale.push({
          destination: name,
          reason: 'restore_point_exceeds_rpo',
          hours_since_sync: Math.round(hoursAgo * 10) / 10,
          rpo_hours: RPO_HOURS,
          last_sync: lastSync.toISOString(),
        });
      }
    }

    if (stale.length > 0) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'stale_restore_points',
        stale,
        total_destinations: destinations.length,
      });
    }

    return res.status(200).json({
      status: 'ok',
      total_destinations: destinations.length,
      all_within_rpo: true,
      rpo_hours: RPO_HOURS,
    });
  } catch (err) {
    return res.status(503).json({ status: 'down', error: err.message });
  }
});

app.listen(3008, () => console.log('VolumeSync health sidecar listening on :3008'));

Python Health Sidecar

# health/volsync_health.py
import os, time
from flask import Flask, jsonify
from kubernetes import client, config
from datetime import datetime, timezone

app = Flask(__name__)

try:
    config.load_incluster_config()
except config.ConfigException:
    config.load_kube_config()

custom_api = client.CustomObjectsApi()
NAMESPACE = os.environ.get('VOLSYNC_NAMESPACE', 'default')
RPO_HOURS = float(os.environ.get('VOLSYNC_RPO_HOURS', 24))
OVERRUN_FACTOR = float(os.environ.get('VOLSYNC_OVERRUN_FACTOR', 2.0))

GROUP = 'volsync.backube'
VERSION = 'v1alpha1'

@app.route('/health/volsync')
def volsync_health():
    try:
        sources = custom_api.list_namespaced_custom_object(
            GROUP, VERSION, NAMESPACE, 'replicationsources'
        )['items']
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    failing = []
    for src in sources:
        name = src['metadata']['name']
        status = src.get('status', {})
        conditions = status.get('conditions', [])
        fail_cond = next(
            (c for c in conditions
             if c.get('type') == 'Synchronizing' and c.get('status') == 'False'
             and c.get('reason') != 'Idle'), None
        )
        if fail_cond:
            failing.append({'source': name, 'reason': 'sync_condition_failed',
                            'message': fail_cond.get('message')})

    if failing:
        return jsonify(status='degraded', reason='replication_sources_failing',
                       failing=failing, total_sources=len(sources)), 503

    return jsonify(status='ok', total_sources=len(sources), all_syncing=True)

@app.route('/health/volsync/destinations')
def volsync_destinations():
    try:
        dests = custom_api.list_namespaced_custom_object(
            GROUP, VERSION, NAMESPACE, 'replicationdestinations'
        )['items']
    except Exception as e:
        return jsonify(status='down', error=str(e)), 503

    stale = []
    for dest in dests:
        name = dest['metadata']['name']
        last_sync_str = dest.get('status', {}).get('lastSyncTime')
        if not last_sync_str:
            stale.append({'destination': name, 'reason': 'never_synced'})
            continue
        last_sync = datetime.fromisoformat(last_sync_str.replace('Z', '+00:00'))
        hours_ago = (datetime.now(timezone.utc) - last_sync).total_seconds() / 3600
        if hours_ago > RPO_HOURS:
            stale.append({'destination': name, 'reason': 'restore_point_exceeds_rpo',
                          'hours_since_sync': round(hours_ago, 1), 'rpo_hours': RPO_HOURS})

    if stale:
        return jsonify(status='degraded', reason='stale_restore_points',
                       stale=stale, total_destinations=len(dests)), 503

    return jsonify(status='ok', total_destinations=len(dests),
                   all_within_rpo=True, rpo_hours=RPO_HOURS)

if __name__ == '__main__':
    app.run(port=3008)

Step 2: Deploy the Health Sidecar in Kubernetes

Deploy the sidecar as a Deployment with appropriate RBAC so it can read VolumeSync CRDs:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: volsync-monitor
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: volsync-monitor-reader
rules:
  - apiGroups: ["volsync.backube"]
    resources: ["replicationsources", "replicationdestinations"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: volsync-monitor-reader
subjects:
  - kind: ServiceAccount
    name: volsync-monitor
    namespace: default
roleRef:
  kind: ClusterRole
  name: volsync-monitor-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: volsync-health-sidecar
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels: { app: volsync-health-sidecar }
  template:
    metadata:
      labels: { app: volsync-health-sidecar }
    spec:
      serviceAccountName: volsync-monitor
      containers:
        - name: health
          image: your-registry/volsync-health:latest
          ports:
            - containerPort: 3008
          env:
            - name: VOLSYNC_NAMESPACE
              value: "default"
            - name: VOLSYNC_RPO_HOURS
              value: "24"
            - name: VOLSYNC_OVERRUN_FACTOR
              value: "2.0"
---
apiVersion: v1
kind: Service
metadata:
  name: volsync-health-sidecar
  namespace: default
spec:
  selector: { app: volsync-health-sidecar }
  ports:
    - port: 3008
      targetPort: 3008

Expose the service via your ingress or use a NodePort/LoadBalancer so Vigilmon can reach it over HTTPS.

Step 3: Configure Vigilmon HTTP Monitors

1. Log in to vigilmon.online and go to Monitors → New Monitor
2. Choose HTTP / HTTPS
3. URL: https://your-cluster-ingress.example.com/health/volsync
4. Check interval: 5 minutes (sync failures are not second-level events)
5. Under Expected response:
   • Status code: 200
   • Response body contains: "status":"ok"
   • Response time threshold: 5000ms
6. Assign your alert channel
7. Save

Add monitors for each health endpoint:

| Monitor URL | Purpose | Interval | Priority | |---|---|---|---| | /health/volsync | ReplicationSource sync failures, overdue syncs | 5 min | P1 | | /health/volsync/destinations | ReplicationDestination RPO drift | 5 min | P1 |

Step 4: Heartbeat Monitoring for Restic Repository Checks

If you use the Restic engine, periodic repository integrity checks (restic check) are critical for verifying that backed-up data is actually restorable. Wire a heartbeat to your scheduled repository check job.

1. In Vigilmon, go to Monitors → New Monitor → Heartbeat
2. Name: volsync-restic-repo-check
3. Expected interval: 24 hours (daily)
4. Grace period: 4 hours
5. Copy the heartbeat URL

Add the heartbeat ping to your CronJob that runs restic check:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: volsync-restic-repo-check
spec:
  schedule: "0 3 * * *"  # Daily at 03:00
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: check
              image: restic/restic:latest
              command:
                - /bin/sh
                - -c
                - |
                  restic check && \
                  curl -fsS "$VIGILMON_HEARTBEAT_URL" || true
              env:
                - name: VIGILMON_HEARTBEAT_URL
                  valueFrom:
                    secretKeyRef:
                      name: vigilmon-secrets
                      key: restic-check-heartbeat-url
                - name: RESTIC_REPOSITORY
                  valueFrom:
                    secretKeyRef:
                      name: restic-secrets
                      key: repository
                - name: RESTIC_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: restic-secrets
                      key: password
          restartPolicy: OnFailure

Step 5: Alert Routing

| Monitor | Alert Channel | Priority | Condition | |---|---|---|---| | HTTP: /health/volsync | Slack + PagerDuty | P1 | ReplicationSource sync failure or overdue | | HTTP: /health/volsync/destinations | Slack + PagerDuty | P1 | Restore point older than RPO | | Heartbeat: Restic repo check | Email + Slack | P2 | Daily integrity check did not complete | | Heartbeat: restore test | PagerDuty | P1 | Restore test failed (backup unrestorable) |

For cross-cluster replication (Rsync-TLS engine), add a TCP port check on the VolumeSync destination endpoint (port 8448 by default) to catch network-level failures between clusters:

1. In Vigilmon, go to Monitors → New Monitor → TCP Port
2. Host: destination cluster ingress
3. Port: 8448
4. Interval: 5 minutes

Summary

VolumeSync replication failures are invisible until you actually need to restore — by which point your RPO may already be violated. External monitoring with Vigilmon gives you proactive visibility into sync health, restore point age, and repository integrity.

| Monitor Type | What It Covers | |---|---| | HTTP: /health/volsync | ReplicationSource sync failures, schedule overruns | | HTTP: /health/volsync/destinations | Restore point age vs. RPO | | TCP: port 8448 | Cross-cluster Rsync-TLS network connectivity | | Heartbeat: Restic check | Daily repository integrity validation | | Heartbeat: restore test | Backup restorability validation |

Get started free at vigilmon.online — your first VolumeSync monitor is running in under two minutes.