tutorial

Monitoring CloudQuery with Vigilmon

CloudQuery syncs cloud asset inventory from AWS, Azure, GCP, and 100+ sources into your database — here's how to monitor sync job success, data freshness, resource count drift, API rate limiting, and destination health with Vigilmon.

CloudQuery is an open source high-performance data integration framework for cloud infrastructure. It extracts data from cloud providers (AWS, Azure, GCP, Kubernetes, GitHub, Datadog, and 100+ sources) and loads it into destination databases (PostgreSQL, BigQuery, Snowflake, Elasticsearch) for security analysis, compliance auditing, and cost optimization. Security and platform engineering teams depend on CloudQuery for cloud asset inventory, security posture management, and compliance reporting. When CloudQuery's sync jobs fail silently, your asset inventory goes stale — compliance queries return outdated data, security misconfigurations go undetected, and cost analysis reflects yesterday's infrastructure. Vigilmon gives you end-to-end monitoring of the CloudQuery data pipeline.

What You'll Set Up

  • Sync job completion rate monitor via cron heartbeat
  • Data freshness alert per resource type
  • Resource count drift detection
  • API rate limiting event tracking
  • Destination database write latency monitor
  • Destination storage utilization alert
  • CloudQuery process health check
  • Compliance query performance monitor
  • Alert channels with appropriate thresholds

Prerequisites

  • CloudQuery CLI (cq) installed and configured with a YAML spec file
  • PostgreSQL or another supported destination database
  • Scheduled CloudQuery syncs via cron or systemd timer
  • A free Vigilmon account

Step 1: Monitor Sync Job Completion

The most critical CloudQuery health signal is whether sync jobs are completing. A sync that fails halfway through — due to API rate limiting, network interruption, or a plugin crash — leaves your inventory partially updated. Use a Vigilmon heartbeat to detect sync failures:

#!/bin/bash
CQ_CONFIG="/etc/cloudquery/config.yml"
CQ_LOG="/var/log/cloudquery/sync-$(date +%Y%m%d-%H%M).log"

# Run CloudQuery sync
cq sync "$CQ_CONFIG" 2>&1 | tee "$CQ_LOG"
SYNC_EXIT=${PIPESTATUS[0]}

if [ "$SYNC_EXIT" -ne 0 ]; then
    echo "CloudQuery sync failed with exit code $SYNC_EXIT" >&2
    # Log error summary
    grep -i "error\|fatal\|failed" "$CQ_LOG" | tail -20 >> /var/log/cloudquery/errors.log
    exit 1
fi

# Check that the log doesn't contain silent failures
if grep -q "error\|failed to sync" "$CQ_LOG"; then
    echo "CloudQuery sync completed with errors — see $CQ_LOG" >&2
    # Don't exit here — partial success may still be valuable
    # But don't send the heartbeat either
    exit 1
fi

# Sync completed cleanly — send heartbeat
curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_SYNC_HEARTBEAT" > /dev/null
echo "CloudQuery sync completed successfully"

0 */6 * * * /usr/local/bin/cq-sync.sh

Create the heartbeat in Vigilmon with a 7-hour grace period for 6-hour sync schedules. Adjust for your sync frequency — the grace period should be your sync interval plus a reasonable completion buffer (30–60 minutes for large AWS accounts).

Step 2: Monitor Data Freshness Per Resource Type

CloudQuery syncs many resource types — but some may fail silently while others succeed. A global sync completion heartbeat won't catch partial failures where aws_ec2_instances synced but aws_iam_users did not. Monitor freshness per critical resource type using database timestamp queries:

#!/bin/bash
DB_HOST="localhost"
DB_PORT="5432"
DB_NAME="cloudquery"
DB_USER="vigilmon"
DB_PASS="strong-password-here"

# Maximum age in hours before alerting
MAX_AGE_HOURS=25

# Critical resource types to check
RESOURCE_TABLES=(
    "aws_ec2_instances"
    "aws_s3_buckets"
    "aws_iam_users"
    "aws_iam_roles"
    "aws_rds_instances"
    "aws_lambda_functions"
)

PSQL="psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -t -c"
export PGPASSWORD="$DB_PASS"

ALL_FRESH=true

for TABLE in "${RESOURCE_TABLES[@]}"; do
    # CloudQuery writes _cq_sync_time to each table
    LATEST=$(${PSQL} "SELECT MAX(_cq_sync_time) FROM $TABLE" 2>/dev/null | tr -d ' ')

    if [ -z "$LATEST" ] || [ "$LATEST" = "NULL" ]; then
        echo "No sync data found in $TABLE" >&2
        ALL_FRESH=false
        continue
    fi

    AGE_HOURS=$(${PSQL} "SELECT EXTRACT(EPOCH FROM (NOW() - '$LATEST'::timestamptz))/3600" \
      2>/dev/null | tr -d ' ' | cut -d. -f1)

    if [ "$AGE_HOURS" -gt "$MAX_AGE_HOURS" ] 2>/dev/null; then
        echo "Stale data in $TABLE: last synced ${AGE_HOURS}h ago" >&2
        ALL_FRESH=false
    fi
done

if [ "$ALL_FRESH" = "true" ]; then
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_FRESHNESS_HEARTBEAT" > /dev/null
fi

0 */3 * * * /usr/local/bin/check-cq-freshness.sh

Create the heartbeat with a 4-hour grace period. This catches individual resource type sync failures that the global sync heartbeat would miss.

Step 3: Detect Resource Count Drift

A sudden large drop in the number of resources in a CloudQuery table signals a sync failure or, worse, accidental deletion of cloud resources. Track resource counts over time and alert on unexpected drops:

#!/bin/bash
DB_NAME="cloudquery"
DB_USER="vigilmon"
DB_PASS="strong-password-here"
COUNT_FILE="/var/lib/cloudquery/resource-counts.json"

export PGPASSWORD="$DB_PASS"

RESOURCE_TABLES=(
    "aws_ec2_instances"
    "aws_s3_buckets"
    "aws_iam_users"
    "aws_rds_instances"
)

# Read previous counts
PREV_COUNTS="{}"
if [ -f "$COUNT_FILE" ]; then
    PREV_COUNTS=$(cat "$COUNT_FILE")
fi

ALL_OK=true
CURRENT_COUNTS="{}"

for TABLE in "${RESOURCE_TABLES[@]}"; do
    CURRENT=$(psql -h localhost -U "$DB_USER" -d "$DB_NAME" -t \
      -c "SELECT COUNT(*) FROM $TABLE" 2>/dev/null | tr -d ' ')

    if [ -z "$CURRENT" ]; then
        CURRENT=0
    fi

    PREV=$(echo "$PREV_COUNTS" | python3 -c \
      "import json,sys; d=json.load(sys.stdin); print(d.get('$TABLE', $CURRENT))" 2>/dev/null)

    # Alert if count drops more than 20%
    THRESHOLD=$(echo "$PREV * 0.80" | bc | cut -d. -f1)

    if [ "$CURRENT" -lt "$THRESHOLD" ] 2>/dev/null && [ "$PREV" -gt "10" ]; then
        echo "Resource count drop in $TABLE: was $PREV, now $CURRENT (threshold: $THRESHOLD)" >&2
        ALL_OK=false
    fi

    # Update current counts JSON
    CURRENT_COUNTS=$(echo "$CURRENT_COUNTS" | python3 -c \
      "import json,sys; d=json.load(sys.stdin); d['$TABLE']=$CURRENT; print(json.dumps(d))" 2>/dev/null)
done

# Save current counts for next run
echo "$CURRENT_COUNTS" > "$COUNT_FILE"

if [ "$ALL_OK" = "true" ]; then
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_DRIFT_HEARTBEAT" > /dev/null
fi

30 */6 * * * /usr/local/bin/check-cq-drift.sh

Run this 30 minutes after each sync completes. Create the heartbeat with a 7-hour grace period. A 20% drop in resource count is a significant signal — it indicates either a sync failure (partial write) or an actual infrastructure change at unusual scale.

Step 4: Track API Rate Limiting Events

Cloud provider APIs impose rate limits — AWS, Azure, and GCP all throttle API requests. When CloudQuery hits rate limits, it retries with backoff, extending sync duration, or fails individual resource types entirely. Monitor rate limiting frequency to catch syncs that are at risk of timeout:

#!/bin/bash
CQ_LOG_DIR="/var/log/cloudquery"
LATEST_LOG=$(ls -t "$CQ_LOG_DIR"/sync-*.log 2>/dev/null | head -1)

if [ -z "$LATEST_LOG" ]; then
    exit 0
fi

# Count rate limit events in the latest sync log
RATE_LIMIT_COUNT=$(grep -ic "rate limit\|throttl\|429\|TooManyRequests\|RequestLimitExceeded" \
  "$LATEST_LOG" 2>/dev/null || echo 0)

echo "Rate limit events in latest sync: $RATE_LIMIT_COUNT"

# Alert if rate limits are excessive (tune threshold for your account size)
MAX_RATE_LIMITS=50

if [ "$RATE_LIMIT_COUNT" -lt "$MAX_RATE_LIMITS" ]; then
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_RATELIMIT_HEARTBEAT" > /dev/null
else
    echo "Excessive rate limiting ($RATE_LIMIT_COUNT events) — sync may be incomplete" >&2
fi

Run this after each sync completes. Create the heartbeat with a 7-hour grace period. High rate limiting events predict future sync failures — address them by adding sync concurrency limits in your CloudQuery config:

# config.yml
tables:
  - "aws_ec2_*"
  - "aws_s3_*"
scheduler:
  concurrency: 500  # Reduce if hitting rate limits

Step 5: Monitor Destination Database Write Latency

CloudQuery writes extracted data in bulk to the destination database. High write latency indicates destination saturation, storage I/O contention, or a growing dataset that needs index optimization. Monitor PostgreSQL write latency:

#!/bin/bash
DB_HOST="localhost"
DB_PORT="5432"
DB_USER="vigilmon"
DB_PASS="strong-password-here"
DB_NAME="cloudquery"

export PGPASSWORD="$DB_PASS"

# Measure time for a representative bulk query
START=$(date +%s%3N)
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" \
  -c "SELECT COUNT(*) FROM aws_ec2_instances WHERE _cq_sync_time > NOW() - INTERVAL '1 hour'" \
  > /dev/null 2>&1
END=$(date +%s%3N)

LATENCY_MS=$((END - START))
echo "Query latency: ${LATENCY_MS}ms"

# Alert if query latency exceeds 30 seconds (30000ms)
MAX_LATENCY_MS=30000

if [ "$LATENCY_MS" -lt "$MAX_LATENCY_MS" ]; then
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_DB_LATENCY_HEARTBEAT" > /dev/null
else
    echo "Database query latency ${LATENCY_MS}ms exceeds threshold ${MAX_LATENCY_MS}ms" >&2
fi

*/15 * * * * /usr/local/bin/check-cq-db-latency.sh

Create the heartbeat with a 20-minute grace period. Growing write latency typically precedes sync failures by hours — catching it early gives you time to add indexes or optimize the destination before syncs start timing out.

Key indexes for CloudQuery PostgreSQL performance:

CREATE INDEX IF NOT EXISTS idx_ec2_sync_time ON aws_ec2_instances (_cq_sync_time);
CREATE INDEX IF NOT EXISTS idx_s3_sync_time ON aws_s3_buckets (_cq_sync_time);
CREATE INDEX IF NOT EXISTS idx_iam_users_sync_time ON aws_iam_users (_cq_sync_time);

Step 6: Monitor Destination Storage Utilization

CloudQuery data accumulates over time. As your asset inventory grows and historical data builds up, PostgreSQL disk usage increases. Alert before storage fills up — a full disk causes sync failures and can corrupt the destination database.

#!/bin/bash
DB_HOST="localhost"
DB_PORT="5432"
DB_USER="vigilmon"
DB_PASS="strong-password-here"
DB_NAME="cloudquery"

export PGPASSWORD="$DB_PASS"

# Get database size in MB
DB_SIZE_MB=$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -t \
  -c "SELECT pg_database_size('$DB_NAME') / 1048576" 2>/dev/null | tr -d ' ')

# Get available disk space on the PostgreSQL data directory
DISK_AVAIL_MB=$(df -BM "$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -t \
  -c "SHOW data_directory" 2>/dev/null | tr -d ' ')" 2>/dev/null | \
  tail -1 | awk '{print $4}' | tr -d 'M')

DISK_TOTAL_MB=$(df -BM "$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -t \
  -c "SHOW data_directory" 2>/dev/null | tr -d ' ')" 2>/dev/null | \
  tail -1 | awk '{print $2}' | tr -d 'M')

USAGE_PCT=$(echo "scale=0; (1 - $DISK_AVAIL_MB/$DISK_TOTAL_MB) * 100" | bc 2>/dev/null | cut -d. -f1)

echo "CloudQuery DB size: ${DB_SIZE_MB}MB | Disk usage: ${USAGE_PCT}%"

# Alert if disk usage exceeds 80%
if [ "$USAGE_PCT" -lt 80 ] 2>/dev/null; then
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_STORAGE_HEARTBEAT" > /dev/null
else
    echo "Disk usage at ${USAGE_PCT}% — approaching limit" >&2
fi

0 * * * * /usr/local/bin/check-cq-storage.sh

Create the heartbeat with a 2-hour grace period. At 80% disk usage, you have meaningful time to add storage, implement retention policies, or archive historical data before hitting 100%.

Step 7: Monitor CloudQuery Process Health

For long-running CloudQuery sync processes (e.g., large AWS accounts that take hours to sync), monitor the process itself to catch crashes:

#!/bin/bash
# Check if a CloudQuery sync process is currently running
CQ_PID=$(pgrep -f "cq sync" 2>/dev/null | head -1)

# Check when the last sync completed successfully
LAST_HEARTBEAT_FILE="/var/lib/cloudquery/last-heartbeat"

if [ -n "$CQ_PID" ]; then
    # Sync is currently in progress — that's expected
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_PROCESS_HEARTBEAT" > /dev/null
elif [ -f "$LAST_HEARTBEAT_FILE" ]; then
    # Not running — check when it last ran
    LAST_RUN=$(cat "$LAST_HEARTBEAT_FILE")
    AGE_HOURS=$(( ($(date +%s) - LAST_RUN) / 3600 ))

    if [ "$AGE_HOURS" -lt 7 ]; then
        # Ran recently — expected to be idle between syncs
        curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_PROCESS_HEARTBEAT" > /dev/null
    fi
    # If age > 7h and not running, don't send heartbeat — alert fires
fi

Update the heartbeat file in your sync completion script:

# At end of successful sync:
date +%s > /var/lib/cloudquery/last-heartbeat

Create the heartbeat with a 8-hour grace period for 6-hour sync schedules.

Step 8: Monitor Compliance Query Performance

Security and compliance teams run SQL queries against CloudQuery data for CIS, SOC2, and PCI compliance checks. Slow queries indicate missing indexes, growing data volumes, or insufficient PostgreSQL resources. Alert when key compliance queries exceed acceptable latency:

#!/bin/bash
DB_HOST="localhost"
DB_PORT="5432"
DB_USER="vigilmon"
DB_PASS="strong-password-here"
DB_NAME="cloudquery"

export PGPASSWORD="$DB_PASS"

# CIS AWS Benchmark 1.1 - Avoid root account usage
QUERY="SELECT COUNT(*) FROM aws_iam_credential_reports
       WHERE root_access_key_1_active = true
       OR root_access_key_2_active = true"

START=$(date +%s%3N)
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" \
  -t -c "$QUERY" > /dev/null 2>&1
END=$(date +%s%3N)
LATENCY=$((END - START))

echo "Compliance query latency: ${LATENCY}ms"

# Alert if compliance query takes more than 60 seconds
MAX_MS=60000

if [ "$LATENCY" -lt "$MAX_MS" ]; then
    curl -fsS "https://vigilmon.online/api/v1/heartbeat/YOUR_COMPLIANCE_HEARTBEAT" > /dev/null
else
    echo "Compliance query latency ${LATENCY}ms exceeds 60s threshold" >&2
fi

0 */4 * * * /usr/local/bin/check-cq-compliance.sh

Create the heartbeat with a 5-hour grace period. Add indexes on columns used in frequent compliance queries:

CREATE INDEX IF NOT EXISTS idx_iam_creds_root
  ON aws_iam_credential_reports (root_access_key_1_active, root_access_key_2_active);

CREATE INDEX IF NOT EXISTS idx_s3_public
  ON aws_s3_bucket_grants (grantee_type, permission);

Step 9: Configure Alert Channels

Set up alert routing in Vigilmon for the CloudQuery pipeline:

  1. Go to Alert Channels and add email, Slack, or PagerDuty.
  2. Apply thresholds:

| Monitor | Alert After | Severity | |---|---|---| | Sync completion heartbeat | 1 missed beat | Critical | | Data freshness per resource type | 1 missed beat | High | | Resource count drift | 1 missed beat | High | | API rate limiting | 1 missed beat | Warning | | DB write latency | 1 missed beat | Warning | | Storage utilization | 1 missed beat | High | | Process health | 1 missed beat | High | | Compliance query performance | 1 missed beat | Warning |

  1. Route sync completion and data freshness failures to your security channel or on-call — stale asset inventory directly impacts security posture visibility.

  2. Route storage and compliance query performance failures to a lower-priority queue — they're important but not immediate incidents.

Step 10: Create a CloudQuery Pipeline Status Page

Give security and platform teams a real-time view of data pipeline health:

  1. In Vigilmon, go to Status Pages → New Status Page.
  2. Group monitors by function:
    • Sync Health: Sync completion, process health
    • Data Quality: Freshness per resource type, resource count drift
    • Infrastructure: Database write latency, storage utilization
    • API Health: Rate limiting events
    • Compliance: Compliance query performance
  3. Set a custom domain (e.g., cq-status.yourdomain.com).
  4. Share with security and platform teams.

Why Monitoring CloudQuery Matters

CloudQuery is your source of truth for cloud infrastructure state. When the pipeline degrades, the consequences compound quietly:

Stale inventory creates false compliance passes: A compliance query run against 3-day-old data may show "no public S3 buckets" when new public buckets were created yesterday. Without freshness monitoring, you don't know your compliance reports are stale.

Resource count drift can signal real incidents: A 40% drop in aws_ec2_instances count might mean a sync failure — or it might mean someone ran terraform destroy on a production environment. A drift alert catches both, and distinguishes them by checking whether a sync ran successfully around the same time.

Rate limiting silently corrupts partial syncs: CloudQuery logs rate limiting events but continues; the sync may "succeed" (exit code 0) while leaving certain resource types with missing data. Log-based rate limit monitoring catches this before it produces a false-clean compliance report.

Vigilmon gives you a complete health signal for every layer of the CloudQuery pipeline — from the sync job itself to the compliance queries your team runs against the output.

Ready to add observability to your CloudQuery pipeline? Create a free Vigilmon account and have your first sync heartbeat running in minutes.

Monitor your app with Vigilmon

Free plan — 5 monitors, no credit card required. Up and running in 60 seconds.

Start free →
← Back to all articles