Monitoring Apache ServiceMix with Vigilmon

Apache ServiceMix is an open source, enterprise-class OSGi-based integration container that combines Apache Karaf (the OSGi runtime), Apache Camel (integration framework), Apache CXF (web services), and Apache ActiveMQ Classic (JMS messaging) into a unified integration platform. Available since 2005, ServiceMix was one of the first enterprise service bus (ESB) implementations in the Java ecosystem. Every ServiceMix integration — from SOAP/REST service exposure to file-based routing to JMS message transformation — runs as an OSGi bundle inside the Karaf container. If a bundle enters a failed state, a Camel route stops unexpectedly, or the ActiveMQ broker crashes, integration flows halt silently while upstream systems continue sending messages into the void. Vigilmon gives you end-to-end monitoring across every layer of your ServiceMix deployment: the Karaf container, OSGi bundle state, Camel route throughput, CXF endpoint availability, and JMS broker health.

What You'll Set Up

• Karaf SSH management port reachability check
• OSGi bundle health monitoring (ACTIVE vs. FAILED bundles)
• Apache Camel route status and throughput monitoring
• ActiveMQ Classic broker TCP health check
• Apache CXF endpoint availability monitoring
• Camel exchange error rate and dead letter queue monitoring
• JVM heap usage and GC pause monitoring
• Karaf feature installation health check
• ActiveMQ queue consumer count monitoring
• ServiceMix log error rate monitoring

Prerequisites

• Apache ServiceMix 7.x running on a Linux or Windows server
• Karaf SSH management interface accessible on port 8101 (default)
• ActiveMQ Classic management console accessible on port 8161 (default)
• CXF endpoints exposed on port 8181 (default HTTP) or custom port
• A free Vigilmon account

Step 1: Monitor Karaf Container Health

Apache Karaf is the OSGi runtime that powers ServiceMix. The Karaf SSH management shell on port 8101 provides administration access — and its reachability is a reliable proxy for container health. If the SSH port is unreachable, the Karaf container has crashed or become unresponsive.

1. Log in to vigilmon.online and click Add Monitor.
2. Set Type to TCP Port.
3. Host: your ServiceMix server hostname or IP.
4. Port: 8101 (Karaf SSH management).
5. Check interval: 1 minute.
6. Click Save.

For a deeper Karaf responsiveness check that verifies the shell is actually processing commands (not just that the port is bound):

#!/bin/bash
# Use sshpass or pre-loaded SSH keys for non-interactive login
RESULT=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "shell:info 2>/dev/null | head -1" 2>/dev/null)

if echo "$RESULT" | grep -q "Karaf"; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_KARAF_HEARTBEAT_ID" > /dev/null
fi

1. Click Add Monitor → Cron Heartbeat.
2. Set expected interval to 2 minutes.
3. Add to cron: * * * * * /opt/scripts/karaf-health.sh.
4. A missed ping means Karaf is not responding to management commands — indicating a hung container even if the TCP port is still bound.

Step 2: Monitor OSGi Bundle Health

All ServiceMix features — Camel routes, CXF endpoints, ActiveMQ integration, custom business logic — are deployed as OSGi bundles. A bundle that fails to start or transitions from ACTIVE to INSTALLED or RESOLVED stops providing its functionality without any external error signal.

#!/bin/bash
# Query bundle state via Karaf shell
FAILED_BUNDLES=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "bundle:list -s 2>/dev/null | grep -v Active | grep -v Resolved | wc -l" 2>/dev/null)

# Also check for bundles in non-ACTIVE state that should be active
INACTIVE=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "bundle:list --state Installed,Failure 2>/dev/null | wc -l" 2>/dev/null)

if [ "${INACTIVE:-0}" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_BUNDLE_HEARTBEAT_ID" > /dev/null
fi

Alternatively, use the Karaf REST management API if the org.apache.karaf.management.rest feature is installed:

#!/bin/bash
BASE="http://localhost:8181/system/console/bundles"

# Felix Web Console (if installed) provides bundle status
BUNDLES=$(curl -s -u karaf:karaf "$BASE.json" | jq -r '.data[].state' | sort | uniq -c)
echo "Bundle states: $BUNDLES"

1. Create a Cron Heartbeat with 5-minute interval.
2. Alert fires when any bundle is in Failure or Installed (not progressed to Active) state — catching dependency resolution failures and missing packages.

Step 3: Monitor Apache Camel Route Health

Apache Camel routes are the core of ServiceMix integrations — each route defines a processing pipeline from an inbound endpoint (file, JMS, HTTP, timer) through transformations to outbound destinations. A stopped route silently drops all messages that would have flowed through it.

#!/bin/bash
# Query route status via Camel JMX through Karaf shell
STOPPED_ROUTES=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "camel:route-list 2>/dev/null | grep -i stopped | wc -l" 2>/dev/null)

if [ "${STOPPED_ROUTES:-999}" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_CAMEL_HEARTBEAT_ID" > /dev/null
fi

For throughput monitoring per route:

#!/bin/bash
ROUTE_ID="myIntegrationRoute"  # Replace with your route ID

STATS=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "camel:route-info $ROUTE_ID 2>/dev/null" 2>/dev/null)

STATUS=$(echo "$STATS" | grep -i "Status" | awk '{print $NF}')
EXCHANGES=$(echo "$STATS" | grep "ExchangesCompleted" | awk '{print $NF}')

if [ "$STATUS" = "Started" ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_ROUTE_HEARTBEAT_ID" > /dev/null
fi

1. Create a Cron Heartbeat with 3-minute interval.
2. A missed ping means at least one Camel route has stopped — messages to that route's inbound endpoint are now being discarded or accumulating in the endpoint buffer.

Step 4: Monitor ActiveMQ Classic JMS Broker

ActiveMQ Classic is bundled with ServiceMix as the JMS message broker for asynchronous messaging between integration routes. If the broker crashes, all JMS-based Camel routes lose their message source, and messages sent by upstream systems are rejected or lost.

1. Click Add Monitor → TCP Port.
2. Host: localhost (ActiveMQ runs embedded in ServiceMix).
3. Port: 61616 (default OpenWire broker transport).
4. Check interval: 1 minute.
5. Click Save.

Add an HTTP check on the ActiveMQ web console to verify the broker management layer is responsive:

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-server:8161/admin/.
3. Expected HTTP status: 200.
4. Keyword check: ActiveMQ.
5. Check interval: 2 minutes.
6. Click Save.

For broker health via the ActiveMQ REST API:

#!/bin/bash
BROKER_HEALTH=$(curl -s -u admin:admin \
  "http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost/BrokerVersion" \
  -H 'Accept: application/json')

STATUS=$(echo "$BROKER_HEALTH" | jq -r '.status // 0')

if [ "$STATUS" -eq 200 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_AMQ_HEARTBEAT_ID" > /dev/null
fi

Step 5: Monitor Apache CXF Endpoint Health

Apache CXF exposes SOAP web services and JAX-RS REST endpoints from within ServiceMix. CXF endpoints are deployed as OSGi bundles and served through the Karaf HTTP service on port 8181. An unavailable CXF endpoint means external clients calling your web services receive connection errors.

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-server:8181/cxf/YourService?wsdl (for SOAP) or http://your-server:8181/cxf/api/your-endpoint (for REST).
3. Expected HTTP status: 200.
4. Keyword check: For SOAP, enter wsdl:definitions; for REST, enter a known response key.
5. Check interval: 1 minute.
6. Click Save.

For the CXF services listing endpoint (useful for verifying all deployed services at once):

1. Click Add Monitor → HTTP / HTTPS.
2. URL: http://your-server:8181/cxf.
3. Expected HTTP status: 200.
4. Keyword check: Available SOAP services or Available RESTful services.
5. Check interval: 2 minutes.
6. Click Save.

Step 6: Monitor Camel Exchange Error Rate

Every message that flows through a Camel route is an Exchange. When processing fails, the exchange either triggers a retry, gets routed to a Dead Letter Channel (DLC), or is logged as a failure. Monitoring the failed exchange count per route tells you whether your integration is experiencing systematic processing errors.

#!/bin/bash
ROUTE_ID="myIntegrationRoute"

STATS=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "camel:route-info $ROUTE_ID 2>/dev/null" 2>/dev/null)

FAILED=$(echo "$STATS" | grep "ExchangesFailed" | awk '{print $NF}')
COMPLETED=$(echo "$STATS" | grep "ExchangesCompleted" | awk '{print $NF}')
TOTAL=$(( ${COMPLETED:-0} + ${FAILED:-0} ))

# Calculate error rate (fail if > 5%)
if [ "$TOTAL" -gt 0 ]; then
  ERR_PCT=$(( ${FAILED:-0} * 100 / TOTAL ))
  if [ "$ERR_PCT" -lt 5 ]; then
    curl -s "https://vigilmon.online/heartbeat/YOUR_ERRORRATE_HEARTBEAT_ID" > /dev/null
  fi
else
  # No exchanges yet — ping so monitor doesn't alert on startup
  curl -s "https://vigilmon.online/heartbeat/YOUR_ERRORRATE_HEARTBEAT_ID" > /dev/null
fi

Also monitor the Dead Letter Queue (DLC) depth — messages that exhausted retries accumulate there:

#!/bin/bash
DLC_DEPTH=$(curl -s -u admin:admin \
  "http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ/QueueSize" \
  -H 'Accept: application/json' | jq -r '.value // 0')

if [ "$DLC_DEPTH" -lt 10 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_DLC_HEARTBEAT_ID" > /dev/null
fi

Step 7: Monitor JVM Heap and Garbage Collection

ServiceMix runs on the JVM, and heap exhaustion crashes the entire container — Karaf, Camel, CXF, and ActiveMQ all fail simultaneously. GC pauses of more than a few hundred milliseconds degrade message processing throughput and can cause JMS consumer timeouts.

#!/bin/bash
JVM_STATS=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "jvm:info 2>/dev/null" 2>/dev/null)

# Use JMX via Karaf shell for heap stats
HEAP=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "java:info 2>/dev/null | grep -i heap" 2>/dev/null)

echo "Heap info: $HEAP"

# Alternatively, use jstat if JVM PID is accessible
SERVICEMIX_PID=$(pgrep -f servicemix | head -1)
if [ -n "$SERVICEMIX_PID" ]; then
  HEAP_USE=$(jstat -gc "$SERVICEMIX_PID" | tail -1 | awk '{print $8 + $6}')
  HEAP_CAP=$(jstat -gc "$SERVICEMIX_PID" | tail -1 | awk '{print $7 + $5}')
  HEAP_PCT=$(echo "scale=0; $HEAP_USE * 100 / $HEAP_CAP" | bc)

  if [ "$HEAP_PCT" -lt 85 ]; then
    curl -s "https://vigilmon.online/heartbeat/YOUR_JVM_HEARTBEAT_ID" > /dev/null
  fi
fi

1. Create a Cron Heartbeat with 2-minute interval.
2. Alert when heap exceeds 85% — above this threshold, GC overhead becomes significant and the risk of OutOfMemoryError and container crash increases sharply.

Step 8: Monitor Karaf Feature Installation Health

ServiceMix capabilities are installed via Karaf feature repositories. A failed feature installation leaves dependent bundles unresolved. While this typically surfaces during deployment, monitoring ongoing feature state ensures nothing silently uninstalls due to resource pressure or OSGi framework events.

#!/bin/bash
# Check for any installed features that are not in 'Started' state
UNSTARTED=$(ssh -o StrictHostKeyChecking=no \
  -o ConnectTimeout=5 \
  -p 8101 admin@localhost \
  "feature:list --installed 2>/dev/null | grep -v Started | grep -v Name | wc -l" 2>/dev/null)

if [ "${UNSTARTED:-999}" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_FEATURE_HEARTBEAT_ID" > /dev/null
fi

1. Create a Cron Heartbeat with 10-minute interval.
2. Feature state changes slowly in a healthy system; a 10-minute interval is sufficient while avoiding unnecessary polling.
3. A missed ping means at least one installed feature is not in Started state — its bundles are not active and whatever capability it provides is unavailable.

Step 9: Monitor ActiveMQ Queue Consumer Count

JMS queues accumulate messages when their consumers stop processing. A consumer count of zero means all messages sent to that queue sit unprocessed — they won't be lost (assuming persistence is enabled), but your integration workflow has effectively stalled.

#!/bin/bash
QUEUE_NAME="YourIntegrationQueue"  # Replace with your queue name

CONSUMER_COUNT=$(curl -s -u admin:admin \
  "http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=$QUEUE_NAME/ConsumerCount" \
  -H 'Accept: application/json' | jq -r '.value // 0')

QUEUE_DEPTH=$(curl -s -u admin:admin \
  "http://localhost:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost,destinationType=Queue,destinationName=$QUEUE_NAME/QueueSize" \
  -H 'Accept: application/json' | jq -r '.value // 0')

# Alert if queue has messages but no consumers
if [ "$CONSUMER_COUNT" -gt 0 ] || [ "$QUEUE_DEPTH" -eq 0 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_CONSUMER_HEARTBEAT_ID" > /dev/null
fi

1. Create a Cron Heartbeat with 3-minute interval.
2. Alert fires when consumer count drops to zero while messages are queued — the Camel route consuming this queue has likely stopped, and the queue will fill up.
3. Repeat this monitor for each critical integration queue in your ServiceMix deployment.

Step 10: Monitor ServiceMix Log Error Rate

ServiceMix uses Pax Logging (the OSGi logging framework) and writes to data/log/servicemix.log. A spike in ERROR log entries is frequently the first observable signal of a system problem — bundle resolution failures, Camel processing exceptions, CXF SOAP faults, or ActiveMQ connection losses all appear in the log before they surface as user-visible outages.

#!/bin/bash
LOG_DIR="/opt/apache-servicemix-7.0.1/data/log"
LOG_FILE="$LOG_DIR/servicemix.log"

# Count ERROR entries in the last 5 minutes
RECENT_ERRORS=$(awk \
  -v cutoff="$(date -d '5 minutes ago' '+%Y-%m-%d %H:%M')" \
  '$0 >= cutoff' "$LOG_FILE" 2>/dev/null \
  | grep -c ' ERROR ' || true)

# Allow up to 5 ERROR entries per 5 minutes before alerting
if [ "${RECENT_ERRORS:-0}" -le 5 ]; then
  curl -s "https://vigilmon.online/heartbeat/YOUR_LOG_HEARTBEAT_ID" > /dev/null
fi

1. Create a Cron Heartbeat with 5-minute interval.
2. Schedule: */5 * * * * /opt/scripts/servicemix-log-check.sh.
3. Tune the threshold (5 errors per 5 minutes) based on your baseline error rate — an integration heavy with retries may have a higher baseline than one with no expected errors.
4. A missed ping means your error rate has exceeded the threshold and warrants log investigation.

Alerting Configuration

Configure alert channels in Vigilmon to reach your team when integration health degrades:

1. Go to Alert Channels in Vigilmon.
2. Add Email as the baseline for all monitors.
3. Add Slack or PagerDuty for the highest-severity monitors: Karaf container, ActiveMQ broker TCP, and Camel route health.
4. Set 2 consecutive failures before alerting to avoid false positives from momentary check timeouts.

Recommended alert thresholds:

| Monitor | Alert Condition | |---|---| | Karaf SSH port (8101) | TCP connection refused | | Karaf shell responsiveness | No response within 10 seconds | | OSGi bundle state | Any bundle in Failure or Installed | | Camel route status | Any route in Stopped state | | ActiveMQ broker (61616) | TCP connection refused | | CXF endpoint | HTTP 5xx or connection refused | | Camel error rate | > 5% of exchanges failing | | Dead Letter Queue depth | > 10 messages | | JVM heap | > 85% for 4 minutes | | Queue consumer count | 0 consumers with queue depth > 0 | | ERROR log rate | > 5 ERROR entries in 5 minutes |

Conclusion

Apache ServiceMix integrates four powerful frameworks — Karaf, Camel, CXF, and ActiveMQ — into a single OSGi runtime, and each layer can fail independently in ways that aren't visible from the outside. A stopped Camel route looks fine from a TCP perspective; a zero-consumer queue accumulates messages silently; a bundle resolution failure disables an entire integration flow without any HTTP error. By monitoring every layer — TCP ports for the Karaf shell and ActiveMQ broker, HTTP checks for CXF endpoints and the ActiveMQ web console, and heartbeats driven by Karaf shell commands for bundle state and Camel route health — Vigilmon gives you visibility across the full ServiceMix stack. Set these monitors up before your first production incident, and you'll catch integration failures in seconds rather than discovering them when a business process has been stalled for hours.