Alerting

Overview

OpenSearch alerting enables automatic notifications when anomalies are detected. Monitors query OpenSearch at a regular interval and fire when conditions are met.

Configured Monitors

Monitor	Detector	Threshold	Channel
Anomaly Monitor - payment-latency-detector	payment_queries	grade ≥ 0.4	Discord
Anomaly Monitor - external-service-latency-detector	external_service_calls	grade ≥ 0.4	Discord
Anomaly Monitor - database-latency-detector	database_queries	grade ≥ 0.4	Discord

CLI Commands

# Check alerting plugin status
opensearch-mgr alerting-status

# List all monitors
opensearch-mgr alerting-list

# View alert history
opensearch-mgr alerting-history
opensearch-mgr alerting-history -n 20     # Show more alerts

# Test Discord webhook
opensearch-mgr alerting-test
opensearch-mgr alerting-test -u <webhook-url>

Discord Notification Format

Alerts are sent as Discord embeds:

🚨 Anomaly Detected — payment-latency-detector
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Severity: MEDIUM (grade: 0.47)
Detected at: 2026-03-02 14:23:00 UTC

Troubleshooting

Alerts not being sent:

• Check throttle settings (default: 5 minutes between alerts per monitor)
• Test webhook: opensearch-mgr alerting-test
• Check monitor is enabled: opensearch-mgr alerting-list

Monitor not triggering:

• Ensure anomaly detection is producing results: opensearch-mgr anomaly-status
• Manually execute monitor:

  curl -X POST "localhost:9200/_plugins/_alerting/monitors/{id}/_execute"

No anomalies to alert on:

• Generate demo data: node /root/opensearch/scripts/generate-anomaly-demo-logs.js --hours 24 --ingest