Semantic Search

Overview

OpenSearch semantic search uses ML embeddings for natural language log queries. Text is converted to 384-dimensional vectors using sentence-transformers/all-MiniLM-L6-v2 and stored in k-NN indexes for cosine similarity search.

ML Model

Setting	Value
Model	huggingface/sentence-transformers/all-MiniLM-L6-v2
Pinned Version	1.0.1 (see compatibility note below)
Embedding Dims	384
Model Group	logrite-ml-models
Space Type	cosinesimil

Caution

PyTorch op compatibility. OpenSearch ML Commons (first seen on OS 2.18; both estates now run 3.3.1) ships a libtorch that does not recognize aten::scaled_dot_product_attention. Registering all-MiniLM-L6-v2 at the default version 1.0.2 produces model_state: DEPLOY_FAILED and breaks Semantic Search end-to-end. Always pin to version: "1.0.1" (or older) when registering this model. Full recovery procedure: Troubleshooting → Embedding model in DEPLOY_FAILED state.

Semantic Pipeline

1. Text extracted from log message
2. ML model generates 384-dim embedding vector
3. Vector stored in embedding field with k-NN index
4. Queries use cosine similarity for semantic matching
5. Results ranked by semantic relevance (not keyword frequency)

Semantic Indexes

Semantic-enabled indexes use _semantic suffix:

Semantic Index	Source Index
payment_queries_semantic	payment_queries
external_service_calls_semantic	external_service_calls
database_queries_semantic	database_queries
exceptions_semantic	exceptions

CLI Commands

# Initial setup (one-time)
opensearch-mgr semantic-setup
# Creates: model group, registers model, deploys model,
#          creates ingest pipeline, creates semantic indexes

# Check status
opensearch-mgr semantic-status

# Reindex existing data
opensearch-mgr semantic-reindex payment_queries
opensearch-mgr semantic-reindex --all

# Perform semantic search
opensearch-mgr semantic "payment timeout errors"
opensearch-mgr semantic "slow database queries" --index database_queries_semantic

# Ensure model is loaded (called by health check cron)
opensearch-mgr model-ensure

Search Query (Direct OpenSearch)

POST /payment_queries_semantic/_search
{
  "query": {
    "neural": {
      "text_embedding": {
        "query_text": "payment timeout errors",
        "model_id": "vZ_rRJsB11WWRaeQeT1C",
        "k": 10
      }
    }
  }
}

Hybrid Search (Neural + Keyword)

POST /payment_queries_semantic/_search
{
  "query": {
    "hybrid": {
      "queries": [
        {
          "neural": {
            "text_embedding": {
              "query_text": "slow payment processing",
              "model_id": "vZ_rRJsB11WWRaeQeT1C",
              "k": 10
            }
          }
        },
        {
          "match": {
            "message": "slow payment processing"
          }
        }
      ]
    }
  }
}

Ingest Pipeline

Documents are automatically embedded on ingestion via a text_embedding pipeline:

PUT _ingest/pipeline/logrite-embedding-pipeline
{
  "processors": [
    {
      "text_embedding": {
        "model_id": "vZ_rRJsB11WWRaeQeT1C",
        "field_map": {
          "message": "embedding"
        }
      }
    }
  ]
}

Model Health Monitoring

The model-health cron is retired

node /root/logrite/cli/model-health-check.js and opensearch-mgr model-ensure ran on the host destroyed on 2026-08-03; nothing schedules them now. The Semantic Search service recovers the model itself (ensureModelDeployed) and reports it on GET /health as checks.model — see Semantic Search → How the model is provisioned now. The original description is kept below.

The model-health-check.js cron runs every 30 minutes:

# Manual run
node /root/logrite/cli/model-health-check.js

If the model is unloaded (e.g., after OpenSearch restart), it:

1. Calls opensearch-mgr model-ensure
2. Waits for model to deploy
3. Sends Discord alert if model fails to load

Upload embedding pipeline (2026-10-09)

The semantic API writes a second ingest pipeline, logrite-upload-embedding, which embeds an upload event’s message into text_embedding with the deployed all-MiniLM-L6-v2 model (384-d, ignore_failure: true). New upload dataset indexes use it as their default_pipeline and map text_embedding as knn_vector, so uploads are searched by meaning. Collector indexes (log-analytics-<ws>-*) have no vector field and are searched by keyword; the API’s responses say which with mode and a KEYWORD_FALLBACK warning — see Semantic Search API.

Collector indexes: the template is ready, the clusters are not (2026-10-19). LogriteOpenSearch exports/templates/log-analytics-template.json now declares text_embedding as a 384-d knn_vector (lucene / hnsw / l2 — naming the method halves the bytes per document against the cluster default) plus embeddingFailed / embeddingError, and exports/apply-pipelines.sh creates logrite-upload-embedding. A knn_vector field cannot be added to an index that exists, so existing collector indexes only gain vectors through a reindex: scripts/semantic-backfill.sh, with the measured costs and the ML breaker it trips (the JVM heap one, at 85%) in docs/SEMANTIC-BACKFILL.md in that repo. As of 2026-10-19 neither the template nor the pipeline script has been applied to dev or prod, so the keyword fallback above still describes every collector index.