AI Service

Overview

The Logrite AI Service is the central intelligence layer. It provides a conversational AI interface backed by NeuralWatt (one pinned model, deepseek-v4.1-flash, since 2026-10-07 — see Model & Fallback; Bring-Your-Own-LLM covers pointing a workspace at its own provider) with multi-turn tool calling, SSE streaming, and SQLite-based conversation history.

The AI assistant is dual-purpose: it specializes in log analysis, troubleshooting, and compliance scanning, but also serves as a general-purpose software engineering assistant. It answers general knowledge questions (programming concepts, architecture patterns, debugging strategies, error explanations) directly without tool calls, and only invokes tools when the user asks about their specific data, logs, or system.

Domain Presets

The service supports configurable domain presets that scope the AI’s knowledge and tool selection for specific platforms:

Domain	Key	Description
Universal	universal	Default — general-purpose across all indexes
Worksoft CTM	worksoft	Worksoft Certify & CTM test automation — prioritizes CTM indexes, understands Certify terminology

Domain presets are fetched via GET /api/v1/domains and selected by the user in the chat input bar. The selected domain injects platform-specific knowledge into the system prompt (terminology, field mappings, tool guidance).

Static presets are defined in src/config/domainPresets.ts (Universal, Worksoft CTM). Custom platforms can be created by users via the Platforms page (/platforms) in the UI or the POST /api/v1/domains/platforms API. Both static and custom domains appear in the domain selector dropdown.

Platform CRUD API

Method	Path	Description
GET	/api/v1/domains	List all domains (static + custom)
GET	/api/v1/domains/platforms	List user-created platforms only
POST	/api/v1/domains/platforms	Create new platform
GET	/api/v1/domains/platforms/:key	Get platform details
PUT	/api/v1/domains/platforms/:key	Update platform
DELETE	/api/v1/domains/platforms/:key	Delete platform

Each platform defines: name, description, associated indexes, custom domain knowledge, and platform-specific follow-up suggestions.

Setting	Value
Technology	Node.js + TypeScript + Express
LLM Provider	Configurable — AI_PROVIDER (default neuralwatt), LLM_MODEL (default deepseek-v4.1-flash). There is no cross-model fallback list; see Model & Fallback
Database	SQLite (local, data/ dir)
Port	9300 (main only)
Repo	Deployd-io/logrite-ai-service
Runs as	The ai-service compose service on each VM (warden and ingest-shim are built from the same repo)
Production	http://aiservice.logrite.ai

Model & Fallback (2026-10-07)

Cortex answers with one pinned model: PRIMARY_MODEL in src/config/env.ts reads LLM_MODEL (legacy alias GROQ_MODEL) and defaults to deepseek-v4.1-flash, pinned by id rather than by the provider’s moving alias. LLM_CHEAP_MODEL and LLM_STRONG_MODEL default to the same id, and image transcription uses it too.

• Lanes, not models. Every call has a class in CALL_POLICY (src/config/values.models.ts) that fixes its lane, reasoning effort and temperature. Interactive classes (chat, route, analysisSync, codeFix, …) run on the standard lane; background classes (judgeAsync, analysisAsync) run on the discounted flex lane (service_tier:"flex"). Chat answers run at temperature 0.
• Same-model fallback only. ResilientProvider retries a transient failure (rate limit, 5xx, timeout) on the same model on the flex lane; it never switches to a different model. When that lane answers, message_end.metadata.tier is "flex".
• Both lanes down. Paths that hold measured facts return those facts followed by The AI step is unavailable right now; these are the measured facts. (AI_UNAVAILABLE_NOTE, src/core/consistency/aiUnavailable.ts) instead of a fluent answer from another model.

The former GROQ_FALLBACK_MODEL / GROQ_FALLBACK_MODELS / LLM_FALLBACK_MODELS keys are no longer read. Full model, lane and per-call policy reference: AI models.

Bring-Your-Own-LLM (2026-07-19; catalogue and routing corrected 2026-08)

A workspace can point Cortex at its own LLM provider instead of the platform model. The provider layer (src/core/providers/) speaks every provider over axios with no vendor SDKs: OpenAICompatibleProvider covers OpenAI, Azure OpenAI, NeuralWatt, Groq, Mistral, Google Gemini and any other OpenAI-wire endpoint, and AnthropicProvider covers Anthropic’s /v1/messages shape. Both normalise back to the platform’s OpenAI-shaped ChatMessage / ToolCall model, so the AgentLoop is unchanged.

The platform default is NeuralWatt deepseek-v4-flash, not Groq. It moved in the 2026-08 migration. byo.platform.provider / byo.platform.model are exactly what the Settings banner prints as “Logrite (built-in)”, so a stale value there tells every workspace that the wrong model is answering its chats.

The provider catalogue

ProviderCatalog (backend) is the single source of the Settings → AI Model dropdown. Eight providers, and the two that need more than a key say so through needsBaseUrl / needsApiVersion rather than through a hard-coded check in the form:

Provider id	Label	Default base URL	Extra field
neuralwatt	NeuralWatt	https://api.neuralwatt.com/v1	—
openai	OpenAI	https://api.openai.com/v1	—
anthropic	Anthropic	https://api.anthropic.com	—
groq	Groq	https://api.groq.com/openai/v1	—
mistral	Mistral	https://api.mistral.ai/v1	—
google	Google Gemini	https://generativelanguage.googleapis.com/v1beta/openai	—
azure_openai	Azure OpenAI	per credential	Base URL and API version
openai_compatible	OpenAI-compatible (self-hosted)	per credential	Base URL

Two of these look like special cases and are not. Google Gemini is reached through its OpenAI-compatible surface — Authorization: Bearer, /models, /chat/completions — so it validates on the ordinary ping branch and reuses OpenAICompatibleProvider with no native adapter. NeuralWatt serves /v1/…, not Groq’s /openai/v1/…; the path shape is the one thing that differs between two otherwise identical OpenAI-wire hosts.

openai_compatible accepts any key shape on purpose. DeepSeek, Together, OpenRouter, Fireworks and most self-hosted gateways all mint sk-… keys, so the browser’s detectProvider deliberately does not map a bare sk- to OpenAI — only the distinctive sk-proj- / sk-ant- / gsk_ / AIza sub-prefixes are confident enough to auto-select. Forcing “openai” on a bare sk- is what used to send an OpenAI-compatible key to api.openai.com and get a 401 back.

Which routes honour the key

Until 2026-08 runWithProviderBundle had exactly one call site — chat. Every other LLM route read AIProviderFactory outside any bundle, which resolves to the platform default by design, so a workspace that had supplied its own key was billed platform spend and had the provider it chose ignored. Five routes now resolve the bundle first and read the factory inside it: chat, code fixes, both agent-builder routes, and the conversation hand-off. Nothing can fail a request that used to succeed — resolveProviderBundle returns null for a missing workspace, missing config, mode=platform or any transport error, and null means the platform default, unchanged.

Two spend paths remain invisible to metering: /agent-builder/parse and the hand-off call generateCompletion, which returns a bare string with usage already discarded. They are routed to the customer’s key; they are not yet measured.

• Per-request resolution. providerResolver.resolveProviderBundle(workspaceId) fetches the workspace’s active credential from the backend (GET /internal/ai-model/resolve, X-Warden-Internal-Token-guarded, LRU-cached ~20s, fail-safe → platform on any error). The decrypted key lives only in this service’s memory, for the length of the request.
• No ChatEngine rewrite. The route wraps the request in an AsyncLocalStorage bundle (providerContext.ts); AIProviderFactory.getProvider/getCheap/getStrong consult it first and fall back to the memoized platform default when absent. A BYO request routes all tiers to the user’s single chosen model, so their key never touches ours and ours never answers for them.
• Storage is backend-owned — the byo_llm_credential table and /api/v1/ai-model/*, mirroring the AI Warden credential vault (AES-256-GCM, per-workspace AAD). Configured in the Settings → AI Model tab; the active model is shown in a status banner.
• One key, stored once. A credential is fingerprinted sha256(workspace:provider:key) and a repeat is refused with 409 naming the existing entry, rather than silently creating a second row that the activation rules would then have to arbitrate.
• Exactly one credential is active per workspace. Activating one deactivates the rest; “Use Logrite’s model” deactivates all of them, which is what switches mode back to platform. Deleting the active credential falls back to the platform model — there is no state in which a workspace has no model.
• Plaintext never comes back. Every browser-facing response is scrubbed to ••last4.

Agent Builder + manual run (2026-07-19)

The AI-Agents Phase 1 execution surface. src/tools/governance.ts classifies every tool read | write | external and enforces read-only least-privilege server-side for Phase 1.

Method	Path	Purpose
GET	/api/v1/agent-builder/tools	Categorized tool catalog (powers the builder UI)
POST	/api/v1/agent-builder/parse	NL description → reviewable agent config (name, systemPrompt, toolsAllowed, schedule); tools constrained to the read catalog with a deterministic fallback
POST	/api/v1/agent-builder/run	Load an agent from the backend (caller bearer), run AgentLoop with toolNames = resolveAllowedTools(agent.toolsAllowed) + tenant toolContext, capture the tool-call log, and POST the execution record back to /api/v1/ai-agents/{id}/executions

See AI Agents Overview for the full system.

Conversational front-door — greetings & small-talk (2026-07-29)

A modern assistant answers “hey”, “how are you”, “who are you”, and “thanks” like Claude/ChatGPT/Gemini would — not by routing them into the log-query machinery. Cortex used to have no first-class handling for these: greetings fell through to the general LLM path (untested, sometimes an awkward deflection to “ask me about your logs”) and “who are you” was only steered by the system prompt (Groq → non-deterministic).

src/utils/conversational.ts::classifyConversational(message) classifies a prompt into greeting | smalltalk | thanks | farewell | assistant_identity, or null for anything that is a real task/knowledge query. It is deliberately narrow: every branch first bails on a task/tech noun (log, error, scan, project, latency, …) and — except identity — requires a short message, so “hey can you list my errors” and “how are you handling timeouts” are never caught (83 unit tests in tests/unit/conversational.test.ts pin the no-false-positive contract).

ChatEngine.handleConversational answers each class:

• thanks / farewell → short, gracious reply, no tool calls, no deflection.
• assistant_identity (“who are you” / “what can you do”) → a consistent, branded capability card (deterministic — reads the same every time) + follow-up chips.
• greeting / smalltalk → warm AND grounded: a light best-effort, JWT-scoped peek at the caller’s own profile + projects + latest scan, phrased by the LLM but constrained strictly to those facts (so it can’t hallucinate) — e.g. “Hey Rahil 👋 You’ve got 3 projects and your most recent scan is #182 on JobSearch. Want the summary, or something else?” — with deterministic follow-up chips derived from the real data. Falls back to a warm template if the LLM is down.

Wiring (processChatStreamingV2): the branch runs right after the JWT-identity answer, gated on !indexName && !canonicalProjectName (an explicitly scoped chat is treated as task-focused, so the scoped handlers — including the project capability menu — still own it) and regardless of CORTEX_LLM_FIRST (a greeting is never a data query). Tenant-safe: identity comes from the verified JWT via _context_*, never the message. Note isIdentityQuestion (“who am I” → JWT account facts) is distinct from assistant_identity (“who are you” → capability card).

LLM-first mode — CORTEX_LLM_FIRST (2026-07-14)

By default Cortex answers most data questions from deterministic templates (byte-identical per user, P2) and only falls through to the LLM for generic/open-ended prompts. That guarantees consistency but has a failure mode: a question the template layer misclassifies gets a confident wrong answer from the wrong template — e.g. “Why was a PR not generated for cSharp?” was mislabeled activity by the LLM route-backstop and answered with an activity breakdown (and leaked another project’s rows), never calling list_project_generator_results.

CORTEX_LLM_FIRST=true flips Cortex into a ChatGPT/Gemini-style engine: after the safety guardrail and the JWT identity answer, every prompt is routed to the LLM agent loop (gpt-oss-120b + the full DB toolset + grounding verifier + tenant scope) instead of the deterministic templates. The model understands any phrasing/typos, picks the right tool, and grounds its answer in that tool’s data.

How it’s wired (ChatEngine.processChatStreamingV2): a single llmFirst flag (from config.cortexLlmFirst). Message-based short-circuits (list_projects, scan-history/id/latest, capability, the project-has-data pre-flight guard) are guarded with !llmFirst; intent-based branches (user_activity/error/performance/saved/static_analysis/activity/analytical) are all neutralized by forcing the classified intent → generic right after classifyIntent, and the LLM route-backstop is skipped (it was the piece that re-labeled generic prompts into a data template). isConceptual is preserved, so a pure how-to question still answers tool-free. Safety nets are unchanged: guardrail, JWT-scoped backend tools, project-scope/serviceName injection, and the grounding verifier all still run. The system prompt gained a generator/PR playbook (why was a PR/generation not created → list_project_generator_results) plus explicit typo-tolerance + “ground in the ACTIVE project only” rules.

Env var	Default	Effect
CORTEX_LLM_FIRST	false	true → LLM agent loop answers every prompt (post-guardrail/identity); deterministic templates are bypassed

Tradeoff: LLM answers are not byte-identical across users/runs (Groq is non-deterministic) — that is the ChatGPT-style behavior this mode intentionally chooses over the P2 determinism guarantee. Rollout: default OFF, so deploying the code changes nothing until the env var is set. Recommended path — enable on the dev/staging ai-service container, verify the six failing prompts (scan summary/critical-high/compliance/report, “why no PR”, activity) now answer correctly and project-scoped, then set it in prod. Fully reversible: unset the var + redeploy. Contract test: tests/unit/llmFirstRouting.test.ts.

Turn trace (/api/v1/trajectory)

Every Cortex turn records what it DID — the routing branch that claimed it, each model call with its TTFT and tokens, each tool call with its arguments and result size — and serves it back. Full detail, including the record model and the OTel field mapping: Turn Trace & Trajectory.

Answer-quality fixes found BY the trace (2026-09-17)

The trace was built first and used as the instrument. Asked “Which service or page is the slowest?” one real turn spent 14 tool calls, 10 model calls, 647,817 tokens and 38 seconds and answered “I cannot identify which service or page is the slowest.” Reading its records found four causes, none of them visible from the answer text:

• The truncation cap destroyed the answer. The agent had already run the right query — trace-analytics-raw-* sorted by durationInNanos descending, whose first row names the slowest service — and truncateToolResult replaced the whole 31,928-byte reply with a 2,000-character preview and “Use more specific filters.” Two causes met there: the cap was a constant sized for a 128K model while the derived input budget is 128,000 tokens, and { success, hits: [...] } — the shape this service’s own search tools return — matched none of the three list handlers and fell through to the branch that DISCARDS rather than trims. The cap is now derived from contextInputTokenBudget, and a list keeps its leading rows in the order the query asked for.
• v2Performance needs a project, correctly — it aggregates one project’s query_exec_ms. A workspace-wide question resolves none, so a sibling path now ranks every service by p95 over the trace family. It requires the project to be ABSENT and a numeric workspace, so it runs only where the first gate already declined and cannot reach the workspace-wide compliance aggregate a previous gate removal turned into 1,196,262 findings.
• The intent is not performance. Probed against the built classifier: slowest queries classifies as performance, which service or page is the slowest as analytical. A narrow predicate (a latency superlative and a noun naming the population) admits the second without moving the classifier, whose changes this programme measures separately.
• An empty window is not an answer. The workspace’s traces are older than “today”, so the ranking widens to every span on record — and says so, in the first sentence and again under the table. A number labelled “today” computed over July is the failure the whole programme exists to stop.

Result on the same question: 0 tool calls, 0 model calls, 1.7 s, naming the slowest service with its p95, its slowest operation, and the window actually covered.

Fixed in the same pass, all found in the live turn store rather than the corpus: list_agents / list_roles / list_templates had never once succeeded (215 calls, every one 401 missing bearer token) because they hand-rolled headers instead of calling backendAuthHeaders; the loop’s duplicate-call short circuit reported 58 of those 401s as successes; 331 merged tables named a row count they did not hold; and The facts provided contain … passed through the mechanics relabel untouched — both mechanics-leak failures in the published failureSplit.

Two defects a screenshot run measured — a ceiling and a rename (2026-09-24)

Ten error screenshots — a Java NPE, a Python KeyError, an exhausted Postgres pool, an nginx 502, an OOMKilled pod, an expired TLS certificate, an expired JWT, a full disk, an unreachable Cloud Agent, and a healthy control — were uploaded through POST /api/v1/attachments and each asked, through POST /api/v1/chatv2, for “(1) a possible root cause analysis, (2) concrete steps to fix it, (3) a short summary, and (4) a conclusion”. Content pickup was 10/10 and every answer was specific to its own image; the control correctly answered “There is no failure to explain. The system appears healthy” and reframed the fix steps as verification. Two defects showed up in the answers anyway, and neither was visible from a single turn.

A rewrite that did not fit was shipped half-written

reAddressAttachment fires when the user asked about their file and the answer came back about telemetry alone. It replaces the whole answer with a fresh single-shot completion — and it was capped at a flat 900 tokens, which is under 3,600 characters, and returned a bare string. So a rewrite the ceiling amputated was indistinguishable from one the model finished, and it shipped, because it was the newer text.

image	took the rewrite?	chars	ending
03-postgres-pool	yes	3,792	…This is a connection
05-k8s-oomkilled	yes	3,589	mid-word
03 re-run (b)	yes	3,639	mid-sentence
01-java-npe	yes	3,003	complete — it fit
03 re-run (a)	no	5,557	complete
every other answer	no	up to 6,126	complete

That is the whole distribution and it is 1:1: every rewrite that did not fit was cut, and no answer that skipped the rewrite was cut at any length. 03 lost the conclusion it had been asked for outright. It looked intermittent because it depends on how long the answer wants to be.

The existing guard could not catch it. “A rewrite shorter than the draft has dropped findings” is the wrong test for this: a truncated rewrite is normally longer than its draft, having grown until the ceiling stopped it.

A ceiling is not a length target

The first fix here derived the ceiling from the draft. That moved the cliff rather than removing it — a rewrite told to keep every figure and add a discussion of the file can legitimately more than double a short draft, and refusing it means falling back to a draft that never mentions the file, losing the one thing the rewrite exists to add. The ceiling is now MAX_PROSE_TOKENS (4,096), which is what all three providers already fall back to when a caller names none. Nothing is spent unless it is generated; the prompt asks for one answer and temperature is 0.

core/providers/completeProse also reports finish_reason === 'length', which every provider already returned and nothing read (AnthropicProvider maps max_tokens onto it). The re-address keeps its complete draft when the rewrite was cut — a shorter whole answer beats a longer broken one — and the grounding repair in core/verify/repair, which carried the identical flat cap, gets the same headroom. There a fallback would be unsafe: the draft is the text that asserted ungrounded figures, which is why the repair is running at all.

A brand rename walked into a path

humanizeOutput renames the log store to its product name — the user-facing name is Analytics, and “OpenSearch” is an implementation detail that must not reach prose. Its safety contract already said it never rewrites code blocks. That was true of which blocks it visits and false of what it did inside one: a text block’s markdown carries fences, inline spans, and the paths and commands a reader is meant to copy.

The disk-full screenshot shows a df row for /var/lib/opensearch. The stored answer said, verbatim:

**`/dev/mapper/vg0-data 920G 920G 0 100% /var/lib/Analytics`** — This `df` row is the
smoking gun: the volume mounted at `/var/lib/Analytics` (where Analytics stores its
data) is **100% full** — 920G used, 0 available.

Three times over, with zero occurrences of “opensearch” anywhere in the response. The rename is right in the prose around it and wrong inside the path, which now names a directory that does not exist — an operator following that step cds into nothing.

utils/codeSpans masks fenced and inline regions with an inert placeholder for the English rules only, and restores them verbatim. The ordering is the point: the tool-name and index-name removals still see the whole text, because a leak in backticks is still a leak, while the brand nouns, the mechanics vocabulary and the whitespace tidy run under the mask. utils/infraLexicon additionally skips a match glued into an unfenced path, filename or environment variable (/var/lib/opensearch, opensearch.yml, OPENSEARCH_JAVA_OPTS) — and only a separator followed by a word character counts as glue, because the commonest trailing separator is the full stop that ends a sentence, and “I searched OpenSearch.” is prose.

One second-order artefact came with it: a model that knows both names writes “your OpenSearch (Analytics) node”, and renaming the first leaves a stutter. The redundant gloss is collapsed.

And one the regression test found

Writing the fenced-block case exposed a third, unrelated defect of long standing. The cleanup that removes an empty inline-code span — the artefact left behind when a leaked tool_call() token is stripped out of backticks — was written /`\s*`/g. That ate fences. Three backticks contain a pair, so every fence marker in every answer lost two of its three backticks and the block stopped being a block; and \s* spans newlines, so it could also weld a backtick on one line to a backtick on the next and delete everything between. It is now exactly two backticks, nothing but blanks between them, and no backtick either side.

Why the metering label moved with the fix

Reading finish_reason means asking chatWithTools for the response envelope where the call used to ask generateCompletion for the content alone. The request is unchanged — every provider builds generateCompletion as exactly that call, with one user message and no tools — but the provider-method name is also the fallback metering label, so the swap would have moved auxiliary spend onto the agent’s line. AICompletionOptions.meterAs names the label explicitly and is never sent to a provider.

/health names the build

Until 23 September 2026 /health reported "version": "1.0.0" from a string literal in the route handler. It had never moved in the service’s history, so the one endpoint whose job is to say what is running answered byte-identically whether the container held the commit from five minutes ago or the one from August. Confirming a fix had shipped meant sshing to the host and reading a docker image tag — the exact thing a health endpoint exists to save you from.

{
  "status": "healthy",
  "service": "logrite-ai-service",
  "version": "1.1.0",
  "sourceId": "b219e40420a35d25183d967f6ec4177865797af4e8cb8eb11959f78b7b9bfd7b"
}

Two identities, because they answer different questions:

• version — what we intend. Read from package.json, moved when the contract moves, and the thing to quote in a bug report. swagger.config.ts held a second copy of the old literal and now reads the same number, so the API docs cannot disagree with the service about its own version.
• sourceId — what is actually there. The content hash of src/ that scripts/source-id.js computes at build time and the Dockerfile copies to /app/SOURCE_ID. Nobody maintains it, so it cannot drift: two containers agreeing on it are running the same code, and a version somebody forgot to bump cannot hide that they do not.

Either field is absent rather than "unknown" when it cannot be read — two builds both reporting "unknown" would look identical, which is the defect this replaces. The lookup never throws; /health is what an uptime monitor calls.

API Endpoints

Method	Path	Description
GET	/health	Health check — and which build is answering (see below)
GET	/api-docs	Swagger UI
POST	/api/v1/chat	Chat with SSE streaming — session JWT only (an API key gets 401)
POST	/api/v1/chat/sync	Chat, one JSON answer — API key or JWT (Chat API)
POST	/api/v1/chatv2	Chat with SSE streaming and project/index context — API key or JWT (Chat API); what Cortex in the app calls
GET	/api/v1/conversations	List conversations
GET	/api/v1/conversations/:id	Get conversation with messages
POST	/api/v1/conversations	Create new conversation
DELETE	/api/v1/conversations/:id	Delete conversation
GET	/api/v1/domains	List available domain presets
POST	/api/v1/chatv2/compose	Composed turn — runs the chat answer and the log-analysis report, fuses them into ONE answer, streams it once (see below)
POST	/api/v1/attachments	Upload one file for a chat turn; classified from its BYTES, not its extension
GET	/api/v1/attachments/:id	Attachment metadata
GET	/api/v1/attachments/:id/content	The bytes, always as an opaque download (Content-Disposition: attachment, nosniff, restrictive CSP)
DELETE	/api/v1/attachments/:id	Delete one the caller owns
GET	/api/v1/attachments/config/limits	The server’s real limits, so a client never guesses them
POST	/api/v1/attachments/uploads/sessions	Open a chunked upload for a file too big for one request — same protocol as the log-analysis door (details)
PUT	/api/v1/attachments/uploads/sessions/:id/parts/:n	One part. Idempotent; X-Part-Sha256 is verified on arrival
GET	/api/v1/attachments/uploads/sessions/:id	Which parts have landed — the resume call
POST	/api/v1/attachments/uploads/sessions/:id/complete	Assemble, then answer the identical 201 {attachment} the single-request POST returns
DELETE	/api/v1/attachments/uploads/sessions/:id	Abandon the upload and free the disk
GET	/api/v1/deployments/signal	Legacy (Coolify). Cheap signal: { active, count, activeServices } — one Coolify call, server-cached 5s. Used by frontend idle polling.
GET	/api/v1/deployments	Legacy (Coolify). List all Logrite services with Coolify deployment status + progress (expensive — ~26 upstream calls)
GET	/api/v1/deployments/:serviceKey	Legacy (Coolify). Service detail: status, commit, active deployment, recent history
GET	/api/v1/deployments/:serviceKey/history?limit=N	Legacy (Coolify). Paginated deployment history (max limit=50)

Chat API

Cortex’s chat is callable directly. Authenticate with an API key (lgp_live_…, created in Sandbox → API keys) or a signed-in session JWT, sent as Authorization: Bearer …. The key acts as the person who created it, in its own workspace. Both routes share the request body:

Field	Required	Notes
message	yes	1–10,000 characters. Missing → 400 INVALID_MESSAGE; blank → 400 EMPTY_MESSAGE; longer → 400 MESSAGE_TOO_LONG
conversation_id	no	Continue a conversation — the id a previous answer returned. Must be a string (400 INVALID_CONVERSATION_ID)
agent_mode	no	auto (default) or single (400 INVALID_AGENT_MODE otherwise)
indexName	no	Scope the question to one log index, e.g. log-analytics-3103-app-4f9e4a (chatv2)
projectName, projectFullName, levels	no	Project and method/class context for code questions (chatv2)
attachment_ids	no	Ids from POST /api/v1/attachments (chatv2)
llm_config	no	{temperature, max_tokens, top_p} (chat, chat/sync). max_tokens must be a whole number from 1 to 4,096, else 400 INVALID_LLM_CONFIG; a non-object llm_config is the same 400. The engine also clamps it to that range. No first-party client sends it

Time limits on a turn (2026-10-23)

Every Cortex turn on /chat, /chat/sync and /chatv2 has hard stops on its model calls (src/core/agentLoop/turnDeadline.ts). Each stop cancels the HTTP call to the provider. A streamed answer needs this: the client’s own 120 s timeout stops counting once headers arrive, so before 2026-10-23 a stream that stalled mid-answer waited on the socket indefinitely.

Stop	Default	Variable
Turn deadline, from the start of the turn	75 s	AGENT_TURN_DEADLINE_MS (0 = off)
One interactive model call	45 s	AGENT_CALL_DEADLINE_MS (0 = off)
The client closes the connection before the answer is sent	—	—

When one fires, the turn answers from what its searches already measured, with the lead “The AI step took too long, so it was stopped; these are the measured facts.” message_end.metadata carries answered: false and unanswered_reason deadline (or client_closed), and the answer is not cached. The 45 s soft budget (AGENT_TURN_MS_BUDGET) is unchanged: it decides when to stop searching and answer, and its clock starts after the first completion. The deadline is the hard stop behind it, and stays under Cloudflare’s ~100 s origin limit. Background calls (flex lane, analyses) are not affected. REQUEST_BUDGET_MS was read by nothing and is removed.

POST /api/v1/chat/sync — one JSON answer

curl -s -X POST https://aiservice.logrite.ai/api/v1/chat/sync \
  -H "Authorization: Bearer $LOGRITE_API_KEY" -H "Content-Type: application/json" \
  -d '{"message": "How do I create an API key?"}'

{
  "status": "success",
  "data": {
    "conversation_id": "92080845-d9b7-4b51-b9ee-da7125a668b4",
    "message_id": "32bcfe57-d4d4-4902-94f5-93563173b3b7",
    "content": "To create an API key in Logrite, follow these steps: …",
    "tool_calls": [],
    "metadata": {"tokens_used": 26706, "tools_executed": 0, "model": "gemma-4-31b",
                 "agent_mode": "auto", "query_type": "explanation", "temperature": 0.3,
                 "cache_hit": false, "response_shape": "data"},
    "results": []
  }
}

tool_calls lists the tools Cortex ran for the answer and results the rows they returned (both empty for a product question like this one). Send conversation_id back to continue the thread. Measured on dev, 2026-10-01; this is also what the MCP cortex_chat tool calls.

Workspace questions work over an API key (feat/sandbox-2, 2026-10-01). Asked “which projects are in my workspace?” over a key, Cortex used to answer “I couldn’t identify your workspace from this session”. The workspace did arrive — introspection supplies it — but a key carries no email, and list_projects built /api/v1/project/{ws}/{email}. It now uses /api/v1/project/workspace/{ws} whenever there is a bearer but no verified email: the backend identifies the owner from the forwarded credential (JWT or key) and returns the same owner-scoped list (48 projects for workspace 3103 on dev). A JWT caller with an email keeps the route it always used, and a model-supplied email is never used on an enforced scope. The project ownership gate (assertProjectOwned) used to allow every key caller unverified; it now checks against the same bearer-identified list.

POST /api/v1/chatv2 — streamed (Server-Sent Events)

Same body, plus the context fields above. The answer arrives as an SSE stream — use curl -N so it is not buffered:

curl -s -N -X POST https://aiservice.logrite.ai/api/v1/chatv2 \
  -H "Authorization: Bearer $LOGRITE_API_KEY" -H "Content-Type: application/json" \
  -d '{"message": "How do I create an API key?"}'

event: message_start
data: {"conversation_id":"8f652362-…","message_id":"b1f15aca-…"}

event: content_delta
data: {"delta":"To create an API key in Logrite, follow these steps:\n\n1. Navigate to the **Sandb"}

event: content_delta
data: {"delta":"ox** section of the platform. …"}

event: content_block
data: {"type":"follow_up_suggestions","id":"b63f…","suggestions":["How do I rotate an API key?", …]}

event: message_end
data: {"conversation_id":"8f652362-…","message_id":"b1f15aca-…","content":"<the whole answer>","title":"How do I create an API key?","metadata":{…}}

Event	Carries
message_start	conversation_id, message_id
content_delta	delta — the next piece of the answer text; concatenate them
tool_use_start / tool_use_result	A tool Cortex is running, and what it returned
content_block	A structured block — follow-up suggestions, tables, charts
message_end	The complete content, a title for the conversation, and metadata
error	The turn failed; the stream ends

message_end.content is the whole answer, so a client that does not render incrementally can ignore the deltas. POST /api/v1/chat streams the same events but accepts only a session JWT.

Deployments API (Coolify proxy) — legacy

Legacy — Coolify no longer deploys anything

These routes read deployment state from Coolify. Coolify was removed from dev on 2026-09-23 and no longer owns any deployable service on prod, so they report nothing useful: measured 2026-10-01, GET /api/v1/deployments/signal answers {"active":false,"count":0,"activeServices":[]} on both estates whatever is deploying, and GET /api/v1/deployments lists Coolify-era services with status: "unknown". Deployment state now lives in the management console (https://management.logrite.ai, GET /api/deployments) — see Deployment platform. The description below is kept for the frontend code that still calls these routes.

The /api/v1/deployments routes proxy Coolify’s REST API (https://cicd.logrite.ai) so the frontend can trigger its Deployment Lock Overlay without exposing the Coolify API token to the browser.

The frontend uses a two-tier strategy: polls /signal at 15s while idle, switches to /deployments at 2s only while a deploy is active. /signal is cached for 5s on the server to shield Coolify from client pile-on.

Config:

COOLIFY_BASE_URL=https://cicd.logrite.ai
COOLIFY_API_TOKEN=<read-only token from Coolify → Keys & Tokens → API Tokens>

The Coolify UUID → service-key map lives in src/config/coolify-apps.ts. Keep this in sync if apps are recreated in Coolify (UUIDs change on recreate).

Progress estimation is heuristic (Coolify does not expose a % field). src/services/coolify.service.ts#estimateProgress inspects the deployment’s status + most-recent log lines and maps to discrete stages (queued 5% → preparing 20% → cloning 30% → building 55% → pushing 75% → starting 85% → health checks 95% → finished 100%). Falls back to maxBatch / 6 if no stage keywords match.

Composed turns — ONE answer, not two (2026-09-23)

A “composed” turn is one where Cortex answers the question and runs the deterministic log-analysis report. Until 2026-09-23 those were two independent answers and the browser concatenated them. No model ever saw both, so the report’s generic prose led and the user’s actual question was answered somewhere underneath it — which is what “Cortex just shows me the API response” meant.

POST /api/v1/chatv2/compose adds the reduce step that was missing. Both halves are produced, then one model is given the question and both witnesses and writes a single answer. The report is not discarded: it is demoted to a collapsed log_analysis block with a tip saying where it went.

It is grounded by construction, not by hope.

Property	How
Deterministic	temperature: 0 on one pinned model, so the same prompt + scope + window still answers identically (the P2 invariant)
No invented figures	The prompt restricts every number, identifier and service name to what appears in one of the two witnesses; groundingVerifier then checks the output
Self-repairing	An ungrounded figure gets exactly one repair attempt
Fails backwards, never forwards	On any failure the turn falls back to the previous concatenation — it never ships an unverified number

The six fallback reasons are named and each is tested: provider-error, route-error, still-ungrounded, a missing question (QUESTION_REQUIRED), an empty provider response, and fusion disabled. Behind CORTEX_FUSION_ENABLED; CORTEX_FUSION_MAX_TOKENS bounds the synthesis.

The UI buffers, then streams. Both halves complete, the fusion runs, and only then does the single answer stream. An answer is never rewritten under a reader who is already partway through it.

Chat attachments

Files attached to a chat turn are classified from their bytes, not their extension.

Kind	Accepted
text / code	any byte-wise-text file, whatever its extension — .log, .sh, .yaml, .conf, …
csv	CSV / TSV
docx / sheet / slides	Word, Excel and PowerPoint (OOXML), read with a dependency-free ZIP + XML extractor
image	sent to the vision model when one is configured; when none is, the file says so rather than appearing empty
pdf	text layer where present — see the caveat below

Legacy .doc / .xls / .ppt (OLE2) are recognised and refused with advice: re-save as the modern format, or as .csv / .pdf. That is deliberate — an honest refusal beats a silent empty extraction.

Installed since 2026-09-24 — and installing it was not the fix

pdf-parse is a dependency of this service now. It is still loaded lazily and the service still starts without it, because PDF is the one type whose parser is allowed to be absent.

⚠ Adding the dependency on its own changed nothing visible, which is the worst shape a bug can have — the fix is applied, the symptom persists, and the library gets blamed. v1 and v2 are not the same library: v1 exports a callable, v2.4 exports a PDFParse class and nothing callable at all. The old loader probed only for default and pdf, resolved undefined against the current release, and went on reporting “PDF text extraction is not enabled on this deployment” over a parser that was installed. The loader now accepts either shape, and destroys the v2 parser after every read — it holds a pdf.js worker open, and one leaked per upload is a slow way to run a long-lived service out of handles.

Limits, and what happens past them

Read them from GET /api/v1/attachments/config/limits — never hard-code them.

Limit	Default
Files per message	10
Bytes per file	100 MB
Bytes per message	250 MB

Different formats may be mixed freely in one message; a single turn carrying .log, .sh, .csv, .docx, .xlsx, .pptx and .png is a supported and tested case.

Both caps are enforced twice, on purpose. The composer checks before a byte moves, so the user learns which file is the problem; the server enforces it because a client check is advice, not enforcement. Past the cap resolveForTurn keeps what fits, stops at the first file that does not, and names what it left behind — and the context block then instructs the model to say so plainly and forbids it writing “all the attached files” over a set it did not receive. A silently truncated set answered as though it were whole is the failure this exists to prevent.

For a log too big for a chat turn, the answer is not a bigger cap: use the log-analysis upload endpoint, which takes 256 MB a file.

SSE Event Stream

Chat responses are streamed via Server-Sent Events:

Event Type	Description
message_start	Initial acknowledgment, assigns message ID
content_delta	Token-by-token text streaming
tool_use_start	Tool execution beginning (includes tool name)
tool_use_result	Tool result returned from downstream service
content_block	Structured JSON block for rich UI rendering
message_end	Completion with token usage metadata
error	Error event with message
debug	Opt-in diagnostic payload (only when ?debug=1 or X-Debug: 1)

Debug mode

Pass ?debug=1 on the chatv2 URL or an X-Debug: 1 request header to opt the request into richer logging. Effects:

• The server emits event: debug SSE lines interleaved with the normal stream (request entry, errors with stack head). The Logrite-UI parses these via parseSSEStream and renders them as a [logrite:cortex] console group.
• auth.middleware.ts echoes the validated JWT subject and loginType as X-Debug-Auth on the response — visible in DevTools Network → Headers.
• ToolExecutor logs every tool invocation ([tool] start ... / [tool] done success= ms= bytes= / [tool] fail ...) at info/warn/error level. These reach the Logrite-UI as part of the debug SSE payload when present.
• validateJwt always logs auth success (with sub + loginType) and failure (with code + reason). Debug toggle is not required for these — they fire on every request.

The Logrite-UI persists the debug flag in localStorage.logriteDebug via window.__logriteDebugToggle(true) so it survives reloads. No request payload bodies, JWT values, or user-supplied query text are ever included in debug output.

Request logging — two lines, and the one that used to be missing (2026-09-25)

logging.middleware.ts emits one line per request:

> POST /api/v1/log-analytics/MigratedWork id=0f3c…
+ POST /api/v1/log-analytics/MigratedWork 200 1204ms id=0f3c…
! POST /api/v1/log-analytics/MigratedWork ABORTED 25301ms id=0f3c… — client disconnected before any reply was sent

+ answered, ~ a 3xx, ! a 4xx/5xx or an abort. Both lines carry the same id, so a caller can quote X-Request-Id and be found.

A conclusion was published from a MISSING log line, and it was wrong

The completion line was on res.on('finish') alone. finish fires only when a response was fully written, so a request whose client walked away left the arrival line and then silence — no completion, no error, nothing.

On 2026-09-25 that silence was read, in a commit message, as proof a request had never arrived. It had: a two-file Postman upload was accepted at 13:18:19 (files=2 … mode=sync) and dropped ~25 s later by its client, which could not wait out a serial vision pass. Absence of evidence was published as evidence of absence, and only a positive line can stop that recurring.

res.on('close') now emits the ABORTED line, and headersSent separates “we had begun answering and the peer went away” from “nothing was ever sent” — a dead stream and a caller that stopped waiting are different failures.

`close` fires after every normal `finish` too

Without the answered guard, every healthy request would grow a second, contradictory line. The test asserts a 200 produces zero ABORTED lines and exactly one +.

supertest cannot witness this

It drives the app over a real socket but owns both ends politely — there is no API on it for “send the request, then destroy the connection before the handler answers”, which is precisely the event under test. tests/unit/requestLogging.test.ts therefore opens a raw socket, hand-writes the request, and destroys it while the handler is deliberately still thinking.

Available Tools

semantic_search

Search logs using natural language via the Semantic Search API.

{
  query: string;    // Natural language query
  index?: string;   // Index to search (auto-detect if omitted)
  k?: number;       // Results to return (default: 10)
}

opensearch_query

Execute direct OpenSearch queries for aggregations and advanced filtering.

{
  query_type: 'search' | 'aggregation' | 'count';
  index: string;
  query: object;
  time_range?: { from: string; to: string };
}

get_scan_results

Retrieve compliance scan results from the Backend API.

{
  scan_id: number;
  user_id: number;
}

Deterministic scan answers (intent-differentiated)

Scan questions are answered by a deterministic branch (handleScanSummary → renderScanAnswer in ChatEngine.ts), not the LLM, so the same prompt is byte-identical for every user (P2). Two fixes landed 2026-07-14:

• Contradiction killed. The answer previously printed the scan DTO severity totals in the header (e.g. 0 critical / 5 high / 5 medium / 5 low) and a report block whose counts came from the keyFindings detail arrays — which can be empty even when totals exist — so it rendered Risk: CLEAN · 0/0/0/0 · no findings directly under a header showing 15 findings. Now the DTO totals are the single source of truth for every count/grade; the detail arrays are used only to name individual findings. When totals exist but the per-finding detail wasn’t captured, the answer says so plainly instead of a false “CLEAN”.
• One prompt → one shape. classifyScanAnswerIntent(message) (in utils/scanQuery.ts, robust to case/punctuation/typos) routes each phrasing to a distinct render: summary (default), report (all severities + next steps), critical-high (only those two), compliance (posture-framed + audit next steps). Both the id path (extractScanId) and the “my latest scan” path (wantsLatestScan) pass the classified intent. wantsScanHistory + handleMyScansList route “show my scan history / list my scans” to a scans table rather than the latest-scan block. Model tag: deterministic-scan-<intent>.

list_projects render: the backend repository field is a nested object ({repositoryId, repoURL, repoName, branches:[…]}); responseBlocks.ts now parses it and shows only the repo name + branch count + a plain date instead of dumping the raw JSON blob into the table cell.

Deterministic project overview (“<Project> details / summary”) (2026-07-29)

A prompt that names a project and asks about the project itself — e.g. “JobSearch project details and its summary”, “tell me about JobSearch”, or just JobSearch — is answered by a deterministic branch (handleProjectOverview in ChatEngine.ts, model tag deterministic-project-overview), not the LLM. It previously fell through to the agent loop as a creative query with a log-oriented project context, so the model searched the project’s logs, found nothing, and replied “I don’t have any details” over a real project.

The branch resolves the target project from the pinned UI context or by matching a name in the prompt against the caller’s real list_projects, then assembles ONE grounded overview from JWT-scoped sources (each best-effort; a missing source is omitted, never a hard error):

• Metadata — id, description, language/stack, branch, repository (from list_projects).
• Latest scan — newest scan for that project with C/H/M/L totals (from list_my_scans).
• Log generation — recorded generator runs (from list_project_generator_results).
• Recent activity — 7-day error/exception total + top error locations (from runErrorSummary; needs clientId).

Routing: isProjectOverviewQuestion / isBareProjectNameLike (utils/listQueries.ts) gate the branch; a DATA_NOUN (errors/logs/scan/performance/…) in the prompt is excluded so log/data questions still flow to the grounded/agent path. The branch is placed after the scan branches (so “summary of scan 42” still wins) and runs regardless of CORTEX_LLM_FIRST (it is grounded truth, not a template guess). The Active Project Context block + a systemPrompt playbook also steer the LLM to list_projects for project-metadata asks on the fallback path. Tests: tests/unit/listQueries.test.ts.

analyze_logs

Trigger AI analysis or retrieve key findings from the Analyzer service.

{
  action: 'get_key_findings' | 'analyze_code';
  scan_result?: object;
  code?: object;
}

web_search

Search the web for real-time information about technologies, platforms, companies, error codes, and documentation. Uses the Brave Search API.

{
  query: string;    // Search query
  count?: number;   // Results to return (default: 5, max: 10)
}

read_url (2026-09-25)

Read ONE page whose address the model already has, and answer from its text. web_search finds pages; this one reads the page it was given, which is the case that arrives whenever a user pastes a link.

{
  url: string;   // the full http(s) URL, exactly as the user gave it
}

Implementation: src/tools/readUrl/ — addresses.ts (IP classification), guard.ts (URL policy + the DNS pin), fetchPage.ts (the fetch and its own redirect handling), extract.ts (HTML→text) and outcomes.ts (the payload contract).

What it refuses, and why an allowlist

The guard is an allowlist over the address space, applied to the resolved IP — not a denylist of hostnames. A denylist is defeated by every alternative spelling of the same address, and there are many: localhost, 127.1, 0x7f000001, 2130706433, [::1], [::ffff:127.0.0.1], 64:ff9b::7f00:1, and any DNS name whose A record simply points there. All of them converge on an IP, which is the only place the check can work.

What is at stake concretely on this estate: 127.0.0.1:9200 is our OpenSearch, holding every tenant’s logs, and 169.254.169.254 is the cloud metadata service, which hands instance credentials to anything that can make an HTTP request from the VM. A URL reader is, by construction, a way to make requests from the VM.

Refused	Code	Reason given to the user
Anything but http/https	SCHEME_NOT_ALLOWED	file:, data:, gopher: are not the web
A URL carrying user:pass@	CREDENTIALS_IN_URL	refused, not stripped — stripping changes what was asked for, and the secret would land in our logs and the model’s context
Loopback, RFC 1918, CGNAT, link-local, multicast, reserved, documentation space, and every IPv6 equivalent	PRIVATE_ADDRESS	names the address and why
A public NAME that resolves anywhere private	PRIVATE_ADDRESS	reports the name and what it resolved to
A host answering with one public and one private address	PRIVATE_ADDRESS	that pairing is the signature of a rebinding attempt, not a misconfiguration to route around
A port other than 80/443	PORT_NOT_ALLOWED	internal services live on odd ports; this layer still holds if one is ever public
An https → http redirect	DOWNGRADE	on a pasted link this is likelier an attack than a mistake
Anything that is not text/html/json/markdown/xml	NOT_TEXT	a PDF decoded as UTF-8 is thousands of characters of plausible garbage, and a model will summarise it

A redirect is a second URL, and the server chooses it

Every HTTP client will follow redirects for you, which means the guard runs on the URL the user supplied and the connection ends up at a URL the server picked. A public host answering 302 Location: http://169.254.169.254/ defeats an otherwise complete SSRF guard, and nothing about the request looks unusual in any log.

So maxRedirects: 0 at the client is load-bearing, and every hop is re-vetted here by the same function that vetted the original — capped at four, with relative locations resolved and then checked like any other. The test asserts on the config actually passed to axios, because a suite that only drives the happy path passes against the broken version.

Resolve once, and connect to the answer

Checking a NAME and then handing that name to an HTTP client leaves a DNS-rebinding window between the two: a record with a one-second TTL can answer publicly for the check and 127.0.0.1 for the connection.

vetUrl() therefore returns the address, and fetchPage() connects to that address via a custom lookup on the http/https agent. The hostname is untouched — Host header, TLS SNI and certificate validation all still use the real name, which is what makes this a pin rather than a rewrite. Replacing the host in the URL instead would break TLS on every https site and quietly disable certificate checking.

The order of two refusals is not cosmetic

Both of these were live, and both were secure while reporting the wrong thing.

http://127.0.0.1:9200/_cat/indices came back as “I can only read pages on ports 80 and 443” — true, and the least useful true thing available. It reads as a limitation to work around rather than as someone probing our data store, and the obvious retry (:80) is a different message for the same underlying answer. Likewise https://public → http://169.254.169.254/ reported the protocol downgrade instead of the destination, which is how a real attempt on the metadata endpoint gets triaged as a TLS quirk and closed.

The address check now runs first. The more serious reason has to be the one reported.

`read_url` must be listed in governance.ts by name

categorize() is explicit for known write/external tools and heuristic otherwise — its external prefixes are web_, install_ and call_. read_url matches none of them, and read_ is not a write prefix either, so without an explicit entry it files as a read-only tenant query and readOnlyToolNames() hands a tool that makes arbitrary outbound HTTP requests to every autonomous agent. Nothing errors; the permission model simply stops being true, because of how the tool is spelled.

A failure must never read as a finding

Three outcomes here look like content and are not, and each carries its own instruction to the model — the same discipline as webSearch/outcomes.ts, where “unavailable is not empty” is written down.

Payload flag	What happened	What the model is forbidden to say
refused_by_policy: true	we declined the address	anything about the page’s contents; any workaround or alternative spelling
retrieval_failed: true + 401/403	the site declined us	that the page is empty, blank or gone
likely_client_rendered: true	we fetched it; it builds its content in the browser	that it is blank or “appears to be a login screen”

403 is the common case, not an edge case

Measured live on 2026-09-25: chat.deepseek.com/share/… answers 403 to this service and 200 in a browser — Cloudflare and friends refuse a datacentre IP with no browser fingerprint. There is no server-side fix for that; what there is a fix for is the report. A model handed text: "" will tell the user the page is blank, about a page the user is looking at as they read the answer.

An empty extraction is not an empty page

A single-page app fetched server-side is 200 OK, real HTML, and perhaps forty words of boilerplate. likely_client_rendered is a ratio heuristic (markup over 1 KB, under 400 characters of prose, under 5%) and is described to the model as one. It does not need to be precise; it needs to stop “we could not see the content” being delivered as “there is no content”.

Live results, 2026-09-25:

URL	Outcome
https://example.com/	200, Example Domain, 145 chars
https://devdocs.logrite.ai/services/ai-service	200, AI Service | Logrite, 136,169 chars
https://chat.deepseek.com/share/xe6ivslws6szgqlekj	403 — the site refused; reported as such, never as empty
http://169.254.169.254/latest/meta-data/	PRIVATE_ADDRESS — “link-local space — this is where cloud instance metadata lives”
http://127.0.0.1:9200/_cat/indices	PRIVATE_ADDRESS — names loopback, not the port

Caps: 2 MB of body, 40,000 characters of extracted text, 4 redirects, 15 s overall and 8 s per hop. No cookies, no Authorization, no forwarded headers — the request carries nothing of ours or of the user’s.

How Cortex AI Works (End-to-End)

“Cortex AI” is the user-facing name for this service. This is the complete path a question takes from the browser to a rendered answer.

Request flow

{/* Editable Mermaid source: diagrams-src/cortex-request-flow.mmd — re-render via mermaid.ink and replace public/images/cortex-request-flow.svg */}

Step-by-step lifecycle

1. UI → AI Service. The Cortex chat box on app.logrite.ai POSTs to https://aiservice.logrite.ai/api/v1/chatv2 over SSE, with a Bearer JWT and project context (projectName, clientId, projectId, levels, indexName, domain, followup_context).
2. Auth + validation. validateJwt and validateChatRequest middleware gate the request, then it reaches ChatEngine.processChatStreamingV2.
3. Query typing + system prompt. detectQueryType selects a temperature profile (analytics 0.0, search 0.1, explanation 0.3, creative 0.2). Project/entity metadata asks (details/overview/about) route to explanation (2026-07-29) so “<Project> details” no longer lands in high-temperature creative. The project name is resolved/registered and a tenant-scoped system prompt is built — telling the model the project’s data lives in log-analytics-{clientId} and trace-analytics-raw-{clientId}, filtered by serviceName. A persona preamble (the UI role dropdown → persona/experienceLevel, or a role alias) is prepended for tone/depth on both V1 and V2 paths (buildPersonaPreamble, config/userPersonas.ts); it changes presentation only, never data access.
4. Query cache. An exact-normalized (and similarity) cache lookup in SQLite; on a hit the stored answer is replayed verbatim for consistency.
5. Deterministic short-circuits (LLM bypassed). Every prompt is first run through the dynamic intent layer (src/core/intent/): normalizePrompt canonicalizes it (lower-case, de-punctuate, expand contractions, light stem) and classifyIntent maps it — via synonym/prefix dictionaries, not brittle literal regexes — to one of error · performance · activity · saved · static_analysis · analytical · generic. This is why any phrasing/typo routes correctly (e.g. “todays performance”, “how is jobsearch performing”). Because Groq is non-deterministic, each data intent is then answered by a fixed, project-scoped query rendered from a template, in order: (1) project-has-data guard, (2) error sweep (summary + Fix-it panel), (3) performance — query_exec_ms p50/p95/p99 + threshold breaches (500/2000/5000 ms) + slowest ops, (4) saved-query (deterministic “not tracked” + redirect), (5) static_analysis — calls the backend /api/v1/code-analysis, lists exception-handling risks, and emits an Auto-fix & open PR Fix-it (auto:true → /codeFixes/auto), (6) activity — aggregated counts/breakdown (volume separated from errors), hour-bucketed window for cross-user consistency, and auto-widen if the serviceName filter is empty. This branch also answers business-creation questions (“how many jobs were created today / candidates registered yesterday”) with an exact count — see below. Any match returns immediately; generic falls through to the agent loop.
6. Agent loop (the general path). If no short-circuit matched, AgentLoop runs up to 10 turns: it streams the prompt + tool schemas to Groq, executes any requested tools via ToolExecutor, feeds results back, and repeats until the model returns a final answer.
7. Tools fetch the real data. Each tool hits one backend: semantic search (:9100), OpenSearch (:9200), the Spring backend (:8080), the analyzer (:9000), or the web. This is where Cortex’s answers actually come from — it does not read application databases (e.g. JobSearch’s MongoDB), only the observability/platform data exposed by these tools.
8. Rendering + persistence. toolResultToBlocks converts results into structured blocks (text, tables, code_fix_action/Fix-it, follow-up chips), which stream to the UI as SSE events (content_delta, content_block, message_end). The conversation and tool executions are persisted to SQLite.

Where each answer comes from

Tool(s)	Data source	Used for
semantic_search	Semantic Search API :9100 (OpenSearch neural)	Natural-language log search
search_otel_logs, opensearch_query, hybrid_search, search_traces, anomaly_detection, alerting, cluster	OpenSearch :9200	Logs, traces, aggregations, anomalies, alerts
projects, my_scans, my_profile, enable_monitoring, initiate_scan, templates, agents, scan_key_findings, project_generator_results, roles, create_workspace, start_github_app_install, verify_monitoring, my_generated_logs	Backend :8080	Platform/account data and actions
analyzer_api	Analyzer :9000	Code scan / key findings
web_search	Google / Brave	External knowledge

Business-creation counts (“how many X were created”)

Cortex answers record-creation questions with a real, exact number — e.g. “how many jobs were created today”, “candidates registered yesterday”, “employers created this week”. This is derived from the observability logs, not from the application database: every instrumented app already logs a method-call line on create (createJob(jobDTO)=…: save query executed in N ms), which the standard analyzer tokenizes as a single token createjob. So the count is a server-side OpenSearch _count (over the full hour-bucketed window — not the 50-row activity sample), and stays project-agnostic with no per-project code and no hardcoded entity list:

• Entity is extracted dynamically from the prompt (extractCreationEntity in intentClassifier.ts): strip creation/activity verbs, time/quantity/function words and the project name, take the first content noun, and depluralize it to a singular root (jobs/jobss→job, candidates→candidate).
• Count (runCreationCount in dataQueries.ts): _count where body matches prefix(<verb><entity-root>) for verbs create/insert/register/add/new/save (e.g. createjob, registeruser). With no entity named (“how many records were created”), it counts the safe verbs create/insert/register only — bare save/add/new would false-match save query/address/news.
• Render (in ChatEngine.ts): when the count is > 0 it leads with “Today, JobSearch had 7 jobs created.”; when it is exactly 0 it gives an honest “I don’t see any jobs created … 0 creation events were logged” plus the activity breakdown — never a fabricated number. The window is hour-floored and the answer is cached, so the same question is byte-identical for every user/tab (P2).

Requirement: creations are only countable when the app actually logs them. If an app doesn’t emit a create log line, the honest “0 created” answer is returned. (For exact DB-level counts on apps that don’t log creates, an optional per-project read-only stats connector is designed but deferred.)

Code Fix (“Fix it” button) pipeline

The Fix-it panel is a separate flow from chat — it goes through the backend, not directly to the AI Service:

{/* Editable Mermaid source: diagrams-src/cortex-codefix-pipeline.mmd */}

Core Architecture

ChatEngine

Main orchestration layer. Manages conversation lifecycle, message persistence, and delegates to AgentLoop.

AgentLoop

Implements multi-turn tool calling:

1. Send user message + conversation history to Groq
2. If LLM requests tools → execute via ToolExecutor
3. Return tool results to LLM
4. Repeat up to 10 turns
5. Stream final response via SSE

ToolExecutor

Routes tool calls to downstream services:

• semantic_search → Semantic Search API (:9100)
• opensearch_query → OpenSearch (:9200)
• get_scan_results → Backend API (:8080)
• analyze_logs → Analyzer (:9000)
• web_search → Brave Search API (external)

Response Blocks

All structured data is returned as content_block events with typed JSON:

{
  "type": "content_block",
  "block": {
    "type": "search_results",
    "data": [ ... ]
  }
}

Project name resolution

resolveProjectName() (src/tools/otelLogs.tool.ts) canonicalizes service/project names used in code_fix_action blocks and tool metadata. The resolution order is:

1. Dynamic map — names previously registered via registerProjectName(<raw projectName from frontend>). Seeded in ChatEngine.handleMessage whenever the chat request carries a projectName.
2. Separator fallback — if the input contains -, _, or ., split and title-case each segment (e.g. auth-service → AuthService).
3. Mixed-case preservation — if the input already has camelCase / PascalCase / an all-caps acronym (e.g. JobSearch, CTM, UserAuth), it is returned verbatim. A previous version lowercased these destructively, which broke downstream case-sensitive lookups against stored branchLinks in the backend (Deployd-io/JobSearch/... vs Deployd-io/Jobsearch/...).
4. Pure-lowercase — capitalize the first letter only (e.g. jobsearch → Jobsearch).

The canonical form is what ends up in code_fix_action.projectName, which the frontend “Fix it” button forwards to the backend /api/v1/codeFixes. The backend also uses a case-insensitive branchLink finder as defense-in-depth, so either layer alone would tolerate casing drift.

The code_fix_action block also carries an optional category — 'bug' (default), 'performance', or 'compliance' — set by createCodeFixActionBlock (src/utils/responseBlocks.ts), plus optional auto/maxFiles. category is forwarded to the analyzer, which specializes the fix (patch vs optimize vs redact); see Backend → Fix-it categories. When auto:true (set by the static_analysis path) the frontend calls /api/v1/codeFixes/auto instead of the single-target endpoint — running the build-verify repair loop + exception-handling hardening and opening one PR only if the project compiles. The error sweep emits category:'bug'; the static_analysis path emits an auto Fix-it.

Project-scope enforcement (server-side, since 2026-05-22)

When POST /api/v1/chatv2 is called with projectName and clientId set, the AI service enforces project scope inside every search tool — not just as a hint to the LLM. The plumbing lives in src/utils/projectScope.ts and is wired into search_otel_logs, semantic_search, hybrid_search, search_exceptions, search_traces, and opensearch_query.

Two behaviors compose on every tool call:

1. Strict index override — resolveIndexesForProject({ llmIndex, userSelectedIndex, clientId, projectName, kind }) picks the index list using this precedence:

   1. _context_indexName (UI dialog selection — explicit user intent) wins always.
   2. The LLM-supplied index argument, if every entry is in IndexRegistry and none is in STALE_LEGACY_INDEXES.
   3. Project tenant indexes (log-analytics-{clientId} for kind: 'logs', trace-analytics-raw-{clientId} for kind: 'traces') — used when LLM picked a stale or non-existent index and project context is present.
   4. IndexRegistry.getSearchableIndices() or the * wildcard as final fallback. When the override fires, the tool response includes a project_index_override: <reason> field for visibility.

2. Soft service_name injection — if the LLM did not supply service_name / service, the tool resolves the effective serviceName via resolveEffectiveServiceName(...) with precedence explicit service_name > authoritative project binding > lower(projectName), then injects it into the OpenSearch filter (matches serviceName.keyword and service field variants). If the LLM explicitly passed a different service_name, that wins (preserves cross-project queries).

   Authoritative binding (since 2026-05-29). When the chat request includes projectId (the frontend threads it from the selected project), resolveProjectBinding(projectId) fetches GET /api/v1/project/{id}/index-binding from the backend (cached 5 min) and uses its serviceName. This fixes the mismatch where lower(projectName) is wrong — e.g. project BankJava_TestRepo whose OTEL service.name is bankjava (not bankjava_testrepo). With no projectId or on a backend error it falls back to lower(projectName), so the change is non-breaking. projectId reaches every tool via toolContext → _context_projectId and is also passed directly into the deterministic error sweep. See Backend → Project → index binding.

STALE_LEGACY_INDEXES blocklist (any of these LLM-picked names triggers the override when project context is active):

exceptions, exceptions_semantic,
payment_queries, payment_queries_semantic,
database_queries, database_queries_semantic,
external_service_calls, external_service_calls_semantic

These names are development-only and don’t exist on the multi-tenant production cluster. The server silently redirects to log-analytics-{clientId}.

search_exceptions skips Phase 1 (exceptions_semantic,exceptions) entirely when a project context is active and goes directly to the tenant log index with severityText.keyword=SEVERE + the project serviceName filter — saves a round-trip and avoids 404s.

The processChatStreamingV2 system-prompt block is also strengthened to name the canonical indexes for the active project and explicitly forbid the legacy names — so the LLM does the right thing on its own and the server enforcement becomes a safety net rather than the primary mechanism.

Verification: tests/unit/projectScope.test.ts (17 cases) covers the helper’s precedence ordering; tests/unit/projectContextInjection.test.ts (7 cases) covers the tool-level injection with mocked axios. For live testing against a real OpenSearch, see scripts/verify-project-scope.js and scripts/verify-project-scope-extended.js.

Pre-flight project-existence guard

opensearch_query is a low-level tool that accepts arbitrary OpenSearch DSL. To stop permissive bodies ({ match_all: {} } + aggregations) from returning every service’s data when the LLM thinks it’s “scoped to a project,” the DSL is wrapped: when _context_projectName is set the tool ANDs a serviceName.keyword/service filter on top of the LLM’s query clause. This applies uniformly to search, aggregation, and count query types.

A second guard runs at the chatv2 entry point — checkProjectHasData(clientId, projectName) in src/utils/projectScope.ts. If the request supplied projectName + clientId, this issues a cheap _count against log-analytics-{clientId} AND trace-analytics-raw-{clientId} filtered by serviceName=projectName.toLowerCase(). If both indexes return zero (and neither errored), processChatStreamingV2 short-circuits with a deterministic “No monitoring data found for project X” message — the LLM is NOT invoked. Behavior:

• The guard runs only for queryType === 'search' or 'analytics' (per detectQueryType). Explanation / creative queries like “what is logrite?” are unaffected.
• The guard is skipped when _context_indexName (user dialog selection) is set — the user explicitly picked an index, so they own that scope.
• The guard fails open: a transport error or timeout on the count call sets checkFailed and lets the LLM run, rather than blocking legitimate users on a transient OpenSearch hiccup.
• The “no data” response lists the indexes that were checked and the per-index doc counts, and explains three possible causes (monitoring not enabled, no logs in retention window, or unexpected serviceName).

The follow-up summarization branch (isFollowUpSummarization === true) also gets a strengthened instruction: when summarizing conversation history under an active project context, the LLM must EXCLUDE rows tagged with a different serviceName. This closes a stale-history leak path where a conversation that previously surfaced jobsearch data could be summarized later under a different project context.

Tests: tests/unit/projectHasData.test.ts covers 18 cases — happy paths (log-only, trace-only, both empty), edge cases (missing clientId, missing projectName, axios error, partial failure, whitespace inputs, missing count field, string count), behavior assertions (uses _count not _search, fires both index checks in parallel via Promise.all), and the opensearch_query DSL-wrapping (search, aggregation, count, and legacy-no-context).

Tenant isolation for non-search tools

list_indexes, list_anomaly_detectors, get_anomaly_results, get_alerting_status, and get_alert_history previously ran cluster-wide queries that exposed other tenants’ resources. They now consume _context_clientId and filter via the shared indexBelongsToTenant(name, clientId) helper (canonical *-{clientId} suffix; case-insensitive; rejects hidden indexes and exact-prefix collisions like logritefoo).

• list_indexes — when _context_clientId is set with no explicit pattern, requests _cat/indices/*-{clientId} directly. When the LLM supplies a broader pattern, defense-in-depth filters the response to entries matching the tenant suffix.
• list_anomaly_detectors — drops detectors whose indices array contains zero tenant indexes (orphan detectors with indices: [] are also dropped).
• get_anomaly_results — fetches the tenant’s detector IDs in a single match_all lookup and ANDs terms: { detector_id: [...] } into the results query. Intersects with detector_name filter when both are supplied. Uses an __no_detectors_for_tenant__ sentinel so empty tenants force a 0-result response rather than falling through to match_all.
• get_alerting_status and get_alert_history share one lookup, tenantMonitors() (src/tools/alerting/tenantMonitors.ts, ai-service 9531265, 2026-10-20). A monitor is the caller’s only when every index in inputs[0].search.indices belongs to the caller, by clientId or by _context_workspaceId (detectorIndexMatchesTenant). Before, one tenant index was enough, so a monitor over two tenants’ indexes was shown, with its alerts, to both. A monitor with no indexes is never the caller’s. A call with neither id makes no request and lists nothing. Each monitor row carries watching_since: enabled_time as ISO, or null for a disabled monitor. An incident before that time could not raise the monitor’s alert, and the incidents guidance tells the model to say the monitor did not exist yet.
• get_alerting_status lists only the notification channels the caller’s monitors send to (triggers[].*.actions[].destination_id). Channels have no tenant owner in OpenSearch, so the tool used to list every config and the CareVault answer named “Worksoft demo alerts”. The ids are read with GET _plugins/_notifications/configs?config_id_list=<ids> and filtered again locally; if that fails, the legacy _plugins/_alerting/destinations/_search is used with an ids query. A referenced id the cluster does not hold is listed with missing: true and left out of summary.notification_channels. If both reads fail, the result carries notification_channels_unreadable: true so “could not read” is not reported as “none”.
• get_alert_history reads alerts through the alerting plugin’s API, one call per tenant monitor (GET _plugins/_alerting/monitors/alerts?monitorId=<id>&alertState=ALL&sortString=start_time&sortOrder=desc, at most 50 monitors; monitors_truncated: true past that). alertState=ALL includes COMPLETED (resolved) and DELETED alerts. An alert is DELETED when its monitor or trigger changed while it was open. The tool used to _search .opendistro-alerting-alerts, a system index: under the security plugin a search there answers 0 hits even as admin (measured on prod 2026-10-20: 9 alert documents, 0 hits). Cortex therefore told the CareVault demo “no alerts on record” while “PHI in CareVault agent logs” was ACTIVE. Since 9531265:
  • monitor_name is matched locally against the tenant’s own monitors: every word must appear in the name. The old second _search asked for 10 hits only.
  • With time_range the tool pages 50 alerts at a time (startIndex) until it passes the window’s start, up to 10 pages. Before, it read the newest limit alerts and then filtered by window.
  • The answer lists monitors (id, name, indices, watching_since).
  • If one monitor’s read fails, the others’ alerts are still returned and that monitor is named in unreadable_monitors. If every read fails, the call fails.
  • If the monitor lookup itself fails, the call fails. It used to report success with no alerts, which reads as “nothing fired”.

Ownership counts only each row’s own id (2026-10-01). The project and scan ownership gates used to collect every id at any depth of the backend payload, so nested templates[].id and repository.branches[].id — small integers from other tables — made unrelated project ids (project 1, for one) pass assertProjectOwned. Owned ids are now data[].id only (with projectId / project_id accepted); a row naming another workspace is skipped; an empty list denies, and a payload that is not a list fails closed. The scan fast path has the same rule — anything else falls through to the backend’s authoritative check.

Empty-result responses across these tools are returned as success: true with result_count: 0, never as an error — so the LLM can frame the “no data” reply correctly without falling back to cross-tenant exploration.

Tests: tests/unit/tenantIsolation.scoping.test.ts and tenantIsolation.enforcement.test.ts cover list_indexes, the anomaly tools and get_alert_history (tenant and workspace scoping, the local name filter, window paging, partial and total read failures, a failed lookup). tenantIsolation.alertingStatus.test.ts covers get_alerting_status: channels limited to the caller’s monitors even when the server ignores config_id_list, the legacy fallback, unreadable channels, a missing channel, a monitor spanning two tenants, and watching_since.

Conversational no-data response (modern UX)

When the pre-flight guard short-circuits, the response is structured as a chat-grade message — three sections (Why you're seeing this, What I checked, What you can do next), per-index document counts, and a follow_up_suggestions content block with clickable prompts the user can pick to recover (“How do I enable monitoring for X?”, “Which projects on tenant Y have data?”, etc.). This matches the agent UX used elsewhere in Cortex-AI (ChatGPT / Gemini-style follow-up chips).

search_exceptions was also updated: when _context_projectName is set and Phase 2 returns 0 docs, the response is success: true with total_hits: 0 and result_count: 0 — not the misleading “No exception or log indexes found” error.

Live verification: scripts/verify-robust.js runs 25 scenarios across positive (JobSearch with seeded data), non-ran (deploydRadar with no data), edge (empty/whitespace project names, cross-service LLM overrides, stale-legacy redirects, cross-tenant lookups), and negative cases (empty query, unknown index, empty-tenant anomaly/alerting). All 25 pass with the rebuilt dist/.

Pinned-index guard: WHY there is no data (2026-09-07)

A chat can be pinned to one index (indexName), which skips the project guard above and runs its own pre-flight — checkIndexHasData in src/utils/projectScope/dataPresence.ts. That call is GET /<index>/_count?ignore_unavailable=true, and the flag is the catch: an index that cannot be read answers HTTP 200 count: 0, byte-identical to a readable index that genuinely holds nothing. A closed index therefore reported as “may be empty or not created yet” while its data sat intact on disk, and the user was sent to debug a collector agent that was never involved.

The discriminator is in the same response: a readable index reports _shards.total >= 1 even when it counts zero; an unreadable one reports 0. When that happens the guard makes one _cat/indices/<index>?expand_wildcards=all call and returns a cause:

state	Meaning	What the user is told
empty	Readable, holds no documents	“The index exists but holds no documents” — check the agent
closed	Exists, data intact, searches refused	“The index is closed… Analytics → Index Management has an open control”
missing	No index under that name	“No index exists under that name”
out-of-scope	Not in the caller’s workspace	“That index is not in your workspace” — asserts nothing about whether it exists

out-of-scope is decided before the _cat probe: belongsToWorkspace(index, clientId) must hold for every name, or the probe is never issued. Confirming that a foreign index exists (or is closed) is itself a cross-tenant disclosure, so that branch says nothing either way. The copy lives in one place, buildPinnedIndexNoDataMessage, and message_end.metadata.pinned_index_state carries the cause for the trace.

Tests: tests/unit/pinnedIndexState.test.ts (9 cases) — each state, the fail-open transport path, the single-call assertion for a readable index, and the copy assertions.

search_otel_logs: the window reported, and the total (2026-09-07)

Two ways the same response used to misstate what it held:

• The window. When the requested window matches nothing, the widen ladder searches wider and declares it in auto_widened. But time_range_applied still reported the requested window in the same payload, and time_note points the model at exactly that field — so a 30-day query over an index whose newest document is older than 30 days returned all-time data under a 30-day label. time_range_applied is now emitted only when it is the window the documents actually came from; a widened search reports time_range_requested plus a time_range_applied_note instead.
• The total. hits.total.value stops counting at 10,000 without track_total_hits, so the ceiling was quoted as a count — on one real 13-index corpus, four indexes sat above it (89.8% of the data), and 43,518 logs were reported as “10,000”. executeSearch now sets track_total_hits: true on every search it issues (an explicit value in the caller’s body still wins) and warns if a gte relation is ever seen again.

Tests: tests/unit/otelWindowAndTotals.test.ts (3 cases).

An uploaded file is searched whole (2026-10-08)

HIPAA re-test S7-C1: the claims log covers 6–22 Sep. The model passed relative_time_hours=720 (the tool description calls it the preferred filter), so the search started on 8 Sep and Cortex reported 2,056 records and 4 errors for a file holding 2,240 and 6. Nothing in the answer said the oldest days were missing.

wholeUploadWindow() (src/tools/otelLogs/uploadWindow.ts) now drops the window when all three hold:

• the search targets exactly one concrete index (no wildcard, no list);
• that index is an uploaded dataset, meaning it carries the _meta.logrite provenance stamp every upload writes (datasetMeta.ts). Live collector streams do not carry it: on dev, 39 of the 40 indexes written to in the last hour are unstamped;
• the user’s own question names no window (“last week”, “today”, “6 Sep”, “2026-09-10”).

In every other case today’s behaviour is kept: an explicit window, a wildcard or multi-index scope, a live stream, a call with no user question (sweeps choose their own window), and a failed provenance read. The count fallback in counts.ts follows the same rule and then says “across the whole uploaded file” rather than “in the last 30 days”.

Tests: tests/unit/uploadWindow.test.ts (11 cases).

The backstop cannot refuse the caller’s own data as cross-tenant (2026-10-08)

HIPAA re-test S7-C3: “Show the clinical notes and procedures for member BEN-1044 in hipaa-claims-app.log” is the caller’s own upload. It was refused in 861 ms with the cross-tenant copy (“I can only access data that belongs to you”), on 7 Oct and again on 8 Oct. The deterministic rules let it through. The advisory LLM backstop read “member” as another account, and because its verdict is cached per prompt, it refused every time.

A backstop cross_tenant verdict is now honoured only when the message names a tenancy noun: workspace, tenant, organisation/org, or another/other/different user, customer, account, client, company or team (honourBackstop() in src/core/safety/guardrail.ts). “Member”, “patient”, “user” and “customer” on their own are the people inside the caller’s own logs.

Tenant isolation does not depend on this classifier. The data tools fence every query to the caller’s indexes (turnScope.ts, resolveScope.ts), so the verdict only ever chose the wording of an answer. The explicit cross-tenant phrasings are still refused by the deterministic rules before the backstop runs.

Tests: tests/unit/guardrailCrossTenantBackstop.test.ts (7 cases).

Raw log samples read the index’s own fields (2026-09-07)

runErrorSamples / runRecentLogSamples / runErrorSummary’s representative hit asked for _source: ['body', …] and sorted on time — the platform’s OTEL names — whatever the index maps. On an uploaded or collector index storing text in message and dates in @timestamp, every sample came back empty, extractOperation('') returned its (other) sentinel, and the table rendered [(other)] where the log line belongs, under a correct error count.

resolveSampleFields(index) (src/services/indexDescriptor/sampleFields.ts) now supplies the _source list, the sort clause (via resolveSortField, so the sort is valid for that index) and the time field. Because a mapping can declare body while no document fills it, sampleText() picks the first non-empty candidate at read time and unwraps a collector’s JSON envelope. The (other) sentinel can no longer reach a location string, and factsToBlocks drops samples with no readable text rather than rendering placeholder rows.

Tests: tests/unit/sampleFields.test.ts (9 cases).

Deterministic error sweep

When a project is selected and the user asks an error-intent question, ChatEngine runs the error search itself with forced filters instead of trusting the LLM to pick the right tool and arguments. This guarantees the project’s errors surface — and the Fix it block renders — every time, for any project, independent of phrasing. Implemented in src/core/ChatEngine.ts (the “Deterministic error sweep” block, ~line 1126).

Trigger conditions (all must hold):

• A canonical projectName + clientId are in context.
• The query is guardable (a search/analytics query type) and the user did not pick a specific index.
• The message matches the error-intent regex: error(s), exception(s), fail/failed/failure/failing, crash(ed), stack trace/stacktrace, bug(s), broken.

What it does:

1. Calls executeSearchOtelLogs directly with severity: 'SEVERE', relative_time_hours: 720 (30 days), size: 25, scoped to the project’s clientId + canonical projectName.
2. If there are hits, builds a deterministic markdown summary — total error count, distinct affected code locations (class → method()), and up to 5 sampled error messages.
3. Emits a table block, a code_fix_action block (the Fix it button, whose levels are the affected class/method pairs), and follow-up chips.
4. Streams the message and returns early — the LLM is never invoked. Metadata records model: 'deterministic-error-sweep'.
5. On zero hits or any error, it falls through to the normal LLM path (fails open).

This is the runtime half of Cortex’s error detection: it reads errors that actually happened from OTEL logs in OpenSearch. To keep that data fresh for the demo without manual URL-hitting, a cron exercises the JobSearch app’s normal + simulate-error endpoints every 15 minutes — see Cron Jobs → generate-traffic.sh. The complementary static half (“find errors by reading the source, without running it”) is now shipped as the static_analysis intent → backend /api/v1/code-analysis → analyzer /analyze-exceptions, surfaced as a findings table + an Auto-fix & open PR button that runs the autonomous build-verify + hardening Fix-it.

Dynamic intent routing & typo tolerance

Routing (which deterministic path a prompt takes) is pure code in src/core/intent/, so the same prompt routes identically for every user/window (P2). It is also tolerant of any misspelling, not a hand-listed set:

• Normalization (normalize.ts) lower-cases, expands contractions, strips possessives/punctuation, then fuzzy-folds each token onto its nearest concept word via a bounded Levenshtein distance (threshold 1 for ≤6 chars, 2 for ≥7), guarded by a stop-word list and a uniqueness margin so real words are left alone. So yersterday/yestrday→yesterday, performnce→performance, errrors→errors, activty→activity all route like the correct spelling — no per-typo entries. A small TOKEN_ALIASES map still handles non-edit-distance shorthands (2day, npe).
• Classifier (intentClassifier.ts) matches the normalized tokens/stems against synonym + prefix dictionaries to pick one intent (error / performance / activity / saved / static_analysis / analytical / generic) and a time window. analytical is the open-ended derive/summarize class (see Open-ended derive/summarize); it is the lowest-precedence data intent — it fires only when no specific data intent (error/performance/activity/…) matched, and never overrides a conceptual question.
• LLM route-only backstop (llmBackstop.ts) — when the deterministic classifier returns generic for a project-scoped, non-conceptual question, the model is asked to pick an intent + window only (constrained JSON, temperature 0). The answer still comes from the deterministic templates, never the model. Results are cached process-wide by canonical prompt, so determinism (P2) holds. On any error it stays generic and falls through to the agent loop.

Time windows are UTC. today and yesterday are real calendar days (calendarDayRange in dataQueries.ts): today = [00:00Z today, top of the current hour], yesterday = [00:00Z yesterday, 00:00Z today). Other windows (hour, 7 days, 30 days, last N units) are hour-bucketed rolling lookbacks. Answer phrasing is grammatical per window — windowPhrase/windowSuffix render “Today, …” / “Yesterday, …” / “In the last 7 days, …” (never the old “In the last today”). The activity and performance short-circuits both use these.

Open-ended derive/summarize — the third pipeline

Cortex was built for search: the ~dozen deterministic templates above answer single-metric lookups exactly and cheaply. But users increasingly ask open-ended derive/summarize questions — “summarize JobSearch health and what’s most likely wrong”, “what’s going on with this service”, “give me an overview / diagnose the root cause”. These are combinatorial and unstructured: they can’t be forced into one rigid template, and the old behavior dropped them into the generic agent loop, which had to orchestrate tools and reason and synthesize in one pass — the place most prone to hallucination and incompleteness.

The fix is a third first-class path (beside deterministic-search and the agent loop): the “plan → ground → synthesize” pipeline (src/core/synthesize/openEndedSynthesis.ts + openEndedPlanner.ts, wired into processChatStreamingV2). The principle is that orchestration moves out of the model — the system fetches grounded facts; the LLM only summarizes over them:

1. Route. classifyIntent returns analytical for synthesis framings (ANALYTICAL_PHRASES: summarize, summary, overview, health, what is going on, analyze, assess, diagnose, root cause, derive, insights, state of, …). Precedence is lowest of the data intents — it never steals error/performance/activity, and isConceptual excludes it, so “what is the performance of jobsearch” stays performance, “what is logrite” stays conceptual, and only “what is going on with jobsearch” becomes analytical.
2. Plan (decompose) — Phase 2. planOpenEnded (in openEndedPlanner.ts) asks the model — via a cheap generateCompletion at temperature 0, constrained to compact JSON — which subset of the grounded primitives the question needs ({"primitives":["errors","performance","creations"],"window":"7d"}) and over what window. So “why is JobSearch slow?” fetches only performance; “what broke today?” fetches errors over today; a broad “summarize health” fetches all three. The output is validated/deduped/filtered to the known primitive set, the window is resolved (calendar-aware via resolveWindowRange), and the choice is cached process-wide by canonical question (P2 determinism). Durability: any planner failure / empty / unparseable output → defaultPlan = the Phase-1 fixed all-three bundle, so Phase 2 never regresses below Phase 1. The catalog is the three fast OpenSearch primitives only; heavy static-analysis and multi-window comparisons are deliberately deferred to keep this path’s latency bounded.
3. Ground (fetch). Gated on project context (canonicalProjectName + clientId), buildGroundedFacts runs only the planned primitives in parallel — runErrorSummary (severity≥17 sweep + top locations), runPerformanceSummary (query_exec_ms p50/p95/p99 + breaches + slowest ops), discoverCreationEntities (creation counts) — over the planned window (default 30 days). These are the same deterministic queries the search templates use, so the facts are real and project-scoped. GroundedFacts.included records which ran, so factsToText renders only those sections.
4. Synthesize. The LLM is called with chatWithTools(messages, [], …) — an empty tool list — and a prompt containing only the original question + the grounded facts (factsToText). Because the synthesizer needs no tool-calling and only modest reasoning, cheap/simple models suffice (it ships on the same openai/gpt-oss-120b primary; the synthesizer is pluggable via AIProviderFactory, so DeepSeek-own-API or a frontier model is a config flip, no architecture change). The answer is grounded by construction — the model can’t invent numbers it was never given.
5. Fallback. Without project context, or on any synthesis error, the branch falls through to the agent loop unchanged — non-breaking.

The reply is emitted via emitDeterministicResponse with model: "synthesis:<model>" (so the layer that answered is visible in metadata), plus a structured server log for triage that now records the plan:

[route] path=open-ended project="JobSearch" intent=analytical plan=planner primitives=[errors,performance,creations] window=30 days synth_model=openai/gpt-oss-120b errors=1574 perf_q=2853 creations=3 ms=2226

plan=planner means the LLM chose the subset; plan=fallback means the planner failed and the fixed all-three bundle was used.

This is the durable answer to the search → derive/summarize mismatch: grounding is by construction (robust to arbitrary phrasing and to model quality), and new capabilities are added as new grounded primitives rather than new per-question templates. Next phase: synthesizer-model tiers (Phase 3 — DeepSeek-own-API / frontier as a provider flip) + formalized routing-trace observability.

Conversation memory & re-grounding (multi-turn) — Phase 1

Earlier, multi-turn chat hallucinated after ~3–4 turns. Root cause: tool RESULTS were persisted to tool_executions but never replayed as role:'tool' messages, so on turn N+1 the model saw its own dangling tool_calls with no results and confabulated; history was also appended unbounded with no token budget. The fix (src/core/context/):

• contextAssembler builds every LLM call under a token budget — system → rolling summary → last grounded facts → recent verbatim turns → current message — and strips dangling tool_calls so the prompt is never malformed.
• Durable per-turn memory: each turn persists a compact tool_summary (agent path) or grounded_facts (analytical path, the factsToText) onto the assistant message (messages.tool_summary / grounded_facts columns; conversations.rolling_summary + …_upto_message_id). conversationMemory.buildMemoryBlocks re-injects them as system blocks so follow-ups reason over what was actually measured.
• tokenEstimator is a dependency-free heuristic that self-calibrates against Groq’s real prompt_tokens (no tokenizer dep).
• rollingSummary folds older turns into an incremental running memory (cheap tier; only new turns; fail-soft) when the budget would otherwise drop them.
• Reference follow-ups (“why?”, “which was worst?”) and explicit “summarize” disable tools and answer from the grounded-facts memory; the analytical path threads the same memory as priorMemory into synthesis.

Richer grounding (more context): the open-ended path now also fetches raw samples (runErrorSamples, runRecentLogSamples, slowest-trace ids) so the synthesizer quotes actual error messages instead of only aggregates; a logs planner primitive surfaces recent raw lines; a deterministic one-shot sufficiency gate broadens a narrow-and-empty plan once; and factsToBlocks renders per-dimension tables (errors, samples, percentiles, slowest ops, creations, logs) so long answers arrive as scannable sections, not a wall of text.

Model tiers: AIProviderFactory.getCheapProvider() (planning/summarization) vs getStrongProvider() (synthesis/agent); defaults keep current behavior. Persistence: set CONVERSATION_STORE=postgres in production (+ a /data mount as a single-node fallback) so history survives redeploys. Live multi-turn check: node scripts/verify-multiturn.js.

Robustness, consistency & scale — Phase 2

• Anti-hallucination grounding verifier (src/core/verify/groundingVerifier.ts). The most common hallucination is a fabricated/drifted NUMBER. The verifier builds the set of numbers actually grounded — from the typed GroundedFacts (open-ended path) or tool-result JSON (agent path) — and cross-checks every significant number the answer asserts (handles thousands separators + ms/%/k; skips small ordinals, year-safe values, and letter-glued labels like p95/v2). Pure + deterministic. VERIFY_MODE: warn (default — observe + log, never mutate), repair (one constrained re-synthesis on the open-ended path, then re-verify; caveat if still off), resynthesize (strict → faithful factsToText fallback). The agent path warns + appends a caveat (re-running a 10-iteration loop to repair is too costly). Verdict is in the [route] log (verify=ok/none).
• Consistency / caching. Open-ended synthesis runs at temperature 0, and an answer cache keyed by question + (clientId, projectId, resolved window range) + verify mode replays byte-identically within a bucket — so time-varying windows (“today”/rolling) partition by their resolved range instead of colliding. Follow-ups bypass the cache (their answer depends on conversation context the key can’t capture); only OK-verified answers are cached.
• Model tiers (routing). Planner + LLM route-backstop run on the cheap tier; synthesis + agent on the strong tier.
• Scale. Postgres Pool is sized (PG_POOL_MAX/idle/conn timeouts) with an idle-error handler; SIGTERM/SIGINT now closes the pool (graceful shutdown). A transient failure is retried on the SAME model on its flex lane — there is no fallback to a different model (see Model & Fallback).
• Prompt-injection defense (utils/sanitizeForPrompt.ts). Log/tool content is attacker-controlled; before a sample enters a synthesis prompt its injection directives (“ignore previous instructions”, role-tag spoofing) are neutralized and it’s labeled DATA-not-instructions.

Phase 2 follow-ups (designed, not yet built): per-conversation write mutex, SSE-disconnect abort into the agent loop, full ENFORCE_TENANT_FROM_JWT route wiring (the flag exists; the chatv2-route validation is pending), the sync-path query-cache scope_key (the V2 answer cache already covers the hot path), and a per-request budget/deadline into the agent loop. Earlier Phase-1 follow-ups remain: traces/anomalies/trend primitives, per-section streaming synthesis with citations, and the legacy /chat+sync history-assembly sites.

Long-context, streaming, sources & language (2026-07-17)

A round of upgrades to make Cortex behave like a modern chat assistant (ChatGPT/Gemini/Claude) — hold context for 15–20+ turns, stream character-by-character, cite web sources, and understand imperfect/multilingual input. All additive and env-overridable; the data/grounded paths are unchanged.

• Longer context window (the “loses the thread after 4–5 prompts” fix). With tools enabled the context assembler’s effective history budget was CONTEXT_INPUT_TOKEN_BUDGET − CONTEXT_TOOL_DEFS_RESERVE − SAFETY. At the old 12000 base that left ~1500 tokens for all prior turns (~2 exchanges). Raised the default budget to 28000 (~17500 for history+memory) and CONTEXT_RECENT_VERBATIM_TURNS 2→14, so ~14 exchanges stay verbatim and older ones fold into the rolling summary. Safe on every configured Groq model (all ≥128K context). Live-verified: a 30-turn chat recalled a turn-1 fact (“PostgreSQL”, “5000 req/s”, “e-commerce”) correctly at turns 18/22/25/30.
• General-purpose rolling summary. The rollingSummary system prompt was log-analysis-specific (“error counts, p95…”), so a general conversation’s memory was empty. Rewritten to capture goals, entities/numbers, and stated preferences/constraints for ANY topic; ROLLING_SUMMARY_MAX_TOKENS 400→700.
• True character-by-character streaming. On the ungrounded general/web path (no clientId/project/index) the verify+repair stream buffering is off, so tokens stream live. The FE typewriter (useSSEChat.js) now drains before finalizing on message_end, so even a buffered/one-shot response animates in (LogRite-orange cursor + dots, #E64100). Fixes “the dots show, then the whole answer appears at once.”
• Web source tags. A new sources response block (src/core/context/webSources.ts → SourcesBlock.jsx) aggregates + de-dupes web_search results into a ChatGPT/Gemini-style “N sources” strip beneath the answer. Caveat: live web_search is currently gated by a Groq-side tool-call limitation — both gpt-oss-120b and llama-3.3-70b intermittently emit a malformed function call that Groq rejects (400). The block renders whenever a call succeeds; when it fails the turn degrades gracefully (below) to a knowledge answer.
• Reliable general answers — tool scoping + graceful degradation. When there’s no tenant/project/index to query, only web_search is offered (the ~25 DB/OpenSearch tools are meaningless there and just tempt the model into malformed/hallucinated calls). If tool-calling still fails after retries, runAgentWithRetry does one final tools-disabled pass with a nudge so the model answers from its own knowledge instead of erroring. This eliminated the <function>the data lookup… leaks and 400 errors on general/tech turns.
• Typo / beginner / multilingual understanding. The system prompt now instructs Cortex to infer intent through typos/poor grammar/beginner phrasing (never correct or mock) and to mirror the user’s language (Hinglish→Hinglish, Hindi→Hindi). The guardrail’s off-topic backstop was narrowed to PURE lifestyle (weather/sports/recipes/poems); technical/company/tooling questions pass. Live-verified: a Hinglish prompt (“bhai rate limiting kaise karu…”) got a full Hinglish reply.
• Length nudge + carry-over hand-off. After CONVERSATION_SOFT_LIMIT_TURNS (default 10) user turns the UI shows a “start a fresh chat” banner (nothing is blocked). POST /api/v1/conversations/:id/handoff builds an in-depth summary of the chat, creates a NEW conversation seeded with it (as the rolling memory + a visible recap message), and returns the new id — so the next chat continues seamlessly. The threshold is emitted in message_end metadata (conversation_soft_limit_turns). Live-verified end-to-end: a cold carried-over chat answered “PostgreSQL + HikariCP + 5000 req/s” with no restating.

Env var	Default	Purpose
CONTEXT_INPUT_TOKEN_BUDGET	28000	Total token budget for history + memory assembly
CONTEXT_RECENT_VERBATIM_TURNS	14	Recent exchanges kept verbatim before folding into the rolling summary
ROLLING_SUMMARY_MAX_TOKENS	700	Max size of the running conversation memory
CONVERSATION_SOFT_LIMIT_TURNS	10	User turns before the “start a fresh chat” nudge (0 disables)

Shareable conversation URLs

Logrite-UI ships shareable chat URLs of the form /logrite-cortex/{chatId} (commit a57e400d). The URL is the single source of truth for the active conversation:

• /logrite-cortex is the “new chat” landing — blank state, no empty DB row is created until the user actually sends a message.
• After the first message creates a conversation server-side, the URL silently rewrites to /logrite-cortex/{newId} via history.replaceState.
• /logrite-cortex/{chatId} deep-links resolve in two passes: useParams() reads the id, the URL effect waits for the conversation list to load, and selectConversation() hydrates the messages.
• Sidebar conversation clicks navigate via URL — browser back/forward “just works.”
• Invalid deep-link ids (no matching conversation in the list) silently fall back to the landing route with a console warning instead of a stuck loader.
• The URL contains only the opaque conversation UUID; no auth tokens, no user ids, no query parameters. Auth remains header-based.

Implementation lives in src/components/SidebarPage/Assistant.jsx (Logrite-UI). useChatLogic is invoked with { skipAutoRestore: true } so the hook’s localStorage.logrite_last_conversation_id auto-restore doesn’t race the URL effect on deep-links.

Persistent storage for conversations

Conversation history lives in SQLite at /app/data/conversations.db inside the ai-service container. On dev /app/data is a named Docker volume (measured 2026-10-01), so a redeploy keeps it. On prod it is staged onto a bind mount as part of the deploy (/opt/logrite-stack/docker-compose.volumes.yml).

Operating it

The cli/*-cli.cjs scripts and logrite-manager.js drove PM2 on the host destroyed on 2026-08-03; they manage nothing now. The service is the ai-service compose service on each VM (ssh dev / ssh prod):

C=$(docker ps -q --filter label=com.docker.compose.project=logrite --filter label=com.docker.compose.service=ai-service)
docker logs -f "$C"                        # logs
docker inspect -f '{{.State.Status}}' "$C"   # status

Deploy by pushing to main (the deploy watcher builds and deploys it on dev), or redeploy from the console at https://management.logrite.ai — or, from the dev VM:

curl -sX POST http://127.0.0.1:8090/api/deploy -H 'content-type: application/json' \
  -d '{"service":"ai-service","host":"dev","ref":"main"}'

Prod moves only when promoted from the console. See Deployment platform.

Database Schema

SQLite database at data/ai-service.db:

Table	Purpose
conversations	Chat sessions with metadata
messages	Individual messages (user, assistant, tool)
tool_executions	Tool call history and results

Environment Variables

NEURALWATT_API_KEY=...         # model key; LLM_API_KEY and GROQ_API_KEY are read as legacy aliases
AI_PROVIDER=neuralwatt         # default
LLM_MODEL=deepseek-v4.1-flash  # default; the one pinned model (legacy alias GROQ_MODEL)
PORT=9300
SEMANTIC_SEARCH_URL=http://localhost:9100
OPENSEARCH_URL=http://localhost:9200
BACKEND_URL=http://localhost:8080
ANALYZER_URL=http://localhost:9000
GOOGLE_SEARCH_API_KEY=AIza...  # Optional — enables web_search (Google CSE; closed to new customers, ends 2027-01-01)
GOOGLE_SEARCH_CX_ID=abc123... # Required with GOOGLE_SEARCH_API_KEY
BRAVE_SEARCH_API_KEY=BSA...   # Optional fallback — Brave Search ($5/month credit ≈ 1,000 queries; card required)
CORTEX_LLM_FIRST=false        # true → LLM agent loop answers every prompt (ChatGPT-style); see LLM-first mode
AGENT_TURN_TOKEN_BUDGET=30000  # agent-loop budget per turn, in tokens spent BEYOND the fixed prompt; 0 disables
AGENT_TURN_MS_BUDGET=45000     # agent-loop wall-clock budget per turn (ms); 0 disables

Testing

cd services/ai-service/dev/logrite-ai-service    # in a MasterControl checkout
npm test               # All tests (hermetic — no running service needed)
npm test -- --verbose  # Verbose output
npm test -- tests/api/chat-scenarios.test.ts  # Specific file

Test suites:

• health.test.ts — Health endpoint
• conversations.test.ts — Conversation CRUD
• chat-scenarios.test.ts — Real-world AI scenarios
• error-handling.test.ts — Error handling paths

Resilience & reliability (2026-07)

• Transient-failure retry — superseded 2026-10-07: the earlier Groq multi-model chain is gone. A 429/5xx/timeout is retried on the same model on the flex lane only; see Model & Fallback.
• Sticky project per conversation — a follow-up that omits the project keeps the conversation’s pinned project (mirrors the index/agent pins), so Cortex doesn’t drift to another project on a 2nd/3rd prompt.
• Canonical empty-state — one internal-free “No logs or exceptions were found for X” message (no tenant/clientId, serviceName filters, or index/DSL internals leaked).
• Workspace-scoped scan access — the AI-service scan-ownership gate now defers to the backend’s workspace authorization (requireScanAccess), so summarizing a workspace- shared scan works without reopening cross-tenant IDOR.
• Stale-registry own-index tolerance — a just-created index the caller owns (log-analytics-<theirWs>-*) is trusted even before the 5-min registry refresh.
• Honest foreign-index redirect — asking for an index outside your workspace now says so instead of silently returning your own project’s data.
• “Fix it” button — only offered when search_otel_logs/search_exceptions results are actually errors/exceptions (not any informational log).

Live stream or uploaded file: the index list says which (2026-10-15)

Cortex picks which log index to search from the inventory in its system prompt. That list gave each index’s name, size and fields, nothing more. On prod workspace 302, asked why users could not sign in to Worksoft Portal, Cortex read log-analytics-302-authenticationservice-ef1613, a file uploaded on 7 Oct whose name matched “AuthenticationService”. Meanwhile log-analytics-302-worksoft-auth was logging the same LDAP error every few minutes, 2,187 times in all. The diagnosis was right about a week-old snapshot and silent about the live outage.

Each log index line now says what kind of data it holds:

`log-analytics-302-worksoft-auth` (10167 docs) — LIVE STREAM, newest event 2026-10-15 09:52 UTC — fields: …
`log-analytics-302-authenticationservice-ef1613` (3000 docs) — UPLOADED FILE AuthenticationService.log (a one-off snapshot, not live, covers 2026-10-06 → 2026-10-07 only) — fields: …
`log-analytics-2905-collector-7551` (3853 docs) — STREAM, QUIET — no event since 2026-10-15 05:58 UTC — fields: …

There are also two rules. When both kinds are present, search the live stream (or both) even if only the file’s name matches the question. Every answer names the index and the dates its evidence comes from.

• How it is read (indexRegistry/freshness.ts): on the registry’s 5-minute refresh, one _msearch per 100 log indexes, size: 0. It asks for the newest and oldest event, the share of events carrying an upload’s analysisId (half or more means an upload), and the top sourceFile. An index whose mapping carries the upload provenance stamp _meta.logrite counts as an upload as well.
• Why from the data, not the mapping: every log-analytics-* index shares one template, so an upload and a collector stream have the same fields.
• QUIET means no event for over 24 hours. A collector that has stopped is not the current state either.
• Absolute times only. “8 min ago” would change the system prompt every minute and defeat the provider’s prompt cache.
• Best-effort: an index whose read failed is described exactly as before, with no label.
• Uploads that share a name are one line. The labels alone were not enough on dev workspace 3103. It has 26 uploaded authenticationservice-* files beside one live worksoft-auth, and its inventory ran to 243,000 characters. Asked about AuthenticationService, Cortex read only the uploads in 2 of 3 runs and ran out of lookups doing it. Live streams are now listed first, and uploads whose names differ only by file number and upload suffix become one line: `<base>-*` — N UPLOADED FILES (e.g. x.log) (one-off snapshots, not live, together covering A → B only; D docs), plus the shared field paths. The pattern stays inside the workspace fence. Workspace 3103’s whole inventory was 1,577 lines; its log section is now 358 lines (49,204 characters).
• An upload names its live twin. Even grouped, the uploads were read in 2 of 3 runs. Their name matched the question (“AuthenticationService”) and the live stream’s (worksoft-auth) did not. The freshness read also takes each stream’s top filePath values. An upload whose file or index name equals the file name in one of a stream’s paths (AuthenticationService_142.log and …\Logs\AuthenticationService.log) now ends with: “the SAME log as the live stream log-analytics-3103-worksoft-auth: for what is happening now, search that”. Directory names never count: /home/deployd/… once tied an uploaded deployd.pem to a load-generator stream. Names under 6 letters, such as app.log, never link. On dev workspace 3103, all seven Worksoft upload groups link to their stream.

Measured on dev workspace 3103. The question was “Users cannot sign in to Worksoft Portal, one account or many?”, in three wordings per build. Count of runs that answered from the live stream:

Build	Answered from the live stream
before	2 of 3 (one read only uploads)
labels only	1 of 3 (two ran out of lookups on 26 uploads)
plus upload groups	1 of 3 (the other two answered from uploads, but dated them)
plus live-twin note	3 of 3, each naming its index and time window

Multi-index retrieval (2026-08)

Cortex can be pointed at several indexes in one turn, and those indexes may hold entirely different kinds of data. Everything in this section exists because the previous design answered a multi-index question by reading one index and narrating the result as if it had read them all — which is the one failure mode that looks exactly like a correct answer.

Routing is a function of what an index HOLDS, not what it is called

familyOf is a table of seven name prefixes; anything else is unknown. That made routing a property of the name, so an uploaded log file, a customer’s vendor index, or any new index type was misrouted until someone shipped a new prefix — and a name that lies was believed. inferCapability now reads the shape off the fields the mapping actually carries, keeps the prefix as a cheap hint, and reports when the two disagree:

Shape	Evidence in the mapping	Tools offered
trace	traceId / spanId / durationInNanos	search_traces, search_exceptions, opensearch_query
finding	a rule/framework plus a disposition	search_compliance_findings, opensearch_query
relay	provider / model / verdict	search_warden_requests, search_compliance_events, opensearch_query
log	a clock, a level and a message	search_otel_logs, opensearch_query
document	free text with no clock — an uploaded file, usually	search_otel_logs, opensearch_query
vector	a knn_vector / *_embedding field	semantic_search, hybrid_search
opaque	nothing recognisable	opensearch_query

The order matters: a trace and a finding both usually carry a clock and would otherwise both read as log.

The semantic tools are added by evidence of a vector field, never by shape. A data-prepper log index has a *_semantic sibling and can be searched by meaning; an uploaded log file has no embeddings at all, so offering the same tool over it can only ever return nothing — which is precisely how a populated index gets reported as empty.

An unreadable mapping falls back to the name and says so (fromName: true). Claiming opaque from an empty field list would strip every 403’d index of every tool that can read it, on the strength of a request we never got an answer to.

ScopeSet — N verified targets, grouped by shape

The scope used to be a scalar that sometimes carried a comma-joined list, and every silent-scope-loss defect had that one root. A scope is now N tenant-verified targets, each with its own mapping and its own capability, grouped BY SHAPE — so a log index and a compliance index in one selection produce two reads that merge, rather than one that wins.

The tenant fence lives at construction and only there. Widening a scope widens the blast radius of a prompt injection, so a foreign index never becomes a target and therefore cannot be named by any downstream query however the model is manipulated. With no verified workspace id, an index named in the user’s prompt is not read at all — a scope driven by user text is also the shape an injection takes.

What a mixed selection no longer does

• Order no longer decides the answer. chooseCountFamily tested startsWith against the raw joined pin, so agent-flags-…,log-analytics-… counted compliance and dropped the logs — and reversing the selection dropped compliance instead. chooseCountFamilies returns every family the pin covers and the empty-hand count runs each.
• Several sources are reported PER SOURCE, never summed. Adding a compliance-finding count to a log-line count produces a total that means nothing and hides which source contributed what.
• A wrong-family entry is deferred, not dropped. resolveIndexesForProject used to strip wrong-family entries and only complain when every entry was wrong, so a log+compliance selection lost the compliance index with no message and no second read. Those are handed back as deferredPins and read through their own tool.
• Each side is filtered with its own vocabulary. severityText ∈ {SEVERE, ERROR} for a log index and severity ∈ {critical, high} for a compliance index, in the same request. One shared filter returns a confident zero for whichever side it does not fit.

A missing index is a failure, not a zero

A search whose target resolves to no index returns HTTP 200 with zero hits, not a 404 — ignore_unavailable=true and, for wildcards, allow_no_indices. The transport could not tell that from an honest empty, so “how many errors are in <index that does not exist>” answered 0: an outage narrated as a clean bill of health. _shards.total is the only field that separates “searched, found nothing” from “searched nothing”, and it is now read.

This is the three-outcome contract applied per target: ok, empty with a reason, fail with a classified cause. Data found beside an unreadable source is still found — and the fault has to be reported alongside it.

An index NAME must not decide intent

normalizePrompt split log-analytics-2905 into log analytics 2905, and log is an activity word; warden-requests-2905 folded warden onto harden and routed a data question to static code analysis. Signals are now read with the index names removed, while normalized keeps the full prompt — it feeds deterministic cache keys, and two questions differing only by index must never share an entry.

Correlating across two indexes

Joins could only be planned on five field names we had chosen, so two indexes that relate on shipmentNo were reported as unrelatable. The fallback learns the key from the two mappings instead: a column both sides declare, both map to an aggregatable key-ish type, and neither uses for time or severity. Typed rather than named, so it holds for a schema nobody here has seen.

A learned key then has to prove it is one. Cardinality cannot be taken from a mapping, so it is read from the sample the pass already fetched: too few distinct values and this is a category, and correlating on it relates every row to every row and calls it a finding. severity is shared by every log index and ERROR passes every plausible looks-like-an-identifier test — following it would correlate every error in the tenant with every other one, on a coincidence.

The evidence ledger covers every read

The retrieval ledger only knew about the call sites somebody had remembered to wire. osClient recorded the grounded data path and the federation gateway recorded platform reads; the ~20 tool files that call OpenSearch or the backend with their own axios did not — and in agent mode those tools are the retrieval, so a turn could search four indexes and ship an envelope saying it had read none.

Attribution now comes from the request URL, at one interceptor, so a tool written next month is on the ledger the day it ships. Writes, mappings, cluster probes and detector inventories are deliberately not evidence: a workspace that was created, or a list of configured detectors, is not data that was found. Platform-database answers are recorded the same way — a fully-populated project card used to arrive with metadata.retrieval absent, telling the reader that the most grounded kind of answer in the product was ungrounded.

Measuring it

tests/fixtures/prompts.retrieval.json is the measurement spine: 77 prompts across 13 classes, each carrying an expectedOutcome (found / honest-zero / unknown) and expectedTargets. scripts/eval-retrieval.js scores retrieval rather than prose — found-rate, false-zero rate (budget 0), unknown-as-zero rate, honest-zero rate and per-target coverage, with coverage read from the retrieval trace so it reflects what ran rather than what the plan said.

The corpus pins intended routes, deliberately unlike the older fixture which pins current behaviour: a corpus that agrees with today can never show a routing defect. The same 13 classes are the backbone of the Cortex multi-index test scenarios written for QA.

Recording every read at one place (2026-08-28)

The evidence record described above was filled in by the functions that performed the reads. That works for as long as every one of them remembers to do it, which is another way of saying it works until someone writes a new one.

It had already stopped working in one place. When a question is answered through the chat endpoint, the retrieval is performed by the assistant’s tools, and none of them recorded anything. The direct path measured full coverage while the path real users take measured 83.7% — not because it retrieved less, but because it wrote down less.

What counts as having read something

Four response shapes arrive over a successful connection and mean “nothing was read”:

• A request the caller allowed to return any status — a 403 arrives on the success path, and a permission failure is not an empty result.
• A pure aggregation, which returns no total and no rows because every figure it computed is somewhere else in the response. Reading only the rows called a query that summarised thousands of documents empty.
• A platform response that returned 200 with a refusal in the body.
• A count the store has capped rather than computed. This is recorded as no figure at all, never as the ceiling, because the record decides which numbers an answer is allowed to state as fact.

Keys are learned from your data, not from a list

Correlating two sources requires knowing which fields mean the same thing. That knowledge used to be a list shipped with the product, which meant it was correct for the field names we had seen and silent for everyone else.

Both the identifier fields and the join keys are now derived from your own index definitions. A field is a candidate when two of your indexes declare it, and it is used only when the values in it are actually key-shaped — so a severity column full of the word ERROR can never be mistaken for something to correlate on. Join keys are matched on field type rather than name, and a derived correlation must be supported by at least ten distinct keys in the sample already retrieved before it is trusted.

Text and free-form date fields are excluded before a query is planned. A speculative probe that errors is a fault, and a fault makes the entire answer unknown — so the cost of guessing wrong is paid by the user’s answer, not absorbed quietly.

Half an answer is not an answer

The hardest case in the honesty contract is not an empty result or a failed one. It is a question that spans two sources where one can be read and one cannot. There is data, the answer is fluent, and the half that was never reached leaves no trace.

A question naming a source that does not exist now records that source as excluded whether the rest of the scope resolved or not. Previously this was reported only when every named source was missing — so a scope that shrank to nothing was disclosed, and a scope that merely shrank was not.

This was found by a measurement written before it was run, which failed on its first execution against the deployed build. That ordering matters: a disclosure check written after looking at the output is a description of the output.

Measured on the deployed build (2026-08-28)

ai-service 4b2e79e, run from inside the running container against the process actually serving traffic:

pass	routing	found	false-zero	unknown-as-zero	honest-zero	coverage	partial-scope
direct, workspace 2905	100% (77/77)	100% (36/36)	0%	0% (0/3)	100% (7/7)	100% (54/54)	100% (4/4)
direct, workspace 3103	100% (77/77)	100% (39/39)	0%	0% (0/3)	100% (8/8)	100% (58/58)	100% (5/5)
store unreachable	100% (73/73)	n/a	n/a	0% (0/3)	n/a	2/3	n/a
the chat endpoint	100% (77/77)	97.9% (47/48)	0% (0/48)	0% (0/9)	28.6% text-inferred	84.9% (45/53)	100% (4/4)

The second row is the one that matters for the derived keys. Workspace 3103 is a different tenant with different index names and a trace family the first does not have, so the correlation logic ran against definitions it was never tuned against and searched every expected source.

What the chat endpoint still cannot do

Recording every read at one place fixed what a recording gap can fix — on the chat path false zeros went to zero and nearly every row now carries a typed record rather than a reading of its own wording. It did not improve how many sources get searched, and the honest reason is that this was never a recording problem.

Every source the chat path misses belongs to the compliance or relay family, and the same questions reach every one of them through the direct path on the same build. A question that names a compliance index while a project is selected is answered from that project’s log index instead — and because that path resolves its scope separately, the index the question named is absent from the record entirely, so nothing marks the answer as partial.

This is the same failure the rest of this page is about, in the one place the fix for it does not reach. It is recorded here rather than described as closed, because a coverage figure that moved from 83.7% to 84.9% is not a fix and should not be read as one.

Retrieval reliability architecture (2026-08-06)

A reliability audit of the grounded data path found 17 defects behind one user-visible symptom: Cortex reporting “0 results everywhere, the logging pipeline appears to be non-functional” for a workspace that actually held 1,196,262 compliance findings. The counts it reported were correct; the conclusion it drew from them was not.

Root causes

1. Three populated index families had zero tool coverage. Measured on dev workspace 2905:

Index	Docs	Reachable by
agent-flags-2905	1,196,262	nothing
relay-compliance-events-2905	645	nothing
warden-requests-2905	524	nothing
log-analytics-2905	0	5 tools
trace-analytics-raw-2905	0	3 tools

Every OpenSearch tool targeted log-analytics-* or trace-analytics-raw-*. 1,197,431 documents were not addressable by any prompt, model, or retry.

2. Failures were indistinguishable from emptiness. 11 of 14 catch blocks in the data layer did console.warn then return null. A 403, a socket timeout, a mapping error and a genuinely quiet system produced the byte-identical value null, which the synthesizer read as “no data”.

The six layers

• L0 — Result envelope. A retrieval is now ok / empty(reason) / fail(reason), as distinct types. The transport type is DataOk | DataFail (it cannot return empty), so the compiler forces every caller to handle failure. On the empty path only, one extra aggregation proves which of NO_DATA_EVER · OUTSIDE_WINDOW · FILTER_EXCLUDED · WRONG_SCOPE applies. A failed probe never upgrades to the strongest claim.
• L1 — Index Descriptor Registry. Reads each index’s real mapping and probes its real values, so severity/time/service fields are chosen by evidence rather than by name. Verified on agent-flags-2905: severityText ∈ {SEVERE, ERROR} matches 0 documents over all time while match_all matches 1,196,262 — that index stores severity ∈ {critical, medium, high}. This registry is also the single source of truth for index exclusion, shared with the frontend so the two lists cannot drift apart again.
• L2 — Family tools. search_compliance_findings, search_warden_requests, search_compliance_events. Descriptor-driven, exact server-side aggregations, unknown filter fields rejected with the valid field list instead of silently dropped, sensitive fields redacted before they reach the model, workspace taken only from the verified JWT.
• L3 — Query correctness. The service filter is case-insensitive (a term query on serviceName.keyword with a lowercased needle never matched "JobSearch"). The time filter now covers startTime (traces) and flaggedAt (compliance findings), which were omitted — so windowed questions over those families matched nothing regardless of content. Widening is time-first and multi-step (1h → 24h → 7d → 30d → 90d → 12mo); it previously took one step and stopped at 30 days while the data was 21 days old. date_histogram honours the caller’s timezone. Rankings come from a real terms aggregation with a widened shard_size — the previous “top error locations” was size: 100 with no sort clause, i.e. an arbitrary sample presented as a ranking.
• L4 — Scope unification and cache hardening. ⚠ Security. The query-cache scope carried no workspaceId; an absent scope meant a shared global bucket; the sync chat path passed no scope at all; and ENABLE_QUERY_CACHE defaults to true. Two users in different workspaces asking a semantically similar question could be served each other’s answer. The cache now fails closed — no workspaceId means no read and no write — and the key includes persona, experience level, role and the exact indices read. Legacy shared rows are purged at boot. Separately, five deterministic branches were gated on !indexName, so pinning an index disabled the quality machinery and routed the user to the least-guarded path; a pin inside the project’s own tenant family no longer does.
• L5 — Claim verification. The existing verifier checks numbers, and the damaging claim contained none — it passed unchecked while the verifier flagged 2905, an index name, as an ungrounded figure. A statement about the state of the user’s system (pipeline broken, nothing ingested, not configured, outage) may now be emitted only when entailed by a verified empty reason with no failed retrieval in the bundle. Deterministic; no LLM.

Retrieval tracing

Every data call emits {op, index, filters, resultKind, reason, docCount, sampled, latencyMs} into a per-request trace. Failures log at error level with the stable prefix [retrieval.fail]; set RETRIEVAL_TRACE_VERBOSE=true to see successes too.

Operations: when a user reports “Cortex says there’s no data but I can see logs in Discover”, grep the AI-service container logs for [retrieval.fail] first. Before this existed, a failing query left nothing behind but a console.warn — which is how these defects accumulated silently.

Role and timezone

• workspaceRole is read from the verified JWT and used to filter the tool list offered to the model, so a withheld capability is never proposed. Read access is unchanged for every role; only create_workspace, start_github_app_install, enable_monitoring and initiate_scan are gated. Tenant isolation remains enforced by workspaceId and the backend bearer — this is capability, not isolation. Previously workspaceRole was read nowhere in the service, so a Viewer and a Maintainer had identical capability in chat.
• timeZone (optional, on the chat request body) drives calendar-day windows and histogram buckets. Absent means UTC. Windows were UTC-only, which shifts an IST user’s “today” by 5.5 hours and drops their entire morning — a frequent cause of a spurious “no data today”.

Env note: OpenSearch → Analytics migration in progress — the service reads ANALYTICS_URL/ANALYTICS_USERNAME/ANALYTICS_PASSWORD with a fallback to the legacy OPENSEARCH_* names (drop the fallback once the Coolify env vars are renamed).

Compliance source streams — every framework has a declared source (2026-08-12)

src/compliance-streams/ keeps the seeded compliance library aligned with the regulations it claims to implement. Until this pass, six frameworks had a source adapter and twenty-three returned null — which the orchestrator reported as “no adapter” and everyone downstream read as “nothing to do here”. Every seeded framework now resolves to an adapter, and the registry makes the difference between them explicit:

Posture	Meaning	Frameworks
live	A real feed with genuine change detection — the orchestrator polls the source’s own version marker and re-ingests only when the regulation actually moved.	HIPAA, GDPR, M-26-14, NIST 800-53, GLBA, COPPA, FERPA (eCFR), EU AI Act, DORA, NIS2 (EUR-Lex)
pinned	A dated, cited record of a source that publishes no feed. It cannot detect upstream change — it expires instead, emitting a review-due clause once the pin outlives its review interval.	SOC 2, PCI, ISO 27001/27701/42001, CJIS, NERC-CIP, FedRAMP, NIST AI RMF, OWASP-LLM, SOX, CCPA, BIPA, Colorado-AI, TRAIGA, NYC-LL144, US-State-Privacy, AI-Data-Gov, GEN

Conflating those two is the failure this design exists to prevent: a nightly “synced, unchanged” on a regulation that moved last week is worse than no stream at all, because it is an assurance nobody has any basis for. So sourcePosture() returns null for an unknown key rather than defaulting to pinned, a pinned fetch() reports contentType: 'derived' with no body, and every pinned source carries a noFeedReason stating why no feed exists (ISO text is copyrighted and sold per copy; state legislatures publish HTML with no stable endpoint; SOX’s real expectations come from PCAOB AS 2201 and COSO).

• genericEcfrAdapter.ts / genericEurLexAdapter.ts — the HIPAA and GDPR adapters were a CFR title+part and a CELEX id with everything else in common, so both are now parameterised. Three US rules and three EU acts share one tested fetch-and-parse path instead of six near-copies.
• pinnedSourceAdapter.ts + pinnedSources.ts — the pin table. reviewIntervalDays is set by how fast the area moves: 90 for AI regulation (phasing in on dated deadlines), 180 for statutes and sector rules, 365 for ISO standards.
• CMMC is deliberately unadapted. The CMMC Program rule is in the CFR (32 CFR 170), so an adapter would work mechanically — and would audit our practice-level rules against program-level clauses about assessment procedure, reporting a wall of drift that reflects the mismatch rather than the rules. KNOWN_UNADAPTED records that with its reason. An honest gap beats a fabricated stream.

Tests: tests/compliance-streams/adapters.test.ts.

A guard that could not tell one scope from another (2026-09-09)

Reported from the product with a screenshot. Two OTEL log indexes pinned in the Cortex composer, the question “any critical insights?”, and an answer that opened:

I found 1,196,262 compliance findings in this workspace. They span 2026-07-03 → 2026-07-16. By severity: critical 1,115,172, medium 80,951, high 139.

Asked again, the same question over the same two indexes answered 1,115,172 grouped by rule; a third run answered 1,196,262 grouped by framework. The complaint was about consistency, and consistency was the symptom rather than the fault.

The mechanism

The emptiness-contradiction guard exists to stop a real and damaging failure: the model writing “no results” over a workspace holding 1.19M findings. It compares denial phrases in the prose against the counts the turn’s tools returned, and when the prose denies data the tools found it replaces the answer with a deterministic census built from those counts.

It had no notion of scope. Three consequences followed, and all three were visible in that one screenshot:

1. The user pinned two log indexes. The model also read the compliance family — which lives in agent-flags-<ws> and is workspace-wide by construction, a family the user never pinned — and the guard took its 1.19M as proof the scoped answer was a lie.
2. The scoped answer said “No critical findings were raised for this index”. That is a statement about one severity bucket, not about the query; the guard’s own header already excludes field-scoped negatives like “no errors”, but its regex allowed thirty characters of anything between the no and the noun, so no critical findings matched the whole-query alternative and bypassed the conservative gate.
3. The replacement discarded the entire analysis. Its first line — “in this workspace” — is the tell: it answers at a scope the user did not choose.

The varying headline number is the same defect seen from the other side. The census renders whichever count-bearing tool happened to return the largest figure that turn, and which compliance tool the model reaches for varies run to run.

The fix

core/verify/guardScope.ts supplies the two things the guard was missing, and both fail open: no tool calls, no pin, or an unrecognisable index leaves behaviour exactly as it was, because a scope check that mis-fires re-opens the bug the guard exists to close.

• Scope attribution. A result is attributed to its tool through tool_call_id, and the three family tools are marked workspace-scoped. Their counts are excluded from the contradiction test only when every pinned index is one the OTEL log tools serve — a user who pins agent-flags-2905 is asking about compliance, and that count must keep firing the guard as it does today.
• Re-entry when the answer names the family. “There are no logs or compliance findings for any time frame” is a lie about the compliance family whatever the user pinned; the model raised that subject itself, so it owns the claim. Matching is on the tool’s full subject phrase, never the bare noun, so a scoped “no critical findings” cannot drag the family back in.
• Severity-qualified denials are field-scoped. A denial carrying a severity word is neutralised before the total-denial test runs. Any unqualified denial elsewhere in the same answer survives untouched and still fires the guard.

The guard is threaded the pinned index from the v2 chat path, which already holds it. The V1 and sync paths carry no pin — indexName is not on ChatRequest — so they pass nothing and behave exactly as before.

Follow-up chips now come from the turn’s own evidence

The chips under that same answer read “Which services are leaking credentials?” and “Why are there so many soc2 findings?” — derived from the census, not from the log analysis. They were generated by the model at temperature 0.4, four of them, after the platform’s own suggestion.

The order is now: a platform-pinned suggestion, then evidence chips computed from the turn’s retrieval ledger, then the model’s, then the deterministic heuristic as backfill. Eight instead of four, and the model runs at temperature 0. Evidence goes before the model because chips are where a user acts, and a chip naming an index the turn never read sends them somewhere the answer cannot follow. Every stage is optional and the list is de-duplicated, so a failure at any one of them shortens the row rather than emptying it.

ledgerNextSteps took a fixed cap of three, so the extra room would have filled with model-invented questions; the cap is the caller’s now and the first three are unchanged.

Two facts the summary counted and never printed

services/logAnalysis/summary.ts computed warnings on every file and showed it on none — so a file that is 40% warnings reported “No error-level events”, which is true and reads as a clean bill of health. It now states the count and share. It also states the average rate across the span, withheld under a minute of span or a single event, where an average is arithmetic rather than a rate.

Measuring consistency, which nothing here did before

Every other harness in this service asks each prompt once and scores the answer against an expectation. At N=1 a wrong answer and an unstable one are indistinguishable, which is why three runs returning three different numbers was reported by a user rather than caught.

npm run eval:consistency asks each prompt N times over one pinned scope and scores agreement: the same set of render-block types, every stated quantity identical, and no run naming a source outside the pin. Prose is deliberately not compared — the model’s wording is not deterministic and never will be, and scoring it would make the harness fail for a reason the product cannot fix, which is how a real regression ends up buried under expected noise.

The guard was half of it — the model was still reading the wrong family

Fixing the guard stopped the census OVERWRITING the analysis. Measured on the deployed build immediately afterwards, three runs of the same question with the same two log indexes pinned:

run	quantities in the answer	what it answered about
1	1,115,021 · 121 · 30	compliance findings by rule
2	1,196,262 · 1,115,172 · 80,951	compliance findings by severity
3	1,351 · 677 · 674 · 104	the two pinned auth-log indexes

Two of three still answered a question the user had not asked, from a family they had not pinned. The guard fix held — the census template was gone — but nothing stopped the model READING the compliance family in the first place, and which family tool it reached for was the run-to-run difference.

A pin of log indexes plus a question that never says “compliance” is not a compliance question, and the three family tools answer a different one: they are workspace-wide by construction, so their counts do not narrow with the pin. They are now withheld for that turn — the same reasoning as the role gate beside them, where filtering the LIST is stronger than refusing afterwards. A compliance-family pin, a mixed pin, an unpinned turn, or a question naming compliance or warden all leave the list untouched.

Applied in both agent implementations. A scope gate on one and not the other means AGENT_ENGINE silently changes the answer.

Confirmed on the redeployed build: four consecutive runs, every tool call targeting only the two pinned indexes, no compliance family in any answer, and scoped clean across all three corpus prompts.

What the harness could not see, and what it still cannot

Its scope check looked for index NAMES, and an answer states compliance counts without ever naming agent-flags-<ws> — so it read scoped=true through the very runs above. It now matches the family’s own vocabulary instead. A metric that cannot see the defect is worse than no metric, because it certifies the bug.

Still varying, and stated rather than counted: numbers and structure. The agent loop issues a different number of tool calls per run — 18, then 9, then 10 on one prompt — so it mentions a different subset of correct, grounded facts and emits a different number of blocks. That is depth varying, not a fact disagreeing: no run contradicted another. The harness cannot currently tell those two apart from prose alone, so the number it prints for those two columns should be read as “did the answers match”, not “was one wrong”. Closing it means constraining the agent loop’s retrieval plan, which changes what the product does and belongs in its own change.

Answering the question you asked about AI Warden (2026-09-10)

Five defects, found by running one workspace’s real AI-Warden questions against Cortex and reading what came back. They are separate faults with one thing in common: each produced an answer that looked well-formed, so nothing on screen suggested checking it.

The question named an index, and Cortex asked which project you meant

Ask “what’s flagged in warden-requests-2905 in the last 30 days” from the Cortex-AI page with no project selected, and Cortex replied that you have 138 projects and should pick one. The index you named was never opened.

Every gate involved was working as designed. Naming a non-log index routes the turn to the analytical path, and that path needs a project — so a project-less turn declined, and the project picker fired. The index name sat in the question the whole time.

An index the question names is now adopted as the turn’s scope — the same value the sidebar sends when you pin one. It resolves through the same tenant fence as every other read: the workspace comes from your session token, never from the text of the question, so naming someone else’s index resolves to nothing. If the name cannot be resolved, or resolves to more than a handful of indexes, the turn routes exactly as it did before.

“Group them by reason” returned the same breakdown as the previous turn

The compliance breakdown card decided its own grouping and never read the question. Asked to group by reason, it returned the verdict census it had already shown — correct numbers, wrong question, and no sign the question had been ignored. reason and action also matched no known breakdown dimension, so every other caller fell through to a default as well.

The card now groups by the dimension you name, and both of those are recognised.

A dimension the data does not carry produced an empty breakdown

“Break these down by framework” over relay requests produced a breakdown heading with nothing under it. Relay requests carry no framework — it is a property of compliance findings — and the search tool dropped the unusable grouping in silence. The result was indistinguishable from an aggregation that ran and found nothing.

The tool now refuses, names the fields that can group this family, and says where the dimension you asked for actually lives. The follow-up chips no longer offer “by framework” on families that have no framework.

A terms count on free text took the whole search down

Related, and worse. Grouping on a field stored as free text is not a bad grouping — it is a hard error from the search cluster, and it kills the entire query: no count, no examples, nothing. The sort clause has been guarded against this for a while; the aggregation had no equivalent guard, which meant a newly-recognised dimension like reason (free text on the findings index) could have destroyed the query it was meant to improve.

Both paths now check that a field can carry a count before asking for one. If it cannot, you get the refusal above; the search itself still runs.

The figures in your question are now checked against the data

A follow-up asking about “the 14 requests” was answered in full when the window held two. The shape of the question was answered while its premise was ratified.

Cortex now corrects a figure that does not match what it counted. The check is deliberately narrow — the number must be attached to a data noun, and the thing it counts must be a bucket Cortex actually counted — because a wrong correction is worse than none.

Records that share a word with your question are not an answer to it

The one that mattered most, because more data made it worse.

Asked “which claims were paid without a human approval, and who released them?”, a workspace with no claims data at all returned seventeen secret-detection findings, presented as the answer. A free-text search matches records containing any word of the phrase, so records carrying the word “human” scored as hits — and 17 results reads exactly like seventeen records about your question. The same question against an empty workspace answered honestly that there were no records.

Cortex now measures how much of your phrase the data actually contains, and says when words are missing from it entirely — here, “claims”, “paid” and “released” appeared in no record.

Deliberately, this reports rather than suppresses. A search for “PHI leak” against records that say “PHI detected in completion” is missing the word “leak” and is still exactly the answer you wanted. Narrowing the search to remove the false positives would reintroduce the false negatives — an answer of “no records” when the records are right there — and that is the more expensive failure of the two.

What is verified, and what is not

All five fixes are covered by tests that pin the defect and the behaviours around it, and the deployed build was verified by its content rather than its version tag.

The live re-run of the original question set is not done. It needs a session token for the environment under test, and it should be repeated before these are considered closed on anything but the code.

A guardrail that only spoke English (2026-09-10)

Cortex screens each prompt before it runs, using fast deterministic rules first and a model only for what the rules do not settle. Every rule was written in English.

Two consequences, and the second is the one that affected ordinary use.

Attacks typed in another language reached the model. “Ignore all previous instructions” was blocked; the same instruction in Hinglish, Hindi, Spanish or French was not. Those prompts were never unguarded — the model backstop judges intent rather than language, and tenant isolation is structural: your workspace comes from your verified session, and no phrasing in any language can move it. But the backstop is optional, billed per call, and slower than a rule that costs nothing.

Ordinary questions in a non-Latin script were refused as gibberish. The gibberish check counted a–z and 0–9. Every Devanagari, Gujarati, Arabic, Chinese, Cyrillic and Tamil prompt therefore scored zero recognisable characters and was answered with the “that does not look like a question” redirect. A rejected prompt short-circuits before the model backstop, so nothing gave it a second opinion — and Cortex’s instruction to reply in the language it was asked in was unreachable for anyone not typing in Latin script.

Both are fixed. The gibberish check now counts letters in any script. Deterministic rules cover Hinglish, Hindi, Spanish, French, German, Portuguese and Italian, and English behaviour is unchanged and pinned by tests.

Care went into the direction that costs more. workspace mein mere logs nahi dikh rahe hain is a user reporting their own data missing — the exact question this product exists to answer — and an early version of the cross-tenant rule blocked it. The rule now requires the other workspace to be named, not merely for the word to appear near a negation. A guardrail that blocks real questions is a worse product than one that misses an attempt the backstop would have caught anyway.

Arabic, Chinese, Cyrillic and Tamil now reach the pipeline, but have no deterministic rules of their own yet; they rely on the model backstop, as any novel English phrasing does.

Logrite’s own internals are not something Cortex discloses (2026-09-30)

Cortex is asked about Logrite constantly, and almost all of it is exactly what it is for: how the API works, how to configure a rule, what a page is showing, what a limit is. One class of question is not. Asking Cortex to describe Logrite’s internal APIs and endpoints, its service or container inventory, its hosting and deployment, its database schemas, its source repositories, or any Logrite-side key, token or credential is a request it declines.

It is declined in two places, because there are two ways it could happen.

The prompt is screened before anything runs. The pre-flight guardrail gained a category for this, alongside the ones it already had for jailbreaks and for the assistant’s own system prompt. Deterministic rules resolve the clear cases at no cost; the model backstop is told about the category too, so a phrasing no rule anticipated is still caught. Rules cover English, Hinglish, Hindi and the Latin-script European languages.

The answer is constrained even when the question was innocent. A prompt-screen only sees the question. Cortex could still volunteer an internal detail while answering something else — a tool result, a log line or an attached file can contain one. So the system prompt carries a confidentiality rule for the other direction: never state those things, not partially, not as an example, and not because they appeared in context.

This is not a rule about discussing Logrite

“How do I use the Logrite API?” is the most common question this product takes, and it is never withheld. Neither is “why is this service erroring?” — that is a support report, not an extraction attempt. The line is between using Logrite and operating it.

Your own infrastructure is your data. Your /internal/… route, your database schema, your container, your hostnames — analysing those is precisely what Cortex is for, and a question about them is unaffected.

The false-positive direction is the one that was tested first. Sixteen ordinary prompts — using the public API, authenticating against it, reporting a 502 on a Logrite host, debugging your internal endpoints, listing your indexes, asking what a stack trace means — are pinned as passing, and so is every suggestion chip the product itself offers. A guardrail that refuses a real question costs more than the disclosure it prevented.

The refusal is in the language you wrote in

Refusal copy was English-only, which meant a Hindi-speaking user who tripped any rule got an English answer — on the one turn where the product is already declining to help them. Every refusal now renders in the language of the question: Hindi, Hinglish, German, Spanish, Portuguese, French, Italian, Dutch, Japanese, Korean, Chinese, Thai, Arabic, Hebrew, Greek and Russian, with US English as the default.

Three properties are worth stating, because each rules out an easier implementation:

• The copy is fixed, never generated. A refusal is written per (category, language) and is byte-identical for every user who trips the same rule in the same language. Translating it with a model would give that up on the one turn the product has already decided not to trust a model with the answer.
• The fallback is per message, not per language. A language with no copy for one category falls back to US English for that message alone. You never see a half-translated sentence or an empty answer.
• A language is only chosen when it is certain. Cortex’s existing language detector decides this, and it reports when it cannot read a question; an uncertain guess answers in English rather than guessing wrong at the worst moment. Gujarati and Tamil reach the pipeline and have no translated copy yet, so they are refused in English rather than in something approximate.

The suggestion chips beneath a refusal stay in English. They are prompts you send back rather than prose you read, and they are the product’s own canonical examples — the intent router is tuned against those exact strings. Cortex reads a prompt in any language, so editing one still gets an answer in yours.

Cortex answers, October 2026 (2026-10-07)

Fixes from the Sandbox test report. Each one makes the same question give the same answer from the same facts.

Questions about an uploaded file are answered from its stored analysis

POST /api/v1/chatv2 takes an optional analysis_id (the Sandbox’s “Continue in chat” on an upload sends it). It is resolved against the caller’s own workspace only; a foreign or expired id is treated as if it had not been sent.

v2UploadFindings (src/core/chatEngine/v2/uploadFindings.ts) runs in streamV2.ts before the index summary. It claims the turn when the first of these finds a stored analysis in the caller’s workspace:

1. the request carries analysis_id;
2. the pinned (or implied) index is a single dataset — no * or , — starting with log-analytics- followed by the caller’s workspace id and -, and that dataset has a stored analysis;
3. the message matches UPLOADED_PHRASE — “I/we (just) uploaded…” or “the log/file I (just) upload…” (past tense after I/we only, so “how do I upload a log?” does not match) — and the latest analysis in the workspace is at most 24 hours old (UPLOAD_RECENT_MS).

When it claims the turn:

• The facts come from the stored analysis (uploadFacts, capped at 6 KB): headline, events analysed, the file’s own level counts (e.g. INFO/WARN/ERROR/FATAL), error and warning totals, time range, likely trigger, findings with samples, and top patterns. Nothing is re-queried, so the same upload always gives the same facts and the same facts_digest.
• The model writes the answer; every number in it is checked against the facts. If the answer has a number the facts do not contain, is empty, or the model is unreachable, the facts are shown as written text instead (when the model is unreachable, AI_UNAVAILABLE_NOTE is added after them).
• A sources block names the dataset: title Uploaded file: followed by the dataset id, linking to /analytics/discover?index= plus the dataset id.
• message_end.metadata.handler is "upload-findings". Follow-ups (“and the warnings?”) use the same analysis again.
• The answer repeats. It is stored in the Redis cortex:upload cache, keyed on the question, the analysis, its facts_digest and the model, so asking again replays it. An “AI unavailable” fallback is never kept.
• It is never truncated. An answer the output ceiling cut off is not shipped.

If the lookup fails, the turn continues down the normal path.

The agent loop also has a new tool, get_log_analysis (src/tools/logAnalysis/getLogAnalysis.tool.ts) — “Read the stored analysis of a log file the user uploaded. Use it first for any question about an uploaded file; it already holds the errors, counts and likely cause.” Parameters, all optional: analysis_id, dataset, latest (boolean; also used when neither of the other two is given). The workspace comes from the verified session, never from the model. It returns {success:true, found:true, analysis} with the same ≤6 KB facts. When nothing matches, or when the id or dataset belongs to another workspace, it returns found:false with “No stored analysis matches. The upload may have expired, or it was analysed in another workspace.”

Reporting verbs are now in ENTITY_STOPWORDS (src/core/intent/vocab.ts): say, says, said, mean, means, tell, show, explain, describe, summarise, summarize, think. Before this, “What do the findings in X say?” ran the error sweep with “say” as the search term and answered No SEVERE errors matching "say" for an upload with 152 errors.

One severity vocabulary

SEVERE, ERROR, ERR, FATAL and CRITICAL (ERROR_INTENT_WORDS, src/tools/otelLogs/queryShape.ts) are now the same filter: each one maps to every error-or-worse level the index uses. Before, ERROR matched only ERROR, while SEVERE matched every error level, so the same upload gave 150 one way and 152 the other. A search_otel_logs call whose query is only a severity defaults to ERROR. An error-level result now starts with (severityHeader.ts):

matched 152 error-or-worse lines (ERROR 150, FATAL 2); showing 10.

The per-level breakdown in brackets is shown only when it adds up to the total. The model is told to report the total as errors, using the level names the index uses, and never to call them SEVERE unless the index does.

Times are when events happened, not bucket boundaries

The request timeZone is passed through to the facts. Trends now carry the first and last matching event, and factsToText (src/core/synthesize/trendTimes.ts) describes a time bucket as a period, not as the moment of an event:

• event span: errors occurred 09:03–09:08 UTC on 5 Oct (14:33–14:38 IST) — the local time in brackets is left out when the zone is UTC; spans across days give day and month at each end;
• daily peak: busiest day was 5 Oct (Asia/Kolkata) with N errors;
• sub-daily peak: busiest hour began 09:00 UTC (14:30 IST) on 5 Oct with N errors.

A workspace with no application logs is told so immediately

v2NoLogsGuard (src/core/chatEngine/v2/noLogsGuard.ts) handles a turn with no project, no pinned index and no attachment that asks about logs — intent error or activity, or the word “log(s)”. It does not handle conceptual or product questions, or questions about compliance or Warden. It counts each application-log family — log-analytics, log-observability, log-cloud-platform, trace-analytics-raw (the workspace index and its -* children) — and answers only when all four were read and all are empty:

There are no application logs in this workspace yet. What I can see: N AI Warden requests in the last 7 days and N uploaded files.

Chips: Upload a log, Connect a collector, Show Warden requests. handler:"no-logs". If any family cannot be read, the guard does not answer. It was added because the agent loop spent 90.9 s and about 102,000 tokens searching for logs that did not exist.

A per-turn budget for the agent loop

Env var	Default	Where
AGENT_TURN_TOKEN_BUDGET	30000	src/config/values.ts → agentTurnTokenBudget
AGENT_TURN_MS_BUDGET	45000	src/config/values.ts → agentTurnMsBudget

0 turns a budget off.

What the token budget counts (2026-10-08). Only what the turn adds: completions plus the evidence each tool result appends to later prompts. The first call’s prompt (system prompt, tool definitions, history and the question, about 27,000 tokens on dev) is resent on every call and is subtracted once per call. Counting it, as the first version did, spent a 30,000 budget before a single tool ran. The dev log read turn budget reached (tokens 55387 >= 30000) after one round. When the opening prompt alone passed 30,000 (a longer history), every first-round call was answered “Not run: this turn reached its search budget”. That is HIPAA re-test H-X13/H-X14 (“both requests were skipped before they ran”), and why replying “retry” (H-X16) only made it worse, since a retry lengthens the history.

The first round always runs, for both budgets. A turn that has run no tool has nothing to answer from, and a first model call that queued on the flex lane (~60 s) used to exhaust the 45 s clock before its own searches started.

When either budget runs out after the first round, AgentLoop stops calling tools and runs one final pass with no tools, answering from what it has already gathered. Tool calls still pending in the current round are not run; each gets {success:false, skipped:true, note:"Not run: this turn reached its search budget."}, so every tool_call_id has a reply. Each round’s results are matched to their calls by tool_call_id rather than by position, so a fallback result added mid-round can no longer attach to the wrong call.

Index family labels

FAMILY_PREFIXES in src/services/indexDescriptor/families.ts covers every Logrite index type. These labels appear when Cortex lists what a workspace contains:

Prefix	Label
log-analytics-	Application logs
log-observability-	Application logs (vendor-forwarded)
trace-analytics-raw-	Traces
agent-flags-	Compliance findings (AI Warden)
warden-requests-	Warden requests
relay-compliance-events-	Relay compliance events
file-	Uploaded files
log-uploads-	Uploaded files (raw)
mcp-activity-	MCP activity
cortex-traces-	Cortex traces
analytics-state-	Saved analytics state
compliance-logs-	Compliance logs
log-cloud-platform-	Cloud platform logs
coding-events-	AI coding sessions
anything else	Other indexes in this workspace (not a Logrite data type — usually created by an integration or by hand)

Sources under knowledge and dataset answers

• Knowledge-pack answers now include a sources block (knowledgeCitations, src/core/chatEngine/v2/sourceBlocks.ts): up to 6 links, de-duplicated by URL, each titled with the pack name, —, and the section. Topics without a public URL (the Logrite pack) are left out.
• Upload-findings and pinned-index answers include dataset sources: one entry per concrete dataset, each linking to Discover.

All of them use the same sources block type as web search, so the chat shows them the same way.

A search that failed is mentioned in the answer

When any tool result in the turn had success:false, v2/repair.ts adds this line to the end of the answer (failedSearchNote, src/core/verify/failedSearchDisclosure.ts). It is added after the cleanup step that removes the model’s own “the search failed” sentences, so it is always shown:

Note: 2 searches could not run (relay requests: timeout; traces: index not found), so this answer may be incomplete.

Up to three failures are listed as source: reason, with each reason cut to 80 characters and secrets removed; more than three adds ; and N more. A repeated call that returned the same failure is counted once. On a live stream the note is sent as an extra content_delta.

message_end.metadata

Every chat message_end now has these fields (messageEndMeta, src/core/consistency/turnMeta.ts):

Field	Meaning
handler	The branch that answered (upload-findings, no-logs, …); if none is set, the model label
model	The model (or deterministic label) that produced the content
tier	standard or flex — the lane that answered
cache	hit, miss or local
facts_digest	Digest of the facts the answer was based on, or null
data_version	Agent path only: the digest of the workspace data the answer was keyed on

For the model, lane and per-call policy, see AI models.

Asking twice gives the same answer — what is stored (2026-10-07)

Three Redis caches, all under cortex:* with a 24-hour TTL, make the agent path, upload questions and log analysis repeat (the full cache table and key parts are on AI models):

Key prefix	What it replays
cortex:agent	agent-path answers
cortex:upload	answers about an uploaded file (question + analysis + facts digest + model)
cortex:logfindings	the log-analysis model pass’s findings

An agent-path answer is stored only when it is the first turn, has no attachments, every tool call succeeded, and the verifier said ok (or verification is off). A turn that ran no tool is keyed without the data version, so new logs do not invalidate a data-free answer; the knowledge-pack pages injected into the turn are part of the key. Before ai-service 2e07c8e the agent cache never stored a single answer: first-turn eligibility read the history, which already held the question.

When the model fails, the call moves from the standard lane to the flex lane of the same model; if both fail, paths that hold measured facts show them followed by “The AI step is unavailable right now; these are the measured facts.”

Live-found fixes (2026-10-07)

• Workspace data tools without a pin (D-1). A workspace-authenticated turn gets the workspace’s data tools even when no project or index is pinned. Web tools are offered only on an un-tenanted turn.
• The question’s own numbers are grounded. The verifier accepts numbers that appear in the user’s question — the line numbers of a pasted stack trace — as it already accepted numbers in an attached file.
• Numbers quoted from retrieved log lines are grounded. An answer that explains log lines quotes them (“exceeded the 3,000 ms timeout”, “port 5432”); those figures were policed as invented because only counts and aggregation values were read, so the recent-errors preset was repaired — and different — on every asking. Numbers present verbatim in the retrieved rows now count as grounded (up to 100 rows, 4,000 characters per field). An invented count is still flagged.
• A previewed tool result keeps its counts (9323feb). A result still over the cap after array trimming used to be replaced by {_truncated, preview: <first 2,000 chars>}, so total_hits survived only inside a string: the model could read it, the grounding verifier could not, and the headline count was policed as invented and repaired on every asking (prod: “140,796 error-level entries”, a 2,380-byte result). The fallback now carries the small top-level fields (≤600 characters each, ≤3,000 in total — counts, the searched window, notes, the index list) beside the preview, and numbers inside a preview are grounded like any retrieved row’s. The truncation lives in core/toolResultTruncation.ts.
• A rolling window is labelled with the window it searched (53befda, prod re-test P-2). “What happened recently?” at 12:37 said “last hour (11:00 → 12:00 UTC)” while quoting a 12:35 error: the model labelled the window from the system-prompt clock, which is rounded down to the hour (kept stable for the prompt hash), and the tool’s real bounds sat after results, past the 2,000-character truncation a busy workspace’s result gets. The tool now returns time_range_applied {from, to, label, note} first, the label rendered in the caller’s zone (07 Oct 2026, 11:37 → 12:37 UTC); the prompt names its clock as rounded; and PROMPT_VERSION moved to 2026-10-07.3, which ends cache replays of answers written under the old prompt.
• A stack trace is not the trace index. A question that carries a pasted stack trace (a “stack trace” phrase, Java at pkg.Cls.m(File.java:N) frames, or a Python Traceback (most recent call last)) no longer names the traces family, and “stack/call/back trace” never does. The explain-trace preset — “…replace with your own trace” — used to pin the caller’s trace-analytics-raw-<ws> index; a pin turns the verifier on for a turn that read nothing, so the answer was repaired, different on each asking and never cached.
• “Reply with exactly…” is formatting. The guardrail treats a request for an exact reply wording as a formatting request, not a jailbreak attempt.
• Tool-call markup never reaches the reader. The model sometimes answers in its own tool-call markup (DSML) instead of the standard tool-call fields. When tools were offered, the markup is parsed into real tool calls; otherwise it is stripped, and the stream filter never shows it. A turn that offers no tools (the agent loop’s final pass) ends with a fixed note saying tools are unavailable, so the model answers in prose.

Cortex close-out (2026-10-09)

Unanswered turns are marked

message_end.metadata (and /chat/sync data.metadata) carry answered (default true). When false, unanswered_reason is one of tools_not_run, tools_failed, provider_unavailable, empty. The HTTP status stays 200. Follow-up chips that would only retry are dropped from an unanswered turn.

• “Run that now”, “retry”, “try again”, “go ahead”, “do it” re-run the previous question with tools on, instead of being read as a reference follow-up.
• The 45 s turn clock starts after the first model completion, so one slow first call no longer spends the whole budget.
• Interactive chat never waits on the flex lane (see AI models). A 429, 502 or 503 (or a reset socket) on the standard lane is retried on the same lane, honouring Retry-After, at most 3 attempts and 12 s of waiting in total (ai-service 96cf104b; prod 2026-10-09 had 16 of 72 turns unanswered on 429s before it). Only then does the turn answer from the facts it already measured, led by “The AI step is unavailable right now; these are the measured facts.”, with answered:false, unanswered_reason:"provider_unavailable". The upload-findings model step and the guardrail backstop retry once on 429.
• A turn stops calling tools once it has spent AGENT_TURN_TOTAL_TOKEN_CAP tokens (default 150000) after round 1, and writes its answer.

Search results carried into chat

A chat request may carry context_results: at most 10 items {id, score, line?, time?, text} with text ≤ 300 characters. They are rendered (sanitised, ≤ 4,000 characters) under “Results the user is asking about” and count as grounded facts, so Cortex can answer from them without a tool call. The Sandbox’s search → Continue in chat sends the top 10 hits this way.

Cortex masking of personal data (2026-10-09)

Search rows (search_otel_logs bodies, stack traces and string fields of _source, except @timestamp, line, level, _index) and Warden example values are masked with labels — [SSN], [MRN], [DOB], [EMAIL], [PHONE], [CARD], [IBAN] — before the model sees them. Within one Cortex turn (or one upload) distinct values are numbered and the same value keeps its number — [EMAIL-1], [EMAIL-2], [MRN-1] — so two masked people can still be told apart; the value itself is never shown. The workspace setting cortex.redaction decides when:

Mode	Masks
auto (default)	when HIPAA is in use in the workspace: its row on GET /api/v1/warden/library/frameworks has enabled: true (a policy that is not Off holds it, or a scan profile uses it); an older backend’s inUse is the fallback
on	always
off	never

cortex.redaction.kinds limits which identifiers are masked (default all seven). Both are read per workspace (cached 60 s). If the settings cannot be read the mode is auto; if the frameworks cannot be read under auto, Cortex masks (fail closed). Set them in Settings → Privacy → Cortex answers.

The system prompt also always says: masked values are never guessed or reconstructed; a person’s name, date of birth or diagnosis is not repeated unless asked for; lines are described by the kind of data they hold; writing personal or health data into logs is itself a compliance risk. Cortex links two events only through a shared id or an explicit causal line — never a shared surname or closeness in time — and reads both search_warden_requests and search_compliance_events for a Warden summary.

Uploads

• Rows from an uploaded file carry line (its line number) and time; “which lines …” questions go to the agent, which reads the whole file.
• get_log_analysis finds an analysis by file_name (case-insensitive, basename too); a message naming a dataset index or a file such as hipaa-clinic-app.log resolves to it. Analyses are kept 90 days (LOG_ANALYSIS_RETENTION_MS).
• A search over a whole upload reports time_range_applied.whole_dataset with the file’s first → last time as its span.
• Timestamps without a zone are read as UTC and the summary says so (timeBasis: "assumed_utc"). Upload facts include latency findings (slow queries).
• The Investigation Plan’s “documents matched” counts the pinned index, and labels wildcard totals as “documents matched across <pattern>”.
• New upload indexes are created with index.knn, the logrite-upload-embedding default pipeline and a 384-d text_embedding field, so semantic search on them is real; when the embedding model route is unavailable the index is created plain (keyword only). See Semantic search.

Routing and answers (CX items)

• A follow-up that names a new subject (“Why was the outreach request blocked?”) searches again; only bare anaphora (“why so many?”) or a bare “summarise this” reuse the previous answer.
• “Explain the … finding” is a data question, not a conceptual one.
• Incidents, outages, downtime and alerts are data nouns (DATA_NOUN_WORDS, 2026-10-20): “What are the recent incidents?” matched the conceptual phrase what are with no data noun, so it ran with no tools (one LLM call, the model answering “No tools are available”). It now keeps its tools, and the tool guide tells the model an incident is a burst of errors with a start and an end: get_alert_history, then a 30-day hourly ERROR/FATAL histogram on log-analytics-<ws>-* with service and message sub-aggregations, then the lines just before and after each burst as the likely cause and the fix. An incident that began before a monitor’s watching_since could not raise that monitor’s alert, so the model says the monitor did not exist yet; it does not say nothing watches the service. The words are not error words, so they never become an error-sweep text qualifier.
• opensearch_query histograms report their bursts (2026-10-20, shapeAggs.ts). A histogram used to be cut to its span and its busiest bucket, so a 30-day query showed one incident and hid the rest; a histogram named e.g. per_15m was printed as a “top 20” of its first 20 buckets in time order. A series is now recognised by its ISO bucket keys as well as its name, and carries bursts: runs of consecutive buckets at least 3× the median of the 12 buckets either side (and ≥ 10 docs), the 12 highest peaks, in time order, each with from, last, buckets, count, peak and its terms sub-aggregations summed across the run (top 3). The neighbourhood median, not the series median, because on a strongly diurnal series (the Worksoft demo) every working hour beats the series median; ranked by peak, not total, because a long working-hours swell out-totals a short outage. Measured on prod: ws 303’s 30 days give exactly its 9 seeded incidents; ws 302’s give all 8 recurring ones, the pinned ldap-rotation and three other runs (peaks 159, 179 and 493 an hour; the incidents peak at 326–519).
• New tools: search_coding_events (AI coding sessions in coding-events-<ws>) and get_collector_health (agents’ last check-in, interval, enabled, and which are unhealthy: never seen, silent for 3 intervals, or a non-ok health state). get_scan_key_findings takes project and query instead of a scan id.
• Health questions (“Which collector is unhealthy?”) no longer list indexes. “Most recent / latest” defaults to the last 24 hours.
• A host name such as logrite-prod-vm pins the search to log-analytics-<ws>-logrite-prod-vm when that index exists. A service filter also matches by prefix (care-management → care-management-agent).
• Derived figures (sums, differences, percentages of grounded numbers) count as grounded. The census names its subject (“3 agents”, “log lines”) and is skipped for drafting requests. A widened window (“no rows this week; showing older data”) is always disclosed.
• A question naming a disabled add-on’s product is answered “not enabled for this workspace”, not from general knowledge.
• Answer cache: keyed on the indexes the answer read, with data freshness in 15-minute buckets (CORTEX_CACHE_DATA_BUCKET_MS, default 900000). Self-contained later turns are cacheable; budget-skipped tool calls no longer prevent storing. message_end.metadata.cache_age_s states the age of a cached answer (0 on a miss).

Knowledge /ask

POST /api/v1/knowledge/ask answers only from topics that match the question (signature match, or two distinct non-vendor terms), without the system-prompt framing, and returns {answer, matched, citations, packsUsed, attribution}. No relevant topic → matched:false, answer:"". The Worksoft pack gained topics for Certify Scheduler, Portal and Results Exporter.

Close-out fix wave 2 (2026-10-09)

• context_results on every door. Search results passed as context_results are honoured by /chat, /chat/sync and the streamed chat route alike.
• Upload search and line lookups go to the agent path. Questions that search inside an upload or ask which lines hold something run tools instead of the fixed upload-findings answer.
• Answer cache. A later turn is eligible when it stands alone (not a reference follow-up or summary of earlier answers). On store, the cache writes a source manifest of the index patterns the turn read, and lookups version the answer against those sources only. Repaired and grounded answers are stored too; only a transient or unclassified tool failure blocks the store (budget-skipped calls and argument refusals the model later corrected do not). Each store decision logs one [agent-cache] line.
• Census only for count/list asks. The “I found N …” census replaces an answer only when the question asks to count or list; other questions keep their answer.
• Masking on every model-facing path. In masking workspaces, index summaries and every tool output reaching the model are masked, not only log rows.
• Identifier phrase search. An identifier made of letters and digits joined by a hyphen or underscore (for example BEN-1044) is searched as a phrase and matched exactly on keyword fields, so it is no longer split into words (BEN-1044 matched 633 records before, 74 after, on dev).
• Grouping by outcome. Warden summaries group by outcome, falling back to its keyword sub-field or to verdict on older indexes.
• Embedding fields are never returned. text_embedding and k-NN fields are excluded from tool results.
• Scans by project (AJ-19). get_scan_key_findings with project picks the latest COMPLETED scan with results and notes when a newer scan is still running.

Audit changes (2026-10-10)

Documented from ai-service main at 75472da.

• Aggregation buckets survive. The final-pass summary keeps arrays named groups, buckets or by_*, and any list whose items are all {key, count|doc_count}; only example rows are listed as omitted. The tool-result compactor accepts only numbers as the total (it can no longer print “false hits”) and appends ; by <grouped_by>: key=count, … for the first 8 groups.
• Evidence rows for aggregations. A grouped tool result adds a table <subject> by <grouped_by> (exact, N total) of up to 25 buckets to the response blocks, plus an examples table of up to 10 rows when the result has examples — so /chat/sync results is no longer empty for an aggregation answer.
• Cache gate: no figure, no cache. An answer is not kept in the agent answer cache (not kept: no figure over N records) when the tools counted records (largest positive count/total_hits > 0) and the answer either contains no figure once ISO dates, times and years are removed, or reads as “read but not answered” (e.g. “cannot be reported”, “none were returned”).
• The plan card replays with the answer. A cached answer stores the Investigation Plan card of the turn that produced it; a cache hit shows that card with one extra step, Replayed answer — stored <ISO time>. Answers cached before this change show no card.
• “Answered” reads the lead. The investigation verdict receives the turn’s verify action, and the “bare census” / “read, not answered” patterns test only the answer’s first paragraph (up to 400 characters), so a caveat sentence further down no longer marks a real answer unanswered.
• LangGraph path. Tool messages are matched to results by call id, and fallback messages are appended after the round, as in the default agent loop.
• enable_monitoring reports applied_templates, dropped_templates and auto_seeded_templates from the backend.
• Worksoft pack. A guide for retrying failed runs (CTM > Executions > Errors > Run Failed Test(s) only); “certify scheduler” and “scheduled suite” are recognised.

Cortex fixes (2026-10-14)

Measured on dev through the MCP use-case run (workspace 3103, Worksoft Certify exporter logs) and verified live on ai-service 62ecc8b.

• /chat/sync honours a pinned index. index_name (what the MCP’s cortex_chat indexName sends) or indexName pins the turn: the tools receive it, the system prompt gets a Selected Index section (as V2 has always done), the answer cache is keyed on it, and the scope disclosures see it. A pin naming another workspace’s index is dropped and logged (deny_tenant). A non-string or longer-than-255 pin is 400 INVALID_INDEX_NAME. Live: a pinned exporter index answers “31 errors in that index”, where it used to answer for all 5.6 M errors in the workspace.
• The budget-stop final pass sees evidence, not only counts. Every tool reply is given to the answer-writing pass as a bounded sample — 10 rows per array, 300 characters per field, 8,000 characters per reply, then 3 rows, then counts — and a row’s duplicate raw _source is dropped where the row has its own body/message. Counts and totals are unchanged.
• A many-index search keeps the status code. With more than 3 indexes, search_otel_logs cuts each row’s body at 300 characters (was 200), which is where a .NET exception line names its status. Live: “why are Certify results failing to export to PI?” now answers “401 Unauthorized is the recurring, systemic failure”, where it used to say the error text “was not returned”.
• An empty search suggests the source’s other indexes. When a search returns nothing, an index_expansion suggestion names populated indexes of the same workspace whose names match the searched one (e.g. a live collector index beside a historical upload). It fires only on an empty result.
• A question that names another workspace says so. “Show me the latest logs from workspace 4102”, asked from 3103, now opens “You can only read workspace 3103; workspace 4102 is not reachable with this credential…”. No data ever crossed — the tools fence to the JWT workspace.

Workspace rules in Cortex (2026-10-15)

Business-rules policies with the Cortex chat feature (see Rules & Policies) are enforced on Cortex turns by src/core/safety/workspaceRules.ts and src/core/chatEngine/workspaceRulesTurn.ts.

• Bundle. fetchCortexBundle(workspaceId, projectId?) calls the backend’s GET /internal/policies/resolve?workspaceId=&feature=cortex&projectId= with X-Warden-Internal-Token, normalised like the relay’s binding (policyNames defaults to {}). LRU cache keyed ws:projectId, 60 s positive / 10 s negative. Only /chatv2 sends a projectId; the backend drops a project outside the workspace, so only workspace-wide policies apply then. startLibraryPoller() runs in src/server.ts so framework guidance is live.
• Prompt pre-check (streaming v1/v2 via guardStreaming, and sync), after the existing safety screen: pattern and prompt-injection rules (preEvaluate), framework request compliance, and the AI judge for plain-English rules and frameworks — the judge only when wardenConfig.llmApiKey is set (otherwise [cortex-rules] judge unavailable is logged once and only the deterministic layers run) and WARDEN_JUDGE_ENABLED is not false. Composed most-restrictive. evaluateFrameworks caps those frameworks to notify. The checked text is the message plus the text of its attachments and context_results (2026-10-21, CX-8); before, only the typed message was checked.
  • block → a deterministic answer (model workspace-rule): ”“<name>” blocks this request.” — never the pattern, the judge’s reason or rule ids; those stay in metadata, and judge reasons are masked (2026-10-21, CX-12). Before, the refusal read “Your workspace rule “<name>” blocks this request: <reason>” and could quote the rule’s regex. The name comes from policyNames (p<id>.<key> → policy id; fw:<FW> for frameworks), then the rule name, then the id. The refusal’s metadata carries workspaceRules = {decision:'block', ruleIds} on all three routes (/chat, /chatv2 in message_end, /chat/sync in data.metadata — sync since 3597c04).
  • flag → the turn proceeds; message_end.metadata.workspaceRules = {decision:'flag', ruleIds}.
  • A judge that errors or times out follows the bundle’s failMode, like the relay: with fail_closed the turn is blocked and the refusal names the first plain-English policy (2026-10-21, CX-5); with fail_open it proceeds. Before, Cortex ignored failMode.
• Answer check. When the bundle has an enabled pattern rule or a framework, the answer is buffered and checked (patterns + response-scope compliance, no judge) before emission. The checked text includes the text of table, chart and source blocks, not only the prose (2026-10-21, CX-7). On block the content becomes “This answer was withheld by your workspace rule “<name>”.”, the held blocks are dropped, and no follow-ups, sources or cache entries are written. A withheld /chat/sync turn answers tool_calls: [] and no scope disclosure (CX-19). Plain-English rules are not judged on answers — the relay does not judge responses either (kept).
• Request-level rule types (rate_limit, token_budget, model_allowlist, endpoint_allowlist) do not count as Cortex rules: a bundle holding only those is treated as empty, so no answer is buffered for them (CX-9).
• Every model door is checked (ai-service ea2eb851 and 3d88fe02, 2026-10-21, dev and prod):
  • /chat/sync cache replay goes through the answer check, re-emits workspaceRules and answered, and reports the model that produced the cached answer (stored in query_cache.model); before, a cached answer skipped the rules (CX-1).
  • POST /api/v1/chatv2/compose checks the question plus the report_body text before the model call and the output after it, and runs redactSecrets + humanizeAssistantText; a withheld answer is the standard withheld text (CX-2). It had no rules and no redaction while CORTEX_FUSION_ENABLED (default true) served it.
  • The agent builder (create and run) and code fixes check the prompt before the model call and the output after it; a block answers 403 {"success":false,"error":"“<name>” blocks this request.","workspaceRules":{"decision":"block","ruleIds":[…]}} (CX-3). The handoff summary is not checked: its input is the already-checked transcript.
• Metadata is the same on every door (CX-18): the v1 stream and /chat/sync carry handler, cache and answered like /chatv2; sync keeps its public cache_hit.
• Server-only tool arguments. ToolExecutor drops every _context_* argument the model supplied before injecting the request context, so a context key the request lacks can never be filled in by the model (ai-service d2509163, CX-24).
• Configuration is visible. With WARDEN_INTERNAL_TOKEN or WARDEN_BACKEND_URL unset, workspace rules are off; the service now logs once at startup [workspace-rules] disabled: <VAR> unset; Cortex turns run without workspace rules, and /health reports workspaceRules: "unconfigured" ("configured" otherwise) (CX-13).
• Keeps the last good bundle while the backend is unreachable (2026-10-16), else fails open. Every successful resolve (including a successful “no rules”) is remembered as last-good, outside the TTL cache. When a later resolve fails and the last-good bundle is younger than CORTEX_RULES_MAX_STALE_MS (default 900000 = 15 min), that bundle is used and re-cached for the 10 s negative TTL, logged at most once a minute per key as [cortex-rules] stale bundle for 3:42 (61s old): resolve failed. Before, a backend deploy (~35 s of 502) switched every workspace rule off for its duration. On a cold cache, or once the last-good is older than the limit, it still fails open: no bundle ([cortex-rules] resolve failed, at most once a minute) and no workspace rules apply; a check that throws lets the turn through. The AI Warden relay is different on purpose: on a cold cache it answers 503 when it cannot resolve a binding; since 2026-10-21 it too serves its last good binding for up to WARDEN_BINDING_MAX_STALE_MS (see AI Warden).

Data federation: the pg.wardenRuleSet / pg.wardenRule resources are replaced by pg.policy (fenced on workspace_id, Owner/Maintainer/Admin, settings hidden) and pg.policyRule (fenced through pg.policy, config and text hidden); the SQL safety patterns deny policy_rule, policy_assignment and policy_version.