Turn Trace & Trajectory

Why it exists

Cortex recorded what it answered — message content, response blocks, the Investigation Plan card — and nothing about what it did. Four of the reliability programme’s open defects are “why did it do that” questions, and one (D66) is filed verbatim as “the payload is evidence of WHAT, not of WHY.”

It is also the OMB M-26-14 §5.5 obligation: an agency must log AI system prompts, model outputs, agent decisions, tool calls and model lineage with enough context to tell expected behaviour from an anomaly. Logrite ships an M-26-14 maturity product; its own agent could not produce that record.

Tip

The trace is what found the defects fixed alongside it. Asked “Which service or page is the slowest?”, a real turn spent 14 tool calls, 10 model calls, 647,817 tokens and 38 seconds grepping log text for the word “slow” — and the trace showed it had already run the correct query and had the answer destroyed by a truncation cap. That turn is now 0 tool calls, 0 model calls, 1.7 seconds.

The record

src/core/trace/record.ts is the single source of truth. One append-only record type covering SYSTEM / USER / CONTEXT / ASSISTANT / TOOL.

Field naming. Where the OpenTelemetry GenAI semantic conventions define an attribute, its name is used verbatim as the field path — gen_ai.tool.name is both the OTel attribute and the OpenSearch query path. Everything else is namespaced logrite.* rather than invented under gen_ai.*, so a reader can tell what was borrowed from what was made up. TTFT and cached input tokens are logrite.* on purpose: neither is a stable span attribute upstream.

Group	Fields
Identity	turnId (is the UsageContext.correlationId, so a trace joins usage_event on cost), messageId, gen_ai.conversation.id, workspaceId, userId
Ordering	turn, step, parentStep, depth
Kind	source, recordType, gen_ai.operation.name (invoke_agent / chat / execute_tool only)
Timing	startedAtMs, endedAtMs, offsetMs, durationMs, timingSource
LLM call	gen_ai.request.model, gen_ai.response.model, gen_ai.usage.{input,output}_tokens, logrite.llm.{ttft_ms,generation_ms,throughput_tok_s,escalation_tier,byo,streamed,provider_cost_usd}
Tool call	gen_ai.tool.name, gen_ai.tool.call.id, logrite.tool.{arguments,result,result_bytes,result_truncated,result_ref}
Routing	logrite.route.branch, logrite.route.intent_{classified,effective}, logrite.scope.*
Turn summary	logrite.turn.{total_ms,iterations,tools_executed,llm_calls,total_tokens,model}, logrite.turn.retrieval.*

recordType ∈ turn_start · user_message · system_prompt · context_inject · route_decision · plan · llm_call · tool_call · turn_end · error.

What the records refuse to claim

These are contracts, not details — each one is pinned by a test.

• step is assigned when a record OPENS, so a tool that finishes late cannot reorder the list.
• A routing decision carries timingSource: 'absent', not a zero duration. “Took no time” and “has no duration” are different claims and only one is true.
• A non-streamed model call carries ttft_ms: null, never 0 — a zero reads as an instant first token.
• A turn that hit the record ceiling is flagged truncated rather than being quietly short.
• A tool result is stored as a redacted preview plus the TRUE byte size and a result_ref into tool_executions. Storing the capped copy twice would make the trace larger than the data it describes.
• The evidence envelope is absent when a turn read nothing — which is not the same as reading and finding nothing, so it is never defaulted to a zero.
• A decision is filed before the work it selected. A deterministic branch emits the turn’s closing record from inside itself, so claimed() reserves its step before running the branch. A declined branch leaves a gap in the numbering — a real turn starts at step 18, and those seventeen gaps are the branches that were evaluated and passed over.

The wire

One event, trace_step, whose payload is the record plus phase: 'start' | 'end'. Not one event per record type: the browser’s SSE reader is a switch with a warning default:, so every extra name is another chance to add the next one without declaring it — which is exactly how debug shipped as as any for months. Both names are now on SSEEventType.

The wire payload and the stored document are the same object. There is no translation step between the live view and the historical view, which is the only way to guarantee they cannot drift.

Opt-in per request — ?trace=1 or X-Cortex-Trace: 1, mirroring the existing debug gate. Every browser already in the field predates trace_step and none of them sends the header, so none of them sees the event. The trace is built either way.

Where it hooks

Site	File	Why there
Turn open	core/chatEngine/agentRun.ts traced()	the one point all three entry paths pass
Routing branch	core/trace/claim.ts wrapping 17 cascade lines	a free function, not a method: streamV2.ts sits under a 300-line ratchet with an empty suppression list
Tool call	core/chatEngine/v2/run.ts	beside the existing tool_use_* emits, so one code path produces both wire shapes
LLM call	core/metering/MeteredProvider.ts	the factory wraps the OUTSIDE of the fallback chain — one logical call, one record — and TTFT has one definition here and four in the adapters
Turn close	core/trace/turnEndHook.ts	composed beside withEvidence, whose header already argued it: six emitters of message_end, and decorating each guarantees the seventh ships without one

Caution

metered() is a no-op when metering is off. The wrapping decision now widens to meteringEnabled() || cortexTraceEnabled, so the billing kill switch does not silently take the trace with it. meter() itself stays behind meteringEnabled.

Storage

cortex-traces-{workspaceId}, written directly from ai-service — the same pattern services/uploads/register.ts already uses for log-uploads-{workspaceId}. The Warden relay’s hop through the Java backend exists because the relay holds no OpenSearch credentials; ai-service does.

• dynamic: false, which matters more here than for uploads: a tool’s arguments are written by a model, so an index that maps on demand has its mapping authored by the LLM.
• Doc id {turnId}:{step} zero-padded — a retried bulk overwrites instead of duplicating, and a prefix scan returns execution order with no sort.
• A bulk 200 carrying errors: true counts as a failure. Treating a partial write as a success is how a store reports coverage it does not have.
• Every call goes through osAxiosConfig and nothing else. Skipping it is not a degraded write but an invisible one — see the AI Warden’s own incident, where writes were rejected by Node before any HTTP response and the product reported 0 requests while serving traffic.

Caution

IndexNames.WS_FAMILIES must contain cortex-traces (logwrite-api/.../util/IndexNames.java). That file records two prior incidents where a missing family made extractWorkspaceSegment return null, the index authorizer fall through to allow, and one workspace read another’s.

Retention is 90 days by ISM policy: a turn trace is an operational artifact, not a legal record.

API — /api/v1/trajectory

Method	Path	Purpose
GET	/turns	one row per turn, newest first; filter by conversationId, q, from, size
GET	/turns/:turnId	every record of one turn, in step order
GET	/messages/:messageId	the trace for an answer — what the UI asks for on a reloaded turn
GET	/turns/:turnId/export	NDJSON, byte-identical to the UI’s Export button
GET	/tool-schemas	the tool schemas, served live rather than copied onto every record
GET	/health	{enabled, storing, written, failed, lastError, lastWriteAt}

JWT + workspace claim. Tenant scoping is the index name and nothing else — derived from the verified token, never from a path, query or body. A foreign turnId and a nonexistent one return the same empty result, because a 403 that differs from a 404 tells a caller which ids are real. A missing index is an empty page; a TLS or 5xx fault is a 502 and is never laundered into one.

No role gate, with one exception: the system prompt is kept to a sha256 by default (CORTEX_TRACE_PROMPTS=hash). This service’s own safety patterns treat “exfiltration (system prompt)” as a jailbreak category, so shipping it to a browser would defeat a guard it already enforces.

The Trajectory view

A Chat | Trajectory pair in the chat header, on all Cortex surfaces, carrying a live step count — the moment a reader most wants it is while a turn is still running.

• A 3-lane strip (Input / Model / Tools) positioned by time across the run.
• The record stream, one row each, with a source badge and turn markers in the gutter.
• An inspector whose tabs depend on what the record is: a routing decision has no payload and a tool has no throughput, and an empty tab invites the reader to wonder what they are missing.
• Duration / Turns / Calls groupings, search, and NDJSON Export.

One adapter (hooks/useTrajectory/trajectoryAdapter.js) serves all three sources — streaming, stored, and degraded-from-an-older-service — and the parity test asserting that live and stored produce an identical record list was written before any component.

CSS-positioned bars, not ECharts: the record list changes many times a second while streaming and the shared wrapper memoises on a full JSON.stringify of its option; that wrapper refuses to resize at zero dimensions; and selection syncs both ways with the list.

The transcript is hidden, never unmounted when the Trajectory tab is open — unmounting throws away scrollTop. useStickToBottom carries the matching guard, because at display:none every dimension is 0 and distanceFromBottom reads as pinned.

Environment

Variable	Default	Meaning
CORTEX_TRACE_ENABLED	true	build records at all
CORTEX_TRACE_SSE_ENABLED	true	allow a caller to opt in to the stream
CORTEX_TRACE_STORE_ENABLED	false	write finished turns to OpenSearch
CORTEX_TRACE_INDEX_PREFIX	cortex-traces-	index family
CORTEX_TRACE_PROMPTS	hash	hash / preview / full
CORTEX_TRACE_TOOL_RESULT_CHARS	2000	preview kept on a record
CORTEX_TRACE_MAX_RECORDS_PER_TURN	400	ceiling; a turn that hits it is flagged
CORTEX_TRACE_RETENTION_DAYS	90	must agree with the ISM policy