Skip to content

Demo Seeder

The demo seeder writes and keeps up the synthetic data in a demo workspace: collector agents and their logs, AI Warden captures, compliance events and agent findings. It is built from the logrite-ai-service image (Dockerfile.seeder, src/demo/), like the Warden relay and the ingest shim, and listens on port 9900.

InstanceWhereWorkspaceSEED_PERSONAS
demo-seederdev, deployed by the console3103 ([email protected])unset (all seven)
seeder-carevaultops/demo/seeders/compose.yml on the dev host, host port 127.0.0.1:9911prod healthcare demo workspacecarevault
seeder-worksoftsame compose project, host port 127.0.0.1:9912prod Worksoft demo workspaceworksoft

The two prod-facing instances are operator-managed (compose project logrite-demo-seeders, not in services.json); ops/demo/seeders/up.sh resolves the pinned demo-seeder image and starts them.

Each persona is one vertical with its own frameworks, collector fleet, traffic shape and hand-written prompts and log lines. Fixtures live at src/demo/fixtures/<key>.json.

KeyCompanyAgentsFrameworks
carevaultCareVault Health (healthcare / virtual care)6HIPAA · SOC2 · PCI · CCPA · AI-Data-Gov · OWASP-LLM · GEN · OPS
ledgerpayLedgerPay Financial (fintech)5PCI · SOC2 · GLBA · SOX · DORA · OPS
clouddeskClouddesk Software (B2B SaaS)4SOC2 · GDPR · CCPA · ISO-27001 · ISO-27701 · OWASP-LLM · GEN · OPS
harborlight-shopfrontHarborlight Commerce (e-commerce)5PCI · GDPR · CCPA · OWASP-LLM · GEN · OPS
federal-demoBureau of Federal Program Integrity5M-26-14 · FedRAMP · CMMC · NIST-AI-RMF · CJIS · AI-Data-Gov · OWASP-LLM · SOC2 · GEN · OPS
ai-native-demoQuillon AI (LLM product)4OWASP-LLM · AI-Data-Gov · EU-AI-Act · ISO-42001 · NIST-AI-RMF · NYC-LL144 · Colorado-AI · SOC2 · GDPR · FERPA · COPPA · OPS
worksoftWorksoft (demo estate) (test automation)8SOC2 · AI-Data-Gov

Every collector agent carries a customer-facing description, sent at provision time. Verdict mixes are allow-heavy on purpose (93–96 % ALLOW per persona), so the data reads like a real tenant while the Blocked filter is never empty.

worksoft backs the Worksoft demo workspace (and joins ws 3103 on dev). It has:

  • eight file agents, one per Certify service (AuthenticationService, Bookmark, CertifyResultsExporter, CertifyResultsViewer, Cryptography, GenDoc, Portal, Scheduler), on hosts WS-CERT-APP01 … WS-CERT-SCH01; each agent ships only its own service’s lines;
  • 67 log lines generated from the Worksoft signature catalogue (src/knowledge/packs/worksoft.signatures.json); 64 match matchWorksoftSignature, 25 of them annotated signatures (Cryptography’s only mined template is a bare IP, so its 3 lines match none);
  • 14 Warden captures (test-authoring prompts, one PII-leak BLOCK, one secret NOTIFY) and 7 findings;
  • time zone UTC−6 and no raw prompt capture.

The lines are rebuilt from templates by scripts/generate-worksoft-persona.js (--check verifies the committed fixture). No line of the real Worksoft corpus is copied: placeholders get RFC 2606/5737 values, stack frames are cut, and any template naming a real user, tenant, database or domain is excluded. A test asserts none of those identifiers appears.

Since ai-service be6e0be (prod ai-service/warden/ingest-shim 2cbcb65b81cb) the worksoft persona tells one story: a Certify estate whose nightly Continuous Testing Manager (CTM) regression breaks because an admin rotated the LDAP bind account svc-certify-ldap without updating Portal.

CTM results go to log-analytics-<ws>-worksoft-ctm-results (dynamic mapping; strings are text with .keyword, durationSec is long). Six suites (O2C, P2P, R2R, L2O, H2R nightly on CERT-01 … CERT-06, plus a weekday Azure DevOps CI smoke three times a day). One document per execution: @timestamp (end), startedAt, executionId (EX-<yyyymmdd>-<suite>-<seq>), ctmRunId (CTM-<yyyymmdd>-<suite>, CI CTM-<yyyymmdd>-CI-<n>), suite, suiteCode, process, application, environment (QA1), machine, result (Passed / Failed / Aborted), durationSec, stepsTotal, stepsFailed, failedStep, errorCategory, errorMessage (the last three null when Passed), triggeredBy, a readable message, level, serviceName: ContinuousTestingManager, service: ctm, host: WS-CTM-01, and the demo markers.

Compliance logs go to compliance-logs-<ws>, the index the MCP compliance_logs tool reads. The seeder creates it when absent (HEAD, then PUT) with the mcp-server INDEX_MAPPING plus demo (boolean) and demoPersona/demoSeed/scenario (keyword); 1 shard, 0 replicas. Fields: timestamp (yyyy-MM-dd HH:mm:ss, UTC), date, level, service, method, param_name, param_value, compliance_type (SOC2-AUDIT), control_id, description, raw_message, workspaceId. 28 routine SOC 2 entries a day (CC6.1, CC6.2, CC6.3, CC6.6, CC7.1, CC7.2, CC8.1).

The incident (scenario: "ldap-rotation", 170 minutes from SEED_INCIDENT_AT):

  • 10 minutes before the window, a CC6.2 WARN records the svc-certify-ldap password change (CHG0042117): the root cause is in the evidence.
  • In the window: worksoft-auth logs The supplied credential is invalid (4/min, 680 lines), bookmark User not found in Certify (170), results-viewer Failed to get step info data (85), results-exporter exporting results to PI … (Unauthorized) (34); CC7.2 and CC6.6 entries repeat hourly; Portal-rooted traces error at 0.7 (plus one extra Portal trace every 5 minutes); every 20th auth error line becomes a severity: high finding (34).
  • CTM executions starting in the window fail at Log on to SAP Fiori Launchpad with Certify login failed: LDAP bind error (The supplied credential is invalid). Measured: nightly pass rate ~95% on a normal day (0.925–0.97 over a week), ~28% in the window. With the incident at 01:40 Chicago every nightly suite starts inside the window (134 of 134 executions).
  • At the window’s end, LdapConfigurations BindUserName … configuration reloaded (auth) and LDAP configuration updated by [email protected] (portal) close it.

Windows hosts. Each worksoft agent has os (Windows Server 2022 Datacenter, kernel 10.0.20348): system_metrics report it, and collector docs carry filePath: C:\Worksoft\<Service>\Logs\<Service>.log. Linux personas are byte-identical to before.

Placing the incident. SEED_INCIDENT_AT (ISO-8601 with a zone; empty means no incident) is set on seeder-worksoft in ops/demo/seeders/compose.yml from WORKSOFT_INCIDENT_AT in ops/demo/seeders/.env. For last night’s 01:40 Chicago:

Terminal window
date -u -d "$(TZ=America/Chicago date +%F) 01:40 CDT" +%FT%TZ # CST after 2026-11-01

Then re-create the seeder (ops/demo/seeders/up.sh) and POST 127.0.0.1:9912/admin/backfill. The backfill first purges the old incident by its tag (purgeScenarioDocs: _delete_by_query term scenario on log-analytics-<ws>-*, agent-flags-<ws>, compliance-logs-<ws> and trace-analytics-raw-<ws>), so moving the date leaves nothing behind. /admin/status reports ctmResults and complianceLogs; /admin/purge covers both new indexes.

Live. Each tick also writes the CTM and compliance slices for the tick window and live findings (from that tick’s collector docs), so agent-flags-<ws> no longer goes stale. Incident lines shipped live carry scenario.

Since ai-service f1e05e4 the carevault and worksoft personas carry recurringIncidents (RecurringIncidentSpec, src/demo/storySpec.ts): four incident types each, each every 14 days on its own phase, so any 30-day window holds 8–9 dated incidents per account, never more than 5 days apart. The schedule is a pure function of the calendar (recurringWindows, src/demo/generate/incident.ts): type N opens at startLocal in the persona’s time zone on every local epoch day d with d % everyDays === phaseDays. The backfill and every live tick agree on where each occurrence is, so nothing needs maintaining.

KeyAccountPhaseLocal startMinutesBurstFinding
cv-rpm-broker-downCareVault (UTC−5)002:2095mqtt subscribe failed 18/min, Back-off restarting failed container 2/minops_svc_k8s_backoff, every 5th Back-off line
cv-fhir-partner-authCareVault409:05130Bundle POST rejected … status=401 24/min, token refresh returned 401 2/min—
cv-claims-db-connectionsCareVault713:4070FATAL: sorry, too many clients already 24/min, portal upstream timed out 6/minops_db_too_many_conn, every 20th
cv-edge-cert-expiredCareVault1106:10150TLS handshake aborted: certificate has expired 20/min, IPsec SA rekey failed 1/minops_net_cert_expired, every 20th
ws-scheduler-loop-crashWorksoft (UTC−6)200:30110Runtime error occurred in main trigger firing loop 6/min, webhook BadRequest 1/minsoc2_system_operations, every 10th
ws-ldap-dc-unavailableWorksoft622:1555655:The LDAP server is unavailable 8/min, bookmark Token is not valid 1/minsoc2_authentication, every 20th
ws-certify-db-processingWorksoft907:4580Error during Processing Certify DB 6/min, viewer Failed to get log header recordset data 2/min—
ws-viewer-baseaddressWorksoft1310:2065An invalid request URI was provided 6/min, Get domainList error 2/min—

The rates are set so each burst’s hour is at least 5× the hours around it. Three CareVault types sit in its busiest hours (300–400 ERROR/FATAL an hour on a full-rate day); at the first rates (8, 12 + 3 and 10 a minute) five of their seven occurrences measured only 3.5–4.3× on prod, so the rates were raised in b10573e. Simulated at the full everyday rate, every occurrence is now ≥5.7× (CareVault) and ≥9× (Worksoft).

Each occurrence writes, on the agents it names:

  • the cause — precursorLines, leadMinutes before the window (a node drain, an expiring client secret, a release scaling denials-worker 6 → 24, an expiring certificate, an appsettings reload, DC patching, a schema upgrade, an upgrade package with an empty authority);
  • the burst — lines, paced at perMinute across the window like the pinned incident;
  • the fix — recoveryLines at the window’s end, 15 s apart;
  • findings for the types with finding: every every-th occurrence of that line, chosen by the sequence number in its id, so backfill and live pick the same lines.

Every document carries scenario: <key>; ids are <agentId>:inc:<seedVersion>:<key>:<startMs>:<seq>. Each type draws from its own rng stream, so the everyday corpus and the everyday findings are byte-identical with or without the incidents (tests/unit/demoSeederIncidents.test.ts). Worksoft’s burst lines are catalogue signatures of the agent’s own service. Traces, CTM, SOC 2 compliance logs and Warden have no recurring overlay; only the pinned ldap-rotation incident drives those.

Monitors. Each type has one OpenSearch query-level monitor (src/demo/monitors.ts) on its agent’s log-analytics-<ws>-<agent> index: every 5 minutes, match_phrase on message over the last 10 minutes, trigger when the count exceeds threshold (half the 10-minute burst; the everyday rate of the same line stays far below it, which the test checks minute by minute over a whole day). No actions, so nothing is notified. upsertMonitor matches an existing monitor by exact name and its first index and refuses an index outside the workspace.

Since ai-service 13380c2 (2026-10-20), the upsert leaves a monitor alone when its stored copy already matches the body (monitorShape). The comparison allows for how the plugin stores a body: match_phrase becomes {query, slop, …} and gte/lte become from/to. When the monitor does change, the replacement keeps each trigger’s id, matched by trigger name. Before, every backfill PUT the monitors without trigger ids. The plugin then minted new ids and moved any alert open on the old trigger to DELETED; the ids changed between the 08:25 and 09:14 runs on 2026-10-20. Measured after the fix: a backfill reports monitors: {created: 0, updated: 0, unchanged: 4} on each prod seeder, and all 8 trigger ids are the same before and after.

No alerts for history. The plugin stamps an alert’s start_time when the monitor runs, so a backdated alert cannot be created truthfully. Backfilled occurrences have their logs, findings and monitor but no alert. Cortex explains the gap with each monitor’s watching_since (its enabled_time, which survives a PUT): an occurrence before it could not raise that monitor’s alert (AI service).

Writing them. The full backfill purges every recurring key (with the pinned one) and upserts the monitors before writing. A monitor failure is recorded in backfill.monitors.error and never fails the data. Two partial backfills (src/demo/partialBackfill.ts) rewrite one slice over SEED_BACKFILL_DAYS and leave the everyday corpus alone, which is about half a million documents for CareVault:

  • POST /admin/backfill-incidents purges each recurring key, upserts the monitors, then writes the incidents’ lines and findings.
  • POST /admin/backfill-findings writes every seeded finding (everyday, the pinned incident, recurring) under its deterministic id. It does no purge, no monitor work and no line writes. It is the repair for seeded findings that something else overwrote.
Terminal window
set -a; . ops/demo/seeders/.env; set +a
curl -s -X POST -H "Authorization: Bearer $CAREVAULT_SEEDER_ADMIN_TOKEN" 127.0.0.1:9911/admin/backfill-incidents
curl -s -X POST -H "Authorization: Bearer $WORKSOFT_SEEDER_ADMIN_TOKEN" 127.0.0.1:9912/admin/backfill-incidents
curl -s 127.0.0.1:9911/health # backfill.kind "incidents", running false, monitors {created, updated, unchanged}
# after anything overwrote seeded findings: the same with /admin/backfill-findings

Dev’s demo-seeder (ws 3103) gets the same incidents from live ticks only; no monitors are created there unless it is backfilled.

Measured on prod, 2026-10-20 (ai-service/warden/ingest-shim 9f5154f, seeders on b10573e):

Checkws 303 (CareVault)ws 302 (Worksoft)
Occurrence days in 30 days9: 09-21, 09-24, 09-28, 10-01, 10-05, 10-08, 10-12, 10-15, 10-198: 09-23, 09-26, 10-01, 10-03, 10-07, 10-10, 10-15, 10-17 (ws-ldap-dc-unavailable lands on the next UTC day, 04:15Z)
Seeded findings with scenario694 (edge 450, claims 168, rpm 76)176 (scheduler 132, ldap-dc 44)
Monitors4, each once; 10 minutes before an occurrence’s last burst line is 2× its threshold, 1 hour before the start and after the end 0–1same
“What are the recent incidents?”tools run (get_alert_history, opensearch_query); all 9 named by date, service and dominant error; the ACTIVE “PHI in CareVault agent logs” alert listedall 8 named, plus the pinned ldap-rotation; only ws 302’s alerts listed

Measured again on prod, 2026-10-20 13:00Z (ai-service/warden/ingest-shim 13380c2, analyzer 54ec836, seeders on 13380c2):

Checkws 303 (CareVault)ws 302 (Worksoft)
get_alerting_status channels“CareVault demo alerts” only (7 monitors)“Worksoft demo alerts” only (6 monitors)
Monitor dry run (_execute?dryrun=true&period_end=<ms>ms)all 4 trigger at the end of their latest burst and stay quiet 1 hour before its start; no alert persistedsame
Analyzer findings on seeded agents after the skip0 (384 in the hour before it)0
“What are the recent incidents?”, fresh runsall 9 named in 5 of 7 answers; 2 more were withheld by the workspace rule “Care-Team Copilot Safety”all 8 named in 5 of 8

The answers that named fewer occurrences searched the last 7 days or one service’s index instead of 30 days across log-analytics-<ws>-*. No answer said that no monitor was configured or that nothing was watching. Answers that mentioned the gap used watching_since, for example “your monitors only started watching on 19 Oct 08:23 GMT, so those two incidents happened before anything was evaluating them”.

A comma list of persona keys from the table above. Empty means all seven. An unknown or malformed key stops the boot with the list of known keys. Relay key ids stay stable whatever subset is selected.

FileContent
SEED_STATE_PATH (default /app/.demo-state/agents.json)provisioned agents and their tokens, relay keys {persona, id, name, key}, provisionedAt
run.json, beside it{live: {enabled, lastTickAt, ticks}}

Bind /app/.demo-state to a host directory owned by 1000:1000 (dev: /var/lib/logrite-data/demo-seeder/state; prod instances: /var/lib/logrite-data/demo-seeder-<persona>/state). On boot the seeder reads run.json and resumes the live stream when it was enabled, so a container recreate needs no admin call.

Terminal window
curl -s localhost:9900/health | python3 -m json.tool
FieldMeaning
workspaceId, seedVersionfrom env
personasthe persona keys this instance owns
statePath, stateFoundwhere state is read from, and whether a state file exists
provisionedAt, provisionedlast provision time and agent count
relayKeysrelay keys held in state
backfillrunning, progress, kind (full, incidents or findings), monitors (created, updated, unchanged, error) and error when the last backfill failed
liveenabled, lastTickAt, ticks, and stale (enabled but no tick for 10 minutes)

After a recreate, expect stateFound: true, live.enabled: true and live.ticks rising.

Host-only: no FQDN, no tunnel route. Every /admin call needs Authorization: Bearer $SEEDER_ADMIN_TOKEN; with the token unset the routes fail closed.

Terminal window
T="Authorization: Bearer $(cat ~/.demo-seeder-token)"
curl -s -X POST localhost:9900/admin/provision -H "$T" -H 'Content-Type: application/json' \
-d "{\"workspaceJwt\":\"$JWT\"}"
curl -s -X POST localhost:9900/admin/backfill -H "$T" # 202; poll /health until backfill.running=false
curl -s -X POST localhost:9900/admin/backfill-incidents -H "$T" # recurring incidents + monitors only
curl -s -X POST localhost:9900/admin/backfill-findings -H "$T" # every seeded finding, nothing else
curl -s -X POST localhost:9900/admin/live/start -H "$T"
curl -s localhost:9900/admin/status -H "$T" # document counts
  • Provision takes a workspace JWT in the body, uses it and stores nothing. It is create-if-absent-else-rotate: an existing agent’s token is rotated, so a lost state file costs one re-provision. resetRuleSets defaults to true and deletes every policy in the workspace first; pass false to keep them.
  • Database agents are created in connector mode (dbConnection.connectorMode: true, no host or credentials stored) and their token is always rotated. An older pull-mode DB agent (rotate answers 409) is deleted without purging its data and re-created under the same key with a new agent id.
  • Backfill refuses to start, and aborts, when OpenSearch free disk is low. The check fails closed: if node stats cannot be read, the backfill stops and /health backfill.error says why. The OpenSearch user needs cluster:monitor/nodes/stats.

Provision gets or creates the workspace’s neuralwatt provider credential (created from SEED_RELAY_PROVIDER_API_KEY when none exists) and mints one Warden relay key per persona (no ruleSetId since 2026-10-15). An existing key with the same name is re-attached; its plaintext is reused from state, or the key is rotated if the plaintext was lost. Backfilled and live captures carry the real relay key ids. With no credential and no env var, provision adds a warning and mints no key. The seeder sends no relay traffic itself.

Fixture rule sets become policies (2026-10-15). Fixture JSON keeps ruleSets[], but each entry is created as a business_rules policy through POST /api/v1/policies (src/demo/ruleSets.ts): appliesTo maps relay → ai_traffic, logs → logs, coding → ai_coding; the persona’s keys and agents become relayKeyIds / collectorIds; nlPolicies become ai_judge items and frameworks become framework items; settings {mode:'inline_blocking', failMode:'fail_open', scanScopes:['request','response'], deterministicRules:false}. See Rules & Policies.

FamilyBackfill (SEED_BACKFILL_DAYS, default 60)Live stream
Collector logs, system_metricsOpenSearch _bulkthe ingest shim POST /ingest, with each agent’s token
Warden captures, compliance events_bulk_bulk
Agent findings_bulk, backdated flaggedAtproduced by the analyzer’s scan, which the shim triggers after each batch

Every document carries {"demo": true, "demoPersona": "<key>", "demoSeed": "<version>"}. Purges refuse any _delete_by_query without the demo: true term or outside the workspace. Documents are a pure function of (SEED_VERSION, persona, date, seq), so a re-run of the backfill upserts rather than duplicates.

PORT_DEMO_SEEDER (9900) · SEED_WORKSPACE_ID · SEED_PERSONAS · SEED_STATE_PATH · SEED_RELAY_PROVIDER_API_KEY · SEED_VERSION · SEED_INCIDENT_AT · SEED_BACKFILL_DAYS (60) · SEED_COLLECTOR_SCALE · SEED_WARDEN_SCALE · SEED_TAPER_FULL_DAYS (7) · SEED_TAPER_FLOOR (0.08) · SEED_TICK_MS (60000) · SEED_MIN_FREE_BYTES_START / _ABORT · SEEDER_ADMIN_TOKEN · OPENSEARCH_URL / _USERNAME / _PASSWORD / _INSECURE_TLS · LOGRITE_BACKEND_URL · COLLECTOR_SHIM_URL.

WARDEN_INTERNAL_TOKEN is no longer read by the seeder; remove it from the seeder’s env.

Enforced by the seeder tests:

  • RFC 2606 names only (.example, .invalid, .test, example.com and similar).
  • RFC 5737 / RFC 1918 addresses only.
  • Published test card numbers only (4111111111111111, 4242424242424242, 5555555555554444), never a random Luhn-valid number.
  • Live ticks append. Row and request ids are keyed on the UTC calendar day (d:<seedVersion>:<persona>:<agent>:<YYYY-MM-DD>:<seq>), so a tick on a new day writes new rows instead of overwriting the same relative day. Each tick sends only documents with since < @timestamp ≤ now; lastTickMs is restored from run.json on boot. No generated document is later than now. Both seeders run with SEED_VERSION: v2.
  • BLOCKs every day. Fixture field wardenTraffic.minDailyBlocks (worksoft 2, carevault 3) forces that many BLOCK captures spread over each day.
  • Re-provision adopts policies. provision with resetRuleSets: false reuses a policy that already exists by name instead of creating a duplicate.
  • Product redaction. Seeded previews and snippets use the shared redaction (phone, dob, mrn, iban, …), not a local email-only mask.
  • /health relayKeysMissing lists personas without a relay key. The worksoft seeder needs WORKSOFT_RELAY_PROVIDER_API_KEY (compose maps it to SEED_RELAY_PROVIDER_API_KEY).
  • Worksoft traces. A fixture traceTopology (Portal → Results API → Certify DB; Scheduler → Certify Execution → Certify DB) generates Data Prepper raw-span documents in trace-analytics-raw-<ws>, in backfill and live; purge covers that index.
  • CareVault gains the business-rules policy (id 20 on prod ws 303) “CareVault Health · Member Services — PHI & Copilot Safety”, which blocks member-supplied identifiers (SSN, MBI, member numbers, full date of birth, card/bank numbers).

Re-seeding order after an upgrade: POST /admin/live/stop → /admin/purge → /admin/provision (resetRuleSets: false) → /admin/backfill (poll /admin/status) → /admin/live/start.

  • Seeded captures carry latency. Every seeded Warden capture writes latency.wardenMs, so “Avg added latency” is measured on demo workspaces (ai-service ccec420).
  • CareVault “No clinical decision-making”. The policy still blocks a diagnosis, a treatment choice or a medication/dose change for a specific person, but drafting a care plan, goals, a risk summary, outreach or a follow-up for a clinician or care team to review and sign off is documentation and is allowed, as long as the draft does not itself choose a new diagnosis, medication or dose. The fixture and prod ws 303 policy 10 carry the same text.