Troubleshooting

Port Conflicts

# Who holds a port. `sudo` is required: the listener is a root-owned
# docker-proxy, and without it ss shows the port as held by nobody.
sudo ss -ltnp | grep :9300

# Which container publishes it
docker ps --format '{{.Names}}\t{{.Ports}}' | grep 9300

Danger

Never free a port by killing what holds it. This page used to say lsof -ti:9300 | xargs kill -9. On a containerised estate the listener is docker-proxy, so that kills the port mapping of a container that is running perfectly — the service stays “up” and becomes unreachable, which is a harder failure to diagnose than the one you started with. Nothing runs on the host any more.

If a port is genuinely held by something stale, find the container and recreate it through the console. A tunnel that 502s is usually a missing port mapping, not a dead service — check docker ps before anything else.

Container Issues

A service will not start:

# The container, its state and its health
docker ps -a --format '{{.Names}}\t{{.Status}}' | grep <service>

# Why it exited — the last 50 lines
docker logs --tail 50 <container>

A service is “up” but not serving:

# Up and unhealthy are different answers, and so is up with no healthcheck
docker inspect --format '{{.State.Status}} {{.State.Health.Status}}' <container>

Caution

Grep logs with a WINDOW, never a total. docker logs <c> | grep -c error cannot distinguish recovered from still broken — a burst of connection errors during a restart is the pool evicting stale connections, which is correct. Use docker logs --since 60s and ask whether it is still happening.

Restart or redeploy:

Use the console — management.logrite.ai — so the action is recorded, logged and announced. A deploy from there also moves the pin, which is what makes a rollback possible afterwards.

Caution

A prod container is not named for its service. It keeps the uuid it was adopted under, so docker ps | grep logrite-redis returns nothing for a service that is running fine. Match on the uuid prefix, or ask GET /api/services?host=prod.

OpenSearch Issues

OpenSearch won’t start:

# Check if running as root (must use opensearch user)
ps aux | grep opensearch

# Check logs
opensearch-mgr logs opensearch

# Verify Java 21 is being used
su -s /bin/bash opensearch -c "java --version"

# Check if port 9200 is in use
lsof -i:9200

OpenSearch red health:

# Check cluster health details
curl -s http://localhost:9200/_cluster/health | jq .

# Check for unassigned shards
curl -s "http://localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason"

# For single-node: set all replicas to 0
curl -X PUT "http://localhost:9200/_settings" \
  -H "Content-Type: application/json" \
  -d '{"index.number_of_replicas": 0}'

ML model not loading:

# Check model status
opensearch-mgr semantic-status

# Force reload
opensearch-mgr model-ensure

# Check ML node status
curl -s "http://localhost:9200/_plugins/_ml/stats" | jq .

Embedding model in DEPLOY_FAILED state — torch op mismatch

If /api/v1/models/deployed on the Semantic Search API returns "No deployed embedding model found" and the model record shows model_state: DEPLOY_FAILED with an error like Unknown builtin op: aten::scaled_dot_product_attention, the registered TorchScript artifact uses a PyTorch op newer than the libtorch shipped with our OpenSearch ML Commons (first seen on OS 2.18/2.19; both estates now run opensearchproject/opensearch:3.3.1). The user-visible symptom is the UI banner “Semantic search service is unavailable — Cannot connect to API at https://search.logrite.ai”.

The service’s own auto-recovery (embeddingService.ensureModelDeployed) cannot fix this on its own: as of the 2026-05-29 fix the code pins version: "1.0.1", but a stale build or a manual registration can still re-introduce 1.0.2. Recover manually by registering and deploying the op-compatible version.

Auth: _ml/models/_register, _deploy, _undeploy, and DELETE are cluster:admin/opensearch/ml/* actions. The app user logrite-app is intentionally denied these and returns 403. Use the admin user (password = the OpenSearch container’s OPENSEARCH_INITIAL_ADMIN_PASSWORD env). Read-only _search/_predict work for logrite-app too.

OS='https://analyticsapi.logrite.ai'
AUTH='-u admin:<OPENSEARCH_INITIAL_ADMIN_PASSWORD>'

# 1. Inspect the broken model record — note its id AND model_group_id
curl -sk $AUTH "$OS/_plugins/_ml/models/_search" \
  -H 'Content-Type: application/json' \
  -d '{"query":{"bool":{"must_not":[{"exists":{"field":"chunk_number"}}]}},"_source":["name","model_state","model_version","model_group_id"]}' \
  | jq '.hits.hits[] | {id:._id, src:._source}'

# 2. Register all-MiniLM-L6-v2 at v1.0.1 in the SAME model group as the broken
#    record (reuse its model_group_id; or omit the field to auto-create one —
#    model_access_control is disabled).
curl -sk $AUTH -X POST "$OS/_plugins/_ml/models/_register" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "huggingface/sentence-transformers/all-MiniLM-L6-v2",
    "version": "1.0.1",
    "model_group_id": "<MODEL_GROUP_ID_FROM_STEP_1>",
    "model_format": "TORCH_SCRIPT"
  }'
# → poll _plugins/_ml/tasks/<task_id> until state=COMPLETED, capture model_id
#   (the new record's internal model_version increments, e.g. "2" — that is the
#    group's revision counter, NOT the pretrained version; ignore it.)

# 3. Deploy
curl -sk $AUTH -X POST "$OS/_plugins/_ml/models/<NEW_MODEL_ID>/_deploy"
# → poll _plugins/_ml/tasks/<task_id> until state=COMPLETED

# 4. Smoke-test
curl -sk $AUTH -X POST "$OS/_plugins/_ml/_predict/text_embedding/<NEW_MODEL_ID>" \
  -H 'Content-Type: application/json' \
  -d '{"text_docs":["hello"],"return_number":true,"target_response":["sentence_embedding"]}' \
  | jq '.inference_results[0].output[0].data | length'   # expect 384

# 5. Verify semantic-api auto-discovers the new deployed model
curl -sk https://search.logrite.ai/health
# expected: {"data":{"status":"healthy","checks":{"opensearch":true,"model":true}}}

# 6. Delete the broken model record (only after the new one is DEPLOYED)
curl -sk $AUTH -X POST   "$OS/_plugins/_ml/models/<OLD_MODEL_ID>/_undeploy"
curl -sk $AUTH -X DELETE "$OS/_plugins/_ml/models/<OLD_MODEL_ID>"

Circuit-breaker gotcha: the OpenSearch container has no cgroup memory limit (docker inspect <c> --format '{{.HostConfig.Memory}}' returns 0); the JVM heap is what bounds it — 4 GiB on dev (OPENSEARCH_JAVA_OPTS=-Xms4g -Xmx4g, set in /opt/logrite-stack/docker-compose.env.yml). The _deploy step transiently spikes heap; an immediate _predict can return 429 circuit_breaking_exception "Memory Circuit Breaker is open" (ML jvm_heap_memory_threshold defaults to 85%). It is transient — clear caches (curl -sk $AUTH -X POST "$OS/_cache/clear") and retry _predict after a few seconds. If it persists, raise the heap through the console’s env editor (/api/env) rather than the breaker threshold.

Note: analyticsapi.logrite.ai and localhost:9200 on the Victor server hit the same OpenSearch node, so running these steps once recovers both prod and local. Semantic Search auto-discovers the deployed model on each /health cycle — no container restart needed. Existing semantic indexes (*_semantic) and ingest pipelines do not pin a model_id in our setup, so swapping the underlying model is transparent.

“View in Discover” returns nothing / 400 Unknown key ... [runtime_mappings]. runtime_mappings is an Elasticsearch-only feature — OpenSearch (we run 3.3.1) rejects it outright with a parsing_exception. Discover only needs a synthetic timestamp when an index has no real date field; for that fallback use OpenSearch’s native derived fields (same painless {lang, source} shape, must be null-safe), not runtime_mappings. Fixed in Logrite-UI/src/components/OpenSearch_Analytics/utils/discoverUtils.js (2026-05-30). In practice the OTEL index log-analytics-logrite has a real time (date) field, so detection uses it directly and no derived field is built.

There was a second, compounding cause: Cortex table rows display the tenant-stripped index name (log-analytics, via toIndexDisplayName), which is not a real index — navigating “View in Discover” to it 404s the mapping fetch and the page then falls back to the broken synthetic-timestamp path. The tool now also emits _indexReal (the unstripped index) and TableBlock.jsx navigates by that. So if Discover ever opens on an index that “doesn’t exist,” check whether a display name lost its -{clientId} suffix.

Backend Build Issues

Maven build fails:

# Ensure Java 17 is being used (not 21)
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
mvn clean package -DskipTests

# Check for dependency issues
mvn dependency:resolve

Backend won’t connect to database:

# The datasource comes from the container env, not a file on the host
C=$(docker ps -q --filter label=com.docker.compose.project=logrite --filter label=com.docker.compose.service=backend)
docker exec "$C" env | grep -E '^SPRING_DATASOURCE_URL='   # never paste the password anywhere

Analyzer / AI Service Issues

LLM provider errors: the provider is chosen by AI_PROVIDER and the model by LLM_MODEL on the ai-service container — one pinned model, no cross-model fallback since 2026-10-07 (see AI models; a degraded /health names a model id missing from the provider catalogue). Check which provider and key are configured — names only:

C=$(docker ps -q --filter label=com.docker.compose.project=logrite --filter label=com.docker.compose.service=ai-service)
docker exec "$C" sh -c 'env | cut -d= -f1 | grep -E "AI_PROVIDER|LLM_|_API_KEY"'
docker exec "$C" sh -c 'echo "$AI_PROVIDER $LLM_MODEL"'
docker logs --tail 100 "$C" | grep -i -E "provider|429|5[0-9][0-9]"

TypeScript build errors:

# Reproduce locally in a MasterControl checkout; the image is built by the deploy pipeline
cd services/ai-service/dev/logrite-ai-service
rm -rf dist/
npm run build

Webhook Server Issues

Webhook not triggering deploys: it no longer deploys anything — see Webhook Server. Dev deploys come from the deploy watcher; check systemctl status logrite-deploy-watcher and the console’s deployment history at https://management.logrite.ai.

C=$(docker ps -q --filter label=com.docker.compose.project=logrite --filter label=com.docker.compose.service=webhook)
docker logs --tail 50 "$C"
curl -s https://dev-webhook.logrite.ai/health

MCP Server Issues

MCP Server crashes on start:

# Logs of the running container
C=$(docker ps -q --filter label=com.docker.compose.project=logrite --filter label=com.docker.compose.service=mcp-server)
docker logs --tail 100 "$C"
curl -s https://dev-mcp.logrite.ai/health

# Reproduce locally in a MasterControl checkout (tests only — never bind 9500 on the host)
cd services/mcp/dev/logrite-mcp-server
pip install -r requirements.txt

Slow Reboot / Boot Hangs

Written for the pre-2026-08-03 host and its Coolify

This section was measured on the cicd host that was reflashed on 2026-08-03, and its fixes are expressed as Coolify UI steps. Coolify no longer runs anything on dev and owns only jobsearch on prod; healthchecks, memory limits and env for every other service now live in the compose files under /opt/logrite-stack/ on each VM and change through the deployment platform, not a UI. The systemd/dockerd tuning still applies to any Docker host.

Symptom: systemctl reboot takes many minutes to come back. On this host (cicd), a freshly-measured baseline shows:

Unit	Boot cost	Notes
Total	~27 min	kernel 58 s + userspace 26 min
docker.service	~20 min	dockerd restoring 22 containers serially on rotational QEMU disk
systemd-tmpfiles-clean.service	~12 min	runs in parallel with docker startup — likely IO contention
containerd.service	~43 s

The 22 containers split roughly as: 7 Coolify infra (autoheal, db, proxy, realtime, redis, sentinel, coolify itself) + 15 Logrite apps (backend, frontend, analyzer ×2, opensearch, opensearch-dashboard, data-prepper ×2, cortex, mcp, semantic-search, etc.).

Status — what’s applied as of 2026-05-22

The full fix from Fix parts 1, 1b, 2, plus an additional tier of tuning, is now live on the cicd host. Listed in apply-order so future operators know what to check.

Lever	State
/etc/docker/daemon.json — dns + dns-opts + live-restore + overlay2	Live. Verified: a container’s /etc/resolv.conf shows nameserver 1.1.1.1; getent hosts github.com resolves instantly.
/etc/systemd/system/docker.service.d/override.conf	Live. TimeoutStopSec=180s, bounded ExecStop, IOSchedulingClass=best-effort, Nice=-5, LimitNOFILE=1048576, TimeoutStartSec=600s.
/etc/systemd/system/systemd-tmpfiles-clean.service.d/override.conf	Live. After=docker.service + IOSchedulingClass=idle.
coolify-apps-autostart.service	Live + fixed 2026-05-22. Requires=docker.service was triggering Dependency failed cascades on slow boots — relaxed to Wants=docker.service + After=docker.service network-online.target. TimeoutStartSec=300s. Dress-rehearsed active (exited) success.
start-coolify-apps.sh	Live + tiered 2026-05-22. Three priority tiers — T1 (frontend, backend, opensearch, analyzer-main, analyzer-dev) waits for healthy before T2 (cortex, semantic-search, data-prepper-{traces,logs}, opensearch-dashboard) before T3 (mcp-server, logrite-webhook, docs, logrite-downtime-tracker). OpenSearch is in T1 because the analyzers depend on it — running concurrently overlaps its 90-120 s shard recovery with backend’s JVM warmup. Per-tier health-wait timeout never blocks boot.
Coolify per-container mem_limit	Live on all 14 resources. opensearch 4 GB · backend 1.5 GB · opensearch-dashboard 1 GB · logrite-cortex 1 GB · analyzer-{main,dev} 768 MB · semantic-search 768 MB · data-prepper-{traces,logs} 768 MB · mcp-server 512 MB · frontend / docs / logrite-webhook / logrite-downtime-tracker 256 MB each (total ~12 GB / 23 GB available, leaves ~11 GB for page cache).
Coolify per-container healthchecks	Live on 14/14 resources. Backend :8080/health, OpenSearch :9200/, OS dashboard :5601/pxe/api/status (basePath is /pxe), data-prepper-{traces,logs} :4900/list (HTTP admin port, not the gRPC 21890/21891), downtime-tracker host=127.0.0.1:8765/ (Docker DNS mis-resolves localhost in this image). All HC start_period 60–120 s; interval 15 s; retries 8.
OpenSearch env tuning	Live. OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:G1ReservePercent=25, bootstrap.memory_lock=true, discovery.type=single-node, indices.recovery.max_bytes_per_sec=200mb, cluster.routing.allocation.disk.threshold_enabled=false. memory_lock=true is safe because mem_limit=4g is set — verify the latter before changing the former or the container OOMs at start. Cluster confirmed GREEN after the redeploy.
logrite-mcp-server + logrite-webhook images	Rebuilt 2026-05-22 with apt-get install -y curl so Coolify’s built-in HC (curl ... || wget ... || exit 1) can run inside the container. python:3.12-slim and node:20-slim base images both lacked it; enabling HC without curl had pushed both into autoheal restart-loops earlier in the session. Commits dc7ffaa (mcp) + 039b020 (webhook).
Boot-cruft systemd units	Disabled 2026-05-22: e2scrub_reap.service (~1 min 8 s; useless inside the QEMU guest), apport.service (~37 s; no desktop, no one reads crash reports), snapd.seeded.service (~41 s; one-shot at install). Kept enabled: snapd.service + snapd.socket — zellij is installed via snap and the active session relies on it. Disabling those would break the user’s terminal multiplexer after the next reboot.

Expected total reboot time after all of the above: 6–8 min (vs the 24m12s baseline). Sub-5 min is below the floor on this rotational rota=1 QEMU disk; SSD-backed storage is the only remaining lever to reach 3-4 min.

Why so slow?

1. Restore-on-startup: Each container with restart: unless-stopped or restart: always is re-launched by dockerd one after another when the daemon starts. OpenSearch JVM, two Spring Boot backends, and a Python MCP server alone contribute several minutes to this serialized chain.
2. Disk is rotational: lsblk -d -o NAME,ROTA shows rota=1 (QEMU virtual disk behaves as HDD), so concurrent I/O during startup is heavily contended.
3. Shutdown is also unbounded: Default docker.service has no TimeoutStopSec, so stop waits the full 90 s default per container and Docker doesn’t escalate to SIGKILL aggressively. This makes the next boot worse if dockerd was killed before flushing state.
4. systemd-tmpfiles-clean.service is a separate, parallel offender (~12 min on this host). Cause is still under investigation — /tmp and /var/tmp are small, so it’s likely IO starvation while dockerd hammers the disk during recovery, not a giant tmp tree.

Fix part 1 — systemd drop-in for graceful, bounded docker shutdown

This addresses only the shutdown path of the next reboot, not startup. It exists so the next reboot starts from a cleanly-stopped state (no replayed write-ahead logs, no stale network sandboxes).

File: /etc/systemd/system/docker.service.d/override.conf (shutdown bits only — the startup bits are in Fix part 2 below):

[Service]
TimeoutStopSec=180s
ExecStop=/bin/bash -c '/usr/bin/docker ps -q | xargs -r /usr/bin/docker stop -t 30 2>/dev/null || true'

Effect:

• Each container gets a graceful 30 s SIGTERM window before SIGKILL (vs the default 10 s).
• Total docker.service stop bounded to 180 s.
• daemon-reload is non-disruptive — no containers restart, only takes effect on the next reboot.

Verify the drop-in is loaded:

systemctl cat docker.service | grep -A2 "override.conf"
systemctl show docker.service -p TimeoutStopUSec   # should print 3min

⚠️ Autostart regression introduced by this drop-in — the ExecStop line runs docker stop <id> on every container before dockerd exits. docker stop flips dockerd’s manually stopped flag in /var/lib/docker/containers/<id>/, and containers with restart: unless-stopped (the Coolify default for project apps) explicitly skip auto-restart in that state. After the next reboot, only restart: always containers (Coolify infra: coolify*, autoheal) come up automatically; the ~14 app containers sit Exited until something starts them. Fix part 1b below closes this gap — install it whenever Fix part 1 is applied.

Fix part 1b — Autostart oneshot for unless-stopped containers (2026-05-19)

A Type=oneshot systemd unit that runs after docker.service and starts every non-running Coolify-managed container in parallel. Idempotent — skips containers that are already running, so safe to also trigger manually.

Files (already present in this repo):

Path	Purpose
scripts/start-coolify-apps.sh	Parallel starter, xargs -P 8. Honors PARALLEL env override.
scripts/coolify-apps-autostart.service	Unit definition (oneshot, After=docker.service Requires=docker.service).
scripts/install-autostart-unit.sh	Sudo installer: copies the unit to /etc/systemd/system/, daemon-reloads, enables.

Install (one-time, on each host that has the Fix part 1 drop-in):

sudo /home/victor/logrite/scripts/install-autostart-unit.sh

Verify:

systemctl is-enabled coolify-apps-autostart.service   # → enabled
systemctl status     coolify-apps-autostart.service   # → loaded; inactive (dead) is correct for a oneshot before its first run

Manual recovery if the oneshot fails on boot — same script runs cleanly any time:

/home/victor/logrite/scripts/start-coolify-apps.sh
journalctl -u coolify-apps-autostart.service -n 100 --no-pager

Measured performance on the cicd host (14 app containers, post-reboot cold state): 29 s parallel (PARALLEL=8) vs ~189 s if started serially. verify-boot.sh reports the unit’s is-enabled / is-active state and flags any app containers still Exited.

Fix part 2 — Auto-start tuning (2026-05-19): hit 5–8 min reboot without manually stopping anything

This is the recommended path going forward. The earlier “stop heavy apps before reboot” workflow (preserved below as an Alternative) shrinks boot time by removing containers from the auto-start set, but it forces a manual Coolify click per heavy service after every reboot. The tuning here keeps all 21 containers on restart: unless-stopped and still lands the full stack in 5–8 min by attacking the three root causes individually.

The three root causes of the 20-min docker.service blame, measured from journalctl -u docker.service:

Phase	Duration	Root cause
Loading containers → stale sandboxes	~8 min	10+ stale network sandboxes from unclean prior shutdown. Fix part 1 (graceful shutdown) prevents these from accumulating.
Per-container DNS retries	~4 min	dockerd asks systemd-resolved per container; resolver returns 127.0.0.53 → “No non-localhost DNS” + retries.
Daemon initialized → systemd Active	~5 min	Network sandbox setup serialized on a rotational disk while systemd-tmpfiles-clean competes for IO.

2a. /etc/docker/daemon.json — daemon-level tuning

Enable live-restore (so planned dockerd restarts don’t kill containers; also reduces what dockerd has to re-validate on startup), short-circuit the DNS retry storm, and pin the storage driver:

{
  "live-restore": true,
  "max-concurrent-downloads": 6,
  "max-concurrent-uploads": 6,
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "default-address-pools": [
    { "base": "10.0.0.0/8", "size": 24 }
  ],
  "dns": ["1.1.1.1", "8.8.8.8", "8.8.4.4"],
  "dns-opts": ["ndots:0", "timeout:1", "attempts:1"],
  "storage-driver": "overlay2",
  "shutdown-timeout": 60
}

Apply (does not restart dockerd — change is read on next dockerd start, i.e. next reboot):

sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.bak.$(date +%s)
sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
{ ...the JSON above... }
EOF
sudo python3 -c "import json; json.load(open('/etc/docker/daemon.json')); print('JSON OK')"

The dns + dns-opts lines are the single biggest startup-time win — they remove the ~4 min DNS retry storm by bypassing systemd-resolved and giving the resolver a 1 s timeout with no retries.

2b. Extended docker.service drop-in — startup tuning

Append to the same /etc/systemd/system/docker.service.d/override.conf from Fix part 1:

[Service]
# --- shutdown tuning (Fix part 1) ---
TimeoutStopSec=180s
ExecStop=/bin/bash -c '/usr/bin/docker ps -q | xargs -r /usr/bin/docker stop -t 30 2>/dev/null || true'

# --- startup tuning (Fix part 2b) ---
IOSchedulingClass=best-effort
IOSchedulingPriority=0
Nice=-5
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=600s

Effect:

• IOSchedulingClass=best-effort priority 0 + Nice=-5 → dockerd wins the IO race against systemd-tmpfiles-clean on the rotational disk.
• LimitNOFILE=1048576 → enough FDs for 21 containers’ simultaneous network sandbox creation.
• TimeoutStartSec=600s → systemd won’t kill dockerd mid-restore if a single slow boot does take 6+ min.

Apply and verify:

sudo systemctl daemon-reload
systemctl show docker.service -p TimeoutStartUSec -p IOSchedulingClass -p Nice -p LimitNOFILE
# Expect: 10min, 2 (best-effort), -5, 1048576

2c. Order systemd-tmpfiles-clean after dockerd

This stops the two services from competing for the same rotational disk during boot. Combined with the IO priority change in 2b, systemd-tmpfiles-clean drops from ~12 min to under 30 s.

File: /etc/systemd/system/systemd-tmpfiles-clean.service.d/override.conf (new):

[Unit]
After=docker.service

[Service]
IOSchedulingClass=idle

Apply:

sudo mkdir -p /etc/systemd/system/systemd-tmpfiles-clean.service.d
sudo tee /etc/systemd/system/systemd-tmpfiles-clean.service.d/override.conf > /dev/null <<'EOF'
[Unit]
After=docker.service

[Service]
IOSchedulingClass=idle
EOF
sudo systemctl daemon-reload

2d. Coolify UI — healthchecks and memory limits

Without these, dependent services crash-loop until OpenSearch is healthy (each retry costs IO during the worst time), and JVM containers evict each other’s page cache. With these, the heavy services come up cleanly without retry storms.

In the Coolify UI (https://cicd.logrite.ai → project logrite), per resource:

Healthchecks tab:

Resource	Test command
backend	curl -fsS http://localhost:8080/health || exit 1
opensearch	curl -fsS 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s' || exit 1
opensearch-dashboard	curl -fsS http://localhost:5601/api/status || exit 1
data-prepper-traces	curl -fsS http://localhost:21890 -o /dev/null || true
data-prepper-logs	curl -fsS http://localhost:21891 -o /dev/null || true
analyzer / mcp / cortex / semantic-search	curl -fsS http://localhost:<port>/health || exit 1

Sensible defaults: interval 15s · timeout 5s · start-period 90s · retries 8. OpenSearch needs start-period 120s because cold shard recovery can exceed 90 s.

Resource limits tab:

Resource	mem_limit	Why
opensearch	4g	JVM 2 GB + off-heap Lucene mmap
backend	1500m	Spring Boot + Hibernate cache
opensearch-dashboard	1g	Node + bundle
data-prepper-*	768m	Armeria gRPC + OS sink
logrite-cortex	1g
mcp-server	512m
semantic-search	768m
analyzer-*	768m
frontend, docs, webhook, downtime-tracker	256m each

Total reserved: ~12 GB out of 23 GB. Leaves ~11 GB for the page cache, which is what makes cold Lucene mmap not thrash.

2e. OpenSearch env vars — fast-start tuning

In the opensearch resource’s Environment Variables tab:

OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:G1ReservePercent=25
bootstrap.memory_lock=true
discovery.type=single-node
cluster.routing.allocation.disk.threshold_enabled=false
indices.recovery.max_bytes_per_sec=200mb

AlwaysPreTouch makes the JVM touch its heap pages at startup — counterintuitive but speeds up steady-state by ~30 % because page faults aren’t paid during query time. memory_lock=true requires the mem_limit from 2d to be set first; otherwise the container is OOM-killed on start.

2f. (Optional) Spring Boot CDS for the backend

Biggest single backend cold-start win: ~60 s → ~15 s. In the backend’s Dockerfile:

# First-run only — generates the CDS archive
RUN java -XX:ArchiveClassesAtExit=/app/cds.jsa -jar /app/app.jar --spring.context.exit=onRefresh || true

# Permanent start command
CMD ["java", "-XX:SharedArchiveFile=/app/cds.jsa", "-jar", "/app/app.jar"]

The .jsa archive lives inside the image so it survives redeploys.

Alternative — Pre-reboot stop heavy apps (faster boot, manual heavy-service start)

If you’d rather skip the Coolify UI work in Fix part 2d/2e, you can keep the original 2026-05-18 workflow: stop the heavy apps before rebooting so dockerd has fewer containers to restore.

Docker’s restart: unless-stopped policy explicitly does NOT auto-restart manually-stopped containers, so they stay down across reboot until you click Start again in Coolify.

In the Coolify UI (https://cicd.logrite.ai → project logrite), click Stop on each heavy app in this order (gentlest dependents first, opensearch last):

1. data-prepper-traces (traces.logrite.ai)
2. data-prepper-logs (logs.logrite.ai)
3. opensearch-dashboard (analytics.logrite.ai)
4. analyzer-main, analyzer-dev, mcp-server, semantic-search, logrite-cortex
5. opensearch (analyticsapi.logrite.ai) — last

After reboot, start OpenSearch first (~3 min to green), then start dependents in parallel.

Post-reboot verification

A ready-made report script lives at /home/victor/logrite/scripts/verify-boot.sh. It is read-only, exits 0 always, and does not require sudo. Run it any time after reboot:

/home/victor/logrite/scripts/verify-boot.sh

It prints, in order:

1. systemd-analyze total boot time.
2. Top 10 boot-time offenders (systemd-analyze blame).
3. docker.service / containerd.service blame lines specifically, with a pass/fail check for “is docker.service sub-minute?”.
4. Whether the drop-in override is present and its effective values.
5. Container inventory and any container reporting health=unhealthy.
6. Backend /health HTTP code and response time.
7. OpenSearch cluster health (best-effort — not an error if OpenSearch is intentionally stopped).
8. A summary comparing the run against the documented targets and baseline.

Targets after Fix part 1 + 2 applied (auto-start path):

Metric	Target	Baseline (2026-05-18)
systemd-analyze total	< 8 min	27 min
docker.service blame	< 3 min	20 min 26 s
systemd-tmpfiles-clean blame	< 30 s	12 min 02 s
Backend HTTP 200 from SSH-up	< 2 min	~5–8 min
OpenSearch yellow from SSH-up	< 5 min	~10 min
All 14 logrite containers up	< 7 min	~25 min

If docker.service is still in the minutes after both fixes, verify:

• The Coolify UI healthchecks from Fix part 2d are actually saved (the resource page should show HEALTHCHECK lines under “Effective config”).
• mem_limit is non-zero for opensearch/backend/data-prepper (docker inspect <c> --format '{{.HostConfig.Memory}}' returns bytes, not 0).
• The dockerd journal Loading containers: done happens within ~90 s of Starting up (sudo journalctl -u docker.service -b 0 | grep -E "Starting up|Loading containers"). If still longer, the page cache is still being evicted by JVM cold starts — re-check that mem_limits are applied.

Per-service reboot timeline (target state)

Times are from T=0 (power restored / reboot fires). Two columns: clean shutdown (typical reboot) vs power outage (kernel killed mid-write, stale sandboxes accumulate, page cache cold). Numbers assume the rotational sda disk; SSD-backed storage cuts all of these by ~40 %.

Phase	Service	URL	Clean reboot	After power outage
Kernel + early systemd	BIOS / GRUB / kernel init	—	0:45	0:50
	systemd userspace, networking, journald, containerd	—	1:30	1:40
Docker daemon	dockerd starting	—	1:30	1:40
	Loading containers (parallel)	—	2:30	3:10
	dockerd Active (systemd)	—	2:45	3:30
	systemd-tmpfiles-clean	—	2:55	3:40
Coolify infra	coolify-db (postgres)	internal	3:00	3:50
	coolify-redis	internal	2:50	3:40
	coolify-proxy (Traefik)	internal	3:00	3:50
	coolify + realtime + sentinel + autoheal	https://cicd.logrite.ai	3:15	4:00
Light Logrite services	frontend	https://app.logrite.ai	3:15	4:00
	downtime-tracker	https://downtime.logrite.ai	3:15	4:00
	webhook	https://webhook.logrite.ai	3:15	4:00
	docs	https://devdocs.logrite.ai	3:20	4:05
Medium Logrite services	backend (with CDS)	https://leapfrogapi.deployd.io	3:45	4:30
	backend (without CDS, current image)	https://leapfrogapi.deployd.io	4:30	5:15
	mcp-server	https://mcp.logrite.ai	3:30	4:15
	analyzer-main	https://analyzer-main.logrite.ai	4:00	4:45
	analyzer-dev	https://analyzer.logrite.ai	4:00	4:45
	logrite-cortex (AI service)	https://aiservice.logrite.ai	4:00	4:45
Heavy Logrite services	OpenSearch yellow	https://analyticsapi.logrite.ai	5:45	7:00
	opensearch-dashboard (waits for OS)	https://analytics.logrite.ai	6:15	7:30
	data-prepper-traces (waits for OS)	https://traces.logrite.ai	6:15	7:30
	data-prepper-logs (waits for OS)	https://logs.logrite.ai	6:15	7:30
	semantic-search (uses OS)	https://search.logrite.ai	6:30	7:45
ALL SERVICES HEALTHY	—	—	~6:30	~7:45

Backend reachable at ~3:45 clean / ~4:30 outage — that’s when logrite.deployd.io serves user traffic. Everything else trails behind the OpenSearch readiness gate.

Coolify Control Plane Down (cicd.logrite.ai 502)

The coolify core container is what serves cicd.logrite.ai and runs auto-deploys on git push. When it’s down, a push reaches GitHub but nothing deploys until the control plane is back — so “I pushed but it’s not live” during a 502 is expected, not a separate bug.

There are three distinct failure modes; identify which one before acting:

docker ps -a --filter name=coolify --format '{{.Names}}\t{{.Status}}'

1. Not running (:8000 bind race) — after a reboot the core loses the :8000 bind race and restart=always gives up after one attempt. Self-healed by /home/victor/logrite/scripts/ensure-coolify-core.sh (idempotent docker start + spaced retries), run at boot via the @reboot cron.

2. Not running (severe leak) — an orphaned root docker-proxy + stale libnetwork :8000 reservation → docker start fails with “port is already allocated”. Needs a Docker daemon restart (operator runbook; the watchdog logs repeated failures to /home/victor/ops/coolify-core-autostart.log and stops there).

3. Running but unhealthy (nginx down inside the container) — the container is Up … (unhealthy), :8000 is bound, but curl http://localhost:8000/api/health fails / cicd returns 502. Inside the image, s6 supervises nginx; an s6 down flag at /run/service/nginx/down keeps nginx stopped, so nothing listens on :8080 (php-fpm stays up). Manual recovery:

   docker exec coolify rm -f /run/service/nginx/down
   docker exec coolify /package/admin/s6-*/command/s6-svc -u /run/service/nginx
   # verify: nginx listening + health flips to healthy + public 200
   docker exec coolify sh -c 'netstat -ltn | grep :8080'
   docker inspect -f '{{.State.Health.Status}}' coolify
   curl -s -o /dev/null -w '%{http_code}\n' https://cicd.logrite.ai/

Auto-recovery (2026-07-01) — event-driven, not blind polling. ensure-coolify-core.sh is health-aware (asserts Docker health and a direct :8080 listener check; a graceful nginx -s reload keeps the socket so it won’t false-fire) and has two modes:

• once (default) — a single reconcile pass, run at boot via the @reboot cron to win the :8000 bind race (mode 1).
• watch — recover now, then verify every 30 s for 5 minutes that it stays up, re-healing any flap; flock-guarded so only one watcher runs at a time.

The recovery is triggered by the downtime event itself, not a timer: the downtime tracker (services-tracker.py) health-probes coolify-core each tick, and the moment it sees it DOWN it fires ensure-coolify-core.sh watch. So we act right after a real outage and then confirm it holds for 5 minutes — there is no blind */3 recovery cron anymore. On unhealthy/not-serving the pass clears the nginx down-flag and s6-svc -u in place, escalating to docker restart coolify (wipes the tmpfs down-flag) behind a 15-minute cooldown.

Monitoring: the downtime tracker tracks the control plane as service coolify-core (probes http://localhost:8000/api/health), so Coolify up/down and RECOVERED appear in the Discord status embeds like every other service — status is computed live per-probe, not hardcoded. This same probe is what triggers the event-driven recovery above.

General Debugging

# What is running, and on which host port
docker ps --format '{{.Names}}\t{{.Status}}\t{{.Ports}}'

# Follow a service's logs
docker logs -f <container>

# Which commit a service is actually running
curl -s http://127.0.0.1:8090/api/services | python3 -m json.tool | grep -A2 '"image"'

# Deploy history, live logs, resources and container logs for BOTH hosts
open https://management.logrite.ai

# Generate AI Root Cause Analysis
./cli/logrite-manager.js rca

OpenSearch: k-NN and ML Commons share libstdc++

Caution

The k-NN plugin’s faiss library and ML Commons’ bundled PyTorch both load libstdc++. After an embedding model is deployed, PyTorch’s older copy (from the ML cache) is loaded first, and the first k-NN index flush then fails with UnsatisfiedLinkError ... libopensearchknn_faiss.so ... GLIBCXX_3.4.20 and the node crashes. This happened on 2026-10-09 at 07:17 UTC on prod and 07:21 UTC on dev, on the first flush of a k-NN upload index. No data was lost (bind-mounted data).

Fix, in place on both hosts: set LD_PRELOAD=/usr/lib64/libstdc++.so.6 (the image’s GCC 14 libstdc++ 6.0.33) for the opensearch service in the env overlay /opt/logrite-stack/docker-compose.env.yml. Verified by deploying the model first, then embedding and flushing a k-NN index: restart count 0, no UnsatisfiedLinkError. plugins.ml_commons.model_auto_redeploy.enable is true again. Do not remove the preload while k-NN and ML Commons run on the same node.