Discord Notifications

Overview

Operational events reach Discord through a single webhook, driven by logrite-ops — a small standard-library Python service that runs on each VM and watches Docker and git (and, on prod, Coolify for jobsearch).

It reports three kinds of thing:

1. Deploys — started, succeeded, failed, with the commit, duration, trigger and a post-deploy HTTP probe.
2. Unexplained restarts and health flips — a container that came back without a deployment to account for it.
3. An hourly roll-up — every service, host metrics, OpenSearch health, and any service running behind its branch.

It also causes deploys: a branch ahead of what is running is deployed automatically. See Automatic deploys below.

This replaced the PM2 notifier

The previous implementation was cli/deployment-notifier.js in LogriteMasterControl: a PM2 event listener, configured from /root/logrite/config/webhook.json. Both are gone. There is no PM2 on either VM and /root/logrite does not exist — see Deployment reality. The JS library (lib/discord-notifier.js) is kept only for reference.

Where it lives

Source	ops/notify/ in Deployd-io/LogriteMasterControl
Installed to	/opt/logrite-ops
Config	/etc/logrite-ops/config.json — mode 0640 root:deployd, never in git
State	/var/lib/logrite-ops/watcher-state.json
Units	logrite-deploy-watcher.service, logrite-status-report.timer

rsync -a ops/notify/ dev:/tmp/logrite-notify/
ssh dev 'sudo /tmp/logrite-notify/install.sh'

install.sh is idempotent and never overwrites an existing config.

Automatic deploys

The watcher reads each application’s branch tip with git ls-remote over the deploy key already on the host, compares it to what is actually running, and when they differ asks the management console to deploy (POST /api/deploy, deploy_backend: "console" in /etc/logrite-ops/config.json). It used to call the Coolify deploy API; the Coolify-era detail below is kept as history.

Why deploys are Coolify-native rather than GitHub Actions

Deploys used to run through each repo’s docker-deploy workflow, which built the image and then called the Coolify deploy webhook. From 2026-08-19 the Deployd-io org was billing-blocked on GitHub Actions, and every run was killed before its first step:

The job was not started because recent account payments have failed or your spending limit needs to be increased.

The job died in about four seconds with zero steps and no log, so a push landed code and deployed nothing — for every service. Reading a ref over SSH is not a billable workflow minute, which is why this path kept working, and it is why this service was written.

That block is history. Since the 2026-09 migration every private repo’s CI runs on the self-hosted logrite-dev runners on this VM, where minutes are free and unmetered — see CI runners. The deploy path deliberately did not go back: all eight docker-deploy.yml workflows stay disabled_manually, because re-enabling one would give dev a third concurrent deploy path alongside the Coolify push webhook and this watcher. Deploys are Coolify-native by choice now, not because a bill is unpaid.

Behaviour worth knowing:

• Some services are built here first. See Services that are built before they are deployed.

• The first run adopts, it does not deploy. Starting by building every drifted app would put eleven simultaneous Docker builds on the shared Coolify host. Startup records what is running as the baseline and reports the drift.

• At most max_parallel_deploys (default 2) are triggered per tick.

• The running image tag beats Coolify’s deployment history. An image built by hand on the box leaves the last finished deployment pointing at an older commit than the code actually serving traffic.

• An unreadable ref is announced, never treated as “no change”. Both are the empty string, and conflating them turns a broken watcher into a silent one.

• Only apps in the auto_deploy allow-list are deployed. Everything else in report_only is watched and reported but never touched, so a new app added in Coolify cannot start auto-deploying before someone says it should.

Services that are built before they are deployed

Most apps are build_pack=dockerfile and Coolify builds them from the repo, so a deploy is self-contained. backend is not. It deploys Dockerfile.deploy, which is one line — FROM ghcr.io/deployd-io/logrite-ai:dev — an image produced by docker-deploy.yml, which has been disabled since 2026-08 and stays that way. Left alone, a backend deploy succeeds, re-uses whatever that image last held, and Coolify tags the new container with the current git sha, so tag and code disagree while the deploy looks green.

prebuild/backend.sh closes that. Before deploying backend the watcher runs the same gates the workflow did, in the same order:

1. mvn -B package — the unit suite and the jar
2. mvn -B checkstyle:check — the 300-line file ceiling
3. checkstyle-ratchet.sh — the suppression list may only shrink
4. stage exactly one jar (cp target/*.jar would silently pick one of several)
5. docker build -f Dockerfile.ci --build-arg GIT_SHA=…, then read /app/BUILD_SHA back out of the image and refuse if it disagrees

A non-zero exit cancels the deploy and posts the tail of the log. Not deploying is the point: deploying anyway would ship the previous image under the new commit’s name. It is also the only gate in the deploy path: checks.yml now runs on the logrite-dev runners and reports a real verdict on the commit, but Actions cannot stop a Coolify deploy, so without the prebuild the backend would deploy automatically with no suite having run against the image being shipped.

The image must be published, not just built

Coolify builds every dockerfile app with docker build --pull — ApplicationDeploymentJob.php (~3830-3846) has no path that omits it. So the FROM in the deploy Dockerfile is resolved from the registry on every deploy, and an image built only on the host is never consulted. Publishing is step 6 of the build, and the ability to publish is probed in about a second before the ten-minute suite, via a registry blob-upload session.

The registry is the deploy host’s own (changed 2026-08-21)

Between 2026-08-20 and 2026-08-21 backend auto-deploy was off, and both halves of the publish path were broken in ways nothing in these repos could fix:

• the workflow that writes ghcr.io/deployd-io/logrite-ai:dev was billing-blocked from 2026-08-19 — every run failed in about four seconds, so the tag froze. It is frozen still, for a different reason: that workflow is deliberately disabled and stayed disabled through the 2026-09 runner migration;
• the host’s own ghcr.io credentials are pull-scoped, so backend.sh could not write the tag either:

error from registry: permission_denied: The token provided does not match expected scopes.

backend.sh correctly exited 8 and the watcher posted ”🔑 cannot publish the built image” rather than deploying something misleading. Everything else passed — 1984 tests, checkstyle, the ratchet, the image build and the /app/BUILD_SHA verification. Publishing was the only missing link.

The resolution was that --pull does not care whose registry it is. A registry:2 now runs on the deploy host (see ops/registry/ in MasterControl), bound to 127.0.0.1:5000, and backend.sh publishes there by default:

IMAGE = ${LOGRITE_REGISTRY:-127.0.0.1:5000}/deployd-io/logrite-ai:<tag>

The backend’s dev Coolify app points at /Dockerfile.devdeploy — the same one-line FROM aimed at that registry. There is no GitHub account, token or billing anywhere in the deploy path now.

Why loopback, and why that is enough

registry:2 with no auth accepts writes from anyone who can reach it, so the port is published to 127.0.0.1 only. That also means plain HTTP works with no /etc/docker/daemon.json edit, because Docker treats 127.0.0.0/8 registries as insecure by default — and therefore no daemon restart, which would take every service on the box down.

It is sufficient because the FROM is resolved by the daemon, not by Coolify’s build wrapper, and Coolify’s build command also passes --network host.

BASE still comes from ghcr.io deliberately: pulling from ghcr.io works on this host and always did — only the push was refused — so mirroring a 13GB toolchain image locally would spend disk on a problem that does not exist.

Set LOGRITE_REGISTRY=ghcr.io to restore the old target on any environment whose token can push there; the publish probe speaks the registry v2 protocol generically rather than ghcr.io specifically.

The build host needs node and esbuild

The suite needs node 22 and esbuild 0.21.5 on PATH, with ESBUILD_BIN exported. The workflow used to npm install -g both on a throwaway runner; since the 2026-09 migration it only verifies /usr/local/bin/esbuild, which ops/runner/provision-toolchain.sh provisions once on the runner host. The workflow’s own comment says why it bothers: EsbuildSyntaxChecker.available() returns false when $ESBUILD_BIN does not run, and JsSyntaxGate then falls back to node --check, which does not parse TypeScript at all — a fallback that once hid a TypeScript failure for an entire session while the job stayed green. Other JS tests self-skip via JUnit assumptions when node is absent.

A build host without them therefore does not fail; it covers less and still reports success. backend.sh refuses to build (exit 7) unless both resolve, before it takes the lock or clones anything:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash - && sudo apt-get install -y nodejs
sudo npm install -g [email protected]

Installed on the dev VM on 2026-08-20. A fresh host — including prod — needs this before its first backend build. In backend.sh ESBUILD_BIN is resolved with command -v rather than hard-coded, because a build host is whatever someone installed; the workflow checks the one fixed path instead, because there a missing binary means provision-toolchain.sh has not been run.

Each build also gets a private temp dir. CollectorScriptRenderTest writes to a fixed java.io.tmpdir-based path, so whichever user runs the suite first owns that file and every later run by a different user fails with AccessDenied — a build broken purely by the build before it.

Why not just point Coolify at Dockerfile.ci?

Because it is not self-contained: it does COPY app.jar app.jar, so a Maven build must have produced the jar first. A one-line Coolify setting cannot do that, which is why the prebuild exists.

The build runs as a background subprocess so a long Maven suite does not stall deploy detection for the other services, holds a flock so two builds cannot race on the same checkout, is capped at 45 minutes, and refuses to start below 12 GB of free disk. Exit codes are meaningful: 2 = another build holds the lock (retry next tick), 3 = not enough disk (bounded retry plus an alert), 7 = the host is missing the JS toolchain, 8 = the image cannot be published (see below), anything else = the gate refused, no retry.

Retries

A failed deploy is retried up to 3 times with a 3-minute backoff, then left alone with a 🛑 message. Retries are per commit, so the next push starts from zero. A transient failure should not stall deploys until someone notices, and a genuinely broken commit must not become a build loop.

Reporting the result back onto the commit

From 2026-08-19 until the 2026-09 runner migration every commit on dev showed a red ✗ and 0 / 3 in GitHub: the checks and docker-deploy workflows, billing-blocked, failing in four seconds having run no steps. Both halves of that have changed. checks runs on the self-hosted logrite-dev runners now, so its tick is a verdict on the code; docker-deploy is disabled_manually and produces no run at all.

Two ticks, two different questions

checks answers “do the gates pass on this code”. It says nothing about whether the commit reached dev — Actions is not in the deploy path, so a green checks sits just as happily on a commit whose build gate refused it and which never deployed. The deploy verdict is the commit status this watcher posts, below. And a job stuck on Queued is a stopped runner service (systemctl status 'actions.runner.*' on deployd-dev), not a billing problem.

The watcher therefore posts the real outcome as a GitHub commit status, which needs no Actions minutes and no GitHub App — only a token with repo:status, set as github_token in the config. With no token configured the feature is off and everything else behaves identically.

when	state	description
prebuild starts	pending	Building and running the gate
deploy starts	pending	Deploying to dev
deploy finishes	success	Deployed to dev
deploy fails	failure	Deploy to dev failed
build gate refuses	failure	nothing deployed
out of disk, or cannot publish	error	an environment problem, not the commit

The context is keyed on the app, not the repository — logrite-ai-service builds ai-service, warden and ingest-shim from a single commit, so a repo-keyed context would have the three overwrite each other and leave the commit reporting whichever posted last. That commit correctly carries three ticks.

Backfilling commits from before it was enabled

sudo python3 /opt/logrite-ops/backfill_status.py --dry-run   # print, post nothing
sudo python3 /opt/logrite-ops/backfill_status.py

It reads Coolify’s own deployment history and reports only what Coolify recorded: finished becomes success, a failed deployment becomes failure, and anything still in flight is skipped rather than guessed at. A green tick that was not earned is worse than the red ✗ it replaces, because someone will trust it.

Notification types

Deploy lifecycle

🔄 Deploying frontend                    (blue — in progress)
   Commit  1041cf52   Branch  dev   Trigger  git push (deploy watcher)

✅ frontend — deploy succeeded           (green)
   Commit  1041cf52   Branch  dev   Duration  1m 24s
   URL  dev-app.logrite.ai
   Post-deploy probe  ✅ HTTP 200 in 286 ms

The Trigger field names the origin: git push (deploy watcher), manual (Coolify UI) (prod’s jobsearch only), API, git webhook, rollback or restart. Every deployment is announced regardless of who started it, so a manual Redeploy is as visible as an automatic one.

A failed deploy is red and carries a Logs link straight to that deployment in Coolify.

Restarts and health

♻️ analyzer restarted                    (amber)
   restarted outside a deployment
   Image tag  1d7e6882   Restart count  3

Restarts within ten minutes of a settled deployment are attributed to it and not reported — replacing a container is what a deploy does, and alerting on it trains people to ignore the channel.

Hourly status report

Sent on the hour by logrite-status-report.timer, with SUPPRESS_NOTIFICATIONS unless something needs a human, so it is scrollable history rather than 24 pings a day.

📊 LogRite dev — 14/16 services healthy
🟢 ai-service · 26 hours · 598c12a5 · HTTP 404 · 385ms
🟢 backend · 14 minutes · d3898b33 · HTTP 200 · 811ms
⚪ jobsearch · 26 hours · ebe7f6f9 · ❌ URL unreachable
…

🖥️ Host & infrastructure
   Load (1/5/15m)  6.12 / 9.05 / 7.43
   Memory  11,535 MB / 32,071 MB (36%)
   Disk /  127 GB / 144 GB (88%), 18 GB free
   OpenSearch  🟢 green · 1 node · 163 shards
   ⚠️ Needs attention  • disk at 88% on /

A container Coolify calls running whose public URL does not answer is counted as unhealthy, not healthy — counting it green is how a status page reports fine through an outage.

Sending an event by hand

Anything a person does that the platform has no record of — a restore, a cache flush, a maintenance window:

logrite-notify --title "Restoring the dev database" --level warn \
    --detail "read-only for about five minutes" \
    --field "operator=rahil" --field "expected downtime=5m"

--level is ok, warn, error or info. Exit status is 0 only when Discord confirms delivery, so a script can react to a failed send.

Operating it

systemctl status logrite-deploy-watcher      # is the watcher alive
journalctl -u logrite-deploy-watcher -f      # what it is doing
systemctl list-timers logrite-status-report  # when the next report goes out
systemctl start logrite-status-report        # send one now

python3 /opt/logrite-ops/status_report.py --dry-run   # print the payload, send nothing
python3 /opt/logrite-ops/deploy_watcher.py --once     # a single tick, then exit

The journal logs the Discord message id for every send, so “the watcher ran and the channel is empty” can be told apart from “the watcher never sent anything”.

Webhook handling

All of it is in ops/notify/discord.py, so no caller has to think about it:

• every structural limit is truncated, never raised on — title 256, description 4096, 25 fields, field value 1024, footer 2048, 10 embeds, and a 6000-character budget across the whole message. Over budget, trailing content is dropped and a footer says how much;
• 429 is honoured via retry_after from the body, falling back to the Retry-After header, and X-RateLimit-Remaining / -Reset-After are used to wait before sending into an exhausted bucket;
• 5xx and transport errors retry with jittered exponential backoff; 4xx does not, because retrying cannot fix a revoked webhook;
• allowed_mentions is pinned to parse nothing, so a commit message containing @everyone cannot ping the server;
• requests use ?wait=true so Discord returns the created message — the only way to tell “delivered” from “accepted and silently discarded”.

Discord’s limits for reference: roughly 30 requests per 60s per webhook, 5 per 5s per channel (shared by every webhook in that channel), 50/s globally per IP. Normal traffic is one message per deploy plus one per hour, so the limiter is for the pathological case.

Configuration

/etc/logrite-ops/config.json, mode 0640 root:deployd:

{
  "environment": "dev",
  "discord_webhook": "https://discord.com/api/webhooks/…",
  "coolify_token": "…",
  "github_token": "github_pat_…  (optional; repo:status, for commit statuses)",
  "coolify_base": "http://localhost:8000/api/v1",
  "coolify_ui": "https://dev-cicd.logrite.ai",
  "poll_seconds": 60,
  "max_parallel_deploys": 2,
  "disk_warn_percent": 85,
  "memory_warn_percent": 90,
  "build_stamp_paths": { "backend": "/app/BUILD_SHA" },
  "auto_deploy": ["frontend", "backend", "ai-service", "…"],
  "report_only": ["opensearch", "jobsearch", "…"]
}

Any key can be overridden per-run with LOGRITE_OPS_<KEY>; the webhook also reads LOGRITE_DISCORD_WEBHOOK.

The API is called over loopback on purpose

coolify_base points at http://localhost:8000 rather than the public hostname. This runs on the Coolify host, so loopback takes Cloudflare, DNS and egress out of the path of the process whose job is to notice when those break. It also sidesteps Cloudflare’s Error 1010, which rejects the default Python-urllib User-Agent outright.

OpenSearch anomaly alerts

Separate from this service: the OpenSearch alerting plugin posts to the same channel through its own destination, configured by opensearch/scripts/create-discord-destination.js.

🚨 Anomaly Detected — payment-latency-detector
Severity: MEDIUM (grade: 0.47)