Cron Jobs & Background Tasks

There is no crontab

Danger

There is no crontab, and none of the /root/logrite/cli/*.js scripts below exist. They ran on the host destroyed on 2026-08-03 and were driven by PM2, which is gone too. sudo crontab -l on the dev VM is empty (measured 2026-09-25). Everything scheduled now is a systemd unit or timer. The per-script sections further down are kept for reading old runbooks.

What actually runs

Measured on the dev VM, 2026-09-25; the alerter and backup rows were added 2026-10-10:

Unit	Kind	What it does
logrite-deploy-watcher	service, always on	Polls each branch tip every 60s, deploys through the console, reports to Discord
logrite-management	service, always on	The console at management.logrite.ai
logrite-reports	service, always on	Static report pages, Tailscale-reachable
logrite-status-report	timer, hourly	Roll-up of every service, the host and anything that has drifted. Since 2026-10-10 it also warns when the host’s OpenSearch ML embedding model is not DEPLOYED (e.g. DEPLOY_FAILED)
logrite-internal-plane-alert@dev, @prod	timer, every 10 min	Announces /internal/mcp/** auth refusals on that host — see below
logrite-disk-guard	timer, hourly	Trims build cache unused for 48h (-af), removes dangling images and exited containers (never --preadopt- backups), trims dead scan checkouts. Old pipeline images are the console janitor’s, daily
actions.runner.*	8 services	Self-hosted GitHub Actions runners, one per repo
logrite-backup (prod VM)	timer, nightly 02:30	Prod snapshot with logical database dumps — see below

systemctl list-units  --type=service --all | grep -i logrite
systemctl list-timers --all             | grep -i logrite
journalctl -u logrite-deploy-watcher -f

Caution

A timer whose service is failed still looks scheduled. list-timers shows the next run whether or not the last one worked, so a job that has failed every hour for days reads as healthy there. Check the SERVICE, not just the timer: systemctl list-units --state=failed.

That is not hypothetical. logrite-status-report polled Coolify, and from the moment Coolify was removed from dev on 2026-09-23 it failed hourly — posting a red “could not reach Coolify” alert into the same Discord channel the watcher uses for real deploy failures, roughly fifty times. Fixed 2026-09-25 by reading the git catalogue instead.

Internal-plane alerter and nightly backups (2026-10-10)

Documented from LogriteMasterControl main at 9ce9a1fa.

Internal-plane alerter (logrite-internal-plane-alert@<host>.timer, every 10 minutes per host). It reads the backend’s /internal/mcp/** authentication refusals and announces them to Discord when at least LOGRITE_INTERNAL_ALERT_THRESHOLD (default 5) fall inside a rolling LOGRITE_INTERNAL_ALERT_WINDOW_MIN (default 10) minute window — so 4 refusals in one run and 4 in the next still cross the threshold once. It records what it has seen only after a delivered announcement (or when below the threshold); a failed send exits 1 and leaves the state untouched, so the next run tries again. The unit runs the installed copy in /opt/logrite-ops/security/ (installed by ops/security/install.sh), and its --self-test checks the backend repository’s origin/main with git grep, not a working tree on whatever branch is checked out.

Nightly prod backup (logrite-backup.timer on the prod VM, OnCalendar=*-*-* 02:30:00, RandomizedDelaySec=600, Persistent=true), installed as root by ops/backup/install_nightly.sh. Each run of ops/backup/prod_snapshot.sh writes a snapshot directory under /home/deployd/logrite-backups/ with volume copies and consistent logical dumps (logical/postgres-all.sql.zst, logical/mongo.archive.gz, logical/redis-dump.rdb) and a log in meta/snapshot.log. Containers are found by their compose service label first, then the Coolify label; a database with no running container is recorded as a failure rather than skipped. The last 7 snapshots are kept, and pruning runs only after a successful snapshot.

systemctl list-timers | grep -E 'logrite-(backup|internal-plane-alert)'

The cron era (historical)

Everything below ran on the host destroyed on 2026-08-03. None of these files, processes or crontab lines exist. They are kept because old runbooks, tickets and Discord threads still refer to them by name — each one names what answers that question today. Nothing here is safe to copy and run.

Open the pre-2026-08-03 cron and PM2 runbooks

model-health-check.js

Replaced by: the hourly logrite-status-report, which warns when the model is not DEPLOYED (2026-10-10). The script itself was deleted from the repository.

Location: /root/logrite/cli/model-health-check.js Schedule: Every 30 minutes Purpose: Verify the OpenSearch ML embedding model is loaded and responsive

What It Does

1. Calls the Semantic Search API /health endpoint
2. Checks if the ML model responds to a test embedding request
3. If model is unloaded: calls opensearch-mgr model-ensure
4. If model fails to reload: sends Discord alert
5. Logs result to audit log

Manual Run

node /root/logrite/cli/model-health-check.js

Cron Entry

*/30 * * * * node /root/logrite/cli/model-health-check.js >> /var/log/logrite/model-health.log 2>&1

---

status-reporter.js

Replaced by: logrite-status-report.timer (hourly). It reads the git catalogue, not PM2 and not Coolify.

Location: /root/logrite/cli/status-reporter.js Schedule: Daily at 09:00 UTC Purpose: Send proactive status reports to Discord

What It Does

1. Queries PM2 for all Logrite process statuses
2. Checks OpenSearch cluster health
3. Checks semantic search model status (via semantic-status)
4. Builds a formatted Discord embed
5. Posts to Discord webhook

Manual Run

node /root/logrite/cli/status-reporter.js
# Or via master CLI:
./cli/logrite-manager.js notify-status

Cron Entry

0 9 * * * node /root/logrite/cli/status-reporter.js >> /var/log/logrite/status-reporter.log 2>&1

---

deployment-notifier.js

Replaced by: ops/management/notifier.py — the component that performs the deploy also announces it, so the two cannot disagree.

Location: /root/logrite/cli/deployment-notifier.js Runtime: Always running (PM2 daemon) Purpose: Listen for PM2 events and send Discord notifications

What It Does

• Connects to PM2 event bus
• Listens for start, stop, restart, exit, error events
• Sends color-coded Discord notifications for each event
• Enriches notifications with service name, branch, port, and uptime

Starting

pm2 start /root/logrite/cli/deployment-notifier.js --name deployment-notifier
pm2 save  # Persist across reboots

Event Types Handled

PM2 Event	Discord Message	Color
start	🚀 Service Started	Green
restart	🔄 Service Restarted	Blue
stop	⏹️ Service Stopped	Yellow
exit	⚠️ Service Exited	Orange
error	🚨 Service Error	Red

---

generate-traffic.sh (JobSearch demo)

Location: /home/victor/logrite/projects/JobSearch/generate-traffic.sh (Victor server only) Schedule: Every 15 minutes Purpose: Keep fresh runtime logs/traces flowing into OpenSearch for the JobSearch demo app — without anyone hitting URLs by hand — so Cortex always has recent errors to surface and offer Fix it on.

Note

This is the “auto-generate traffic” (Option A) approach to keeping Cortex’s error data live. Cortex reads runtime error logs from OpenSearch, which only exist when the buggy code path actually executes. This cron stands in for real user traffic so those errors keep arriving. (The complementary “errors from the code itself” / static analysis approach — reading source without running it — is a separate, not-yet-shipped build tracked in new-arch_or_extend/cortex-static-code-analysis-b2.md.)

What It Does

1. Probes http://localhost:9081/ and exits quietly (exit 0) if the app is down (e.g. between deploys/reboots) — never errors the cron.
2. Normal traffic — GET /jobs/search?term=… across 8 terms (spring, java, python, react, devops, kubernetes, terraform, aws), plus /jobs, /employers, /candidates/search?term=engineer. Populates logs + traces with successful queries.
3. Error traffic — hits the app’s simulate-error endpoints (/candidates/simulate-error, /jobs/simulate-error, /employers/simulate-error, /proposals/simulate-error, /jobs/simulate-search-error), each throwing a real, traceable exception (Arithmetic, NumberFormat, NullPointer, IndexOutOfBounds, ClassCast) so a fresh, varied set of SEVERE logs stays in the recent window, each tracing to a real class/method for the Fix-it levels.
4. Appends <timestamp> generate-traffic: ok to traffic.log.

Consumer

Cortex’s deterministic error sweep reads these SEVERE logs and renders a summary + Fix it button. See AI Service → Deterministic error sweep.

Manual Run

/home/victor/logrite/projects/JobSearch/generate-traffic.sh

Cron Entry

*/15 * * * * /home/victor/logrite/projects/JobSearch/generate-traffic.sh >> /home/victor/logrite/projects/JobSearch/traffic.log 2>&1

To remove the feature entirely: delete the crontab line and the script.

Adding a scheduled task

Write a systemd unit and a timer; do not add a crontab entry. A unit gets you logs in journalctl, a recorded exit status, and a failed state something can alert on — a cron line gets you an email nobody reads.

/etc/systemd/system/logrite-example.service

[Unit]
Description=LogRite example task
[Service]
Type=oneshot
WorkingDirectory=/opt/logrite-ops
ExecStart=/usr/bin/python3 /opt/logrite-ops/example.py

/etc/systemd/system/logrite-example.timer

[Unit]
Description=Run the LogRite example task hourly
[Timer]
OnCalendar=hourly
Persistent=true          # catch up a run missed while the box was down
[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now logrite-example.timer
sudo systemctl start logrite-example.service   # prove it works before waiting an hour

Caution

Make it exit non-zero AND say why. status_report.py returned 1 from a branch with no message at all, so systemctl start produced exit 1 with empty stdout and stderr — a failure that reports nothing is one nobody can act on, and it sat that way for two days.

Summary

Task	Where it lives now	Schedule
Deploy watcher	logrite-deploy-watcher.service	continuous, 60s poll
Management console	logrite-management.service	continuous
Hourly status roll-up	logrite-status-report.timer	hourly
Disk guard	logrite-disk-guard.timer	hourly
Model health check	gone — the console’s overview answers this continuously	—
Deployment notifier	gone — deploy notifications are emitted by the thing that performs the deploy (ops/management/notifier.py), so they cannot outlive a control plane being removed	—