SOP-08 — Monitoring and availability measurement (Sentinel)
Owner: Owner-operator (Document Owner and Records Owner for every SOP; no delegate exists). Version 2026-10-09 · effective 2026-09-25 · reviewed every 180 days · last reviewed 2026-10-09 · next review due 2027-04-07.
Purpose
Detect and, where safe, heal platform faults; measure availability honestly for the agreements that quote it.
Scope
Services, jobs, disk, sources, disclosure surfaces, SOP review dates.
Procedure
- Every 15 minutes: check database reachability, scheduler backlog, stuck jobs, disk use (prune under cap, then raise disk_nearly_full), held files not read, live endpoints down, Daily Replay slot, self-arming jobs, telemetry roll-up, SOP review dates.
- Heal what is safe (re-queue, re-arm, prune, re-submit OTS). Never heal by fabricating a source result.
- Record every run and finding to db.sentinel_runs; publish at /audit. Email the owner on new findings.
- Availability for the annual agreements is the fraction of Sentinel runs in the month where the roster/receipt endpoints answered; it is published, not promised.
- Sources board (/api/public-record/screening-sources, shown on /, /methodology, /trust and the reviewer pages): the last snapshot is served at once and the twenty live endpoints are re-probed in the background about every ten minutes; the snapshot is kept in db.sourcesboard_snapshots so a freshly started service answers instantly. A public page never waits on a government server; checked_at says when the probe ran.
- Bootstrap state: each run reads the process's startup record; 'failed' raises bootstrap_failed with the error text, because a failed bootstrap leaves scheduled jobs unarmed while every page still answers.
- Government Data Status (second product, same telemetry): the 15-minute `down` list is republished at /government-data-status as incidents (start, end, duration) with the vantage stated on every output — one reader, one key, one network path; not an agency's own status; no cause inferred. Alerts: free daily digest (double opt-in) and Pro ($19/month, Stripe subscription, source 'govdata') with e-mail + webhook within 15 minutes of open/close and CSV history to 90 days. Webhook deliveries are logged with the HTTP status; a failed post is recorded, not retried silently. Every e-mail carries an unsubscribe link that works without login.
Records this procedure produces — and who keeps them current
| Record | Where | Updated by | Update trigger | Retention | Public |
|---|
| Sentinel run and findings | db.sentinel_runs (/api/audit/sentinel) | system | every 15 min | indefinite | True |
| Sources board snapshot | db.sourcesboard_snapshots | system | ~10 min | latest only | True |
| Government Data Status subscribers, incidents, webhook log | db.govdata_subscribers; db.govdata_incidents; db.govdata_webhook_log; db.govdata_state | system | 15 min / daily | subscribers until unsubscribed; incidents indefinite | False |
Evidence
/audit · /api/audit/sentinel · /api/public-record/screening-sources
Change log
| Date | Change |
|---|
| 2026-10-06 | Written as SOP; sop_review_overdue finding added. |
| 2026-10-06 | Incident: a reviewer reported /api/record/trust failing to load in three review cycles. Cause: the sources board probed 20 live endpoints inside the request when its 10-minute cache had expired (measured 16.4 s cold, 0.2 s warm), long enough for clients to give up. Fix: serve last snapshot, refresh in background, persist snapshot, warm at start. |
| 2026-10-09 | bootstrap_failed check added after the 2026-10-09 index incident (SOP-09). |
| 2026-10-09 | Owner: produce other revenue streams. Government Data Status launched from Sentinel's own telemetry (backend/govdatastatus.py). Nothing new is read from any agency; the product is a view of checks we already make. Doctrine holds: counts and dates, vantage stated, no inference about the agency. |
Quality Manual · All SOPs · Records Register