SOP-12 — Backups, disk, recovery and off-site copies

Owner: Owner-operator (Document Owner and Records Owner for every SOP; no delegate exists). Version 2026-10-26 · effective 2026-09-28 · reviewed every 180 days · last reviewed 2026-10-26 · next review due 2027-04-24.

Purpose

Keep the registry recoverable, keep an off-site retention-locked copy of what was registered each day, and keep the host from filling up (a full disk stopped the database once).

Scope

MongoDB data, local backups, media and held files, and the daily off-site manifest copy.

Procedure

1. Provider backs up MongoDB. The platform keeps rolling local backups of its own collections under a size cap; Sentinel prunes oldest first and raises disk_nearly_full when pruning is not enough.
2. Held public-record files are re-fetchable from source and are not the system of record.
3. Recovery = redeploy from version control + latest database backup. Both have been rehearsed: redeploy (routine) and a restore drill on 2026-10-06 (backend/tests/restore_drill.py: latest archive restored into a scratch database, 239 collections / 233,562 documents in 9.2 s, 300 of 300 restored receipt bodies re-hashed to their registry hash, scratch dropped). The drill runs against the local backup set in the preview environment; a restore from the provider's production backup has not been exercised and remains open.
4. Restore drill cadence: quarterly, and before any annual agreement is signed; each run's figures are written to this change log. Drop any leftover scratch database before a run — a half-finished scratch made a second attempt fail on index conflicts.
5. Provider-backup restore rehearsal: once a quarter the owner restores the most recent hosting-provider backup into a scratch database from the provider console, runs the same count-and-hash comparison as the local drill, and records date, backup id and result here. The build environment has no provider-console access; this step is the owner's and is open until the first recorded result.
6. Off-site copy, daily at 05:40 UTC (job offsite_copy_daily, backend/wormexport.py): the platform writes one file per day — a canonical-JSON manifest of every receipt registered that day (id, SHA-256, time, signature state) plus the newest signed transparency-log head — to a Backblaze B2 bucket (S3 API, us-east-005, bucket owned by Alpha Covenant Holdings LLC, Object Lock enabled, server-side encryption on) with Object Lock COMPLIANCE retention of WORM_RETENTION_YEARS (7). Before recording 'stored' the job reads the retention back from the store; the row (day, manifest hash, key, version id, lock mode, retain-until) is written to db.offsite_copies and published at /api/public/offsite-copies. The manifest contains no personal data beyond what the public registry already prints; receipt bodies are not copied.
7. What the copy proves and does not: it is a dated, locked record of which receipt hashes existed on which day, independent of this host and this database. It does not by itself restore receipt bodies (those come from the database backup); it lets a restored body be checked against the hash that was locked off-site. The lock is stated as a fact about the store, with its mode and date; the words 'immutable', 'WORM-compliant', 'SEC 17a-4' and 'FINRA 4511' are not used.
8. Failure handling: a run that cannot write records status 'failed' with the error class on the same public row; the manifest hash is still published from the registry so the day is never silent. If the store rejects a write for two consecutive days the owner reviews the key, bucket and lock settings in the Backblaze console; the platform never retries with a different bucket. Credentials live only in environment settings (SOP-09 / Secrets policy); the Backblaze master key is not used by the platform and the owner rotated it on 2026-10-25 after it appeared in a screenshot.
9. Verification from outside: anyone may fetch /api/public/offsite-copies/manifest/{day}.json, hash it with SHA-256 and compare to manifest_sha256 on the row; the owner may additionally confirm key, size and retain-until in the Backblaze console. First stored copy: 2026-10-24 manifest (95 receipts), written 2026-10-25 17:23 UTC, COMPLIANCE lock to 2033-10-23, read back with matching hash, delete attempt refused by the store.

Records this procedure produces — and who keeps them current

Evidence

/audit · /api/trust/policies/security · /api/public/offsite-copies

Change log

Quality Manual · All SOPs · Records Register