SOP-03 — Source onboarding and held-file refresh

Owner: Owner-operator (Document Owner and Records Owner for every SOP; no delegate exists). Version 2026-10-08 · effective 2026-10-04 · reviewed every 90 days · last reviewed 2026-10-08 · next review due 2027-01-06.

Purpose

Add only sources read from the publishing agency's own file or endpoint; keep held copies dated, hashed and refreshed on a stated cadence.

Scope

Every check listed on /methodology; held files (LEIE, SAM extract, state lists, UFLPA, 1260H, FARA, PHMSA, NRC, UN) and live endpoints.

Procedure

A new source needs: the agency's own URL; a parser that keeps rows as printed; a match method written in words; a not_read state for failure; a knownlimits entry; a changewatch TRACKED entry; a filehistory entry; a sourcesboard line.
Sources reachable only by bypassing a bot wall, CAPTCHA or access control are not added; they are listed as not read with the reason (FERC, CBP EAPA, OSHA SVEP, FCC Covered List).
Held files are fetched on their job cadence (24 h for federal lists), hashed (file_sha256), and stored with the source's Last-Modified or printed date. A fetch that fails leaves the previous copy in place and marks the file not_read until the next success; receipts built meanwhile print the held copy's date so the reader can see its age.
changewatch records every change in a held file's hash as a public source-change event and triggers recurring rosters within the hour.
Removing a source requires 30 days' notice on /methodology and in the SOP change log (/api/trust/sops/changes) unless the agency withdrew it.

Records this procedure produces — and who keeps them current

Record	Where	Updated by	Update trigger	Retention	Public
Held file copy with sha and date	`db.fedlists_meta, db.<source>_rows, filehistory`	system on each fetch	job cadence	current + change history	True
Source-change log	`db.source_changes (/api/audit/source-changes)`	system	any hash change	indefinite	True
Known limits	`knownlimits.py (/api/public-record/known-limits)`	owner	any source added, removed or found unreadable	indefinite	True

Evidence

/methodology · /api/audit/source-changes · /api/public-record/known-limits

Change log

Date	Change
2026-10-06	Written as SOP.
2026-10-06	Source-removal notice now points to /methodology and the SOP change log; the /changelog page it cited does not exist.
2026-10-07	Source freeze adopted (reviewer, owner): no new held file or lookup without a named buyer. Exception made the same day with the buyer named — federal construction / prevailing wage column set (DOL Wage & Hour enforcement cases by printed name, official API) for PennDOT and federal-aid primes; the DOL debarment list and OSHA Severe Violator log refuse automated reads (HTTP 403) and are printed not read with the official link, never as no record. The free DOL API key returned HTTP 429 during the first production-style test; that state prints as did not answer — owner action: DOL Open Data key with higher quota.
2026-10-08	Reference reads added that are not party lookups and so sit outside the source freeze: eCFR versioner API (regulation text as served, amendment dates, hash) and Federal Register API (documents naming a form number), cached 7 days / 24 h; FAC finding counts by type and year from the existing FAC source. Governed by SOP-15.

Quality Manual · All SOPs · Records Register