Goal
Does cmd/verify find real leaks in real court filings, and how often does a filing that has been
redacted still leak?
Method
- CourtListener’s RECAP archive, public search API: documents whose docket-entry description contains “redacted”, available as a PDF, newest first, at most 60 pages. The query names five districts; the 200 documents came from four — S.D.N.Y. 106, D.D.C. 43, C.D. Cal. 32, N.D. Ill. 19 — and S.D. Tex. gave none.
- One
cmd/verify -f <file>per document, no target list: redaction-classes and metadata only. - Every failure was opened and read by eye. The first pass failed 21 documents; what each one was is in recap-failures.tsv.
RECAP rerun
None of the 200 filings has more than one revision. Each contains exactly one %%EOF and one
startxref; the four /Prev matches are outline entries. So no filing fails prior-revision, and there
was nothing to spot-check: this class has never fired on a real filing. With prior-revision added
the result was 195 pass and 5 fail, the same 5 as the run before it. The table below is from a
rerun on 2026-10-26, after the white-overwrite exemption was withdrawn and with strokes, including
an /Ink annotation’s, counted as covers. It fails the 12 filings named below and no others.
Results
| Checked | Pass | Fail | Refused | Error |
|---|---|---|---|---|
| 200 | 188 | 12 | 0 | 0 |
- 2 are real leaks, and they are one document. A chart of search hits was filed twice, as two different files. Black bars sit over email addresses, and the text is still under the bars, in a font the tool cannot read as text. Copy-paste gives control codes; nobody here tried to recover the addresses from the glyph shapes.
- 3 are not leaks. Each is a signature image over the printed name and title it reproduces.
- 7 are not leaks. Each is a white box with text drawn on top, over glyphs the filer did not
mean to hide. 498111275 and 498080982 are one byte stream: a D.D.C. form whose
____ District of ____line is overwritten withDistrict of Columbia(30 characters under the box). 497288353 and 498155703 are two DOJ letters on one letterhead template, whose white box sits over 2 characters of template residue. 497565661, 497565667 and 497565668 carry one memo endorsement, whose box covers the/of the letter’s/s/signature (1 character).
The 200 files are 186 distinct byte streams, and some filings repeat under different bytes, so about 180 distinct documents were checked: 1 leaking document in about 180.
Limitations
Image content, OCR layers, form fields. So the 3 signature fails and the 7 white-box fails stay fails until a person looks, and a leak inside a scanned image passes. The rate is among filings whose filer called them redacted, not all filings: a description match says the filer redacted something, not that a cover was painted.
