Goal. Assemble and verify the 13-paper genotype→phenotype-discordance case set that the downstream analysis validates against: load each paper’s per-record extraction table, load the direction classification, and check that both reproduce exactly from the committed files — no re-extraction, no network, no withheld inputs.
Inputs (all committed, all reachable without the withheld publisher PDFs):
data/processed/discordance_case_classification.csv — one row per paper: DOI, phenotype system, discordance direction, and the n_records_extracted count.
data/case_records/EXTRACT_<paper>_records.csv (13 files) — the machine-readable per-record table for each paper (schema: gene, variant, rsid, genotype_or_zygosity, phenotype, effect_or_association, population, source_table).
What this notebook is not. It does not re-run the extraction — that step reads the withheld publisher PDFs and is frozen (shown, not executed) in docs/NB3_case_assembly_provenance.md §2–3. This notebook is the machine-checkable half of that provenance record: it re-derives the headline numbers (694 records; 3 D1 / 5 D2 / 5 both) from the files actually committed to the repository, so the claim is auditable by anyone who checks the repo out, with no paper access required.
Provenance. Full acquisition, extraction methodology, the md→PDF correction, and the classification scheme are documented in docs/NB3_case_assembly_provenance.md. This notebook cites that document rather than repeating it, and focuses on the reproducibility check.
Key terms — so this notebook stands on its own (you shouldn’t need the other notebooks to read this one).
Genotype→phenotype-discordance case set (the “validation-case set”) — 13 published pigmentation papers, hand-curated for this project, in which the genetics and the pigmentation phenotype do not line up the way the textbook would predict. This notebook only verifies that the curated tables reproduce from the committed files; the set itself is the empirical base that the project’s later “rescue” analysis — connecting hard-to-explain pigmentation loci into the melanin-making gene network — is validated against.
Discordance direction — D1 / D2 / both — how each paper’s genotype–phenotype mismatch runs (defined in the provenance doc, restated here). D1: a person carries the usual causal variant but does not show the expected phenotype (reduced/incomplete penetrance). D2: the phenotype is present but the usual causal variant is absent (a different gene, an atypical genotype, or a compound/single-allele cause). both: the paper documents cases in each direction, often at one locus. The headline “3 D1 / 5 D2 / 5 both” is the count of papers in each category.
Withheld publisher PDFs / “frozen” extraction — the source papers are copyrighted and are not committed to the repo, so the step that reads them to build the per-paper record tables cannot be re-run here; it is shown-but-not-executed (“frozen”) in docs/NB3_case_assembly_provenance.md. This notebook deliberately touches only the committed extraction outputs, so anyone can audit the headline numbers without any paper access or network.
EXTRACT_<paper>_records.csv / n_records_extracted — one machine-readable table per paper listing the extracted genetic records (grain: one variant × source table); n_records_extracted is that paper’s claimed row count stored in the classification table, which Step 3 checks against the actual row count of the committed file.
Setup
Load libraries and resolve the repo root so the notebook runs the same way whether launched from notebooks/ or from the repo root (matches the convention in Notebook 2).
Show code
import pandas as pdfrom pathlib import PathROOT = Path.cwd().parent if Path.cwd().name =="notebooks"else Path.cwd()PROC = ROOT /"data"/"processed"RECORDS = ROOT /"data"/"case_records"CLASSIFICATION_CSV = PROC /"discordance_case_classification.csv"assert CLASSIFICATION_CSV.exists(), f"missing {CLASSIFICATION_CSV}"assert RECORDS.is_dir(), f"missing {RECORDS}"print("ROOT resolved:", ROOT.name) # repo folder name only — avoid leaking a local absolute pathprint("classification file:", CLASSIFICATION_CSV.name)print("case_records dir:", RECORDS.name)
discordance_case_classification.csv carries one row per paper: the DOI, phenotype system, discordance-direction call (D1 / D2 / both, defined in docs/NB3_case_assembly_provenance.md §4), and n_records_extracted — the row count of that paper’s committed records CSV at the time the classification was pinned.
Step 2 — Load every per-paper records CSV and count rows
For each paper, find its committed EXTRACT_<paper>*_records.csv under data/case_records/ (some filenames carry a _records suffix, some don’t — the classification table’s extract_artifact column names the paper’s .md extract; the matching .csv sits alongside it) and read the actual row count. This is the independent check: n_records_extracted in the classification table is a claim; the row count of the committed file is the fact.
Show code
def committed_rows(paper: str) ->tuple[int, str]:# Row count of paper's committed records CSV(s) under data/case_records/, and the filename(s) used. hits =sorted(RECORDS.glob(f"EXTRACT_{paper}*.csv"))ifnot hits:raiseFileNotFoundError(f"no committed records CSV found for {paper!r} under {RECORDS}") total =sum(len(pd.read_csv(h)) for h in hits) names =", ".join(h.name for h in hits)return total, namescls[["committed_rows", "committed_file"]] = cls["paper"].apply(lambda p: pd.Series(committed_rows(p)))cls[["paper", "n_records_extracted", "committed_rows", "committed_file"]]
Step 3 — Verify: does n_records_extracted match the committed file, paper by paper?
This is the reconciliation check documented in docs/NB3_case_assembly_provenance.md §6. It must pass with zero mismatches — if it doesn’t, the classification table is out of sync with the committed record files and needs to be re-pinned before anything downstream trusts it.
Show code
mismatch = cls.loc[cls["n_records_extracted"] != cls["committed_rows"]]assert mismatch.empty, (f"row-count mismatch between classification and committed files:\n"f"{mismatch[['paper', 'n_records_extracted', 'committed_rows']]}")print("PASS: n_records_extracted matches the committed records CSV for all", len(cls), "papers")
PASS: n_records_extracted matches the committed records CSV for all 13 papers
Step 4 — Reproduce the headline numbers: 694 records, 3 D1 / 5 D2 / 5 both
Sum the per-paper counts for the canonical record total, and tally discordance_direction for the direction split. Both are asserted against the values reported in docs/NB3_case_assembly_provenance.md §1 and §6 (an earlier “511” headline is retired there — it did not reproduce from the committed files).
Show code
total_records =int(cls["committed_rows"].sum())direction_counts = cls["discordance_direction"].value_counts().to_dict()print("canonical record total:", total_records)print("direction tally:", direction_counts)assert total_records ==694, f"expected 694 total records, got {total_records}"assert direction_counts == {"both": 5, "D2": 5, "D1": 3}, f"expected 3 D1 / 5 D2 / 5 both, got {direction_counts}"print("PASS: 694-record total and 3 D1 / 5 D2 / 5 both split both reproduce from committed files")
canonical record total: 694
direction tally: {'both': 5, 'D2': 5, 'D1': 3}
PASS: 694-record total and 3 D1 / 5 D2 / 5 both split both reproduce from committed files
Step 5 — Per-paper summary table
Paper, DOI, phenotype system, discordance direction, and record count for all 13 cases — the table that documents which published finding backs each entry in the validation-case set. Each DOI resolves directly to the source paper on the publisher site (https://doi.org/<doi>).
Full evidence for each direction call (verbatim quotes, page/table locations) is in the corresponding EXTRACT_<paper>.md under data/case_records/, and the classification scheme (D1 / D2 / both) is defined in docs/NB3_case_assembly_provenance.md §4.
Outstanding provenance sign-offs (cheap, not yet closed)
Two items from docs/NB3_case_assembly_provenance.md §7 remain open. Both are inexpensive to close and are noted here rather than actioned, since they involve editing/retiring files outside this notebook’s own inputs:
Item B — 11 of 13 extracts are missing an md→PDF correction change-log entry. Only Ang 2023 and Abbatangelo 2026 currently record, inside their own EXTRACT_*.md, that an earlier md-derived extraction wave was superseded by a re-read of the authoritative publisher PDF (§3 of the provenance doc). The fix is a one-line change-log entry backfilled into each of the other 11 EXTRACT_*.md files, naming the authoritative source (typeset PDF, pages/tables read) and stating that any md-derived predecessor was superseded. No data changes — this is a documentation backfill.
Items D/E — retire superseded duplicate record CSVs from the artifact store. The two-wave extraction history (§3, §6 of the provenance doc) left md-era and PDF-era record CSVs (plus _stats / _gwas_leads splits for some papers) coexisting in the artifact store. The canonical file per paper is the one committed under data/case_records/ and read by this notebook (Step 2); the reconciliation in Step 3 confirms every canonical file’s row count matches the pinned classification. The remaining action is retention policy, not re-verification: retire or archive the non-canonical duplicates in the store so a downstream reader finds exactly one file per paper. This notebook does not delete anything — store-artifact retirement is a PI-authorized action outside notebook scope; this cell only records that the canonical set is already unambiguous (Steps 2–4 above) and names what should be retired once that authorization is given.
Summary
The 13-paper validation-case set assembled in discordance_case_classification.csv and data/case_records/EXTRACT_*_records.csv reproduces exactly from the committed files, with no paper access and no network required:
Every paper’s n_records_extracted in the classification table matches the row count of its committed records CSV, with zero mismatches (Step 3).
Full acquisition, extraction-methodology, and classification-scheme provenance — including the frozen (unexecuted) per-paper extraction driver that depends on the withheld publisher PDFs — is in docs/NB3_case_assembly_provenance.md.