Reproduce the GWAS Catalog pigmentation pull

Result: output/catalog/pigmentation_gwas_catalog.csv (1,072 lead SNPs). Unlike 01a–01c this is a live network pull, not an offline raw-file transform: the raw source is the NHGRI-EBI GWAS Catalog download endpoint, queried by trait ontology. The derivation logic lives in scripts/gwas_catalog.py (config-driven, provenance-grade); this notebook is the thin, auditable entry point that runs it and asserts the payoff anchors.

Provenance (from docs/specs/gwas_catalog.spec.md): 10 frozen EFO/OBA/MONDO roots in scripts/traits_pigmentation.json, includeChildTraits=true. The catalog is a live resource, so the access timestamp is the version key; the frozen copy committed here was pulled 2026-07-08T01:15:41Z and is what the assertions below check. Re-running the pull live will produce a newer snapshot (more rows as the catalog grows) — that is expected; compare against the frozen .meta.json, don’t expect byte-identity.

Key terms — so this notebook stands on its own (you shouldn’t need the other notebooks to read this one).

  • GWAS Catalog — the NHGRI-EBI GWAS Catalog, a public database of published genome-wide association study (GWAS) results. This notebook pulls pigmentation hits from its live download endpoint by querying for trait terms.
  • Lead SNP — the single top-associated genetic variant that stands in for an association signal at a locus (the reported “hit”). The pull returns 1,072 of them.
  • Trait ontology roots / includeChildTraits — standardized trait/disease vocabularies (EFO, MONDO, etc.) that give each phenotype a stable ID. The pull names a handful of pigmentation “root” terms and sets includeChildTraits=true so every sub-trait beneath them is swept in, so no pigmentation phenotype is missed.
  • Frozen pull / snapshot (timestamp as the version key) — because the Catalog is a live, growing resource with no fixed release number, the code saves a dated copy (“frozen”) and treats the query timestamp (queried_utc) as its version. Downstream notebooks read this frozen copy so the input is deterministic offline; a fresh live re-run will legitimately return more rows.
  • .meta.json — a sidecar file recording when and how the pull was made. Reproducibility is checked against this metadata, not by expecting a byte-identical file.
  • Payoff anchors / assertions — specific expected values (e.g. the row count and named SNPs) the notebook re-checks against the frozen snapshot; if the assertions pass, the pull is confirmed reproduced.
  • 01a–01c — sibling notebooks that transform static, already-downloaded files offline. Unlike them, this notebook (01d) performs a live network query to build its output.

Step 1 — Load the frozen pull and assert its provenance + anchors

We validate the committed frozen snapshot. To regenerate a fresh pull, run python scripts/gwas_catalog.py (requires network to www.ebi.ac.uk); the script stamps a new queried_utc and archives the snapshot. The frozen file is the reproducible input for all downstream notebooks so the network is deterministic offline.

Show code 
import pandas as pd, json
FROZEN = "output/catalog/pigmentation_gwas_catalog.csv"
META   = "output/catalog/pigmentation_gwas_catalog.csv.meta.json"
cat = pd.read_csv(FROZEN, dtype={"pvalue": str})   # p-values kept as STRINGS (underflow-safe)
meta = json.load(open(META))
print("rows:", len(cat), "| columns:", len(cat.columns))
print("frozen access timestamp (version key):", meta.get("queried_utc") or meta.get("access_utc") or meta)
assert len(cat) == 1072, f"expected 1072 frozen rows, got {len(cat)}"
Show code 
# Anchor assertions — the pull aborts if any is absent (payoff loci + a skin-colour control)
ANCHORS = {"rs1426654":"SLC24A5", "rs12913832":"HERC2", "rs1805007":"MC1R"}
for rs, gene in ANCHORS.items():
    assert (cat["rsid"] == rs).any(), f"anchor {rs} ({gene}) missing from pull"
print("all anchors present:", ANCHORS)
# payoff-locus association counts (from the spec):
for g in ["MC1R","OCA2","HERC2"]:
    print(f"  {g}: {(cat['mapped_gene'].astype(str).str.contains(g, na=False)).sum()} associations")
Back to top