REVIEW 0.2.0-rc3
Evolving Dark Energy or Supernovae Systematics?
2408.07175v3 · G. Efstathiou
Reproduced in scope
The supernova fits and common-sample magnitude offset reproduce under the inferred distance convention and stated prior variants.
Limit The offset does not establish a full-sample calibration error; correlated cross-catalogue systematic uncertainty remains unmeasured.
Full report Checking method and assumptions Code and evidence package
What these results mean
- Reproduced in scope: the named checks support the reported result, with the stated exceptions; not a certification of the whole paper.
- Partly reproduced: some named results reproduce while important parts remain unverified.
- Confirmed finding: the specific discrepancy has author or independent researcher confirmation; downstream consequences may remain unknown.
- Technical finding — not externally confirmed: a supported, explicitly limited discrepancy without external confirmation.
- Attempt documented: the completed work and reasons it stopped are recorded. Failure to finish does not establish a paper error.
What we checked
- Supernova sample counts, selected cosmological fits, magnitude offsets among shared objects and printed numerical comparisons.
- The role of the distance convention and selected priors; not a full correlated calibration analysis.
| Check | How it was checked | Result | Evidence assessment | Evidence |
|---|---|---|---|---|
| The selected supernova counts and five-year Dark Energy Survey sample composition reproduce. Report reference: C1 | Checked against source material | Reproduced in this check | Supports this assessment | Evidence 1 |
| The fits reproduce under an inferred luminosity-distance convention that the paper does not state. An alternative redshift convention misses one printed fit improvement. Report reference: C2 | Checked against source material | Reproduced in this check | Supports this assessment | Evidence 1 |
| Fit statistics and improvements reproduce within tolerance. Some printed best-fitting points lie slightly above the minima along a nearly flat parameter direction. Report reference: C3 | Checked against source material | Reproduced in this check | Supports this assessment | Evidence 1 |
| The stated approximate significance conversions are supported within their stated context and parameter counts. Report reference: C4 | Checked against source material | Reproduced in this check | Supports this assessment | Evidence 1 |
| The Planck comparison reproduces when the prior includes the plotted contours. One printed parameter range cannot have produced that plot. Report reference: C5 | Checked against source material | Difference identified | Supports this assessment | Evidence 1 |
| Shared-object counts and mean offsets reproduce. The quoted offset error covers scatter only; several table errors and sample-description slips remain. Report reference: C6 | Checked against source material | Difference identified | Supports this assessment | Evidence 1 |
| After shifting the Dark Energy Survey sample, no significant preference for evolving dark energy remains in this check. The printed point is not the minimum; the fit improvement is 2.89 rather than the legend's 2.3. Report reference: C7 | Checked against source material | Difference identified | Supports this assessment | Evidence 1 |
| Appendix statements hold under a flat prior on the Hubble constant. That unstated prior affects the reported comparison level. Report reference: C8 | Checked against source material | Difference identified | Supports this assessment | Evidence 1 |
| Most quoted numbers match. One decimal-place typo remains, and two sample offsets with different weighting and membership are compared as though they were the same quantity. Report reference: C9 | Checked against source material | Difference identified | Supports this assessment | Evidence 1 |
“Supports this assessment” means the evidence supports the stated result, including a discrepancy or limitation. It does not mean the paper's claim passed.
Run the packaged code
mkdir replay-work && python3 -m zipfile -e legacy-original.zip replay-work && cd replay-work && python3 reproduce.py --allUse the package instructions for dependencies, data and limits. This command repeats the packaged checks; it does not necessarily reproduce the whole paper.
AI used and recorded resources
Partial records · 8/8 sessions with complete token counts · 8/8 with activity time recorded
OpenAI Codex · gpt-6-astra · reasoning: high
Anthropic Claude · claude-opus-5-5 · reasoning: high · settings reported by the project owner
Token breakdown and what the time measures
- New input
- 754.6K
- Input read from cache
- 97.2M
- Input written to cache
- 1.2M
- Output, including reasoning
- 394.4K
- Recorded agent activity
- 4.7 h
- Scientific computing time
- Not established by these records
- Human time
- Not recorded
Recorded case work before 2026-11-10; later shared collection preparation is excluded. Tokens count processed traffic, including repeated cached input, not unique text or a monetary cost. Reasoning tokens are already part of output.
Session time sums the recorded session windows, including waits and parallel sessions. Activity time sums only explicitly logged activity durations and may include tool waits. Neither is elapsed project time, scientific CPU/GPU time, or human labour. Missing, estimated or overlapping records are not silently counted as zero or added to measured totals. 0 records are excluded from sums; remaining records can still lack particular counters. This is a partial accounting snapshot, not a full cost estimate.
Claude contributed the first pass; Codex contributed separate checks and criticism. Later packaging and informed corrections are distinguished in the audit record. A separate AI check is not external scientific confirmation.
Technical scope, original wording and review history
Reference labels such as C1 or N3 identify checks and notes in the report, not physical quantities. Workflow letters denote the first pass (A), separate checking (B) and criticism (C). P2 means package validation; G2 means the rule to commit a plan before scientific execution. V-numbers identify internal workflow steps. S0–S3 are internal issue-severity labels, not statistical significances.
Original scope and exclusions
- Sample selection and counts for Pantheon+ and DES5Y (C1)
- χ² of the Planck ΛCDM cosmology and the ΛCDM fits to both catalogues (C2)
- The w0waCDM fits and their χ² improvements (C3)
- The significance statements in the text and abstract (C4)
- The supernova-only posteriors and where Planck ΛCDM lies in them (C5)
- Table 1: the magnitude offsets of supernovae common to both catalogues, and the low-redshift minus DES difference (C6)
- DES5Y with 0.04 mag subtracted from its low-redshift supernovae: fits and posterior (C7)
- Appendix A: DESI DR1 BAO alone and combined with each supernova catalogue (C8)
- Numbers quoted from Planck 2018, DESI 2024 and the DES analysis of the offset (Vincenzi et al. 2025) (C9)
Not checked: Whether the offset seen in the common supernovae implies a calibration error in the full DES5Y sample. That needs the light curves, bias-correction simulations and cross-catalogue systematic covariance, which the public distance tables do not carry. The paper's 'can bring … into good agreement' is conditional, and the shift experiment supports it; The DES analysis's own decomposition of the offset (selection, host-mass and intrinsic-scatter terms), beyond checking that the quoted numbers match their source; The statement that binned and unbinned likelihoods give nearly identical results (l.257), and the attribution of the remaining asymmetry of the shifted posterior to supernovae at z > 1 (l.425–426); The paper's own chains and plotting settings, which are not public. Our posteriors are grids over the prior box; for the Pantheon+ posterior we used a w_a range that contains the plotted contours (note N2); The ± 0.008 mag on the offset is the scatter of the common supernovae only, without correlated systematic errors between the catalogues, so it is not a calibrated significance
Original claim wording
C1 · C1 in fixed reviewed version: Counts and DES5Y composition exact (1417 and 1754 after the cuts; A and B independently)
C2 · C2 in fixed reviewed version: Fidelity gate passed under D_L = (1+zHD) D_C(zHD), the paper's unstated convention; (1+zHEL) misses the DES5Y Delta chi2 (5.49 vs 5.1) (N1)
C3 · C3 in fixed reviewed version: w0waCDM chi2 and improvements within tolerance; printed best-fit points 0.14/0.15 above the minima along the flat direction (N3)
C4 · C4 in fixed reviewed version: ~2.2 sigma 2.15 (printed numbers) / 1.98; roughly 3 sigma 2.76 (k=3); ~2 sigma 2.29 (k=2)
C5 · C5 in fixed reviewed version: Planck point inside 1 sigma for Pantheon+ with a wa range containing the plotted contours (0.429/0.383/0.070); the printed range [-4,-0.4] cannot have been used (N2); DES5Y thresholds pass
C6 · C6 in fixed reviewed version: All counts and means reproduce, differential -0.0360 +- 0.0080 (scatter only); Table 1 DES/FOUND errors x10 typo, three z_median, '150 Foundation' (N4-N6)
C7 · C7 in fixed reviewed version: Shifted DES5Y: LCDM Omega_m 0.310, no significant preference (1.19 sigma); printed w0wa point not a minimum, Delta chi2 2.89 vs legend 2.3 (N7)
C8 · C8 in fixed reviewed version: Appendix statements hold under the flat-H0 prior; unstated H0 prior moves the DESI+Pantheon+ w0-wa level 0.74 -> 0.84 (N10; protocol rule S0)
C9 · C9 in fixed reviewed version: 12 of 13 quoted numbers match; 0.0344 for 0.344 (N8); V25's weighted Foundation-DES -0.042 compared with the unweighted all-low-z offset (N9)
Scientific abbreviations
DESI: Dark Energy Spectroscopic Instrument; BAO: baryon acoustic oscillations; SN: supernovae; CMB: cosmic microwave background; DESY5/DES5Y/DY5: Dark Energy Survey five-year supernova sample; PP: Pantheon+; PD: Planck + DESI; PDPP: Planck + DESI + Pantheon+. CPL, JBP and BA name the Chevallier–Polarski–Linder, Jassal–Bagla–Padmanabhan and Barboza–Alcaniz dark-energy parametrizations. EXP and LOG mean exponential and logarithmic parametrizations. Full-plik refers to the full Planck high-multipole likelihood; lite is its compressed approximation. ESS is effective sample size; IC means initial conditions; MG means modified gravity. NEC is the null energy condition; GR is general relativity; FLRW denotes Friedmann–Lemaître–Robertson–Walker cosmology. GB means Gauss–Bonnet. AIC and DIC are model-comparison information criteria. Further symbols are defined in the linked paper and protocol.
See original source-payload README and recorded P2; not rerun during assembly. Unpacks the unchanged original candidate into a new work directory and runs its documented scoped replay. Package validation checked the artifact and inherited the earlier replay after verifying the relevant files were unchanged; it did not rerun the scientific calculation.
Original provenance disclosure
Prepared with AI assistance. Historical Claude first pass and Codex independent review/critique remain distinct; this assembly reads saved evidence and is neither blind B nor new P2. No scientific code was run in this assembly. Before the replay command below, extract legacy-original.zip into a new directory and read its README.md and requirements.txt for installation, dependency, data and scope instructions.
Tools: Claude (historical A); Codex (historical B/C and current informed assembly)
Human involvement: Owner selected the preparation scope and approved the predecessor release published on 10 November 2026; the owner approved publication of this editorial revision on 11 November 2026