Experiments#
The journal is the running log. These are the experiments proper: each one is a standalone, reproducible write-up that someone else (or future me) could re-run and check.
How an experiment is written up#
One file per experiment, same section order, so they are comparable:
Question: the hypothesis, in one or two sentences. What would confirm it, what would falsify it.
Why it matters: the assumption being tested, not the machinery.
Subject under test: model, backend, system version (pin the commit), paradigm, temperature. Everything needed to know what was measured.
Reproduction: exact commands. Copy-paste to stand it up.
Variables: independent (what is changed), dependent (what is observed, with operational definitions), controlled (what is held fixed).
Protocol: how a run is conducted, sample size N, how outcomes are classified. Decide this before running.
Threats to validity: confounds and limits, stated plainly.
Results: the data, tagged with N. Single runs are anecdotes; only rates are results.
Interpretation: tentative, separated from the data.
Status / next: what is settled, what is pending.
Log: dated lab-notebook entries as the experiment progresses.
Two house rules, learned the hard way:
No conclusions from n=1. At non-zero temperature a single run flips run-to-run. Replicate, then report the split.
Falsify, do not assert. “It did not work here” is not a refutation of the idea; check whether the mechanism was even present.