Sample-Size Planner
How many runs a test needs before its answer means anything, and what the runs you can afford will actually show. Three tries each is almost never enough.
Inputs
Averages
From a few practice runs; Descriptive Stats works it out. In the same unit as the measurement.
Same unit as the measurement.
Same unit. A difference too small to change a match result isn't worth testing for.
Success rates
A guess is fine; 50% is the cautious choice if you have no idea.
The gap between the two is what the test has to detect.
How sure
How often the test may report a difference that isn't really there.
How often the test should detect the difference, when it is really there.
Used for the reverse question: what that many runs can show.
Results
Runs needed
—
Runs in total
—
With the runs you can do
—
Is that enough?
—
Show how the plan changes
How the number of runs changes as the question gets finer.
Add runs
How well you know the answer, run by run, for the question you picked. The curve moves steeply at first and then flattens out: the first few extra runs buy a lot, and later ones buy less and less, because precision grows with the square root of the runs. The dashed line across is what you asked for; the dashed line up is the number of runs that gets you there.
How this is calculated
Every test is fighting the same thing: runs vary. Two setups can look different after three tries purely by luck. More runs shrink the luck, but slowly, because the uncertainty in an average falls with thesquare root of the number of runs. To halve it you need four times as many.
Pinning down an average
The smallest number of runs whose 95% interval is no wider than the margin you asked for. It uses the t distribution rather than the normal one, because at the sample sizes a team can afford the two differ, and the normal shortcut promises more than it delivers.
margin = t × spread ÷ √runsComparing two setups
Two things can go wrong. The test can report a difference that isn't real (a false alarm), or miss one that is. The planner holds the first to 5% and catches a real difference 80% of the time, the usual choices in science and engineering, and finds how many runs each setup needs for both.
runs per setup ≈ 2 × (t₁ + t₂)² × (spread ÷ difference)²The ratio in that formula is the heart of it. Halve the difference you care about and you need four times the runs. Make the robot more consistent and you need fewer.
With your numbers
Success rates
Rates need far more attempts than times do, because each attempt only says yes or no. Telling 70% from 85% reliably takes well over a hundred attempts per setup. That isn't the planner being cautious; it is how little a single yes-or-no carries.
Make a notebook page
Turns this calculation into a dated engineering notebook page: the setup, data, results, figures, working and sources, numbered and cited. You add the goal, the decision and the reflection in your own words, plus photos and sketches. Print it, paste it into Google Docs, or download a Word file or slides.
Save this run, and compare
Keeps what is on screen so you can change something and see both sides of the change. Saved in this browser only, never uploaded.
Sources & assumptions
Methods. Averages: the t-based interval, and the two-sample t-test planned by the iterated-t form of the standard formula. Rates: the Wilson interval (the same one the reliability tools report), and the pooled two-proportion test without continuity correction.
Checked by simulation (scripts/verify-stats.mjs): running each planned test twelve thousand times detects the difference 80.3% to 81.3% of the time for averages and 80.3% to 83.2% for rates, where the target is 80%. Near 100% success the rate plan runs a few attempts long, which is the safe direction to be wrong in. A plain normal-theory plan, without the t correction, reached only 72.6% in the same test.
Assumes independent runs under the same conditions. A battery draining through the session breaks that assumption quietly.Confidence intervals explain what the percentages mean.
Where this goes next
Explore further on the full mapWhere the answers on this page can be used, and which tools could have given you the numbers it asks for.
Comes after
Descriptive Statsfills in your run-to-run spread herePID ComparisonCompare Test VersionsOutlier DetectorYou are here
Sample-Size Planner
a number carries across the next question it raises