Question
A museum is trialling eight conservation treatments. It will publish a success claim for any treatment whose pre-specified hypothesis test rejects its null hypothesis. The director wants the probability of making at least one false success claim anywhere in the eight-test programme to be at most 5%.
Assume the test results are independent across treatments, and that each test with a true null hypothesis falsely rejects with probability \(\alpha\). The set of eight tests is fixed before anyone looks at the results.
What is the largest equal per-test significance level the museum can use while maintaining the stated familywise error guarantee? Compare it with the Bonferroni per-test choice, which does not need the independence assumption. Explain why testing every treatment separately at 5% does not satisfy the director's promise.
Get hint
Hint 1
The complement of at least one false rejection is that no true-null test rejects. Independence lets you multiply the no-rejection probabilities.
Check your answer
Solution
The familywise error rate (FWER) is the probability of one or more false rejections across the fixed family, not the error probability for one isolated test.
When all eight null hypotheses are true, independence gives
\[
P(\text{no false rejections})=(1-\alpha)^8,
\quad \mathrm{FWER}=1-(1-\alpha)^8.
\]
Requiring this to be at most 0.05 yields
\[
(1-\alpha)^8\ge0.95
\quad\Longleftrightarrow\quad
\alpha\le\boxed{1-0.95^{1/8}\approx0.00639115}.
\]
This is the exact Šidák threshold under independence. If only \(m\le8\) of the hypotheses are true, their rejection events are still independent under the stated model and \(\mathrm{FWER}=1-(1-\alpha)^m\le1-(1-\alpha)^8\); thus the all-true case is worst.
Bonferroni instead applies the union bound:
\[
\mathrm{FWER}=P(\text{at least one true-null rejection})
\le \sum_{i\in\text{true nulls}}P(\text{reject }i)
\le8\alpha.
\]
It therefore sets \(\alpha_{B}=0.05/8=\boxed{0.00625}\), slightly more conservative but valid even if tests are dependent. At this value, independent all-null tests yield \(1-(1-0.00625)^8\approx0.04892<0.05\).
At a naive 5% level on each of eight independent true-null tests, the overall chance of a false claim is
\[
1-0.95^8\approx\boxed{0.3366},
\]
far above the director's 5% limit. Picking the family after looking at observed results would require a different analysis; the specified family is fixed in advance.
Assumptions
There are exactly eight pre-declared tests, each with type-I error probability \(\alpha\) under its own true null.
The true-null rejection events are independent; Bonferroni is separately discussed without this restriction.
The goal is strong familywise control at 0.05, meaning at least one false rejection among the true null hypotheses, regardless of which of the eight are actually true.
Test threshold calibration is idealised as continuous, allowing any real \(0\le\alpha\le1\).
Final Answer
Exact independent-test threshold \(\alpha_*=1-0.95^{1/8}\approx0.00639115\), or 0.6391% per test. Bonferroni uses \(0.05/8=0.00625\), or 0.625%. Uncorrected 5%-level tests yield familywise risk \(1-0.95^8\approx33.66\%\) when all eight nulls hold.