Question
A clinic is trialling a new appointment system. Before collecting data, its protocol names shorter waiting times as the primary benefit and fewer missed appointments as a secondary benefit worth claiming only if the primary one is supported. For each outcome, the clinic uses a valid p-value for a null hypothesis of no improvement. It wants familywise probability of making any false benefit claim to be at most 5%.
The prespecified rule is: test the waiting-time null at level 0.05; only if it is rejected, test the missed-appointment null at the same level 0.05. The eventual p-values are 0.03 and 0.04, respectively.
Which claims may the clinic make under its rule? Prove that the prespecified two-test sequence controls familywise error at 5% even if the p-values are dependent. Finally, explain what can go wrong if the clinic examines the p-values first and chooses the smaller one to test first.
Get hint
Hint 1
The second test occurs only if the first test passes, so any false rejection must include rejection of the first true null in the prespecified order.
Check your answer
Solution
The protocol places the waiting-time null \(H_1\) before the missed-appointment null \(H_2\). The clinic first sees \(p_1=0.03\le0.05\), so it rejects \(H_1\) and proceeds. It then sees \(p_2=0.04\le0.05\), so it also rejects \(H_2\). Both benefit claims are permitted by the prespecified hierarchy; this is not a claim that the probability they are both true is 95%.
To prove strong familywise error control, consider the unknown set of true nulls. If none are true, no false rejection is possible. Otherwise let \(H_j\) be the first true null in the fixed order. To reject any true null, the procedure must first reject \(H_j\): if it stops before that point, it never reaches a true null, and if it reaches later true nulls, it has already rejected \(H_j\). Hence
\[
P(\text{at least one false rejection})
\le P(p_j\le0.05)\le0.05,
\]
where the last inequality is the validity of the p-value under its true null. This argument does not assume independence between tests. Each hypothesis is tested at 5%, but the fixed gate prevents unrestricted multiple chances to make a false claim.
The fixed order is essential. If both nulls are true and their p-values are independent Uniform(0,1), selecting whichever p-value is smaller to test first and then using the 0.05 threshold makes some rejection whenever their minimum is at most 0.05. Its probability is
\[
P(\min(p_1,p_2)\le0.05)=1-(0.95)^2=\boxed{0.0975},
\]
which exceeds 5%. The clinic therefore cannot choose its hypothesis priority based on the observed results and retain the prespecified fixed-sequence guarantee.
Assumptions
The two hypotheses and their test order are fixed before observing any outcome data.
Each supplied p-value is valid (super-uniform) whenever its corresponding null is true.
The illustrative 9.75% counterexample assumes both nulls are true and their p-values independent Uniform(0,1).
Final Answer
Reject both nulls under the fixed 0.05 hierarchy. Familywise error is \(\le5\%\) for any true/false configuration; selecting the order after seeing p-values can inflate it (to 9.75% for independent uniform null p-values).