Question
Eighteen customers try two handle designs and give each a comfort score on the same scale. Each customer's result is paired: subtract the old-handle score from the new-handle score. Ten customers give a positive difference, three give a negative difference and five give exactly zero.
Before collecting the scores, the workshop decided to use a one-sided sign test for whether the new handle is preferred. Under its null hypothesis, conditional on a nonzero difference, each customer's sign is an independent fair coin; the magnitude of a score difference is not modeled. Zero differences are treated as ties and excluded, not counted as half-wins.
What is the exact p-value for more positive than negative differences, and would the workshop reject at a 5% significance level? What do the five ties do to the test?
Get hint
Hint 1
Condition on the 13 customers who reported nonzero differences. Under the null, the number of positive signs is \(\operatorname{Binomial}(13,\frac12)\).
Check your answer
Solution
Of the 18 pairs, five have zero differences, leaving \(m=10+3=13\) informative signs. Because the null assigns independent probability 1/2 to either sign conditional on nonzero differences, \(K\), the number of positive signs among those 13, is \(\operatorname{Binomial}(13,\frac12)\). The prespecified directional alternative asks whether \(K\) is unusually large.
The exact one-sided p-value is
\[\begin{aligned}\Pr_0(K\ge10)&=\sum_{k=10}^{13}\frac{\binom{13}{k}}{2^{13}}\\
&=\frac{\binom{13}{10}+\binom{13}{11}+\binom{13}{12}+\binom{13}{13}}{8192}\\
&=\frac{286+78+13+1}{8192}=\frac{378}{8192}=\frac{189}{4096}\approx0.04614.\end{aligned}\]
This is below 0.05, so the test rejects the null of symmetric nonzero signs in favour of a tendency toward positive differences, under the stated sampling assumptions.
A zero difference cannot be called a positive or negative sign. Conditioning on \(m=13\) removes the ties without inventing a direction for them. It also means only 13 rather than 18 observations enter the binomial test, generally reducing its power; the test makes no claim about the magnitude of any comfort improvement. Direct enumeration of the \(2^{13}\) equally likely sign sequences finds 378 with at least 10 positive signs, verifying the calculation.
Assumptions
Customer pairs are independent; conditional on being nonzero, their signs are independent and equiprobable under the null.
The one-sided direction and exclusion of exact zeros were set in advance.
The 18 customers are a fixed-size sample, not a sample stopped when a significance threshold was reached.
Final Answer
Effective sample size is 13; one-sided exact \(p\)-value \(=\frac{189}{4096}\approx0.04614\). Reject the sign-symmetry null at 5%. The five ties contribute no signs and reduce the information available.