Question
A recycling plant is choosing between two binary classifiers, A and B, that predict whether an item is contaminated (1) or clean (0). Its evaluation labels \(L\) come from rushed human inspections, not from the true status \(Y\). An independent audit finds \(P(L=1\mid Y=0)=0.10\) and \(P(L=0\mid Y=1)=0.40\). Assume that, conditional on the true status, inspection errors are independent of either model's prediction.
The following entries are proportions of all evaluated items, not conditional percentages. Each model was evaluated on the same population:
| Model A: \(L=0\) | Model A: \(L=1\) | Model B: \(L=0\) | Model B: \(L=1\) | |
|---|---|---|---|---|
| Predict 0 | 0.61 | 0.29 | 0.40 | 0.10 |
| Predict 1 | 0.04 | 0.06 | 0.25 | 0.25 |
Which model has higher true accuracy, after correcting for label noise? Derive the correction rather than estimating it from raw agreement. When does this correction become mathematically impossible, and why does the conditional-independence assumption matter?
Get hint
Hint 1
For a fixed model prediction \(m\), split the observed event \(\{M=m,L=1\}\) according to the unobserved true class \(Y\). The independence assumption makes the audit's error rates usable in both pieces.
Check your answer
Solution
Let \(M\in\{0,1\}\) be either classifier's prediction, \(Y\) truth, and \(L\) the noisy reference. Write \(\alpha=0.10\), \(\beta=0.40\), and \(d=1-\alpha-\beta=0.50\). Conditional independence of \(L\) and \(M\) given \(Y\) implies, for each prediction value \(m\),
\[ q_m:=P(M=m,L=1)=\alpha P(M=m,Y=0)+(1-\beta)P(M=m,Y=1). \]
Since \(p_m=P(M=m,Y=0)+P(M=m,Y=1)\), inversion gives
\[ P(M=m,Y=1)=\frac{q_m-\alpha p_m}{d},\qquad P(M=m,Y=0)=\frac{(1-\beta)p_m-q_m}{d}. \]
For A, \((p_0,q_0)=(0.90,0.29)\) yields \(P(M=0,Y=1)=0.40\) and \(P(M=0,Y=0)=0.50\); \((p_1,q_1)=(0.10,0.06)\) yields \(P(M=1,Y=1)=0.10\), \(P(M=1,Y=0)=0\). Therefore A's true accuracy is \(0.50+0.10=0.60\).
For B, \((p_0,q_0)=(0.50,0.10)\) yields \(P(M=0,Y=1)=0.10\), \(P(M=0,Y=0)=0.40\). For \((p_1,q_1)=(0.50,0.25)\), the corresponding entries are \(P(M=1,Y=1)=0.40\), \(P(M=1,Y=0)=0.10\). Hence B's true accuracy is \(0.40+0.40=0.80\). Both inversions imply the same true prevalence \(P(Y=1)=0.50\), a useful consistency check. In contrast, raw agreement with the noisy reference is \(0.61+0.06=0.67\) for A and \(0.40+0.25=0.65\) for B, which ranks them incorrectly.
The label channel maps true class masses to noisy class masses through the matrix \(\left(\begin{smallmatrix}1-\alpha&\beta\\\alpha&1-\beta\end{smallmatrix}\right)\), with determinant \(1-\alpha-\beta\). If \(\alpha+\beta=1\), the two columns coincide: the reference supplies no class information and the inversion is not identifiable.
Why conditional independence is essential. Knowing only \(P(L\mid Y)\) does not determine \(P(L\mid Y,M)\). To see this even for an invertible channel, suppose \(P(Y=1)=1/2\) and keep the same audit rates. One possible predictor sets \(M=Y\) exactly. Another, for every observed \(L\), draws \(M\) independently of \(Y\) from the conditional distribution \(P(Y\mid L)\). Both predictors have the same joint distribution of \((M,L)\), since both satisfy \(P(M\mid L)=P(Y\mid L)\); both share the same \((Y,L)\) law and audit rates. Yet the first is 100% truly accurate, while the second is only \(58/91\approx63.74\%\) accurate. The second construction violates the required conditional independence, showing that the observed label table and marginal noise rates alone cannot identify true risk.
Assumptions
The two confusion tables are joint shares of the evaluation population, not conditional on model prediction.
Under the stipulated assumption, label noise depends on truth but not on a model prediction once truth is held fixed.
The class-noise rates are population probabilities; any finite-sample uncertainty in the audit is outside the question.
The true class 1 means contaminated, and model accuracy is \(P(\text{prediction equals truth})\).
Final Answer
Observed-label accuracies are \(67\%\) (A) and \(65\%\) (B), but corrected true accuracies are \(60\%\) (A) and \(80\%\) (B); choose B. The channel is noninvertible when \(\alpha+\beta=1\), where \(\alpha=P(L=1\mid Y=0)\) and \(\beta=P(L=0\mid Y=1)\). Without conditional independence, these marginal label-error rates do not identify true model risk.