Question
A hospital is testing a model that predicts the number of visits a patient will make next month. Its dataset contains three records per patient, and it randomly sends each row to training with probability 1/2 or validation otherwise. The model is allowed to recognise a patient's identifier. A row under evaluation is known to be in validation, but the patient's other two rows are split independently.
For patient i, the recorded target in row j is \(Y_{ij}=\mu+U_i+\varepsilon_{ij}\). Here \(\mu\) is known, the patient-specific effect \(U_i\) has mean 0 and variance 9, and the independent row noises \(\varepsilon_{ij}\) have mean 0 and variance 4. Patient effects and noises are mutually independent. When the training set contains \(k\ge1\) rows for the same patient, the model predicts a held-out row using the average training target for that patient; if \(k=0\), it predicts \(\mu\). Evaluate errors against the unobserved true held-out Y, not against its noise-free expectation.
Find the expected squared prediction error of a random validation row. Compare it with the squared-error risk when predicting a completely new patient, who has no training rows. Explain how the validation result would change if there were many records per patient, still using a 50/50 row split.
Get hint
Hint 1
Condition on K, the number of the other two records from the same patient that entered training. Given the evaluated row is held out, \(K\sim\operatorname{Binomial}(2,1/2)\).
Check your answer
Solution
Conditional on one target row being in validation, its two companion rows independently enter training with probability 1/2. Thus \(K\sim\operatorname{Binomial}(2,1/2)\), with \(P(K=0,1,2)=(\frac14,\frac12,\frac14)\).
For \(K=0\) the predictor is \(\mu\), so the held-out error is \(U_i+\varepsilon_{ij}\). Independence and zero means give \(\mathrm{MSE}=\operatorname{Var}(U)+\operatorname{Var}(\varepsilon)=9+4\)=\(13\).
For \(K=k\ge1\), the training mean for that patient equals \(\mu+U_i+\frac1k\sum_{\mathrm{train}}\varepsilon_{i\ell}\). The held-out target minus this mean is \(\varepsilon_{ij}-\frac1k\sum_{\mathrm{train}}\varepsilon_{i\ell}\). The patient effect cancels, while training and test noises are independent. Therefore
\[\mathrm{MSE}(K=k)=4+\frac4k\].
In particular, \(\mathrm{MSE}(K=1)=8\) and \(\mathrm{MSE}(K=2)=6\). Averaging over K gives
\[\mathrm{MSE}_{\mathrm{row}}=\frac14\cdot13+\frac12\cdot8+\frac14\cdot6\]
=\(\frac{13}{4}+4+\frac32\)=\(35/4=8.75\).
For a new patient, no identifier-specific training average exists, so the rule predicts \(\mu\) and its risk remains 13. The row-split score understates this target risk by \(13-8.75=4.25\) squared-visit units.
For \(m\) records per patient and training probability \(t=1/2\), the companion-row count is \(K\sim\operatorname{Binomial}(m-1,t)\). The familiar-patient fraction among validation rows is
\[P(K\ge1)=1-(1-t)^{m-1}=1-2^{-(m-1)}\].
The full row risk is
\[(1-t)^{m-1}(9+4)+\sum_{k=1}^{m-1}C(m-1,k)t^k(1-t)^{m-1-k}\cdot4(1+1/k).\]
As \(m\) grows, \(P(K=0)\to0\) and \(K\to\infty\) in probability. Hence the row-split risk approaches 4, the irreducible held-out noise variance, while the new-patient risk remains 13. Splitting by patient ID rather than by row would estimate the intended new-patient performance more directly.
Assumptions
Each patient has exactly three rows initially; companion rows are assigned to train independently at probability 1/2.
The model uses only the specified per-ID sample-mean rule and a known population baseline \(\mu\); no hyperparameter estimation is included.
Patient effects have variance 9, row noises variance 4, and all stated independent quantities have mean zero.
New-patient deployment contains no training row with that patient identifier.
Final Answer
Row-split validation \(\mathrm{MSE}=\frac{35}{4}=8.75\); new-patient \(\mathrm{MSE}=13\). With \(m\) records per patient, \(\Pr(\text{a held-out patient has been seen in training})=1-2^{-(m-1)}\), and the row-split MSE tends to 4 as \(m\to\infty\) while new-patient MSE remains 13.