Question
A modelling team predicts a customer's next-week spending minus the overall expected spending, in dollars. Call this centred outcome \(Y\). In truth, customers' outcomes are independent, with mean \(0\) and variance 25 \(\text{dollars}^{2}\), and marketing category provides no predictive information.
Each category in the team's evaluation dataset contains exactly five customers. For a customer \(i\), the team constructs a feature \(Z_i\) equal to the average of all five observed outcomes in that category, including \(Y_i\) itself, before dividing the dataset into training and evaluation sets. It then naively predicts \(Y_i\) using \(Z_i\).
Derive \(\operatorname{Cov}(Z_i,Y_i)\), the mean squared prediction error \(\mathbb E[(Y_i-Z_i)^2]\), and the apparent improvement over always predicting zero. Compare these with a valid deployment-time feature \(Z_i^*\) formed by averaging the outcomes of five different, independently observed training customers from the same category. What is the covariance and mean squared error if \(Z_i^*\) is used unchanged, and what is the best linear predictive weight on \(Z_i^*\) under this no-signal model?
Get hint
Hint 1
Expand the category mean as \(Z_i=(Y_i+\sum_{j\ne i}Y_j)/5\). Only one term in that sum is correlated with \(Y_i\).
Check your answer
Solution
Fix one customer \(i\) in a five-customer category. With all five outcomes mutually independent and centred, write \(Z_i=\tfrac15Y_i+\tfrac15\sum_{j\ne i}Y_j\). The other four outcomes are independent of \(Y_i\), so
\(\operatorname{Cov}(Z_i,Y_i)=\tfrac15\operatorname{Var}(Y_i)=25/5=\boxed{5\text{ dollars}^2}\).
The average's variance is \(\operatorname{Var}(Z_i)=5(25)/25=5\) \(\text{dollars}^{2}\). Its prediction error has mean zero and variance
\(\mathbb E[(Y_i-Z_i)^2]=25+5-2(5)=\boxed{20\text{ dollars}^2}\).
The baseline prediction zero has MSE \(\mathbb E[Y_i^2]=25\). The apparently reduced error is 5 \(\text{dollars}^{2}\), or 20% of baseline risk. This is not learned information about future customers: the evaluated customer's answer has been put into their own feature.
Now build \(Z_i^*\) from five independent other labelled customers in the same category. The no-signal hypothesis makes \(Z_i^*\) independent of \(Y_i\). It has mean zero and variance 5, so \(\operatorname{Cov}(Z_i^*,Y_i)=\boxed{0}\) and using it directly produces \(\mathbb E[(Y_i-Z_i^*)^2]=25+5=\boxed{30}\). More generally, using \(wZ_i^*\) gives MSE \(25+5w^2\), minimised at \(w=\boxed{0}\) with risk 25. A properly constructed evaluation/deployment feature therefore has no predictive value here. Valid alternatives include computing category means on disjoint training rows, cross-fitting, or leave-one-out encoding with the evaluated outcome excluded; all must prevent label reuse from the scored row.
Assumptions
All customer outcomes have the same mean zero and variance 25 \(\text{dollars}^{2}\) and are mutually independent, including within marketing category.
Category membership is fixed, and categories carry no genuine signal.
Leaky feature includes the row's own labelled outcome; clean feature uses five independent other labelled customers. Zero/linear predictors have no fitted intercept because outcomes are centred.
Final Answer
Leaky self-including encoding: Cov\((Z_i,Y_i)=5\) \(\text{dollars}^{2}\), MSE \(20\) \(\text{dollars}^{2}\) versus zero-prediction baseline 25 (illusory 20% improvement). Independent encoding: Cov\((Z_i^*,Y_i)=0\), unchanged-mean MSE 30, and optimal linear weight 0, giving MSE 25.