Thirty-six models, and the simplest fix won.
Disability is a minority outcome in national survey data, so a model tuned for raw accuracy learns to predict "no disability" every time. It looks 90% accurate and is completely useless. The real question is not which classifier — it is which classifier paired with which imbalance fix.
On 9,241 households described by 145 features from the 2023 American Housing Survey, disability is the minority class. Optimise for accuracy and the model discovers the shortcut immediately: always answer no. High score, zero value.
The usual response is to reach for SMOTE and move on. But the choice of classifier and the choice of balancing strategy interact — a balancer that rescues a decision tree can wreck a linear model. Testing them separately tells you very little. So I tested every combination.
- Prepared the survey data. 9,241 households × 145 features, cleaned and encoded, with a stratified train/test split held constant for everything that followed.
- Defined the grid. Nine classifiers crossed with four balancing strategies — none, random oversampling, SMOTE, and random undersampling — giving 36 complete pipelines.
- Trained every combination on the identical split. This is the part that makes the comparison mean anything. One split, one preprocessing path, 36 outcomes that can actually be ranked against each other.
- Ranked on minority-aware metrics. F1 and recall on the disability class, never on overall accuracy — the metric that made the problem invisible in the first place.
| Dimension | Value |
|---|---|
| Data | 9,241 × 145 |
| Grid | 9 classifiers × 4 balancers |
| Pipelines trained | 36 |
| Ranking metric | Minority-class F1 / recall |
| Best balancing strategy | Random undersampling |
Random undersampling — throwing majority rows away, the crudest option on the list — beat SMOTE and every synthetic alternative. The expensive technique is not automatically the right one.
That result is less surprising than it first sounds. SMOTE interpolates between minority neighbours in feature space, and with 145 mostly categorical survey features, the points it invents can land somewhere no real household lives. Undersampling never invents anything. It pays for that with variance, which is the trade this dataset happened to favour.
This write-up is shorter than the other two on purpose. The full comparison heatmap across all 36 pipelines, the winning model's confusion matrix, and its exact final F1 are not published here yet — and I would rather leave a gap than fill it with a number I haven't rechecked against the held-out set.
Happy to walk through the notebook directly in the meantime.