Dataset splitting

Welcome to Machine Learning Fundamentals. Most of this course is about supervised learning: you have a table of examples, each with some input columns (the features) and one column you care about (the target), and you want a program that predicts the target for rows it has never seen. When the target is a number, such as a price, a delivery time or a blood test result, the task is called regression. When the target is a category, such as spam or not spam, it is called classification.

A model that scores 99% on the rows it was trained on tells you almost nothing about how it will do next month on rows it has never seen. Dataset splitting is how you get an honest estimate: you hide some data from the model, train on the rest, and measure on what was hidden. This lesson covers the train/validation/test pattern, holdout versus k-fold cross-validation, and the special splits you need when rows are grouped or ordered in time.

The intuition: practice exams and the final exam

Think of a student preparing for an exam. If the teacher hands out the exact exam questions as homework, the student can memorise the answers and score 100% without understanding anything. A fair teacher keeps three sets of questions apart:

• Homework (training set): the student studies these freely.
• Practice exams (validation set): used to decide how to study — more flashcards, fewer, a different textbook. The student sees the score, so over many attempts these questions slowly stop being "unseen".
• The final exam (test set): taken once, at the end. Its score is the one you report.

Machine learning works the same way. The model trains on the training set, you compare models and settings on the validation set, and the test set gives the one number you trust because nothing was ever tuned against it.

How it works

Train, validation and test

Each split answers a different question:

The validation set is the reason the test set stays clean. Every time you look at a score and change something because of it, information from that data leaks into your decisions. After 50 rounds of tweaking against the same validation set, its score is optimistic. The test set protects you from that, but only if you do not tweak against it too.

Holdout versus k-fold cross-validation

A holdout split cuts the data once, for example 80% train and 20% validation. It is fast and easy to reason about, but the score depends on which rows happened to land in the 20%. On a few hundred rows, that luck matters.

K-fold cross-validation removes most of the luck. It splits the data into k equal folds, trains k models, and each model is scored on the one fold it did not see:

fold 1: [VAL] [trn] [trn] [trn] [trn]
fold 2: [trn] [VAL] [trn] [trn] [trn]
fold 3: [trn] [trn] [VAL] [trn] [trn]
fold 4: [trn] [trn] [trn] [VAL] [trn]
fold 5: [trn] [trn] [trn] [trn] [VAL]

CV score = (score_1 + score_2 + ... + score_k) / k

Every row is used for validation exactly once and for training k − 1 times. You report the mean and the standard deviation across folds; the standard deviation tells you how much the score moves with the data. The cost is k fits instead of one. k = 5 or k = 10 are the usual choices.

The typical workflow combines both: hold out a test set once, then run k-fold cross-validation on the remaining data for all model selection.

Stratification

In classification, a random split can give folds with different class ratios. If 10% of your rows are fraud and one fold gets 4% while another gets 16%, the fold scores are not comparable. Stratified splitting keeps the class proportions the same in every fold. Pass stratify=y to train_test_split, and use StratifiedKFold for cross-validation. Scikit-learn already uses StratifiedKFold when you pass cv=5 with a classifier, but it does not shuffle by default, so build the splitter explicitly when your rows are sorted.

Group splits

Sometimes several rows belong to one entity: five scans from the same patient, 200 transactions from the same customer, ten photos of the same product. If one patient's scans land in both training and validation, the model can recognise the patient rather than the disease, and the score is inflated. GroupKFold takes a groups array and guarantees that every group sits entirely in one fold. StratifiedGroupKFold does the same while trying to keep class ratios balanced, and GroupShuffleSplit is the holdout version.

Time-series splits

When rows are ordered in time and the model will predict the future, a shuffled split lets the model train on Friday and be tested on Wednesday. That is not how it will run in production. TimeSeriesSplit always puts the validation fold after the training rows, with an expanding training window. Its gap parameter leaves a buffer of rows between train and validation, which matters when features use lagged values or labels arrive late.

How big each split should be

There is no single correct ratio. What matters is the absolute number of rows in each evaluation set, because the uncertainty of an accuracy estimate shrinks with the square root of the count:

standard error of accuracy ≈ sqrt(p · (1 − p) / n)

p = the accuracy you measure
n = number of rows in the evaluation set

At p = 0.95 and n = 100, the standard error is about 0.022, so a 2-point difference between two models is noise. At n = 10,000 it is about 0.002. Rough guidance:

For rare classes, count the minority rows: a test set with 12 positive examples cannot measure recall with any precision, whatever percentage it is.

Worked example

The script below uses the breast cancer dataset (569 rows, 62.7% benign). It builds a three-way split, measures how much a single holdout score moves with the random seed, compares plain and stratified k-fold, checks that GroupKFold keeps groups apart, prints the windows TimeSeriesSplit produces, and finally scores the test set once. The model is a scaled logistic regression (lesson 11 covers it in depth).

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (
    train_test_split, KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit, cross_val_score,
)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)
print(f"Rows: {len(y)}, positive rate: {y.mean():.3f}")

# 1. Three-way holdout: 60 / 20 / 20, stratified on the label
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval, test_size=0.25, stratify=y_trainval, random_state=42)
print(f"Train {len(y_train)}, val {len(y_val)}, test {len(y_test)}")

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))

# 2. How much does a single holdout score move with the seed?
scores = []
for seed in range(20):
    Xa, Xb, ya, yb = train_test_split(X_trainval, y_trainval, test_size=0.25, random_state=seed)
    scores.append(model.fit(Xa, ya).score(Xb, yb))
print(f"Holdout accuracy over 20 seeds: min {min(scores):.3f}, max {max(scores):.3f}, std {np.std(scores):.3f}")

# 3. 5-fold CV, plain vs stratified
for name, cv in [("KFold", KFold(5, shuffle=True, random_state=42)),
                 ("StratifiedKFold", StratifiedKFold(5, shuffle=True, random_state=42))]:
    s = cross_val_score(model, X_trainval, y_trainval, cv=cv)
    rates = [y_trainval[val].mean() for _, val in cv.split(X_trainval, y_trainval)]
    print(f"{name:16s} mean {s.mean():.3f} +/- {s.std():.3f}  fold positive rates {np.round(rates, 3)}")

# 4. Group split: pretend every 5 consecutive rows come from one patient
groups = np.arange(len(y_trainval)) // 5
splits = {
    "KFold": KFold(5, shuffle=True, random_state=42).split(X_trainval),
    "GroupKFold": GroupKFold(n_splits=5).split(X_trainval, y_trainval, groups),
}
for name, folds in splits.items():
    overlap = [len(set(groups[tr]) & set(groups[va])) for tr, va in folds]
    print(f"{name:10s} patients in both train and validation, per fold: {overlap}")

# 5. Time-series split: training window grows, test always comes later
tss = TimeSeriesSplit(n_splits=4, test_size=50)
for i, (tr, va) in enumerate(tss.split(X_trainval)):
    print(f"TimeSeriesSplit fold {i}: train rows {tr[0]}-{tr[-1]}, test rows {va[0]}-{va[-1]}")

# 6. Final, one-time check on the untouched test set
model.fit(X_trainval, y_trainval)
print(f"Test accuracy (touched once): {model.score(X_test, y_test):.3f}")

Output:

Rows: 569, positive rate: 0.627
Train 341, val 114, test 114
Holdout accuracy over 20 seeds: min 0.947, max 0.991, std 0.013
KFold            mean 0.971 +/- 0.024  fold positive rates [0.604 0.692 0.527 0.703 0.604]
StratifiedKFold  mean 0.978 +/- 0.010  fold positive rates [0.626 0.626 0.626 0.626 0.626]
KFold      patients in both train and validation, per fold: [61, 63, 57, 61, 60]
GroupKFold patients in both train and validation, per fold: [0, 0, 0, 0, 0]
TimeSeriesSplit fold 0: train rows 0-254, test rows 255-304
TimeSeriesSplit fold 1: train rows 0-304, test rows 305-354
TimeSeriesSplit fold 2: train rows 0-354, test rows 355-404
TimeSeriesSplit fold 3: train rows 0-404, test rows 405-454
Test accuracy (touched once): 0.982

Reading the output

• The holdout score is a lottery ticket. The same model on the same data scores anywhere from 94.7% to 99.1% depending only on the seed. If you compared two models using one holdout split each, a 4-point difference could be pure luck.
• Stratification steadies the folds. Plain KFold produced folds with 52.7% to 70.3% benign cases; StratifiedKFold kept every fold at 62.6%. The fold-to-fold standard deviation dropped from 0.024 to 0.010.
• Groups leak without GroupKFold. With plain KFold, about 60 of the simulated patients had rows on both sides of the split in every fold. GroupKFold brought that to zero.
• TimeSeriesSplit never looks backwards. Each validation window starts after the last training row, and the training window grows by 50 rows per fold.
• The test score is reported once. 98.2% is within the range cross-validation predicted. Had it been far below, that would be a sign that something in the model selection overfitted the validation data.

Reference: scikit-learn splitters

Common pitfalls

• Splitting after preprocessing. Fitting a scaler or imputer on all rows and then splitting lets test statistics leak into training. Split first, and put preprocessing inside a Pipeline. Lesson 2 covers this in detail.
• Reusing the test set for decisions. If you looked at the test score, changed a feature, and looked again, it is now a validation set. Spot it by asking whether any choice in your code depended on the test score.
• Shuffling time-ordered data. A model that scores far better under KFold than under TimeSeriesSplit is probably using future information.
• Ignoring groups. If your dataset has an ID column that repeats (user_id, patient_id, session_id), check whether a random split puts the same ID on both sides. A score that drops sharply under GroupKFold is the warning sign.
• Sorted data with shuffle=False. Many datasets come sorted by class or date. KFold without shuffling then produces folds with no examples of one class.
• Reporting the mean without the spread. "0.97" and "0.97 ± 0.05" describe very different levels of confidence.

Checklist for your project

1. Decide what "new data" means in production: new rows, new users, or future dates. Choose the splitter that mimics that.
2. Carve off the test set first, with the same splitter logic, and do not look at it again until the end.
3. Use stratified k-fold (5 or 10 folds) on the remaining data for every comparison.
4. Put all preprocessing inside a Pipeline so it is refit inside each fold.
5. Count the rows, and the minority-class rows, in each evaluation set and estimate the standard error.
6. Fix random_state on every splitter so results are reproducible.
7. Report mean and standard deviation across folds, then the single test score.

Key takeaways

• The training score measures memory; only data the model has not seen measures generalisation.
• Use the validation set (or cross-validation) for decisions and the test set once, for the final number.
• K-fold cross-validation averages out the luck of a single holdout split and gives a spread you can report.
• Stratify classification splits; use GroupKFold when rows share an entity and TimeSeriesSplit when you predict the future.
• Size evaluation sets by absolute row count, not percentage: the standard error shrinks with sqrt(n).

Try it yourself

1. Change the holdout loop to use test_size=0.10 and then 0.40. How do the min, max and standard deviation of the scores change?
2. Replace KFold with RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42) in step 3 and compare the reported spread.
3. Sort X_trainval and y_trainval by label, then run KFold(5) without shuffling. Print the fold positive rates and explain what you see.