Measure overfitting with validation curves

Reading about overfitting is one thing; watching it appear on a chart is another. In this lab you vary one setting, the maximum depth of a decision tree, and watch training and validation scores pull apart.

The dataset is scikit-learn's built-in breast cancer dataset: 569 rows, 30 numeric features, two classes. It ships with scikit-learn, so no download is needed.

Step 1: split the data

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
 
X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=0
)
print(X_train.shape, X_val.shape)

stratify=y keeps the class balance equal in both parts. In a real project you would also hold back a test set; see train, validation and test sets.

Step 2: sweep the depth

from sklearn.tree import DecisionTreeClassifier
 
depths = range(1, 16)
train_scores, val_scores = [], []
for d in depths:
    tree = DecisionTreeClassifier(max_depth=d, random_state=0).fit(X_train, y_train)
    train_scores.append(tree.score(X_train, y_train))
    val_scores.append(tree.score(X_val, y_val))
 
for d, tr, va in zip(depths, train_scores, val_scores):
    print(f"depth={d:2d}  train={tr:.3f}  val={va:.3f}  gap={tr - va:.3f}")

Step 3: plot it

import matplotlib.pyplot as plt
 
plt.plot(list(depths), train_scores, marker="o", label="training")
plt.plot(list(depths), val_scores, marker="o", label="validation")
plt.xlabel("max_depth")
plt.ylabel("accuracy")
plt.legend()
plt.title("Decision tree: training vs validation accuracy")
plt.savefig("validation_curve.png", dpi=150, bbox_inches="tight")

Step 4: read the chart

Look for three regions:

1. Small depth: both scores are lower. The tree is too simple: underfitting.
2. Moderate depth: validation accuracy is at or near its best.
3. Large depth: training accuracy reaches 1.000, while validation stays flat or drops. The gap widens: overfitting.

Write down the depth with the best validation score and the size of the gap there.

Step 5: check how noisy one split is

Change random_state=0 in Step 1 to 1, 2 and 3 and re-run. The "best depth" may move. That is the reason for cross-validation:

from sklearn.model_selection import StratifiedKFold, validation_curve
 
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
tr, va = validation_curve(
    DecisionTreeClassifier(random_state=0), X, y,
    param_name="max_depth", param_range=list(depths), cv=cv,
)
for d, t, v in zip(depths, tr.mean(axis=1), va.mean(axis=1)):
    print(f"depth={d:2d}  train={t:.3f}  cv_val={v:.3f}")

Expected outcome

• A chart where training accuracy climbs to 1.0 and validation accuracy peaks earlier.
• A written note of the depth you would choose, and why.
• An observation of how much the best depth changes between random splits, and how cross-validation smooths that out.

Extensions

1. Replace accuracy with scoring="f1" in validation_curve and compare. See precision, recall and F1.
2. Try min_samples_leaf instead of max_depth as the complexity setting.
3. Replace the tree with RandomForestClassifier and see how averaging many trees changes the gap.