Ensembling SGTs — Random SG Forests

RandomSGForestClassifier / RandomSGForestRegressor bag many SGTs, each trained on a bootstrap resample with a random feature subset per split, and average their predictions. This usually beats a single tree at the cost of interpretability. This notebook shows the accuracy gain and the knobs that control it.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sgtlearn import SGTClassifier, RandomSGForestClassifier

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)

tree = SGTClassifier(max_depth=3, random_state=0).fit(X_tr, y_tr)
forest = RandomSGForestClassifier(
    n_estimators=50, max_depth=3, max_features="sqrt", random_state=0, n_jobs=-1
).fit(X_tr, y_tr)

print(f"single tree  test={tree.score(X_te, y_te):.3f}")
print(f"forest (50)  test={forest.score(X_te, y_te):.3f}")

single tree  test=0.877
forest (50)  test=0.959

What to notice

The forest averages 50 decorrelated trees, so its test accuracy is typically higher and more stable than a single tree of the same depth. Compare the two printed numbers.

The knobs

• n_estimators — number of trees. More = better, with diminishing returns.

• bootstrap — resample rows per tree (True) for diversity; max_samples sets the resample size.

• max_features — features considered per split ("sqrt", "log2", int, float). Lower = more diverse trees.

• n_jobs — parallel tree fitting (-1 = all cores).

# More trees, then a smaller per-split feature set for extra diversity.
for n in (5, 25, 100):
    f = RandomSGForestClassifier(
        n_estimators=n, max_depth=3, max_features="sqrt", random_state=0, n_jobs=-1
    ).fit(X_tr, y_tr)
    print(f"n_estimators={n:3d}: test={f.score(X_te, y_te):.3f}")

# bootstrap + max_samples control resampling.
f = RandomSGForestClassifier(
    n_estimators=50, max_depth=3, bootstrap=True, max_samples=0.6,
    max_features="log2", random_state=0, n_jobs=-1
).fit(X_tr, y_tr)
print(f"bootstrap 60% rows, log2 feats: test={f.score(X_te, y_te):.3f}")

n_estimators=  5: test=0.912
n_estimators= 25: test=0.953
n_estimators=100: test=0.965
bootstrap 60% rows, log2 feats: test=0.953

How does it compare to a classic random forest?

scikit-learn’s RandomForestClassifier bags axis-aligned CART trees (each node is a single x < t threshold); RandomSGForestClassifier bags SGTs whose nodes apply a shape function to a feature. Two things set an SGT forest apart — the shape-function splits and optional TAO refinement — so we separate them, using phoneme (nasal-vs-oral vowel classification from five continuous acoustic features that relate non-monotonically to the class).

from sklearn.datasets import fetch_openml
from sklearn.ensemble import RandomForestClassifier

# Phoneme: 5404 samples, 5 continuous acoustic features, binary target.
Xp, yp = fetch_openml("phoneme", version=1, return_X_y=True, as_frame=False, parser="liac-arff")
Xp_tr, Xp_te, yp_tr, yp_te = train_test_split(Xp, yp, test_size=0.3, random_state=0, stratify=yp)

# Same settings for all three. tao_n_runs=0 isolates the SGT architecture;
# the default (tao_n_runs=10) adds TAO refinement on top.
common = dict(n_estimators=50, max_depth=3, max_features=None, random_state=0, n_jobs=-1)
sg_arch = RandomSGForestClassifier(tao_n_runs=0, **common).fit(Xp_tr, yp_tr)   # shape functions only
sg_tao  = RandomSGForestClassifier(**common).fit(Xp_tr, yp_tr)                 # + TAO (the default)
rf      = RandomForestClassifier(**common).fit(Xp_tr, yp_tr)                   # classic CART forest

print(f"SGForest, no TAO     test={sg_arch.score(Xp_te, yp_te):.3f}")
print(f"SGForest, with TAO   test={sg_tao.score(Xp_te, yp_te):.3f}")
print(f"sklearn RandomForest test={rf.score(Xp_te, yp_te):.3f}")

SGForest, no TAO     test=0.811
SGForest, with TAO   test=0.818
sklearn RandomForest test=0.790

With TAO off, the SGT forest already beats the classic forest — each node’s shape function bins one feature into several regions, capturing non-monotone structure that a single axis-aligned threshold misses. Turning TAO on (the default) refines the routing rules for a further bump.

For a single interpretable model instead, see the quickstart and structure vs. accuracy. TAO also refines every tree in a forest — see TAO › Forests.