Feature importance¶
Two ways to rank features for an SGT:
Built-in impurity importances —
feature_importances_on a tree (or mean/std across a forest). Fast, and aligned with the logical features used at fit.Permutation importance — sklearn’s
permutation_importanceon held-out data. Model-agnostic; with categoricals, put one-hot encoding inside aPipelineso columns are shuffled before encoding.
This notebook covers both.
1. Built-in impurity importances¶
After fit, each tree exposes normalized importances over logical features — the same order as processed_features_. Forests expose the mean and standard deviation across base trees.
When to use this. Prefer built-in importances when the tree was grown without TAO (
tao_n_runs=0). Impurity importances are accumulated from the ShapeCART splits during growth. TAO later rewrites routing rules without recomputing those importances, so after TAO they can drift from how the refined model actually uses features. If you ran TAO (the default), prefer permutation importance (part 2) instead.
[1]:
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sgtlearn import SGTClassifier
from sgtlearn.ensemble import RandomSGForestClassifier
data = load_breast_cancer()
X, y = data.data, data.target
names = list(data.feature_names)
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.3, random_state=0, stratify=y
)
# tao_n_runs=0: importances match the grown tree (recommended for this API).
clf = SGTClassifier(max_depth=4, tao_n_runs=0, random_state=0).fit(X_tr, y_tr)
print(f"test accuracy: {clf.score(X_te, y_te):.3f}")
pf = clf.processed_features_
order = np.argsort(clf.feature_importances_)[::-1]
print(f"\n{'feature':30s} {'importance':>10s} type")
for i in order[:8]:
feat = pf.features[i]
print(f"{names[i]:30s} {clf.feature_importances_[i]:10.4f} {feat['type']}")
test accuracy: 0.895
feature importance type
mean concave points 0.4263 continuous
mean radius 0.3612 continuous
worst perimeter 0.1272 continuous
worst radius 0.0670 continuous
mean texture 0.0182 continuous
worst concave points 0.0000 continuous
worst smoothness 0.0000 continuous
worst compactness 0.0000 continuous
Index i of feature_importances_ always matches pf.features[i] / pf.logical_names[i]. With a feature_dict, those names are your logical keys (including multi-column categoricals).
[2]:
forest = RandomSGForestClassifier(
n_estimators=20, max_depth=3, tao_n_runs=0, random_state=0, n_jobs=1
).fit(X_tr, y_tr)
print(f"{'feature':30s} {'mean':>8s} {'std':>8s}")
for i in order[:5]:
print(
f"{names[i]:30s} "
f"{forest.mean_feature_importances_[i]:8.4f} "
f"{forest.std_feature_importance_[i]:8.4f}"
)
feature mean std
mean concave points 0.1524 0.2071
mean radius 0.1280 0.2314
worst perimeter 0.1177 0.2178
worst radius 0.0713 0.1351
mean texture 0.0000 0.0000
2. Permutation importance with categoricals¶
sklearn.inspection.permutation_importance shuffles one input column at a time. If you one-hot encode before fitting and then call it on the expanded matrix, each dummy is scored separately (and rows can leave the one-hot simplex).
Put OneHotEncoder inside a Pipeline instead. Sklearn permutes the original categorical column; the pipeline re-encodes on each repeat. That is the standard workaround for categorical features — and it works with SGTClassifier because the estimators are sklearn-compatible.
The Adult census example below (subsampled) keeps categoricals as strings in X.
[3]:
from sklearn.compose import ColumnTransformer
from sklearn.datasets import fetch_openml
from sklearn.inspection import permutation_importance
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
CAT_COLS = ["sex", "education", "marital-status", "occupation"]
NUM_COLS = ["age", "hours-per-week", "capital-gain", "capital-loss"]
df = fetch_openml("adult", version=2, as_frame=True, parser="auto").frame
df = df.dropna(subset=[*CAT_COLS, *NUM_COLS, "class"]).sample(n=4000, random_state=0)
X = df[NUM_COLS + CAT_COLS].copy()
for c in CAT_COLS:
X[c] = X[c].astype(str)
y = (df["class"].astype(str) == ">50K").to_numpy()
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.3, random_state=0, stratify=y
)
pipe = Pipeline(
[
(
"prep",
ColumnTransformer(
[
("num", "passthrough", NUM_COLS),
(
"cat",
OneHotEncoder(handle_unknown="ignore", sparse_output=False),
CAT_COLS,
),
]
),
),
(
"clf",
SGTClassifier(
max_depth=4,
inner_max_depth=2,
inner_max_leaf_nodes=8,
min_samples_leaf=5,
random_state=0,
tao_n_runs=0,
),
),
]
)
pipe.fit(X_tr, y_tr)
print(f"test accuracy: {pipe.score(X_te, y_te):.3f}")
result = permutation_importance(
pipe, X_te, y_te, n_repeats=10, random_state=0, scoring="accuracy"
)
order = np.argsort(result.importances_mean)[::-1]
print(f"\n{'feature':16s} {'mean':>8s} {'std':>8s}")
for i in order:
print(
f"{X_te.columns[i]:16s} "
f"{result.importances_mean[i]:8.4f} "
f"{result.importances_std[i]:8.4f}"
)
test accuracy: 0.817
feature mean std
capital-gain 0.0482 0.0053
marital-status 0.0465 0.0055
education 0.0326 0.0064
capital-loss 0.0152 0.0020
sex 0.0000 0.0000
occupation 0.0000 0.0000
hours-per-week 0.0000 0.0000
age 0.0000 0.0000
Each row is one original column of X — including categoricals like marital-status — not a one-hot dummy. The same pattern works for forests: swap in RandomSGForestClassifier as the pipeline’s final step.
A standalone script of this Adult walkthrough is in examples/permutation_importance_categorical.py.
Choosing a method¶
Situation |
Prefer |
|---|---|
Tree/forest fitted with |
Built-in |
Model refined with TAO |
Permutation importance on held-out data |
Raw categoricals in |
|
Already one-hot + |
Built-in importances over logical groups; or group-aware permutation if you stay on the expanded matrix |