Robustness API

uplift_bench.robustness.permutation

Permutation feature importance for uplift.

Standard permutation importance (Breiman 2001) shuffles a feature column and measures how much the loss degrades. For uplift the obvious knob to turn is the Qini coefficient: a feature that genuinely drives heterogeneous treatment effect should hurt Qini when shuffled. A feature that only drives the outcome (not uplift) shouldn't.

That distinction is why we don't reuse sklearn's permutation_importance: sklearn shuffles relative to a model's score on Y, which conflates the two.

permutation_uplift_importance(model, X, treatment, outcome, *, n_repeats=5, seed=0)

Per-feature permutation importance computed against Qini.

Parameters:

Name	Type	Description	Default
model	UpliftModel	A fitted UpliftModel.	required
X	DataFrame	Held-out evaluation set (don't reuse the training fold).	required
treatment	DataFrame	Held-out evaluation set (don't reuse the training fold).	required
outcome	DataFrame	Held-out evaluation set (don't reuse the training fold).	required
n_repeats	int	How many shuffles per feature. 5 is enough for ranking; bump to 30+ for tight error bars.	5
seed	int	RNG seed. Each (feature, repeat) pair derives a child seed via SeedSequence so the shuffles are independent but reproducible.	0

Returns:

Type	Description
DataFrame	Columns: feature, baseline_qini, mean_qini_drop, std_qini_drop, n_repeats. Sorted descending by mean_qini_drop. Positive drop = important feature.

uplift_bench.robustness.feature_drop

Drop-feature stability of Qini.

For each feature (or group of features), refit the model with that column removed and measure the change in Qini on a held-out set.

Different from permutation importance: * Permutation: how much does the fitted model rely on this feature? * Drop-feature: how much does removing the feature change the model that gets fit? Captures interaction effects permutation misses.

Slow — costs N_features model fits — so we expose n_jobs for joblib parallelism and recommend running it only on the final candidate models, not during sweep.

feature_drop_stability(model_builder, X_train, t_train, y_train, X_eval, t_eval, y_eval, *, groups=None, n_jobs=1)

Refit the model dropping each feature (or group) and report Qini delta.

Parameters:

Name	Type	Description	Default
model_builder	ModelBuilder	Zero-arg factory returning a fresh, unfit UpliftModel. We need a factory rather than a model instance because we have to re-fit from scratch for each drop.	required
X_train	DataFrame	Training and held-out folds.	required
t_train	DataFrame	Training and held-out folds.	required
y_train	DataFrame	Training and held-out folds.	required
X_eval	DataFrame	Training and held-out folds.	required
t_eval	DataFrame	Training and held-out folds.	required
y_eval	DataFrame	Training and held-out folds.	required
groups	dict[str, list[str]] | None	Optional mapping of group_name → list of column names to drop together. If None, drops one feature at a time.	None
n_jobs	int	joblib parallelism. Each fit is independent.	1

Returns:

Type	Description
DataFrame	Sorted descending by qini_delta (most-important groups first).

uplift_bench.robustness.learning_curve

Learning curve: Qini vs training-set size.

Refit the same model on increasing fractions of the training set and report Qini on a fixed eval set. Useful for two questions:

1. "Will more data help?" — flat curve at the high end means we've saturated; rising curve means we're data-limited.
2. "Is the model overfitting at small N?" — early Qini > late Qini is the pathological signature of meta-learners that haven't been regularised for sample size (often X-learner with deep CatBoost).

We always sort the train set by a fixed permutation (seeded) and take prefixes — so smaller training sets are subsets of bigger ones, which makes the curve interpretable.

learning_curve(model_builder, X_train, t_train, y_train, X_eval, t_eval, y_eval, *, fractions=(0.1, 0.25, 0.5, 0.75, 1.0), seed=0)

Compute Qini at each training-set fraction.

Returns a DataFrame with columns: fraction, n_train, qini.

uplift_bench.robustness.overlap

Propensity overlap diagnostics.

Two checks:

1. Propensity histogram: estimate e(X) with a calibrated classifier and look at the distributions for treated vs control. Heavy tails near 0 or 1 mean we have observations with no comparable counterpart in the other arm — IPW-flavoured estimators (X / R / DR) will explode there.

2. Effective sample size: ESS = (sum w_i)^2 / sum w_i^2 with w_i = 1 / e(x_i) for treated, 1 / (1 - e(x_i)) for control. ESS / n is a single-number summary of the same problem.

We compute these with a fresh, simple classifier (sklearn HistGradientBoosting by default) so the diagnostic doesn't depend on which meta-learner the user picked.

overlap_diagnostics(X, treatment, *, seed=0, n_splits=5, clip=(0.05, 0.95))

Estimate propensity OOF and report overlap diagnostics.