Metrics API

uplift_bench.metrics.qini

Qini coefficient and Qini curve.

Definition (Radcliffe 2007, "Using control groups to target on predicted lift", Direct Marketing Analytics Journal). Sort observations by predicted uplift descending. Walking down that ordered list, plot:

x-axis : cumulative population share k / N
y-axis : (n_treated_responders[:k] - n_control_responders[:k]
         * n_treated[:k] / n_control[:k])  / N

The raw Qini area is the integral between this curve and the random-targeting diagonal. The normalised Qini coefficient divides that by the area achievable by a perfect-ordering model, so it sits in roughly [-1, 1] and is comparable across datasets — this is the convention used by scikit-uplift.metrics.qini_auc_score and what we expose as qini_coefficient.

The perfect-ordering curve is constructed by sorting observations on y * (2t - 1): positive responders in the treated group first, negative responders in the control group last. This is the standard sklift convention (Gutierrez & Gerardy 2017, "Causal Inference and Uplift Modeling: A Review of the Literature").

Two implementation notes that bit me before:

1. The ratio n_treated[:k] / n_control[:k] is undefined at k=0 and at the edge case where a prefix has zero control rows. We clamp using np.where instead of try/except to keep the function vectorised.

2. Ties in the score must be broken consistently across runs — otherwise qini(score, t, y, seed=0) and the same call later return slightly different numbers. We resolve ties via numpy's stable sort on the negated score, which preserves insertion order.

QiniCurve dataclass

The cumulative uplift curve and its Qini coefficient.

population_share and cumulative_uplift describe the curve itself; use them for plotting. qini_coefficient is the normalised scalar (raw area / perfect area), bounded in roughly [-1, 1]. qini_raw is the un-normalised area for cross-package comparability.

qini_curve(score, treatment, outcome)

Compute the Qini curve and the normalised Qini coefficient.

Parameters:

Name	Type	Description	Default
score	NDArray1D | list[float]	Predicted individual treatment effect (or any scalar to rank by).	required
treatment	NDArray1D | list[int]	Binary 0/1 array of treatment indicators.	required
outcome	NDArray1D | list[int]	Binary 0/1 array of observed outcomes.	required

Returns:

Type	Description
QiniCurve	Curve plus the normalised coefficient qini_coefficient and the un-normalised qini_raw.

qini_coefficient(score, treatment, outcome)

Normalised Qini coefficient — bounded in roughly [-1, 1].

Equivalent to qini_curve(...).qini_coefficient.

qini_raw(score, treatment, outcome)

Un-normalised Qini area (Radcliffe 2007 raw definition).

Useful when comparing to legacy packages that don't normalise.

uplift_bench.metrics.auuc

Area Under the Uplift Curve (AUUC).

Distinguished from Qini by the y-axis: AUUC plots the raw difference of treated and control responder counts at each prefix, rather than the re-weighted incremental uplift used by Qini. The two coincide when the treatment/control sample sizes are perfectly balanced; they diverge as the imbalance grows.

We report AUUC normalised by the theoretical max (perfect-ranking AUUC) so the reported number lives in roughly [-1, 1] and is comparable across datasets. The unnormalised area is also returned for users who need it.

auuc(score, treatment, outcome)

Compute AUUC and the underlying curve.

uplift_bench.metrics.uplift_at_k

uplift@k — average uplift in the top-k targeted fraction.

Used by marketers who only have budget for the top X% of customers. The question is: among those, what's the realised lift over a random pull of the same size?

Implementation detail: we compute the difference of means of the outcome between treated and control rows inside the top-k. This is the natural estimator under random assignment; under confounded assignment it's biased but that's what every public benchmark uses, so we match for comparability.

uplift_at_k(score, treatment, outcome, k)

Compute uplift in the top-k fraction.

Parameters:

Name	Type	Description	Default
score	NDArray1D | list[float]	Predicted treatment effect (or anything to rank by, descending).	required
treatment	NDArray1D | list[int]	Binary 0/1 arrays.	required
outcome	NDArray1D | list[int]	Binary 0/1 arrays.	required
k	float	Fraction in (0, 1]. e.g. 0.10 → top decile.	required

Returns:

Type	Description
float	Realised mean(y | T=1, top-k) - mean(y | T=0, top-k). NaN if either sub-group is empty in the top-k (the right answer — the metric is undefined, not zero).

uplift_bench.metrics.decile

Per-decile uplift table.

Splits the population into k equal-size buckets ordered by predicted uplift, reports the realised uplift in each bucket. The natural diagnostic to look at after Qini/AUUC: monotone-decreasing buckets confirm the model has useful ordering; a flat or zigzagging table tells you the "good" Qini was luck on the head and tail.

decile_table(score, treatment, outcome, n_buckets=10)

Build a per-decile (or n-tile) uplift table.

Parameters:

Name	Type	Description	Default
score	NDArray1D | list[float]	Same shapes as elsewhere.	required
treatment	NDArray1D | list[float]	Same shapes as elsewhere.	required
outcome	NDArray1D | list[float]	Same shapes as elsewhere.	required
n_buckets	int	Number of equal-size buckets. Default 10 (deciles).	10

Returns:

Type	Description
DataFrame	Columns: bucket (1..n_buckets, 1 = highest score), n_treat, n_ctrl, mean_y_treat, mean_y_ctrl, uplift, n_total.

uplift_bench.metrics.bootstrap

Bootstrap confidence intervals.

We support two variants:

• Percentile — the simple one. Take the empirical 2.5/97.5 quantiles of the bootstrap distribution. Cheap and good enough for symmetric metrics on big samples.
• BCa (bias-corrected accelerated) — Efron's improvement on percentile. Adjusts for bias and skewness using a jackknife estimate of acceleration. Substantially more accurate on small samples or skewed metrics like Qini, which is why we recommend it as the default.

Plus paired_bootstrap_test: given two metric values computed from the same data (e.g. Qini for model A vs Qini for model B), is A significantly better than B at level alpha? Implemented as a bootstrap on the difference of metrics, sharing the resampled indices between A and B (paired) so the comparison cancels out shared noise.

Implementation notes:

• We use joblib for parallelism. For metrics that take ~1 ms each and a default n_boot=1000, the overhead of a process pool dominates; we default n_jobs=1 and let the user opt in.
• The bootstrap RNG is seeded so reruns are bit-identical. Inside parallel workers we derive child seeds via SeedSequence.

bootstrap_ci(metric, score, treatment, outcome, *, n_boot=1000, alpha=0.05, method='bca', seed=0, n_jobs=1)

Bootstrap a confidence interval for a single metric.

Parameters:

Name	Type	Description	Default
metric	MetricFn	Callable taking (score, treatment, outcome) → float.	required
score	NDArray1D	Aligned arrays.	required
treatment	NDArray1D	Aligned arrays.	required
outcome	NDArray1D	Aligned arrays.	required
n_boot	int	Number of bootstrap resamples. 1000 is the textbook minimum for a 95% CI; bump to 5000 for tighter Qini CIs on small datasets.	1000
alpha	float	Significance level. 0.05 → 95% CI.	0.05
method	CIMethod	'percentile' or 'bca'.	'bca'
seed	int	Bootstrap RNG seed for reproducibility.	0
n_jobs	int	joblib parallelism. >1 helps when metric itself takes more than a few ms; for cheap metrics the IPC overhead dominates.	1

Returns:

Type	Description
BootstrapCI	Point estimate (computed on the original data), lower, upper.

paired_bootstrap_test(metric, score_a, score_b, treatment, outcome, *, n_boot=1000, seed=0)

Test whether metric(A) > metric(B) significantly.

Uses paired bootstrap (same indices for both) so the comparison cancels shared sampling noise. Returns:

• observed_diff - metric(A) - metric(B) on the original sample.
• ci_low/ci_high — 95% percentile CI of the resampled difference.
• p_value_one_sided — bootstrap test p-value for H0: metric(A) ≤ metric(B), H1: metric(A) > metric(B). Computed via the centered bootstrap distribution (Efron & Tibshirani 1993, §16.4): we recenter diffs to be a draw from the null and ask how often the recentered statistic exceeds observed_diff.

The recentered formula simplifies algebraically to mean(diffs <= 0) — but only if the centering shift observed_diff is the only shift between empirical and null distribution. We make that assumption explicit by computing it as such.