Statistical Testing & Calibration

4.4 Statistical Testing & Calibration

Statistical Testing

The simudyne.engine.statistics module provides non-parametric tests for comparing model outputs, distribution distance metrics, and multiple-comparison corrections.

Two-Sample Tests (each returns a TestResult with statistic, p_value, reject_null, effect_size):

Function	Signature	Description
`ks_two_sample`	`(x, y, alpha=0.05) -> TestResult`	Kolmogorov-Smirnov: max CDF difference
`anderson_darling_two_sample`	`(x, y, alpha=0.05) -> TestResult`	Anderson-Darling: weighted tail sensitivity
`permutation_test`	`(x, y, statistic_fn=None, n_permutations=10000, seed=42, alpha=0.05) -> TestResult`	Permutation: exact p-value via resampling
`mann_whitney_u`	`(x, y, alternative="two-sided", alpha=0.05) -> TestResult`	Mann-Whitney U: rank-based location test

Bootstrap Methods:

Function	Description
`bootstrap_ci(data, statistic_fn, n_bootstrap=10000, alpha=0.05, seed=42)`	BCa confidence interval for any statistic
`bootstrap_two_sample(x, y, statistic_fn, n_bootstrap=10000, seed=42, alpha=0.05)`	Two-sample bootstrap test

Distribution Distances (each returns a float):

Function	Signature	Description
`wasserstein_distance`	`(x, y) -> float`	Earth mover's distance (L1 optimal transport)
`energy_distance`	`(x, y) -> float`	Energy distance (Szekely & Rizzo)
`mmd_distance`	`(x, y, kernel="rbf", bandwidth=None) -> float`	Maximum Mean Discrepancy with RBF kernel
`jsd_distance`	`(x, y, n_bins=50) -> float`	Jensen-Shannon divergence (symmetric KL)

Multiple-Comparison Corrections (for controlling family-wise error rate when running many tests):

Function	Signature	Description
`bonferroni_correction`	`(results: List[TestResult], alpha=0.05) -> MultipleTestResult`	Conservative: alpha/n
`holm_correction`	`(results: List[TestResult], alpha=0.05) -> MultipleTestResult`	Step-down Holm-Bonferroni
`benjamini_hochberg_fdr`	`(results: List[TestResult], alpha=0.05) -> MultipleTestResult`	BH-FDR: controls false discovery rate
`benjamini_yekutieli_fdr`	`(results: List[TestResult], alpha=0.05) -> MultipleTestResult`	BY-FDR: handles dependent tests

ABC-SMC Calibration

The ABCSMCCalibrator performs likelihood-free Bayesian calibration using Approximate Bayesian Computation with Sequential Monte Carlo. It does not require a likelihood function --- instead, it compares simulated features to observed features using a distance metric, and uses importance sampling with adaptive epsilon contraction to converge on the posterior distribution of parameters.

class ABCSMCCalibrator:
    def __init__(
        self,
        priors: List[Prior],            # Prior distributions for each parameter
        feature_fn: Callable,           # Function that extracts features from traces
        epsilon_decay: float = 0.9,     # Per-generation epsilon contraction rate
        min_acceptance_rate: float = 0.01,  # Stop if acceptance rate drops below this
    ) -> None

    def calibrate(
        self,
        model_cls: Type[ABMModel],
        obs_features: Dict[str, float],    # Observed feature values to match
        base_config: Dict[str, Any],       # Base model config (non-calibrated params)
        n_particles: int = 500,            # Number of particles in the population
        n_generations: int = 10,           # Maximum number of SMC generations
        mc_seeds: int = 5,                 # Seeds per particle evaluation
        max_workers: int = 0,              # Parallel workers (0 = auto)
        progress_callback: Optional[Callable] = None,
    ) -> ABCSMCResult

The Prior dataclass supports five distribution types:

Distribution	Parameters	Description
`"uniform"`	`low`, `high`	Flat prior over [low, high]
`"log_uniform"`	`low`, `high`	Uniform in log-space (for scale parameters)
`"truncated_normal"`	`mean`, `std`, `low`, `high`	Gaussian truncated to [low, high]
`"beta"`	`alpha`, `beta`, `low`, `high`	Beta distribution scaled to [low, high]
`"scipy"`	`scipy_dist`	Any frozen `scipy.stats` distribution

The ABCSMCResult provides posterior analysis:

Method / Property	Return Type	Description
`map_estimate()`	`Dict[str, float]`	Maximum a posteriori parameter values
`posterior_mean`	`Dict[str, float]`	Weighted posterior mean
`posterior_std`	`Dict[str, float]`	Weighted posterior standard deviation
`posterior_ci_95`	`Dict[str, Tuple]`	95% credible intervals
`effective_sample_size`	`float`	ESS of weighted particles
`summary_table()`	`str`	Formatted table of all posteriors
`posterior_samples(n)`	`np.ndarray`	Weighted resamples from posterior
`posterior_predictive(feature_fn, n)`	`Dict[str, ndarray]`	Predictive distribution of features

The compare_models() function calibrates multiple competing models and ranks them by complexity-penalised distance, implementing the "description over fitting" principle: a simpler model that fits nearly as well is preferred over a complex model that fits marginally better.

4.4 Statistical Testing & Calibration​