Monte Carlo Evaluation & Experimentation

4.2 Monte Carlo Evaluation & Experimentation

MCRunner

The Monte Carlo runner executes a model across multiple independent seeds and aggregates results. Each seed produces an independent realisation of the stochastic process, and the aggregate statistics characterise the model's distributional behaviour.

class MCRunner:
    def __init__(self, max_workers: Optional[int] = None) -> None

    def run(
        self,
        model_cls: Type[ABMModel],
        config: Dict[str, Any],
        mc_spec: Dict[str, Any],           # {"seeds": 100, "steps": 200, "burn_in": 10}
        metrics_spec: Optional[List[Dict]] = None,  # Metrics to compute per seed
        trace_fields: Optional[List[str]] = None,    # Time series fields to capture
    ) -> MCResult

The MCResult contains per-seed results and aggregate statistics:

@dataclass
class MCResult:
    model_class: str
    seeds_run: int
    seeds_completed: int
    seeds_failed: int
    total_steps: int
    burn_in_steps: int
    observation_steps: int
    wall_time_seconds: float
    seed_results: List[SeedResult]
    aggregate_metrics: Dict[str, Dict[str, float]]  # metric -> {mean, std, min, max, ...}
    failure_cases: List[Dict[str, Any]]

    @property
    def crash_rate(self) -> float
    def get_metric_values(self, metric_name: str) -> np.ndarray

Each SeedResult contains the seed number, success flag, time series traces (as Dict[str, np.ndarray]), computed metrics (as Dict[str, float]), and wall time.

The ProcessPoolMCRunner extends MCRunner with configurable parallelism:

class ProcessPoolMCRunner:
    def __init__(self, max_workers=None, parallel_seeds=True, batch_size=0) -> None

    def run(self, model_cls, config, mc_spec, metrics_spec=None,
            trace_fields=None, progress_callback=None) -> MCResult

ExperimentRunner

The experiment runner performs parameter sweeps: it varies model parameters systematically and runs Monte Carlo evaluation at each combination to map the parameter-to-output landscape.

Three sweep strategies are provided:

Strategy	Description	Use Case
GridSweep(*param_ranges)	Exhaustive Cartesian product of all parameter values	Low-dimensional spaces (2-3 params)
LatinHypercubeSweep(*param_ranges, n_samples=50)	Space-filling Latin Hypercube Sampling	Medium-dimensional spaces (4-8 params)
SobolSweep(*param_ranges, n_samples=64)	Quasi-random Sobol sequence	High-dimensional spaces, sensitivity analysis

Each sweep operates on ParameterRange objects:

class ParameterRange:
    def __init__(
        self,
        name: str,                        # Parameter name (key in config)
        values: Optional[List[Any]] = None,  # Explicit value list (for grid)
        min: Optional[float] = None,      # Range minimum (for LHS/Sobol)
        max: Optional[float] = None,      # Range maximum
        steps: int = 10,                  # Number of grid steps (if values not given)
    ) -> None

The runner:

class ExperimentRunner:
    def __init__(self, max_workers: int = 0) -> None

    def run(self, model_cls, base_config, sweep, mc_seeds=10, steps=100,
            metrics_spec=None, trace_fields=None, progress_callback=None) -> ExperimentResult

    def run_sensitivity(self, model_cls, base_config, param_ranges,
                        metric_name, mc_seeds=10, steps=100, ...) -> Dict[str, float]

    def run_sobol_analysis(self, model_cls, base_config, param_ranges,
                           metric_name, mc_seeds=10, steps=100, n_samples=64,
                           n_bootstrap=100, ...) -> SobolResult

The ExperimentResult provides best_combination(metric_name, maximize=True) to find optimal parameters and summary_table() for display. The SobolResult provides first-order and total-order Sobol sensitivity indices with bootstrap confidence intervals, and most_influential() returns the parameter name with the highest total-order index.

---