Features & Validation

4.3 Features & Validation

The feature system is the SDK's unit of measurement, validation, and calibration. Rather than comparing raw time series visually, models are evaluated by extracting quantitative features (stylized facts) from their output and comparing these to empirical targets.

Feature Protocol

A Feature is a named, described callable that extracts a single scalar from model output traces:

@dataclass
class Feature:
    name: str                                         # Unique identifier
    description: str                                  # Human-readable explanation
    calculator: Callable[[Dict[str, np.ndarray]], float]  # Extraction function
    required_fields: Tuple[str, ...] = ()             # Trace fields needed
    target_range: Optional[Tuple[float, float]] = None  # Empirical target range
    domain: str = "general"                           # Domain tag
    unit: str = ""                                    # Units of measurement
    metadata: Dict[str, Any] = field(default_factory=dict)

Features are composed into FeatureSet collections:

class FeatureSet:
    def __init__(self, name: str, features: Optional[List[Feature]] = None) -> None

    def add(self, feature: Feature) -> None
    def compute_all(self, traces: Dict[str, np.ndarray],
                    cache: Optional[Dict[str, float]] = None) -> FeatureResult
    def compute_from_mc(self, mc_result, trace_field_map=None, cache=None) -> FeatureResult
    def validate(self, traces, ground_truth=None, model_name="unnamed") -> ValidationReport
    def compare(self, result_a: FeatureResult, result_b: FeatureResult,
                label_a="A", label_b="B") -> ComparisonReport
    def distance(self, traces, targets: Dict[str, float],
                 weights=None, distance_fn=None) -> float

The validate() method checks each feature's computed value against its target_range and returns a ValidationReport with an overall score (0.0-1.0), a letter grade (A-F), and per-feature results:

@dataclass
class ValidationReport:
    model_name: str
    n_features_tested: int
    n_features_passed: int
    pass_rate: float
    results: List[FeatureValidationResult]
    overall_score: float
    grade: str  # "A" (>0.9), "B" (>0.8), ..., "F" (<0.5)

    def summary_table(self) -> str
    def failed_features(self) -> List[FeatureValidationResult]

The distance() method computes the weighted sum of normalised distances between computed features and target values. This is the loss function used by the ABC-SMC calibrator: minimising FeatureSet.distance() calibrates the model to match observed data.

Pre-Built Domain Libraries

Three domain-specific feature libraries ship with the SDK. Each provides a pre-configured FeatureSet with empirically-validated target ranges:

Financial Features (simudyne.engine.features.contrib.financial) --- 8 features for financial market models:

Note: Domain feature libraries were moved from simudyne.engine.features.{domain} to simudyne.engine.features.contrib.{domain} in v0.7.1. The old import paths (simudyne.engine.features.financial, etc.) remain as deprecation shims and will continue to work but emit a DeprecationWarning.

Feature	Target Range	Description
fat_tails	[2.0, 5.0]	Hill tail index of return distribution
volatility_clustering	[0.05, 0.20]	Autocorrelation of squared returns at lag 1
leverage_effect	[-0.50, -0.05]	Correlation between returns and future volatility
return_unpredictability	(-0.05, 0.05)	Absolute autocorrelation of returns at lag 1
long_memory	[0.55, 0.90]	Hurst exponent of absolute returns
excess_kurtosis	[1.0, 50.0]	Excess kurtosis of return distribution
negative_skewness	[-0.50, -0.10]	Skewness of return distribution
correlation_regime	varies	Cross-correlation structure across regimes

Supply Chain Features (simudyne.engine.features.contrib.supply_chain) --- 4 features:

Feature	Target Range	Description
bullwhip_effect	[1.5, 5.0]	Order variance amplification ratio
cascade_fraction	[0.1, 0.5]	Fraction of nodes affected by disruption
inventory_variance_ratio	varies	Inventory variance relative to demand variance
demand_amplification	varies	Demand signal amplification through tiers

Epidemiology Features (simudyne.engine.features.contrib.epidemiology) --- 4 features:

Feature	Target Range	Description
basic_reproduction_number	[1.0, 10.0]	R0 estimated from early exponential growth
epidemic_peak_timing	None	Tick at which infection count peaks
attack_rate	varies	Final fraction of population infected
epidemic_duration	varies	Ticks from first to last infection

---