Multirun Setup

The RunnerBackend and ModelRunner classes need to be used together to setup and run a simulation that can run multiple times.

This guide assumes you have a Model created which implements simudyne.core.Model The following imports are assumed

Imports required for multirun (Java)

import simudyne.core.exec.runner.ModelRunner;
import simudyne.core.exec.runner.RunResult;
import simudyne.core.exec.runner.RunnerBackend;

The RunnerBackend class is needed to set up the ModelRunner. The default RunnerBackend can be created by using the RunnerBackend create method. This will create an instance of the RunnerBackend that can be used to run the model locally.

Create a model runner (Java)

RunnerBackend runnerBackend = RunnerBackend.create();

Create the ModelRunner for a specific model with RunnerBackend#forModel passing the class of the model or RunnerBackend#forConfig passing the ModelConfiguration for the model

Create a model runner for a specific model (Java)

ModelRunner modelRunner = runnerBackend.forModel(myModel.class);

Setting up the multirun

Use the ModelRunner to create 'definitions' which define the parameters of the multiruns, as well as the export path to be used and config to set for all runs.

Settings for a ModelRun

modelRunner
    .forExportPath("Some/absolute/path")
    .withConfig("configField", "configValue");

ModelRunner Definitions

There are three types of ModelRunner Definitions that can be used.

• BatchDefinitionsBuilder
• ModelSamplerDefinitionsBuilder
• ScenarioDefinitionsBuilder

BatchDefinitionsBuilder

This is used to run a model multiple times for a set number of ticks. A single set of inputs can be specified that will be applied to all runs before setup.

The default data output for batch runs.

• Data output of all runs in held in memory and aggregated which can be inspected at the end of the run. To change this, set the config field "core.return-data" to false.
• Data output is not written to file (parquet). To change this, set the config field "core.parquet-export.enabled" to true, or simply set the export path using the method forExportPath on the modelRunner (as shown above).

Define the parameters for the Batch run by creating an instance of BatchDefinitionsBuilder and setting this instance as the run definition to use for the model runner.

  BatchDefinitionsBuilder runDefinitionBuilder = 
    BatchDefinitionsBuilder.create()
    .forRuns(100) // a required field, must be greater than 0.
    .forTicks(50) // a required field, must be greater than 0.
    .withInput("preference", 0.75) // can be used as many or little times depending on how many input fields you want to specify.
    .forGeneratorSeed(467854386543L); // This is an optional field - if set, this will be the root seed that is used to set the seeds for the individual runs. If not set, the seed will be pulled from the Simudyne SDK properties file.
    
  modelRunner.forRunDefinitionBuilder(runDefinitionBuilder)

Setting input values for variables in custom objects, or variables in the global state is more complex. This is explained further below (Setting nested input values).

The seeds can also be defined for the individual runs. If specifying the seeds for the runs, the number of seeds provided must match the number of runs specified.

  BatchDefinitionsBuilder runDefinitionBuilder = 
    BatchDefinitionsBuilder.create()
    .forRuns(3) // a required field, must be greater than 0.
    .forTicks(50) // a required field, must be greater than 0.
    .forSeeds(2344, 95846, 474654);    

  modelRunner.forRunDefinitionBuilder(runDefinitionBuilder)

ModelSamplerDefinitionsBuilder

This is used to run a model multiple times, for a set number of ticks. The input fields can be set for the individual runs by drawing from a list of samples.

The data output for model sampler runs

• Data output of all runs is not held in memory and aggregated which so cannot be inspected at the end of the run
• Data output is written to file (parquet)

The sample input values to use can be generated by the SDK by providing the SDK with the bounds for your input fields to generate samples for, or they can be provided by the user.

SDK generated samples

  ModelSamplerDefinitionsBuilder runDefinitionsBuilder =
    ModelSamplerDefinitionsBuilder.create()
    .forRuns(5)
    .forTicks(100)
    .forSamples(9)
    .withBoundedDimension("gridSize", 12, 100)
    .withBoundedDimension("price", 0.2, 0.5);

  modelRunner.forRunDefinitionBuilder(runDefinitionBuilder)

This will generate 9 integer samples between the values of 12 and 100 to apply to the gridSize field, and 9 double samples between the values of 0.2 and 0.5 to apply to the price field when running the model. Each sample set will be run for 5 runs, so this configuration will run for a total of 45 runs.

User specified samples

  ModelSamplerDefinitionsBuilder runDefinitionsBuilder =
    ModelSamplerDefinitionsBuilder.create()
    .forRuns(100)
    .forTicks(100)
    .withListedDimension("gridSize", 12, 34, 67, 34, 100)
    .withListedDimension("price", 0.1, 0.2, 0.3, 0.4, 0.5);

  modelRunner.forRunDefinitionBuilder(runDefinitionBuilder)

This will pair up the samples provided for gridSize and price so that we have 5 unique sample definitions, and run each of these sample defintions 100 times, for a total of 500 runs.

The seeds for the runs can either be set by setting the generator seed which will be used to create the seeds for the individual runs, or by providing a list of seeds explicitly.

Full Example:

  ModelSamplerDefinitionsBuilder runDefinitionsBuilder =
    ModelSamplerDefinitionsBuilder.create()
    .forRuns(3)
    .forTicks(100)
    .withListedDimension("gridSize", 12, 34, 67)
    .withListedDimension("price", 0.1, 0.2, 0.3)
    .forSeeds(123,345,678);

  modelRunner.forRunDefinitionBuilder(runDefinitionBuilder)

This will create the following set of runs.

The ModelSampler cannot be used to set input fields of global state fields.

ScenarioDefinitionsBuilder

This is used to build a set of scenarios for running the model, where every scenario defines input values that can be set at various ticks throughout the model.

The data output for model sampler runs:

• Data output of all runs is not held in memory and is aggregated which therefore cannot be inspected at the end of the run
• Data output is written to file (parquet)

Multiple scenarios can be built using the builder, with each scenario requiring a unique name, and the ending of each scenario definition being marked with done()

  ScenarioDefinitionsBuilder runDefinitionsBuilder =
    ScenarioDefinitionsBuilder.create()
      .createScenario("firstScenario").done()
      .createScenario("secondName").done();

  modelRunner.forRunDefinitionBuilder(runDefinitionBuilder)

Once created, each scenario can be setup with the runs, seeds, ticks and inputs to use when running the model, as with batch runs. Differently to the the batch runs, the scenario runs can also have the input values specified to be applied before running various ticks of the model.

  ScenarioDefinitionsBuilder runDefinitionsBuilder =
    ScenarioDefinitionsBuilder.create()
      .createScenario("firstScenario")
        .forRuns(5)
        .forTicks(50)
        .withInput("griSize", 8, 1000)
        .done();

  modelRunner.forRunDefinitionBuilder(runDefinitionBuilder)

This will set the value of gridSize to 1000 before running tick 8.

Setting input values for variables in custom objects, or variables in the global state is more complex. This is explained further below (Setting nested input values).

Running the multirun

ModelRunner.run will return a MultirunController, which is a view onto the async running simulation. This can be used to cancel the simulation, check its progress, and get the output of the simulation.

MultirunController run = modelRunner.run();

run.cancelRun(); // This will cancel the run

run.getProgress(); // This returns a value between 0 and 1 which reflects the progress of the simulation

run.awaitOutput(); // Block and wait for the simulation to complete and all output to be written to file
run.asyncOutput(); // This returns the future that can be used to track the completion of writing all output data to file.

run.asyncResult();// This returns the future that can be used to track the run.
RunResult runResult = run.awaitResult(); // This returns the run results object (explained below)

To understand the RunResult output, see Multirun Output

Running a distributed multirun

Using Spark

Running a distributed multirun simulation depends on different packages and imports.

These elements can be found in the spark requirement tutorial and the spark runner tutorial tutorials.

To setup a distributed multirun, create a new instance of the SparkRunnerBackend as the RunnerBackend. All other methods are the same as for the local RunnerBackend.

Multirun with spark (Java)

RunnerBackend runnerBackend = new SparkRunnerBackend();
ModelRunner modelRunner = runnerBackend.forModel(myModel.class);

Extracting information from the RunResult is explained in Multirun Output

Setting nested input values

When the input fields are nested objects, as is the case with fields inside the global state, setting the input values is slightly more complicated. A Map of key-values needs to be used to as the value for the input field.

For example, if there is a field "fillFactor" inside the global state object, setting this field would work as follows.

 HashMap<String, Object> valueMap = new HashMap<>();
    valueMap.put("fillFactor", 0.55);

    RunnerBackend.create()
        .forModel(GameOfLife.class)
        .forRunDefinitionBuilder(
            BatchDefinitionsBuilder.create().forRuns(2).forTicks(2).withInput("system", valueMap))
        .run();

The input field has a name "system" (the outer field name for all global state values), and the value set for "system" is actually a Map, which has a key "fillFactor" and the value to set for it.