2.4.1

01st June 2021

This is a minor release of the software that includes a few fixes to Parquet output as well as some smaller request features.

Upgrading

Upgrading can be done by changing the simudyne.version setting in your pom.xml to 2.4.1. You MUST also switch any instance in your pom.xml where a library like simudyne-core-abm_2.11 exists to simudyne-core-abm_2.12 - This is because 2.4 is now using Scala 12.

There should be no breaking changes going from 2.4.0 to 2.4.1, but if upgrading for the first time see below.

Fixes

• Parquet Output is no longer nested within an additional structure. This should support more common readers requirements and lessen data impact.
• Fixed an issue where some Hive configuration was using default localhost which for some deployments could pose an issue.
• If there is no simudyneSDK.properties file a warning will now notify the user, this is to mitigate confusion on why certain settings may not be working as the software would simply revert back to defaults.
• Additional checks have been made to not cause issues with missing properties that users may not require (such as Hive password / username)

Additions

• Parquet Output now supports the ability to work with INT's, previously usage of LONG or DOUBLE was required.
• A ScenarioDefinitionsBuilder has been added - previously this was only possible via Scala instead of Java like other Builders.
• The Programmable Random Number Generator (prng) now has the ability to getRandom in order to generate a RNG for usage with the default seed set by the PRNG, and a getSeed function which is preferable to grabbing to seed from configuration (which can be overwritten)

2.4.0

25th January 2021

This is a major relase of the software, including updated data handling and a more powerful console.

Upgrading

Upgrading can be done by changing the simudyne.version setting in your pom.xml to 2.4.0. You MUST also switch any instance in your pom.xml where a library like simudyne-core-abm_2.11 exists to simudyne-core-abm_2.12 - This is because 2.4 is now using Scala 12.

We have been careful in 2.4 to avoid any other major breaking changes, but there are still a few things to be aware of. Most notably will be changes to data structure and format of parquet output.

Parquet Output

We've updated the process for handling data output via Parquet. This involves usage of new libaries both to improve performance but also availabilty to easily use in both other data formats and output methods in 2.4 and 2.5

Hive Integration

Changes to our output libraries allows us to now connect via JDBC to a Hive database using the exact same structure of parquet output. Users can simply modify their settings to move local output to Hive.

H2 Output

Updates to data format will allow storage to H2

Chart Improvements

Console charts now have better handling for negative values, as well as optional bounding in order to easily analyze your data.

Import updates and bug fixes

Several classes may have been moved, so their imports will have to be updated to reflect that. The easiest way to do that is to remove the old imports which are broken, and with your mouse over the erroring classes, press alt + enter (this only works if using Intelij). Intelij will now be able to find and add the new import for you.

Additional various small fixes to other bugs have been added, mostly concerning the console.

2.3.0

16th September 2019

2.3 includes multiple new modules, new APIs, and a more powerful console. We have been careful in 2.3 to avoid any major breaking changes, but there are still a few things to be aware of. A change in imports in particular will be a breaking change for most models, and a change in the paquet output config field.

Data output with output channels

More flexible, detailed data can now be output and written to parquet, without having to use annotations and output at every tick.

ModelRunner

The model runner has been updated, allowing users to create different types of multiruns, where previously, only batch runs were possible.

Parquet output

Previously, parquet output was managed via a config flag "core.parquet-export". This has now been changed to "core.parquet-export.enabled"

Agent context

Agents now have a view onto the model context, which provides information about the current tick and gives access to the output channels.

Import updates

Several classes have been moved, so their imports will have to be updated to reflect that. The easiest way to do that is to remove the old imports which are broken, and with your mouse over the erroring classes, press alt + enter (this only works if using Intelij). Intelij will now be able to find and add the new import for you.

2.2.0

29th March 2019

Immutable Schema

2.2 is beginning preparations for future improvements for data integration. As part of this, models will need to have a fixed type once they are initialised. We have introduced a new method, init() that can be implemented for models, which is called just as the model is created. After this method has been called, all agent and link types should be registered, and all accumulators created. Demonstrations of this and other improvements can be found in the new Trading Tutorial.

To ease upgrading you can disable this restriction for this release, by disabling the feature in your configuration. Note that this flag is planned for removal in a future release, and so you will be warned when it is disabled.

# In your simudyneSDK.properties file

feature.immutable-schema = false

Migrating for this involves moving any accumulator creation to an init() method on your model, as well as registering agent and link types. For example, in the Trading tutorial model, this is done as so:

public void init() {
  createLongAccumulator("buys", "Number of buy orders");
  createLongAccumulator("sells", "Number of sell orders");
  createDoubleAccumulator("price", "Price");

  registerAgentTypes(Market.class, Trader.class);
  registerLinkTypes(Links.TradeLink.class);
}

Java 9+ Compatibility

The Simudyne SDK has been made warning free for versions of Java 9 and above. As part of the restrictions this introduces, annotated fields and methods (e.g. your @Variable and @Input fields) must now be marked as public. Users of Java 8 will be unaffected by this change.

Deprecations

SeededRandom.create()

Calling SeededRandom.create() without any arguments is now deprecated. The behaviour of this method is to return a new SeededRandom instance, using the seed configured in your project, or a random seed if there is no seed configured. However, this method was easy to use incorrectly, as calling it twice is almost certainly an error (returning two separate PRNG with the same seed, and so generating the same sequences of outputs independently).

The advised replacement is to use ModelContext.getPrng(). Within your AgentBasedModel class, you can get the context by calling getContext(). This has identical behaviour to SeededRandom.create(), except that multiple calls will return the same instance of SeededRandom. If you want to keep the previous behaviour, you can call SeededRandom.create(long seed) explicitly with the same seed.

Field Groups

Field groupings set through field annotations are no longer used on the console. These annotation properties have been deprecated to maintain compatibility, and will be removed in a future version. Specifically this related to the @ModelSettings(groups = {}) setting, and field level annotation settings such as @Variable(group = 0).

AgentBasedModel#registerAgentType

This method has been deprecated in favour of AgentBasedModel#registerAgentTypes which can take any number of classes.

Console

The console has been given a visual update, providing a more polished interface and better indication of types of data through colouring. Along with this there are a large number of improvements and new features, here are the highlights:

Time Control

• The new time bar at the top of the side panel can be moved to a point in the models past. This syncs with all point-in-time visualisations such as the network, so you can see these more detailed visualisations at any point in the models timeline.
• Time series charts can now be zoomed, and a new time bar beneath each time series chart allows the zoomed view to be panned through time.

Network View

• Network visualisation has been rebuilt to use hardware acceleration, allowing much larger networks of agents to be visualised.
• A circular layout option has also been added, which is much faster to layout, and better for visualising some types of networks.

Charts

• Time series charts can now be toggled between bar and line chart views, with bar chart view being the default.
• Agent attributes can now be aggregated as histograms, showing the distribution of the attribute across the agents at a point in time. Combined with the time bar control, you can now see how distributions evolve as your model runs.
• Balance sheet and order book visualisations for the new experimental Financial Tool kit.
• Charts can now show real time value, rather than tick numbers. The mapping from tick to time can be controlled through the @ModelSettings annotations on the Model class.

Sidebar

• Sidebar can now be collapsed, giving more space for visualisations.
• Inputs now better indicate when they are read only.

Installation

Global Install Support

License files can now be placed in a users home directory, allowing all Simudyne SDK projects for that user to find the license. This eases installation to be a once-per-machine task, rather than once per project. Existing installations will continue to work, but it is advised for everyone to update to the new approach

License Server Support

The SDK now supports a remote licensing server. This allows the license placed on a users machine to contain only simple user identification, with the server controlling aspects such as expiration. This removes the need for rotating license files on expiry, as they can be extended by the server independently. This approach will be rolled out first to academic users of Simudyne. The existing licenses, which requires no network connection, will continue to function and are not planned to be removed, in support of enterprise environments.

New Modules

Model Scaffolding

Model Scaffolding is a tool for generating the initial skeleton of an ABM, using a definition of the model in the JSON data format. This is a step towards simplifying the initial creation of an ABM, making it easier to declare agents and links quickly. It is distributed within the new simudyne-maven-plugin module, which is a plugin for the Maven build system. We will be looking to roll out new tooling at the build stage in the future, to continue improving the modeling experience.

New APIs

Time Series Variables

Agents now support Time Series Variables through windowed values, which are values that track a value through a certain number of ticks.

Empirical Distributions

Empirical distributions can now be loaded from a Source, using the new SeededRandom.empiricalFromSource(Source source) API. This allows you to easily construct distributions from source data, structured as a histogram, and then use them in your model.

ModelContext

A new API object is available in all models, called ModelContext. In AgentBasedModel this is available by calling getContext(), and returns the context of the current model. Through this context you can access additional APIs relating to the model, including the current tick (through getTick) and the root PRNG (through getPrng).

Model.init

Model.init() is a new lifecycle method available on all Models. This acts similarly to a constructor, however contextual APIs such as ModelContext will be available. This interacts with the new Immutable Schema restrictions, as after init() has been called, methods affecting the data structure of the model can no longer be called.

Model Sampler

The Model Sampler is a new interface onto multi-run, allowing more declarative execution of collecting samples from a model over a given parameter space. Currently this is only available through the REST API.

Experimental Modules

In the pursuit of newer features to end users, but clear communication of stability and readiness for production, we have a new classification of modules: experimental. These are modules that are not guaranteed to be stable, either in the implementation or their API, and so are not suitable for production use. They have however been tested internally by us, and are at the point where external users should be able to usefully try and test them, and provide feedback on whether these modules are suitable.

Financial Tool Kit

This module provides two new APIs for defining domain specific agent-based models. The first API allows the definition of a system of traders and markets, where the markets hold an internal order book which is executed on via messages from traders. The second is a lending model, where a system of borrowers and lenders take out loans and make repayments, tracked through balance sheets.

SparkGraph

SparkGraph has been reclassified as an experimental module, with no other changes.

DistributedGraph

DistributedGraph is a new implementation of a distributed execution engine for very large networks of agents. It is designed to be more performant than SparkGraph, and is available for initial testing.

Other Enhancements

Command Line Interface

The CLI mode is a new addition that allows models to be run from the command line, rather than using the webserver. While the server is still used to run the task, once it is finished it will exit. Enabling this new mode is striaghtforward, as you just need to pass program arguments through to the existing Server.run call. If any arguments are passed, the server will try and execute them, otherwise if no arguments are passed the server will start as usual.

Parquet Directory Structure

For parquet output on batch and multi-run, the SDK needs to output into a set of directories containing all of the results. In previous versions this was done by grouping results into separate directories per run. This was friendly to humans, as it was clear where the results for a given run are located, but in practice these results are often used from query engines such as SparkSQL. For these systems, files should be grouped by their data structure (columns).

This release adds a new configuration setting to control the output structure, with the default the new query-friendly group-by-type.

Model Health Checks

The server is now able to perform a set of health checks on models when it is starting up, to advise on potential problems. With this release there exists a check for Repeatable Models which attempts to detect if your model is deterministic with respect to the configured seed. If it is not, then it will log a warning advising that the model fails the test.

2.1.0

7th November 2018

This major release of the Simudyne SDK is focused on three main themes:

1. Improved data input / output for simulation models.
2. An enhanced console for better visualisation of agent-based model output.
3. Better modelling APIs for agent-based models, and the introduction of a new System Dynamics modelling API.

Breaking Changes

• License files are now required; please see the licensing section below. Existing contract holders will be contacted regarding license provisioning.
• The messaging API has been updated. Please see the migration notes further down these release notes for instructions on upgrading.

Licensing

The Simudyne SDK now requires an individually issued license file to operate. This license contains certain characteristics and restrictions, such as validity dates and contractual CPU core limits. This file must be placed and referenced correctly in a Simudyne project for the Simudyne libraries to function. These license files should be requested from Simudyne.

Data Import and Export

Data about the current state of a simulation can be retrieved as JSON via the REST API. The Simudyne SDK can also export all simulation data to Parquet files for further analysis.

Console

Network visualisation view

The network visualisation tile has been removed. Network visualisation now has its own tab, which paves the way for enhanced network visualisations and analytics in forthcoming releases.

Agent Attribute Aggregation

Agent attributes can now be aggregated (mean or total) and represented as line charts on the console.

Agent Attribute Time-Series

Agent attributes can be represented as line charts on the console. This allows the user to observe the evolution of an agent attribute (e.g. income) through time.

Tile Search Tool

Tiles are made available through a new search tool, allowing the user to toggle on and off the visualisations they want presented on the console.

Modelling API

BlankLink Removal

The default link type BlankLink has been removed. When building links via connectors, link type must always be defined and declared. This has been done to encourage the good modelling practice of naming the defined relationships, as this can become confusing when transitioning from simple models to more complicated ones.

Messaging API

For broadcastMessage, a new fluent API now provides the capability to filter and customise messages sent based on the link the message will be sent along. The capability to broadcast along all link types has been removed, as this was rarely the modeller's intention once multiple link types are defined. Broadcasting along links could cause confusing and unexpected behaviour.

//////
// 2.0
//////
cell.broadcastMessage(new Messages.Aliveness(cell.alive),
                      Links.Neighbour.class);

// Any message received is inside a wrapper Message type.
Message<Messages.Aliveness> m = cell.getMessageOfType(Messages.Aliveness.class);

//////
// 2.1
//////

// Using a generic message class.
cell.getLinks(Links.Neighbour.class)
    .send(Messages.Aliveness.class, (msg, link) -> msg.alive = cell.alive);

// Using a specialised message class (shown below)
// Here the value given is assigned into the body
// of the constructed messages automatically.
cell.getLinks(Links.Neighbour.class)
    .send(Messages.Aliveness.class, cell.alive);

// New functionality, customising and filtering messages sent
// using link information.
cell.getLinks(Links.Neighbour.class)
    /* filter messages sent based on link */
    .filter(link -> link.property < 100)
    /* can set message body based on link attributes */
    .send(Messages.Aliveness.class,
          (msg, link) -> msg.setBody(link.property > some_condition));

// Any message received is now the message type itself
Messages.Aliveness m = cell.getMessageOfType(Messages.Aliveness.class);

For direct messaging with sendMessage, use the same message construction API based around Agent#send, but use Messaging#to to define the subsequent destination.

//////
// 2.0
//////

cell.sendMessage(new Messages.Aliveness(cell.alive), sender);

//////
// 2.1
//////

cell.send(Messages.Aliveness.class, cell.alive).to(sender);

Message and Link Classes

User-defined message and link types should now extend the built-in Message and Link classes. These classes should not define any constructor that requires parameters. There are also built-in special classes for messages containing a single primitive type, e.g. Message.Boolean, Message.Double, etc. When using these types, there is a convenient shorthand when constructing messages, as shown above.

//////
// 2.0
//////

public class Messages {
    public static class Aliveness {
        public boolean alive;

        public Aliveness(boolean alive) {
            this.alive = alive;
        }
    }
}

//////
// 2.1
//////

public class Messages {
    // Generic Message
    public static class Aliveness extends Message {
        public boolean alive;
    }

    // Extending the special message class
    public static class Aliveness extends Message.Boolean {}
}

Links should similarly extend the Link class. There is currently a single specialisation of link, Link.Empty, that may provide optimisation benefits in the future.

System Dynamics API

System Dynamics is an approach to understand complex nonlinear systems via the creation of stocks, flows, delays, and feedback loops. Originally developed in the 1950's to model industrial or factory processes, it has been used to understand and map large, complex systems to enact policies in the fields of trade, banking, regulation, and more.

A new System Dynamics API has been introduced, allowing users to implement system dynamics models in Simudyne.

Auto-Compilation

The Simudyne auto-compiler allows models to be automatically compiled when changes are made. For now, this must be explicitly enabled via opt-in configuration of the development server.

2.0.5

22nd August 2018

This is a minor improvement release, including a compatibility update for deployment on Spark 2.2.x.

Dependency Changes

• json4s has been downgraded to version 3.2.11 to match Spark 2.2.x.

Module changelogs

core-abm

• Agent#setEnvironment signature has been updated to be consistent with that in Vertex, resolving a warning showing in some IDEs.
• Performance for frequently used LongAccumulator and DoubleAccumulator when running on machines with high core counts has been improved.

core

• Annotating final or effectively final values such as Scala vals as inputs now gives a clearer, eager error detailing the issue.

2.0.4

2nd August 2018

This is a bug-fix release, most importantly fixing issues relating to execution on Spark using YARN.

Dependency Changes

• akka and akka-http have been updated to their latest versions (2.5.13 and 10.1.3 respectively)
• akka-slf4j is now included as a dependency.
• akka-http-testkit is no longer included as a dependency.

Module Changelogs

core-abm

• When loading links from a Source, ids are now validated to be in the expected range of IDs.

nexus-server

• Timeout for long operations has been increased from 20 seconds to 120 seconds.

core-graph-spark

• Resolved issues with execution on Spark using YARN, related to Kryo ClassLoaders.
• Resolved issues related to serialisation occurring with Spark 2.2.0.

2.0.3

26th June 2018

This is a critical bug-fix release relating to deployment on Cloudera Spark clusters.

Dependency Changes

• Spark dependency is now properly marked as "provided" scope.
• fastutil library has been downgraded from version 8.1.1 to 6.3, to match commonly deployed ambient libraries.

console

• String input fields should no longer be null when set from the console.

2.0.2

8th June 2018

Breaking Changes

This release is primarily focused on bug fixes but does include a minor API change to be aware of when upgrading. Data injectors for links now receive the agent itself rather than an InitContext. This change allows link attributes to be set based on the agent, and for agents to be altered based on their links. Access to the seeded PRNG and agent ID is then available directly from the agent, e.g. Agent#getPrng().

Group<MyAgent> = generateGroup(MyAgent.class, 100)

// Before
myGroup.fullyConnected(otherGroup, MyLink.class,
    // receives types InitContext and MyLink
    (initContext, myLink) -> { ... });

// After
myGroup.fullyConnected(otherGroup, MyLink.class,
    // receives types MyAgent and MyLink
    (myAgent, myLink) -> { ... });

Behavioural Changes

This release includes some improvements which can affect the behaviour of models.

• For APIs where a class is specified for selection, all subclasses of that type will also be selected. See specific changes in core-abm for further information.
• When loading external data, user data injectors are now run after the external data has been assigned to the instance.

Console

• The network tile is now always shown as the first tile when displayed.
• The network tile can now appropriately display boolean attributes.
• The fan chart shown for multi run results should no longer report NaN as the current value.
• The headings for agent tables should now always align with the corresponding columns.
• Values of type int and long will no longer show decimal places in the agent table.
• Inputs of type String are now supported in the console.
• Exception modals can now be dismissed.

nexus-server

• The default nexus timeout, controlled by nexus-server.nexus-lifetime, has been increased from 30 to 60 minutes.
• Progress for batched model runs is now calculated from the total progress on ticks, rather than the total progress on runs, which should give a smoother progress update when running a high number of ticks.
• Batch runs now use the nexus-server.batchrun-lifetime configuration property to determine how long the server stores the results, and this is now timed from when the results are generated rather than when the batch processing is started.
• Exceptions thrown during a batch run are now caught and reported. This will also include the root PRNG seed for the run that failed.

core-abm

• [Behaviour Change] AgentSelection, as returned by AgentBasedModel#select(Class<AgentType>), now includes subclasses of the given agent type in the selection.
• [Behaviour Change] Messages and Links retrieved from an agent, e.g. from Agent#getMessagesOfType(Class<MessageType>), now include subclasses of the given type.
• [Behaviour Change] The data injector run by loadGroup(Class<AgentType>, Source, SerializableConsumer<T>) now runs after data from the source has been injected, rather than before.
• [Breaking Change] Connector data injectors now are given the Agent instance itself, rather than an InitContext. This allows links to be created based on attributes of the agent, and also for agents to be updated based on link properties.

core-graph

Note that changes within this module should not directly affect any users of the API unless noted under core-abm, unless you are using this lower level implementation directly.

• [Behaviour Change] VertexSelection now includes subclasses of the given agent type in the selection.
• [Breaking Change] Vertex is now an abstract class, rather than an interface. Methods relating to accessing links, such as Environment#getLinks() have moved from the Environment interface to the Vertex class.
• [Breaking Change] Graph#addVertices now receives an InitContext rather than a ConnectionInitializer
• simudyne.core.graph.SerializationLevel and related configuration setting core-abm.serialization-level have been removed.

2.0.1

17th May 2018

This is a minor bug fix release, primarily resolving issues with the network tile display. Improvements have also been made to the ABM API, allowing more flexible definition of actions, as well as injection of attributes to agents loaded from external sources.

Console

• Improved attribute selection UI for network tile.
• Improved display for hovering agents in the network tile.
• Network tile should now reliably update when links in the network change.
• Agent table now shows rounded numbers for floating point values.
• Upgrading console version should no longer require a browser cache clear.

core-abm

• Actions now also run against all subclasses of their defined agent class.

• Added AgentSystem#loadGroup(Class<AgentType>, Source, Consumer<AgentType>) (as well as AgentBasedModel#loadGroup(...) shortcut method).

  • This allows customisation of agent attributes when loaded from a Source, in the same way as generating agents from scratch allows.

2.0.0

April 19th 2018

First major release of the new Simudyne SDK 2.0.