The Simudyne SDK has built in output generators for all of it's varying run types, and in multiple different formats. The following pages will be helpful in understanding if there are certain restrictions, requirements, or configurations for certain output formats. However most information on using these outputs is the same regardless of format.
\nBy default, when running a batch run, Agent and Link data is not serialised, and so not output to parquet. This is to reduce the amount of data being held in memory when sending the batch results to the console. If the data is being output to parquet and does not need to be viewed on the console, the in memory data storage can be turned off allowing the Simudyne SDK to export Agent and Link data to parquet as well as the general Model data. This is done by setting the config field core.return-data to false.
For large model runs that produce a lot of data, setting this config field to false will also reduce the amount of memory being held by the simulation, which can help avoid potential OutOfMemory exceptions and improve the efficiency of the model.
\nIf the data does not need to be displayed on the console, but Agent and Link data is not needed, the config fields core-abm.serialize.agents and core-abm.serialize.links should be set to false, to avoid generating uncessary data.
Scenario runs do not hold the data in memory because they are not managed by the console, and the data cannot be viewed on the console. This means that Agent and Link data is serialised by default, and so should be explicitly turned off if not needed. (Use the config fields core-abm.serialize.agents and core-abm.serialize.links to control this.)
Data export format for scenario runs is controlled via the POST request sent to start the scenario run. (See the scenario REST specification for more details on the POST request here.)
\nBy default the scenario will output data as JSON files. To specify the output format as parquet, set the 'format' field in the 'output' section of the POST request.
\n{\n //Other scenario json fields\n \"output\": {\"uri\": \"/path/to/export/to\" , \"format\": \"parquet\"}\n}The model sampler will always output data to parquet. As with scenarios, the data is not held in memory, so Agent and Link data is serialised by default and should be explicity turned off if not needed using the config fields core-abm.serialize.agents and core-abm.serialize.links.
When exporting data to parquet, the folder layout can be specified in the config using the config field core.export.folder-structure. There are two options supported for this field, group-by-type and group-by-run. If no value is specified, it will default to group-by-type.
When the output folder structure is group by type, folders are created for each output table type, and a output file for each run is created inside these folders.
\nFor this example, the root export directory passed through the config field core.export-path is /exportFolder. The output is also shown as parquet files, but could be JSON/CSV/etc.
Group by type batch output folders
\n/exportFolder/\n {simulation_id}/\n runs/\n root/\n run000.parquet\n run001.parquet\n run002.parquet\n root__system__Agents__Cell\n run000.parquet\n run001.parquet\n run002.parquet\n metadata.json\n finished.jsonGroup by type scenario output folders
\n/exportFolder/\n {simulation_id}/\n runs/\n root/\n scenario0run0001.parquet\n scenario0run0002.parquet\n scenario0run0003.parquet\n root__system__Agents__Cell\n scenario0run0001.parquet\n scenario0run0002.parquet\n scenario0run0003.parquet\n metadata.json\n finished.jsonThe model sampler output folders will match the scenario output folders.
\nWhen the output folder structure is group by runs, folders are created for each simulation run, and a output file for each table type is created inside these folders.
\nFor this example, the root export directory passed through the config field core.export-path is /exportFolder.
Group by run batch output folders
\n/exportFolder/\n {simulation_id}/\n runs/\n run000/\n root001.parquet\n root__system__Agents__Cell001.parquet\n run001/\n root001.parquet\n root__system__Agents__Cell001.parquet\n run002/\n root001.parquet\n root__system__Agents__Cell001.parquet\n metadata.json\n finished.jsonGroup by run scenario output folders
\n/exportFolder/\n {simulation_id}/\n runs/\n scenario0.run0/\n root001.parquet\n root__system__Agents__Cell001.parquet\n metadata.json\n finished.jsonThe model sampler output folders will match the scenario output folders.
\nA metadata file is added to the data export giving details about the data. The metadata contains
\nThis is an empty file created at the end of a run to let you know that no new output files will be created in this directory.
\nAs well if you are working with a SQL/Hive location you will NEED to specify this output folder for these files alongside the corresponding URL for connection to the external tables.
\n