{"data":{"markdownRemark":{"html":"

Data about the current state of the simulation can be retrieved as a JSON via the REST API. The Simudyne SDK can also export all simulation data to Parquet files for further analysis.

\n
Note for Windows users
\nYou will need a file named `winutils.exe` to be able to use Parquet on Windows.\n

You can find it in the hadoop-winutils directory here,\nor you can copy-paste the following URL into your browser : http://content.simudyne.com/$web/hadoop-winutils-master.zip.

\n

Once you have downloaded the hadoop-winutils , run the Winutils_setup.bat batch file to set your environment variable accordingly.

\n

If you already have an installed version of Hadoop and just lack the winutils.exe, you can add it to your C:\\hadoop-x.x.x\\bin directory manually.

\n

When using Parquet on Windows, the system will try to access ...\\hadoop-winutils\\bin (or ...\\hadoop-x.x.x\\bin if you already had hadoop installed) to find the file winutils.exe.\nIf you are getting error messages like Shell Failed to locate the winutils binary in the hadoop binary path,\ncheck that your HADOOP_HOME environment variable is set and that your winutils.exe is located in the bin directory inside the directory of the HADOOP_HOME destination.\nFor instance, if the location of your hadoop-winutils directory is C:\\hadoop-winutils, then HADOOP_HOME must be C:\\hadoop-winutils.

\n

Batch run parquet export

\n

The Simudyne SDK will not export files to Parquet by default. To enable, set the value of the config field core.parquet-export in the SimudyneSDK.properties file to true. (More about Model Config.)

\n

The path to create the Parquet files in should be provided in the config field core.parquet-export-path. This can be a HDFS path, or a local file system path. If no value is specified for core.parquet-export-path, the Parquet files will be dumped to a tmp directory, or the hdfs home if running with spark.

\n

By default, when running a batch run, Agent and Link data is not serialised, and so not output to parquet. This is to reduce the amount of data being held in memory when sending the batch results to the console. If the data is being output to parquet and does not need to be viewed on the console, the in memory data storage can be turned off allowing the Simudyne SDK to export Agent and Link data to parquet as well as the general Model data. This is done by setting the config field core.return-data to false.

\n

For large model runs that produce a lot of data, setting this config field to false will also reduce the amount of memory being held by the simulation, which can help avoid potential OutOfMemory exceptions and improve the efficiency of the model.

\n

If the data does not need to be displayed on the console, but Agent and Link data is not needed, the config fields core-abm.serialize.agents and core-abm.serialize.links should be set to false, to avoid generating uncessary data.

\n

Scenario run parquet export

\n

Scenario runs do not hold the data in memory because they are not managed by the console, and the data cannot be viewed on the console. This means that Agent and Link data is serialised by default, and so should be explicitly turned off if not needed. (Use the config fields core-abm.serialize.agents and core-abm.serialize.links to control this.)

\n

Data export format for scenario runs is controlled via the POST request sent to start the scenario run. (See the scenario REST specification for more details on the POST request here.)

\n

By default the scenario will output data as json files. To specify the output format as parquet, set the 'format' field in the 'output' section of the POST request.

\n
{\n  //Other scenario json fields\n  \"output\": {\"uri\": \"/path/to/export/to\" , \"format\": \"parquet\"}\n}
\n

Model sampler parquet export

\n

The model sampler will always output data to parquet. As with scenarios, the data is not held in memory, so Agent and Link data is serialised by default and should be explicity turned off if not needed using the config fields core-abm.serialize.agents and core-abm.serialize.links.

\n

Interactive run parquet export

\n

In most cases, it will be unecessary to output parquet data when running interactive runs. Therefore, by default parquet data will not be exported when running interactive runs, even if the config field core.parquet-export is true. If parquet output is required for interactive runs, the config field feature.interactive-parquet-output should be set to true, in addition to the config fields core.parquet-export and core.parquet-export-path.

\n

When running an interactive run, the parquet files will be closed (and ready for reading) when the interactive run is deleted or restarted.

\n

Files created

\n

Multiple Parquet files could be created for each simulation run. The root Parquet file, will contain all output fields related to the model. This includes the values of global fields and accumulators. All complex objects that are nested in the JSON output of the simulation are flattened. For example, if a model's JSON contained nested fields as follows

\n

Model JSON output

\n
{\n  \"someValue\": 23,\n  \"system\":{\n    \"aglobal\" : 24,\n    \"anAccumulator\": {\n      \"count\": 25,\n      \"value\": 26\n    }\n  }\n}
\n

The Parquet root file created for this would contain the following fields.

\n

| someValue | system__aglobal | system__anAccumulator__count | system__anAccumulator__value |

\n

The field name is made up of the path to the JSON field where every element is seperated by a double underscore __.

\n

If a model's JSON output contains arrays of objects, such as Agents or Links, these are exported to seperate Parquet files. (One file per Agent or Link type.) The name of the Parquet file will be the path to the agent.

\n

Model JSON output with agents and links

\n
{\n  \"someValue\": 23,\n  \"system\":{\n    \"Agents\" : {\n      \"Cell\": [\n        {\n          \"alive\": false,\n          \"_id\": 0\n        },\n        {\n          \"alive\": true,\n          \"_id\": 1\n        }\n      ]\n    },\n    \"Links\": {\n      \"Neighbour\" : [\n        {\n          \"_to\" :123,\n          \"_from\": 256\n        },\n        {\n          \"_to\": 256,\n          \"_from\": 123\n        }\n      ]\n    }\n  }\n}
\n

Parquet files:

\n

root

\n

| someValue |

\n

root__system__Agents__Cell

\n

| alive | _id |

\n

root__system__Links__Neighbour

\n

| _to | _from |

\n

Fields added

\n

Every Parquet table will also include a field tick which tells you which tick this data was produced for and a field seed that tells you the random number generator seed being used to for this run of the simulation.

\n

Output directory structure

\n

When exporting data to parquet, the folder layout can be specified in the config using the config field core.parquet-export.folder-structure. There are two options supported for this field, group-by-type and group-by-run. If no value is specified, it will default to group-by-type.

\n

Group by type structure

\n

When the parquet folder structure is group by type, folders are created for each parquet table type, and a parquet file for each run is created inside these folders.

\n

For this example, the root export directory passed through the config field core.parquet-export-path is /exportFolder.

\n

Group by type batch output folders

\n
/exportFolder/\n    {simulation_id}/\n        runs/\n            root/\n              run000.parquet\n              run001.parquet\n              run002.parquet\n            root__system__Agents__Cell\n              run000.parquet\n              run001.parquet\n              run002.parquet\n            metadata.json\n            finished.json
\n\n

Group by type scenario output folders

\n
/exportFolder/\n    {simulation_id}/\n        runs/\n            root/\n              scenario0run0001.parquet\n              scenario0run0002.parquet\n              scenario0run0003.parquet\n            root__system__Agents__Cell\n              scenario0run0001.parquet\n              scenario0run0002.parquet\n              scenario0run0003.parquet\n            metadata.json\n            finished.json
\n\n

The model sampler output folders will match the scenario output folders.

\n

Group by run structure

\n

When the parquet folder structure is group by runs, folders are created for each simulation run, and a parquet file for each table type is created inside these folders.

\n

For this example, the root export directory passed through the config field core.parquet-export-path is /exportFolder.

\n

Group by run batch output folders

\n
/exportFolder/\n    {simulation_id}/\n        runs/\n            run000/\n              root001.parquet\n              root__system__Agents__Cell001.parquet\n            run001/\n              root001.parquet\n              root__system__Agents__Cell001.parquet\n            run002/\n              root001.parquet\n              root__system__Agents__Cell001.parquet\n            metadata.json\n            finished.json
\n\n

Group by run scenario output folders

\n
/exportFolder/\n    {simulation_id}/\n        runs/\n            scenario0.run0/\n              root001.parquet\n              root__system__Agents__Cell001.parquet\n            metadata.json\n            finished.json
\n\n

The model sampler output folders will match the scenario output folders.

\n

metadata.json

\n

A metadata file is added to the data export giving details about the data. The metadata contains

\n\n

finished.json

\n

This is an empty file created at the end of a run to let you know that no new parquet files will be created in this directory.

\n

Outputting to HDFS

\n

When exportting to a cluster there are a few things to make sure you have set up ahead of time.

\n\n

For more detailed information working with Spark/HDFS in your model(s) please refer to Spark Setup

","headings":[{"value":"Batch run parquet export","depth":2},{"value":"Scenario run parquet export","depth":2},{"value":"Model sampler parquet export","depth":2},{"value":"Interactive run parquet export","depth":2},{"value":"Files created","depth":2},{"value":"Fields added","depth":2},{"value":"Output directory structure","depth":2},{"value":"Group by type structure","depth":3},{"value":"Group by run structure","depth":3},{"value":"metadata.json","depth":2},{"value":"finished.json","depth":2},{"value":"Outputting to HDFS","depth":2}],"frontmatter":{"title":"Parquet data export","toc":null,"experimental":null}},"site":{"siteMetadata":{"title":"Simudyne Docs","latestVersion":"2.6"}}},"pageContext":{"absolutePath":"/home/vsts/work/1/s/content/2.2/reference/data_output/parquet_output.md","versioned":true,"version":"2.2","kind":"reference","pagePath":"/reference/data_output/parquet_output","chronology":{"prev":{"name":"Data Output","path":"/reference/data_output"},"next":{"name":"Avro schema","path":"/reference/data_output/avro_schema"}},"lastUpdated":"2026-04-21T13:56:54.836Z"}}