{"data":{"markdownRemark":{"html":"

Data about the current state of the simulation can be retrieved as a JSON via the REST API. The Simudyne SDK can also export all simulation data to Parquet files for further analysis.

\n

Exporting the data to Parquet

\n

The Simudyne SDK will not export files to Parquet by default. To enable, set the value of the config field core.parquet-export in the SimudyneSDK.properties file to true. (More about Model Config.)

\n

The path to create the Parquet files in should be provided in the config field core.parquet-export-path. This can be a HDFS path, or a local file system path. If no value is specified for core.parquet-export-path, the Parquet files will be dumped to a tmp directory, or the hdfs home if running with spark.

\n
Note for Windows users
\nYou will need a file named `winutils.exe` to be able to use Parquet on Windows.\n

You can find it in the hadoop-winutils directory here,\nor you can copy-paste the following URL into your browser : http://content.simudyne.com/$web/hadoop-winutils-master.zip.

\n

Once you have downloaded the hadoop-winutils , run the Winutils_setup.bat batch file to set your environment variable accordingly.

\n

If you already have an installed version of Hadoop and just lack the winutils.exe, you can add it to your C:\\hadoop-x.x.x\\bin directory manually.

\n

When using Parquet on Windows, the system will try to access ...\\hadoop-winutils\\bin (or ...\\hadoop-x.x.x\\bin if you already had hadoop installed) to find the file winutils.exe.\nIf you are getting error messages like Shell Failed to locate the winutils binary in the hadoop binary path,\ncheck that your HADOOP_HOME environment variable is set and that your winutils.exe is located in the bin directory inside the directory of the HADOOP_HOME destination.\nFor instance, if the location of your hadoop-winutils directory is C:\\hadoop-winutils, then HADOOP_HOME must be C:\\hadoop-winutils.

\n

Files created

\n

Multiple Parquet files could be created for each simulation run. The root Parquet file, named root.parquet will contain all output fields related to the model. This includes the values of global fields and accumulators. All complex objects that are nested in the JSON output of the simulation are flattened. For example, if a model's JSON contained nested fields as follows

\n

Model JSON output

\n
{\n  \"someValue\": 23,\n  \"system\":{\n    \"aglobal\" : 24,\n    \"anAccumulator\": {\n      \"count\": 25,\n      \"value\": 26\n    }\n  }\n}
\n

The Parquet root file created for this would contain the following fields.

\n

| someValue | system__aglobal | system__anAccumulator__count | system__anAccumulator__value |

\n

The field name is made up of the path to the JSON field where every element is seperated by a double underscore __.

\n

If a model's JSON output contains arrays of objects, such as Agents or Links, these are exported to seperate Parquet files. (One file per Agent or Link type.) The name of the Parquet file will be the path to the agent.

\n

Model JSON output with agents and links

\n
{\n  \"someValue\": 23,\n  \"system\":{\n    \"Agents\" : {\n      \"Cell\": [\n        {\n          \"alive\": false,\n          \"_id\": 0\n        },\n        {\n          \"alive\": true,\n          \"_id\": 1\n        }\n      ]\n    },\n    \"Links\": {\n      \"Neighbour\" : [\n        {\n          \"_to\" :123,\n          \"_from\": 256\n        },\n        {\n          \"_to\": 256,\n          \"_from\": 123\n        }\n      ]\n    }\n  }\n}
\n

Parquet files:

\n

root

\n

| someValue |

\n

root__system__Agents__Cell

\n

| alive | _id |

\n

root__system__Links__Neighbour

\n

| _to | _from |

\n

Fields added

\n

Every Parquet table will also include a field tick which tells you which tick this data was produced for and a field seed that tells you the random number generator seed being used to for this run of the simulation.

\n

Output directory structure

\n

Folders are created in the root export directory passed through the config field core.parquet-export-path, to seperate and identify the data for different runs.

\n

Scenario run folders

\n
exportFolder/\n    {simulation_id}/\n        runs/\n            scenario0.run0/\n              root.parquet\n              root\\_\\_system.parquet\n            metadata.json
\n\n

Batch/Interactive run folders

\n
exportFolder/\n    {simulation_id}/\n        runs/\n            run0/\n              root.parquet\n              root\\_\\_system.parquet\n            metadata.json\n            DONE
\n\n

Metadata.json

\n

A metadata file is added to the data export giving details about the data. The metadata contains

\n","headings":[{"value":"Exporting the data to Parquet","depth":2},{"value":"Files created","depth":2},{"value":"Fields added","depth":2},{"value":"Output directory structure","depth":2},{"value":"Metadata.json","depth":2}],"frontmatter":{"title":"Parquet data export","toc":null,"experimental":null}},"site":{"siteMetadata":{"title":"Simudyne Docs","latestVersion":"2.6"}}},"pageContext":{"absolutePath":"/home/vsts/work/1/s/content/2.1/reference/data_output/parquet_output.md","versioned":true,"version":"2.1","kind":"reference","pagePath":"/reference/data_output/parquet_output","chronology":{"prev":{"name":"Data Output","path":"/reference/data_output"},"next":{"name":"Avro schema","path":"/reference/data_output/avro_schema"}},"lastUpdated":"2026-04-21T13:56:54.831Z"}}