{"data":{"markdownRemark":{"html":"

This page explains how to setup and use the Simudyne SDK to distribute a single simulation on Spark.

\n

Spark graph backend

\n

The spark graph backend allows you to run a single large graph on a Spark cluster.\nRunning a distributed graph simulation depends on the package core-graph-spark which needs to be imported in your project:

\n

pom.xml

\n
<dependency>\n    <groupId>simudyne</groupId>\n    <artifactId>simudyne-core-graph-spark_2.11</artifactId>\n    <version>${simudyne.version}</version>\n</dependency>
\n

To enable Simudyne SDK using Spark as the backend implementation of the SDK, you need to uncomment the following line in your properties file:

\n

simudyneSDK.properties

\n
### CORE-ABM-SPARK ###\ncore-abm.backend-implementation=simudyne.core.graph.spark.SparkGraphBackend
\n

Then you need to configure the properties related to core-abm on Spark.\nYou have two possibilities to configure them:

\n\n

Some properties are already listed with default values in simudyneSDK.properties:

\n
### CORE-ABM-SPARK ###\ncore-abm-spark.master-url = local[*]\ncore-abm-spark.checkpoint-directory = /var/tmp\ncore-abm-spark.log-level = WARN\n# core-abm-spark.spark.executor.memory = 2g\n# core-abm-spark.spark.sql.shuffle.partitions = 24
\n

You must be aware that a property set in the simudyneSDK.properties file will override the one passed to the spark-submit.

\n

You can then submit your job using spark-submit. Here is a example with some configurations options:

\n
spark-submit --class Main --master <sparkMasterURL>  --deploy-mode client --files simudyneSDK.properties,licenseKey name-of-the-fat-jar.jar
\n

This command will run the main function of the class Main and distribute it on Spark. You can then access the console through the config parameters nexus-server.hostname and nexus-server.port.

\n

They default to localhost and 8080. You can also interact with the server through the REST API

\n

spark-submit allows you to configure Spark. You need to choose a configuration that best suits your cluster.\nTo learn more about Spark configuration, refer to the official documentation.

\n

Some useful resources can be found on Cloudera's website.

","headings":[{"value":"Spark graph backend","depth":2}],"frontmatter":{"title":"Spark Graph","toc":null,"experimental":true}},"site":{"siteMetadata":{"title":"Simudyne Docs","latestVersion":"2.6"}}},"pageContext":{"absolutePath":"/home/vsts/work/1/s/content/2.4/reference/distributed_computation/spark_graph.md","versioned":true,"version":"2.4","kind":"reference","pagePath":"/reference/distributed_computation/spark_graph","chronology":{"prev":{"name":"Spark setup","path":"/reference/distributed_computation/spark_setup"},"next":{"name":"Distributed Graph","path":"/reference/distributed_computation/distributed_graph"}},"lastUpdated":"2026-04-21T13:56:54.852Z"}}