{"data":{"markdownRemark":{"html":"

Deploying from Development to Spark

\n

This page explains how to setup and use the Simudyne SDK to run on top of Spark, for distributing your models.

\n
\nSpecific Versions\n\n

Please note that you should not use version 2.4.0-2.5.0 of the SDK if you wish to use Spark. This is because version 2.4 uses Scala 12 which is only supported by version 3 of Spark, and the libraries to complement this Spark 3 support are included in version 2.5.2+

\n
\n

To run your model, you will need to build a fatJar file which will carry your model, the Simudyne SDK and all the necessary dependencies. You will then need to upload it to the master node of your cluster where you can submit your Spark jobs.

\n

Some more dependencies and shading are needed for your project and some will be provided by Spark.

\n

pom.xml

\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<project xmlns=\"http://maven.apache.org/POM/4.0.0\"\n         xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n         xsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\">\n    <modelVersion>4.0.0</modelVersion>\n\n    <groupId>simudyne</groupId>\n    <artifactId>simudyne-maven-java</artifactId>\n    <version>1.0-SNAPSHOT</version>\n\n    <properties>\n        <maven.compiler.source>1.8</maven.compiler.source>\n        <maven.compiler.target>1.8</maven.compiler.target>\n        <simudyne.version>2.5.2</simudyne.version>\n    </properties>\n\n    <repositories>\n        <repository>\n            <id>simudyne.jfrog.io</id>\n            <name>simudyne.jfrog.io</name>\n            <url>https://simudyne.jfrog.io/simudyne/releases</url>\n        </repository>\n    </repositories>\n\n    <dependencies>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-nexus-server_2.12</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core_2.12</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core-abm_2.12</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n\t\t<dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core-graph_2.12</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core-graph-spark_2.12</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core-runner-spark_2.12</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core-abm-testkit_2.12</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>org.slf4j</groupId>\n            <artifactId>slf4j-log4j12</artifactId>\n            <version>1.7.25</version>\n        </dependency>\n        <dependency>\n            <groupId>org.apache.hadoop</groupId>\n            <artifactId>hadoop-common</artifactId>\n            <version>2.7.5</version>\n        </dependency>\n        <dependency>\n            <groupId>org.apache.avro</groupId>\n            <artifactId>avro</artifactId>\n            <version>1.8.2</version>\n        </dependency>\n    </dependencies>\n\n    <build>\n        <plugins>\n            <plugin>\n                <groupId>org.codehaus.mojo</groupId>\n                <artifactId>exec-maven-plugin</artifactId>\n                <version>1.2.1</version>\n                <configuration>\n                    <mainClass>Main</mainClass>\n                </configuration>\n            </plugin>\n            <plugin>\n                <groupId>org.apache.maven.plugins</groupId>\n                <artifactId>maven-shade-plugin</artifactId>\n                <version>3.2.1</version>\n                <executions>\n                    <execution>\n                        <phase>package</phase>\n                        <goals>\n                            <goal>shade</goal>\n                        </goals>\n                        <configuration>\n                            <outputDirectory>target</outputDirectory>\n                            <shadedArtifactAttached>true</shadedArtifactAttached>\n                            <shadedClassifierName>allinone</shadedClassifierName>\n                            <filters>\n                                <filter>\n                                    <artifact>*:*</artifact>\n                                    <excludes>\n                                        <exclude>META-INF/*.SF</exclude>\n                                        <exclude>META-INF/*.DSA</exclude>\n                                        <exclude>META-INF/*.RSA</exclude>\n                                    </excludes>\n                                </filter>\n                            </filters>\n                            <transformers>\n                                <transformer implementation=\"org.apache.maven.plugins.shade.resource.AppendingTransformer\">\n                                    <resource>reference.conf</resource>\n                                </transformer>\n                                <transformer implementation=\"org.apache.maven.plugins.shade.resource.ManifestResourceTransformer\">\n                                    <main-Class>Main</main-Class>\n                                </transformer>\n                            </transformers>\n                        </configuration>\n                    </execution>\n                </executions>\n            </plugin>\n        </plugins>\n    </build>\n</project>
\n
\nSpecific dependencies will be needed depending on your usage of Spark. Please read the related sections:\n\n
\n

Here is the command to build your fatJar file :

\n

Maven

\n
mvn -s settings.xml compile package
\n

You can then upload this jar file to your master node as well as the simudyneSDK.properties file and you license file.\nBy default the license file will be looked for in the directory you called spark3-submit from, you can change this behaviour using the configuration core.license-file.

\n

Spark Model Runner

\n

The spark model runner allows you to run the same simulation multiple time on a spark cluster to create a distributed multirun.\nRunning a distributed multirun simulation depends on the package core-runner-spark which needs to be imported in your project:

\n

pom.xml

\n
<dependency>\n    <groupId>simudyne</groupId>\n    <artifactId>simudyne-core-runner-spark_2.12</artifactId>\n    <version>${simudyne.version}</version>\n</dependency>
\n

To enable Simudyne SDK to use the Spark runner, you need to uncomment the following line in your properties file:

\n

simudyneSDK.properties

\n
### CORE-RUNNER ###\n# core-runner.runner-backend = simudyne.core.exec.runner.spark.SparkRunnerBackend
\n

Then you need to configure the properties related to core-runner-spark.\nYou have two possibilities to configure them:

\n\n

Some properties are already listed with default values in simudyneSDK.properties:

\n
### CORE-RUNNER-SPARK ###\ncore-runner-spark.master-url = local[*]\ncore-runner-spark.log-level = WARN\n# core-runner-spark.executor.memory = 2g\n# core-runner-spark.partitions = 24\n#core-runner-spark.task.cpus = 1
\n

You must be aware that a property set in the simudyneSDK.properties file will override the one passed to the spark3-submit, for this reason we recommend only using properties based on configuration for testing, and to use spark-submit based configuration when possible.

\n

You can then submit your job using spark3-submit. Here is a example with some configurations options:

\n
spark3-submit --class Main --master <sparkMasterURL>  --deploy-mode client --files simudyneSDK.properties,mylicense.license path-to-fat-jar.jar
\n
\n

Including the Simudyne SDK properties and license

\nThe Simudyne SDK cannot run without the config field and license on every node, in order to do this, we pass the files to every node using the parameter `--files` (as in the command above). Edit the command to pass the full path to the actual properties files and license file so these can be found and copied onto the clusters nodes.\n
\n

This command will run the main function of the class Main and distribute it on Spark. You can then access the console through the config parameters nexus-server.hostname and nexus-server.port.

\n

They default to localhost and 8080. You can also interact with the server through the REST API

\n

spark3-submit allows you to configure Spark. You need to choose a configuration that best suits your cluster.\nTo learn more about Spark configuration, refer to the official documentation.

\n

Some useful resources can be found on Cloudera's website.

\n
\nIf you wish to use the runner directly, and not use the console nor the REST API, you can use the SparkModelRunner\n
\n

Spark Distributed Graph

\n

The distributed graph backend allows you to run large graphs on a cluster of machines.\nRunning a distributed graph simulation depends on the core-graph-distributed package which needs to be imported in your project:

\n

pom.xml

\n
<dependency>\n    <groupId>simudyne</groupId>\n    <artifactId>simudyne-core-graph-distributed_2.12</artifactId>\n    <version>${simudyne.version}</version>\n</dependency>
\n

To let Spark manage nodes of the Distributed Graph implementation, add the following:

\n

simudyneSDK.properties

\n
core-abm.backend-implementation=simudyne.core.graph.spark.SparkGraphBackend\nfeature.immutable-schema=false
\n

Please be aware that properties set in simudyneSDK.properties file takes precedence over options passed to spark3-submit.

\n

You can then submit your job using spark3-submit. Here is an example:

\n
spark3-submit --class Main --master yarn  --deploy-mode client --files simudyneSDK.properties,licenseKey hdfs://path/name-of-the-fat-jar.jar
\n

This command will run the main function of the class Main and distribute it on Spark. You can then access the console through the config parameters nexus-server.hostname and nexus-server.port.\nThey default to localhost and 8080. You can also interact with the server through the REST API

\n

spark3-submit allows you to configure Spark. You need to choose a configuration that best suits your cluster.\nTo learn more about Spark configuration, refer to the official documentation.

\n

Some useful resources can be found on Cloudera's website.

\n

Limitations

\n
\n

The Distributed graph comes with limitations, most notable being:
\n

  • no support for Immutable Schema, please set feature.immutable-schema=false
    • \n
  • no support for dynamic agent creation/stopping
    • \n
  • no support for SystemMessages
    • \n

    You also might want to disable the health check for models by setting nexus-server.health-check to false in order to avoid their distribution.

    \n
    \n

    For more see tuning and debug

    ","headings":[{"value":"Deploying from Development to Spark","depth":2},{"value":"Spark Model Runner","depth":2},{"value":"Spark Distributed Graph","depth":2},{"value":"Limitations","depth":2}],"frontmatter":{"title":"Spark Setup","toc":null,"experimental":true}},"site":{"siteMetadata":{"title":"Simudyne Docs","latestVersion":"2.6"}}},"pageContext":{"absolutePath":"/home/vsts/work/1/s/content/2.6/reference/distributed_computation/spark_setup.md","versioned":false,"version":"2.6","kind":"reference","pagePath":"/reference/distributed_computation/spark_setup","chronology":{"prev":{"name":"Distributed Requirements","path":"/reference/distributed_computation/distributed_requirements"},"next":{"name":"Spark Tuning & Model Debugging","path":"/reference/distributed_computation/tuning_debugging"}},"lastUpdated":"2026-04-21T13:56:54.869Z"}}