{"data":{"markdownRemark":{"html":"

This page explains how to setup and use the Simudyne SDK to run on top of Spark, for distributing your models.

\n

Spark requirements

\n

The first requirement is to install Spark , running standalone or on top of Hadoop YARN. The required version is Spark 2.2.

\n

We recommend using the version of Spark running on Cloudera products : https://www.cloudera.com/products/open-source/apache-hadoop/apache-spark.html

\n

Once Spark is installed you can check it is running correctly launching the Spark-shell in a terminal :

\n
./bin/spark-shell
\n

\n \n \n \n \n \n \n \n \n \n \n \n \n \n

\n

You need to identify your Spark master URL which points towards the master node of your cluster. Above, the master URL indicates Spark is running locally (master = local[*]).\nThe master URL should generally be a spark://host:port type of URL on a standalone cluster.

\n

To run your model, you will need to build a fatJar file which will carry your model, the Simudyne SDK and all the necessary dependencies. You will then need to upload it to the master node of your cluster where you can submit your Spark jobs.

\n

Some more dependencies and shading are needed for your project and some will be provided by Spark.

\n

pom.xml

\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<project xmlns=\"http://maven.apache.org/POM/4.0.0\"\n         xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n         xsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\">\n    <modelVersion>4.0.0</modelVersion>\n\n    <groupId>simudyne</groupId>\n    <artifactId>simudyne-maven-java</artifactId>\n    <version>1.0-SNAPSHOT</version>\n\n    <properties>\n        <maven.compiler.source>1.8</maven.compiler.source>\n        <maven.compiler.target>1.8</maven.compiler.target>\n        <simudyne.version>2.2.0</simudyne.version>\n    </properties>\n\n    <repositories>\n        <repository>\n            <id>simudyne.jfrog.io</id>\n            <name>simudyne.jfrog.io</name>\n            <url>https://simudyne.jfrog.io/simudyne/releases</url>\n        </repository>\n    </repositories>\n\n    <dependencies>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-nexus-server_2.11</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core_2.11</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n        <dependency>\n            <groupId>simudyne</groupId>\n            <artifactId>simudyne-core-abm_2.11</artifactId>\n            <version>${simudyne.version}</version>\n        </dependency>\n    </dependencies>\n\n    <build>\n        <plugins>\n            <plugin>\n                <groupId>org.codehaus.mojo</groupId>\n                <artifactId>exec-maven-plugin</artifactId>\n                <version>1.2.1</version>\n                <configuration>\n                    <mainClass>Main</mainClass>\n                </configuration>\n            </plugin>\n            <plugin>\n                <groupId>org.apache.maven.plugins</groupId>\n                <artifactId>maven-shade-plugin</artifactId>\n                <version>3.1.0</version>\n                <configuration>\n                    <artifactSet>\n                        <excludes>\n                            <exclude>META-INF/*</exclude>\n                            <exclude>com.fasterxml.jackson.core:jackson-databind</exclude>\n                            <exclude>com.fasterxml.jackson.module:jackson-module-scala_2.11</exclude>\n                            <exclude>com.twitter:chill_2.11</exclude>\n                            <exclude>io.netty:netty-all</exclude>\n                            <exclude>org.apache.commons:commons-math3</exclude>\n                            <exclude>org.apache.commons:commons-lang3</exclude>\n                            <exclude>commons-net</exclude>\n                            <exclude>org.slf4j:slf4j-api</exclude>\n                            <exclude>org.xerial.snappy:snappy-java</exclude>\n                            <exclude>org.scala-lang:scala-library</exclude>\n                        </excludes>\n                    </artifactSet>\n                    <transformers>\n                        <transformer implementation=\"org.apache.maven.plugins.shade.resource.AppendingTransformer\">\n                            <resource>reference.conf</resource>\n                        </transformer>\n                    </transformers>\n                    <relocations>\n                        <relocation>\n                            <pattern>org.json4s</pattern>\n                            <shadedPattern>shaded.json4s</shadedPattern>\n                        </relocation>\n                    </relocations>\n                </configuration>\n                <executions>\n                    <execution>\n                        <phase>package</phase>\n                        <goals>\n                            <goal>shade</goal>\n                        </goals>\n                    </execution>\n                </executions>\n            </plugin>\n        </plugins>\n    </build>\n</project>
\n
\nSpecific dependencies will be needed depending on your usage of Spark. Please read the related sections:\n\n
\n

Here is the command to build your fatJar file :

\n

Maven

\n
mvn -s settings.xml compile package
\n

You can then upload this jar file to your master node as well as the simudyneSDK.properties file and you license file.\nBy default the license file will be looked for in the directory you called spark-submit from, you can change this behaviour using the configuration core.license-file.

\n

Spark model runner

\n

The spark model runner allows you to run the same simulation multiple time on a spark cluster to create a distributed multirun.\nRunning a distributed multirun simulation depends on the package core-runner-spark which needs to be imported in your project:

\n

pom.xml

\n
<dependency>\n    <groupId>simudyne</groupId>\n    <artifactId>simudyne-core-runner-spark_2.11</artifactId>\n    <version>${simudyne.version}</version>\n</dependency>
\n

To enable Simudyne SDKto use the Spark runner, you need to uncomment the following line in your properties file:

\n

simudyneSDK.properties

\n
### CORE-RUNNER ###\n# core-runner.runner-backend = simudyne.core.exec.runner.spark.SparkRunnerBackend
\n

Then you need to configure the properties related to core-runner-spark.\nYou have two possibilities to configure them:

\n\n

Some properties are already listed with default values in simudyneSDK.properties:

\n
### CORE-RUNNER-SPARK ###\ncore-runner-spark.master-url = local[*]\ncore-runner-spark.log-level = WARN\n# core-runner-spark.executor.memory = 2g\n# core-runner-spark.partitions = 24\n#core-runner-spark.task.cpus = 1
\n

You must be aware that a property set in the simudyneSDK.properties file will override the one passed to the spark-submit, for this reason we recommend only using properties based on configuration for testing, and to use spark-submit based configuration when possible.

\n

You can then submit your job using spark-submit. Here is a example with some configurations options:

\n
spark-submit --class Main --master <sparkMasterURL>  --deploy-mode client --files simudyneSDK.properties,mylicense.license path-to-fat-jar.jar
\n
\n

Including the Simudyne SDK properties and license

\nThe Simudyne SDK cannot run without the config field and license on every node, in order to do this, we pass the files to every node using the parameter `--files` (as in the command above). Edit the command to pass the full path to the actual properties files and license file so these can be found and copied onto the clusters nodes.\n
\n

This command will run the main function of the class Main and distribute it on Spark. You can then access the console through the config parameters nexus-server.hostname and nexus-server.port.

\n

They default to localhost and 8080. You can also interact with the server through the REST API

\n

spark-submit allows you to configure Spark. You need to choose a configuration that best suits your cluster.\nTo learn more about Spark configuration, refer to the official documentation.

\n

Some useful resources can be found on Cloudera's website.

\n
\nIf you wish to use the runner directly, and not use the console nor the REST API, you can use the:\n

SparkModelRunner

\n
","headings":[{"value":"Spark requirements","depth":2},{"value":"Spark model runner","depth":2}],"frontmatter":{"title":"Spark setup","toc":null,"experimental":null}},"site":{"siteMetadata":{"title":"Simudyne Docs","latestVersion":"2.6"}}},"pageContext":{"absolutePath":"/home/vsts/work/1/s/content/2.3/reference/distributed_computation/spark_setup.md","versioned":true,"version":"2.3","kind":"reference","pagePath":"/reference/distributed_computation/spark_setup","chronology":{"prev":{"name":"Distributed Computation","path":"/reference/distributed_computation"},"next":{"name":"Spark Graph","path":"/reference/distributed_computation/spark_graph"}},"lastUpdated":"2026-04-21T13:56:54.843Z"}}