{"data":{"markdownRemark":{"html":"

This page explains how to setup and use the Simudyne SDK to run on top of Spark, for distributing your models.

\n

The first requirement is to install Spark , running standalone or on top of Hadoop YARN. The required version is Spark 2.2.

\n

We recommend using the version of Spark running on Cloudera products : https://www.cloudera.com/products/open-source/apache-hadoop/apache-spark.html

\n

Once Spark is installed you can check it is running correctly lauching the Spark-shell in a terminal :

\n
./bin/spark-shell
\n

\n \n \n \n \n \n \n \n \n \n \n \n \n \n

\n

You need to identify your Spark master URL which points towards the master node of your cluster. Above the master URL indicates Spark is running locally (master = local[*]). The master URL should generally be a spark://host:port type of URL, on a standalone cluster or yarn if you use Hadoop YARN.

\n

You can then start your project from one of the quickstart projects, preconfigured for Spark :

\n\n

Clone or download the repository and setup your credentials like a standard simudyne project.

\n

Uncomment the following line in your properties file to enable Simudyne SDK using Spark as the backend implementation of the SDK :

\n

simudyneSDK.properties

\n
### CORE-ABM-SPARK ###\ncore-abm.backend-implementation=simudyne.core.graph.spark.SparkGraphBackend
\n

You have then two possibilities to configure Spark properties :

\n\n

You must be aware that a property set in the simudyneSDK.properties file will override the one passed to the spark-submit.

\n

To run your model, you will need to build a fatJar file which will carry your model, the Simudyne SDKand all the necessary dependencies. You will then need to upload it to the master node of your cluster where you can submit your Spark jobs.

\n

Here is the command to build your fatJar file :

\n

Maven

\n
mvn -s settings.xml compile package
\n

SBT

\n
sbt assembly
\n

You can then upload this jar file to your master node via SSH and then submit your job with :

\n
spark2-submit --class Main --master <sparkMasterURL>  --deploy-mode client --num-executors 30 --executor-cores 5 --executor-memory 30G --conf \"spark.executor.extraJavaOptions=-XX:+UseG1GC\" --files simudyneSDK.properties name-of-the-fat-jar.jar
\n

You should set --num-executors, --executor-cores, --executor-memory parameters according to your own cluster resources. Useful resource : http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/

","headings":[],"frontmatter":{"title":"Spark setup","toc":null,"experimental":null}},"site":{"siteMetadata":{"title":"Simudyne Docs","latestVersion":"2.6"}}},"pageContext":{"absolutePath":"/home/vsts/work/1/s/content/2.0/reference/distributed_computation/spark_setup.md","versioned":true,"version":"2.0","kind":"reference","pagePath":"/reference/distributed_computation/spark_setup","chronology":{"prev":{"name":"Distributed Computation","path":"/reference/distributed_computation"},"next":{"name":"Alternative Setups","path":"/reference/alternative_setups"}},"lastUpdated":"2026-04-21T13:56:54.827Z"}}