Skip to main content

Distributed Requirements

Running on Hadoop​

The Simudyne SDK only requires that you have a Hadoop based environment with HDFS, Yarn, and Spark 4.2.0+ installed onto your cluster. Optionally if you wish to use Parquet with Hive that will also need to be installed see here on how to configure.

Specific Versions
  • Spark 2 - use version 2.3.x of Simudyne SDK
  • Spark 3 - use version 2.5.2 – 2.6.x of Simudyne SDK
  • Spark 4.2.0 - use version 2.7.0+ of Simudyne SDK (Scala 2.13, Hadoop 3.5.0)

Please note that you should not use version 2.4.0-2.5.0 of the SDK if you wish to use Spark. This is because version 2.4 uses Scala 2.12 which is only supported by Spark 3, and the libraries to complement this Spark 3 support are included in version 2.5.2+. Version 2.7 upgrades to Spark 4.2.0 and Scala 2.13

Because of the usage of the driver/worker nodes JDK 25+ is required on all nodes, however most recent versions of Hadoop installs should satisfy this.

Our recommended setup is to make usage of Cloudera CDP for connecting to your existing or Azure/AWS environments. A Data Engineering template with inclusion of Spark is recommended for usage with Simudyne's SDK.

However, as long as you have a valid Hadoop cluster with Spark, Spark on Yarn, and HDFS you should be able to work with the SDK. The core requirement is being able to submit a spark job that includes a singularly packaged jar file of your simulation along with any configuration or data lakes included.

Setting up Spark​

The first requirement is to install Spark , running standalone or on top of Hadoop YARN. **The required version is Spark 4.2.0+ **

We recommend using the version of Spark running on Cloudera products : https://www.cloudera.com/products/open-source/apache-hadoop/apache-spark.html

Once Spark is installed you can check it is running correctly launching the Spark-shell in a terminal :

./bin/spark-shell

You need to identify your Spark master URL which points towards the master node of your cluster. Above, the master URL indicates Spark is running locally (master = local[*]). The master URL should generally be a spark://host:port type of URL on a standalone cluster.

Last updated on