Experimental features are early versions for users to test before final release. We work hard to ensure that every available Simudyne SDK feature is thoroughly tested, but these experimental features may have minor bugs we still need to work on.
If you have any comments or find any bugs, please share with support@simudyne.com.
Introduction
This tutorial covers the basic steps to running the Simudyne SDK via a Python Subprocess and from a Jupyter Notebook. Models built in the Simudyne SDK are packaged as executable Java JAR files, meaning they can be run from the command line or from a subprocess in another application.
Variations in Local Environment
Note that Jupyter Notebook and the Python Subprocess libraries are not supported by Simudyne. This tutorial is just one example of how to run your models with an external tool. If the example does not function in your local environment, using the basic guidance provided on running any executable Java JAR file should be the foundation to arriving at your solution.
Sections covered
This tutorial is structured in 3 sections:
Creating & Running an executable JAR
Python subprocess
Jupyter Notebook
1. Creating & Running an executable JAR
Creating a JAR
The first step is to package, you will need to build a fat jar which will carry your model, the Simudyne SDK and all the necessary dependencies.
More information on how to package your JAR file please reference the CLI Model Runner
To package your Simudyne SDK model, run the below command from the project root.
Maven
mvn -s settings.xml clean compile package
Maven will create the jar and place it in the target directory (unless you have specified otherwise in your pom settings). Unless you have given it a specific name, the generated jar will be named <some-artifact-id>-<some-version>-jar.
Running a JAR
Running the JAR can be done from the command line by executing the below command (pointing to the location of the packaged JAR):
Running a JAR
java -jar target/sample-artifact-1.1.0.jar
The example above is a general example, for more info please reference the Running the model section of the CLI Model Runner section.
2. Python subprocess
Once the Simudyne SDK model has been packaged as an executable JAR file, it is possible to call this file via a Python Subprocess. It is recommended to read through the Python Subprocess Documentation before beginning this section.
Before starting with the Python subprocess, a new MainNoConsole.java should be added to your project as it is recommended to have the SDK run headless for ease of returning results.
The following Java code can be added to your project and customized to your model and inputs:
Now that the SDK model has been configured for headless mode, the code for the Python subprocess can be defined.
Below is a provided utils.py Python module that can be imported, and whose functions can be called to run the Simudyne SDK and return the results. As mentioned above, there have been attempts to account for changes in your local environment, but it's possible this code will not function 'out of the box' on your machine.
The first step is to create a notebook_settings.json file:
The following Python Module can be imported into your Jupyter Notebook and called to run your simulations and analyse your results.
utils.py
import glob
import pandas as pd
import numpy as np
import subprocess
import shutil
import os
import platform
#Run SDK using Popen - checks operating system and returns resultsdefrun_sdk(params, maven_repo, java_cmd='java', debug=True, delete_tempfile=True):
_pwd = os.getcwd()
os.chdir(params['notebook_settings']['sdk_path'])
_config_export_path_250 ='simudyne.core.parquet-export-path'
_repos = get_repos_2_5_0()
_parquet_path = params['config'][_config_export_path_250]if _config_export_path_250 in params['config']else'output'if os.path.isdir(_parquet_path):
shutil.rmtree(_parquet_path)
_sep =Noneif platform.system()=='Windows':
_sep =';'elif platform.system()=='Darwin':
_sep =':'else:raise ValueError('{0} not supported.',format(platform.system()))
_config =[java_cmd]+['-D{0}={1}'.format(k, params['vm'][k])for k in params['vm']]+['-classpath',
_sep.join(['target/classes']+['{1}/{0}'.format(i, maven_repo)for i in _repos]),'MainNoConsole',
get_input(params['config']),
get_input(params['input'])]if platform.system()=='Windows':
o = subprocess.Popen(_config, stdout = subprocess.PIPE, stderr=subprocess.STDOUT)if o.stdout != b'':
_o = o.stdout.read().decode('utf-8').split('\n')if debug:[print(s)for s in _o]if o.stderr isnotNone:
_o = o.stderr.read().decode('utf-8').split('\n')if debug:[print(s)for s in _o]elif platform.system()=='Darwin':
o = subprocess.run(_config, capture_output=True, check=False)if o.stderr != b'':
_o = o.stderr.decode('utf-8').split('\n')if debug:[print(s)for s in _o]if o.stdout != b'':
_o = o.stdout.decode('utf-8').split('\n')if debug:[print(s)for s in _o]
_res = get_res(output_folder=_parquet_path)if delete_tempfile:
shutil.rmtree(_parquet_path)
os.chdir(_pwd)return _res
#Get map of inputs defget_input(_map):return','.join(['{0}={1}'.format(a,str(b).lower()iftype(b)==boolelse b)for a,b in _map.items()])#Return results from output pathdefget_res(output_folder='output'):
_sep =Noneif platform.system()=='Windows':
_sep ='\\'elif platform.system()=='Darwin':
_sep ='/'else:raise ValueError('{0} not supported.',format(platform.system()))
res ={}for f in glob.glob('{0}/**/*/*.csv'.format(output_folder), recursive=True):
_f = f.split(_sep)
_name = _f[-2]
_run = _f[-1].split('.csv')[0]if _run notin res:
res[_run]={}
df = pd.read_csv(f)
df = pd.concat({i: pd.Series(df.loc[i])for i in df.index}, axis=1).T
df['time']= pd.to_datetime(df['time'])
res[_run][_name]= df
run_ids = res.keys()
dataset_ = np.unique([list(res[k].keys())for k in res])return res, np.sort(list(run_ids)), dataset_
defget_repos_2_5_0():return['simudyne/simudyne-nexus-server_2.12/2.5.0/simudyne-nexus-server_2.12-2.5.0.jar','org/scala-lang/scala-library/2.12.8/scala-library-2.12.8.jar','simudyne/simudyne-core-exec_2.12/2.5.0/simudyne-core-exec_2.12-2.5.0.jar','com/jcabi/jcabi-aspects/0.22.5/jcabi-aspects-0.22.5.jar','com/jcabi/jcabi-log/0.17/jcabi-log-0.17.jar','org/aspectj/aspectjrt/1.8.7/aspectjrt-1.8.7.jar','javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar','mysql/mysql-connector-java/8.0.26/mysql-connector-java-8.0.26.jar','simudyne/simudyne-console_2.12/2.5.0/simudyne-console_2.12-2.5.0.jar','com/typesafe/akka/akka-actor_2.12/2.6.10/akka-actor_2.12-2.6.10.jar','com/typesafe/config/1.4.0/config-1.4.0.jar','com/typesafe/akka/akka-stream_2.12/2.6.10/akka-stream_2.12-2.6.10.jar','com/typesafe/akka/akka-protobuf-v3_2.12/2.6.10/akka-protobuf-v3_2.12-2.6.10.jar','org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar','com/typesafe/ssl-config-core_2.12/0.4.2/ssl-config-core_2.12-0.4.2.jar','org/scala-lang/modules/scala-parser-combinators_2.12/1.1.2/scala-parser-combinators_2.12-1.1.2.jar','com/typesafe/akka/akka-remote_2.12/2.6.10/akka-remote_2.12-2.6.10.jar','com/typesafe/akka/akka-pki_2.12/2.6.10/akka-pki_2.12-2.6.10.jar','com/hierynomus/asn-one/0.4.0/asn-one-0.4.0.jar','org/agrona/agrona/1.7.2/agrona-1.7.2.jar','com/typesafe/akka/akka-slf4j_2.12/2.6.10/akka-slf4j_2.12-2.6.10.jar','com/typesafe/akka/akka-actor-typed_2.12/2.6.10/akka-actor-typed_2.12-2.6.10.jar','com/typesafe/akka/akka-http_2.12/10.1.3/akka-http_2.12-10.1.3.jar','com/typesafe/akka/akka-http-core_2.12/10.1.3/akka-http-core_2.12-10.1.3.jar','com/typesafe/akka/akka-parsing_2.12/10.1.3/akka-parsing_2.12-10.1.3.jar','com/github/pathikrit/better-files_2.12/2.17.1/better-files_2.12-2.17.1.jar','com/github/pathikrit/better-files-akka_2.12/2.17.1/better-files-akka_2.12-2.17.1.jar','com/fasterxml/jackson/core/jackson-core/2.9.9/jackson-core-2.9.9.jar','com/fasterxml/jackson/core/jackson-databind/2.9.9/jackson-databind-2.9.9.jar','com/fasterxml/jackson/dataformat/jackson-dataformat-avro/2.9.9/jackson-dataformat-avro-2.9.9.jar','com/fasterxml/jackson/core/jackson-annotations/2.9.9/jackson-annotations-2.9.9.jar','com/fasterxml/jackson/module/jackson-module-scala_2.12/2.9.9/jackson-module-scala_2.12-2.9.9.jar','com/fasterxml/jackson/module/jackson-module-paranamer/2.9.9/jackson-module-paranamer-2.9.9.jar','org/json4s/json4s-native_2.12/3.6.9/json4s-native_2.12-3.6.9.jar','org/json4s/json4s-core_2.12/3.6.9/json4s-core_2.12-3.6.9.jar','org/json4s/json4s-ast_2.12/3.6.9/json4s-ast_2.12-3.6.9.jar','org/json4s/json4s-scalap_2.12/3.6.9/json4s-scalap_2.12-3.6.9.jar','org/json4s/json4s-jackson_2.12/3.6.9/json4s-jackson_2.12-3.6.9.jar','commons-io/commons-io/2.6/commons-io-2.6.jar','com/beust/jcommander/1.72/jcommander-1.72.jar','simudyne/simudyne-core_2.12/2.5.0/simudyne-core_2.12-2.5.0.jar','org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar','org/apache/commons/commons-lang3/3.7/commons-lang3-3.7.jar','commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar','commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar','com/google/guava/guava/29.0-jre/guava-29.0-jre.jar','com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar','com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar','org/checkerframework/checker-qual/2.11.1/checker-qual-2.11.1.jar','com/google/errorprone/error_prone_annotations/2.3.4/error_prone_annotations-2.3.4.jar','com/google/j2objc/j2objc-annotations/1.3/j2objc-annotations-1.3.jar','com/typesafe/scala-logging/scala-logging_2.12/3.9.0/scala-logging_2.12-3.9.0.jar','org/scala-lang/scala-reflect/2.12.4/scala-reflect-2.12.4.jar','org/scala-lang/modules/scala-java8-compat_2.12/0.9.0/scala-java8-compat_2.12-0.9.0.jar','org/eclipse/collections/eclipse-collections-api/9.2.0/eclipse-collections-api-9.2.0.jar','org/eclipse/collections/eclipse-collections/9.2.0/eclipse-collections-9.2.0.jar','au/com/bytecode/opencsv/2.4/opencsv-2.4.jar','org/apache/arrow/arrow-vector/0.12.0/arrow-vector-0.12.0.jar','org/apache/arrow/arrow-format/0.12.0/arrow-format-0.12.0.jar','org/apache/arrow/arrow-memory/0.12.0/arrow-memory-0.12.0.jar','joda-time/joda-time/2.9.9/joda-time-2.9.9.jar','com/carrotsearch/hppc/0.7.2/hppc-0.7.2.jar','io/netty/netty-buffer/4.1.22.Final/netty-buffer-4.1.22.Final.jar','com/google/flatbuffers/flatbuffers-java/1.9.0/flatbuffers-java-1.9.0.jar','com/google/protobuf/protobuf-java/3.6.0/protobuf-java-3.6.0.jar','io/netty/netty-common/4.1.42.Final/netty-common-4.1.42.Final.jar','com/typesafe/akka/akka-stream-typed_2.12/2.6.10/akka-stream-typed_2.12-2.6.10.jar','simudyne/simudyne-core-abm_2.12/2.5.0/simudyne-core-abm_2.12-2.5.0.jar','org/apache/parquet/parquet-avro/1.11.1/parquet-avro-1.11.1.jar','org/apache/parquet/parquet-column/1.11.1/parquet-column-1.11.1.jar','org/apache/parquet/parquet-common/1.11.1/parquet-common-1.11.1.jar','org/apache/yetus/audience-annotations/0.11.0/audience-annotations-0.11.0.jar','org/apache/parquet/parquet-encoding/1.11.1/parquet-encoding-1.11.1.jar','org/apache/parquet/parquet-hadoop/1.11.1/parquet-hadoop-1.11.1.jar','org/apache/parquet/parquet-jackson/1.11.1/parquet-jackson-1.11.1.jar','org/apache/parquet/parquet-format-structures/1.11.1/parquet-format-structures-1.11.1.jar','javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar','com/googlecode/concurrentlinkedhashmap/concurrentlinkedhashmap-lru/1.4.2/concurrentlinkedhashmap-lru-1.4.2.jar','simudyne/simudyne-core-graph_2.12/2.5.0/simudyne-core-graph_2.12-2.5.0.jar','it/unimi/dsi/fastutil/6.5.7/fastutil-6.5.7.jar','simudyne/simudyne-core-abm-testkit_2.12/2.5.0/simudyne-core-abm-testkit_2.12-2.5.0.jar','io/eels/eel-core_2.12/1.2.4/eel-core_2.12-1.2.4.jar','com/sksamuel/exts/exts_2.12/1.49.0/exts_2.12-1.49.0.jar','com/univocity/univocity-parsers/2.4.1/univocity-parsers-2.4.1.jar','com/h2database/h2/1.4.196/h2-1.4.196.jar','io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar','io/dropwizard/metrics/metrics-jvm/3.1.2/metrics-jvm-3.1.2.jar','io/eels/eel-hive_2.12/1.2.4/eel-hive_2.12-1.2.4.jar','io/eels/eel-orc_2.12/1.2.4/eel-orc_2.12-1.2.4.jar','org/apache/orc/orc-core/1.4.0/orc-core-1.4.0.jar','io/airlift/aircompressor/0.3/aircompressor-0.3.jar','org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar','org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar','log4j/log4j/1.2.17/log4j-1.2.17.jar','org/apache/hadoop/hadoop-common/2.7.5/hadoop-common-2.7.5.jar','org/apache/hadoop/hadoop-annotations/2.7.5/hadoop-annotations-2.7.5.jar','commons-cli/commons-cli/1.2/commons-cli-1.2.jar','xmlenc/xmlenc/0.52/xmlenc-0.52.jar','commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar','commons-codec/commons-codec/1.4/commons-codec-1.4.jar','commons-net/commons-net/3.1/commons-net-3.1.jar','commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar','javax/servlet/servlet-api/2.5/servlet-api-2.5.jar','org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar','org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar','org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar','javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar','com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar','com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar','org/codehaus/jettison/jettison/1.1/jettison-1.1.jar','com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar','javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar','javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar','javax/activation/activation/1.1/activation-1.1.jar','org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar','org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar','com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar','asm/asm/3.1/asm-3.1.jar','commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar','net/java/dev/jets3t/jets3t/0.9.0/jets3t-0.9.0.jar','org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar','org/apache/httpcomponents/httpcore/4.1.2/httpcore-4.1.2.jar','com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar','commons-lang/commons-lang/2.6/commons-lang-2.6.jar','commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar','commons-digester/commons-digester/1.8/commons-digester-1.8.jar','commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar','commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar','org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar','org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar','com/google/code/gson/gson/2.2.4/gson-2.2.4.jar','org/apache/hadoop/hadoop-auth/2.7.5/hadoop-auth-2.7.5.jar','org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar','org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar','org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar','org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar','org/apache/curator/curator-framework/2.7.1/curator-framework-2.7.1.jar','com/jcraft/jsch/0.1.54/jsch-0.1.54.jar','org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar','org/apache/curator/curator-recipes/2.7.1/curator-recipes-2.7.1.jar','com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar','org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar','org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar','io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar','org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar','org/apache/avro/avro/1.8.2/avro-1.8.2.jar','com/thoughtworks/paranamer/paranamer/2.7/paranamer-2.7.jar','org/xerial/snappy/snappy-java/1.1.1.3/snappy-java-1.1.1.3.jar','org/tukaani/xz/1.5/xz-1.5.jar']
Once these files have been added and adapted to your environment and model you can begin testing in a Jupyter Notebook.
Jupyter Notebook
In this section we will build a simple example from scratch to demonstrate how to call your model.
Pull up a blank Jupyter Notebook and import your utils.py module and define the params for your model as below:
MyNotebook.ipynb
from utils import*#Load notebook settingswithopen('notebook_settings.json','r')as f:
notebook_settings = json.load(f)
seed =12345678910
level ='WARN'#for the logger#create dict of VM and input parameters
params ={'vm':{'seed': seed,'runs':1,'ticks':100,'level': level},'config':{'simudyne.core.export-path':'Output_{0}'.format(seed)},'input':{'input1':1,'input2':100,'input3':1000},'notebook_settings': notebook_settings
}
Now that you have set the notebook_settings.json, VM params, and model input params you can call your run_sdk() function and return your results dict with the data from your runs:
To better understand the structure/contents of the res, run_ids, and dataset_ return variables, reference the get_res() function in your utils.py file.
Conclusion
Running the SDK from an external tool requires the ability to call a Java executable JAR file. We provided this example in Python using Jupyter notebook as this is a popular analysis tool with users of the Simudyne SDK. Other languages and applications have similar features in place for running subprocesses or making callouts to Java and should likely have no problem running your Simudyne SDK models as well.