{"data":{"markdownRemark":{"html":"

The other data formats on Data Export all make usage of the default built-in output that is managed either via configuring of what data is serialized, or through output channels Output Channels. However there may be the case where these types of outputs do not fit into your data science workflow. The below will detail a brief example using H2 - an in-memory relational database to explain how this can be accomplished.

\n

Simudyne SDK as of version 2.4 makes usage of Eel a toolkit for manipulating data in the Hadoop ecosystem making usage of outputs in parquet, orc, csv in locations such as local, HDFS, or Hive tables. Existing users can output both local and/or distributed on a cluster, users will not need to change anything about their code in order to make usafe of this new output technology and enjoy the performance benefits that it provides.

\n

However in order to develop a customized data pipeline there are additional options via usage of Eel being included in the SDK that users should feel free to take advantage of.

\n

Explaining Source & Sink

\n

The core data structure in Eel is a DataStream consisting of a Schema and n rooms containing values for each field in the schema. Conceptually, a DataStream is similar to a table in a normal reltational database.

\n

With Eel, DataStreams can be read from a variety of Sources, and correspondingly can be written out to Sinks. These Source/Sink types can cary from Hive, JDBC Database, or collections in Java, JSON, or Parquet.

\n
\n

Supported Eel Projects

\n By default, Eel Core allows for the variety of basic Source/Sink formats such as Parquet/CSV/etc. We also include Hive and Orc to allow users to both natively work with Hive via our SDK or easily work with Orc formatted files.\n \n However, HBase, Kafka, and Kudu are not included. Should a user wish to work with these technologies they must do so at their own risk. If you wish to include these parts of the Eel SDK's libraries you should make sure to EXCLUDE both Hive/Hadoop/Log4J packages from that library in order to not cause issues with conflicting versions.\n
\n

Here is a basic example of the code structure for a Source/Sink:

\n
val source = CsvSource(new Path(\"input_historical.csv\"))\nval sink = ParquetSink(new Path(\"output_npv.pq\"))\nsource.toDataStream().filter(_.get(\"npv\") > 0).to(sink)
\n

What's happening here is that we are reading in a csv file containing presumably multiple columns/rows of data. Then we define a Parquet output that will contain just the NPV value. We then take the source and create a DataStream, and then we can process that stream with actions such as map, filter, take, drop. Here we add a filter to output the stream only if the NPV value is greater than 0. If this is true it will output the contents of the row in the same schema format as the input, but to a parquet Sink via the .to() method.

\n

Defining a Custom Schema

\n

Note if you wish to make usage of Custom Output Channels please refer to our (Reference Docs) and use the standard Parquet or Hive outputs.

\n

However, because of the versatility using Eel, it might be faster or make more sense to define a custom Source/Sink operation within your model that does\nnot have to handle the serailization used by various Simudyne SDK structures. As such, please refer to this short example on how if using your own custom\nSource/Sink code on defining a schema.

\n
val personDetailsStruct = Field.createStructField(\"PERSON_DETAILS\",\n  Seq(\n\tField(\"NAME\", StringType),\n\tField(\"AGE\", IntType.Signed),\n\tField(\"SALARY\", DecimalType(Precision(38), Scale(5))),\n\tField(\"CREATION_TIME\", TimestampMillisType)\n  )\n)\nval schema = StructType(personDetailsStruct)\n\nval rows = Vector(\n  Vector(Vector(\"Fred\", 50, BigDecimal(\"50000.99000\"), new Timestamp(System.currentTimeMillis()))),\n  Vector(Vector(\"Gary\", 50, BigDecimal(\"20000.34000\"), new Timestamp(System.currentTimeMillis()))),\n  Vector(Vector(\"Alice\", 50, BigDecimal(\"99999.98000\"), new Timestamp(System.currentTimeMillis())))\n)
\n

This population of a the rows Vector can instead be done as an agent action within the step function.

\n
static Action<Person> updateSalary() {\n    return Action.create(\n        Person.class,\n        p -> {\n\t\t\tinterim_row = Vector(\"Fred\", 50, BigDecimal(\"50000.99000\"), new Timestamp(System.currentTimeMillis()))\n\t\t\trows.addAll(interim_row)\n\t\t}\n\t);\n}
\n

This example structure can then be written out to a Parquet Sink like this:

\n
DataStream.fromValues(schema, rows).to(ParquetSink(parquetFilePath))
\n

Creating an H2 Sink via JDBC

\n

If you are working with H2 you will need to include the driver into your code by adding org.h2.Driver to your imports.

\n

This example given a database string and a tableName will return a JdbcSink which as shown above can be the output. For H2 the database string should look something like this jdbc:h2:~/test

\n
def createJDBCSink(dbName: String, tableName: String): JdbcSink = {\n\tval driver = \"org.h2.Driver\" \n\n\ttry {\n\t  Class.forName(driver)\n\t} catch {\n\t  case _: Throwable => println(\"JDBC Driver not Found\");\n\t}\n\n\tval dataSource = new BasicDataSource()\n\tdataSource.setDriverClassName(driver)\n\tdataSource.setUrl(dbName)\n\tdataSource.setUsername(\"username1\")\n\tdataSource.setPassword(\"password1\")\n\tdataSource.setPoolPreparedStatements(false)\n\n\tnew JdbcSink(() => dataSource.getConnection(), tableName)\n}
","headings":[{"value":"Explaining Source & Sink","depth":2},{"value":"Defining a Custom Schema","depth":2},{"value":"Creating an H2 Sink via JDBC","depth":2}],"frontmatter":{"title":"H2 and JDBC","toc":null,"experimental":null}},"site":{"siteMetadata":{"title":"Simudyne Docs","latestVersion":"2.6"}}},"pageContext":{"absolutePath":"/home/vsts/work/1/s/content/2.6/reference/data_export/h2.md","versioned":false,"version":"2.6","kind":"reference","pagePath":"/reference/data_export/h2","chronology":{"prev":{"name":"Hive via Parquet","path":"/reference/data_export/hive"},"next":{"name":"Run and Deploy","path":"/reference/run_deploy"}},"lastUpdated":"2026-04-21T13:56:54.868Z"}}