Loading Data

Last updated on 16th July 2024

Loading Data

This part will get you started with populating models using data.

Loading agents from data

Let's take a simplified version of the agent class City and let's say you have the following kind of data where each City has:

  • a name, that is a String
  • a population, that is an int
  • a surface, that is a float

More precisely the data is organised like so:

id name population surface
23323443 Kodiak 783 904 20 229
48217039 Firebrick 13 003 4320
37410058 Onalaska 2 377 843 61 839
93749281 Ak-Chin Village 453 150 17 323
55283047 Blacktail 1 921 613 49 315
... ... ... ...

Here, there is an additional field called id (which is a unique number) that identifies each record.

On the type of data supported

The Simudyne SDK supports consistent schema of data : the records present in your data should not be duplicated and there must exist a field that identifies each individual record.

Each record will be mapped to an agent, and each field of a record, except the indexing field (that is defaulted to id), will be mapped to the eponymous agent attribute.

Hence, if your data does not have a field identifying each of your record, you should add a new id column to your dataset to use.

We can come up with a simplified version of the Agent class City described above to match this data:

loading

To import data, you have to annotate your fields like so so that records in your data can be mapped to Cities.

Class definition (Java)

class City extends Agent {
  @Constant
  String name;

  @Constant
  int population;

  @Constant
  float surface;

  //...
}

Finally, to import your agents in your model, you have to use the system loadGroup() method.

You have to specify a Source to use ; this object is dedicated to import data from your file or data base. Let's use a CSVSource to import the data from a CSV file !

Is there only Source for CSV (CSVSource) out there ?

In our example we suppose the data is stored in a CSV so we use a CSVSource to trigger it. However, you can use other Source -- like JDBCSource -- to import data from other data storage. Check the dedicated section in the following.

Windows user

The `/` in the paths needs to be replaced with `\`.

Creating data group (Java)

// Creating a Source to import data -- you can use another type of source too
CSVSource citySource = new CSVSource("/path/to/myCities.csv");

Group<City> cities = loadGroup(City.class, citySource);

And now, all the Cities contains as records in your data are imported into your model !

On IDs present in data vs IDs used in the AgentBasedModel

The IDs present in the AgentBasedModel when importing data are not used any more as data gets remapped to agent that are given a new ID adapted to the state of the system.

Thus, IDs present in your data still remain important as they identify agents; but won't be used in the system once the records are imported.

What if I do not have enough of fields for the agent that I want to model ?

If you do not have enough fields in your dataset compared to what you would like to have in your model for an agent class, this is not restrictive to use data import. The fields present into your data will be mapped to some of the fields of your agents, leaving the remaining ones with their default values.

Let's now imagine that you want to introduce Roads between your Cities.

links from data

And let's say your have the same dataset of Cities ...

id name population surface
23323443 Kodiak 783904 20229
48217039 Firebrick 13003 4320
37410058 Onalaska 2377843 61839
93749281 Ak-Chin Village 453150 17323
55283047 Blacktail 1921613 49315
... ... ... ...

... and that you have another a data sets of Roads linking Cities together:

from to length
23323443 48217039 23.32
37410058 23323443 343.3
23323443 55283047 43.13
55283047 93749281 33.38
... ... ...

Here, from and to identify respectively the source agents and the target agents.

The remaining fields will be injected in the link at their creation. In our context, there is just one remaining field, length, that represent the length of a Road from the City identified by the ID from to the City identified by the ID to.

You can import those links in your model and between your two groups using the connect() with a 'Source'.

Connecting a Group Using Data (Java)

CSVSource citySource = new CSVSource("/path/to/myCities.csv");
CSVSource roadSource = new CSVSource("/path/to/myRoads.csv");

Group<City> cities = loadGroup(City.class, citySource);

cities.loadConnections(cities, Road.class, roadSource);

You can also connect two different Groups as well:

Connect two Groups using Data

CSVSource citySource = new CSVSource("/path/to/myCities.csv");
CSVSource countrySource = new CSVSource("/path/to/myCountry.csv");

CSVSource linkSource = new CSVSource("/path/to/myLinks.csv");

Group<City> cities = loadGroup(City.class, citySource);
Group<Country> countries = loadGroup(Country.class, countrySource);

cities.loadConnections(countries, MyLink.class, linkSource);

More about Sources

If you have data you want to import from a database, you can use a JDBCSource that supports MySQL, SQLite as well as PostGreSQL. Here is an example for SQLite:

Using JDBC Sources (java)

String connectionString = s"jdbc:sqlite:path/to/myDataBase.db";

JDBCSource citySource = new JDBCSource(connectionString, "Country", "username", "password");

Java JDBC connection string examples

How about supporting other Sources ?

For now, Source have been implemented for CSV files and relational databases. In the future, more solutions will be covered, especially non-relational databases a la NoSQL.