Propose some data structures for scattering model data

[gavinmacaulay]

One of echoSMs objectives is to encourage the use of common data structures (and perhaps file formats) for organism shapes and properties. The aim is to make it easier to use (and reuse) organism data across multiple models and implementations.

Currently, echoSMs uses:

• TOML-formatted files for:

  • Body shapes for the KRM model
  • Body shapes for the DWBA and SDWBA models
  • Benchmark model parameters given in the Jech et al paper

• STL-formatted files for triangular surface meshes

• CSV-formatted files for:

  • Model results for the Jech et al benchmarks
  • Model results for various KRM and DWBA test models

Previous examples of fisheries acoustics file formats indicate that specifying a data structure rather than a file format is a better long-term solution as it gives greater flexibility and ability to adapt to new file formats and to accommodate the varying requirements of institutional data policies and preferences.

Developing a data structure then requires information on:

• the data required to run scattering models
• whether storage of raw organism shape data is necessary (e.g., x-ray data)
• metadata required to understand the model data (e.g., units, coordinate conventions, data source, attributions, etc)
• relevant relationships between these data

File formats that can readily store proposed data structures should be kept in mind and examples provided. Use of existing file and data formats should be preferred over developing new ones (this significantly reduces the effort required to use those formats in programming languages).

[jmjech]

Hi Gavin,
I've been playing with the schema a bit and created a schema that incorporates all your fields, but reorganizes them. If I understand correctly, your schema defined a dataset as a collection of specimens and then the shape data for each specimen. I was having a bit of difficulty with that organization, so I thought of defining a dataset as the data and metadata for a single shape, i.e., anatomical feature. For me, it's a bit cleaner and easier to follow the schema.

This is the draft schema. I'm new to json schema, so I most likely have some misuse of syntax (It does check out with json.tools, so you can at least read and parse it), especially with unevaluatedProperties and arrays vs objects.

draft_datastore.json

I wanted to get it to you before our meeting on Tuesday so we can discuss then, or provide any comments/responses here.

mike

[gavinmacaulay]

Some updates after talking to @jmjech:

• A link to this issue (and hence the alternative schema above) has been added to the echoSMs schema documentation (to make readers aware of a different way to structure the shape and meta data).
• Units have been added to the current schema in echoSMs. Regardless of how the schema is eventually structured, having explicit units is important for long-term usability.