VODML Mapping vs ModelInstanceInVot

[lmichel]

Both proposals (VOML Mapping, ModelInstanceInVot) have similar structures. There is nevertheless a major difference that is justified here.

• Fig 1 shows a dataset made with a block of metadata on the top of the data block and mapped with VOML Mapping
  • Data are mapped in the TEMPLATE block that maps data for one row.
  • Metadata are located in the GLOBALS block along with the coordinate frames.
  • Issues
    • The parser has to browse 2 different mapping blocks to retrieve all the components of the dataset
    • The parser has no easy way to discrimine which GLOBALS element is part or not of the dataset mapping.
• Fig 3 shows the same dataset mapped with ModelInstanceInVot
  • Both data and metadata are mapped within the TABLE_TEMPLATE blocks
  • The data mapping is enclosed in a TABLE_RAW_TEMPLATE block
  • Gains
    • All elements related to the dataset mapping are located in a single block (TABLE_TEMPLATE), this makes the parser job easier.
    • All elements not related to any dataset are located in the GLOBALS block
    • We can have several TABLE_RAW_TEMPLATE blocks within a single TABLE_TEMPLATE. This allows to tell the parser to iterate several times on the same data table e.g. to extract data subsets (selection by filter in time-series/gaia_multiband).

The advantage of the ModelInstanceInVot mapping structure is even more obvious for multi-table VOTables.

• Fig 2 shows the mapping structure for a VOTable containing 2 tables, each one with its own mapping (e.g. source + detections).
  • The GLOBALS block contains now the meta data of the 2 datasets in addition to the coordinate frames.
  • Issues
    • Same as before
    • The parser has no easy way to identify which GLOBALS element is part of dataset 1 ort 2
• Fig 4 shows the same datasets mapped with ModelInstanceInVot
  • Each table is mapped in a specific TABLE_TEMPLATE.
  • Gains
    • Same as before
    • No longer risk to mix up meta data of both datasets.

[mcdittmar]

I'm taking a closer look to compare the annotation schemes today.

I notice that the preview PDF in the ModelInstanceInVot repository is not current (2020-08-18).
The one under doc (2020-09-15) appears to match the current serializations.

Is that correct?

[lmichel]

The CI failed! I’ve been trying to run it again

I'm taking a closer look to compare the annotation schemes today. I notice that the preview PDF <https://github.com/ivoa-std/ModelInstanceInVot/releases/download/auto-pdf-preview/vodml-instance-vot-draft.pdf> in the ModelInstanceInVot repository is not current (2020-08-18). The one under doc <https://github.com/ivoa-std/ModelInstanceInVot/blob/master/doc/model-instance-in-vot.pdf> (2020-09-15) appears to match the current serializations. Is that correct?

yes, the one under the doc is always uptodate

[Bonnarel]

Good clarification Laurent, but I think there is a typo in your text above. TABLE_RAW_MAPPING shoukd write "TABLE_ROW-TEMPLATE", shouldn't it ?

[lmichel]

right, fixed

[mcdittmar]

I wrote this up some days ago.. forgot to push the green button!.
I spent the day studying the ModelInstanceInVot syntax so that I could, hopefully, provide a more informed response.

General comment:
One of the main reasons I'm using the Mapping syntax, is that it was designed/vetted against a set of requirements/cases covering the expectations on any syntax. To my knowledge, these requirements on our Annotation syntax have not changed, so I'm reviewing this scheme against my understanding of the Mapping syntax requirements. It is not quite fair to evaluate one document based on the requirements of another, but in this case, we are trying to determine what is gained/lost by the various options.

MODEL_INSTANCE:
Your comments seem to come from the perspective that the annotation represents "the dataset". The Model_Instance description follows that view in that you can provide for no more than 1 'name' and 'uri' for "the mapped model.". It is further supported by INSTANCE with dmrole="root", which "can be used to tell the parser whic class has to be instanciated first.". If the annotation contains more than one "root" instance (eg: a MANGO and a Cube/TimeSeries), the MODEL_INSTANCE name and uri attributes have little meaning.

It is a specific goal of our Annotation Syntax to 'Annotate files to multiple models' (Mapping doc; Section 2.3.2). This can be either different products ( MANGO, Cube ) or different versions of the same model. I think this is a feature we want to preserve, and I am expecting the case in Issue #29 does just this?

GLOBALS/TEMPLATES in the Mapping syntax:
GLOBALS holds 'direct instances' (COLUMN is not allowed under GLOBALS) so that each instance may be given an ID which can be referenced by other instances

• There can be more than one GLOBALS (where ModelInstanceInVot can have only 1).
  • This allows you to bundle GLOBALS into logical sets.
    • 1 for Coordinate Systems
    • 1 for Photometry Filters
    • 1 for each DatasetMetadata instance needed

TEMPLATES hold all instances which are created per row of the associated TABLE, so looks equivalent to the ModelInstanceInVot TABLE_ROW_TEMPLATE but at a higher level

• There can be more than one TEMPLATES as well. In my usage, I think this has only come into play when there is more than one TABLE

So, in your first pair of diagrams.. you cannot consider the 1 TABLE to represent 1 Dataset, or even 1 'root' object.
Instead, what the Mapping syntax does is separate the elements into sections where parsers must act differently to generate the instances

• GLOBALS - simpler, single direct instance generated from LITERAL or CONSTANT/PARAM and
• TEMPLATES - where it must manage the TABLE data and possibly contend with KEYS to generate multiple instances

TABLE_MAPPING..(TABLE_TEMPLATE in diagrams?):
This element confuses me.
The ModelInstanceInVot document states that the annotation is independent of the data structure

• Section 2.2: Requirements: "mapping structure must be independent of the data structure"
• Section 3.1: Mapping Block Structure: "The mapping construction rules are independant from any model or data layout"

But this element appears to tie very directly with the VOTable serialization structure

• Section 3.1: "There is one <TABLE_MAPPING> per mapped <TABLE>"
• Section 3.2.3: "contains the mapping statements of the data contained in one TABLE"
• Section 3.2.3: "A TABLE cannot be referenced by more than one TABLE_MAPPING"

How does this accommodate an INSTANCE whose content is distributed among multiple TABLEs?
This is another specific goal of the Annotation Syntax, "It is often the case that two or more files or tables represent different pieces of information regarding the same astronomical sources of objects." (Mapping doc; Section 2.3.3). So, the data required to create any given INSTANCE may be spread among different TABLEs, even for the GLOBALS.

In the second pair of diagrams.. there can certainly be INSTANCES in GLOBALS which contain elements from multiple TABLES.

Summary:
The ModelInstanceInVot examples above may look and behave more cleanly, but if I understand the descriptions properly, they fail to satisfy some basic requirements of the Annotation Syntax.

1. We may include annotation to more than one model (different models, different versions of one model).
2. The Annotation must be independent of the serialization structure. (ie: cannot isolate per TABLE)

Additional Comments:

• TABLE_MAPPING tableref attribute: "first resolved against the TABLE identifier (@id), then against the table name".
  • there are some workshop examples where the TABLE has no ID (and I've needed to add one).
  • several also had no ID on FIELDs, which I found surprising
  • this fallback in the spec may help accommodate these cases without modifying the original TABLE content.
• FILTER/JOIN/GROUPBY
  • These all seem to be some action related to using the PRIMARYKEY/FOREIGNKEY of the Mapping syntax.
  • I don't understand GROUPBY, but I rather like FILTER/JOIN as a more intuitive element (since I'm not a database person).
  • I'd like to focus some time on cases which use these

[lmichel]

Long post multiple answers

** Mapping usecase**

The use case sections in both document differ, so you comparison is teneous.
The ModelIntanceInVot use case is simple:

• You have a model , you have a VOTable: ModelIntanceInVot allows clients to build model instances populated with VOtable data.
• Once you have this instance , implemeting your use cases is straighforward

In addition to this, we have a couple of requirements

• The mapping must be designed in a way that facilitate the annotation process
• This must work with most of the VOTables one can find in the market. If you come with a VOTable where your Cube data are randomly spread out over multi tables, ModelIntanceInVot will turn into some troubles

To facilitate both annotation and parsing tasks, ModelIntanceInVot is based on a drawer approach.

• One drawer for the global object
• One drawer for each model you want to map (only one at the moment, see below)
  So that the annoter knows where to put things and the clients knows where to look for. If it extracts one {TABLE_TEMPLATE} , it knows it gets everything it needs to build the instance mapped there.

[lmichel]

MODEL_INSTANCE:

Recently ModelIntanceInVot has been downgraded to support the annotation of only one model. The {MODEL} blocks has been removed.
The reasons for this was:

• The lack of use-cases
• Simplification of the syntax to help people to take possession of the cases

We could easily move back to a multi-model version:

• Restoring the {MODEL} block
• Allowing multiple {TABLE_TEMPLATE} for one {TABLE}

The root role is just an indication given by annoter.
Starting parsing by the root node is not mandatory. My own code ignores it by the way.
I've to admit that feature brings more confusion than help.

[lmichel]

GLOBALS

The ModelIntanceInVot {GLOBALS} block can hold as many {INSTANCE} as needed.
This is the drawer for the global objects.
{GLOBALS}/{INSTANCE} have no specific role.
They must have an ID to be referable from objects within {TABLE_MAPPING} blocks.

I would say that your {TEMPLATE} matches my {TABLE_ROW_TEMPLATE}. I added one level {TABLE_MAPPING} to encompass both metadata and data.
Beyond the benefits of the drawer structure this allows to support models with mutiple data collection. For instance, gaiamultiband has 3 data arrays, one per filter, each populated with a specific {TABLE_ROW_TEMPLATE}:

{TABLE_MAPPING}
    metatata
    {TABLE_ROW_TEMPLATE} for filter B    
    {TABLE_ROW_TEMPLATE} for filter RP    
    {TABLE_ROW_TEMPLATE} for filter BG

[lmichel]

TABLE_MAPPING..(TABLE_TEMPLATE in diagrams?):

The annotation is independent of the data structure in sense of the hierarchy the mapping elements (INSTANCE ATTRIBUTE COLLECTION) does not depend on the way that data arranged into the table.
If you remane columns or change their order or move value fron FIELDS to PARAMS (factorisation: Vizier often does it) that hierarchy dies not change, only the ATTRIBUTE@ref attribute has to be modifed.

This feature allows annoters to work with pre-defined blocks which is a very important requirement.

• This allows e.g. Vizier to demonstrate the capacity to generate annotated VOTable on the fly.
• This opens a window to annotate TAP responses on the fly.

If you have multiple tables, let's say source + detections:

• You map sources in a {TABLE_MAPPING} related to the source table
• You map detections in a {TABLE_MAPPING} related to the detection table
• You set a JOIN statement at the right place of the {TABLE_ROW_TEMPLATE} of the source {TABLE_MAPPING}

If you have meta-data spread over multiple tables (any use case?) , you can use the @dmref/@id pattern

[lmichel]

Summary

1. Answered above
2. I do not see which dependency you are talking about

• One mapped model => one {TABLE_MAPPING}

Additional Comments:

You are right to say that ModelInstance should support {TABLE} without name or ID either. A default behavior must defined (e.g. no name -> take the first)

GROUPBY

There is an example here

ex: {GROUPBY}, allows to retrieve per-source data from tables containing a mix of source data

[mcdittmar]

GROUPBY

There is an example here

ex: {GROUPBY}, allows to retrieve per-source data from tables containing a mix of source data

Without something consuming it, it's hard for me to see what comes out of it. It looks like a SORT method, producing a long collection of instances rearranged so that the instances with common 'filter' criteria are together. I'm not sure what the benefit of that is.

In the ZTF example you mentioned, appears to create:

• a collection of MangoObject (now 'Source') grouped by identifier
  • so we have multiple Source instances all id=a, then all id=b, then all id=c, etc
• each Source has an associated VOModelInstance which is a 'tsd:tsdata' (TimeSeries)
  • the tsdata instance contains a collection of tsd:Point, but I'm not sure how many Points are in the Collection
  • The GroupBy says the top level Source is instantiated from each row of the table, so there should be only 1 Point in each tsdata instance... generated by the row creating the top level Source instance.

So each Source has associated TimeSeries containing 1 Point?

The document example uses role:test.lightcurve generating a series of Collections of 'test:photometric.point', first points with "R" filter, then points with "G" filter, then points with "V" filter. I would expect in such an example to create 3 instances of LightCurve, one with the "R" points, one with the "G" points, and one with the "V" points.. but there is no such Instance.

If I'm interpreting this correctly, the only effect GroupBy has is to sort the records. This isn't a model feature, but a client side operation. eg: give me all your Source records in this region, sorted by source_id. The last clause is a client request which has nothing to do with the Source model.

Bottom line: if this is all correct, and GroupBy is ~= Sort, then it is a feature outside the scope of "Mapping the data to data models" which the Annotation is supposed to support.

[Bonnarel]

Hi Mark, Partial answer There is only one filter (zg) in the ZTF example you are pointing. If we would have 3 filters we would indeed create 3 different light curves (tsdata) Cheers François Le 07/04/2021 à 05:11, Mark Cresitello-Dittmar a écrit :

…

*GROUPBY* There is an example here <https://github.com/ivoa/dm-usecases/blob/main/usecases/time-series/fb%2Bml-proposal/TimeSeriesMangoZTF.annot.xml> ex: {GROUPBY}, allows to retrieve per-source data from tables containing a mix of source data Without something consuming it, it's hard for me to see what comes out of it. It looks like a SORT method, producing a long collection of instances rearranged so that the instances with common 'filter' criteria are together. I'm not sure what the benefit of that is. In the ZTF example you mentioned, appears to create: * a collection of MangoObject (now 'Source') grouped by identifier o so we have multiple Source instances all id=a, then all id=b, then all id=c, etc * each Source has an associated VOModelInstance which is a 'tsd:tsdata' (TimeSeries) o the tsdata instance contains a collection of tsd:Point, but I'm not sure how many Points are in the Collection o The GroupBy says the top level Source is instantiated from each row of the table, so there should be only 1 Point in each tsdata instance... generated by *the* row creating the top level Source instance. So each Source has associated TimeSeries containing 1 Point? The document example uses role:test.lightcurve generating a series of Collections of 'test:photometric.point', first points with "R" filter, then points with "G" filter, then points with "V" filter. I would expect in such an example to create 3 instances of LightCurve, one with the "R" points, one with the "G" points, and one with the "V" points.. but there is no such Instance. If I'm interpreting this correctly, the only effect GroupBy has is to sort the records. This isn't a model feature, but a client side operation. eg: give me all your Source records in this region, sorted by source_id. The last clause is a client request which has nothing to do with the Source model. Bottom line: if this is all correct, and GroupBy is ~= Sort, then it is a feature outside the scope of "Mapping the data to data models" which the Annotation is supposed to support. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMP5LTA3CMW5R2OC3XCSCOTTHPEOZANCNFSM4ZXJMUMA>.

[msdemlei]

On Tue, Apr 06, 2021 at 08:11:40PM -0700, Mark Cresitello-Dittmar wrote: Bottom line: if this is all correct, and GroupBy is ~= Sort, then it is a feature outside the scope of "Mapping the data to data models" which the Annotation is supposed to support.

+1 for "outside the scope" for me (even if it's really not sorting). If we start down this path, we'll eventually put in all of relational algebra into the annotation scheme. We already have ADQL for that, let's not re-invent SQL-in-XML (a.k.a ADQL 1.0, which fortunately didn't really happen). No, I am convinced that you simply will not be able to normalise all tabular structures out there so that a hypothetical client will always receive the same structure after "decoding" the annotation (ignoring for the moment that I claim very few clients will even want that). Which, of course, is another point for adopting small, flexible, independent, per-domain DMs and annotations...

[lmichel]

You might be worry to see such dataset (ZTF) circulating, but you have to admit that they do exist.

This is one of the examples we have been committed to work on 2 years ago in the frame of the TDIG.
ModelInstanceInVot proposal he been initiated by this work.
This is a use-case we worked on and there no plan or ambition either to extend the mapping syntax to any sort of QL feature.

The role of GROUP_BY is just to tell the parser how to extract per-source data from a table containing data related to multiple sources.
I believe that Vizier contains similar catalogs (@gilleslandais can confirm?).

In this case, using a model + annotation allows to do something which is not possible otherwise.