Extended reach goal science-motivated metrics

[aimalz]

Because people like thinking about ancillary, optional metrics for other potential challenge goals, I'm making this issue as a place for that discussion. Here are the ones that have come up already, and feel free to mention more in the comments:

• Early lightcurve challenge (which may focus more on maximizing true positives)
• Anomaly detection (which may focus more on minimizing false negatives)
• Class-specific metrics (a "best in class" as opposed to "best in show" metric for those who only aim to classify one object type)
• Hierarchical classes (distinguishing between sub-classes of a particular class)

However, this is just to get it out of your system -- please do not work on these until there is significant progress toward the main goal! We can implement them in follow-up challenges in the future*, but there won't even be a first version of the challenge unless we prioritize the single, agreed-upon goal of the full lightcurve challenge.

*_{We can also include at least some of them in the first version of the challenge on an opt-in basis, but we can't even progress with the Kaggle/Ramp process until we make a choice for the official metric, so please restrain yourselves for now.<\sub>}

[aimalz]

Another that came up in today's call was to divide test set objects based on sky location and compare metrics between them to see how sensitive classifiers are to dense/sparse areas of the sky.

• Sky sensitivity (classification quality as a function of region of sky, i.e. proximity to galactic plane)

[aimalz]

Here's another ideas from a dropped Slack discussion:

• Per-class metric extrema (anticipate a better idea of how to do this after challenge results are in, but it can be considered for a future version)