Metric names and icons

[ecomodeller]

The skore project is designed to make evaluation of ML models easy.
One feature is nice skill tables were the direction of better is indicated for clarity.

skore uses a simple dict to store this info.
https://github.com/probabl-ai/skore/blob/b5f84adb03cfb659233d3085739b33ab086710d9/skore/src/skore/sklearn/_estimator/metrics_accessor.py#L27-L38

Should we add a similar feature to ModelSkill?

[jsmariegaard]

def metric(best=None):
    """Decorator to attach a 'best' attribute to metric functions."""
    def decorator(func):
        func.best = best  # Set the best value
        return func
    return decorator

@metric(best=0)
def mean_error(obs, model):
    return (model - obs).mean()

@metric(best="+")
def r2_score(obs, model):
    return 1 - ((obs - model)**2).sum() / ((obs - obs.mean())**2).sum()

# User-defined metric without 'best'
def user_metric(obs, model):
    return (obs - model).std()

# Function to get best value safely
def get_best(metric_func):
    """Returns the 'best' attribute if it exists, otherwise None."""
    return getattr(metric_func, "best", None)

# Usage
print(get_best(mean_error))  # Output: 0
print(get_best(r2_score))    # Output: "+"
print(get_best(user_metric)) # Output: None (no best defined)

[ecomodeller]

Is this how you envision this decorator to be used for user defined metrics?

from modelskill.metrics import metric, add_metric

@metric(best="-")
def silly(obs,model):
    return np.random.uniform()

add_metric(silly) # this step seems unnecessary, could this be part of the decorator?

cc.skill(metrics=[silly]).style()

[jsmariegaard]

Nice. I think the add_metric could be part of the decorator too and potentially more behaviour in the future. Seems like a nice way to register functions as metrics :-)