torch-crps

Implementations of the Continuously-Ranked Probability Score (CRPS) using PyTorch

Background

The Continuously-Ranked Probability Score (CRPS) is a strictly proper scoring rule. It assesses how well a distribution with the cumulative distribution function $F$ is explaining an observation $y$

$$ \text{CRPS}(F,y) = \int _{\mathbb {R} }(F(x)-\mathbb {1} (x\geq y))^{2}dx \qquad (\text{integral formulation}) $$

where $1$ denoted the indicator function.

In Section 2 of this paper Zamo & Naveau list 3 different formulations of the CRPS.

Incomplete list of sources that I came across while researching about the CRPS

• Hersbach, "Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems"; 2000
• Gneiting et al.; "Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistis and Minimum CRPS Estimation"; 2004
• Gneiting & Raftery; "Strictly Proper Scoring Rules, Prediction, and Estimation"; 2007
• Zamo & Naveau; "Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts"; 2018
• Jordan et al.; "Evaluating Probabilistic Forecasts with scoringRules"; 2019
• Olivares & Négiar & Ma et al; "CLOVER: Probabilistic Forecasting with Coherent Learning Objective Reparameterization"; 2023
• Vermorel & Tikhonov; "Continuously-Ranked Probability Score (CRPS)" blog post; 2024
• Nvidia; "PhysicsNeMo Framework" source code; 2025
• Zheng & Sun; "MVG-CRPS: A Robust Loss Function for Multivariate Probabilistic Forecasting"; 2025

Application to Machine Learning

The CRPS can be used as a loss function in machine learning, just like the well-known negative log-likelihood loss which is the log scoring rule.

The parametrized model outputs a distribution $q(x)$. The CRPS loss evaluates how good $q(x)$ is explaining the observation $y$. This is a distribution-to-point evaluation, which fits well for machine learning as the ground truth $y$ almost always comes as fixed values.

For processes over time and/or space, we need to estimate the CRPS for every point in time/space separately.

There is work on multi-variate CRPS estimation, but it is not part of this repo.

Implementation

The integral formulation is infeasible to naively evaluate on a computer due to the infinite integration over $x$.

I found Nvidia's implementation of the CRPS for ensemble preductions in $M log(M)$ time inspiring to read.

👉 Please have a look at the documentation to get started.