metrics
metrics
Metrics for evaluating the difference between a model and an observation.
Functions
| Name | Description |
|---|---|
| r2 | Coefficient of determination (R2) |
| rmse | alias for root_mean_squared_error |
| root_mean_squared_error | Root Mean Squared Error (RMSE) |
| c_rmse | alias for circular root mean squared error |
| c_root_mean_squared_error | Circular root mean squared error |
| urmse | Unbiased Root Mean Squared Error (uRMSE) |
| c_urmse | alias for circular unbiased root mean squared error |
| c_unbiased_root_mean_squared_error | Circular unbiased root mean squared error |
| bias | Bias (mean error) |
| c_bias | Circular bias (mean error) |
| mae | alias for mean_absolute_error |
| mean_absolute_error | Mean Absolute Error (MAE) |
| c_mae | alias for circular mean absolute error |
| c_mean_absolute_error | Circular mean absolute error |
| mape | alias for mean_absolute_percentage_error |
| cc | alias for corrcoef |
| corrcoef | Pearson’s Correlation coefficient (CC) |
| rho | alias for spearmanr |
| spearmanr | Spearman rank correlation coefficient |
| ev | alias for explained_variance |
| explained_variance | EV: Explained variance |
| nse | alias for nash_sutcliffe_efficiency |
| nash_sutcliffe_efficiency | Nash-Sutcliffe Efficiency (NSE) |
| kge | alias for kling_gupta_efficiency |
| kling_gupta_efficiency | Kling-Gupta Efficiency (KGE) |
| max_error | Max (absolute) error |
| c_max_error | Circular max error |
| mef | alias for model_efficiency_factor |
| model_efficiency_factor | Model Efficiency Factor (MEF) |
| si | alias for scatter_index |
| scatter_index | Scatter index (SI) |
| pr | alias for peak_ratio |
| peak_ratio | Peak Ratio |
| willmott | Willmott’s Index of Agreement |
| hit_ratio | Fraction within obs ± acceptable deviation |
| lin_slope | Slope of the regression line. |
r2
metrics.r2(obs, model)Coefficient of determination (R2)
Pronounced ‘R-squared’; the proportion of the variation in the dependent variable that is predictable from the independent variable(s), i.e. the proportion of explained variance.
\[ R^2 = 1 - \frac{\sum_{i=1}^n (model_i - obs_i)^2} {\sum_{i=1}^n (obs_i - \overline {obs})^2} \]
Range: \((-\infty, 1]\); Best: 1
Notes
r2 = nash_sutcliffe_efficiency(nse)
Examples
>>> obs = np.array([1.0,1.1,1.2,1.3,1.4])
>>> model = np.array([1.09, 1.16, 1.3 , 1.38, 1.49])
>>> r2(obs,model)
np.float64(0.6379999999999998)rmse
metrics.rmse(obs, model, weights=None, unbiased=False)alias for root_mean_squared_error
root_mean_squared_error
metrics.root_mean_squared_error(obs, model, weights=None, unbiased=False)Root Mean Squared Error (RMSE)
\[ res_i = model_i - obs_i \]
\[ RMSE=\sqrt{\frac{1}{n} \sum_{i=1}^n res_i^2} \]
Unbiased version:
\[ res_{u,i} = res_i - \overline {res} \]
\[ uRMSE=\sqrt{\frac{1}{n} \sum_{i=1}^n res_{u,i}^2} \]
Range: \([0, \infty)\); Best: 0
c_rmse
metrics.c_rmse(obs, model, weights=None)alias for circular root mean squared error
c_root_mean_squared_error
metrics.c_root_mean_squared_error(obs, model, weights=None)Circular root mean squared error
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| obs | ArrayLike | Observation in degrees (0, 360) | required |
| model | ArrayLike | Model in degrees (0, 360) | required |
| weights | ArrayLike | Weights, by default None | None |
Notes
Range: [0, 180]; Best: 0
Returns
| Name | Type | Description |
|---|---|---|
| float | Circular root mean squared error |
urmse
metrics.urmse(obs, model, weights=None)Unbiased Root Mean Squared Error (uRMSE)
\[ res_i = model_i - obs_i \]
\[ res_{u,i} = res_i - \overline {res} \]
\[ uRMSE = \sqrt{\frac{1}{n} \sum_{i=1}^n res_{u,i}^2} \]
Range: \([0, \infty)\); Best: 0
See Also
root_mean_squared_error
c_urmse
metrics.c_urmse(obs, model, weights=None)alias for circular unbiased root mean squared error
c_unbiased_root_mean_squared_error
metrics.c_unbiased_root_mean_squared_error(obs, model, weights=None)Circular unbiased root mean squared error
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| obs | ArrayLike | Observation in degrees (0, 360) | required |
| model | ArrayLike | Model in degrees (0, 360) | required |
| weights | ArrayLike | Weights, by default None | None |
Notes
Range: [0, 180]; Best: 0
Returns
| Name | Type | Description |
|---|---|---|
| float | Circular unbiased root mean squared error |
bias
metrics.bias(obs, model)Bias (mean error)
\[ bias=\frac{1}{n}\sum_{i=1}^n (model_i - obs_i) \]
Range: \((-\infty, \infty)\); Best: 0
c_bias
metrics.c_bias(obs, model)Circular bias (mean error)
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| obs | ArrayLike | Observation in degrees (0, 360) | required |
| model | ArrayLike | Model in degrees (0, 360) | required |
Notes
Range: \([-180., 180.]\); Best: 0.
Returns
| Name | Type | Description |
|---|---|---|
| float | Circular bias |
Examples
>>> obs = np.array([10., 355., 170.])
>>> mod = np.array([20., 5., -180.])
>>> c_bias(obs, mod)
np.float64(10.0)mae
metrics.mae(obs, model, weights=None)alias for mean_absolute_error
mean_absolute_error
metrics.mean_absolute_error(obs, model, weights=None)Mean Absolute Error (MAE)
\[ MAE=\frac{1}{n}\sum_{i=1}^n|model_i - obs_i| \]
Range: \([0, \infty)\); Best: 0
c_mae
metrics.c_mae(obs, model, weights=None)alias for circular mean absolute error
c_mean_absolute_error
metrics.c_mean_absolute_error(obs, model, weights=None)Circular mean absolute error
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| obs | ArrayLike | Observation in degrees (0, 360) | required |
| model | ArrayLike | Model in degrees (0, 360) | required |
| weights | ArrayLike | Weights, by default None | None |
Notes
Range: [0, 180]; Best: 0
Returns
| Name | Type | Description |
|---|---|---|
| float | Circular mean absolute error |
mape
metrics.mape(obs, model)alias for mean_absolute_percentage_error
cc
metrics.cc(obs, model, weights=None)alias for corrcoef
corrcoef
metrics.corrcoef(obs, model, weights=None)Pearson’s Correlation coefficient (CC)
\[ CC = \frac{\sum_{i=1}^n (model_i - \overline{model})(obs_i - \overline{obs}) } {\sqrt{\sum_{i=1}^n (model_i - \overline{model})^2} \sqrt{\sum_{i=1}^n (obs_i - \overline{obs})^2} } \]
Range: [-1, 1]; Best: 1
See Also
spearmanr np.corrcoef
rho
metrics.rho(obs, model)alias for spearmanr
spearmanr
metrics.spearmanr(obs, model)Spearman rank correlation coefficient
The rank correlation coefficient is similar to the Pearson correlation coefficient but applied to ranked quantities and is useful to quantify a monotonous relationship
\[ \rho = \frac{\sum_{i=1}^n (rmodel_i - \overline{rmodel})(robs_i - \overline{robs}) } {\sqrt{\sum_{i=1}^n (rmodel_i - \overline{rmodel})^2} \sqrt{\sum_{i=1}^n (robs_i - \overline{robs})^2} } \]
Range: [-1, 1]; Best: 1
Examples
>>> obs = np.linspace(-20, 20, 100)
>>> mod = np.tanh(obs)
>>> rho(obs, mod)
np.float64(0.9999759973116955)
>>> spearmanr(obs, mod)
np.float64(0.9999759973116955)See Also
corrcoef
ev
metrics.ev(obs, model)alias for explained_variance
explained_variance
metrics.explained_variance(obs, model)EV: Explained variance
EV is the explained variance and measures the proportion [0 - 1] to which the model accounts for the variation (dispersion) of the observations.
In cases with no bias, EV is equal to r2
\[ \frac{ \sum_{i=1}^n (obs_i - \overline{obs})^2 - \sum_{i=1}^n \left( (obs_i - \overline{obs}) - (model_i - \overline{model}) \right)^2}{\sum_{i=1}^n (obs_i - \overline{obs})^2} \]
Range: [0, 1]; Best: 1
See Also
r2
nse
metrics.nse(obs, model)alias for nash_sutcliffe_efficiency
nash_sutcliffe_efficiency
metrics.nash_sutcliffe_efficiency(obs, model)Nash-Sutcliffe Efficiency (NSE)
\[ NSE = 1 - \frac {\sum _{i=1}^{n}\left(model_{i} - obs_{i}\right)^{2}} {\sum_{i=1}^{n}\left(obs_{i} - {\overline{obs}}\right)^{2}} \]
Range: \((-\infty, 1]\); Best: 1
Notes
r2 = nash_sutcliffe_efficiency(nse)
Nash, J. E.; Sutcliffe, J. V. (1970). “River flow forecasting through conceptual models part I — A discussion of principles”. Journal of Hydrology. 10 (3): 282–290. https://doi.org/10.1016/0022-1694(70)90255-6
kge
metrics.kge(obs, model)alias for kling_gupta_efficiency
kling_gupta_efficiency
metrics.kling_gupta_efficiency(obs, model)Kling-Gupta Efficiency (KGE)
\[ KGE = 1 - \sqrt{(r-1)^2 + \left(\frac{\sigma_{mod}}{\sigma_{obs}} - 1\right)^2 + \left(\frac{\mu_{mod}}{\mu_{obs}} - 1\right)^2 } \]
where \(r\) is the pearson correlation coefficient, \(\mu_{obs},\mu_{mod}\) and \(\sigma_{obs},\sigma_{mod}\) is the mean and standard deviation of observations and model.
Range: \((-\infty, 1]\); Best: 1
Notes
Gupta, H. V., Kling, H., Yilmaz, K. K. and Martinez, G. F., (2009), Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling, J. Hydrol., 377(1-2), 80-91 https://doi.org/10.1016/j.jhydrol.2009.08.003
Knoben, W. J. M., Freer, J. E., and Woods, R. A. (2019) Technical note: Inherent benchmark or not? Comparing Nash–Sutcliffe and Kling–Gupta efficiency scores, Hydrol. Earth Syst. Sci., 23, 4323-4331 https://doi.org/10.5194/hess-23-4323-2019
max_error
metrics.max_error(obs, model)Max (absolute) error
\[ max_{error} = max(|model_i - obs_i|) \]
Range: \([0, \infty)\); Best: 0
c_max_error
metrics.c_max_error(obs, model)Circular max error
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| obs | ArrayLike | Observation in degrees (0, 360) | required |
| model | ArrayLike | Model in degrees (0, 360) | required |
Notes
Range: \([0, \infty)\); Best: 0
Returns
| Name | Type | Description |
|---|---|---|
| float | Circular max error |
Examples
>>> obs = np.array([10., 350., 10.])
>>> mod = np.array([20., 10., 350.])
>>> c_max_error(obs, mod)
np.float64(20.0)mef
metrics.mef(obs, model)alias for model_efficiency_factor
model_efficiency_factor
metrics.model_efficiency_factor(obs, model)Model Efficiency Factor (MEF)
Scale independent RMSE, standardized by Stdev of observations
\[ MEF = \frac{RMSE}{STDEV}=\frac{\sqrt{\frac{1}{n} \sum_{i=1}^n(model_i - obs_i)^2}} {\sqrt{\frac{1}{n} \sum_{i=1}^n(obs_i - \overline{obs})^2}}=\sqrt{1-NSE} \]
Range: \([0, \infty)\); Best: 0
See Also
nash_sutcliffe_efficiency root_mean_squared_error
si
metrics.si(obs, model)alias for scatter_index
scatter_index
metrics.scatter_index(obs, model)Scatter index (SI)
Which is the same as the unbiased-RMSE normalized by the absolute mean of the observations.
\[ \frac{ \sqrt{ \frac{1}{n} \sum_{i=1}^n \left( (model_i - \overline {model}) - (obs_i - \overline {obs}) \right)^2} } {\frac{1}{n} \sum_{i=1}^n | obs_i | } \]
Range: \([0, \infty)\); Best: 0
pr
metrics.pr(obs, model, inter_event_level=0.7, AAP=2, inter_event_time='36h')alias for peak_ratio
peak_ratio
metrics.peak_ratio(
obs,
model,
inter_event_level=0.7,
AAP=2,
inter_event_time='36h',
)Peak Ratio
PR is the mean of the largest-N individual ratios of identified peaks in the model / identified peaks in the measurements (N number of events defined by AAP). PR is calculated only for the joint-events, ie, events that ocurr simulateneously within a window +/- 0.5*inter_event_time.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| inter_event_level | float | Inter-event level threshold (default: 0.7). | 0.7 |
| AAP | Union[int, float] | Average Annual Peaks (ie, Number of peaks per year, on average). (default: 2) | 2 |
| inter_event_time | str | Maximum time interval between peaks (default: 36 hours). | '36h' |
Notes
\(\frac{\sum_{i=1}^{N_{joint-peaks}} (\frac{Peak_{model_i}}{Peak_{obs_i}} )}{N_{joint-peaks}}\)
Range: \([0, \infty)\); Best: 1.0
willmott
metrics.willmott(obs, model)Willmott’s Index of Agreement
A scaled representation of the predictive accuracy of the model against observations. A value of 1 indicates a perfect match, and 0 indicates no agreement at all.
\[ willmott = 1 - \frac{\frac{1}{n} \sum_{i=1}^n(model_i - obs_i)^2} {\frac{1}{n} \sum_{i=1}^n(|model_i - \overline{obs}| + |obs_i - \overline{obs}|)^2} \]
Range: [0, 1]; Best: 1
Examples
>>> obs = np.array([1.0, 1.1, 1.2, 1.3, 1.4, 1.4, 1.3])
>>> model = np.array([1.02, 1.16, 1.3, 1.38, 1.49, 1.45, 1.32])
>>> willmott(obs, model)
np.float64(0.9501403174479723)Notes
Willmott, C. J. 1981. “On the validation of models”. Physical Geography, 2, 184–194.
hit_ratio
metrics.hit_ratio(obs, model, a=0.1)Fraction within obs ± acceptable deviation
\[ HR = \frac{1}{n}\sum_{i=1}^n I_{|(model_i - obs_i)|} < a \]
Range: [0, 1]; Best: 1
Examples
>>> obs = np.array([1.0, 1.1, 1.2, 1.3, 1.4, 1.4, 1.3])
>>> model = np.array([1.02, 1.16, 1.3, 1.38, 1.49, 1.45, 1.32])
>>> hit_ratio(obs, model, a=0.05)
np.float64(0.2857142857142857)
>>> hit_ratio(obs, model, a=0.1)
np.float64(0.8571428571428571)
>>> hit_ratio(obs, model, a=0.15)
np.float64(1.0)lin_slope
metrics.lin_slope(obs, model, reg_method='ols')Slope of the regression line.
\[ slope = \frac{\sum_{i=1}^n (model_i - \overline {model})(obs_i - \overline {obs})} {\sum_{i=1}^n (obs_i - \overline {obs})^2} \]
Range: \((-\infty, \infty )\); Best: 1