xskillscore.mae_test¶
-
xskillscore.
mae_test
(forecasts1, forecasts2, observations=None, dim=[], time_dim='time', alpha=0.05)¶ Returns the Jolliffe and Ebert MAE significance test.
Tests whether forecasts1 and forecasts2 have different mean absolute error (MAE) at significance level alpha. https://www.cawcr.gov.au/projects/verification/CIdiff/FAQ-CIdiff.html
Note
alpha
is the desired significance level and the maximum acceptable risk of falsely rejecting the null-hypothesis. The smaller the value of α the greater the strength of the test. The confidence level of the test is defined as 1 - alpha, and often expressed as a percentage. So for example a significance level of 0.05, is equivalent to a 95% confidence level. Source: NIST/SEMATECH e-Handbook of Statistical Methods. https://www.itl.nist.gov/div898/handbook/prc/section1/prc14.htm- Parameters
forecasts1 (xarray.Dataset or xarray.DataArray) – first forecast to be compared to the observations
forecasts2 (xarray.Dataset or xarray.DataArray) – second forecast to be compared to the observations
observations (xarray.Dataset or xarray.DataArray or None) – observations to be compared to both forecasts. if None, assumes that arguments forecasts1 and forecasts2 are already MAEs. Defaults to None.
time_dim (str) – time dimension of dimension over which to compute the temporal correlation. Defaults to
'time'
.dim (str or list of str) – dimensions to apply MAE to. Cannot contain
time_dim
. Defaults to [].alpha (float) – significance level alpha that forecast1 is different than forecast2.
- Returns
xarray.DataArray or xarray.Dataset – is the difference in MAE significant? boolean returns
xarray.DataArray or xarray.Dataset – Difference in xs.mae reduced by
dim
andtime_dim
xarray.DataArray or xarray.Dataset – half-width of the confidence interval at the significance level
alpha
.
Examples
>>> f1 = xr.DataArray(np.random.normal(size=(30)), ... coords=[('time', np.arange(30))]) >>> f2 = xr.DataArray(np.random.normal(size=(30)), ... coords=[('time', np.arange(30))]) >>> o = xr.DataArray(np.random.normal(size=(30)), ... coords=[('time', np.arange(30))]) >>> significantly_different, diff, hwci = xs.mae_test( ... f1, f2, o, time_dim='time', dim=[], alpha=0.05 ... ) >>> significantly_different <xarray.DataArray ()> array(False) >>> diff <xarray.DataArray ()> array(-0.01919449) >>> hwci <xarray.DataArray ()> array(0.38729387) >>> # absolute magnitude of difference is smaller than half-width of >>> # confidence interval, therefore not significant at level alpha=0.05 >>> # now comparing against an offset f2, the difference in MAE is significant >>> significantly_different, diff, hwci = xs.mae_test( ... f1, f2 + 2., o, time_dim='time', dim=[], alpha=0.05 ... ) >>> significantly_different <xarray.DataArray ()> array(True)
References