xskillscore.sign_test
- xskillscore.sign_test(forecasts1, forecasts2, observations=None, time_dim='time', dim=[], alpha=0.05, metric=None, orientation='negative')
Returns the Delsole and Tippett sign test over the given time dimension.
The sign test can be applied to a wide class of measures of forecast quality, including ordered (ranked) categorical data. It is independent of distributional assumptions about the forecast errors. This is different than alternative measures like correlation and mean square error, which assume that the metrics were computed from independent samples. However, skill metrics computed over a common period with a common set of observations are not independent. For example, different forecasts tend to bust for the same event. This procedure is equivalent to testing whether a coin is fair based on the frequency of heads. The null hypothesis is that the difference between the median scores is zero.
- Parameters
forecasts1 (xarray.Dataset or xarray.DataArray) – forecasts1 to be compared to observations
forecasts2 (xarray.Dataset or xarray.DataArray) – forecasts2 to be compared to observations
observations (xarray.Dataset or xarray.DataArray or None) – observation to be compared to both forecasts. Only used if
metric
is provided, otherwise it is assumed that both forecasts have already been compared to observations and this input is ignored. Please adjustorientation
accordingly. Defaults to None.time_dim (str) – time dimension of dimension over which to compute the random walk. This dimension is not reduced, unlike in other xskillscore functions. Defaults to
'time'
.dim (str or list of str) – dimensions to apply metric to if
metric
is provided. Cannot containtime_dim
. Ignored ifmetric
is None. Defaults to [].alpha (float) – significance level for random walk.
metric (callable, str, optional) – metric to compare forecast# with observations if
metric
is not None. Ifmetric
is None, assume that forecast# have been compared observations before usingsign_test
. Make sure to adjustorientation
ifmetric
is None. Usemetric=categorical
, if the winning forecast should only be rewarded a point if it exactly equals the observations. Also allows strings to be convered toxskillscore.{metric}
. Defaults to None.orientation (str) – One of [
'positive'
,'negative'
]. Which skill values correspond to better skill? Smaller values ('negative'
) or larger values ('positive'
)? Defaults to'negative'
. Ignored ifmetric== categorical
.
- Returns
xarray.DataArray or xarray.Dataset – boolean whether
forecast1
is significantly different toforecast2
.xarray.DataArray or xarray.Dataset – walk values shows how often
forecast1
is betterforecast2
.xarray.DataArray or xarray.Dataset – confidence boundary for a random walk at significance level
alpha
.
Examples
>>> f1 = xr.DataArray(np.random.normal(size=(30)), ... coords=[('time', np.arange(30))]) >>> f2 = f1 + 2 >>> o = xr.DataArray(np.random.normal(size=(30)), ... coords=[('time', np.arange(30))]) >>> significantly_different, walk, confidence = xs.sign_test( ... f1, f2, o, time_dim='time', metric='mae', orientation='negative' ... ) >>> walk.plot() [<matplotlib.lines.Line2D object at 0x...>] >>> confidence.plot(color='gray') [<matplotlib.lines.Line2D object at 0x...>] >>> (-1 * confidence).plot(color='gray') [<matplotlib.lines.Line2D object at 0x...>] >>> walk <xarray.DataArray (time: 30)> array([ 1, 0, 1, 2, 1, 2, 3, 4, 5, 6, 5, 6, 7, 6, 7, 8, 9, 10, 9, 10, 11, 12, 13, 12, 11, 12, 13, 14, 15, 14]) Coordinates: * time (time) int64 0 1 2 3 4 5 6 7 8 9 ... 20 21 22 23 24 25 26 27 28 29 >>> significantly_different <xarray.DataArray (time: 30)> array([False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]) Coordinates: * time (time) int64 0 1 2 3 4 5 6 7 8 9 ... 20 21 22 23 24 25 26 27 28 29 alpha float64 0.05
References
DelSole, T., & Tippett, M. K. (2016). Forecast Comparison Based on Random Walks. Monthly Weather Review, 144(2), 615–626. doi: 10/f782pf