Model Evaluation

After obtaining base forecasts or coherent reconciled forecasts, you can use evaluate the forecasting accuracy through pyhts.Hierarchy.accuracy_base(), pyhts.Hierarchy.accuracy(), or pyhts.TemporalHierarchy.accuracy() (for temporal hierarchies).

Hierarchy.accuracy_base(real, pred, hist=None, levels=None, measure=None)

Calculate the forecast accuracy of base forecasts.

Parameters

  • real – real future observations, array-like of shape (forecast_horizon, m)

  • pred – forecast values, array-like of shape (forecast_horizon, n)

  • levels (Union[str, Iterable[str], None]) – which levels. None means all levels.

  • hist – historical time series.

  • measure (Optional[List[str]]) – list of measures, e.g., [‘mase’], [‘mse’, ‘mase’].

Return type

DataFrame

Returns

forecast accuracy of base forecasts.

Hierarchy.accuracy(real, pred, hist=None, levels=None, measure=None)

Calculate the forecast accuracy.

Parameters

  • real – real future observations, array-like of shape (forecast_horizon, m)

  • pred – forecast values, array-like of shape (forecast_horizon, m)

  • levels (Union[str, Iterable[str], None]) – which levels, None means all levels.

  • hist – historical time series.

  • measure (Optional[List[str]]) – list of measures, e.g., [‘mase’], [‘mse’, ‘mase’].

Returns

forecast accuracy of reconciled forecasts.

TemporalHierarchy.accuracy(real, pred, hist=None, measure=None)

function to compute forecast accuracy

Parameters

  • real (ndarray) – univariate time series at forecast periods

  • pred (dict) – dict containing forecasts, either reconciled or base

  • hist (Optional[ndarray]) – univariate historical time series

  • measure – measures

Returns

Assuming ht is a defined hierarchy, model is a fitted HFModel, test is real observations in the forecasting horizon. train is the history bottom time series.

>>> forecasts = model.predict(h=12)
>>> base_forecasts = model.generate_base_forecasts(h=12)
>>> ht.accuracy_base(test, base_forecasts, hist=train, levels=[0], measures=['mase', 'rmse'])
>>> ht.accuracy(test, forecasts, hist=train, levels=[0], measures=['mase', 'rmse'])

Supported forecasting measurements:

pyhts.mase(y_true, y_pred, history, m)
pyhts.rmse(y_true, y_pred, history, m)
pyhts.mse(y_true, y_pred, history, m)
pyhts.mape(y_true, y_pred, history, m)
pyhts.mae(y_true, y_pred, history, m)
pyhts.smape(y_true, y_pred, history, m)