Evaluate a forecaster

Introduction

Once we have trained a forecaster, it is time to evaluate the quality of forecasting on data. In this guidance, we demonstrate how to evaluate a forecaster in detail. The evaluate result is calculated by actual value and predicted value.

We will take TCNForecaster and nyc_taxi dataset as an example in this guide.

Setup

Before we begin, we need to install chronos if it isn’t already available, we choose to use pytorch as deep learning backend.

!pip install --pre --upgrade bigdl-chronos[pytorch]
!pip uninstall -y torchtext # uninstall torchtext to avoid version conflict

Forecaster preparation

Before the evaluating process, a forecaster should be created and trained. The training process is introduced in the previous guidance Train forcaster on single node in detail, therefore we directly create and train a TCNForecaster based on the nyc taxi dataset.

📝Note

The metrics for evaluating are specified when forecaster is created. The default metric is “mse”.

Please refer to API documentation for more information on metrics.

Evaluating

When a forecaster is trained, we provide with evaluate to evaluate it on data. The data support following formats:

1. numpy ndarray

2. xshard term

3. pytorch dataloader

4. bigdl.chronos.data.TSDataset (recommended)

The batch_size will not affect evaluate result but will affect resources cost and the default value is 32. When forecaster is non-distributed, aggregating of multiple output values can be defined through multioutput. Besides, quantize can be set to True to use the quantized model to predict.

# get TSDataset for training and testing
tsdata_train, tsdata_test = get_data()
# get a trained forecaster
forecaster = get_trained_forecaster(tsdata_train)

# evaluate using forecaster
forecaster.evaluate(tsdata_test)

📝Note

The evaluate result above is calculated by scaled data.

Evaluating on unscaled data(Optional)

For example, after using scale on TSDataset, the data is scaled. If you need to evaluate on unscaled data, we provide with the following method.

Please refer to API documentation for more information on Evaluator.evaluate.

# get TSDataset for training and testing
tsdata_train, tsdata_test = get_data()
# get a trained forecaster
forecaster = get_trained_forecaster(tsdata_train)
(x, y) = tsdata_test.to_numpy()

from bigdl.chronos.metric import Evaluator
yhat = forecaster.predict(x)
yhat_unscaled = tsdata_test.unscale_numpy(yhat) # or other customized unscale methods
y_unscaled = tsdata_test.unscale_numpy(y) # or other customized unscale methods
Evaluator.evaluate("mse", y_unscaled, yhat_unscaled)