Tune forecaster on single node#
Introduction#
In this guidance, we demonstrate how to tune forecaster on single node. In tuning process, forecaster will find the best hyperparameter combination among userdefined search space, which is a common process if users pursue a forecaster with higher accuracy.
Chronos support forecasting model’s hyperparameter tuning in 2 sepaerated APIs (i.e. Forecaster.tune
and AutoTSEstimator
) for users with different demands:




Single Node 
✓ 
✓ 
Cluster 
X 
✓ 
Performanceawared Tuning 
✓ 
X 
Feature Selection 
X 
✓ 
Customized Model 
X 
✓ 
Forecaster.tune
provides easier and more strightforward API for users who are familiar with Chronos forecasters, it is recommened to try this method first.
We will take AutoformerForecaster
and nyc_taxi dataset as an example in this guide.
Setup#
Before we begin, we need to install chronos if it isn’t already available, we choose to use pytorch as deep learning backend.
[ ]:
!pip install pre upgrade bigdlchronos[pytorch,automl]
!pip uninstall y torchtext # uninstall torchtext to avoid version conflict
Data preparation#
First, we load the nyc taxi dataset.
Currently, tune func only support Numpy Ndarray input.
[ ]:
from sklearn.preprocessing import StandardScaler
from bigdl.chronos.data import get_public_dataset
def get_tsdata():
name = 'nyc_taxi'
tsdata_train, tsdata_valid, _ = get_public_dataset(name)
stand_scaler = StandardScaler()
for tsdata in [tsdata_train, tsdata_valid]:
tsdata.impute(mode="linear")\
.scale(stand_scaler, fit=(tsdata is tsdata_train))
return tsdata_train, tsdata_valid
tsdata_train, tsdata_valid = get_tsdata()
input_feature_num = 1
output_feature_num = 1
lookback = 20
horizon = 1
label_len = 10
train_data = tsdata_train.roll(lookback=lookback, horizon=horizon, label_len=label_len, time_enc=True).to_numpy()
val_data = tsdata_valid.roll(lookback=lookback, horizon=horizon, label_len=label_len,time_enc=True).to_numpy()
train_data and val_data is compose of (x, y, x_enc, y_enc) as we set time_enc=True
which is only necessary for Autoformer.
Tuning#
The first step of tuning a forecaster is to define forecaster with space parameters.
There are several common space choices:
space.Categorical
: search space for hyperparameters which are categorical, e.g. a = space.Categorical(‘a’, ‘b’, ‘c’, ‘d’)
space.Real
: search space for numeric hyperparameter that takes continuous values, e.g. learning_rate = space.Real(0.01, 0.1, log=True)
space.Int
: search space for numeric hyperparameter that takes integer values, e.g. range = space.Int(0, 100)
How to change these hyperparameters might be tricky and highly based on experience, but lr, d_model, d_ff and layers or similar parameters usually has a great impact on performance.
[ ]:
import bigdl.nano.automl.hpo.space as space
from bigdl.chronos.forecaster.autoformer_forecaster import AutoformerForecaster
autoformer = AutoformerForecaster(input_feature_num=input_feature_num,
output_feature_num=output_feature_num,
past_seq_len=lookback,
future_seq_len=horizon,
label_len=label_len,
seed=1024,
freq='t',
loss="mse",
metrics=['mae', 'mse', 'mape'],
lr = space.Real(0.0001, 0.1, log=True),
d_model=space.Categorical(32, 64, 128, 256),
d_ff=space.Categorical(32, 64, 128, 256),
e_layers=space.Categorical(1,2),
n_head=space.Categorical(1,8))
Then just call tune
on the training data and validation data!
In addition to data, there are three parameters which need to be specified : n_trials, target_metric and direction(or directions for multiobjective HPO).
n_trials
: number of trials to run. The more trials, the longer the running time, the better results.
target_metric
: the target metric to optimize, a string or an instance of torchmetrics.metric.Metric, default to ‘mse’. If you want to try a multiobjective HPO, you need to pass in a list, for example [‘mse’, ‘latency’] in which latency is a builtin metric for performance.
direction
: in which direction to optimize the target metric, “maximize” or “minimize”, default to “minimize”. If you want to try a multiobjective HPO, you need to set direction=None, and specify directions which is a list containing direction for each metric, for example [‘minimize’, ‘minimize’].
there are other two parameters which you may change their default values : epochs and batch_size.
[ ]:
autoformer.tune(train_data, validation_data=val_data,
n_trials=10, target_metric='mse', direction="minimize")
Then, you can see the whole trial history by calling search_summary()
.
[ ]:
autoformer.search_summary()
After tune
, the model parameters of autoformer is initialized according to the best trial parameters. You need to fit the model again.
[ ]:
autoformer.fit(train_data, epochs=4, batch_size=32)
# evaluate on val set
evaluate = autoformer.evaluate(val_data)
print(evaluate)
Save and load(Optional)#
After tuning and fitting, you can save your model by calling save
with a filename.
[ ]:
autoformer.save(checkpoint_file="best.ckpt")
Then, when you need to load the model weights, just call load()
with corresponding filename.
[ ]:
autoformer.load(checkpoint_file="best.ckpt")
Or if there is a new session, just define a new forecaster with six necessary parameters: input_feature_num, output_feature_num, past_seq_len, future_seq_len, label_len, and freq, then load
with corresponding filename.
[ ]:
new_autoformer = AutoformerForecaster(input_feature_num=input_feature_num,
output_feature_num=output_feature_num,
past_seq_len=lookback,
future_seq_len=horizon,
label_len=label_len,
freq='s')
new_autoformer.load(checkpoint_file="best.ckpt")