A whole workflow in production environment after my forecaster is developed#

In model developing process, TSDataset is used to preprocess(including feature engineering, data sampling, scaling, …) the raw data the postprocess the predicted result(majorly unscaling). This post provides a way by which users could replay the preprocessing and postprocessing in production environment(e.g. model serving).

In this guide, we will

  1. Train a TCNForecaster with nyc_taxi dataset and export the model in onnx type and save the scaler.

  2. Show users how to replay the preprocessing and postprocessing in production environment.

  3. Evaluate the performance of preprocessing and postprocessing

  4. More tips about this topic.

Forecaster developing#

First let’s prepare the data. We will manually download the data to show the details.

[ ]:

# run following
!wget https://raw.githubusercontent.com/numenta/NAB/v1.0/data/realKnownCause/nyc_taxi.csv

Then we may load the data to pandas dataframe and carry out preprocessing through TSDataset. You could refer to

for details.

[ ]:

from sklearn.preprocessing import StandardScaler
from bigdl.chronos.data import TSDataset, get_public_dataset
import pandas as pd

# load the data to pandas dataframe
df = pd.read_csv("nyc_taxi.csv", parse_dates=["timestamp"])

# use nyc_taxi public dataset
train_data, _, test_data = TSDataset.from_pandas(df,
                                                 dt_col="timestamp",
                                                 target_col="value",
                                                 repair=False,
                                                 with_split=True,
                                                 test_ratio=0.1)

# create a scaler for data scaling
scaler = StandardScaler()

# preprocess(generate datetime feature, scale and roll samping)
for data in [train_data, test_data]:
    data.gen_dt_feature(features=["WEEKDAY", "HOUR", "MINUTES"])\
        .scale(scaler, fit=(data is train_data))\
        .roll(lookback=48, horizon=24)

Developing a forecaster on this data is quite easy. You may refer to other how-to guide for more detailed information.

[ ]:

from bigdl.chronos.forecaster import TCNForecaster  # TCN is algorithm name

# create a forecaster
forecaster = TCNForecaster.from_tsdataset(train_data)

# train the forecaster
forecaster.fit(train_data)

Forecaster API is for quick iteration during the development, when a forecaster is developed with satisfying accuracy and performance, users may prefer to export the model to formats that are easier to deploy in production environment (e.g., ONNX, openVINO, torchscript, …). We choose to use ONNX here as an example. You may refer to other how to guides for more details.

[ ]:

# save the forecaster in onnx type
forecaster.export_onnx_file(dirname="nyc_taxi_onnx_model", quantized_dirname=None)

If a scaler is used during the preprocessing process, then it should be saved for production environment.

[ ]:

import pickle

# save the scaler
# There are many ways, we use pickle here
with open('scaler.pkl','wb') as f:
    pickle.dump(scaler, f)

In production environment#

When you have successfully developed a model your are really satisfied with, then it’s time to deploy your model. With the highly optimized model you just exported, the whole process could be suprising easy.

There are 2 possibilities for deployment: 1. You will use the model in a monolithic application, where the input, model inference and output is located on the same server. 2. You will use a server-client model, where you may want to adopt some model serving tools (e.g., torchserve, OpenVINO server, Triton, …). This means that users will separate model inference with other workload.

For the first choice, you may directly call some inference engine API (e.g., onnxruntime, OpenVINO, …) in your application. For the second choice, this may depends on different model serving tools’ procedure. We have an example to serve a forecaster on torchserve.

For both choicies, it’s common to have a single sample data to come, here we use the last sample of nyc taxi dataset as example

[ ]:

# generate data to predict in a local csv file
_, _, test_data = get_public_dataset("nyc_taxi")
test_data.df[-48:].to_csv("inference_data.csv")

Then we could load all the items we need, this includes the scaler and onnx file we just dumped, and the data to be inferenced.

[ ]:

import onnxruntime

# load the scaler
with open('scaler.pkl', 'rb') as f:
    scaler = pickle.load(f)

# load the onnx file to onnxruntime
session = onnxruntime.InferenceSession("nyc_taxi_onnx_model/onnx_saved_model.onnx")

# load the data to be predicted
df = pd.read_csv("inference_data.csv", parse_dates=["timestamp"])

The preprocess process should be the same as how you processed your data when developing the forecaster, except

📝Note

There are 2 exceptions here that should be followed carefully

  • Please make sure to set deploy_mode=True when creating TSDataset through TSDataset.from_pandas, TSDataset.from_parquet and TSDataset.from_prometheus, which will reduce data processing latency and set necessary parameters for deployment.

  • For scale, please make sure using the scaler you dumped and loaded back.

[ ]:

def preprocess_during_deployment(df, scaler):
    tsdata = TSDataset.from_pandas(df,
                                   dt_col="timestamp",
                                   target_col="value",
                                   repair=False,
                                   deploy_mode=True)
    tsdata.gen_dt_feature(features=["WEEKDAY", "HOUR", "MINUTES"])\
          .scale(scaler)\
          .roll(lookback=48, horizon=24)
    data = tsdata.to_numpy()
    return tsdata, data

For postprocessing, if scaler is used, then unscale_numpy is needed.

[ ]:

def postprocess_during_deployment(data, tsdata):
    return tsdata.unscale_numpy(data)

They users could predict the data easily by a clear process.

[ ]:

tsdata, data = preprocess_during_deployment(df, scaler)
data = session.run(None, {'x': data})[0]
processed_data = postprocess_during_deployment(data, tsdata)