Export the OpenVINO model files to disk#
Introduction#
When a forecaster is accelerated by OpenVINO, we can save the OpenVINO model files to disk by calling export_openvino_file
. In this guidance, we demonstrate how to export the OpenVINO model files to disk in detail.
We will take TCNForecaster
and nyc_taxi dataset as an example in this guide.
Setup#
Before we begin, we need to install Chronos if it isnβt already available, we choose to use pytorch as deep learning backend.
[ ]:
!pip install --pre --upgrade bigdl-chronos[pytorch]
# install OpenVINO
!pip install openvino-dev
# fix conflict with google colab
!pip uninstall -y torchtext
πNote
Although Chronos supports inferencing on a cluster, the method to export model files can only be used when forecaster is a non-distributed version.
Only pytorch backend deep learning forecasters support openvino acceleration.
Forecaster preparation#
Before the exporting process, a forecaster should be created and trained. The training process is introduced in the previous guidance Train forcaster on single node in detail, therefore we directly create and train a TCNForecaster
based on the nyc taxi dataset.
Export the OpenVINO model files#
When a trained forecaster is ready and forecaster is a non-distributed version, we provide with export_openvino_file
method to export the OpenVINO model files to disk. The export_openvino_file
method has 2 parameters: dirname
is the location to save the OpenVINO files, and quantized_dirname
is the location to save the quantized OpenVINO model files when you have a quantized forecaster.
[ ]:
from pathlib import Path
# get data for training and testing and validating
train_data, test_data, val_data = get_data()
# get a trained forecaster
forecaster = get_trained_forecaster(train_data)
# quantize the forecaster
forecaster.quantize(calib_data=train_data,
val_data=val_data,
framework="openvino")
# create a directory to save openvino files
dirname = Path("ov_files")
dirname.mkdir(exist_ok=True)
ckpt_name = dirname / "fp32_openvino"
ckpt_name_q = dirname / "int_openvino"
# export the openvino files
forecaster.export_openvino_file(dirname=ckpt_name, quantized_dirname=ckpt_name_q)
πNote
When
export_openvino_file
is called, the forecaster will automatically build an OpenVINO session with default settings. So you can directly call this method without callingpredict_with_openvino
first. But when you want to export quantized openvino model files, you should quantize the forecaster by callingquantize
method first.If you just need to export fp32 openvino files, you could specify
dirname
only and setquantized_dirname
toNone
:forecaster.export_openvino_file(dirname=ckpt_name, quantized_dirname=None)
The files exported will be saved at ov_files
directory.
There are 3 files in each subdirectory:
nano_model_meta.yml
: meta information of the saved model checkpointov_saved_model.bin
: contains the weights and biases binary data of modelov_saved_model.xml
: model checkpoint for general use, describes model structure
You only need to take .bin
and .xml
file for futher usage.