View the runnable example on GitHub

Save and Load OpenVINO Model in TensorFlow#

This example illustrates how to save and load a TensorFlow Keras model accelerated by OpenVINO. In this example, we use a pretrained EfficientNetB0 model. Then, by calling trace(model, accelerator="openvino"...), we can obtain a model accelarated by OpenVINO method provided by BigDL-Nano for inference. By calling save(model_name, path) , we could save the model to a folder. By calling load(path, model_name), we could load the model from a folder.

First, prepare model. We use an EfficientNetB0 model (model_ft in following code) pretrained on Imagenet dataset in this example.

[ ]:

from tensorflow.keras.applications import EfficientNetB0

model_ft = EfficientNetB0(weights='imagenet')

Accelerate Inference Using OpenVINO

[ ]:

import tensorflow as tf
from bigdl.nano.tf.keras import InferenceOptimizer

ov_model = InferenceOptimizer.trace(model_ft,
                                    accelerator="openvino"
                                    )

x = tf.random.normal(shape=(2, 224, 224, 3))
# use the optimized model here
y_hat = ov_model(x)
predictions = tf.argmax(y_hat, axis=1)
print(predictions)

Save Optimized Model. The saved model files will be saved at “./optimized_model_ov” directory. There are 3 files in optimized_model_ov, users only need to take “.bin” and “.xml” file for further usage:

  • nano_model_meta.yml: meta information of the saved model checkpoint

  • ov_saved_model.bin: contains the weights and biases binary data of model

  • ov_saved_model.xml: model checkpoint for general use, describes model structure

[ ]:

InferenceOptimizer.save(ov_model, "./optimized_model_ov")

Load the Optimized Model

[ ]:

loaded_model = InferenceOptimizer.load("./optimized_model_ov", model_ft)

Inference with the Loaded Model

[ ]:

# use the optimized model here
y_hat_ld = loaded_model(x)
predictions_ld = tf.argmax(y_hat_ld, axis=1)
print(predictions_ld)