View the runnable example on GitHub
Accelerate TensorFlow Inference using OpenVINO#
You can use InferenceOptimizer.trace(..., accelerator='openvino')
API to enable the OpenVINO acceleration for TensorFlow inference. It only takes a few lines.
Let’s take an EfficientNetB0 model pretrained on ImageNet dataset as an example. First, we load the model:
[ ]:
from tensorflow.keras.applications import EfficientNetB0
model = EfficientNetB0(weights='imagenet')
To enable OpenVINO acceleration for your TensorFlow inference, the only change you need to made is to import BigDL-Nano InferenceOptimizer
, and trace your TensorFlow model to convert it into an OpenVINO accelerated module for inference:
[ ]:
import tensorflow as tf
from bigdl.nano.tf.keras import InferenceOptimizer
ov_model = InferenceOptimizer.trace(model,
accelerator="openvino",
input_spec=tf.TensorSpec(shape=(None, 224, 224, 3))
)
📝 Note
input_spec
is the parameter for OpenVINO accelerator to know the shape of the model input. Here, we specify our test dataset to consist of images with \(224 \times 224\) pixels and \(3\) input channels.Please refer to API documentation for more information on
InferenceOptimizer.trace
.
You could then do the normal inference steps with the model optimized by OpenVINO:
[ ]:
x = tf.random.normal(shape=(2, 224, 224, 3))
# use the optimized model here
y_hat = ov_model(x)
predictions = tf.argmax(y_hat, axis=1)
print(predictions)
📚 Related Readings