View the runnable example on GitHub

OpenVINO Inference using Nano API#

You can use OpenVINOModel class in Nano to load an OpenVINO model and run inference. It only takes a few lines.

To run inference on OpenVINO model with Nano, the following dependencies need to be installed first:

[ ]:

# for BigDL-Nano
!pip install --pre --upgrade bigdl-nano # install the nightly-built version
!source bigdl-nano-init

# for OpenVINO
!pip install openvino-dev

📝 Note

We recommend to run the commands above, especially source bigdl-nano-init before jupyter kernel is started, or some of the optimizations may not take effect.

Let’s take a resnet18-xnor-binary-onnx-0001 model pretrained on ImageNet dataset from the Open Model Zoo as an example. First, we download the model using omz_downloader:

[ ]:

!omz_downloader --name resnet18-xnor-binary-onnx-0001 -o ./model

To load the model and run inference, the only change you need to make is to import BigDL-Nano OpenVINOModel, and pass the xml path of model to it:

[2]:

from bigdl.nano.openvino import OpenVINOModel

ov_model = OpenVINOModel("model/intel/resnet18-xnor-binary-onnx-0001/FP16-INT1/resnet18-xnor-binary-onnx-0001.xml")

Then you could call ov_model(x) to run inference.

[ ]:

import numpy as np

x = np.random.randn(1, 3, 224, 224)
# call ov_model directly to run inference
y_hat = ov_model(x)
predictions = y_hat.argmax(axis=1)
print(predictions)