QAIC execution provider¶

QAIC Execution Provider for ONNX Runtime enables hardware accelerated execution on Qualcomm AIC100 chipset. It leverages AIC compiler and runtime API packaged in apps, platform SDKs.

Setup¶

• Create a json specification file for onnxruntime as shown below

  {
  "base_image": "ubuntu20",
  "applications": ["onnxruntime"],
  "python_version": "py38",
  "sdk": {
      "qaic_apps": "/path/to/apps/sdk.zip",
      "qaic_platform": "/path/to/platform/sdk.zip"
  }
  

  }

• Launch a docker container for an image built with this specification, following instructions in Docker

• In docker container, onnxruntime_qaic build (onnxruntime version 1.13.1 with qaic EP integration) will be available at /opt/qti-aic/integrations/qaic_onnxrt/onnxruntime_qaic.

Configuration Options¶

Parameters supported in model settings yaml file¶

Usage¶

Python¶

Here are few basic commands you can use with ONNX Runtime and QAIC.

Load a model¶

import onnxruntime as ort
provider_options = []
qaic_provider_options = {}
qaic_provider_options['config'] = '/path/to/yaml/file'
qaic_provider_options['device_id'] = aic_device_id
provider_options.append(qaic_provider_options)
session=onnxruntime.InferenceSession('/path/to/onnx/model', sess_options,
                                           providers = providers, provider_options = providers_options)

This will bind your model to AIC100 chip, with qaic exectuion provider.

Perform Inference¶

# Perform inference using OnnxRuntime
results = sess.run(None, {'input_name': input_data})

In the above code replace 'input_name' with name of model input node and input_data with the actual input data.

C++¶

Load a Model¶

#include <onnxruntime_cxx_api.h>
#include <qaic_provider_factory.h>


// Set environment as required
Ort::Env env(ORT_LOGGING_LEVEL_ERROR, "test");
// Initialize session options, create session
Ort::SessionOptions session_options;
session_options.SetIntraOpNumThreads(1);
session_options.SetGraphOptimizationLevel(
    GraphOptimizationLevel::0);
auto s = OrtSessionOptionsAppendExecutionProvider_QAic(
        session_options, "/path/to/yaml/file", aic_device_id);

Ort::Session session(env, "/path/to/onnx/model", session_options);

Perform inference¶

// Run the model

auto output_tensors = session.Run(Ort::RunOptions{nullptr},
input_names.data(), &input_tensor, 1, output_names.data(), 1);

In the above code, replace "/path/to/onnx/model/" to the path for your onnx file. Also ensure data and shape of your input tensor match the requirements of your model.

End-to-end examples¶

Install additional packages in the container for running End-to-end examples

apt-get update
apt-get install -y python-yaml libpng-dev

pip3 install --upgrade pip
pip3 install opencv-python pyyaml scipy

End to end examples (cpp and python) for resnet50 are available at - /opt/qti-aic/integrations/qaic_onnxrt/tests/.

Running the ResNet C++ sample¶

Compile the Sample Resnet C++ test using build_tests.sh script. By default, test is built using libs from onnxruntime_qaic release build. To enable debugging, re-build onnxruntime_qaic project in Debug configuration and run ./build_test.sh with debug flag.

build_tests.sh [--release|--debug]

Run the executable. The commands below set the environment and run the ResNet-50 model with the provided image on QAic or CPU backend. The program outputs the most probable prediction class index for each iteration.

cd build/release
./qaic-onnxrt-resnet50-test -i <path/to/input/png/image>
                            -m  ../../resnet50/resnet50.yaml

Test options

Running the ResNet Python sample¶

Run test_resnet.py at /opt/qti-aic/integrations/qaic_onnxrt/tests/resnet50
python test_resnet.py --model_config ./resnet50/resnet50.yaml
                      --input_file </path/to/png/image>

Test options

Running models with generic QAic EP test¶

test_qaic_ep.py is a generic test runner for compilation, execution on AIC100.

Run test_qaic_ep.py at /opt/qti-aic/integrations/qaic_onnxrt/tests/ -

python test_qaic_ep.py --model_config ./resnet50/resnet50.yaml
                           --input_file_list </path/to/input/list>

Test options

Execution through Onnxruntime test framework¶

• QAic EP is enabled for execution with onnx_test_runner, onnxruntime_perf_test.

• For model directory requirements and comprehensive list of options supported, refer to onnxruntime perf test documentation.

• onnx_test_runner documentation.

• Sample testdata can be downloaded here.

cd /opt/qti-aic/integrations/qaic_onnxrt/onnxruntime_qaic/build/Release

./onnxruntime_perf_test -e qaic -i 'config|/path/to/resnet50.yaml aic_device_id|0' -m times -r 1000 /path/to/model.onnx

./onnx_test_runner -e qaic /path/to/model/dir