Simple ONNX RT C++ Example for the NPU

A simple C++ program which runs an ONNX model on the CPU and then on the NPU (if the system configuration allows it).

Requirements

To build the example, the following software must be installed:

Ryzen AI 1.4
Cmake
Visual Studio 2022

Instructions

Build the example

conda activate <RAI env>
compile.bat

Run the example

run.bat

Note: it is not necessary to run the example from within the RyzenAI conda environment.

Notes

The CMakeLists.txt copies all the required files (DLLs, XCLBINs) from the RyzenAI installation tree into the folder where the executable is created. With this approach, there is no runtime dependency on the RyzenAI installation tree. Once built, the example can be run on machines without RyzenAI installed.
This example shows that it is possible to run without any dependency on Python.
The example uses the test_model.onnx model from the quicktest folder.
Like the Python quicktest, this example runs the model with random data and doesn't test the output results.
The application uses the VAI-EP session options to specify the XCLBIN, and it explicitly unsets the XLNX_VART_FIRMWARE and XLNX_TARGET_NAME env vars.
The example uses the helper functions in npu_util.cpp to check whether VAI-EP 1.4 can be used on the deployment machine, and to determine the type of NPU present (PHX/HPT or STX/KRK).
The test program only supports models with exactly 1 input node and 1 output node. But the code can easily be extended to support an arbitrary number of nodes if need be.

Sample Output

Sample output of a successful run:

>run.bat

-------------------------------------------------------
Running quicktest on CPU
-------------------------------------------------------
Creating ORT env
Initializing session options
Creating ONNX Session
ONNX model : c:\temp\quicktest_cpp\build\Release\test_model.onnx
  input -1x3x32x32
  output -1x10
Dynamic batch size detected. Setting batch size to 1.
Running the model
-------------------------------------------------------
Test PASSED!
-------------------------------------------------------

-------------------------------------------------------
Performing compatibility check for VitisAI EP 1.4
-------------------------------------------------------
 - NPU Device ID     : 0x1502
 - NPU Device Name   : NPU Compute Accelerator Device
 - NPU Driver Version: 32.0.203.252
Environment compatible for VitisAI EP

-------------------------------------------------------
Running quicktest on NPU
-------------------------------------------------------
Creating ORT env
Initializing session options
Configuring VAI EP
Creating ONNX Session
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250221 12:48:40.532923 76424 vitisai_compile_model.cpp:1046] Vitis AI EP Load ONNX Model Success
I20250221 12:48:40.532923 76424 vitisai_compile_model.cpp:1047] Graph Input Node Name/Shape (1)
I20250221 12:48:40.532923 76424 vitisai_compile_model.cpp:1051]          input : [-1x3x32x32]
I20250221 12:48:40.532923 76424 vitisai_compile_model.cpp:1057] Graph Output Node Name/Shape (1)
I20250221 12:48:40.532923 76424 vitisai_compile_model.cpp:1061]          output : [-1x10]
[Vitis AI EP] No. of Operators :   CPU     2    NPU   18e
[Vitis AI EP] No. of Subgraphs :   CPU     1    NPU     1 Actually running on NPU     1
ONNX model : C:\temp\quicktest_cpp\build\Release\test_model.onnx
  input -1x3x32x32
  output -1x10
Dynamic batch size detected. Setting batch size to 1.
Running the model
-------------------------------------------------------
Test PASSED!
-------------------------------------------------------