In this article, I’m going to share my experience building an interruptible voice runtime on an NVIDIA Jetson Orin Nano. The goal of this project was to create a system that can handle voice commands in real time, with the ability to interrupt ongoing processes based on user input. The user presses a button and says a voice command like “Turn right”, voice activity detection (VAD) detects the speech, Whisper transcribes the utterance on the GPU, and a JSON command with a latency breakdown is emitted. I use a closed grammar with a small set of commands, so the transcribed text is matched against the grammar to determine the intent.

In this article I am going to use the MobileNetV2 deep learning model from the ONNX model zoo [1] to benchmark the inference performance on NVIDIA Jetson Orin Nano Developer Kit. The main focus is to compare the performance of FP32, TF32, FP16 and INT8 precision modes in terms of latency, throughput, accuracy and power consumption.

The goal is to provide insights into efficient model deployment on edge devices, and to share lessons learned from the benchmarking process. The article is intended for Edge AI enthusiasts, engineers, and researchers who are interested in optimizing deep learning models for edge devices.