Proof of concept for automatic tool inventory using an XR headset (HoloLens 2). The objective is to detect which tools are missing from a case in realtime.

A deep learning object detector (YOLO) is trained with tools arranged inside a toolbox/case. The demo uses DIY tools, while the target application is surgical instrument tray inventory.

A key requirement is offline operation (no internet connection). For fully on-device deployment, the model is exported to ONNX and executed with WinML (Windows Machine Learning). In addition to detecting each tool, the system estimates the tray/case pose (position and orientation) in 3D space. This enables detections to be anchored in the environment and visualized as a 3D overlay for the user, providing an intuitive representation of the inventory status.