smartXR Lab

Mon, 01 Jan 0001 00:00:00 +0000

The smartXR Lab equips our research for patient care. We have the following devices:

Structured Light 3D Scanners (Artec Leo and AutoScan Inspec)
3D Printer (Creality SLA printer)
Mixed/Augmented Reality Glasses (HTC VIVE Pro 1 & 2, Varjo XR-3, Meta Quest 3, Microsoft HoloLens 2, and Apple Vision Pro)
Electromagnetic Tracking Systems (NDI Aurora and NDI 3D Guidance trakSTAR)
Optical Tracking Systems (OptiTrack with PrimeX41 cameras and NDI Polaris Vega)
Ultrasound Scanner (GE Healthcare Logiq E10)
Phantoms (Karstadt models and cranio-maxillofacial surgical phantom)

The lab was presented during the symposium of the Zentrum für virtuelle und erweiterte Realität in der Medizin (ZvRM). Watch the presentation on YouTube.

Structured Light 3D Scanners

Our structured-light scanners are the Artec Leo and the AutoScan Inspec. The Artec Leo is a handheld scanner, easy to use and real-time capable, that quickly captures detailed 3D models with high precision. It is well suited for scanning large objects like limbs or even entire rooms.

The AutoScan Inspec, on the other hand, is a tabletop scanner designed for industrial quality control and inspection. Its high-precision scans allow accurate measurement and analysis of complex and small surgical instruments, such as tweezers, scissors, or blades.

Both scanners use structured light technology, making them safe to use on human subjects. Additionally, we provide 3D scanning spray for highly reflective and shiny objects to optimize surface scan quality.

From left to right: AutoScan Inspec, Artec Leo and 3D scanning spray from AESUB

3D Printer

Our resin printer is a stereolithography (SLA) printer from Creality that prints precise, detailed 3D models.

Stereolithography is an additive manufacturing process that uses UV light to selectively harden the resin layer by layer. It can print parts with small features, tight tolerance requirements, and smooth surface finishes. We use the printer to make replicas of our 3D structured-light-scanned objects, CTA/MRI-based surface models, manually created cutting guides or AI-generated patient-specific implants, accurately and within days. For even more rapid prototyping we plan to permanently install an FDM printer as well.

Creality 3D printer, its basin and print plate

Mixed and Augmented Reality Glasses

Our smartXR Lab provides a range of mixed reality devices, each with its own capabilities, with a focus on multi-user applications. Mixed reality entails the fusion of the real and virtual worlds, achieved through two approaches: immersive optical-pass-through (mixed reality) and optical-see-through (augmented reality). By combining real and virtual elements, mixed reality goes beyond the capabilities of virtual reality.

Our lab features multiple HTC VIVE Pro 1 and 2, which use base station infrared scanners for user tracking, while the head-mounted display (HMD) captures the surrounding environment. These devices allow for superimposing filmed and registered reality onto the virtual world. The high-end Varjo XR-3 mixed reality device does not require a base station and also incorporates hand tracking.

Besides the HTC VIVE we have acquired 16 Meta Quest 3 mixed reality headsets, which are versatile, wireless, and have a great price/performance ratio, making them ideal for development and research in a clinical setting.

Additionally, we have 10 HoloLens 2 (HL2) devices, which offer optical-see-through functionality. With HL2, users experience minimal discomfort typically associated with mixed reality devices, as the real world is still perceptible through their eyes. These wireless devices include hand tracking, simultaneous localization and mapping technology, and an onboard computer.

Our latest additions are two Apple Vision Pro units. With them, we develop for Android, Microsoft and Apple based devices.

From left to right: a HL2 visualizing the insides of the patient, the HL2, HTC VIVE Pro 2 and the Varjo XR-3

Electromagnetic Tracking Systems (EMT)

Our smartXR Lab offers the NDI Aurora and 3D Guidance trakSTAR systems, which generate an electromagnetic field and measure changes within it, providing 3D tracking of devices based on these changes. Both track micro sensors embedded in a wide range of medical instruments (ultrasound probes, endoscopes, catheters, guidewires, and even the tip of a needle) in real time and through complex anatomical structures. When integrated into original equipment manufacturer image-guided surgery or interventional systems, the Aurora links patient image sets to the 3D space by localizing the relative positions and orientations of instruments within the operative field. We plan to combine the devices with our mixed and augmented reality headsets, to use them for assessing stand-alone tracking systems, and to keep our research close to current hospital standards.

Both NDI EMT systems and an overview of trackable probes for research

Optical Tracking Systems (OTS)

For motion capture in mixed reality we use the NDI Polaris Vega and an OptiTrack system with the latest PrimeX41 cameras. These systems track spherical infrared-reflective markers by triangulation between at least two cameras. The NDI Polaris Vega is a portable system with high accuracy. The OptiTrack cameras capture live motion within our smartXR Lab at submillimetre accuracy. This brings real-world motion into medical simulations, interactive applications and motion analysis.

OptiTrack system on the left, on the right a trackable surgical instrument and the NDI Polaris Vega

Ultrasound Scanner

The GE Healthcare Ultrasound Scanner Logiq E10 is the lab’s ultrasound system. Its API will allow integration with our Artificial Intelligence and Mixed Reality solutions, as well as our tracking systems, for interactive visualization and real-time guidance. Equipped with an electromagnetic tracking system for the ultrasound probe, the Logiq E10 delivers precise imaging during procedures. Its limited 3D scanning and CT/MRI fusion capabilities are a foundation we aim to expand in the projects to come, along with needle tracking and further features.

The Logiq E10 with color doppler to visualize blood flow in the portal vein on the left, on the right the electromagnetic field generator at the top and an overview of ultrasound probes on the bottom

Practice Phantoms

Our phantoms are the models on which we develop, train and demonstrate our medical Mixed Reality. With our 3D printers, we can create (miniature) models for our research. More often we use our larger-scale Karstadt models, Ms. and Mr. White, as well as our cranio-maxillofacial surgery phantom from GTSimulators.

Both Mr. White and our surgery phantom have undergone high-dose CT scans, so their imaging data can be used in our Mixed Reality applications. We plan to expand our set of phantoms, particularly for ultrasound applications, and to pair them with our Mixed Reality devices and tracking systems.

Our surgical simulator phantom on the left, in the middle Mr. White with the HL2, on the right an AR-enriched Mr. White

AI-guided Therapies | IKIM MML