White Paper

Augmented Reality with DICOM Data

Transforming CT and MRI scans into 3D anatomical models and overlaying them onto the real world for immersive medical education, surgical planning, and patient communication.

Innovation DICOM AR
Innovation Category
Apr 2026 Published
Anusruthy M P Lead author

Overview

In this paper, we introduce a method for visualizing and interacting with medical DICOM data using Augmented Reality (AR). By transforming CT and MRI scans into 3D anatomical models and overlaying them onto real-world settings, AR allows for intuitive and immersive exploration of medical structures. This approach enhances spatial understanding, improves medical education, aids in surgical planning, and facilitates effective patient communication. By integrating Unity3D, Vuforia, and 3D Slicer, the system offers a non-contact, real-time, and interactive solution for viewing anatomical data with increased clarity and precision in clinical and educational environments.

Introduction

The proposed method integrates Augmented Reality (AR) with DICOM-based medical imaging data to develop interactive visualization tools for the healthcare industry. Using AR, three-dimensional anatomical models generated from CT scans can be superimposed onto real-world environments, enabling immersive learning, enhanced diagnostics, and better patient engagement. The workflow employs Unity3D, Vuforia, and 3D Slicer to convert DICOM images into 3D models and overlay them using AR on mobile devices or head-mounted displays.

Augmented Reality

Augmented Reality (AR) is a technology that overlays virtual objects—such as 3D models, text, or animations—onto the real world in real time. Unlike Virtual Reality (VR), which creates a fully simulated environment, AR enhances the user's perception of their physical surroundings.

Key components of an AR system include:

  • Real-world environment: The physical space captured by a device’s camera.
  • Virtual content: 3D models, images, or animations to be overlaid.
  • AR device: Smartphones, tablets, headsets, or smart glasses for viewing the augmented scene.
AR has gained significant traction in medical training, surgical simulation, patient education, and the visualization of DICOM data.

Types of Augmented Reality

1. Marker-Based AR

Uses predefined visual markers (images or QR codes) to trigger and align virtual objects.
Marker-based AR demonstration. A printed book cover acts as the marker, and a digital 3D character model is rendered standing on top of it, with labels pointing to the digital 3D object and to the marker.
Marker-Based AR

2. Markerless AR

Also known as SLAM-based AR, it uses device sensors (camera, gyroscope, accelerometer) to track surfaces and positions without markers.
Markerless AR surface detection on an Android phone. The camera view shows a wooden table with a circular placement reticle detected on its surface.
Markerless AR

3. Projection-Based AR

Projects digital content directly onto physical surfaces to enhance real-world objects.
A pair of Google Glass smart glasses on a dark surface, showing the slim head-mounted frame and the prism display module that projects content into the wearer's field of view.
Projection-Based AR

4. Superimposition-Based AR

Overlays virtual 3D models on recognized real-world objects, such as anatomical regions on the human body.
An Android phone held in landscape orientation. A red 3D car model is rendered standing on a stand on the floor in the camera view, with a label pointing to the car 3D object.
Superimposition-Based AR

Goals of the Proposed System

The primary objective is to demonstrate a healthcare AR use case in which anatomical models created from DICOM data are accurately superimposed onto body regions by integrating Unity3D with Vuforia and 3D Slicer.

Technology Stack

The proposed system uses a combination of software tools to process DICOM data, generate 3D anatomical models, and visualize them through Augmented Reality. The following technologies form the core of the workflow:

I. Unity3D

Unity3D is a robust cross-platform game engine for developing 2D, 3D, AR, and VR applications. It supports multiple platforms, including Android, iOS, Windows, macOS, and WebGL. Key features include:
  • C# scripting
  • Prefab-based object management
  • Support for physics, lighting, animation, and UI systems
  • Seamless integration with AR SDKs such as Vuforia, ARKit, and ARCore

Integration of Vuforia

Vuforia is a widely adopted AR SDK that provides robust capabilities for marker tracking, object recognition, and real-time augmented visualization. Steps include:
  • Register on the Vuforia Developer Portal
  • Obtain API keys
  • Import the Vuforia package into Unity
  • Configure AR camera and image targets
  • Assign 3D content to the tracked targets

II. 3D Slicer

3D Slicer is an open-source platform for processing, segmentation, and visualization of DICOM datasets. It enables:
  • Loading CT/MRI DICOM images
  • Segmenting anatomical structures
  • Exporting 3D models (STL/OBJ) for AR use
  • Visualizing complex 3D medical data
These models can then be imported into Unity for real-time AR superimposition.

Development Challenges

Developing AR applications for medical datasets involves several challenges:
  • Understanding AR concepts such as tracking, SLAM, and rendering
  • Platform-specific APIs, e.g., ARKit (iOS), ARCore (Android)
  • Hardware limitations, including camera resolution and device processing power
  • Accuracy of alignment, especially for superimposition on human body parts
  • Handling large 3D models derived from DICOM segmentation

Sample Demonstrations

1. Face-Tracking App (Superimposition)

This demo integrates Unity 3D and 3D Slicer to create an Android application that overlays a 3D skull model (derived from CT data) on the user's face in real time.
Four frames from the face-tracking application, showing a 3D skull model derived from CT data superimposed on a user's face and tracking it from several angles.
Face-Tracking App (Superimposition)
Benefits
  • Accurate anatomical representation
  • Interactive visualization
  • Real-time engagement for medical learners
Applications
  • Medical education
  • Cosmetic surgery planning
  • Forensic reconstruction

2. 3D Spinal Model from CT Scan (Marker-Based AR)

This demo uses marker-based AR to visualize a 3D spinal cord model generated from DICOM data.
A 3D spinal column model generated from CT DICOM data, rendered in augmented reality above a printed marker card held in the user's hand.
3D Spinal Model from CT Scan (Marker-Based AR)
Benefits
  • Anatomically accurate CT-based model
  • Interactive examination (rotate, zoom, dissect)
  • Useful for both patient and clinician education
Applications
  • Pre-operative surgical planning
  • Patient communication
  • Anatomy training

3. Human Body Tracking App (Superimposition)

Using ARKit’s body-tracking capabilities, this demo overlays virtual anatomical elements on the user's body.
Key Features
  • Full-body skeletal tracking
  • Real-time gesture and movement detection
  • Interactive anatomical visualization
Repository used: https://github.com/Unity-Technologies/arfoundation-samples

Version History

Version Date Summary of Changes
1.1 09-December-2025 Updated documentation, refined the Technology Stack section, and corrected AR workflow descriptions.

Conclusion

While augmented reality offers a powerful way to visualize medical data, combining AR with DICOM-based 3D models significantly enhances the accuracy and usefulness of anatomical representation. Simple AR overlays alone may not ensure proper alignment or detail, especially when working with complex structures derived from CT and MRI scans. By integrating AR frameworks such as Unity3D and Vuforia with 3D Slicer’s segmentation and model-generation capabilities, the system becomes more precise, reliable, and clinically meaningful. This multi-layered approach enables a clearer understanding of anatomical structures, supports medical education, improves patient communication, and assists in pre-operative planning. As AR technologies continue to advance, their role in medical visualization and healthcare training will become increasingly valuable, offering practical, effective solutions for transforming medical imaging into interactive, real-world experiences.

Read the original white paper This page carries the full content of the paper. Download the PDF for the original typeset version.
Download PDF

Authors & Contributors

Anusruthy M P Anusruthy M P Senior Software Engineer
Vishak Kurup Vishak Kurup Technical Architect

References

  1. https://github.com/Unity-Technologies/arfoundation-samples

Work with us

Exploring AR for your
imaging or training workflow?

Tell us about your project. We'll tell you honestly whether and how we can help.

Start a Conversation View All White Papers