Skip to Content

Human Experience and Design

Microsoft Research

Touchless Interaction in Medical Settings

  • Surgeon using system
  • Surgeon using system
  • Surgeons at Addenbrooke's hospital, Cambridge

With advances in medical imaging over the years, surgical procedures have become increasingly reliant on a range of digital imaging systems for navigation, reference, diagnosis and documentation. Interacting with images in these surgical settings offers particular challenges arising from the need to maintain boundaries between sterile and non-sterile aspects of the surgical environment and practices. Traditional input devices such as keyboard, mouse and touch screen surfaces are reliant on physical contact. Such contact-based interaction introduces the possibility for contaminated material to be transferred between the sterile and non-sterile. This constraint creates difficulties for surgical staff who are scrubbed up and are dependent upon others to manipulate images on their behalf. This can create inefficiencies, which in turn can entail potential medical complications. Additionally, it can interfere with the surgeon’s interpretive and analytic use of the images.

The aim of the project then was to explore the use of touchless interaction within surgical settings, allowing images to be viewed, controlled and manipulated without contact by using camera-based gesture recognition technology. In particular, the project sought to understand the challenges of these environments for the design and deployment of such systems, as well as articulate the ways in which these technologies may alter surgical practice. While our primary concerns were with maintaining conditions of asepsis, the use of these touchless gesture-based technologies offer other potential uses. For example, such systems offer interesting possibilities for interacting with emerging 3D imaging technologies. They also enable new possibilities for how the surgeons spatially configure themselves with respect to the various screens within surgical settings by enabling interaction at a distance. Such technologies then also offer potential to re-imagine the spatial environments within which image-based surgery takes place.

  • Surgeon at Guy's and St. Thomas' Hospital, London

Working with two partner hospitals, we developed two systems for manipulating image data. One was in partnership with vascular surgeons at Guy’s and St. Thomas’ hospital in London, and the other with neurosurgeons at Addenbrookes Hospital in Cambridge. Both systems supported surgeons to manipulate 3D renderings in ways which are sensitive to their existing practices and teamwork.

See below for a video of the system we developed for vascular surgeons.

This work was funded by MS Connections in collaboration with the Computer Vision group at MSRC, Kings College London and Lancaster University.  

News Coverage

Trial of "touchless" gaming technology in surgery, Adam Brimelow, BBC News Health (May 31, 2012)

Touchless technology put to test by surgeons (video), Adam Brimelow, BBC News (May 31, 2012)

Kinect imaging lets surgeons keep their focus, MacGregor Campbell, New Scientist, Tech (May 17, 2012) 

برنامج 4 تك - الحلقة - Tech program 4 - Episode 73 - BBCArabicNews (May 11, 2012) [in Arabic]

Interacting without Touching, Inside Microsoft Research (March 8, 2012)

Microsoft's TechFest Trots Out 'What is Now Possible' for Computers, The Seattle Times, Business/Technology (March 7, 2012)

Microsoft Installe Kinect Dans les Salles D'operation, 01net (March 7, 2012) [in French]

Microsoft Shows Off Kinect-Based Projects at TechFest Research Fair, The Tech Journal (March 6, 2012)

Microsoft showcases new Kinect-centric projects at its TechFest Research Fair, ZDNet (March 6, 2012)