Handsfree Decision Support: Toward a Non-invasive Human-Computer Interface

Eric Horvitz and Michael Shwe

Abstract:

We describe our efforts to develop and test a prototype handsfree human–computer interface that allows healthcare personnel to access computer-based records and decision support without interfering with their daily clinical interactions and procedures. The methods are based on the integration of a computer-based decision-theoretic diagnostic system, speaker-independent speech-understanding, and a head-mounted, transmissive personal display that has the ability to embed translucent computer-based data, images, and inference results in a user’s field of view. The handsfree techniques highlight opportunities for user interfaces that can provide real-time and interactive decision support with minimal changes to existing patterns of clinical activity. We review the components of the system and describe the system’s performance and limitations. We describe methods for enhancing speech understanding for noisy environments by coupling the output of the Bayesian inference procedures with user models to provide methods that dynamically influence the language models used in speech recognition. Finally, we present several areas of ongoing and future research and development.

Keywords: Wearable computing, head-mounted display, decision support for trauma care, wearable Bayesian inference, speech recognition in medicine, time-critical decision support.

In: E. Horvitz and M. Shwe, Handsfree Decision Support: Toward a Non-invasive Human-Computer Interface, Nineteenth Annual Symposium on Computer Applications in Medical Care. Toward Cost-Effective Clinical Computing, November 1995.

Trauma-care modeling effort (model employed in handsfree system): E. Horvitz and A. Seiver. Time-Critical Action: Representations and Application. Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, August 1997.

Author email: horvitz@microsoft.com

AI-powered handsfree trauma decision-support system (1992). Trauma care system (with Eric Horvitz playing the role of paramedic and Jim White starring as an injured cyclist). The prototyping effort focused on the combination of speech recognition, a Bayesian-network--based trauma-care diagnostic system that reasoned about the criticality of injuries from observations, and that continues to apply value-of-information based on the current informational context, to recommend the next best patient findings to evaluate. The prototype was implemented on an early portable PC that was attached to the caregiver's belt.