Improving the view of the world; law enforcement and augmented reality technology.

Today's rapidly changing society is driven by higher and higher levels of technological advancement. For good or bad, technology brings change. The events of September 11, 2001, have served to fuel the debate concerning the role that technology should play in people's lives, particularly future battles against terrorism in a free society.

[ILLUSTRATION OMITTED]

The rate of change itself, however, is changing, significantly compressing the time that it takes for new technologies to take hold and alter the way in which people live and interact with each other. Rapidly emerging technologies also afford criminals and terrorists new opportunities for exploiting, disrupting, and harming society. While human intelligence and traditional policing methodologies continue as important aspects of law enforcement, high-tech systems are becoming a principle, and undeniably necessary, means of maintaining domestic security. To employ emerging technologies effectively and thwart their illicit use by criminals and terrorists, law enforcement officers need to understand fully the state of current technological capabilities and how those capabilities will change in the coming decades.

One of these emerging and powerful technologies, augmented reality (AR), uses wearable components to overlay virtual (computer-generated) information onto individuals' real-world view or into their real-world experiences in a way that improves and enhances their abilities to accomplish a wide variety of tasks and missions. Still in the early stages of research and development, AR combines the real and the virtual, displaying information in real time, in a way that enhances the individual abilities of people operating in the real world.

Each weekend, millions of auto racing and football fans view one popular type of AR on television. Broadcasters display driver and speed information tagged to race cars hurtling around a track, as well as super-impose yellow first-down lines on a football field, to help fans better understand the real-time action of the events. Another popular AR application is the heads-up display (HUD), common in military fighter aircraft. While not a wearable application, the HUD superimposes aerodynamic and other aircraft system data onto the forward view of the cockpit canopy, allowing the pilot to see critical information about the flight situation while maintaining focus on the enemy target.

[ILLUSTRATION OMITTED]

By using virtual graphics, three-dimensional maps, textual annotations, auditory information, and haptic (touch) sensations in a coordinated real-time presentation, AR brings together a variety of technologies to display information to individuals in a way that instantly applies to a given task or situation. The use of AR technology may positively influence any situation enhanced or helped by the visual, audible, or haptic display of information not available or detectable by normal human senses. (1)

A fully interactive AR system may derive information from a multitude of sources. Data can be transmitted wirelessly from a computer network, accessed from the wearable computer carried by the AR user, acquired from embedded devices within a surrounding intelligent environment, and obtained from wearable sensor arrays scanning the immediate or visible location of the user. Linking individual users together on a wireless network could allow them to view each other's location and status to coordinate activities and take supportive action when appropriate.

Augmenting reality in this way also allows for some uniquely tailored applications in specific circumstances. Users of AR systems can block out certain aspects of the real world that might either detract from the task at hand, a process known as diminished reality, or they can filter out confusing information and see things not normally visible to the unaided eye, a freeze-frame process called mediated reality.

THE AR SYSTEM

Fundamentally, an AR system consists of a wearable computer, a head-mounted display (HMD), and tracking and sensing devices, along with advanced software and virtual three-dimensional-rendering applications. Depending on the intended use, the basic system could incorporate a wide range of other components adapted for specific operational functions. (2)

AR is a mobile technology designed to improve situational awareness and speed human decision making. To accomplish this, the human-machine interface must streamline the process of input and output so that the user can maintain focus on the task at hand with minimal distraction. Traditional methods of computer input control (keyboard, mouse, and trackball) and output receptors (monitors and flat-panel displays) prove difficult, if not impossible, to use in a dynamic mobile environment. AR system development revolves around the use of modern interface devices, features, and methodologies that allow the user to concentrate on real-world tasks while seamlessly enhancing the real-world experience with useful data.

The coherent integration of supplemental visual data to the user via a see-through HMD constitutes the principle means of achieving an enhanced human-machine interface. High-quality HMDs for AR use are still under development, and several different approaches to accomplishing the display of visual data exist. An optical see-through HMD is a semitransparent display that allows the user to see the real world directly, projecting visual data on the inside of the screen in front of the user's eyes. A video see-through HMD has an opaque display and uses head-mounted video cameras to provide the real-world view, incorporating both the video and virtual data onto the opaque screen. Each approach has benefits and drawbacks for law enforcement use, depending on the application and operational environment where used.

Another option for both optical and video see-through HMDs is a one-display configuration that mounts a single display over one eye, leaving the other eye completely unobstructed. The latest development in HMD technology, the virtual retinal display (VRD), uses low-powered laser light projected directly on the retina to display information. Providing the benefits of the optical see-through display, the VRD can exhibit high-resolution graphics, even in bright sunlight.

To be effective, such devices must overlay textual and graphical data precisely within the user's field of view, correctly associated with relative real-world objects. For virtual images to be correctly and accurately overlaid or "registered" on the HMD, the AR system must have the capability to continuously track the user's head movements, exact position, viewing direction, and real-world orientation. (3)

AR systems employ a variety of mechanisms to accomplish this tracking, such as mechanical, magnetic, acoustic, inertial, and optical sensors or a hybrid combination of several of these technologies. Of particular concern to law enforcement is the need for the tracking system to function accurately outdoors in open terrain, as well as indoors. Outdoor tracking proves a much more difficult problem for AR systems and depends heavily on GPS (the U.S. Department of Defense's Global Positioning System), dead-reckoning techniques, (4) compasses, and gyroscopes to achieve accurate image registration in unprepared environments. (5) Sensing the entire environment in real time using a hybrid tracking system to accurately determine the location of the user, as well as natural and manmade terrain features, is a necessity and a hurdle that technology has yet to fully overcome.

Voice-activation, speech-recognition, and text-to-speech technologies take advantage of the most natural form of human communication. The use of sonification--the translation of nonaudible data into an acoustic signal--can facilitate rapid interpretation and comprehension of data. Haptic interfaces allow users to feel various data components that lend themselves to touch or heat sensations. Augmented cognition programs using artificial intelligence could provide a comprehensive situational context to the user, combining location, the presence of other people and objects, actions presently occurring, user goals, and other situational components to help determine the best course of action.

[ILLUSTRATION OMITTED]

While several current research programs in the United States are examining AR technology, until recently, none of them had explored the application of AR to policing. The Futures Working Group currently is working with the Naval Research Lab (NRL) in Washington, D.C., to examine AR applications for policing. Based upon NRL's battlefield augmented reality system for the U.S. military, a law enforcement augmented reality system test-bed project will look at potential uses of AR in policing and explore the various components and configurations to best serve the needs of the law enforcement community.

LAW ENFORCEMENT APPLICATIONS

Preliminary research indicates a number of likely law enforcement applications where AR technology could advance officer performance well beyond current levels. The true strength of AR will rest with its ability to improve the situational awareness of an individual officer or networked team of officers by employing a number of related technologies, alone or in combination, to accomplish law enforcement-related tasks and missions. Devices and technologies that could be incorporated into a law enforcement AR system include night and thermal imagers; biometric, chemical/biological, and explosive sensors; identification and tracking of moving and stationary objects; and real-time speech translation and text-to-speech systems.

Patrol duties, special weapons and tactics (SWAT) operations, investigative situations, training efforts, and management issues provide merely a partial list of potential uses. Only the knowledge and creativity of law enforcement officers will limit the actual number and type of eventual applications for AR technology.

Patrol Duties

Uniformed patrol officers will have many potential uses for AR technology over a wide range of scenarios. Among them are a variety of applications to facilitate interaction with a diverse population and to detect criminals and crimes in progress.

* Real-time language translation, along with data on cultural customs and traditions, could strengthen police ties with minority groups and improve information flow to police.

* The immediate display of real-time intelligence about crimes and criminals in the patrol area could foster appropriate patrol actions to reduce crime.

* Facial, voiceprint, and other biometric recognition data of known criminals would allow officers to identify wanted subjects merely by observing people on the street.

* Integration of chemical, biological, and explosive sensors could notify officers immediately of any local contamination and recommend appropriate protective measures that they could take for themselves and the public.

* The accessibility of scalable, three-dimensional maps (complete with building floor plans, sewer system schematics, public utility informations, and public transportation routes) could improve situational awareness and response to problems.

* The availability of patrol car operator data and regional traffic management information could make driving safer and more efficient, especially in pursuit and rapid-response situations.

SWAT Operations

AR could make SWAT operations safer and more effective. Basically, it could improve situational awareness during dynamic and dangerous incidents, enhance communication between team members, and provide better coordination with command personnel.

* Advanced audio could moderate the audible intensity of gunshots and explosions, but provide superior hearing capabilities over long distances.

* Advanced optics could provide zoom, thermal, and infrared imaging for the location and apprehension of fleeing criminals.

* Identification friend or foe (IFF) technology, worn by every law enforcement officer, could reduce or eliminate friendly fire casualties by visually, audibly, or haptically highlighting fellow officers.

* The human-machine interface could expand to include robots and other mechanical devices that could extend human capabilities to remote locations through physical, virtual, and haptic interfaces. (6)

Investigative Situations

AR could enhance investigators' abilities to gather information, follow leads, and visualize large amounts of data. In turn, it would lead to an increase in the number of crimes solved and to the quicker identification and capture of dangerous criminals and terrorists.

* Speaker-recognition technology, under development, will give investigators the ability to accurately match voices against known criminals. (7)

* With advanced optics, investigators could lip-read from great distances in situations where listening devices would prove impractical. (8)

* Thermal imaging might improve interrogations by helping to indicate the truthfulness of subjects' statements. (9)

* AR video, audio, and sensing devices used to visualize blood patterns, blood stains, and other sensor-detectable forensic data could enhance crime scene investigations.

* Forensic pathology could benefit from various advanced medical imaging techniques to visualize traumatic penetrating wounds before physical autopsy. (10)

* The coordinated use of robots, unmanned ariel vehicles (UAVs), and law enforcement officers managed through an AR network could enhance surveillance operations.

Training Efforts

Training programs could use AR to simulate dangerous law enforcement environments by blending real-world equipment and fellow trainees into realistic scenarios. Interactive simulations, such as shoot/don't shoot scenarios and use-of-force demonstrations, can add further realism to police training efforts.

Management Issues

Leadership from the front is a commonly talked about principle rarely employed in law enforcement. The primary reason for this lies in the difficulty leaders have with accessing and visualizing all of the available supervisory and management data while mobile.

* Using video feed from their personnel on the street, supervisors potentially could see what their people are seeing in real time.

* A three-dimensional map display of the community could contain location, activity, and status information.

[ILLUSTRATION OMITTED]

* During critical incidents, supervisors could monitor the physiologic status of all personnel and make decisions concerning tactical deployment and dynamic action based upon those officers, both mentally and physically, best capable to perform.

* The use of visual, audible, and haptic cues from the supervisor could enhance the coordination of widely dispersed units.

IMPLICATIONS FOR POLICING

AR remains an emerging technology unsuitable for law enforcement use today. Also, a number of issues with the technology currently exist that could make adoption by the law enforcement community and acceptance by the public a difficult prospect. Public sanction of civilian police with a futuristic appearance could present a problem. Indeed, the apparent physical melding of law enforcement officers with powerful technology may induce a visceral negative reaction by some citizens. (11)

Acceptance by law enforcement officers themselves may prove even more problematic. Bulk and mobility issues associated with the additional equipment could cause many officers to reject the advent of AR systems, whereas appearance itself creates a major concern for numerous officers and law enforcement administrators. Many law enforcement agencies may not adapt readily to the detraction from the traditional police appearance that historically has connoted professionalism. Additionally, the AR system must be mobile, lightweight, and compact, conforming to the user's body in a way that makes it unobtrusive and nonhindering to the employment of other law enforcement equipment. It must be rugged and capable of withstanding extremes of cold and heat, as well as the rigors of street police work, attributes that the technology does not posses today. In addition, like many of the technologies available today, the cost of AR systems may inhibit agencies from purchasing them, at least initially. To this end, the law enforcement community will need to address all of these potential problems and weigh the risks and benefits of employing AR technology.

CONCLUSION

Augmented reality is a completely human-centered technology. Unlike the quest for artificial intelligence, robotics, or some other "smart-technology" research designed to eliminate the need for humans in favor of decision making by computers, AR will enhance human performance directly, allowing people to work both faster and smarter and in full control of technology, instead of it controlling them.

AR could give law enforcement officers tremendous physical and sensory enhancement that remains completely under the user's control and responsibility. In this post-September 11th age, characterized by the criminal and terrorist exploitation of existing technologies and the serious threat that they pose to free societies, AR could become a potent tool for improved policing. The future prevention and timely termination of crimes and terrorist attacks may depend on the individual law enforcement officer's ability to rapidly process and analyze available data and take immediate action in an extremely short time frame, precisely the kind of potential capability offered by AR. Research and development efforts are underway that dramatically will improve the underlying technology in a few years. To take full advantage of the enhanced capabilities when they occur, the law enforcement profession should understand these efforts fully so it can develop the policies and strategies necessary for effective implementation. The Futures Working Group AR project, in partnership with the Naval Research Lab, will study the potential of AR and help foster the effective implementation of such systems for law enforcement use.

Endnotes

(1) W. Robinett, "Synthetic Experience: A Proposed Taxonomy," Presence: Teleoperators and Virtual Environments 1, no. 2 (Spring 1992): 229-247.

(2) D. McAllister, L. Nyland, V. Popescu, A. Lastra, and C. McCue, RealTime Rendering of Real-World Environments (1999); retrieved February 10, 2002, from ftp://ftp.cs.unc.edu/pub/publications/techreports/99-019.pdf.

(3) M. Bajura, Merging Real and Virtual Environments with Video See-Through Head-Mounted Displays (Ph.D. diss., University of North Carolina, 1997); retrieved February 1, 2002, from http://citeseer.nj.nec.com/cache/papers/cs/1615/ftp:zSzzSzftp.cs.unc. eduzSzpubzSzpublicationszSztechreportszSz98-036.pdf/bajura97merging.pdf.

(4) "The determination without the aid of celestial observations of the position of a ship or aircraft from the record of the courses sailed or flown, the distance made, and the known or estimated drift." Merriam Webster's Collegiate Dictionary 10th ed. (1996), s.v. "dead reckoning."

(5) R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre, "Recent Advances in Augmented Reality," IEEE Computer Graphics and Applications (November/December 2001); retrieved February 9, 2002, from http://www.computer.org/cga/cg2001/g6toc.htm.

(6) J. Colgate, W. Wannasuphoprasit, and M. Peshkin, "Cobots: Robots for Collaboration with Human Operators," in Proceedings of the International Mechanical Engineering Congress and Exhibition, in Atlanta, Georgia, 1996, DSC-Vol. 58, 433-39; retrieved February 15, 2002, from http://lims.mech.nwu.edu/publications/jecolgateIMECE96.Colgate.Wannasuphoprasit.Peshkin.html.

(7) Q. Jin and A. Waibel, "Applications of LDA to Speaker Recognition," presented at the International Conference on Speech and Language Processing, Beijing, China, October 2000; retrieved February 10, 2002, from http://www.is.cs.cmu.edu/mie/.

(8) U. Meier, R. Stiefelhagen, J. Yang, and A. Waibel, "Towards Unrestricted Lipreading," International Journal of Pattern Recognition and Artificial Intelligence 14, no. 5 (2000): 571-785; retrieved February 16, 2002, from http://www.is.cs.cmu.edu/mie/.

(9) K. Patch, "Hot Spots Give Away Lying Eyes," Technology Research News, January 23, 2002; retrieved February 23, 2002, from http://www.trnmag.com/Stories/2002/012302/Hot_spots_give_away_lying_eyes_012302.html.

(10) E. Viire, H. Pryor, S. Nagata, and T. Furness, "The Virtual Retinal Display: A New Technology for Virtual Reality and Augmented Vision in Medicine," in the Proceedings of Medicine Meets Virtual Reality in San Diego, California, January 28-31, eds. J.D. Westwood, H.M. Hoffman, D. Stredney, and S.J. Weghorst (Amsterdam, Berlin, Oxford, Tokyo, and Washington, DC: IOS/Ohmsha Press, 1998), 252-257.

(11) S. Mann and H. Niedzviecki, Cyborg: Digital Destiny and Human Possibility in the Age of the Wearable Computer (Canada: Doubleday, 2002).