An Augmented Reality System to Create ‘Digital Graffiti’

One of the indelible images from the Hurricane Katrina disaster in 2005 was that of first responders in boats and helicopters plucking stranded New Orleans-area residents off rooftops in flooded parts of the city.

In the days following the initial rescue efforts, National Guardsmen and other workers conducted door-to-door searches for other survivors. Along the way, they spray painted arrows and large X’s on walls, marking them with numbers, letters and symbols to indicate who had gone through the premises, when they had conducted the search, and any resulting discoveries.

The process, though rudimentary, was necessary given the extent of the flooding and the lack of communication equipment and tools to methodically track search-and-rescue teams’ progress. But over time, the hand-painted signs became illegible as subsequent search parties added their own annotations. Historical information that may have been pertinent for future relief efforts was all but erased.  

Six years later, portable electronics and computer vision technologies have advanced to the point where a person could mark up those same locations digitally. But rather than using a smartphone or an iPad to do the trick, researchers at Lockheed Martin Corp.’s Advanced Technology Laboratories have developed a backpackable augmented reality system that permits users wearing a head-mounted video display to sketch notes and drop icons virtually in the air.

“We’re looking at augmented reality as a way to blend the real or natural world with the digital or virtual world,” said John Sausman, program manager of the “digital graffiti” effort.

The wearable prototype comprises a commercial laptop that connects to a pair of sunglasses with a built-in video display. Users wearing the glasses will see a video representation of the real world captured by two cameras located on the frame’s opposite sides.

On the video display, option menus pop up on top of the images being captured, just as they do on digital camera displays. By donning a glove that is designed for tracking hand gestures, users can make their selections.

“It’s a simple flicking of the wrist that lets you navigate through the menus,” said Sausman.

In a demonstration of how troops might use the system on patrol, a program engineer points his gloved hand at a vehicle. It calculates the range by detecting how his arm is tilted. A pie-shaped menu of tasks appears on screen. He rotates his hand left or right to navigate and selects an icon. To “drop” it on the vehicle, he turns his head slightly until the icon floats into position. He then locks in its location using commercial GPS. The object can be tagged with additional information, such as color characteristics, license plate number and a photo.

Others arriving later could pull up those same annotations and add to the data.

“The markings are at the site, not in somebody’s notebook or laptop,” said Sausman. If the original user returns later, only to discover the vehicle is gone, his notes will still persist in the vacated location.

The Lockheed team is researching the augmented reality system for use in other domains. Potential applications could be in the manufacturing or maintenance fields. Researchers are investigating how they could present 3-D models, technical drawings or parts diagrams over the real-world environment to help the human user in assembling a component or repairing a part.

“Can we overlay those, so they know exactly where they need to be looking, or diagnose problems before they’ve even taken the panel off?” asked Sausman.

To enable that vision, one of the challenges that would need to be solved is in the display technology itself. The digital graffiti team wants video displays that don’t block the user’s entire view of the real world, but instead are transparent so that the icons appear over his natural field of view.

Head-mounted displays will improve in the near future, said James Donnelly, director of sales, defense and augmented reality at Vuzix Corp., which provided the Lockheed researchers with its commercial 920 video eyewear for the project. The company makes a tactical video display that is incorporated into troops’ protective eyewear. It plugs into a radio receiver used by joint terminal attack controllers on the ground to call in fires on a target. The company has fielded thousands of the monocle devices to the explosive ordnance disposal community, which deploys them to troops who use them to see through the “eyes” of their bomb disposal robots.

The next-generation system, he said, will resemble a miniature version of the heads-up displays that pilots now rely upon in commercial planes and fighter jets. Donnelly gave National Defense a sneak preview of the small postage stamp-size optical viewer, which will be incorporated into consumer eyewear. A projector about the size of a dime and a half-inch deep will display holograms on the glasses.

The Defense Advanced Research Projects Agency is funding the development of the holographic optical display system as part of its persistent close-air support, or PCAS, program. The device will overlay a series of symbols onto a target that the joint terminal attack controller has in view so that he can determine the potential for any collateral damage incurred by bombs.

“At the end of the day, they want an unmanned A-10 flying above with the video feed from a pod underneath,” said Donnelly. “They want to give the JTAC the ability to fire from the device.”

The new system is about 18 months away from hitting the market. Integrating the display technology into comfortable eyewear is the challenge, said Donnelly. “Once you get it to a point where it looks like a pair of Oakleys, everyone will wear it,” he said. “This see-through stuff will do that.”

Topics: Science and Engineering Technology, Simulation Modeling Wargaming and Training