With the advent of touch-screen technologies, navigating the digital world is no longer limited to typing on a keyboard and clicking on a mouse. Today, we drag our fingers across handheld devices and tabletop displays to interact with electronic data. In the near future, we may command such systems without touching them at all.
The same researchers who developed the touch capability for Microsoft Surface are exploring ways to interface with computers and other digital gadgets using hand gestures.
“We’re tracking specific kinds of motions,” says Hrvoje Benko, a researcher on the Redmond, Wash.-based Microsoft Research team, during a recent road show.
Inside a small makeshift planetarium, he stands behind an omni-directional projector that is producing a 360-degree image of the nighttime sky. He places his hands into the light beam so that they cast a shadow on the dome above — the better to demonstrate his motions in the darkened space.
“We can literally reach into the image and move the sky,” he says as he touches his index fingers to his thumbs in a pinching motion and moves his hands apart to “pull” the edges of the sky outward. The overhead image expands, zooming in on a cluster of distant stars. He reverses the motion, pushing his hands towards each other. This time, the sky contracts and zooms out beyond the Milky Way galaxy.
The secret lies inside the projector where a wide-angle lens focuses the data onto the 360-degree environment. The team has added an infrared camera that shares the projector’s lens to detect a person’s gestures in mid-air. Anywhere the images are projected, the camera can also “see.” Algorithms allow the system to correlate the holes made by a person’s hands to computer functions.
Touching the thumbs and index fingers together to form a large teardrop-shaped hole signals to the computer to listen for a vocal command. “Crab Nebula,” Benko says into a headset microphone. The planetarium fills with the requested telescope imagery. Using the same method, he switches to a replica of his team’s office building, pulls up real-time video footage of the planetarium’s exterior and immerses viewers in a snow globe simulation.
“More and more, we’re seeing the emergence of other kinds of data that might be more suitable for other form factors and displays,” he says.
There could be complex, three-dimensional battlefield awareness data that a ground commander might want to examine. “You don’t have to gesture crudely at a flat display and not understand what we’re talking about in 3-D,“ says Andy Wilson, the senior researcher on the team. “We can actually walk into the dome and I can take you there. By putting yourself into the data, you can get a better sense of what’s happening.”
Researchers say that a corporate conference room could be another outlet for their omni-directional projector. “You can think about having something like this on the ceiling, and all of a sudden, every surface in the room is potentially [a display],” says Benko. “You can put content anywhere you want.”
But the brightness and resolution of current projector technologies are limiting factors. To convey intentions through gestures, one must stand close to the projector. “What we’d like to do is relax that a little bit and start interacting more in the full volume of the space,” says Wilson. Ultimately, researchers want conference attendees to be able to interact with the visual information no matter where they are sitting in relation to the projector. They could walk up to an image projected on a wall, for example, and manipulate it by touching it and moving it around with their hands. “The spatial dimension comes into play,” says Benko.
The team is not wedded to projectors and it points out that researchers are looking at interactions with displays embedded in other form factors. The growing popularity and affordability of high definition displays means that American households in the future likely will contain an ecosystem of multiple and disparate display technologies. How those might be tied together and how we will interface with them also are being investigated.
“We’re saying there will be a variety of different devices. You’ll still have a keyboard, probably, to deal with Excel spreadsheets,” says Wilson. But gestures or other control mechanisms could allow people to interact more collaboratively with other types of information, such as the latest vacation photos on a giant flatscreen display along a hallway, he adds. “The opportunities are there to make it a highly interactive, shared experience,” says Benko.