Typing on a virtual keyboard floating in the air sounds cool until your arms give out. This is the reality of many AR (Augmented Reality) and MR (Mixed Reality) headsets today. You’re either fumbling with handheld controllers or holding your hands up for too long, causing fatigue.
Researchers at Tohoku University believe they have found a better way. The team has developed a system that turns any normal flat surface into a touch input area for AR and MR headsets. No additional sensors. No special markings. Nothing to set up.
How does it work?
When you press a fingertip on a hard surface, the skin under your fingertip turns white for a brief moment. This phenomenon is known as the blanching effect and researchers have developed a method to identify it using the headset cameras.
The researchers trained an AI model to recognize this color change in real time. When the camera detects a bleaching effect, it registers a touch input that matches what appears on the surface in the AR environment.
“This research means that common surfaces around us, walls, desks or partitions, can be used as a touch input area,” said Guanghan Zhao, who led the study. “In addition, this method does not require special sensors, markers or additional devices. It can be easily used by anyone.”
Does it work well?
User studies as part of the research found that the system reliably recognized touch input on several common surface materials and participants completed interaction tasks accurately.
“Our primary goal was to develop a technology that enables touch input on everyday physical surfaces for AR and MR without the need for special hardware,” said Guanghan. And it is much more comfortable to place your fingers on a surface than to hold your hands in the air.
The results were presented last week at the 33rd IEEE Conference on Virtual Reality and 3D User Interfaces in South Korea. The paper has also been approved for publication in the IEEE Computer Society Digital Library.
