Google, not content with just pursuing augmented reality applications, has developed a novel way to allow people to see through a user’s VR headset.
Often, when demoing VR applications to people, a camera and green-screen is combined to capture what a user is doing within a game. This mixed-reality video allows the potential audience to see exactly how you would interact with your virtual surroundings and helps those without a VR headset to understand how a game may work. The problem is that, as you can see in the GIF demoing the game Job Simulator below, the VR headset blocks most of the user’s facial expressions. Is he having fun? Does he hate the endless hell of a simulated working environment?
Luckily for us, as detailed over at Google’s blog, researchers have figured out a novel way to show off people’s faces and facial expressions. First, Google creates a number of 3D scans of a person’s face. These scans capture a number of variations such as blinking, looking in different directions, etc. Then, with the help of an HTC Vive modified to include eye-tracking, Google is able to combine the 3D face-models with the eye tracking data to simulate how the person is reacting to what they’re doing. This is then inserted into the mixed-reality video, allowing people to see how the user is reacting in real time.
Capturing a user’s facial expressions is pretty efficient as well, with each scan taking less than a minute to complete and requiring only a webcam. As the VR industry continues to evolve, it’s expected that we’ll start to see eye-tracking becoming native to future headsets, which may make this technology more available to everyone. Though Google’s current focus with the technology has only been on the development of these mixed-reality videos, it should still appeal to the growing VR software industry as they continue to try to figure out how to market their programs to users who may not yet have a VR headset.
That said, future iterations of the technology could be a boon to both video conferencing and multiplayer games. The tech could allow your face to be inserted in-game, allowing you to see how your friend is reacting as you explore a VR environment together. Or, it could simply let you see the facial expressions of the person you’re hanging out with overseas. Sooner than later, you’ll finally be able to see the horror on your friend’s face as an alien bursts out of their chest…
Images: Owlchemy Labs, Google