The name of the game is S.T.A.R. Ops. It’s an iPad game, and it’s available right now on the iTunes store. Its premise sounds simple: pick up objects, manipulate them, and strategically put them down somewhere else. But S.T.A.R. Ops has a twist: its “virtual” environment directly correlates to the player’s ‘real" surroundings.
To play S.T.A.R. Ops, a player’s iPad must be equipped with the Structure Sensor, a 3D scanner for mobile devices. Developed and designed by Occipital, a software startup based out of Boulder, Colo., the Structure Sensor scans and digitizes players’ immediate surroundings. When you move with the tablet, your position in the game’s virtual world changes, depending on where the iPad’s camera is pointing.
By translating physical motion into a game world, S.T.A.R. Ops blurs the line between “augmented reality,” where virtual elements are superimposed over the real world, and “virtual reality,” where players are immersed in a virtual world. And it’s a step toward solving virtual reality’s biggest issue: physicality.
It’s one thing to play a video game, but to actually feel like you’re physically in a virtual world requires more than just a controller or even a technologically advanced headset strapped to your face. It requires physical feedback and motion sensing technology more advanced and/or expensive than what any consumer-facing virtual reality company has yet come up with. But they’re not blind to the problem.
SEEING IS BELIEVING
“Positional tracking in limited form exists in some VR headsets today, but you’re usually constrained to stay behind your monitor or in a small area,” Occipital co-founder and CEO Jeffrey Powers told Playboy. The latest version of the highly publicized Oculus Rift headset, for example, aims an external camera at players to track their movements in the real world and translate them into virtual space.
“But not with our tech,” Jeffrey continued. “Instead of mounting cameras in a player’s environment to look at the user, we’re actually doing the opposite; looking out at the world to infer how the player is moving.” In other words, rather than using a hand-held controller to command an avatar, the player becomes the avatar—in a way, at least.
The very phrase “virtual reality” evokes science fiction imagery, of Star Trek-esque holodeck adventures, where the digital representation of people is indistinguishable from the real. And though we’re many years away from this level of realism, Occipital’s tech takes a step in that direction, in a form that’s accessible, affordable and portable.
Most major companies are gun shy when it comes to VR, with the most notable exception of Google—although the search company has its own dogs in this fight. Nintendo made an early foray into VR, but instead of a gaming revolution the 1995 disaster called the Virtual Boy became a running joke and a cautionary tale to developers whose ambitions might outpace their abilities.
So instead, Occipital turned to consumers on Kickstarter, where they raised over $1 million for the Structure Sensor.
It was Oculus VR’s shockingly successful 2012 Kickstarter campaign for its Oculus Rift headset that once again put virtual reality in the spotlight and paved the way for Occipital and countless other companies. Virtual reality failed to take hold before, but there’s one key difference today: the technology may have finally caught up with the ambition.
Sony, Valve, Samsung, Microsoft, Google, HTC and others have thrown their hats in the virtual reality ring, and it’s through their efforts that tech like the Structure Sensor will reach its full potential.
“With the technology we’re building, you can actually import the real world as your game world,” Powers said. “It’s already possible with Structure Sensor to capture photorealistic 3D models of objects and people, and it’s becoming possible to capture entire scenes too. If you put a Structure Sensor and an Oculus Rift together, you can literally copy your real world and use it as a VR game world.”
“I recently read that the size of Minecraft is closing in on four billion square kilometers, which is massively larger than the earth itself,” he continued. “Imagine mounting a Structure Sensor on a VR headset and then navigating the Minecraft universe by walking around. The only problem is that the real world would run out before the Minecraft world would.”
That space problem—running out of real-world room, because after all, our imaginations have always been larger than reality—is being tackled by companies like Omni. Omni’s “Virtuix” is essentially a circular treadmill that keeps players in VR environments from running into walls. It’s like a people-sized hamster wheel for gamers.
But that’s only one aspect of the physical feedback that will be required for true VR. The world around us changes based upon our reactions to it, and what we feel and perceive is a crucial part of the bigger picture. What VR needs is sensors that detect physical impulses, then convert that information into gameplay—and vice versa.
Jocelyn Scheirer is one individual who is working to develop such technologies. She’s a role model of female STEM (Science, Technology, Engineering, and Math) involvement. She cut her teeth at the MIT Media Lab, where she and fellow PhD student Teresa Marrin Nakra collaborated with the Boston Pops Orchestra to create a “Conductor’s Jacket,” loaded with biological and motion sensors, that translated conductor Keith Lockhart’s feelings and motions into visual art.
Two years later, Scheirer thought of a new take on an old project; instead of the performer communicating his reactions to the audience, what if the audience could communicate their feelings to the performer? She created the Galvactivator, a wearable glove that measures a person’s overall arousal through a biosignal known as skin-conductance.
At first glance, the Galvactivator looks like a snazzy update of Nintendo’s Power Glove, another of the company’s failed experiments. But the Galvactivator’s possible applications to gaming are self-evident.
“My late colleague, the wonderful Dr. Carson Reynolds, formerly at Tokyo University, digitized a Galvactivator and fed the signal into the computer game Quake,” Scheirer recalled to Playboy. “It was designed so that the avatar representing the user would jump back every time the game detected that he had been startled by another character in the game. It was then that we saw how useful the knowledge of affective response could be.”
Scheirer’s tech could theoretically be paired with the force feedback gloves that others working on VR have created, from the Dexmo “exoglove” to Rice University researchers’ “Hands Omni” haptic gloves. These advancements make it possible to “feel” what your avatar in a virtual world is touching, and they could eventually lead to VR as real as the fantastical OASIS in Ernest Cline’s novel Ready Player One, where players wear haptic sensors all over their bodies.
For now, all this potential is largely untapped. So what’s the next practical step for VR?
“Cameras, cameras, and more cameras are the future,” Jeffrey Powers explained. “The line is blurring between what’s real and what’s virtual, and a lot of this is thanks to the power of tiny cameras that are able to capture the world around us.”
“But cameras are starting to do more than look outward,“ he continued. "They’re looking inward, too. People are already working on cameras that look at your hands so you can interact with your fingers. Multiple tiny cameras will let you make eye contact in virtual environments. Sensitive cameras are actually able to see the color variation on your face to measure your pulse.”
Soon, Scheirer will be working with NYU Professor Ken Perlin to incorporate her sensors into a VR experience. Over the years, she’s fine-tuned the Galvactivator into a more aesthetic, robust, wearable ring called the BE Sensor. Her primary focus is her new company Bionolux, which will address mental health issues through physiological sensor technology. But she also recognizes the possibilities for her product.
“The approach to VR is unique because of the focus on freedom of motion, and seamlessly blending virtual and physical stimuli,” Scheirer says. “Ken is using a motion capture lab and the Samsung Gear VR and Galaxy Note 4 so that the user can move freely, wirelessly through the space, while the virtual and physical worlds respond to user movement, voice, and now, nervous system activity.”
Imagine a game that becomes more difficult or easy as the player becomes more stressed. Maybe the gun could shake, or the player’s health could deplete. Perhaps more enemies could appear, or the types of enemies that spawn could vary. Or a game character could react to a physical cue like a blush—just like a real person.
But whether technological pioneers like Powers and Scheirer measure their users’ environments or measure the users themselves, they’re all working towards that idealized, ever-receding goal: creating a digital reality that is similar to, if not identical to, our real one.
Wing-Man has written about video games and popular culture since 2013, and has been published in multiple online and print publications. Follow him on Twitter to learn more.