The sounds of traffic echo down a dusty street in the Syrian city of Aleppo. A food vendor is taking orders at his rickety cart. A boy momentarily rests on his bike at the sidewalk corner. The pitter-patter of dozens of footsteps helps drown out the ceaseless honking, but only a little. Someone with a handheld camera is recording footage alongside the street. A little girl appears onscreen, her left cheek painted with an image of the Syrian flag, singing a little song in a timid, uneven voice.
15 yards to her left, a rocket strikes and explodes.
There is screaming. The little girl is gone from the camera’s narrow field of vision. People flee left and right as quickly as they can through a choking wall of dust. Already, bodies lay on the ground in unnatural and ghastly positions.
This is real life, but it’s also virtual. The people who died on that street have long since been buried, but their final moments and the memories of those who survived live on through Project Syria, a virtual reality simulation.
This is the work of Nonny de la Pena. Called “The Godmother of Virtual Reality” by Engadget and the Guardian. This is her new mission in life: to expand our understanding of the stories that seem to flit by on the television screen while we’re too busy cooking, chatting or flipping through Twitter. This “immersive journalism,” as de la Pena called it during our interview, could change how we view the nightly news.
“What if I could present you a story that you would remember with your entire body and not just with your mind? My whole life as a journalist, I’ve really been compelled to try to make stories that could make a difference and maybe inspire people to care,” de la Pena began in a 2015 Ted Talk focused on her work.
The concept of boots-on-the-ground firsthand reporting is as old as journalism itself. The ability to weave a story with whatever tools you have (be they words, a camera, a VR headset, or some yet-undefined technology) and make someone feel like they just lived the experience is often regarded as one of the greatest achievements of storytelling. Take the CBS Radio series You Are There, which eventually transitioned to television in 1953 with the legendary newscaster Walter Cronkite:
From his New York studio, Cronkite would give a short introduction, speaking as if he were an actual anchor during each event, such as the Great Chicago Fire of 1871, referencing the “new city, which is now only 30 years old.” In the age of three channels, the idea that “the most trusted man in America” was going to send you off to the Salem witch trials or the Nazi Party’s annual rally in Nuremberg was incredibly novel. Celebrity actors in the guise of historical figures (James Dean appearing as Jesse James’ killer Robert Ford) provided “interviews” to reporters clad in contemporary suits.
But as much of a landmark for storytelling as You Are There might have been, its title was just a precursor to more visceral depictions provided by increasingly mobile technology. Stage plays gave way to 24/7 news stations, which gave way to democratized platforms like YouTube and streaming services, and now de la Pena’s virtual recreations. Each iteration was inevitably met with cries of “video killed the radio star”-esque worries, yet each went on to transform the state of media in profound ways.
Such a process was never easy for de la Pena. Even now, with multiple partnerships and commissions from global news organizations, her team’s pace seemed to match that of the rapid news cycle. For her, the adventure began after a career dedicated to storytelling in documentaries, television dramas, and stints at New York Times and Newsweek.
ON THE SCENE
It was a particularly frustrating day in Los Angeles. The depressingly long line of people waiting outside a local food pantry certainly knew it. The wait was beginning to take too long for some, and the woman in charge of dispensing food eventually began to shout “there’s too many people! There’s too many people!” As the distressing commotion continued, one man in line experienced a nosedive in his blood sugar levels, causing him to collapse into a diabetic coma, spasming on the ground as others watch on in fright.
De la Pena was elsewhere, but soon enough, one of her interns came running into the office with tears in her eyes. The intern had been on scene as part of the pair’s ongoing project to record real audio from Los Angeles food pantries for their first VR project. Her distraught state was enough to convince de la Pena of that moment’s power.
Hunger in Los Angeles became the Sundance Film Festival’s first VR documentary in 2012. Using donated models of humans and what little money she had, and cashing in on all the favors owed to her, de la Pena’s team digitally recreated the scene for filmgoers to physically walk around in VR.
“It was really Sundance that pushed [virtual reality] forward,” de la Pena said.
At least one participant left in tears, shocking even de la Pena.
The reaction was visceral, to put it lightly. Participants instinctively avoided getting too close to the collapsed man, bending down to get a better look as he twitched, all while listening to the real, distressing audio captured that day. At least one participant left in tears, shocking even de la Pena. While significantly different from the 360 degree videos that pervade Facebook (the rudimentary graphics used in de la Pena’s digital recreations would certainly never be mistaken for the real world), the sense of presence afforded by de la Pena’s simulations impart a much stronger impact on viewers than a simple rotational, yet flat video.
The headset used during Sundance acted as a replacement for the $50,000 version de la Pena wasn’t allowed to take outside the studio. (An “unofficial intern” by the name of Palmer Luckey would later end up designing the replacement, originally pursuing a degree in journalism. Nine months later, Palmer would go on to found Oculus VR, launching a Kickstarter campaign for the Oculus Rift, the product that would spark the current wellspring of consumer-level VR headsets.)
This was just the beginning for de la Pena, though. Over the following years, her company, Emblematic Group, would be contracted to create numerous similar projects. De la Pena would tackle Border Patrol officer brutality with Use of Force, wherein players live out an eyewitness account of officers beating and tasering a migrant named Anastasio Hernandez to death. In Kiya, using recordings of two separate 911 calls, de la Pena recreated the scene of a deadly case of domestic abuse for Al Jazeera America.
One thing remains consistent throughout each project: de la Pena’s intent to evoke a sense of empathy, of developing a deeper understanding of worlds we otherwise pay no mind to. A lack of empathy may be what keeps a predominantly white audience from understanding the plight of police brutality in black communities, or the often vicious and degrading gauntlet that hundreds of women seeking abortions must face. It’s not necessarily about imparting a particular message over another, it’s about giving people a more informed perspective to utilize when assessing the facts of a situation.
LIVING WITH PREJUDICE
“Everyone is a prisoner of his own experiences. No one can eliminate prejudices—just recognize them,” famed American broadcast journalist Edward R. Murrow said during a broadcast in 1955.
As each new step in technology is made, there are of course doubters. One can’t necessarily blame organizations and individuals who’ve yet to adopt VR for thinking there’s an inherent ethical problem when a simulated recreation is being shown in place of actual footage. De la Pena, along with a number of her industry peers, kindly disagree.
“The ethics are about the newness, right? People think of print as being so ethical,” de la Pena said.
De la Pena refers to the case of Kitty Genovese, a New York City bar manager who was stabbed to death outside her apartment in 1964. As legend has it, roughly 38 residents heard Genovese screaming for help as she died, none of them coming to her aid. Somewhere along the way, the poorly researched anecdotes became grounds for incredibly inflammatory newspaper reports, calling New Yorkers monsters and establishing a hellish image of the city. Time and more responsible research would eventually show that only two individuals, both with histories of carelessness or meek behavior, failed to act, according to a 2014 report in the New Yorker. The damage was done, however, and the faulty reporting birthed what is commonly referred to as the “Bystander Effect,” which implies that the more witnesses there are to a crime, the less likely any of them are to help.
The concept rings throughout de la Pena’s work, which places viewers into scenarios where they are literally helpless to alter the course of reality. It sparks a variety of reactions.
I don’t think ethics should be the barrier going forward.
Vice President, Poynter Institute
“We’ve sort of given up on the idea of objectivity,” de la Pena said. “For me, the harder thing here isn’t an ethical approach, but more about how do we create better transparency so that our audience can have critical thinking when they see these pieces and can make some judgement calls for themselves.”
The Poynter Institute, a nonprofit journalism school, owner of the Tampa Bay Times, and an oft-cited source in matters of journalistic ethics, has at least one key administrator who agrees, particularly on the issue of 360 degree cameras.
“I don’t think ethics should be the barrier going forward,” Poynter Institute Vice President Kelly McBride told me. “In 360 video, there’s the complication of where to put the equipment. The presence of the equipment changes the scene. It’s just like in science when you’re observing something. The nature of your observation changes what happens.”
For both de la Pena and McBride, the crux of the argument seems to rest on the idea that all of journalism is essentially re-creation.
“There is a tradition in journalism of recreating things,” McBride said. “We recreate a narrative text, we recreate a scene. In both cases, the writer who’s writing this narrative wasn’t there, but he’s done the reporting to recreate a scene.”
“I would suggest VR is definitely going to be used for propaganda,” de la Pena said. “One of the biggest challenges in VR is how do we offer that transparency, because people right now will accept a lot of it because it can be felt so viscerally. That’s my one big concern.”
WELCOME TO NORTH KOREA
I’m standing in a press box just slightly above the immense Kim Il Sung Square in Pyongyang, North Korea. Foreign media is clumped together shoulder-to-shoulder, restricted to this spot for a massive military parade. There’s the legions of lockstep soldiers saluting their leader Kim Jong Un, and formations of ground and aerial vehicles carrying missiles of unknown origin. As the square grows full, a final stretch of North Korean citizens march past waving flags, balloons, and flowers.
Any other recording of this parade might showcase the expansive landscape, filled with the colors of military uniforms and extravagant dresses. Instead I’m looking directly into the eyes of each North Korean citizen, hints of desperation cracking through the veneer of false optimism that any but the most brainwashed know has always been there. Suddenly, the military might washes away, the knowledge of prison camps and mass starvation rising back to the top, and an extravagant spectacle is reduced to a mere charade.
This is the work of Jaunt, a “cinematic VR” company in partnership with ABC News, among others organizations like CBS and Sky News. Jaunt President Cliff Plumer has helped produce a number of these news vignettes, all with a specially designed 360 degree camera exclusive to Jaunt’s studio. Much like any other virtual reality experience, the videos allow viewers to pan around in any direction as a reporter’s narration unfolds a particular story. Beyond the piece on North Korea, ABC News’s VR division, in coordination with Jaunt, has already covered elements of the European migrant crisis, earthquake recovery efforts in Nepal, and archaeologists’ efforts to protect Syrian artifacts threatened by war.
Despite this extensive work on journalistic storytelling (Jaunt also produces Pixar-like VR films and PR material for figures like First Lady Michelle Obama), Plumer actually laughed at the notion of 360 degree video “counting” as VR.
“It’s not,” Plumer told me. “It just doesn’t have the same sense of immersion as VR, so it’s totally different. I think it’s an introduction to VR.”
Plumer cites the same concerns that many VR developers have, which is the difficulty of explaining the medium’s potential to people who’ve yet to experience it firsthand. The news content Jaunt produces is, in essence, an easy road towards convincing users to try out their decidedly more “virtual” content.
Even though he thinks 360 degree video doesn’t reach the strength of fully immersive simulations, Plumer (whose background includes administrative roles at LucasFilm, Oculus VR, and Telltale Games) feels that there is a power in the mobility their work provides that few others in the field can touch.
“[Reporting] is about getting access and getting close to what the story is and what’s going on in that scene,” Plumer said. “But what happens in editorial and post-production is just as important. We’ve done a pretty good job of developing tech and a workflow to turn things around in hours, not days or weeks. Because if you’re telling a news story, it’s relevant that it’s current.”
It remains to be seen what form of virtual storytelling viewers will flock to the most, or if de la Pena, Jaunt, or other companies will be able to carve out their own niches. Both de la Pena and Jaunt are currently devoting resources to training reporters on these new technologies, developing questions like, as de la Pena puts it: “What is the interview of the future?” and how handheld controllers can give viewers even greater agency within a story.
De la Pena recently received a “Visionary Award” from the Academy of Art University of San Francisco, alongside Oculus VR co-founders Palmer Luckey and Brendan Iribe. Future innovators will likely see equal reward as a result of this decade’s groundbreaking work. But it’s Joseph Pulitzer, the namesake behind the most well known journalism award, who unknowingly touted this new form of journalism best:
“Put it before them briefly so they will read it, clearly so they will appreciate it, picturesquely so they will remember it and, above all, accurately so they will be guided by its light.”
One can only imagine what will happen when we live it.
Joseph Knoop is a freelance games journalist and part-time comic book geek. His favorite games include cute animals, so Metal Gear Solid 3: Snake Eater probably counts. Talk progressive metal and jazzhop with him on Twitter @JosephKnoop.
RELATED: Gamer Next Door’s Adventures in Virtual Reality at E3 2016