Multicamera film tech advancing fast

Today, if you're looking online for tips on improving your golf swing, you might search YouTube for videos of professional golfers demonstrating good form. You peer at the tiny figure swinging a club on your laptop screen. If you're lucky, the filmmaker chose a good camera angle.

Get ready for a dramatic improvement. Pretty soon, as volumetric capture technology ("volucap") takes hold, you'll be able to slip on your latest-generation VR (virtual reality) or MR (mixed reality) goggles and practice your swings with your favorite 7-iron in your backyard, while watching a life-sized hologram image of one of the world's best professional golfers repeat swings right in front of you. If you walk toward the hologram, he'll appear to get larger — back up, and he'll get smaller.

But here's the new and extraordinary thing: You'll be able to walk all the way around him, watching him practice his swing from all sides — including from behind. If you like, you can even get up on a ladder and watch him from above. You'll be watching a 3D hologram of the golfer, not a film. And as the technology continues to develop, you'll be able to digitally interact with the hologram, too.

Dozens of cameras, fast computers

The way it works is straightforward in principle, though it takes a lot of cameras, software trickery and computer power to make it work in practice, as Sven Bliedung, CEO of Volucap, explained to DW during a visit to the startup's windowless premises, deep in a building at Studio Babelsberg, a film industry complex in Potsdam, Germany.

"Dozens of cameras — 32 in total — simultaneously film an actor from all sides, including from above. They generate about two terabytes of data per minute. It's later combined by special software into digital files that generate a moving hologram of the actor," Bliedung explained. As he spoke, we stood on a three-meter-diameter circular stage inside a brightly lit white cylindrical room about five meters in diameter, with dozens of cameras poking through its wall from all sides and from a suspended ceiling.

This process is becoming known as "volucapping" an actor or scene. Volucapping a professional golfer currently means filming the golfer practicing swings on the stage in Volucap's cylindrical white recording studio, analogously to how actors are filmed with a single camera against the background of a green screen in standard filming.

"The hologram can then be spliced into separately filmed background scenes, or into computer-generated backgrounds of any kind, and viewed from any angle in any digital environment," Bliedung said. That includes virtual reality, mixed reality and augmented reality (AR) environments. "We can also generate 2D renderings for use in conventional films and games."

This means Volucap holograms can be adapted for use on conventional flat smartphone screens, too. Instead of being able to walk around a life-sized hologram as you would while wearing VR goggles, you're able to spin the recorded image around on your hand-held screen and see it from all angles. A free app called "Magenta VR" from Telekom recently featured "Fantaventura," a short piece about a 1990s German pop-rap group called The Fantastic Four, which made use of Volucap holograms.

Unlimited applications

Now imagine surgeons performing surgeries so medical students can learn from them as if they were in the room, and volucapping all the equipment in the operating room as well. Or technologists demonstrating work processes for trainees — "a key early market for us," Bliedung said. Or volucapping professional cricket or baseball players demonstrating batting form. Or ballerinas, guitarists, archers, acrobats ... the list goes on.

Because Volucap holograms are made with real human actors, they're more realistic-looking than computer-generated action figures. Bliedung believes game developers will soon switch over to volucapped holograms en masse, and film directors will replace human "extras" with digital avatars to save on costs.

Volucap-generated holograms could also be used as skins for computer-generated actions which means it will be possible to have holograms of volucapped actors appear to do things they didn't actually do during the recording session. Deep fakery has no inherent limits. 

At present, hiring Volucap for a film shoot that generates two minutes of moving digital hologram content costs about €25,000 ($27,400). It's cheaper than trying to generate a realistic 3D animated digital avatar for use in game engines, Bliedung says, and generates better results.

"As VR goggles continue to improve, becoming lighter, more comfortable, and higher-resolution, they'll become more widely used," Bliedung said, "and volucapped holograms will find ever more applications. This is the future of films, apps and gaming."

Few competitors so far

Microsoft and Scatter are the only other companies besides Intel and Volucap that currently have volumetric capture production studios. Intel has a studio in Los Angeles, and Microsoft has its Microsoft Mixed Reality Capture Studio in San Francisco. Scatter is based in New York City.

"Intel in the USA has built a much larger setup where 30 people can be filmed simultaneously," Bliedung said. "But the trade-off is that the resolution of individual actors is much lower than what we can do in our studio. In terms of high-resolution images, lighting quality and other things, we're the world leader."

For Volucap, an important milestone will be to develop a transportable version of its equipment, to allow filming in locations other than their Babelsberg stage. In fact, the 2-year-old startup, is as much a laboratory as a production facility — part of an "innovation cluster" meant to explore the possibilities of the new technology.

"Film directors will have to develop new modes of storytelling using 3D moving holograms," according to Bliedung. "It won't work to simply copy 2D film storytelling to VR interactive films with volucapped holograms ... we're just at the beginning."