3D Video Conferencing Uses Full Body Reconstruction to Allow Video Chats Using Realistic Avatars

video conferencing in 3d

It is a truth universally acknowledged…that 3D technology works only in two dimensions.

For all the headlines and retweets that a holographic appearance by a long-dead rock star generates, there’s a sad fact behind every such story–3D technology doesn’t do backsides.

No matter how convincing the illusion, all current 3D imagery relies on visuals projected onto 2D screens – be they transparent screens that create the effect that the hologram is moving in front of real world objects, or stereoscopic images that use shading and the ancient painter’s trick of shrinking horizons to generate depth.

If you were to ever view such spectacles from alongside or behind the display you’d see nothing but a clear pane of glass.

So when one of the world’s foremost telecommunications research centers turned its attention to 3D video conferencing, it did away with the idea of casting holograms into the real world, and instead focused on bringing the real world into a virtual reality setting.

The result is 3D video calls you can move around within, and 3D video callers you can walk around–all the way around.

Which means 3D is about to get back.

Video Conferencing Engineered in Germany

The research center in question is the Fraunhofer Institute for Telecommunications, Berlin. The name may not be as recognizable as some Silicon Valley startups, but these Europeans are no novices when it comes to video conferencing technology.

In 2012 they introduced the Virtual Eye Contact Engine, which uses an array of cameras to track facial and eye movement and let video callers look each other square in the face while staring into their screens, not up at the camera.

And before that, they were one of the first to demonstrate how HD quality video conferencing could be transmitted over standard internet connections using innovative audio and visual codecs.

So there’s reason to believe them when they say what they’re about to unveil at upcoming trade fairs in Berlin and Amsterdam will be the most convincing and immersive 3D video calling yet created.

The technology, currently going by the slightly ominous title of 3D Human Body Reconstruction, will recreate real world places and people in 3D virtual reality, and allow video callers to speak to each other in real time using realistic scans of their entire bodies.

Video Calling with 3D Avatars

Fraunhofer’s promised 3D breakthrough relies on an array of 20 stereoscopic cameras–that is, cameras with two lenses capturing slightly different angles of an object–that circle their subject and record every inch of their body. That image is then projected into a virtual reality world where it appears as a stand-in for video callers, rather than a live camera stream. It’s a bit like hosting a meeting in Second Life, but with an emphasis on reality, not escapism.

This is the same principle the Smithsonian employed to cast their 3D model of President Obama, and that USC’s Institute for Creative Technologies used to make a 3D video of Supersize Me star Morgan Spurlock.

Where Fraunhofer has them both beat is that their 3D models can walk and talk in real time, acting as instant avatars for real world people making real-world video calls. The researchers are also working on Real-Time 3D Scene Reconstruction to give these virtual interlocutors somewhere to sit, walk, and explore during their chat.

Of course, to become ubiquitous, a full-body imaging system like this will have to be compact enough to fit within a standard office, or it will likely remain a trade-fair oddity. The USC version, which is not live, takes up an entire lighting stage.

It’s also going to take a lot computer power to crunch all that streaming information and paste together so many different angles of the same subject into a working model that can react quick enough to replicate human conversation. So it’s likely these avatars will be pre-recorded full-body masks with mouths that move in tune with live audio, and heads and bodies that react to auto-tracking, like the funny cat and KISS masks you can don in Snapchat and some video calling apps. Here, though, the focus is on recreating an authentic version of the caller.

But if it works convincingly, it’ll create a virtual video conferencing system that is far more powerful and versatile than anything two people could recreate from a face-to-face meeting.

Video Conferencing with Backside

Fraunhofer currently plans to immerse callers in this virtual world through VR headsets, probably not unlike the basic models currently on the market. This removes the most obvious reminder that you’re not actually sitting face-to-face with your colleague during current video conferences: the edge of the screen.

Instead, you are both sharing the same 3D world. You could get up from your chair and move around freely without going out of camera shot. You can gesticulate with your entire body, and should you move past or around each other you’d be presented with a rounded human body, not as a flat face that jerks unnaturally so as to always appear front on.

And once the real life scene reconstruction element is added you’d have plenty of reason to move. While the location may be a faithful rendering of your office, each component is now digital and so can used to store electronic files, display video, and be stretched, squeezed, and manipulated into any shape needed for demonstration purposes.

You’re also free to defy the laws of physics. You could each fly up to the ceiling to get a bird’s-eye view of a map or blueprint. You could move furniture with the flick of a wrist, or replace it instantly with a snap of your fingers.

If Fraunhofer’s European trade fair tour bears fruit, we could all be shaking our 3D-rendered backsides while dancing on the ceiling during video calls within a few years.

Subscribe to VC Daily