Apple’s Augmented Reality Is Coming to FaceTime, Using Light Field Camera Tech

Augmented Reality Light Field Camera Tech

Not all ideas granted patent protection are destined to change the world.

Strange inventions like the kissing shield, the snake-walking collar, and an apparatus that gives you a pat on the back, for instance, may be safely enshrined as the property of their inventors, but there’s little chance you’ll ever see any of them in use.

Tech giant Apple, on the other hand, has had more success in bringing patented ideas to world-changing fruition. In early June, the world’s most valuable brand secured tech patents for 68 new ideas it hopes will be as pioneering as the iPad or the iPhone.

One, in particular, could have a big impact on the future of video calling: Apple’s augmented reality. And this high-tech magic could come to the humble FaceTime app.

What Makes It Possible? Apple’s Light Field Cameras

The patent in point is a new method of creating augmented reality within a live video chat. Augmented reality is basically the ability to overlay graphics onto a screen, and ranges from the simple scores pasted in the corner of your NFL broadcast, up to the cartoon augmented reality of Pokemon Go.

Apple has won patent protection for a new way of using light field cameras, included in the iPhone 7 Plus–although it has been used by other brands as well–to project images into a live video chat.

The iPhone version of the tech overlays the images from two cameras to create a sense of depth, resulting in the “bokeh” effect where the subject of a photo is held in sharp focus while the background is blurred.

Using more cameras in the array, and focusing them on objects other than the main subject–like your fellow caller, in a video chat situation–makes it possible to radically expand the visuals of a video call.

Apple’s Augmented Reality Could Add Surroundings to a Video Chat

Apple’s first use of the tech may be to bring the depth of field feature from the iPhone 7 to a video call. It would increase the immersive nature of a call by making the person on screen stand out in contrast to a blurred background. Such technology is already available in webcams such as the Razor Stargazer and the 4k video-enabled Logitech Brio, which use infrared sensors to separate a caller from their surroundings. It’s also the key ingredient to the Windows Hello facial recognition software.

Apple could push things further by displaying alternate–but equally live–backgrounds behind a video caller. It only has to combine the face-on stream of a traditional camera with the stream from a secondary camera that takes in more of participant’s environment to add extra context. You’d be able to see a closeup of your friend’s face within a wider view of the room, café, or park where they were sitting. Conversely, the light field cameras could cut out a video caller’s image and project it onto a different background in true augmented reality style.

So rather than have the scores plastered over your NFL broadcast, your video calling friend’s image could sit there, instead, and you could watch the game on a stream from your computer and overlay your conversation on top of the action. Or, as with Pokemon Go, you could potentially project that cutout over the live viewpoint of an iPhone so it appears that your friend is sitting on the chair opposite you in a café.  It’s a much more personal experience than having your friend encased within the confines of a chat window.

No matter to what extent this tech is used, it at least offers the promise of improving on the current functionality of Apple’s one and only video calling app, FaceTime.

Bringing Augmented Reality to FaceTime

FaceTime has seen little evolution since its introduction almost seven years ago. It was–and is–built for simplicity and rapid-fire video calling, but its current state, as we discussed in our Skype for iPhone review, means that rival Skype is probably the better bet in most cases.

Adding an augmented reality feature, however, would be a major step forward. Even if it were initially limited to the desktop, due to the processing limitations of a smartphone, the concept of easily switching out backgrounds and projecting remote callers directly into your real world would certainly make this default technology a lot more interesting. And useful.

Certainly more useful than the patented flush toilet for dogs.

You know you want to read more posts like this on cutting edge tech and where it might be headed. If so, submit your email address (in the box in the sidebar to the right) and we’ll send our newsletter to your inbox once a week with our latest articles.

Subscribe to VC Daily