A Google Translate Sign Language App Brings Automated Interpretation to Video Calls

0
1987
Woman using Google translate sign language app

At its most basic, video conferencing is a digital recreation of the real world. It brings people together face-to-face over a distance so they can talk to each other as though they were sharing a dinner table.

But recreating the face-to-face experience can sometimes mean bringing existing real-world difficulties along for the ride. If you use sign language during an in-room conversation with people who don’t understand it, for instance, you’ll still need to use it on a video call.

Video calling can become more than just a reproduction of reality, it can become an improvement on reality.

However, if you embrace the digital format of video conferencing, you can solve a real-world problem in a virtual world. Video calling can become more than just a reproduction of reality, it can become an improvement on reality.

That’s the methodology behind a new app that acts like a Google translate sign language interpreter. Using computer vision and artificial intelligence, the app’s Dutch designers have produced a device that instantly converts sign language into speech or text. It’s quick enough to use in live conversation, and it’s a vast improvement over existing services that require the attention of a live human interpreter to act as a third video caller.

Automated Sign Language Interpretation

VC Daily has previously discussed the long-form version of this kind of translation service. The solutions we encountered were of the “first phase” variety, where old methodologies are simply grafted on to a new platform or delivery system. In these kinds of interpretation services, traditional human sign language interpreters were hauled in to act as conduits during a video call, upping the number of video callers to three people.

List of sign language learning appsThis solution is currently in use in some schools in the U.S. The example we focused on was a school in Utah that used a sign language interpreter to translate the teacher’s words over video conferencing for hearing impaired students. In this arrangement, the interpreter would follow along with the day’s lesson via a video call and translate both the student’s questions and, if need be, the teacher’s responses. In reality, every exchange had to travel some 300 miles back and forth before it was understood.

The second example was an initiative of the NYPD. In order to build better relations between officers and the 200,000 deaf or hard-of-hearing New Yorkers, police trialed a mobile version of the above classroom setup. In this scenario, police would carry a dedicated app on their phones that allowed a remote sign language interpreter to help them communicate. The solution brings with it the logistical problem of sharing the interpreter–limited by the smartphone–between two people, but is nonetheless a noble effort.

This new app we mentioned above, however, does away with all those digital theatrics by enabling translation through AI.

Google Translate for Sign Language

Ok, so it’s not Google Translate–the app has nothing to do with Google–but thinking of it as a kind of Google Translate for sign language makes the concept a little easier to envision. The app is produced by European company Evalk and (according to previous articles, we see no mention of the app’s name on the Evalk site) is called GnoSys. It uses computer vision and neural networks–computer systems with multiple interconnecting points that mimic the animal brain–that recognize the shapes and movements of sign language to feed information into an algorithm that can translate those images into speech.  

Incorporating GnoSys into your video calls could make all kinds of social and professional scenarios possible.

The app requires no specialized devices and all the neural networking and algorithmic number crunching takes place in the vastness of the Cloud–which is just a public server with a lot of memory. The app will become available sometime in 2019 and is expected to cost $1 a day, $4 for a week, and $11 for a whole month. There are also subscription packages for small and medium-sized business that top out at $450 a month for multiple users.

For that price, a sign language user can instantly sign into their smartphone screen and have their words read out by the phone. Presuming it works (if you enjoy a laugh go twist the Skype translator in knots to see how easily confused these kinds of services can be) it’s a massive improvement over the human models we mentioned above.

Incorporating GnoSys into your video calls could make all kinds of social and professional scenarios possible.

Video Calling Translation Service

Provided GnoSys works–again, that’s the key to the whole thing–it would stand well above any other hearing-impaired app we’ve encountered on a video calling platform. In fact, the only reliable service we’ve come across is offered by BlueJeans and it’s a one-way flow that provides subtitles. You’d have to couple the GnoSys with this kind of device so that the sign language user could “hear” the other half of the conversation through subtitles.

Ideally, the GnoSys app would mean better access to remote meetings, to telecommuting, and to interactions with online video portals.

But the GnoSys app does offer the hearing-impaired a service they can access anywhere, at any time, without requiring the video host to make special arrangements. The device can be placed in front of the user (and therefore between the user and their usual webcam) and the translation is “spoken” aloud by the computer (let’s hope that voice is more of the Siri variety than the robotic tones of the late Stephen Hawking).

Ideally, the GnoSys app would mean better access to remote meetings, to telecommuting, and to interactions with online video portals such as those used by banks and e-commerce traders. The app would also make it possible for the hearing impaired to participate in social video calls with friends and family who have yet to learn sign language.

Above all else, it offers a private alternative to a human interpreter, which should someday soon be relegated to the failed first phase of digital sign language conversation. No longer would that interpreter to privy to intimate conversations with a teacher or a police officer that might cause the user embarrassment.

GnoSys is an app that could make video calling even better than speaking to someone in person.

Subscribe to VC Daily