Interview with Technical Director Thomas Holl

Speech recognition is the exciting new feature at  Babbel. It’s not only fun – it’s also amazingly efficient for learning a new language. But how does it work? I got the low down from our Technical Director Thomas.

Crisi: What does the new speech recognition tool do?

Thomas: Basically, we use pronunciation samples recorded by our native speaking course editors and compare your pronunciation to theirs. As always with Babbel, you get instant feedback. The closer your pronunciation is to this example, the more points you get on a scale from 0 to 100. If you get more than 50 points, you’re good enough to be generally understood.

Crisi: But if you just compare two sounds, is that really speech recognition?

Thomas: Sure, we recognize what you say. We’re now sitting in front of the screen and we are talking but you see that the score is 0 all the time. Now, try saying arrivederci.

Crisi: Arrivederci

Thomas: Nice, 78 points.  Better than Aldo Raine in “Inglorious Basterds” (see details here). Remember the hilarious scene where Brad Pitt is trying to speak Italian? We ran his pronunciation through our analysis and as you might expect he scored pretty low. But I’m digressing, sorry. Back to our little test. Your pronunciation is about 78% exact compared to our reference sample. That’s pretty good.

Crisi: Still, it’s only about comparing sounds, not about understanding what I say.

Thomas: Well, there are different sub-types of speech recognition. One is speech-to-text or voice control. That’s what you’d use to enter text or commands if you can’t use a keyboard. Recognizing words and evaluating their pronunciation is another sub-type, and that’s the technology that makes sense for language learning. We can use it for pronunciation training and for building new interactive exercises.

Crisi: So, what’s the technical challenge in this sub-type of speech recognition?

Thomas: Well, it’s not as easy as it sounds – no pun intended. It’s actually not enough to just compare two sounds. It’s a little like telling how similar two people look in two different photos. The audio samples are usually pretty different: a woman has a higher voice than a man and the tempo of speech also differs a lot. And then you have a number of artifacts…

Crisi: Artifacts?

Thomas: Noises and characteristics that are caused by the environment or the technical setup: rumbling, hissing, other sounds mixing into the voice. Most people don’t have a high-end microphone connected to their computer and in our case we just use the built-in mic on my laptop. The audio quality of what the system is hearing is pretty poor.

Crisi: So to make the speech recognition work properly, our users need to have a good mic and be in a quiet room?

Thomas: No, that’s the point: we can also work with cheap microphones and filter out noise in the immediate environment. That’s part of the challenge.

Crisi: Sounds like a lot of filtering and levelling…

Thomas: Yes, that also, but there’s more: We have to distil the “core” of the voice sample and then match that to the original. To do that, the system needs to figure out when you start and stop speaking. You don’t have to press any key to start and stop recording; we do the matching in real-time.

Crisi: So everything we say into the system here is somehow analyzed?

Thomas: Right. Just look at the level: every sound input is analyzed and matched to the sound we’re looking for. In this case, arrivederci.

Crisi: 55 points

Thomas: Ok, yours is better than mine. But you see that the word was recognized among all the other things we said.

Crisi: Is this unique technology? Are there other software product that do this?

Thomas: There are a number of software products that do have speech recognition. Some of them also are of decent quality.

Crisi: So what’s so special about the Babbel speech recognition?

Thomas: Well, it’s online and works in your browser.

Crisi: Does this mean that everything we say here is sent to the Babbel servers and analyzed there?

Thomas: No, the whole audio processing is done instantly, directly in the browser. We don’t have to send the audio to the server and that’s why we can give instant feedback.

Crisi: Do I have to install a plugin or something?

Thomas: You don’t. It’s all done in Flash. 97% of all browsers have the Flash plugin pre-installed. As we use the latest version, you might have to do an update, but that’s very quick. Other than that, you just need a microphone like the one that’s built into my laptop.

Crisi: Babbel has been online since January 2008. Why did it take so long to add this feature?

Thomas: We needed the new Flash Player 10.1 because before that it wasn’t possible to do audio processing locally. It would have been necessary to either send all the audio to the server for analyses or to use a custom browser plugin.

Crisi: What’s wrong with a custom browser plugin?

Thomas: First of all, you have to install new software on your computer. And then you have compatibility issues. There are some rare solutions that offer real-time speech recognition in a browser plugin, but most of them won’t work on your Mac and none of them are compatible with all browsers. Flash is already there, the plugin works fine and it’s available for all platforms.

Crisi: How about the iPhone? You can’t use Flash technology on that platform, can you?

Thomas: No, but the Babbel iPhone apps work natively on the iPhone anyway.

Crisi: Natively?

Thomas: The Babbel apps are built specifically for the iPhone and don’t need a browser or plugin to work. That’s called a “native” application. We can build our algorithm directly into the app.

Crisi: That’s not related to Native Instruments, the software company you used to work for?

Thomas: (laughs): No, not directly. But for being an audio software company, Native Instruments definitely is a great name because the software works natively on the computer.

Crisi: I guess we don’t have to understand that completely. But speaking of audio software: has your audio expertise (along with that of the other Babbel founders) been crucial for this new feature or is it something entirely different than building DJ tools?

Thomas: Both. Of course working on beat detection and time stretching for music and building a speech recognition tool are two different things. On the other hand, we couldn’t have done this in-house without our background.

Crisi: So who actually implemented the new feature?

Thomas: Most of it was done by Toine Diepstraten, one of the Babbel founders. He and I started working together on audio software in our first company, d-lusion, more than 10 years ago. Toine is one of the best developers and audio specialists I’ve ever met. It’s fantastic to have him on board for this project. He did have to do quite some research but without his expertise, this would never have been possible. But this way we have state-of-the art technology that can compare with any other implementation.

Crisi: You sound very convinced

Thomas: From a technical point of view, this is a great piece of software. We actually got some recognition from Adobe, the makers of the Flash Player. They were pretty impressed by our solution.

Crisi: Will this be a focus for Babbel from now on, or do you plan to work on other types of features?

Thomas: It is a very important feature because now we can do everything online that traditional e-learning software can do locally. And we don’t need installation or updates and we have a very lively online community that goes together with the self-directed learning…

Crisi: But?

Thomas: It’s important but it’s not the end. We’ll keep working and adding new features.

Crisi: Can you say what’s next for Babbel?

Thomas: Sorry, but for that we’ll have to turn off the mic.

Crisi: No problem.
–

When visiting these places, a translator could definitely come in handy. There is a new iPhone app which could have been a wonderful solution, but it comes with a catch: The iSpeak application ($2 for each language) offers translation from English to Spanish, French, German, Italian, Portuguese, Dutch, Polish, Swedish and vise versa – you type in a sentence, and it translates it and “speaks” the translation out for you. The only problem is that iSpeak relies on the Google translation engine – meaning you have to be connected to the internet. Which you may not be as a tourist without a contract with a local provider (or without wireless/WLAN, near the Victoria Falls for example).

But also as a bibliophile who’s packed up and crossed oceans for good more than once, I can attest to one of books’ major detriments: weight. So my interest was piqued to hear yesterday that Google has just made 1.5 million ebooks from Google Book Search available on mobile devices.

Now, as of now these are only books in the public domain, meaning pretty old stuff; I can’t say that I often have the urge on a train ride to work to peruse say, Beowulf. Also the transfer of older scanned books to text for easier reading on a cellphone can often result in a bit of a verbal mishmash, as the LA Times notes.  But it seems that Amazon.com is also on the case. They announced simultaneously that they are now working on making contemporary and out-of-print titles that are already digitized for the Kindle e-reader for access on mobile phones as well.

In the meanwhile,  for those too impatient to wait to read the latest airport novel without all the fuss of pages, Amazon is scheduled to unveil the latest version of Kindle on Monday.

The Victoria University press office speaks of Sulzberger’s work as a “revolutionary approach” – but isn’t being exposed a language and learning it bit by bit the most well known way to learn a language anyhow?

(via WaterCoolerGames)

The following statement of Kristian over at web-translations made me wonder. He was writing about how much is too much in language learning, and ends with the following: “As for me, well, I speak 6 languages…English, French, small talk, MSN speak, some basic programming languages (do they count? They should as they have strict syntax like any other language) and, of course, the language of Luurve.”

Yes, what about “code”? Does it count? There are several dozens of important programming languages and many more derived dialects plus some pidgin codes. And one would think, well, you don’t speak them literally, you hack them into the keyboard. But there are stories about conferences with lots of tech geeks where insider jokes and whole presentations consists just of spoken code. Over here for example, some programmer wishes to speak code.

“I was having a conversation with my boss a week ago, and I had a lot of trouble expressing myself. I think the reason I can’t communicate is because I spend 14 hours a day programming. When it comes to code, I’m multilingual. But for speech, I’m absolutely useless. “Here, this.. thingy.. goes.. here.. into.. this.”. A co-worker came into my office an hour ago, and asked me ‘Should I be.. doing.. this.. thing?’ and then wrote some code on the whiteboard. My response was ‘No no, you should use the run-time class macro.’. The ‘run-time class’ macro? That’s how it came out in speech, but what I said was ‘RUNTIME_CLASS macro’.”

Though it is English, many people won’t understand the meaning of “runtime_class”. Yes, Kristian is right, there is a syntax behind it, but it follows another logic than the spoken language that the code is based on. Karthik Hariharan gives a good explanation in a comment to this post, why for him code doesn’t count as a “real” language: “I think that there’s a fundamental difference in the usage of a programming language versus a spoken/written language. A programming language is a tool to accomplish a task, while a spoken language is a medium for communication.”

I’ve always found it curious that the Americans have no centralized institution which establishes the end-all be-all of language. I mean, something along the lines of the German Rechtschreibungen, grammars that all of which incorporated a rather catastrophic spelling reform mandated by an official agreement between German-speaking countries in 1996. Or the Real Academia Española (the Royal Spanish Academy) which purports to maintain propriety, elegance, and purity in the Spanish language, and consistently has conferences all over Spain and Latin America deliberating which words are worthy of inclusion. The North American language, however, is a bit federated, you could say… if not Balkanized.

For the Brits, one of the closest things to language royalty – along with Oxford’s, of course – would be Collins’ Dictionary, which has recently gotten positively ruthless in cutting words it deems obsolete. The Times along with other linguistic luminaries have taken up the case to save “endangered” words from institutional oblivion, by using them in public, and so reviving them.

Meanwhile, an English website Wordia is taking a grassroots, let’s say Yankee, approach to the matter, redefining the dictionary itself by acknowledging the diversity and subjectivity of words. Not surprisingly, seems Rupert Murdoch is involved. Individual users upload their own video definitions of whatever word they choose, a “democratic” spirit that was recently critiqued by the Independent.

Above see one bloke’s take on “ineffable”.

After 4 years of field-testing, the MILLEE Project hopes to go big time in India. MILLEE - Mobile and Immersive Learning for Literacy in Emerging Economies – grew out of Matthew Kam’s doctoral thesis at the University of California, Berkeley. His idea was to use phones and educational computing to improve English language learning for children living in rural areas and in urban slums in India. In a recent interview at shareideas.org, Kam talked about teaching 8 to 14 year old Indian kids using mobile phones:

One particular challenge was the time and energy it took us to explain our earlier games to the kids. The games were unconsciously Western in their designs and didn’t match the expectations and experiences that rural kids have about games. We therefore studied the traditional village games that they play every day, and designed our subsequent games based on these village games. This has made a big difference in making our game designs more intuitive.

For anyone interested in game-and educational design, there are quite a bit of research papers available on their website. MILLEE is now in talks with mobile-phone companies about a possible sponsorships, and even are hoping to get funds from an Indian government program. US- telecommunications company Verizon has already provided them with seed money for investigating how to target Spanish-speaking immigrants in the U.S with their concept.