Barbie's digital makeover

Toy manufacturer Mattel has joined forces with San Francisco startup ToyTalk to develop an interactive Barbie doll that has a microphone, a speaker and Wi-Fi capability.

Thanks to speech recognition software developed by ToyTalk, the new Barbie will be able to have two-way conversations with children by transmitting audio data over the Internet. But critics fear this may lead hackers - or worse - straight into children's bedrooms. The Wi-Fi capability is especially controversial in light of the recent Samsung smart television spying debacle.

DW: Hello Barbie can listen to children's conversations and adapt to them over time, referencing things discussed in the past. Can you explain how this technology works?

Oren Jacob: Let's say you're playing a modern mobile game and the game's system remembers your state, the level you're at, or the way you set up your character - really most things mobiles do today from Candy Crush to Angry Birds. If you take that kind of similar concept of remembering a child's favorite color, ice cream, cupcake or favorite place they went on vacation, those are things that we can write into the conversation, and reference later on.

Barbie is asking questions about children's aspirations and what they're excited and dream about, while also responding and staying in character.

So the kid is having this conversation and the information is being transmitted over the Internet via a Wi-Fi connection?

When the child presses down Barbie's belt buckle and says, "Hi Barbie, how are you?" the microphone is then live and the audio is transitioned over Wi-Fi in someone's house to the Internet connection. This then goes to our servers in the cloud.

And ToyTalk servers get that audio and try to recognize it as speech. We do our best to convert what the child says as an audio and process that as a sentence and understand the language being spoken, then decide that Barbie is in a good mood and should say something back. And the audio of Barbie streams back down the same Internet connection to Barbie's speaker and then you hear Barbie say to you, "Hi I'm fine, how are you today?"

So when the child is talking to the doll, the speech gets recognized and the algorithm, the software, basically generates the response?

The software chooses which response of the ones we've written and already recorded that we're going to send back down. And that makes it very different from things like Siri and Google Now and Cortana, which search the open Web and generate an answer with text to speech, which kind of sounds like a robot, and talk back to you. We don't do that.

Everything Barbie says has been written and recorded before, but the algorithms on the cloud decide at that moment which responses to send back to this particular Barbie in this particular conversation.

What all is being kept or stored on the cloud?

The audio recordings of the children themselves are stored so we can build better recognition for kids going forward, and that's the data that parents have control over. When you start using Hello Barbie, a parent creates an account that's verified through an email verification loop, and that password-protected place is where all the data is stored.

As a parent, you can hear what your child said last week to Barbie, and most parents do that and have a great time and laugh their heads off because it's super-fun to listen to what kids have to say.

What about concerns about hackers, or even worse. Worst-case scenario: a pedophile hacker. Is that at all possible?

Anything is possible - let's talk about probable. None of Barbie's content is stored on the device. Barbie's content is stored on our cloud services, and that's a relatively secure place that would require a pretty sophisticated method to get in between. All transmissions back and forth from the doll to the cloud are on secure connections, with https protocols.

And the parents are in control of that data at all times. So anything that doesn't feel right, they can scrub it all clean and it's gone. That's a pretty secure system design overall. Data privacy and security is our primary concern.

What's the next level, after this gets rolled out and is working … what's the next move?

There's a lot of work to really build what amounts to synthetic conversation. That's the reason we founded the company, to explore that as a creative endeavor. And this is the first major product coming out in that regard.

Getting to languages beyond English is going to be the next major step forward: Mandarin, Spanish, German, Hindi, Hebrew … we could do 40 languages over the years to come.

The big technology companies that work in the United States have done that well for adult speech, which is why we have such sophisticated assistance on mobile devices today. But they haven't done that for kids. That language is different, so developing that for children will take a long time to come.

You mentioned developing the character. What is Barbie actually like?

[Laughs] Barbie is fun, she is eager, she is certainly curious. And Barbie has some opinions too. It's been fun seeing the writing team at Mattel really start to dig into the character. A modern take on who Barbie is and how she engages young girls and boys in a positive and inquisitive way is pretty exciting to see being developed right now.

Oren Jacobs is the CEO of ToyTalk in San Francisco. Hello Barbie is expected to cost around $75 (67 euros) and will be released in late fall 2015.