Toyota YUI Project

Multimodal interactions are not a new idea as most of the people get used to the combination of a visual modality and a voice modality. The in-car infotainment system is one of the most common applications of multimodal interactions design. It makes use of several input and/or output modalities, including visual interaction, audio interaction and haptic interaction.

We synergize and capitalize on the strength of different modalities to create an intuitive and holistic car experience. It’s an immersive experience filled with things people can see, hear and feel. In that case, it requires an enormous amount of UX effort to make it happen altogether in a short period of time. The UX proportion aims to enrich the car experience and ensure driving safety without being overwhelming. The multimodality interactions are deployed in a diverse format of medium.

‍Sight
• Yui display: It visually shows different emotions or states of Yui, like being imaginative and eager or talking and awaiting.
‍
• In-vehicle lights: Embedded lighting strips on the roof and floor area use dynamic illumination to achieve human-machine interactions, including the light color change of manual and autonomous driving mode. The footwells will also light up to better indicate the passenger that Yui is talking to.

‍Hearing
• Yui speech: Yui will lead the conversation with the passengers and provide personally interesting information based on human input.
‍
• Music playlist: In collaboration with the digital music service provider, pre-set persona-based music playlists will be triggered across scenarios.
‍
• Sound effect: The environment-based sound effect serves as ambient noise to fill in the gap between speeches.

‍Smell
• Fragrance emitting: fragrance emitting systems will release relaxing and refreshing scents when passengers feel tired or worried. 

‍Touch 
• In-seat massage: the seating function is designed to provide a natural and gentle back massage to either keep the passengers awake or alert.

Designing a 30 mins car ride is much like designing a show that has a beginning, middle and finale. Starting from onboarding, Yui will guide passengers to experience the functionality of the car, introduce landmarks along the road and lead the conversation until the end. Scenarios have been updated for several rounds in terms of the order and content by the UX team, but the structure remains the same. In addition, to create a more personalized voice-based experience, we also come up with three types of persona based on different user input. Each time when users need to make a choice or give an answer, their feedback will lead them to a path that offers persona-based content. The complexity of scenarios and persona requires a lot of effort to clear things up with both writers and developers. And this is where UX Designers come into play.

Partnered with creative writers and other UX Designers, I translated part of the stories into flowcharts to make sure the logic behind the conversation is complete for further development. This kind of visualized documentation also helped me a lot when communicating with the whole team. When I found an incomplete path or a wrong condition, a flowchart was always a good start point and a reference. Below is a basic flowchart with a legend that I created as an example. We also used flowcharts for code implementation and QA testing later on. 

When designing voice-forward systems, how to make Yui talk in a natural human way is our priority.  As research shows female voices can increase acceptance and trust, the Text-to-Speech (TTS) uses a young woman’s voice across all three languages. Some high-level rules are set to create a friendly and approachable image of the voice agent. The English content is created by the UX writers first. Then the Japanese and Chinese scripts are developed based on English. We have vendors taking care of the basic translation. I create a language-specific guideline for the translators to understand the overall context, Chinese linguistic details and technology limitations of TTS.

When it comes to localization and speech tuning, rephrasing and polishing the scripts takes most of the time. There’re some reasons behind this. One is that without a certain background, the quality of the translation only meets the basic requirements. The other is due to the ephemeral nature of oral speech, the listeners can’t consume long sentences that carry too many messages. Contents that are exceeding the capacity of short-term memories will cause information overload and make people upset. Additionally, letting the TTS read scripts and listen to it is very different from yourself reading and listening to it. So the typical workflow would be like utilizing the Speech Synthesis Markup Language (SSML) and throwing marked contents into the tuning tool piece by piece. Then playing each sentence several times to either adjust SSML tags, such as adding <break> or restructure the whole sentence, such as breaking down big chunks into short ones.</break> 

Because of the limited resource in the project, I also played a QA role and did the Chinese scenario testing. Running scenarios one by one or altogether is another story than just listening to each sentence. Besides, this whole Yui tour is led by a voice agent feeding information along the way. And considering the linear and ephemeral nature of the conversation, salient pieces of information should be given upfront and repeated later if necessary.  So the balance between the delivery of messages and the flow of conversation is another key point for building a natural VUI. 

A few reflections on my working experience with the project:
‍
Ask as many questions as I can
Don’t be afraid to ask questions about things that you don’t know. It’s not you’re junior or fresh but it’s almost impossible to have someone knowing everything in the project. The best practice for me is identifying the right person and getting help from them. With the “get things done” kind of mindset, most people are willing to help you out. 
‍
Learn how your coworkers work
Knowing the work style of others and interacting with them in different ways make collaboration more efficient and quick. Someone would get to the point with just a few words while others may prefer a doc with details. I was learning by watching people at work during the first few months when the office wasn’t locked down. Then the situation changed rapidly and everything was online. 
‍
Take initiatives and ownership 
Sometimes, the responsibilities are not exclusively defined and someone needs to step out of the blurred line and do things. For example, there was no designated QA role either from TRI or TEMC side for the Chinese demo, so I took the responsibilities to do test runs.