Project: Interact naturaly and emotionaly with virtual environments

Nowadays, computers have taken an important place in our day-to-day life. Software meet our common needs, varying from making bank operation to music listening. However, during the last few years, solutions have been developed to serve deeper expectations. One striking example is the growing number of social networking platforms, such as FaceBook, Twitter, etc. Those new kind of applications allow people to meet each other, keep in contact with friends, or even share with the entire world. Along with social networks, another kind of applications has an exponential growing market place: E-Learning solutions. Those applications intend to meet various educational purposes, should they stand in the framework of academic or lifelong learning._x000D__x000D_There exist different supports for social networks. Virtual worlds are a very new way for socializing. They allow users to embody an avatar evolving in a three dimensional representation of a real or imaginary place, in which they usually can meet other users and interact with them. In this case, such applications digitally extend users social life. E-Learning solutions can appear as simple as forms to fill-up or be developed using technologies from the Computer entertainment field. The later type of E-learning solutions is known as serious games. They aim at merging educational content in a gaming design, allowing users to actively learn and improve their skills._x000D__x000D_Both, virtual worlds and serious games, aim at facilitating work meetings, training employees, and more generally learning. It should be stressed that lifelong learning has been acknowledged as a strategic objective since “Europe faces challenges in adapting to globalization and the shift to knowledge-base economies” (Proposal for a Recommendation of the European Parliament and of the Council on “key competences for lifelong learning”, Commission of the European Communities, 10.11.2005). The Lisbon European Council in March 2000 stressed that “a European framework should define the new basic skills to be provided through lifelong learning: IT skills, foreign languages, technological culture, entrepreneurship and social skills”. This challenge has been regularly restated, most recently at the European Council of March 2005. _x000D__x000D_Virtual worlds and serious games offer a good technological answer to this challenge since they give users virtual experiences of real situations. If actual existing solutions reached a satisfying level of physical immersion, the next steps consist in providing users with a higher level of interaction both with other users and with virtual humans populating the digital environments. Today's applications lack of verbal and emotional interactions. Filling this gap would give the virtual experience a better realism. For instance, avatar's lips and face animation should be coherent not only with the phrasing but also with the emotional message (anger, pity, etc.). Thus, spoken interactions (in other words, dialogs) are an important aspect to focus on in order to improve users' experience. More precisely, synthesized speech and face animations should take into account verbal and non-verbal components (COly emotions) to fully represent speaker intentions. Allowing users’ avatars but also virtual humans to handle emotions will definitely improve the immersiveness of the virtual worlds and serious games._x000D__x000D_Artefacto has built a strong expertise on virtual worlds and serious games. Different solutions have been developed to manage virtual worlds, design serious games and merge both technologies. However, emotional aspects are still missing. Emo-Speech has been set-up to fill this lack. Artefacto, Acapela and two french research teams PAROLE and TALARIS from the INRIA LORIA Lab. gathered to collaborate on this challenge: how to transcript human emotions to give natural conversational capabilities to avatars and virtual humans? Emo-Speech will tackle this problem and provide a complete solution from sound and face capture to natural speech synthesis and natural face animation. The final demonstrator of the project will be seen as a set of plug-ins for Artefacto and Acapela solutions. It will enable_x000D_ * the efficient capture of a real actor’s speech (real actor's video + sound and images) and its reproduction with optional alterations,_x000D_ * the development of virtual worlds or serious games with autonomous characters having credible conversational capabilities,_x000D_ * the synthesis of realistic speech, either corresponding to that captured from real actors or to artificial conversational agents ._x000D_

Acronym	Emo-Speech (Reference Number: 5218)
Duration	01/04/2010 - 30/05/2014
Project Topic	The CO objective of the project is to develop an innovative way for users to interact in a natural manner with virtual humans: use speech and emotions.
Project Results (after finalisation)	In the context of this project, we proposed an innovative interaction scheme to interact naturally integrating an aspect that is missing in the computer based training software, namely emotional aspects. _x000D_To fulfill our commitments, we have proposed a common platform to develop and test the technologies. Artefacto has provided with an adapted version of a Serious Game, realized in 2010. _x000D_Integrated into that Game, Synalp (formerly TALARIS) Team has integrated a dialog management library that allowed first for training an autonomous agent to interact within the themes tackled in the Serious Game, and second for interacting intelligently with the users. Parole Team has integrated a voice recognition system to allow for natural interaction from the users. The characters of that particular could not be adapted easily to reach a realistic animation of the face using the Text To Speech technology developed by Acapela during the project. Hence, based on a second tool has been built by Artefacto to integrate the technology developed by Acapela. This development led to the creation of a Serious Game built on top of that TTS technology to provide our customer with a flexible management of the voices in their game. _x000D_All the Ps could promote the results of the project either through Publications in Journals/Conferences, demonstrations in Business/Scientific Conference, Workshop organized around the Dialog Management system, products commercialization for industrial Ps. _x000D_
Network	Eurostars
Call	Eurostars Cut-Off 3

Project partner

Number	Name	Role	Country
3	ACAPELA GROUP S.A.	Partner	Belgium
3	Artefacto	Coordinator	France
3	Institut National de la Recherche en Informatique et Automatique	Partner	France