Project: Autonomous Lifelong learnIng intelLigent Systems

Acronym	ALLIES (Reference Number: ANR-17-CHR2-0004)
Duration	01/01/2018 - 01/01/2018
Project Topic	Standard machine learning systems require massive data and huge processing infrastructures, but the main limitation to their spreading comes from the need of the empirical and rare knowledge of an experienced data scientist able to set and adjust their behaviour over time. The ALLIES project will lay the foundation for development of autonomous intelligent systems sustaining their performance across time. Such unsupervised system will be able to auto-update and perform self-evaluation to be aware of the evolution of its own knowledge acquisition. It should adapt to a changing environment by following a given learning scenario that balances the importance of performance on past and present data to avoid unwanted regression. Such systems could not be developed without adapted metrics and protocols enabling their objective and reproducible evaluation. This evaluation should continuously assess the performance on the given task and quantify the effort required to reach it in terms of unsupervised data collected by the system and of interaction with humans in the case of active-learning. The ALLIES project will develop, evaluate and disseminate those metrics and protocols. They will be available to European actors via an open evaluation platform dedicated to reproducible research. An evaluation campaign and a workshop will be organised to engage the community on this path. By publicly releasing the evaluation protocols and data, by setting up a dedicated evaluation platform and by developing autonomous systems for two tasks: machine translation and speaker diarization, we believe that the ALLIES project will boost the development of intelligent lifelong learning systems in Europe.
Project Results (after finalisation)	The main outcome of the ALLIES project consists in producing and validating an evaluation method (metrics, protocols, environment) for autonomous systems. A clear evaluation infrastructure for autonomous systems evaluation: metrics, protocols and evaluation data will be released to the community to support the development of autonomous systems in machine translation, speaker diarization but also other field which can benefit from the experience acquired during this project. ALLIES Reproducible Online Platform for evaluation of lifelong learning intelligent systems: we intend to iterate on the design of the existing BEAT platform to support iterative workflows with support for GPU computing. This revised platform will allow for outside parties to visualize, keep track and reproduce results produced by ALLIES project. The BEAT Platform will also function as a display for project progress and will contain freely accessible material allowing not only reproducibility, but also exploration, cross-pollination and certification of results obtained through the project. At the end of the project, the platform will include data, metrics, protocols to evaluate speaker diarization and machine translation systems and will constitute a reference for further extension to other modalities. The outcomes from the evaluation campaign and workshop organized for those two modalities will benefit the entire machine learning community as it will be the first attempt to evaluate system evolution across time. The three elements above aim at structuring the machine learning community and provide guidance and feedback for the development of autonomous systems. Two use cases of autonomous systems in action: self-evolutive machine translation and speaker diarization systems will be developed and evaluated across time within the platform presented above. Those two tasks addressed by ALLIES have been chosen for two reasons. First, they require very different algorithms and evaluations that will demonstrate the genericity of the proposed evaluation framework. Second, language technologies are expected to encounter a major economic development in the coming years and the development of autonomous systems for those tasks will benefit European companies and institutions in their development.
Network	CHIST-ERA III
Call	CHIST-ERA Call 2017

Project partner

Number	Name	Role	Country
1	University of Maine	Coordinator	France
2	National Metrology and Testing Laboratory	Partner	France
3	Polytechnic University of Catalonia	Partner	Spain
4	IDIAP Research Institute	Partner	Switzerland