My name is Antonio and I am a researcher working on Human-Machine Interaction and Dialogue Systems. I would like to present you the open source plugin I am publishing on the basis of the work I have been doing with UE4 in the last years at the University of Padua and at the University of Naples Federico II (Italy). The plugin is named Framework for Advanced Natural Tools and Applications with Social Interactive Agents (FANTASIA).
FANTASIA is a re-engineered version of a set of tools I have been developing for research purposes and it aims at easily integrating in UE4 powerful AI services and libraries for interaction management and knowledge representation. These are exposed to Blueprints to help people concentrate on the development of interaction models and reduce the time spent on technical issues as much as possible. The first version of the plugin will include the following features:
Access to the Automatic Speech Recognition service provided by Microsoft Azure
Access to the Text-to-Speech service provided by Microsoft Azure
Access to Natural Language applications developed in Azure LUIS
Access to the Text-to-Speech service provided by Amazon Polly (with lipsync data support)
Access to instances of the Neo4j graph database
I have a lot of prototypical code to restructure and include in future versions of the plugin to provide machine learning capabilities (both local and cloud-based), probabilistic decision systems and semantic 3D capabilities, among other things. In general, I hope this contribution will integrate with the efforts made by Epic towards the creation of Metahumans by providing the infrastructure needed to make them behave in a believable way, other than look great as they already do.
The plan is to publish the first public beta of the plugin on March 8th 2021. I am currently recording a small set of video tutorials that will show how to use this first set of tools to close the loop in the creation of a very simple humanoid conversational agent. Meanwhile, I am setting up some social channels to keep people informed about the development process. If you like the idea, you may visit the FANTASIA website or you can check out the Youtube channel, which currently contains a promo video of the plugin.
I hope this contribution will be able to help other people in the academy to leverage on the tools provided by Epic to make UE4 a powerful environment for the development of Embodied Conversational Agents. I also hope the provided interfaces will help the general community to work more easily with powerful tools that really have the potential to give rise to exciting new applications!