Framework for Advanced Natural Tools and Applications with Social Interactive Agents (FANTASIA)

Hello, everyone.
My name is Antonio and I am a researcher working on Human-Machine Interaction and Dialogue Systems. I would like to present you the open source plugin I am publishing on the basis of the work I have been doing with UE4 in the last years at the University of Padua and at the University of Naples Federico II (Italy). The plugin is named Framework for Advanced Natural Tools and Applications with Social Interactive Agents (FANTASIA).

FANTASIA is a re-engineered version of a set of tools I have been developing for research purposes and it aims at easily integrating in UE4 powerful AI services and libraries for interaction management and knowledge representation. These are exposed to Blueprints to help people concentrate on the development of interaction models and reduce the time spent on technical issues as much as possible. The first version of the plugin will include the following features:

Access to the Automatic Speech Recognition service provided by Microsoft Azure
Access to the Text-to-Speech service provided by Microsoft Azure
Access to Natural Language applications developed in Azure LUIS
Access to the Text-to-Speech service provided by Amazon Polly (with lipsync data support)
Access to instances of the Neo4j graph database

I have a lot of prototypical code to restructure and include in future versions of the plugin to provide machine learning capabilities (both local and cloud-based), probabilistic decision systems and semantic 3D capabilities, among other things. In general, I hope this contribution will integrate with the efforts made by Epic towards the creation of Metahumans by providing the infrastructure needed to make them behave in a believable way, other than look great as they already do.

The plan is to publish the first public beta of the plugin on March 8th 2021. I am currently recording a small set of video tutorials that will show how to use this first set of tools to close the loop in the creation of a very simple humanoid conversational agent. Meanwhile, I am setting up some social channels to keep people informed about the development process. If you like the idea, you may visit the FANTASIA website or you can check out the Youtube channel, which currently contains a promo video of the plugin.

I hope this contribution will be able to help other people in the academy to leverage on the tools provided by Epic to make UE4 a powerful environment for the development of Embodied Conversational Agents. I also hope the provided interfaces will help the general community to work more easily with powerful tools that really have the potential to give rise to exciting new applications!


Exciting, keep us updated!

Hello, everyone.
After some final polishing and setting up of the social channels, the first public beta version of FANTASIA is online. You can download it from the Github repository here

A presentation video showing the installation process is on the Youtube channel

By the end of the week I will publish another video to show how to integrate a graph database (Neo4j) in Unreal using the Blueprint nodes provided by FANTASIA. I will also work on the documentation side to provide more insight about the plugin. For some preliminary information, you can check the “Features” section on the dedicated website

Have fun!

Hello, everyone.
I just posted a new video tutorial showing how to access graph based knowledge in Neo4j from UE4 using FANTASIA. In the next videos, I will show how to use Automatic Speech Recognition and Natural Language Processing from Microsoft Azure to guide data extraction. Here’s the link

Check it out and have fun!

Hello, eveyone!
A new video tutorial is online on the Youtube channel. This one shows how to combine the Microsoft Azure Automatic Speech Recognition service with parameterized queries in Neo4j to extract data in a flexible way from UE4. This is another step in the series of tutorials that will present how to build a simple Embodied Conversational Agent in UE4 using the FANTASIA plugin. Check it out and have fun!

Hello, everyone!
A new video is online on the Youtube channel. This tutorial shows how to create a basic application that allows streaming audio from UE4 to Microsoft Language Understanding Intelligent Service (LUIS). This way, you can use AI to analyse speech transcriptions and extract both user intentions and named entities (like movies and actors). The video also shows how to query Neo4j using LUIS feedback to extract information, so that you can easily query your data using Natural Language Understanding.

Happy Easter and have fun!

Hello, everyone!
The final video of the Basic tutorials series is online! In this video, I will show how spoken commands interpreted by Microsoft LUIS generate a query to Neo4j and how a synthetic voice is obtained with Amazon Polly. Audio and lipsync data are, then, used to animate a Genesis 8 3D character in the Unreal Engine!

From now on, I will be publishing advanced tutorials to cover functionalities and aspects that were not included in this first example. Also, I will be working on linking FANTASIA with Metahumans!

I hope you will enjoy the video. Have fun!