Announcing LLMAI Plugin - AI Realtime Text, Voice, and function calling!

Distribution UE Marketplace
, , , , ,
1

LLMAI_Titlescreen_001
LLMAI_Titlescreen_0011920×1080 123 KB

We just dropped something pretty exciting, at least to me, hopefully others will find it useful and or interesting.

Video Demo here

LLMAI, a plugin that lets you have real-time AI conversations and function calling directly in Unreal Engine 5.6 at runtime.

What’s the big deal?

  • Real-time voice chat with OpenAI (yes, you can literally talk to your game!)

  • Function calling system - AI can actually control your game, spawn objects, change settings

  • Two complete demos including an AI-controlled arcade game that responds to voice commands

  • Production-ready with full source code and comprehensive docs

Perfect for prototyping AI NPCs, voice-controlled interfaces, or just having fun conversations with your projects. The AI can even play games for you!

Check it out: LLMAI Plugin | Fab

This version works with OpenAI Realtime API with all the voices, we are working on various AI provider support, along with local hosted integrations, so stay turned!

Questions, feedback, or cool use cases welcome!

Would love to see what the community builds with this.

Cheers!

2

image
image1763×1219 135 KB

We’re on our way to adding LocalAI realtime API capability.

Unfortunately the current version of LocalAI’s realtime is basically just a stub and nowhere near complete, so we’re working on updating that too! With the first step pictured above, we have connections properly established, some more error checking, and text mode communications (with qwen3-4b thinking model on the backend, but ofcourse you can configure LocalAI to your own needs)

This will allow for much greater scalability and versatility since LocalAI can run on client PC or network infrastructure as needed.

Stay tuned!