Volley Studios - Local Multimodal LLM

Distribution Product Discussion
1

Local Multimodal LLM is a native Win64 Unreal Engine 5.8 C++ plugin for building private, offline AI-driven characters. It turns player speech or text into locally generated and voiced NPC dialogue without requiring a cloud inference service.

Youtube Demo:

Metahuman demo level with multiple model configurations. (Demo level assets are not included.)

GitHub Equivalent Release:

Github open source code, documentation, and coding agent integration.

User Guide:

Direct link to the github user guide, additional documentation also included inside that folder.

The Starter Core package includes a preconfigured Gemma 4 E2B text model, Parakeet speech recognition, Pocket TTS, four starter voices, CPU and Vulkan inference backends, matching native runtimes, documentation, and applicable licensing notices.

Core features

  • Local GGUF inference through llama.cpp

  • Separate memory and context sessions for multiple characters using one loaded model

  • Streamed subtitles and sentence-level speech generation

  • Push-to-talk and optional voice-activity detection

  • Native Parakeet speech-to-text and Pocket text-to-speech

  • Character sheets, grounded world context, conversation memory, relationships, and controlled dynamic lore

  • Typed and allow-listed Unreal function calls validated by game code

  • Conversation routing using explicit targeting, facing direction, conversation state, and proximity

  • Contextual speech vocabulary, rollback, configurable guard patterns, and provider-neutral speech interfaces

  • Blueprint components and a documented public C++ API

  • Packaging validation, checksums, model notices, and an AI-agent integration guide

Getting started

Enable Local Multimodal LLM and Unreal Engine’s Audio Capture plugin, compile the project once, and add the Local LLM, Microphone, and Text To Speech components to a persistent actor. Load the included model and create one character session for each NPC. Complete Blueprint and C++ instructions are included above in the documentation or the included starter pack.

Technical scope

  • Unreal Engine 5.8.1

  • Win64 development and packaged games

  • CPU and Vulkan included

  • Optional CUDA accelerator distributed separately

  • No network connection required at runtime

  • No Python runtime required

  • Source code and documentation included

  • Experimental vision API available with a developer-supplied matching projector

Disclosures:

  • Local model output is nondeterministic. Developers remain responsible for authoritative game state, validation of mutating actions, and any additional content moderation required for their intended audience.

  • The demonstration environment, MetaHumans, animations, facial-animation setup, Qwen, and NeuTTS are integration examples and are not included. Original plugin source and documentation are licensed under Apache 2.0. Bundled models, voices, and third-party runtimes retain their respective licenses and notices.

  • Requires Unreal Engine’s built-in Audio Capture plugin for microphone input. It is automatically enabled by the supplied .uplugin descriptor.

  • Development used an AI coding assistant under human direction and review. The package does not include AI-generated visual game assets, characters, environments, or demo scenes.

  • Pocket TTS v0.1 initializes one reference voice per Text To Speech component. Use one component per distinct Pocket voice reference; dynamic single-component reference switching is planned.

Contact me through my email: volleyballersvr@gmail.com or by commenting on my YouTube page if you need assistance (I'll get back as soon as I can). Integration video coming soon if there is interest.

Created by Volley Studios.

2

Let me know if anyone is interested in this project. It’s free (Apache 2.0). And my goal is to enable open-ended RPG conversations in which characters can respond according to their identities, knowledge, relationships, conversation histories, and the state of the game world without sending dialogue or microphone audio to an external service.

3

It is interesting , i think for ai newbies it may be scary all those terms and for advanced users the latency and memory may be their concern, what hardware is the minimal intended ? and is llama running async while inferencing ? the demo is great !

1 Like
4

Thank you! I think it depends on what sort of demands your game already has, but I’d estimate like a RTX 3060 (desktop) or equivalent should run close to real-time with a final product (that’s just a guess, I’ll have to rent some time on a lower-end GPU or maybe someone will tell me if they use it). The worse your hardware, the more latency. I have it run CPU inference as a fallback, so you don’t even need a GPU technically to run my specific plug-in, but in that case, it’d take like 15 seconds or something for a response.

Yes, llama.cpp inference runs asynchronously on it’s own dedicated thread. The generated text from llama.cpp is streamed back to UE without blocking the game thread. It does compete with the game thread for the GPU but the inference thread is created at Unreal’s TPri_BelowNormal thread priority. So it should defer to the game thread.

The starter pack should be pretty much plug and play for beginners. They can use my blueprint picture to start an AI conversation with a character (STT->TTS), I tried to make that blueprint as minimal as possible. But I also recommend using a coding agent (AI) to help. I left explicit instructions for AI agents, if you have a coding agent configured in your project.

I’m working on a version 2 right now to make the blueprints just a tad simpler, I want it to be extremely easy to plug and play. That’s why I included configured models inside the plug-in.