Cactus AI Framework Plugin for UE5.6 for Local LLM & VLM as Generative AI

Development Programming & Scripting C++
,
1

Cactus AI is a framework to run local LLM, VLM and TTS on your computer. It is especially good for low powered, edge devices such as mobile phones but it works on desktop platforms, too. This is one of the reasons why it is a good choice for UE5.

So, I integrated it to UE5.6 and share it with you guys.

Features:

  • Single question
  • Multi-turn conversations
  • Image processing
  • Conversation export for persistent conversations.
  • Blueprint exposed

Use Cases:

  • You can talk with your NPCs just like humans.
  • Digital twin / factory simulation projects that can analyze runtime generated data.
  • AR & MR Projects that can understand surroundings. (It will be good with Quest 3)

Roadmap:

  • Currently it supports Windows but I will include Android, too.
  • It is work-in-progress state. Expect some problems !

Limitations:

  • Cactus AI doesn’t support buffer import for processing. It requires actual files. (I tried virtual file mapping and it didn’t work) So, you need paths. I shared another plugin to export images. So, you will be fine and also I opened an issue about it. They said they will look at it with next updates.
  • I don’t have Apple and Linux devices. So, I will only support Android and Windows
  • But library itself supports Apple ecosystem.
  • There will be no support about UE4 or older versions than 5.6.

Dependencies

I can help you if you have problems about Unreal side but for framework related problems, ask them to Cactus’ github repo !

Plugin have sample blueprints in plugin’s content folder. You can look at them to understand workflow.

1 Like
2

Hi! I liked a lot your project, I´m trying to learn more about AI. Do you have any tips from where to start?

3

If your question is about AI itself (such as model choosing, fine tuning and etc.) you should look at Cactus AI’s own discord channel and Github page. I am not in their team. I am just an acquaintance of them.

If your question is about Unreal implementation, you can ask me. But before that I suggest you to look at plugin’s content folder. I put some samples.

Other than that “how can I start to AI” is a vague expression and depends on what you want to do. For example do you want to create AI integrated products or do you want to work on AI itself ?

4

I wanted to make some NPCs that remembers conversations etc. I saw the sav file, would be great. I´m trying to install the plugin right now, created a new third person template project (5.6.1). It asked to compile and I got an error. Will try again

image
image406Ă—156 2.55 KB

Update: It compiled! :smile: I just updated the visual studio

5

To be more specific, I wanted an companion npc to be believable. To be able to input some more information (or unlock) in it. Also I wanted to activate it with triggers in the world, so the NPC could “react” to some things without being asked by the player. Another thing is that It has to be light for the processing, since there is aleady a game running. I don´t know if I´m dreaming too big, but that´s what I was thinking

6

  1. I already integrated “remembering conversation history”. You need to just need to use import and export functions.

  2. You can implement “interacting with world without being asked” feature on your own. Use line trace or collision box to detect surroundings. And based on what they see, you have to generate a prompt. There is no template for that. it depends on your imagination.

    For example: Greeting command
    If Hit Actor = player, get its experience level and use that value in a prompt string with append to have different greetings based on player’s level.

    You don’t need to use image processing to understand environment. It will be stupidly expensive and unnecessary.

  3. To trigger something from AI:
    There is no “runtime” agent like way because there is no constant or predictable way to do it for every user. You can send an email or do some editor side operations, because workflows, classes, functions and buttons are same for everyone. For example, import something to editor has a specific workflow. only variable is paths. But not in runtime. Runtime mechanics are subjective to its developer and it causes ambiguity.

    Let’s assume we solved that. We developed something special only for specific project. You need to be sure that your model will give you correct answer without hallucinations everytime.

    If you can solve them (I mean, all function and variable names are ready, LLM is fine tuned… and these are not my or plugin’s job), I have another plugin to allow you to call a function, set a variable or get its values with only string without actual variable reference.

    For example, your generated text should be like this.

    Call “APlayer::BowAnimation”, set “bending” level to “20”, “print” “Greetings Player” on “UNPCWidget::Conversation_Text”

    If you have that string and objects like that, AI can even trigger things (for example animations) on its own but making sure that LLM will generate exactly these strings everytime (with fine tuning or prompts) is your job.

In short, sky is the limit and there is no automatic way. Just mess around.

7

Thanks a lot! It´s been some busy days studying ai.. very nice!
It´s working, the plugin is working great! great work! The responses are taking a while, around 25 seconds depending on the question. I think the model is not the best for an NPC, I downloaded another one with 1B from huggingface, but it keeps talking about my case or something like that. And now I´m thinking about what would be the best way to make the model talk like the character. I´m searching a chat model in the huggingface, but no luck until now. But I´m happy with the progress!

image
image1920Ă—824 145 KB

8

You can’t get too much speed with complex personality prompts on local libraries that run on end-user hardwares. You guys get used to Gemini, ChatGPT and Claude. They use billion dollars worth hardwares.

Of course there are ways to make local LLMs faster. but there is nothing to do on Unreal’s side. Cactus team can do it if they see fit with their scope. For example, they can increase usages of hardware specific instruction sets or some advanced hardware accelerated calculation systems.

But these are not Cactus Framework’s job. This is a simple, easy-to-use and implement library solution especially for edge devices when privacy required or there is no network connection.

More dependency means more weight and complexity.

Also, UE5 is very problematic about third party libraries. Because sometimes already included third party libraries cause clashes and we can’t update them without engine modifications. So, you wouldn’t be able to use it.

And I will be honest with you. If I needed to integrate that much libraries to UE5 such as CuBlast and solve all the problems, I wouldn’t share it as a free & open source project. Because my goodness has its limits :rofl:

You can increase the thread counts, though but you have to be careful with it. Because Unreal already use game thread, render thread, network thread (if your project is multiplayer), audio thread and maybe you can use some allocated threads (such as FRunnables) in the future. So, if you have 8 core 16 thread computer, don’t give more than half.

Also, you can use more advanced models but they need more RAM.

If your game requires clever, philosophical conversations, local LLMs might not be good for you (or at least not with every hardwares). Design your project in a way that weirdness of AI halucinations are good for you. Consume and use that… madness.

9

Could you make a video guide for us special folk
Using input text to NPC text and text to voice Output…in blueprints