Cactus AI is a framework to run local LLM, VLM and TTS on your computer. It is especially good for low powered, edge devices such as mobile phones but it works on desktop platforms, too. This is one of the reasons why it is a good choice for UE5.
So, I integrated it to UE5.6 and share it with you guys.
Features:
- Single question
- Multi-turn conversations
- Image processing
- Conversation export for persistent conversations.
- Blueprint exposed
Use Cases:
- You can talk with your NPCs just like humans.
- Digital twin / factory simulation projects that can analyze runtime generated data.
- AR & MR Projects that can understand surroundings. (It will be good with Quest 3)
Roadmap:
- Currently it supports Windows but I will include Android, too.
- It is work-in-progress state. Expect some problems !
Limitations:
- Cactus AI doesn’t support buffer import for processing. It requires actual files. (I tried virtual file mapping and it didn’t work) So, you need paths. I shared another plugin to export images. So, you will be fine and also I opened an issue about it. They said they will look at it with next updates.
- I don’t have Apple and Linux devices. So, I will only support Android and Windows
- But library itself supports Apple ecosystem.
- There will be no support about UE4 or older versions than 5.6.
Dependencies
-
Library itself is here. I did some changes in order solve MSVC related problems and created pull request. You can solve them if you want to build library by yourself and good at C++ but probably you won’t need because I will update it if there is an important update.
GitHub - cactus-compute/cactus: Cross-platform framework for deploying LLM/VLM/TTS models locally on smartphones. -
nlohmann json: Already included. Framework requires json to format prompt. Actually we can do it with Unreal’s json implementation (for example LLM side) but if we need to ask something to Cactus team, using generic codes will makes everyone’s job easier. For this reason I used nlohmann for VLM sideç But you can delete it if you want. In that case just remember to implement your own json system. You can use LLM side as sample.
-
Extended Variables: One of my other plugins. It allows you to extract texture buffers and export them. GitHub - Frozen-Projects/ExtendedVars: UE5 plugin for converting some variables to each other, exporting bitmaps, runtime font load, getting widgets as textures and getting color arrays of textures.
I can help you if you have problems about Unreal side but for framework related problems, ask them to Cactus’ github repo !
Plugin have sample blueprints in plugin’s content folder. You can look at them to understand workflow.