I’m trying to build a genuinely usable local AI agent for Unreal Engine 5.8, not just prove MCP can call a tool.
My goal is to be able to talk naturally to an agent and say things like:
“Inspect this scene and block out two rows of buildings with a road in the middle.”
“Create a proper 50mm medium close-up of Antonio and visually verify it.”
“Look at this Sequencer setup and fix what’s wrong.”
I want the agent to inspect, reason, act, verify, and finish without me having to hand-train a rule for every new type of request.
My main machine is UE 5.8 on Windows with an RTX 4090. I bought a 64GB M1 Max Mac specifically to run local models as an AI appliance. The Mac is serving Ollama over the LAN.
Models I’ve tested include Qwen-family models and Ornith 1.5 35B-A3B. I currently have Qwen2.5-Coder-32B-Instruct Q6_K and Ornith Q6 available locally.
I’ve tried Claude Code, Qwen Code, Cline, OpenClaw, and my own custom harness.
Claude Code is designed around very large system prompts/tool contexts and is a poor fit for these 30–35B local models. The local models become slow or indecisive once the tool/context surface gets large.
Qwen Code can connect to MCP, but with large Unreal tool catalogs the model starts looping or spending too much time deciding what to call. I’ve also hit wrapper-format issues where Qwen correctly emits a structured tool call, but my harness fails to parse it and incorrectly marks the task completed with 0 MCP calls.
Cline connects and can use Unreal MCP, but with my local model it tends to over-plan, dump schemas, fall back to Python, and spend minutes talking to itself instead of acting.
I built a custom harness because none of those were usable enough.
The weird part is that my custom harness is exceptionally good at one narrow domain: camera placement.
My benchmark is:
“Create a new CineCameraActor named CAM_KIMI_TEST. Position and aim it to create a proper medium close-up of Antonio using his current position. Set the focal length to 50mm. Visually verify the actual camera render and correct it if necessary.”
The current harness does this in about 30 seconds, 6 MCP calls, 0 corrections, with measured visual verification. Nothing else I’ve tested comes close to it for this task.
So I know a local model can be useful when the wrapper/tool architecture is good.
The problem is everything outside that optimized path.
The current harness became a treadmill of:
ask a new type of question → model takes a bad route → add another rule/regex/tool guard → next request exposes another failure.
I no longer believe the model is the only problem. I think the general wrapper architecture is the bigger problem.
I audited three Unreal MCP projects:
- remiphilippe/mcp-unreal
- IvanMurzak/Unreal-MCP
- VibeUE
The architecture I’m moving toward is:
user request
→ agent interprets intent
→ capability search
→ retrieve only 2–4 relevant capability cards/tool groups
→ expose a small tool surface
→ execute one tool call
→ inspect result
→ continue
Camera work stays pinned to my existing optimized camera path and bypasses the registry completely.
I’m borrowing ideas from:
- VibeUE: capability/skill packs and “NOT for” routing
- IvanMurzak: proper tool registry and metadata instead of inferring read/write behavior from names
- mcp-unreal: scored, budgeted capability/documentation lookup and non-Python reflection fallback
I’m deliberately NOT using Python as the universal executor because local models tend to disappear down that rabbit hole.
I’ve now proven another important point:
Reducing tool count alone did not solve it.
With Ornith, I reduced initial exposure to only 3 tools for inspection and 6 for scene actions, enforced one call per turn, removed direct Python exposure, and prevented repeated-call loops.
That reduced looping, but it still did not reliably finish broad tasks.
With Qwen2.5-Coder-32B, the very first inspection test showed a different problem: Qwen immediately selected the correct-looking tool and emitted:
<tools>{"name":"get_level_summary","arguments":{}}</tools>
but the wrapper failed to execute it, reported 0 MCP calls, and marked the task completed.
That reinforced my suspicion that the wrapper/general-agent layer is the real bottleneck.
My question is:
Has anyone actually solved this for a 27B–35B local model in Unreal 5.8?
I’m not asking whether MCP “works.” I know it works.
I’m asking whether anyone has a local setup that can reliably handle broad multi-step Unreal tasks without:
- exposing hundreds of tools
- constantly falling back to Python
- needing task-specific regex/rules
- looping on tool selection
- requiring a fresh chat for every tiny subtask
- using Claude/GPT/Gemini as the reasoning engine
If yes, what is the architecture?
Specifically:
- Are you using progressive tool/capability discovery?
- A second router model?
- Skill packs?
- Tool schemas generated dynamically?
- A custom MCP proxy?
- Qwen Code/Cline with aggressive include/exclude lists?
- Native Epic ToolsetRegistry directly?
- Something else entirely?
I’d especially like to hear from anyone running this on a 64GB Apple Silicon machine or a single 24GB GPU.
I’m less interested in model recommendations than in the wrapper/agent architecture that actually made the local model reliable.
Thanks
Jeff