How Are People Making Local LLMs Reliably Control Unreal Engine 5.8 Through MCP?

[Jrt0013]

I’m trying to build a genuinely usable local AI agent for Unreal Engine 5.8, not just prove MCP can call a tool.

My goal is to be able to talk naturally to an agent and say things like:

“Inspect this scene and block out two rows of buildings with a road in the middle.”

“Create a proper 50mm medium close-up of Antonio and visually verify it.”

“Look at this Sequencer setup and fix what’s wrong.”

I want the agent to inspect, reason, act, verify, and finish without me having to hand-train a rule for every new type of request.

My main machine is UE 5.8 on Windows with an RTX 4090. I bought a 64GB M1 Max Mac specifically to run local models as an AI appliance. The Mac is serving Ollama over the LAN.

Models I’ve tested include Qwen-family models and Ornith 1.5 35B-A3B. I currently have Qwen2.5-Coder-32B-Instruct Q6_K and Ornith Q6 available locally.

I’ve tried Claude Code, Qwen Code, Cline, OpenClaw, and my own custom harness.

Claude Code is designed around very large system prompts/tool contexts and is a poor fit for these 30–35B local models. The local models become slow or indecisive once the tool/context surface gets large.

Qwen Code can connect to MCP, but with large Unreal tool catalogs the model starts looping or spending too much time deciding what to call. I’ve also hit wrapper-format issues where Qwen correctly emits a structured tool call, but my harness fails to parse it and incorrectly marks the task completed with 0 MCP calls.

Cline connects and can use Unreal MCP, but with my local model it tends to over-plan, dump schemas, fall back to Python, and spend minutes talking to itself instead of acting.

I built a custom harness because none of those were usable enough.

The weird part is that my custom harness is exceptionally good at one narrow domain: camera placement.

My benchmark is:

“Create a new CineCameraActor named CAM_KIMI_TEST. Position and aim it to create a proper medium close-up of Antonio using his current position. Set the focal length to 50mm. Visually verify the actual camera render and correct it if necessary.”

The current harness does this in about 30 seconds, 6 MCP calls, 0 corrections, with measured visual verification. Nothing else I’ve tested comes close to it for this task.

So I know a local model can be useful when the wrapper/tool architecture is good.

The problem is everything outside that optimized path.

The current harness became a treadmill of:

ask a new type of question → model takes a bad route → add another rule/regex/tool guard → next request exposes another failure.

I no longer believe the model is the only problem. I think the general wrapper architecture is the bigger problem.

I audited three Unreal MCP projects:

• remiphilippe/mcp-unreal
• IvanMurzak/Unreal-MCP
• VibeUE

The architecture I’m moving toward is:

user request
→ agent interprets intent
→ capability search
→ retrieve only 2–4 relevant capability cards/tool groups
→ expose a small tool surface
→ execute one tool call
→ inspect result
→ continue

Camera work stays pinned to my existing optimized camera path and bypasses the registry completely.

I’m borrowing ideas from:

• VibeUE: capability/skill packs and “NOT for” routing
• IvanMurzak: proper tool registry and metadata instead of inferring read/write behavior from names
• mcp-unreal: scored, budgeted capability/documentation lookup and non-Python reflection fallback

I’m deliberately NOT using Python as the universal executor because local models tend to disappear down that rabbit hole.

I’ve now proven another important point:

Reducing tool count alone did not solve it.

With Ornith, I reduced initial exposure to only 3 tools for inspection and 6 for scene actions, enforced one call per turn, removed direct Python exposure, and prevented repeated-call loops.

That reduced looping, but it still did not reliably finish broad tasks.

With Qwen2.5-Coder-32B, the very first inspection test showed a different problem: Qwen immediately selected the correct-looking tool and emitted:

<tools>{"name":"get_level_summary","arguments":{}}</tools>

but the wrapper failed to execute it, reported 0 MCP calls, and marked the task completed.

That reinforced my suspicion that the wrapper/general-agent layer is the real bottleneck.

My question is:

Has anyone actually solved this for a 27B–35B local model in Unreal 5.8?

I’m not asking whether MCP “works.” I know it works.

I’m asking whether anyone has a local setup that can reliably handle broad multi-step Unreal tasks without:

• exposing hundreds of tools
• constantly falling back to Python
• needing task-specific regex/rules
• looping on tool selection
• requiring a fresh chat for every tiny subtask
• using Claude/GPT/Gemini as the reasoning engine

If yes, what is the architecture?

Specifically:

• Are you using progressive tool/capability discovery?
• A second router model?
• Skill packs?
• Tool schemas generated dynamically?
• A custom MCP proxy?
• Qwen Code/Cline with aggressive include/exclude lists?
• Native Epic ToolsetRegistry directly?
• Something else entirely?

I’d especially like to hear from anyone running this on a 64GB Apple Silicon machine or a single 24GB GPU.

I’m less interested in model recommendations than in the wrapper/agent architecture that actually made the local model reliable.

Thanks

Jeff

[Memeseco]

Hi Jeff i been working on something very similar for 3 years now and just hooked up my MCP tools literally this week, to my locally hosted “Agnostic LLM API”(so it can point to any model) with guardrails and a fast API i have built called the “ironknight”. I have mainly been working on spawning actors in a reliable way and adding custom gameplay tags. (included an image)

If you want to share research, collaborate share tips, feel free to reach out to me.
Is my public discord, Feel free to find “memeseco” the owner. ( Ouract )

“wrapper/agent architecture that actually made the local model reliable.”

I have been working on my own Kernel and framework for provable trust and the difference between a chat and a protocol. My ai has the ability to posses actors in the scene and talk through the LLM for narrative purposes. I am currently adding MCP and tools through python.

I have my own git repo and a few academic papers i am working on.

Otherwise best of luck.

[pavyny (pavyny dev)]

I’m on cloud models (Claude Code), not local ones, so I can’t speak to the harness side, but a big share of what you’re describing isn’t the model, it’s the toolset surface itself.
Things that cut my failure rate a lot:

Don’t feed the model describe_toolset output. For a full toolset it can run into six figures of tokens - that’s the catalog your 32B is drowning in. The toolsets are plain Python on disk: Engine/Plugins/Experimental/Toolsets/<X>/Content/Python/<pkg>/toolsets/*.py.

A grep for def [a-z_]+\( gives you every tool with its signature and docstring for pennies. C++ toolsets (e.g. physics) - same trick, grep the headers for UFUNCTION. This is effectively the “progressive discovery” you’re asking about, done outside the protocol.

Get schemas lazily via validation errors. Calling a tool with deliberately incomplete args returns the full JSON schema of that one tool. So the loop is: call → error+schema → correct call. One caveat: if every param has a default, the tool silently runs as a no-op instead of erroring, then it’s back to the sources.

Expect the schemas to lie. Argument names diverge between toolsets and inside one toolset (path vs folder_path vs asset_paths), “optional” fields are often required in practice, and find_actors truncates at 20 results without saying so. A small model burns its context re-guessing these; hard-code the known ones into your system prompt instead.

For multi-step edits, use ProgrammaticToolset - one sandboxed Python script instead of N tool calls is exactly the “bypass” you mention and it’s built in.

I keep a register of ~50 of these traps per-toolset and with workarounds. Here: GitHub - PavelVyny/ue58-mcp-field-notes: Field notes on driving the Unreal Editor through the native UE 5.8 MCP server - what the docs do not cover. · GitHub
it’s the stuff that bites regardless of which model drives the calls.

[Jrt0013]

Hey, thanks for reaching out. It definitely sounds like we’ve been attacking a lot of the same problems from different directions.

My setup right now is a local model server feeding Unreal 5.8 through Epic MCP, and I’ve been testing different local models against a very strict inspect → decide → act → verify → stop loop.

The biggest issue I’ve run into isn’t basic tool calling anymore. I can get models to call Unreal tools reliably. The harder problem is judgment and execution discipline — things like acting before inspecting, overcorrecting camera moves, getting stuck in retry loops, or claiming a shot is good when the verifier says otherwise.

I’ve also been experimenting with wrapper/agent rules and hardening the MCP interaction so the model doesn’t waste time on tool discovery or malformed calls.

Your Ironknight architecture and the “chat vs protocol” idea sound especially interesting to me. I’d definitely be interested in comparing notes on guardrails, verification after state changes, limiting tool scope, and how you’re making actor manipulation reliable.

[WildeHoney]

intersting. im having similar results.
its a mixed bag, layer interpretation confuses output to terminal in vscode, not ue. which isnt helping; and as you mentioned over time stops handling tool calls correctly. its also mixed results with/without “thinking” … go figure =/
take it with a grain of salt. im self taught.