Local LLMs I’ve Tested for Unreal Engine 5.8 + Epic MCP

Development Programming & Scripting AI
1

Local LLMs I’ve Tested for Unreal Engine 5.8 + Epic MCP

I’m trying to get a fully local LLM to control Unreal Engine 5.8 through Epic’s MCP stack.

The target is not just basic tool calling. I need the model to reliably complete a real agent loop:

inspect scene/frame → understand what is wrong → choose a correction → execute it → verify the actual result → stop

My current architecture is:

Kimi Code → local model → Epic Unreal MCP → Unreal Engine 5.8

Unreal runs on Windows. The recent local models are hosted on an M1 Max Mac with 64 GB unified memory.

Over time I also cleaned up the MCP interface significantly. This is important because some early failures turned out to be interface/schema failures rather than pure model reasoning failures.

The current permanent Kimi agent contains:

  • exact call_tool examples
  • exact toolset_name / bare tool_name rules
  • known schema traps
  • camera tool argument names
  • transform preservation rules
  • ProgrammaticToolset guidance
  • cinematography/composition rules

So the later tests are substantially cleaner and more meaningful than the early ones.

Models actually tested

Model Runtime / Quant Approx. performance Result Main failure / conclusion
Qwen3-30B-A3B-Instruct-2507 llama.cpp, Q8_0 ~45.6 tok/s decode Best fast executor so far Excellent when given explicit instructions. Passed exact reply, structured tool loop and decisiveness. Given an exact camera delta, it applied and verified it in ~33 s. When required to make its own artistic/spatial correction, it gathered real evidence but skipped set_actor_transform, fabricated post-correction measurements, and declared success without actual verification.
Qwen3-Coder-30B-A3B-Instruct llama.cpp, Q8_0 ~44.9 tok/s decode Excellent tool interface, unsafe autonomous behavior Raw tests were excellent: exact reply PASS, structured tool loop PASS, decisiveness PASS. Real Unreal sanity test failed badly: after correctly answering a simple read-only get_current_level request, it continued autonomously, found actors, described tools, placed a cinematic camera, rendered/projected, and modified the scene without permission. It also fabricated verification claims contradicted by the actual tool output.
Ornith 1.5 35B-A3B Ollama, Q6_K ~45–50 tok/s raw decode Best judgment seen, but impractically slow Earlier artistic camera run eventually produced the best composition of the local models, but took ~17.8 minutes / 36 turns. With the cleaned interface it still hit a 180 s cap before reaching a correction. Hard output caps broke its reasoning instead of making it usable interactively.
NVIDIA Nemotron 3.5 Lightning 30B-A3B llama.cpp, Q8_0 + MTP ~50 tok/s Fast and tool-capable, but indecisive Passed structured tool loops and was fast, but in real Unreal camera work it repeatedly inspected/verified instead of committing to the edit. Better instructions reduced schema failures but did not fix decision-making.
Qwen2.5-Coder-32B-Instruct Ollama, Q6_K, dense 32B ~9.5 tok/s @32K Rejected Too slow and, more importantly, the GGUF/Ollama combination did not emit structured tool_calls. It produced <tools>...</tools> or raw JSON as content. Agent loop could not execute tools.
Qwen3.8-27B llama.cpp Q6_K ~10.5 tok/s Rejected for speed Tool-loop behavior was workable, but decode speed was nowhere near interactive.
Qwen3.8-27B MLX ~11.8 tok/s Rejected for speed Slightly faster than llama.cpp, still much too slow for multi-turn Unreal work.
OpenResearcher-30B-A3B llama.cpp, Q6_K ~50 tok/s Rejected Fast and capable of tool calls, but heavily biased toward research behavior. Exact-reply and decisiveness tests failed; unsuitable as a concise real-time Unreal agent. Deleted afterward.
Kanana-2-30B-A3B-Instruct-2601 llama.cpp, Q8_0 ~36 tok/s short context, ~19 tok/s at heavier context Rejected / deleted Basic tests and a tiny Unreal read passed, but under realistic context it was slower and less stable than Qwen3. It also showed prompt-sensitive MCP envelope behavior and fell back into discovery. Not competitive enough to keep.
Muse Glimmer 30B Q4_K_M Slow in prior Windows test Failed camera benchmark Ran to the step cap without successfully completing the camera task. Repeated failed camera-placement calls; scene remained unchanged.
Qwen3 14B local — Too weak for the original raw MCP surface Tested early. Not reliable enough when exposed to the large/raw Unreal tool surface. Could behave differently with the cleaned interface, but smaller/faster wrong does not solve the core problem.
~30B Qwen-class model on RTX 2080/shared RAM local Windows Extremely slow Rejected for hardware/runtime speed Partial/shared-memory offload on the RTX 2080 was completely impractical.

Detailed results

1. Qwen3-30B-A3B-Instruct-2507 Q8_0

This is still the strongest fast execution baseline.

Raw performance

Approximate measured results:

  • decode: ~45.6 tok/s
  • prompt processing: ~392 tok/s at ~12K tokens
  • RAM: ~34 GB
  • GPU offload: essentially full
  • TTFT: very good

Synthetic tests

It passed:

  • exact reply
  • structured tool loop
  • decisiveness
  • immediate tool execution with zero pre-call prose

This was the first model that passed all three synthetic tests cleanly.

Explicit camera-transform test

When given an exact numeric correction, Qwen3 performed very well.

It:

  • called set_actor_transform
  • applied the supplied delta
  • preserved camera rotation
  • rendered/projected afterward
  • completed in roughly 33 seconds / 6 turns

This proved Qwen3 is a legitimate fast executor.

Autonomous judgment test

The problem appeared when I stopped giving it the answer.

It gathered real projection/render evidence and correctly identified several composition issues:

  • face too far left
  • torso too tightly cropped
  • headroom concerns

But then it failed the actual agent loop.

It:

  1. calculated a proposed correction
  2. never called set_actor_transform
  3. jumped directly to a final report
  4. fabricated post-correction measurements
  5. declared the frame successful despite never applying or verifying anything

It even claimed new image coordinates that no Unreal tool had ever returned.

So the failure mode is:

inspect → diagnose → calculate → SKIP WRITE → fabricate verification → report PASS

That is not acceptable for an autonomous Unreal agent.

Conclusion

Qwen3-2507 = best fast executor, but not reliable autonomous visual/spatial agent.

2. Qwen3-Coder-30B-A3B-Instruct Q8_0

This is the latest model tested.

The result was interesting because the raw model looked nearly perfect at first.

Model / download

Repository:

ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF

File:

qwen3-coder-30b-a3b-instruct-q8_0.gguf

Size:

32,483,933,856 bytes / 30.25 GiB

Architecture:

  • qwen3moe
  • 30.53B total parameters
  • ~3.3B active
  • 48 blocks
  • 128 experts
  • 8 experts active/token

Native context:

262,144

Benchmarked at:

32,768

llama.cpp version:

b10615

It was fully compatible.

Raw performance

Metric Qwen3-Coder
Load time ~15.0 s
RSS ~32.7 GiB
Wired memory ~35.2 GiB
GPU offload all 48 layers
Device utilization ~98%
Prompt speed 440.7 tok/s @10K
Prompt speed 524.6 tok/s @6.6K
Decode 44.88 tok/s
Small-prompt TTFT 0.276 s
10K-prompt TTFT 22.87 s

Performance is essentially at parity with Qwen3-2507.

Exact reply test

PASS

Exact output:

QWEN3 CODER READY

About 0.43 seconds.

Structured tool loop

PASS

This was important because Qwen3-Coder uses an XML-style internal tool syntax, but llama.cpp b10615 includes a dedicated parser for it.

The API correctly returned an actual structured tool_calls array.

The model:

  • called add_numbers
  • supplied correct structured arguments
  • received the result
  • continued normally
  • produced the correct result

Zero pre-tool prose.

Decisiveness test

PASS

  • exactly one tool call
  • zero prose
  • correct arguments
  • time to tool call: about 1.58 s

At this point it looked extremely promising.

Unreal sanity test

This is where it failed badly.

The task was intentionally trivial:

Get the current Unreal level and stop.

Expected:

one call_tool to:

SceneTools.get_current_level

Then stop.

First call

It did exactly the right thing.

Returned:

/Game/Noir/Maps/L_BackroomTable_01

So interface discipline was excellent.

Then it ignored the task boundary

Instead of stopping, it continued autonomously.

It:

  • searched for actors
  • read transforms
  • called describe_toolset
  • invoked cinematic tools
  • ran place_cinematic_camera
  • rendered/projected/inspected
  • modified an unrelated camera

The run reached:

  • 120.6 seconds
  • 13 turns
  • 12 MCP calls

despite a test limit of:

  • 30 seconds
  • 3 turns

Scene mutation during read-only request

It moved:

CineCameraActor_25

from origin:

(0,0,0)

to approximately:

(268.17, 297.51, 103.65)

The target benchmark camera:

CAM_ANTONIO_DIRECTED_01

was untouched.

The accidental mutation was identified and could be reverted because its original state was known.

Fabricated verification

Qwen3-Coder also showed the same verification-honesty problem as Qwen3-2507.

Its final summary claimed:

headroom properly maintained with no head cropping

But the actual verifier result returned:

headCropped = true

It also asserted:

  • look room quality
  • eyes near upper third

without corresponding evidence in the returned tool fields.

So this model did something arguably worse than Qwen3-2507.

Qwen3-2507 tends to under-execute.

Qwen3-Coder over-executed, changed the scene without authorization, then misreported what its own verification said.

Conclusion

Qwen3-Coder = speed PASS, tool calling PASS, schema discipline PASS, scope discipline FAIL, verification honesty FAIL.

The full artistic camera judgment benchmark was not run because it already failed the read-only sanity gate.

3. Ornith 1.5 35B-A3B Q6_K

Ornith has shown the strongest artistic/compositional judgment of the local models tested.

The problem is speed at the agent level.

Earlier artistic camera test

An earlier run took approximately:

  • 36 turns
  • 1069 seconds
  • 17.8 minutes

It eventually produced a materially better composition than Nemotron/Qwen.

It maintained:

  • better look room
  • better 3/4 character readability
  • better framing judgment

But 18 minutes is unusable for interactive work.

Clean-interface rerun

After the MCP interface was fixed:

  • 0 malformed calls
  • 0 schema slips
  • 0 list_toolsets
  • 0 describe_toolset
  • 0 toolset-name omissions

So the plumbing was no longer the problem.

Even then Ornith hit a 180-second hard cap before it reached a correction.

It completed only a few evidence-gathering turns.

Why it is slow

Its raw decode rate is actually fine:

~45–47 tok/s

The problem is that Ornith generates extremely long reasoning turns.

Example:

  • ~2,015-token reasoning turn
  • ~43 seconds just decoding that turn

Another generation continued for more than 120 seconds before being killed.

Even a trivial prompt like “reply READY” caused it to narrate its reasoning first.

Output-cap test

I also tried forcing short output.

Approximate per-turn cap:

250 tokens

That did not fix it.

It simply truncated the reasoning before the model could complete the judgment.

Conclusion

Ornith is not practically interactive on this machine.

It remains useful as a judgment reference, but not as the main Unreal agent.

4. NVIDIA Nemotron 3.5 Lightning 30B-A3B Q8_0

Nemotron was one of the fastest models tested.

Approximate decode:

~50 tok/s

Prompt processing was also extremely fast.

It passed the basic tool-loop test.

The problem was real-task behavior.

On Unreal camera work it tended to:

  • inspect
  • inspect again
  • verify
  • rethink
  • delay commitment

rather than simply make the correction.

Better prompting dramatically improved its tool discipline, but not its actual judgment/commitment.

One cleaned-up camera test got it down from an 88-turn mess to about:

9 turns / 71 seconds

which proved the interface matters enormously.

However, on genuine artistic correction tasks it still failed to commit and execute reliably.

Conclusion

Nemotron = fast, structurally capable, but too indecisive for autonomous artistic Unreal work.

Deleted afterward.

5. Qwen2.5-Coder-32B-Instruct Q6_K

Dense 32B.

At 32K context:

~9.5 tok/s

Already too slow.

But the bigger problem was tool calling.

Instead of structured API tool calls, it emitted:



<tools>
{"name":"...","arguments":{...}}
</tools>

or raw JSON as ordinary assistant content.

The API returned:

tool_calls: []

This happened through:

  • Ollama OpenAI-compatible endpoint
  • Ollama native /api/chat

So no agent harness could actually execute the calls without adding another parser/repair layer.

Conclusion

Too slow and incompatible with the required structured tool-call pipeline.

Deleted.

6. Qwen3.8-27B

Tested in both:

  • llama.cpp Q6_K
  • MLX

Approximate speeds:

  • llama.cpp: ~10.5 tok/s
  • MLX: ~11.8 tok/s

Tool-loop behavior was workable.

But the speed was nowhere near usable for multi-turn Unreal work.

Conclusion

Rejected for latency.

Deleted.

7. OpenResearcher-30B-A3B

Approximate decode:

~50 tok/s

Tool-loop behavior worked.

But it was strongly biased toward its research-agent behavior.

It failed:

  • exact response
  • decisiveness
  • concise action behavior

It wanted to behave like a deep-research system rather than a real-time Unreal editor agent.

Conclusion

Wrong behavioral tuning.

Deleted.

8. Kanana-2-30B-A3B-Instruct-2601

Kanana was also tested and discarded.

It passed basic raw tests and a tiny Unreal read test, but it did not hold up well enough under more realistic context.

Observed issues included:

  • slower effective speed under heavier context
  • prompt-sensitive MCP envelope behavior
  • fallback into discovery
  • less robust than Qwen3

Approximate observed performance:

  • ~36 tok/s under lighter context
  • ~19 tok/s under heavier context

The full camera test was not worth pursuing further.

Conclusion

Not competitive enough with Qwen3 to keep.

Deleted.

9. Muse Glimmer 30B

Muse was tested earlier in the project.

In one Unreal camera benchmark it:

  • ran to the turn cap
  • repeatedly failed camera-placement calls
  • never successfully completed the target task
  • left the scene unchanged

It was also significantly slower than the better MoE candidates.

Conclusion

Not useful for this workflow.

Deleted.

10. Qwen3 14B

Tested early in the project.

It was not reliable enough against the original large/raw Unreal tool surface.

That test predates the cleaned MCP interface, so it is possible the model would perform somewhat better today.

However, I have not revisited it because the problem I’m trying to solve is not merely latency.

A faster model that still makes the wrong artistic/spatial decisions is not useful.

Older Qwen3-Coder history

There is one important historical result.

I previously used a Qwen3-Coder 30B-A3B 4-bit MLX build in an older camera harness.

That model did complete camera tasks successfully.

Examples included:

  • roughly 20 calls / 2 corrections
  • later rerun around 13 calls / 1 correction

Some failures around that period turned out to be bugs in my own harness.

However, this was a different agent stack, so I do not count it as directly comparable with the current raw Epic MCP setup.

The latest Q8_0 test described above is the cleaner current result.

MCP/interface findings

This testing exposed a major issue unrelated to model intelligence.

Epic MCP exposes a gateway-style tool:

call_tool

Kimi’s native schema says:

  • toolset_name may be omitted
  • use list_toolsets
  • use describe_toolset

That behavior is reasonable generically, but disastrous for smaller local models.

They start doing:

list → describe → describe → forget schema → retry → context explosion

Some toolset descriptions are huge.

Fix

I added a small authoritative section directly to the permanent Kimi system profile.

For example:



call_tool({
  "toolset_name": "<exact toolset>",
  "tool_name": "<bare tool name>",
  "arguments": {...}
})

And explicit rules:

  • always use toolset_name
  • use bare tool_name
  • never put dotted names into tool_name
  • do not call list_toolsets or describe_toolset for known tools
  • preserve full actor transform
  • camera tools use camera_name / subject_name
  • never guess refPath

This transformed Qwen3’s behavior.

Before:

  • schema slips
  • discovery loops
  • wasted turns
  • malformed gateway calls

After:

A tiny Unreal sanity test completed in:

  • 12 seconds
  • 2 model turns
  • 1 MCP call
  • 0 discovery
  • 0 schema slips

So the MCP interface itself is now mostly under control.

ProgrammaticToolset

Epic’s ProgrammaticToolset turned out to be very useful.

Exact toolset:

editor_toolset.toolsets.programmatic.ProgrammaticToolset

Key execution tool:

execute_tool_script

A script defines:



run() -> Dict[str, Any]

and receives a pre-injected:



execute_tool(...)

Important restrictions:

  • no Unreal Python module
  • no filesystem
  • no network
  • limited imports
  • orchestration only

But one programmatic call successfully executed 8 internal Unreal tool operations.

I tested:

  • actor lookup
  • transform read
  • temporary transform edit
  • verification
  • restoration
  • final verification

all in one MCP script call.

The actor was restored exactly.

This is significant because it can collapse:

8 model ↔ MCP round trips

into:

1 model decision + 1 scripted execution

That should help a lot with larger scene-edit tasks.

What is no longer the main problem

At this point I do not think MCP plumbing is the primary blocker anymore.

The latest models can:

  • form correct gateway calls
  • preserve schemas
  • avoid discovery loops
  • access real scene evidence
  • run deterministic batches

The remaining failures are mostly model behavior.

The actual model problem

I need a model that can reliably do:

inspect → understand → decide → execute → verify → stop

The failures now fall into three categories.

Qwen3-2507

Fast enough.

But:

under-executes

It may understand the problem, calculate a correction, then skip the actual write and fabricate the verification.

Qwen3-Coder

Fast enough.

Excellent structured tool use.

But:

over-executes

It may answer the requested task correctly, then continue autonomously into unrelated scene edits.

It also fabricated verification contradicted by the actual verifier result.

This is arguably more dangerous than under-execution.

Ornith

Best artistic judgment observed.

But:

over-deliberates

It can spend minutes thinking before committing to an action.

Current local models kept on the Mac

Before the Qwen3-Coder test I had reduced the permanent inventory to:

Qwen3-30B-A3B-Instruct-2507 Q8_0

Current fast executor baseline.

Ornith 1.5 35B-A3B Q6_K

Slow judgment/reference model.

The new:

Qwen3-Coder-30B-A3B-Instruct Q8_0

is currently still on disk pending deletion/retention decision after the benchmark.

It is not currently running.

The normally restored live model after testing is:

Ornith 1.5 35B-A3B Q6_K

What I am actually looking for

I am not looking for the model with the best coding benchmark.

I need a local model that combines:

Qwen3 speed

with

Ornith-level judgment

with

strict scope discipline

and

honest verification

It needs to reliably complete:

inspect Unreal → reason about the real scene → decide what should change → actually make the change → inspect the changed state → report only what the tools prove → stop

Ideally within:

30–90 seconds

and certainly under:

~2 minutes

for a bounded camera or scene task.

Acceptance criteria for the next model

Any future model has to pass all of these.

Tool interface

  • structured tool calls
  • correct toolset_name
  • bare tool_name
  • no malformed JSON
  • no discovery loops

Scope control

  • do exactly what was requested
  • do not invent additional tasks
  • do not mutate unrelated actors
  • stop when the task is complete

Judgment

  • correctly interpret real frame/scene evidence
  • choose a geometrically/artfully sensible correction
  • avoid shotgun multi-axis changes without justification

Execution

  • actually perform the requested edit
  • do not skip the write step

Verification

  • re-read/render/project after the edit
  • only report values that were actually returned
  • never fabricate post-change measurements
  • never claim PASS when the verifier says otherwise

Latency

  • interactive enough for real Unreal work
  • preferably under one minute
  • approximately two minutes maximum for a bounded task

Bottom line

The testing has narrowed the problem considerably.

Qwen3-2507

  • fast
  • excellent explicit executor
  • unreliable autonomous close-the-loop behavior

Qwen3-Coder

  • fast
  • excellent tool calling
  • dangerous scope expansion
  • unreliable verification honesty

Ornith

  • strongest artistic judgment observed
  • far too slow

Nemotron

  • fast
  • indecisive

Kanana

  • not competitive enough

Qwen2.5-Coder

  • too slow / tool-call incompatibility

Qwen3.8

  • too slow

OpenResearcher

  • wrong behavioral tuning

Muse

  • failed practical camera work

So the problem is no longer:

“Can a local 30B model call Unreal tools?”

Yes. Several can.

The real unanswered question is:

Which local model can make the right autonomous decision, execute it, verify the actual result honestly, respect scope, and stop — while remaining interactive?

That is the model I’m still looking for.