Unreal Agent is an in‑editor AI agent copilot for Unreal Engine 5 that doesn’t just suggest ideas—it builds actual content and systems inside your project.
Running as a dockable tab in UE5.6, it connects to OpenAI’s GPT‑class models and drives the editor through Python, scene queries, reflection, screenshots, and external generators. In testing, the agent has successfully created complete PCG assets from scratch (nodes, links, parameter wiring) and authored advanced, layered materials—all from natural‑language prompts.
Describe what you want (“Create a PCG graph that scatters rocks on slopes and grass in the valleys”, “Build a master material with triplanar mapping, distance‑based blending, and a ready‑to‑use instance”), and Unreal Agent plans, executes, and verifies the changes directly in your level and content browser.
Demo Video
What the Agent Can Do
Design and wire up systems
Build PCG assets end‑to‑end: create the asset, add nodes, connect pins, and set parameters via Python.
Author complex materials and material instances, including layered setups, masks, and parameter collections.
Script editor workflows: batch‑rename assets, set up folders, spawn and configure actors, and more.
See and understand your world
scene_query – Search the level by class, name, label, or component types and operate on the matching actors.
viewport_screenshot – Capture the current viewport; the agent uses the image to reason about layout and results.
GetSelectedActorsSummary – Focus the agent on exactly what you’ve selected for surgical edits.
Drive Unreal through powerful tools
python_execute – The editor superpower
Runs Python scripts inside Unreal Editor to create and modify actors, Blueprints, PCG graphs, materials, and assets.
Uses a structured JSON result (status, message, details) so the agent can reason about what actually happened.
scene_query – X‑ray vision for your level
Returns compact JSON describing matching actors (name, label, class, location).
Lets the agent target exactly the right objects for follow‑up actions.
viewport_screenshot – Visual ground truth
Captures the current viewport as a PNG.
Unreal Agent can “look” at the shot to improve layout, spacing, and verification.
reflection_query – Live API explorer
Inspects any UClass and returns a schema of its properties and functions.
The model can discover new APIs on the fly and then immediately use them in Python.
file_search – UE5.6 Python brain extension
Searches a UE 5.6 Python API vector store for relevant docs and examples.
The agent reads real docs before writing code, increasing correctness on complex editor APIs.
web_search – Scoped internet assistant
Calls OpenAI’s web search tool for broader context, tutorials, and edge‑case details.
Great for “how do people usually…”‑type questions around Unreal workflows.
replicate_generate (optional) – Content on demand
Talks to Replicate’s HTTP API to generate images, 3D, SFX, music, speech, or video.
Returns local file paths; Python helpers (unrealgpt_mcp_import) turn those into Texture2D, StaticMesh, and SoundWave assets, ready to use in your materials, levels, or PCG setups.
Voice & images – Rich control surface
Voice input via Whisper (/v1/audio/transcriptions): speak your intent, let the agent handle the rest.
Attach local reference images as additional context for layout, look‑dev, and content generation.
Closed‑loop, agentic behavior
After python_execute, the agent is instructed to verify its work with scene_query and viewport_screenshot instead of blindly chaining more code.
Tool‑loop detection, execution timeouts, and result‑size limits keep runs safe and predictable.
If you want an AI that can own tasks like setting up PCG systems, crafting advanced materials, and automating editor workflows—rather than a passive chat window—Unreal Agent gives you a deeply integrated, open source foundation built specifically to drive Unreal Engine 5.
Disclaimer: PythonScriptPlugin and AudioCapture engine plugins must be enabled for Unreal Agent plugin for function properly.