While on Unreal 5.2, I started writing tools, built on the PythonScriptPlugin and coded partially in C++ and partially in Python, to support importing data from a different game engine into Unreal. I’ve polished and improved that framework across Unreal 5.3, 5.4 and 5.5. Importing data requires multiple processing steps, with only the final step actually creating or updating Unreal assets. All of the preliminary parts of processing are performed on a background thread using AsyncTask for AnyThread. This has worked without issue for 2 years.
While integrating Unreal 5.6, however, it was discovered that my import framework no longer works. The background processing fails with “Attempted to access Unreal API from outside the main game thread.” A quick search through the code revealed a new check in PyUtil.cpp’s InvokeFunctionCall. There’s a comment in the code that “Python editor code that creates threads and interact with the Unreal API will always hit this case”, but I’ve been running Python processing on background threads for years without crashing. I’m assuming something significant changed in how Unreal 5.6 interacts with Python and that this new check is simply the tip of the iceberg--a guard put in place to protect those other changes.
As far as I can tell, this change in Unreal 5.6 is completely undocumented. While many changes related to Python are called out in the 5.6 release notes, nothing indicates Python scripts (effectively) can no longer be run on a background thread. That feels like a change that should’ve been called out explicitly.
I suspect it’s unlikely this restriction will get rolled back, since I’m assuming it’s part of broader changes to how Unreal integrates Python, so the purpose of this EPS question is more to ask how I should try to adapt to this new restriction. Is there any way I can still perform this processing in the background? Is there any way I can tick the editor, while I’m doing my Python processing, to allow the editor to continue to function? What sort of expectations does Epic have around how complex, long-running Python processing, which requires a variety of “simple” function calls that are generally just reading data should be written? How can I continue to provide a good user experience (read: an editor that isn’t hung for minutes at a time) without being able to do my processing in the background?
// An example interface from C++ to implement in Python
class UPythonBase : public UObject
{
GENERATED_BODY()
public:
static UPythonBase* Get()
{
TArray<UClass*> DerivedClasses;
GetDerivedClasses(StaticClass(), DerivedClasses);
if (DerivedClasses.IsEmpty())
{
return nullptr;
}
return DerivedClasses[0]->GetDefaultObject<UPythonBase>();
}
UFUNCTION(BlueprintImplementableEvent)
void DoSomething();
};
// This part is implemented in Python
import unreal as ue
@ue.uclass()
class PythonImpl(ue.PythonBase):
@ue.ufunction(override=True)
def do_something() -> None:
print(ue.SystemLibrary.get_engine_version())
// Back in C++, we call our function async:
AsyncTask(ENamedThreads::AnyThread, []
{
if (UPythonBase* Python = UPythonBase::Get())
{
Python->DoSomething();
}
}
// This will trigger a Python error from InvokeFunctionCall in PyUtil.cpp
That additional check for the game thread was added as part of some refactoring I made to process python bindings on multiple threads and I discovered that inside InvokeFunctionCall, there was some usage of FEditorScriptExecutionGuard which is not thread-safe because it plays with global values. So we added some checks along with an error if anything was touching this function in a non thread-safe way.
Running anything outside the game thread requires careful considerations, and I’m far from a python expert to offer guidance on what can and cannot be touched from different threads, and if you have all the functionality accessible from Python to properly handle the pitfalls when interacting with UObjects and scripts from worker threads.
If you comment those checks and the error reporting, is everything working like it did?
Thanks for reporting this, I’m a big fan of multithreading myself so I can understand your frustration. I’ll talk to some people here to see what they think.
Thanks for your response! One of the first things I did when I found the new check was comment it out, just to see what would happen, and my Python code immediately crashes the editor without it. The same Python runs without issue on 5.5 and earlier, which is why I assumed there were more significant changes to how Unreal 5.6 integrates Python, and this check was to guard those.
My Python scripts are primarily interacting with a REST-like API for another game engine. Importing data from another engine is a bit like peeling an onion, where I have a set of data to start with, and then as I load that data it exposes other data I’ll also need, down through some arbitrary number of layers until I’ve figured out the overall shape of everything I’m going to import. That’s performed as an “analysis” step, prior to actually running an import. A designer/developer reviews that data, after the analysis is complete, and makes a final call on what assets to import, where in Unreal they want them created, etc. _That_ processing all happens on the game thread already, since it’s going to be creating/updating/deleting assets and doing other things that have always been restricted to the game thread. Looking up the starting data and then performing analysis were being done on background threads.
I’ve been able to keep the starting data lookup on a background thread by going through and removing all usages of the “unreal” Python module from it (aside from simply creating struct instances and setting their properties, which does still work on a background thread since it doesn’t involve InvokeFunctionCall). Analysis needs to do too many things that can’t readily be done without using the Unreal API, though; I can’t really make a similar change there.
In the absence of any way to move the logic back to a background thread, is there any good way to tick the editor? In a commandlet, for example, you have CommandletHelpers::TickEngine, which does a variety of things. The closest thing I’m aware of that’s at all similar is FSlowTask::TickProgress. Analysis doesn’t have any way of knowing how much work will need to do done, so I can’t show a useful progress bar, but I could still pop up a slow task dialog, enter a single progress frame and then tick it. Are there any other mechanisms available?
Can you share a callstack of where it fails next when you remove the checks? Maybe its something we just need to understand better in order to allow it to work.
FTSTicker::GetCoreTicker().AddTicker() offers a way to register a tick.
Yes, ticking the editor is very prone to gotchas, you will end up with editor code running inside your functions and it maybe cause side-effects you don’t want… like a GC being called from inside your calls or worse.
The best way is to turn everything on its head and just perform a small amount of work in an editor tick every frame. It’s what most systems do.
I’m happy if it resolves your issue, you will certainly achieve best stability if everything can be time sliced inside the game thread itself.
Sorry for the slow response. Circling back to this, I moved my processing back to AsyncTask (how it was in 5.5) and commented out the PyUtil.cpp InvokeFunctionCall check. The crash turned out to be a new check(IsInGameThread()) in FEditorScriptExecutionGuard::FEditorScriptExecutionGuard that was also added in 5.6 (commit ef8dc1e1a1a3887cf66739c3e0da610f06fbfc97 in GitHub). If I comment out that check as well, my analysis processing runs on the background thread without issue. I’m somewhat tempted to “just” (my favorite word in programming) run with those checks disabled, since, for how I’m using Python in a background thread, everything seems to work, but I don’t like to introduce engine divergences if there’s any way to avoid them.
Re: ticking, FTSTicker looks like it’s addressing a different need. I’m not looking to perform my own processing per-tick; I’m looking to tick the editor itself so it can do whatever things in a given tick it might decide to do. That way, if my analysis is running on the game thread, I can essentially relinquish it.
I guess perhaps I’m thinking about ticking the wrong way, though. Rather than trying to look for a way to let the editor tick because I told it to, perhaps I should, when I hit a point where I would have called some theoretical TickEditor(), schedule a “run next frame” type task and return. That would (I think) allow the game thread to run other tasks pending on it, and then essentially resume my processing. And, I guess, if I was doing _that_, then maybe FTSTicker _is_ addressing my need, because then I could simply do some nibble of processing and, if there’s more to do, return true from my function so it’ll run again next tick. Is that what you mean? Probably that’s how I should have approached the problem all along.
Unfortunately, it’s not really viable for us to break this down that way, so we’ll just have to live with blocking the game thread and hanging the editor during processing. (There’s no time-deterministic way to chunk the processing, which means the editor would still behave oddly and have periods of varying length where it was hung because there’s no good way to preempt the processing and yield back.) Trying to break the processing down that way would make it significantly more complicated, as well, and increase the maintenance costs of it.
I appreciate the clarifications, though, and the context is stull useful for future knowledge. If push comes to shove, we may just make some local edits to disable the new checks. I’m guessing they were added because some folks had Python integrations where they were trying to do processing on multiple background threads at once, or maybe background and game thread, or whatever, leading to concurrent Python calls, and I can see how that would be problematic. Our usage isn’t like that; we do everything on a single thread (which is likely why we never had any issues with running things in the background on 5.2-5.5).