Hello,
We encountered a problem with checking references when deleting an actor in the editor (version 5.5).
This call
AssetToolsModule.Get().FindSoftReferencesToObjects(ActorsToDeletePaths, SoftReferencingObjectsMap);
can take up to 15 minutes (calling LoadPackage on thousands packages) or sometimes cause OOM. We know about bCheckReferencesOnDelete flag but switching it off seems too risky.
I’m trying to get better understanding of what’s going on there and if there’s any way to optimize it. I’ve noticed that in PopulateAssetReferencers when
AssetRegistryModule.Get().GetReferencers(OldPackageName, Referencers, …);
gets called, we pass the path (OldPackageName) to the whole level, which leads to adding tons of packages to ReferencersMap. I’m wondering, shouldn’t OldPackageName point to __ExternalActors__ (since there’s “One File Per Actor”)? I mean something like
FName OldPackageName = AssetToRename.Asset->GetPackage()->GetFName();
instead of
FName OldPackageName = AssetToRename.OldObjectPath.GetLongPackageFName();
which gives us whole level package?
Also there’s another confusing thing about checking referencers. We created a test level. There are two actors Cube1 and Cube2 and there’s an actor holding soft reference to Cube1. Now the problem is when I delete Cube1, FindSoftReferencesToObjects finds nothing and there’s no warning. Then when I change soft ref to Cube2 and try deleting Cube2, FindSoftReferencesToObjects finds the referencer. That’s because I changed it in runtime and GetDirtyWorldPackages added altered package to ExtraPackagesToCheckForSoftReferences. Shouldn’t AssetRegistryModule.Get().GetReferencers(…) call give us referencing package in the first case?
[Attachment Removed]
Steps to Reproduce
Try to delete an actor in a huge level and notice that FindSoftReferencesToObjects can take really long.
Also in some cases FindSoftReferencesToObjects doesn’t find that deleted actor is referenced.
[Attachment Removed]
Update:
I’ve been debugging FindSoftReferencesToObjects a bit and it seems our soft reference is filtered out in FDependsNode::IterateOverDependencies (IterateDependencyList<PackageFlagWidth> to be precise). DependencyProperties has EDependencyProperty::Hard flag set. When I remove this flag (in this particular dependency) using debugger, Soft Ref is found and a warning is displayed. Seems like somehow it’s set incorrectly
[Attachment Removed]
Hello Jan,
Is this hitch occurring in a level with a lot of actors in it, or one that’s just spatially “huge”? I’ve found this tracked issue which seems to line up with what you’re highlighting here- would you agree? The issue has been backlogged, but the investigation you’ve done here may provide the information needed to resolve the ticket.
Have a great weekend.
[Attachment Removed]
Yes it looks similar, there are a lot of actors in our case too, although in out measurements FindSoftReferencesToObjects takes most of the time, not GetActorReferenceMap.
[Attachment Removed]
Hi
I’m late in the conversation but we seem to have the same problem as the first delete that we’re doing is taking at least 1m30s as it loads a lot assets (Computing References). It happens even when adding a basic shape of the engine content like a Cone and deleting it. If I use an Editor Utility Widget with a Destroy Actor it does it right away without checking ref. I don’t know if this small detail can help.
cheers
[Attachment Removed]
Hi guys,
The current implementation of checking references upon delete scales extremely poorly in its current incarnation and although there might be a few quick optimization that could improve the situation, a scale-able fix require a change in approach and that’s probably why the current public jira backlogged. I would suggest turning off the check on delete as it is quite disruptive to iteration for content creator. However, if you are worried about stranded references, then I would suggest implementing a save or submit validator to do a similar check.
The current validation uses the asset registry to query which packages might hold a reference to the world in question but then load any referencing package to understand if anything in it actually referenced this specific actor being deleted. This predates the addition in asset header of full soft path references. We unfortunately haven’t adapted the Asset Registry to scrape that information yet, but were it done then no actual package would need to be loaded to do the validation and a simple query would which should scale a lot better. Until we add the required information to the asset registry however, a validator that does something similar could be implemented by reading the extra necessary information follow what PackageReader does.
Sorry for the long delay in answering, hopefully that can help you guys address the problem until we get to it ourselves.
Cheers
Francis
[Attachment Removed]
Hi Jan,
we ran into similar issue with a huge amount of actors, we’re now using the Engine edit i shared in [this [Content removed]
I investigated FBlueprintEditorUtils::GetActorReferenceMap and found out it creates a complex ref database for all actors, my idea was to “pre filter” by only caring about ref to deleted actors.
It went down from several seconds to less than 1 second for us and our level artists are now happy =).
UUnrealEdEngine::DeleteActors changes in our Engine :
// If we want to warn about references to the actors to be deleted, it is a lot more efficient to query
// the world first and build a map of actors referenced by other actors. We can then quickly look this up later on in the loop.
if (bWarnAboutReferences)
{
//@CYA EDIT replace whole world ref testing to "only deleted ref" (goes from 6s to 0.3s to delete something in a 100K actor scene)
// FBlueprintEditorUtils::GetActorReferenceMap(InWorld, MutableView(ClassesToIgnoreDeleteReferenceWarning), ReferencingActorsMap);
for (FActorIterator ActorIt(InWorld); ActorIt; ++ActorIt)
{
AActor* CurrentActor = *ActorIt;
// do not care about references of actors that will be deleted
if (nullptr != CurrentActor && !ActorsToDelete.Contains(CurrentActor))
{
if (!ClassesToIgnoreDeleteReferenceWarning.ContainsByPredicate([CurrentActor](const UClass* ClassToIgnore)
{
return CurrentActor->IsA(ClassToIgnore);
}))
{
TArray<UObject*> References;
FReferenceFinder Finder(References);
Finder.FindReferences(CurrentActor);
AActor* Parent = CurrentActor->GetAttachParentActor();
while (nullptr != Parent)
{
References.Add(Parent);
Parent = Parent->GetAttachParentActor();
}
for (AActor* DeletedActor : ActorsToDelete)
{
if (References.Contains(DeletedActor))
{
TArray<AActor*>& Refs = ReferencingActorsMap.FindOrAdd(DeletedActor);
Refs.Add(CurrentActor);
}
}
}
}
}
//@CYA END
Hope that helps.
Regards
Cø
[Attachment Removed]
Nevermind i missed the part where you said
> measurements FindSoftReferencesToObjects takes most of the time, not GetActorReferenceMap.
Sorry for the noise.
[Attachment Removed]
Hi Jan,
I could use some clarification on this, as attempting to re-create this hasn’t led to any hitches on my end. I created a level with tens of thousands of actors, each referencing one another at random. Deleting an actor from the level in editor didn’t result in any hitching.
What is the level structure like? Is it a world partition level? How many references do the actors have to one another? Within the test project, the only hitching I found was when deleting levels while testing, which did take a few minutes.
If necessary, we can take the ticket confidential to share more sensitive details about your setup.
[Attachment Removed]
Thanks for sharing this! I checked if it changes anything in our case, but unfortunately it didn’t.
[Attachment Removed]
Hi Jeremy,
Yes, we basically have one huge world partition level, around 250k actors.
I don’t think we have exceptionaly many referencing actors, although I’m not sure how to check this right now. However, I’m deleting a static mesh which has no referencers and it still takes about 15 min.
When deleting said mesh PopulateAssetReferencers adds about 2500 entries to AssetsToRename[0].NotRenamedReferencingPackageNames (seems like a lot of packages that are not really related to the static mesh). Most entries are added when calling AssetRegistryModule.Get().GetReferencers with OldPackageName pointing to our world partition level. Then GatherReferencingObjects loops over NotRenamedReferencingPackageNames and loads each package which can take 15 mins.
[Attachment Removed]
Hi Jan,
I’m still having trouble reproducing this. I have a map with >100k actors with static meshes and various soft references across them, but it’s still taking basically no time to delete one of those actors.
Looking at the functions you mentioned, I noticed a couple things:
- I can’t find AssetsToRename[0].NotRenamedReferencingPackageNames anywhere in the solution- is this a mistake? I did find AssetToRename.NotRenamedReferencingPackageNames used in PopulateAssetReferencers, but it never seems to have more than one element in it.
- I don’t see GatherReferencingObjects iterating over a big collection of packages when an actor is deleted in the test level.
Are you able to provide a minimal project reproducing the issue?
[Attachment Removed]
Sorry I meant first and only element of AssetsToRename collection, there’s no AssetsToRename[0] in the actual code. Sorry for not being clear.
Maybe there’s actually something wrong with our level, if PopulateAssetReferencers finds tons of packages everytime we call it (or your repro misses something, but I don’t know what it is). It might be difficult to create minimal project, I’ll try understanding why all these packages are added to NotRenamedReferencingPackageNames first.
[Attachment Removed]
Hi Jan,
I did some more testing on my end, and think I might have something close to what you’re seeing.
I had actors in a regular level and a OFPA level both reference a third level, which I opened and deleted an actor in. This caused the actors within the OFPA level to have their individual packages loaded (not the level package), but caused the regular level to have its entire package be loaded.
Perhaps you have non-OFPA levels referencing the OFPA level? You said initially that you expect _ExternalActors_ to be referenced, but that only happens if a OFPA level contains refs to the level you’re deleting actors in.
[Attachment Removed]
Hmm, I don’t think I see this in our case. I see many of the assets found by PopulateAssetReferencers are our Level Sequences (about 35% of all assets fund). These Level Sequences hold soft refs to our level, so AssetRegistryModule.Get().GetReferencers(OldPackageName… call finds them and then GatherReferencingObjects loads them.
[Attachment Removed]
You could also use bCheckReferencesOnDelete flag to turn ref checking off. But we’d still like the editor to warn us about broken references.
[Attachment Removed]
Yeah me too I’m afraid to create problems with that flag but maybe adding a “safe delete” option when it’s environment asset that you know that have no ref …
[Attachment Removed]
Hi Jan,
I did some more digging and debugging on my end. I haven’t reproduced the huge wait time, but I think I’ve found the culprit behind what you’re seeing. I suspect the slowdown’s simply due to the sheer complexity of the level you’re working in.
When debugging the actor deletion path, I noticed there’s no specialized handling of OFPA actors. UUnrealEdEngine::DeleteActors() naively constructs the FSoftObjectPath of the actor to delete based on the actor as it lies in the loaded editor level. This means its outer is the PersistentLevel and explains why you’re seeing the level as the outer, as opposed to how the uncooked actor actually lives on the disk as its own package in _ExternalActors_. As a result, it gets the path to the actor embedded in the editor level and continues to soft reference checking without a reference to the actor package.
This could be improved with a specialization for OFPA actors when creating the soft object path during deletion. I will check with the team to see if this would be a feature request or a bug report.
I did also reproduce the second issue you mentioned in your original post, and found an existing tracked issue for it (https://issues.unrealengine.com/issue/UE-209058). It says a fix is currently targeted for 5.8, but that is subject to change. Keep an eye on that link for updates.
[Attachment Removed]
Thank you for investigating the issue and for the link!
I’m not sure I understand how this specialization for OFPA actors should look like.
We’ve tried constructing the FSoftObjectPath from the UPackage* reference (rather than using FSoftObjectPath(UObject*) constructor) in DeleteActors. This gives path to _ExternalActors_. But it didn’t really work. I guess it would be more work than just naively changing the path in this one place in code. Please let me know if there are any plans to change this.
Looking at the issue you linked, I started thinking, maybe we should actually turn off reference checking. According to the issue, referencer has to be dirty to be reported. So it seems it won’t work most of the time anyway, am I right?
[Attachment Removed]