Learning Agents Inclusive Union Action Problem/Question

Development Programming & Scripting
, , ,
1

Hello. I faced a problem working with inclusive union actions when implementing Imitation Learning recording. I am working with stock UE 5.7.2.
So my action schema currently looks like this:

  • root: exclusive union (EU) of 3 top level actions: locomotion, combat, NULL (when inference decides that nothing must be done by an agent, which is a desirable optional behavior)
  • combat: EU of attack and dodge
  • locomotion: EU of blocking locomotion (jumps, mantles) and non-blocking locomotion (various combineable locomotion actions)
  • non blocking locomotion: INCLUSIVE UNION of move, rotate, set speed, upper body animations (which itself is an EU)

In my current setup I have a player imitator character, and for each relevant imitator player action (move, rotate, etc) I push these actions to IL recording actions buffer to be gathered by LA controller (in ULearningAgentsController::EvaluateAgentController_Implementation) on next controller update (each 0.1s). And non-blocking locomotion actions are combineable in this actions buffer, e.g. if durung 100ms imitator moved and rotated right after it then harvested action would be IU of move + rotate. But it can also be that there’s only move or rotate or there’s no pending IU non-blocking locomotion action at all. When IL controller gathers actions, I use ULearningAgentsActions::Make_X_Action functions to make the FLearningAgentsActionObjectElement instances.

THE PROBLEM:
On each IL recording manager update, there is a stacktrace of calls:

UE::Learning::Action::SetVectorFromObject
ULearningAgentsController::EvaluateController
ULearningAgentsController::RunController

and in UE::Learning::Action::SetVectorFromObject for IU action case there’s a check
check(SubElementOffset + SchemaParameters.Elements.Num() == OutActionVector.Num());

and this check gets triggered for my locomotion IU every time. The only way I found to prevent this check from being triggered is to pass all subactions of locomotion IU no matter if imitator did all of them or not. For that I updated my IU to wrap all subactions into optional because I believe it would be easier to distinguish non-taken action as a null optional action rather than an actual action with “null” value. But it kind of makes things ugly. Like, when I specify IU subactions I provide base probabilities for each of them. And then, when I wrap each action into an optional I also provide a probability for each of them, but the probability of each IU subaction element must always be 1 since it is now optional action that does the coin flip.

Another idea that I had is to basically remove the check from UE::Learning::Action::SetVectorFromObject, but

  1. idk if it’s a good idea. Like, I don’t understand ML and all of its implications that much to be certain that this is a bug and/or not intended behavior and it can be safely removed without any other errors arise later on during training/inference.
  2. It cannot be easily done as majority of things to get to that check are either not virtual or not exposed by module API or both, which means I would have to basically copy-paste the whole LA framework to my project from engine folder just to change 1 line (and, frankly, get better debugging support).

I also check logs including LogLearning category each debug session and, as far as I can tell, there are no other issues with my observations and actions schemas. (there were, but I addressed all of them)

THE QUESTION:
Is it exactly the way IU is expected to work? I mean I assumed that IU means “this action can contain SOME of these subactions but they all are probabilistic, i.e. they can be present or absent in whatever combination”. But if I must always provide all subactions of IU as an imitator then I might as well change it to a struct action of multiple optionals. It would at least make more sense as in this case I would only have probabilities of optionals and not optionals AND IU subactions.

2

Can you send a code snippet or BP screenshot of the SpecifyActions and your EvaluateAgentController? I think it would help me understand your exact setup. Thanks.

3

I can, but I’m afraid you would have to spend some time to understand the code base.

I have already changed the code to struct with optionals, but

Here’s where I specified inclusive union:

Here’s where the maps of IU subactions schema is created:

Here’s ULearningAgentsController::EvaluateAgentController, but the call to ULearningAgentsActions::Make_X_Action is behind abstractions here

I can’t provide a link to how I was making an IU action because bugs-free version of it (except for the problem of the topic) was in between commits, but here’s how it looked like:

FLearningAgentsActionObjectElement FAction_Locomotion_NonBlocking::GetAction(ULearningAgentsActionObject* InActionObject,
		AActor* AgentActor, ULearningAgentsInteractor_Combat* Interactor) const
	{
		TMap<FName, FLearningAgentsActionObjectElement> Map;
		Map.Reserve(1 + PendingSimultaneousActions.Num());
		Map.Add(GetActionName(), GetActionInternal(InActionObject, AgentActor));
		for (const auto& PendingAction : PendingSimultaneousActions)
		{
			auto LocomotionAction = StaticCastSharedPtr<FAction_Locomotion_NonBlocking>(PendingAction.Value);
			Map.Add(PendingAction.Key, LocomotionAction->GetActionInternal(InActionObject, AgentActor));
		}
		
		
		auto InclusiveLocomotionActions = ULearningAgentsActions::MakeInclusiveUnionAction(InActionObject,
			Map, Key_Action_Locomotion_NonBlocking);
		auto LocomotionExclusiveUnionAction = ULearningAgentsActions::MakeExclusiveUnionAction(InActionObject, Key_Action_Locomotion_NonBlocking, 
			InclusiveLocomotionActions, Key_Action_Locomotion);
		
		return ULearningAgentsActions::MakeExclusiveUnionAction(InActionObject, Key_Action_Locomotion, LocomotionExclusiveUnionAction,
			Key_Action_All);
	}