Hello. So one of the observations I have is a conv2d observation. Initially I tried to use a continuous observation as its subobservation because I thought it made the most sense. That schema fragment looks like this:
FConv2dObservationParams Conv2DParamsLocal = Config.Conv2dParams;
Conv2DParamsLocal.InputWidth = Columns;
Conv2DParamsLocal.InputHeight = Rows;
Conv2DParamsLocal.InChannels = Config.Grids.Num();
auto RaindropContinuousObservation = ULearningAgentsObservations::SpecifyContinuousObservation(InObservationSchema,
Rows * Columns * Config.Grids.Num(), 1.f, Key_Observation_Surrounding_LIDAR_Raindrop_Continuous);
auto RaindropObservationsConvolved = ULearningAgentsObservations::SpecifyConv2dObservation(InObservationSchema,
RaindropContinuousObservation, Conv2DParamsLocal, Key_Observation_Surrounding_LIDAR_Raindrop_Convolved);
It compiles and, from what I could tell, schema is ok: I managed to get Imitation Recording running, I recorded and saved some data, there were no errors or warnings in logs.
But then, when I tried to create a policy object (ULearningAgentsPolicy::MakePolicy), I’ve got a nullptr exception at NNERuntimeBasicCpuModel.cpp:5432.
FConv2dInstance::FConv2dInstance(const FConv2dLayer& InConv2dLayer) : Conv2dLayer(InConv2dLayer)
{
SubLayerInstance = Conv2dLayer.SubLayer->MakeInstance();
}
void FConv2dInstance::SetMaxBatchSize(const uint32 MaxBatchSize)
{
NNE_RUNTIME_BASIC_TRACE_SCOPE(NNE::RuntimeBasic::Private::FConv2dLayerInstance::SetMaxBatchSize);
-> SubLayerInstance->SetMaxBatchSize(MaxBatchSize); // this is line 5432
SubLayerOutputBuffer.SetNumUninitialized(MaxBatchSize * Conv2dLayer.SubLayer->GetOutputSize(), EAllowShrinking::No);
}
The SubLayerInstance is nullptr and constructor above where this variable is assigned is never called (or at least I never had a breakpoint triggered there).
I haven’t manage to understand why does this happen yet, but I tried to replace the continuous observation with a static array of floats observation as conv2d subobservation aaand… it worked. Well, at least, creating policy object doesn’t crash anymore. So this:
FConv2dObservationParams Conv2DParamsLocal = Config.Conv2dParams;
Conv2DParamsLocal.InputWidth = Columns;
Conv2DParamsLocal.InputHeight = Rows;
Conv2DParamsLocal.InChannels = Config.Grids.Num();
auto RaindropItemObservation = ULearningAgentsObservations::SpecifyFloatObservation(InObservationSchema, 1.f, Key_Observation_Surrounding_LIDAR_Raindrop_Item);
auto RaindropArrayObservation = ULearningAgentsObservations::SpecifyStaticArrayObservation(InObservationSchema, RaindropItemObservation,
Rows * Columns * Config.Grids.Num(), Key_Observation_Surrounding_LIDAR_Raindrop_Array);
auto RaindropObservationsConvolved = ULearningAgentsObservations::SpecifyConv2dObservation(InObservationSchema,
RaindropArrayObservation, Conv2DParamsLocal, Key_Observation_Surrounding_LIDAR_Raindrop_Convolved);
fixes the crash, and when the policy object is created, SubLayerInstance has a valid object inside. However, I still never saw the constructor to trigger so idk where is this object is created 
.
So the question is - is it how things are expected to work and static array of floats is the only viable observation here, or am I doing something wrong or is anything wrong with the current state of LA/NNERuntimeBasicCpu plugin?
For context, I use stock UE 5.7.2. I’m also attaching 2 screenshots with stack traces from calling ULearningAgentsPolicy::MakePolicy down to the place where I’ve been getting nullptr exception. One screenshot is with continuous observation and the other is with static array.
