Course: Neural Network Engine (NNE)

General Tutorial & Course Discussions
22

Thanks @ranierin I was able to run our model on ORT CPU.
Things were a bit tricky since the model has dynamic shapes, thankfully since our code is in C++, once we learned the NNE abstraction layers, we got there decently quickly.

In case others are trying the same thing, here is a quick snippet of what I ended up doing:

In our class:

TArray<FNeuralNetworkTensor> Inputs;
TArray<FNeuralNetworkTensor> Outputs;

// manually set the input shape and data.
Inputs.SetNum(1); // We know that our model only has 1 input tensor
Inputs[0].Shape = { 1, static_cast<int32>(NumSamples), 1 };
Inputs[0].Data.SetNum(NumSamples);
// copy the data to the input tensor so the model can process it
FMemory::Memcpy(Inputs[0].Data.GetData(), SourceToProcess.data(), NumSamples * sizeof(float));

Finally, if you have dynamically output shapes, don’t forget to manually set them and allocate the data in them. Here is an example where we know that all our output shapes have a dynamic size but at that point we know what the size will be (same as the input tensor shape example where we know the number of samples at that point).

    Outputs.SetNum(Model->NumOutputs());
    for (auto i = 0; i < Outputs.Num(); i++)
    {
        auto Shape = Model->GetOutputShape(i);
        if (Shape.Num() == 0)
        {
            break;
        }
        Outputs[i].Shape = Shape;

        int32 Volume = 1;
        for (int32 j = 0; j < Shape.Num(); j++)
        {
            if (Shape[j] < 1)
            {
                // -1 means dynamic size, we need to use our calculated size
                Volume *= CalculatedOutShapeSize;
                Outputs[i].Shape[j] = CalculatedOutShapeSize;
            }
            else
            {
                Volume *= Shape[j];
            }
        }
        
        if (Volume>0)
		{
			Outputs[i].Data.SetNum(Volume);
		}
    }

In this example, it’s important to note that we set both the specific shape size (for the dynamic shape) AND also the shape data size which is then used to receive the model output.

finally, don’t forget to call SetInputs as per the tutorial before calling RunSync (or whatever you implemented to run the model).

Model->SetInputs(Inputs); // needed so the input bindings are properly set
Model->RunSync(Outputs);
// get your data out of Outputs, Outputs[0].Data will have the data of our first output tensor.

So while there is some work needed to convert existing code (or learn how to use models), the fact that NNE handles all the dependency and runtime management is a huge help, thanks @ranierin and team!

1 Like