Hello, I’m new to UE and currently using version 5.4.4.
After reviewing much of the discussions here, I have a few questions:
-
I noticed there are tutorials for Neural Network Inference (NNE) in version 5.3, but I haven’t been able to find any for version 5.4. Have these not been released yet?
-
Does UE’s NNE offer any tools for handling large language models (LLMs), such as a “tokenizer” for converting text to token IDs?
Specifically, LLMs go through an auto-regressive process, where the model generates tokens iteratively until an end-of-sequence token is produced, feeding previously generated tokens back into the model. Is this process supported?
The reason I ask is that when testing an ONNX-converted LLM model in UE 5.3,
I could only identify the following input and output Tensor Shapes
Input Shape: input-ids, attention-mask, position-ids
Output Shape:logits
However, I wasn’t able to observe the entire auto-regressive generation process that results in a final long sequence text output.
- If support for the auto-regressive generation process and tokenizers is not yet available,
would I need to implement these features myself at the C++ level?