[Feature Request] Official Machine Learning API Support. (e.g. cuDNN, tflow, tflow Lite)

General Feedback & Requests
1

cuDNN is Nvidia’s official machine learning driver for their processors: NVIDIA cuDNN | NVIDIA Developer

Tensorflow is Google’s open-source, optimized, and constantly-updated machine learning API built on top of those drivers. It has unofficial plugin support here: GitHub - getnamo/tensorflow-ue4: TensorFlow plugin for Unreal Engine 4

However, this plugin lacks things like Lite support and only gets updated with the free time of its contributors. I want to make an argument that this is an essential feature for the future of this medium, for research purposes and for the creative and gaming end. I won’t focus on game AI as that’s an obvious one, including things like speech/dialogue generation or planning against you.

Graphics Optimization
Ray tracing is just one thing. The new tensor processors on the market and the vast number of graphical applications for tensor-based models make machine learning the next computing frontier including, say, shader optimization. The official Nvidia example of shader optimization is the ray tracing, however that’s a very basic example. Compute shaders like for particle generation may also be implemented to increase visual fidelity while drastically increasing speed: FluidNet example You can generate unique shaders that use minimal computations so that a phone could run it real-time: Universal Style Transfer example This math is applicable to a very wide set of problems, this is just one implementation (thus the “Universal” tag).

These models are fine-tunable and you can switch out neural net architectures that might be better for the job (e.g. a Capsule net vs a CNN, numbers of layers, layer resolution, activation functions, encoders & decoders / different signal analysis methods, etc.).

Art Generation
You can generate rough 3D models from datasets: CNN example, 3DR2N2 example, ShapeNet database
You can generate segmented 3D scenes from point cloud data: PointNet example
ARCore uses a sparse point cloud to do its job, and there’s open source photogrammetry software like Meshroom that could integrate pointNet or 3DR2N2. It has many AR applications, too.

You can generate animations: Mode-Adaptive Neural Nets, Motion Transfer,Phase Functioned NNs, FastMotion Transfer based off Universal Style Transfer

Hell, you can even generate a whole game: Nvidia example - Same team that did the first Motion Transfer example linked above.

Research Visualization
I know that people at NASA, people in universities, and people like me like to use Unreal for visualization and engineering application purposes. It’s easy to make UIs, toy with physics, and leverage C++ with blueprints. With protein modeling, tumor identification, drone navigation, etc., 3D visualization is growing to be an essential part of how the researcher understands, interacts with, and communicates their work. Most of ML is going into helping with simulation and optimization, something Unreal is made for.

In Conclusion
So while people are out there building these plugins and research implementations in bits and pieces across many frameworks, why not unify and optimize the effort with the studio’s resources?

The above research shows modern hardware is capable of running much of this real-time for any application. When you’re decoding a learned data set, say for shaders, then that can be much faster than traditional linear and differential equations. A few more years of hardware updates and there won’t be any performance issues. I cannot express enough what kind of doors this could open for the creative, research, and gaming world. You really have to learn a bit about the 2018 ML field to see the vast applications and where it’s all going, and for all the little details in game and simulation and real-world systems.

Functions to Add/Optimize
Neural nets are more or less just pipelines for regression and data comparison that can be plugged and unplugged. This isn’t comprehensive so help me add on. Consider this the dream list.

Data Structures:
Tensors - these are just n-dimensional matrices. Options for padding, dynamic typing, GPU or CPU-based (i.e. single vs double precision), etc.

General Functions:
Summing/Averaging/Masking/Dot/Cross Product/Other basic matrix math (already present) but extended for tensors, including tensor products
Covariance/Cross-correlation
Autocorrelation
Convolution
Eigenvalues/Eigenvectors
Feature Transforms e.g. WCT,Wavelet Transforms, Affine Transforms
Regression Statistics & Weighting
Gradient Descent

Noise Reduction/Loss Functions:
Max Ent
Cross Ent
L1/L2 Loss
SSIM
Mean Square Error
Margin/Soft Margin
NLL
log(cosh(x))
Squashing
Gradient Boosting

Activation Functions:
ReLU
LReLU
tanh
sigmoid
Identity
Softmax

NN Templates:
Examples

Other:
Renormalization Groups - fundamental but uncommonly used by the ML scene (because y’all need to keep up).

2

The problem with all of these is that games, especially games that have enough budget to mess around a lot with machine learning, also tend towards extreme optimization. And when I say extreme, I mean the most optimized games may be second only to high frequency trading as the most optimized code on the planet. The Last of Us had a bunch of shaders re-written by hand in assembly just to get it to run.

APIs like Tensorflow just aren’t meant to output such things. I mean, they might be optimized compared to some roll your own machine learning stuff, but a lot of that just ends up being created in Python and the like, IE more about easy editing and getting it to work that actually having it run fast for say, when your trying to deploy the NN to get the most performance. Not to say that there aren’t a bunch of particular use cases UE users might have for using these things. I can easily imagine some people happy to run third party, generalized machine learning APIs in conjunction with UE projects.

But that being said, I see other, purpose built machine learning tools, with heavy optimization for next gen consoles and PC, as far more relevant to the largest users of UE. There’s already people releasing their own machine learning based plugins for UE that are meant for games from the ground up. EG Deep Motion for animation and Physics Forests for fluid sim. These sorts of things feel far more appropriate for games in particular than building something from Tensorflow. But as I said I can see use cases still. Though whether it’s enough for Epic to bother with official support IDK, I guess it depends on how many requests they receive for such.

3

I’m definitely not the expert in APIs or what mixes best with game engines, tflow’s just the most robust open source example I know of. Point is it’s an open source API that gives you high level access to fairly optimized parallel computing and signal analysis functions for machine learning, stuff that takes a while to write from scratch if you’re one of the handful with the know-how. It’s just a bunch of prewritten cuDNN and now TensorRT functions and wrappers really. You can always go in and optimize what are essentially templates since it’s open source, just like we do with Unreal already.

What I see is these frameworks are almost daily becoming much better generalized for average hardware and machine learning algorithms as a whole are getting easier to think with. As well, Julia language for example is ground-up written to take full advantage of the gpu, and you can have that talk with C++ and python programs to write shaders or other compute functions with better parallel computing support at the base, meaning it’s totally possible to have simultaneously optimal and generalized code in this day and age. I see dozens of use cases with ML for 3D in general, many of which are not feasible yet for the common hardware (thus a shareholder wanting results for this quarter and not after 2 years) but that’s not the point. The examples you listed again are just specific implementations of ML modeling where there could be a general framework. Look at the Universal Style Transfer math, it’s about as modular and general as you can get, thus it transfers really well to other problems. Most ML problems fall under different a few classifications (e.g. visual-spatial, recurrent/memory learning, attention, etc.) which have many different competing general models (e.g. CNNs vs CapsNets, or ResNet vs AlexNet, RNNs vs LSTMs, Manifold Learning vs Deep Learning, different rectifiers/decoders, different layering/routing, different transforms, denoising, etc etc) for doing the computing job with good performance and minimal loss. The rest is about adapting those to specific problems.

This is gonna be a big part of the future if we learn how to utilize it carefully. I’m pretty sure even fine-tuned shaders like the ones for the Last of Us could be found through ML optimization without the same kind of man-hours required for an insane assembly shader (and of course we won’t really have the hardware limitations that will necessitate such a job but there’s always the next level), there’s a huge window for exploration and many many layers in graphical computing where that kind of modeling can and should be applied. Granted, we have to use priors that took a ton of man-hours to base optimizations off of, but there is a toooooooon of code, as well as data and phd theses laying around. Like I’m baffled there aren’t more people recognizing what we’re sitting on here.

The creative space is a lot more than using the tools, it’s arguably just as much (if not more) in the creation of the tools.

4

Sure, but a lot of what you discussed are offline tasks to later be deployed to the game. Which means it might be more interesting to integrate such tools into the editor than into runtime. You need high optimization for inferencing only if your running a fully packaged game, at least in general. Even Drivatar, MS’s driving neural net for Forza games that’s been around for forever now, runs offline creation then deploys the optimized NN profiles for running in games.

It’s not to say there’s not space for doing realtime learning in games. Black and White did all the way back when, programmed by one of the founders of Deep Mind before he went off and did that. But it’s hard enough to get many interesting things out of neural nets as it is, and that’s with banks of the highest end GPUs available. If you want interesting learning in realtime you probably need more optimization, and less generalization, than any of these open source tools will provide. Basically with games the rule if often “look at what people are doing not in realtime. Now try to make a hacky version that runs a thousand times faster.” with the “thousand times faster” not being exaggerated in the least. There’s now an EA cloud platform, and a new company being made by ex EA employees, that both claim NN’s and “the cloud” will “enable game experiences like never before” including things like “making you believe your talking to a real person”. And they are f*ing kidding themselves, I’m so glad they’re going to suddenly be better at everything than everyone else on the planet for the sake of a game. But a little more down to earth visions seem more practical.

EG being able to use open source, friendly tools for fooling around in offline tasks, in the editor, is really interesting. Could you use it for level creation? NPC creation? Could you fool around NN’s in development and only bother to optimize them later, when you package off the game? Easy, official API support for the editor seems like a better way to go at the moment than bothering anything with runtime.

5

That’s what I am suggesting. Most of the examples I cited provide pre-trained models (which would be great to create in-editor for the rapid iteration and visualization abilities) to the user, then the user sees what the model reconstructs from its learning for whatever functional task it provides (e.g. style transfer or computer vision). Training is inherently slow and resource intensive, especially for some of the graphical tasks I highlighted, but trained models have the potential to provide better graphics optimization (like the Eulerian Smoke CNN, but with an even better NN model). Autoencoders have a ton of possible uses, for example, and could end up replacing or creating complex shading and compute systems. I think RTX uses an autoencoder too, and it involves some live unsupervised training in runtime pretty sure. The “specialization” of a neural net generally comes down to what regressions, transforms, and data structures are you using that best represent the thing you’re trying to change or control, most of it is generic otherwise and basically dependent on how good the coder or API is with hardware optimization - and how good your resolution needs to be.

And as for EA, or any of these companies (don’t even get me started on blockchain), mystification and overpromising always benefits the one lining their pockets and not the reality of the math and engineering, and art. That shouldn’t stop creative infrastructure from moving forward for everybody else, I believe the vast majority of us have been distracted from the most important parts of this medium - the community and enabling creativity. One of the major things that used to stop me from designing graphical applications was not having engines like Unreal around, and slogging through basic graphics engine structures and shaders was obtuse and not fun for me. Now think about ML having that kind of ease and then that bleeding into the creative space.

Ah here’s another great one I forgot about, StyLit for live hand-drawing style transfer onto 3D models: http://stylit.org/
I think Disney uses this one for their cloud lighting (and it’s near-realtime capable already, or can be baked): Deep Scattering

6

Looks like some people from SIGGRAPH 2018 get it: GitHub - smartgeometry-ucl/dl4g: Example code for the Siggraph Asia Tutorial CreativeAI

7

Unity have bloated their boat so much with things like this that they had to start over with a Unity “Tiny” framework… Just like Visual Studio vs Visual Code.

While people are asking for more and more things like this to be built-in, I wish EVERYTHING were plugins, so I could easily disable all the bloated stuff I particularly have NO intention to use in Unreal.

An interesting scene that is growing amongst engine users is a group of devs using smaller less known engines because Unity and Unreal “-have too much, mostly things my game will never need and is a waste to build my project on top of all that zombie code…” (thus Unity now starting their “Tiny” project trying to take back that group of devs).

Things like in OP probably need core pieces to be injected into the core engine, can’t be just a plugin, and for that I am against Epic implementing these things making grow “engine cost” (runtime performance cost of engine below the game I’m building) and I can’t opt out by simply ticking a plugin off.

8

An official plugin? Maybe someone else could make it and put it on the marketplace?

9

That makes the most sense. And Unreal has gotten much better about the bloat over time.

10

i hope to see epic doing something on this topic , there is no good informations about machine learning in ue4 too , if they made an official api for this they will add lot of reference and resource for that too . i hope they focus on tensorflow for ue4 in future