[PLUGIN] TensorFlow

[getnamo]

Hello devs!

For the past couple of months I've been working on bringing TensorFlow to UE4, enabling the use of state of the art machine learning in our projects. The plugin has reached something I think is ready to try out, but it is very much a work in progress so please check it out and give some feedback if you can!

Download
Plugin


Examples Project


Limitations
Windows platform only atm. Consider contributing to add support for your platform!

Resources
Latest documentation is always available at the github repositories.

Plugin
https://github.com/getnamo/tensorflow-ue4
Examples Project
https://github.com/getnamo/tensorflow-ue4-examples


Examples - Mnist

Made a simple mnist recognition example.


Basic MNIST softmax classifier trained on begin play with sample training inputs streamed to the editor during training. When fully trained, UTexture2D (1-3) samples are tested for prediction.

you can even use your browser on your phone or desktop to draw some shapes to try recognition



Examples - Basic adding in TensorFlow
Want just the basics? check out the basic.umap map which uses addExample.py to add float array together. This is a good entry point to see how to pass UE4 blueprint data into and out of python tensorflow.




Examples - Upcoming Pong with QLearning AI

Still training the pong AI with sped up time dilation, but you can track updates in the qlearn branch of the examples repository.




Let me know what you think and what machine learning examples you'd like to see in the future!

[n00854180t]

getnamo - This is awesome!

There are certain really cool projects that use TF that I'd love to be able to integrate into a game, e.g., Google's NLP parser SyntaxNet.

Training a net to differentiate a person stepping vs just leaning in VR would also be really cool (though there's a nice asset for doing this already on the marketplace VR Footstep Plugin), and allow people to use full body avatars in VR without having to have extra trackers other than hands and head.

[duyth]

Cool stuff
Would be awesome if we could use this to Train enemy AI in a RPG game /or enemies in RTS game and then save the progress

[HeadClot]

This is awesome!

[rYuxq]

Cool stuff
Would be awesome if we could use this to Train enemy AI in a RPG game /or enemies in RTS game and then save the progress

You actually can

[rYuxq]

I am wondering if the new editor scripting capabilities can be used to directly integrate tensorflow into our projects?

[RoxyRypler]

It would be cool to see an LSTM for sequence learning in this! enemy predicting the players actions and acting on it.

[IOchair]

Just curious: Can this plugin work after packaging?

[getnamo]

Packaging support is a known missing feature. Progress is tracked here: https://github.com/getnamo/tensorflow-ue4/issues/16, contributions welcome

[HaBe2305]

First of all, thank you very much getnamo for your tremendous amount of work you're giving away basically for free (the couple of bucks on the marketplace are merely symbolic in this regard). I've started to play with Reinforcement Learning and tensorflow using Unity and the Learning Agents preview they provide, but now have switched to Unreal because of - well, many reasons.

So already working with tensorflow, I was and still am excited that the Unreal environment also has integration already. Big thanks again for making this possible!

I also was used to having tensorboard for better understanding what is going on with my learning agent. I've tried for a day now to get tensorboard running (writing out logs with the simple MNIST example from https://www.tensorflow.org/guide/sum...nd_tensorboard worked flawlessly), but I'm totally lost now, also because I'm just beginning writing python code, coming from a C++ and C# background. Have you tested running tensorboard already and could kindly give some guideline or tips how to make it work?