Announcement

Collapse
No announcement yet.

TensorFlow

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • started a topic [PLUGIN] TensorFlow

    TensorFlow

    Hello devs!

    For the past couple of months I've been working on bringing TensorFlow to UE4, enabling the use of state of the art machine learning in our projects. The plugin has reached something I think is ready to try out, but it is very much a work in progress so please check it out and give some feedback if you can!

    Download
    Plugin


    Examples Project


    Limitations
    Windows platform only atm. Consider contributing to add support for your platform!

    Resources
    Latest documentation is always available at the github repositories.

    Plugin
    https://github.com/getnamo/tensorflow-ue4
    Examples Project
    https://github.com/getnamo/tensorflow-ue4-examples


    Examples - Mnist

    Made a simple mnist recognition example.


    Basic MNIST softmax classifier trained on begin play with sample training inputs streamed to the editor during training. When fully trained, UTexture2D (1-3) samples are tested for prediction.

    you can even use your browser on your phone or desktop to draw some shapes to try recognition



    Examples - Basic adding in TensorFlow
    Want just the basics? check out the basic.umap map which uses addExample.py to add float array together. This is a good entry point to see how to pass UE4 blueprint data into and out of python tensorflow.




    Examples - Upcoming Pong with QLearning AI

    Still training the pong AI with sped up time dilation, but you can track updates in the qlearn branch of the examples repository.




    Let me know what you think and what machine learning examples you'd like to see in the future!

  • replied
    Originally posted by jimmyw_404 View Post
    Love your plugin. I'm thinking about making something quite similar that merely visualizes different information from TensorFlow in an Unreal Engine 3D world.

    I don't know exactly how it'll look yet, but some combination of these two videos:

    https://www.youtube.com/watch?v=3JQ3hYko51Y

    https://www.youtube.com/watch?v=AgkfIQ4IGaM

    I've looked pretty hard and haven't seen anything close to what I'm thinking of building. If I move forward with it I think your plugin will help jumpstart my progress, thanks!
    Sounds like a cool project! The only thing I'm aware of that is similar using this plugin is this german thesis work by Marcel Bock https://www.semanticscholar.org/pape...eefdf65d362b10, article: https://www.dlr.de/sc/en/desktopdefa...74_read-53313/

    Might want to take a look for inspiration.

    Leave a comment:


  • replied
    Originally posted by getnamo View Post

    The plugin is using an embedded python setup, so a separate tensorflow installation is managed by the plugin. What you do need to do is ensure you've installed the cuda and cudNN pre-requisites for gpu https://www.tensorflow.org/install/gpu#windows_setup. Please check https://github.com/getnamo/tensorflo...llation--setup
    Love your plugin. I'm thinking about making something quite similar that merely visualizes different information from TensorFlow in an Unreal Engine 3D world.

    I don't know exactly how it'll look yet, but some combination of these two videos:

    https://www.youtube.com/watch?v=3JQ3hYko51Y

    https://www.youtube.com/watch?v=AgkfIQ4IGaM

    I've looked pretty hard and haven't seen anything close to what I'm thinking of building. If I move forward with it I think your plugin will help jumpstart my progress, thanks!

    Leave a comment:


  • replied
    I managed to get a TF2 version of addExample.py working, but I'm a bit confused on how to correctly implement
    the operation self.c = self.a + self.b

    Code:
    import tensorflow as tf
      import unreal_engine as ue
      from TFPluginAPI import TFPluginAPI
        classExampleAPI(TFPluginAPI):
        #expected optional api: setup your model for training
      defonSetup(self):
        self.a = tf.Variable([0.0], tf.float32)
      self.b = tf.Variable([0.0], tf.float32)
        self.op = tf.Variable(True, tf.bool)
      pass
      
      #expected optional api: parse input object and return a result object, which will be converted to json for UE4
      defonJsonInput(self, jsonInput):
      
      print(jsonInput)
        self.a = tf.dtypes.cast(jsonInput['a'], tf.float32)
      self.b = tf.dtypes.cast(jsonInput['b'], tf.float32)
        ifself.op:
      return tf.add(self.a, self.b).numpy().tolist()
      
      else:
      return tf.subtract(self.a, self.b).numpy().tolist()
        
      
        #custom function to change the op
      defchangeOperation(self, type):
      if(type=='+'):
      self.op =True
        elif(type=='-'):
      self.op =False
        defgetVersion(self, jsonInput):
                ver = tf.__version__
      print(ver)
      return("GPU Available: ", tf.test.is_gpu_available())
     
    #expected optional api: start training your network
      defonBeginTraining(self):
      pass
       
      #NOTE: this is a module function, not a class function. Change your CLASSNAME to reflect your class
      #required function to get our api
      defgetApi():
      #return CLASSNAME.getInstance()
      return ExampleAPI.getInstance()

    Leave a comment:


  • replied
    Originally posted by StevePeters View Post
    I'm getting an issue of ImportError: DLL load failed: The specified module could not be found. similar to getnamo above, however I have tensorflow gpu successfully installed on my computer. I installed tensorflow with Anaconda. Could that have something to do with the plugin not finding it?
    The plugin is using an embedded python setup, so a separate tensorflow installation is managed by the plugin. What you do need to do is ensure you've installed the cuda and cudNN pre-requisites for gpu https://www.tensorflow.org/install/gpu#windows_setup. Please check https://github.com/getnamo/tensorflo...llation--setup

    Leave a comment:


  • replied
    I'm getting an issue of ImportError: DLL load failed: The specified module could not be found. similar to getnamo above, however I have tensorflow gpu successfully installed on my computer. I installed tensorflow with Anaconda. Could that have something to do with the plugin not finding it?

    Leave a comment:


  • replied
    Originally posted by envenger View Post
    I have an issue, no matter what i return, the answer is always None.
    Click image for larger version Name:	Capture.PNG Views:	1 Size:	87.5 KB ID:	1610884

    How is https://github.com/getnamo/tensorflow-native-ue4 version of your plugin currently?

    I have a trained model but using python plugin is a huge pain to test.
    That function doesn't return as you think. The python TensorflowComponent wraps callbacks such that they will automatically call back on json_input_gt_callback whether you use multi-threading or not. If you do have multi-threading on you wouldn't be able to receive the answer within a function callback anyway. You need to listen to json_input_gt_callback function which has the json results you're looking for. See https://github.com/getnamo/tensorflo...ponent.py#L101 for the python logic handling this. You can modify that section to return the results directly if you don't use multi-threading.

    I generally haven't used this plugin with c++ inference, typically developing and calling json input from BP is more amenable to ML prototyping. That said I think a refactor is in order which will allow the tensorflow component to be called natively which would simplify cases like these (and using the same api to call remote python servers), this refactor may be a while though as I don't have free opensource time in the near term.

    Originally posted by envenger View Post
    Each time i compile without closing the project, the socketIO plugin crashes the project.
    Any logs from such crashes would be welcome as a new issue under https://github.com/getnamo/socketio-client-ue4/issues, please give a detailed repro if possible.

    Originally posted by envenger View Post
    How is https://github.com/getnamo/tensorflow-native-ue4 version of your plugin currently?
    ​​​​​​​
    Same issue with time currently, but you can try using the c_api natively right now (try any https://github.com/Neargye/hello_tf_c_api, e.g. https://github.com/Neargye/hello_tf_...ession_run.cpp), it just doesn't have any ue4 specific data formats or conveniences implemented. If you get some good workflow going consider contributing.

    Leave a comment:


  • replied
    I have an issue, no matter what i return, the answer is always None.
    Click image for larger version  Name:	Capture.PNG Views:	1 Size:	87.5 KB ID:	1610884

    How is https://github.com/getnamo/tensorflow-native-ue4 version of your plugin currently?

    I have a trained model but using python plugin is a huge pain to test.
    Each time i compile without closing the project, the socketIO plugin crashes the project.
    Last edited by envenger; 04-22-2019, 06:33 PM.

    Leave a comment:


  • replied
    Originally posted by envenger View Post
    I am getting the following error when opening a component after installing tensorflow 2.0
    Click image for larger version

Name:	Capture.PNG
Views:	1
Size:	167.2 KB
ID:	1605275
    open an issue at https://github.com/getnamo/tensorflow-ue4/issues and give detailed steps to reproduce behavior.

    Leave a comment:


  • replied
    I am getting the following error when opening a component after installing tensorflow 2.0
    Click image for larger version

Name:	Capture.PNG
Views:	1
Size:	167.2 KB
ID:	1605275

    Leave a comment:


  • replied
    Another simple question although this is more related to the python plugin but I didn't know where to ask the question.
    https://github.com/20tab/UnrealEnginePython/issues/697

    I have a py file inside a folder inside scripts, or any folder inside content folder.
    Then I add this path to additonal path under python but when making a pyactor and adding that class to python module, I am unable to make it work.

    Any idea how to make something like that work?

    Leave a comment:


  • replied
    Originally posted by envenger View Post
    How does the package scaling version work?

    Suppose I package a version and I use CUDA and cudNN but the PC it would be running on doesn't have CUDA or cudNN but has a card that supports them, would it still work?
    How would it work on a system not having a GPU can it detect and run the processor version?
    See https://github.com/getnamo/tensorflow-ue4/issues/39 for a discussion about this problem. The plugin currently assumes you distribute the correct version by specifying it in the upymodule.json. There could be some ways of detecting the compute capability of the computer it runs on and selectively pull dependencies via pip and that enhancement would be a good contribution to the plugin. For now I'd recommend using the cpu version if you're unsure of environment.

    Alternatively if you do have a model that is more inference focused, there may be mileage from using the tensorflow c_api to run on say a frozen model or .pb file, there are native dll distributions available that come with the requisite gpu binds already directly embedded in a tensorflow.dll. In https://github.com/getnamo/tensorflow-native-ue4 I'm using those dlls and exploring how a more native tensorflow plugin might look. In it's current state, it will correctly load the dll and you can call the c_api from c++, but there is currently no example or unreal specific helping code (Still very WIP). I suspect this approach should be more amenable to being embedded in games with far fewer dependencies, while letting you train/research using e.g. the python based tensorflow-ue4 plugin.

    Finally another common approach is to run the tensorflow code on a server/cloud instance and to just use networking (e.g. the socket.io plugin) to pass data back and forth. That comes with it's own drawbacks though (requiring internet connection, scaling costs).
    Last edited by getnamo; 03-26-2019, 12:30 AM.

    Leave a comment:


  • replied
    How does the package scaling version work?

    Suppose I package a version and I use CUDA and cudNN but the PC it would be running on doesn't have CUDA or cudNN but has a card that supports them, would it still work?
    How would it work on a system not having a GPU can it detect and run the processor version?

    Leave a comment:


  • replied
    Originally posted by David.Bromberg View Post

    Thanks! I take it this means this plugin could be used for inference on tensorflow networks that have been built outside of Unreal, assuming we can find a way to package the .pb files? I imagine there'd still need to be some work to get tensorflow itself to compile on consoles as well, but this is a good start. Thanks!
    That's the idea, here is some example c_api for loading a graph and running a session: https://github.com/Neargye/hello_tf_...ession_run.cpp. Wrapping that into a more unreal way would be the next step to make things easier.

    In terms of packaging the .pb files, it's not a big issue you can just add your pbs as runtime dependencies and they'll get packaged along with other things, see https://github.com/getnamo/tensorflo...w.Build.cs#L32 on how that's done for the script files.

    Keep in mind that another way people do machine learning is to run it as a cloud service and just pipe data to your server and get results back, then you don't have to worry about compatibility for your platform of choice.
    Last edited by getnamo; 03-15-2019, 03:51 AM.

    Leave a comment:


  • replied
    Originally posted by getnamo View Post

    https://github.com/getnamo/tensorflow-native-ue4 is now functional if you want to use the tensorflow c_api directly, but there are no convenience unreal type bindings yet. Will come back to this later.
    Thanks! I take it this means this plugin could be used for inference on tensorflow networks that have been built outside of Unreal, assuming we can find a way to package the .pb files? I imagine there'd still need to be some work to get tensorflow itself to compile on consoles as well, but this is a good start. Thanks!

    Leave a comment:

Working...
X