Announcement

Collapse
No announcement yet.

Perception Neuron Template

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Perception Neuron Template

    Hi MoCap Interests

    this is the discussion thread of my unofficial "Third Person C++ Perception Neuron" template and plugin which I have published for free under the MIT license on Github.
    Perception Neuron (TM) is a motion capture suit from Noitom financed by a Kickstarter campaign in 2014.
    Further details about the suit here: https://neuronmocap.com
    In the meantime also an official plugin version was released => Discussion thread.

    Please use this thread to discuss the technical details about this unofficial template and plugin.

    Download:
    Github Repository
    Latest template version: v0.5.4
    Latest plugin version:
    v0.3.0 (UE4.10)
    v0.3.0 (UE4.11)
    v0.3.0 (UE4.12)
    v0.3.1 (UE4.13)
    v0.3.2 (UE4.14)
    v0.3.3 (UE4.15)
    v0.3.4 (UE4.16)
    v0.3.4 (UE4.17)

    Features Template/Plugin:
    - Live connection over TCP to Axis Neuon player or any other BVH server
    - Playback of locally stored BVH files with stop, pause and rewind
    - Support for multiple instances and different sources (can be shared)
    - Easy to use custom skeleton support with T-Pose, A-Pose or Pose with negative Y axis
    (Just map bone names in event and anim graph. Since plugin v0.3.0 bone coordinate systems are mapped automatically)
    - No dependency to third party libraries and open source solution (MIT license)
    - Remove pelvis drift
    - In-Game and/or In-Editor animation recording (not part of free version, send me a PM if you are interested)

    Changes Template:
    v0.5.4:
    - Remove UE4.16 build warnings

    v0.5.3:
    - Update to Plugin v0.3.3

    v0.5.2:
    - Plugin enable Fix for UE4.14

    v0.5.1:
    - Compile Fixes for UE4.13

    v0.5.0:
    - Cleanup of all Animation BP and switch to new NeuronReadArray function which improves the support of custom skeletons.
    - Remove old custom bone mapping functions (are now deprecated)
    - Add function in Animation BP to remove pelvis drift on ground
    - Change project to UE 4.10
    - Update to Plugin v0.3.0

    v0.4.0:
    - Network and player controller actors now support manual configuration of reference skeleton instead using external BVH reference file.
    - To have an example how to add custom skeletons, the Mannequin now uses the new bone map feature of the plugin.
    For a "HowTo" please have a look into the Mannequin BP function "Init Bone Map".
    - Update to Plugin v0.2.0

    v0.3.0:
    - Source code restructure: C++ code is now placed in a plugin to be more project independend.
    This plugin can be used in any of your projects and there is no need to create a C++ project anymore.
    The binary plugin version 0.1.0 can be downloaded from here:
    https://github.com/Cyx69/TP_ThirdPer...ugin_0.1.0.zip
    - Further comments and minor improvements

    v0.2.0:
    - Playback Controller added.
    Now you can directly play BVH files placed ín your content folder instead connecting to a BVH server.
    See new Blueprint nodes Play and Pause.
    To have an example Neuron character is now connected to a player controller instead to a network controller.
    - Neuron and TPP_Hero are now part of a character blueprint (To have an example for that)
    - Hips bone of TPP_Hero mesh is used to translate complete actor (capsule). Actor rotation is still a problem and is not connected.
    - Add support for standard BVH format
    - Add support for disabled displacement and/or enabled reference bone in motion line
    - Add DNS name support
    - Add Blender source file for Neuron character
    - Further comments and minor improvements

    v0.1.0: Initial version

    Changes Plugin:
    v0.3.4:
    - Remove UE4.16 build warnings

    v0.3.3:
    - Do not resolve HostName if HostName is already a valid IPv4 address

    v0.3.2:
    - Plugin enable Fix for UE4.14

    v0.3.1:
    - Compile Fixes for UE4.13

    v0.3.0:
    - Add better custom skeleton handling. No need to map bone coordinate systems by hand anymore, just map the bone names in anim and event graph.
    - Mark old neuron read functions as deprecated
    - Add BVH player rewind node
    - NeuronRead functions are now able to read the skeleton translation and rotation in component space
    - Compile pre build libraries with UE4.10
    - Further comments and minor improvements

    v0.2.0:
    - It is now possible to initialize Reference skeleton manually and initialization with external BVH reference file is optional.
    - Support all rotation orders and add missing XZY, YZX, ZYX rotations
    - Better support for custom skeletons: Bone coordinate system can now be mapped in blueprint.
    - Fix a synchronization bug when using Neuron format
    - Further comments and minor improvements

    v0.1.0: Initial version (based and included in template v0.3.0)


    Videos:
    Live connection between Perception Axis Player and Unreal Engine 4 (Part 2)
    Live connection between Perception Axis Player and Unreal Engine 4
    Perception Neuron in Unreal Engine 4 Video from galaxyexpress998
    Speech about Perception Neuron and this Plugin at Unreal Meetup #3 Franken (in german) and the used slides (in english)
    Recording Solution (not part of free version, send me a PM if you are interested)

    Applications/Games/Demos based on this Plugin:
    Blood Mine VR Demo (Video)

    Quick Installation Notes Template:
    - Download the template from above link
    - Unpack the ZIP archive to your UE4 template folder, for e.g. "C:\Program Files\Unreal Engine\4.9\Templates\TP_ThirdPersonNeuron".
    Make sure that the project files are placed and visible under "...\Templates\TP_ThirdPersonNeuron" after unzip.
    - Launch Unreal Editor and click on "New project".
    - Under C++ you should now see a template called "TP Neuron".
    - Create a new project with the above template selected.

    Quick Installation Notes Plugin:
    (Not needed if you use the template)
    - Download the plugin under: https://github.com/Cyx69/TP_ThirdPersonNeuron/releases
    - Copy the ZIP File to your closed Unreal project and unpack it directly in the main folder
    - You should see now a new directory "Plugins\PerceptionNeuron" and under "Content" two new folders "Neuron" and "NeuronPlugin"
    - Launch Unreal Editor
    - Under "Edit/Plugins/Project" you should see now a new plugin
    - Add the Neuron Blueprint to your level (See Neuron/Character/Mesh)
    - Add a Network or Player Controller Blueprint to your level (See NeuronPlugin/Blueprints)
    - Connect Neuron Blueprint and Controller with each other in your Level Blueprint (See also "Blueprint Installation Notes.pdf")
    - If you use a Network Controller change the Neuron Animation Blueprint accordingly

    Requirements:
    - Unreal Engine 4.10 or higher
    - MS Visual Studio 2015 (for template)
    - Axis Neuron SW Player (Free version is available here: https://neuronmocap.com/downloads)
    or alternatively you can use my free minimal BVH send server here: https://github.com/Cyx69/bvhsend/releases

    Configuration:
    To connect from the template/plugin to the Axis Neuron SW following settings should be done in the Axis Neuron SW:
    - Open File|Settings dialog
    - Activate under Broadcasting BVH and choose port 7001
    - Configure under Output Format:
    * Frequency reducing: 1
    * Block type: String
    * Rotation: YXZ
    * Activate 'Displacement'
    * Deactivate 'Reference'
    Per default the template/plugin connects to 'localhost' (127.0.0.1) and to port 7001.
    You can change these settings in the Controller Blueprint if you want to use a separate PC or other ports.
    And you can add further controllers if you want to retrieve motion data from additional suits or BVH server.

    Donate:
    If you find the template/plugin or parts of it useful, please support me and the further development and make a donation here:
    Donate

    Best regards
    Heiko Fink aka Cyx
    Last edited by Cyxx; 08-21-2017, 02:01 PM. Reason: New release for UE4.15
    Follow me on Twitter: @heikofink69 | Game: Crazy Max VR | Plugins: Perception Neuron Plugin and Template, Texture Merge | Demo: Blood Mine VR with PN

  • #2
    This is an amazing gift to the community and exactly what I was waiting for in order to preview a motion in the engine. Can't wait to try it out. Thanks Cyx!

    From a workflow perspective though how do you see this fitting into an animation pipeline? In its current form is its purpose simply to preview motion or do you see potential in it to grab a raw sequence directly in UE4 and start using it in Persona? (I would be blown away if we could already do this). As I understand it, and I could be wrong, you'd still need to save out the motion from Axis Neuron, do some clean up (which might include skeletal translation as I think all the bones have different names in Axis Neuron), and import the fbx into UE4. Does that sound right?

    I'm still getting my head around the Axis Neuron -> UE4 workflow and would love to hear more views on it. Admittedly all I'm able to do at the moment is get some nice mocap in Axis Neuron and save it as fbx. I haven't the foggiest how to apply it properly to the UE4 mannequin. When I tried to import the fbx onto the mannequin control rig it was a mess. I did manage to make it less messy by renaming the skeleton from the fbx to match the mannequin but figured there must be a better way/process.
    Last edited by Naveed; 09-22-2015, 09:31 AM.
    twitter: @NaveedHIQ @Guvnorgames

    Comment


    • #3
      @Naveed:
      My focus so far was to get a live connection between UE4 and Axis Player and use it later also in a VR environment.
      But I think it should be possible to store the values of the rotation and translation array for e.g. in a file and playback it later.
      Some small enhancements in the Event Graph of the Animation BP should do it.

      But may be there is a better way for your purpose if you have some Blender KnowHow:
      - Export your animation for instance in BVH format and import it in Blender. Apply a simple mesh to it, export it as FBX and import this to Unreal.
      - Then retarget your animation to the Mannquin skeleton like it is described here: https://docs.unrealengine.com/latest...ons/index.html
      Follow me on Twitter: @heikofink69 | Game: Crazy Max VR | Plugins: Perception Neuron Plugin and Template, Texture Merge | Demo: Blood Mine VR with PN

      Comment


      • #4
        Well Cyxx,

        I have to say that I'm super impressed by this

        I got everything up and running in short order and was able to test some basic data that I captured earlier this week with one of the suites we have.
        Other than some finger movement oddities it works very well. I'll get around to doing some live capture tests later tomorrow but first thing tomorrow I'll be sending you my donation. Great job ! You'll certainly be hearing more from me shortly

        MrNexy

        Comment


        • #5
          Less then a month and I'll be receiving my kits...can't wait to test it out with UE4 livestreaming!!!

          Thanks again Heiko, this is great!

          Guys don't forget to donate, this is a marvelous gift
          ENTER REALITY

          VR Solutions

          Contact us for more informations

          Comment


          • #6
            It sounds awesome! As soon as I've got my kits I will test and donate it.

            It's such a great thing to test your animations directly in your game environment.

            Game Design - Photogrammetry - Programming
            Tutorials · Twitter · Twitch

            Comment


            • #7
              Thanks, will try it.

              Comment


              • #8
                Heiko, this is awesome, thanks for sharing this. I've tested it with recordings and live streaming, works fantastically! Donation done, too
                VR Shooter Guns - Arcade Shooter for Vive
                Magic Tavern - A Magic Shooting Gallery for Virtual Reality
                Unreal Meetup Franken - Unreal Engine 4 Meetup
                Hands for VR: Basic - For Vive and Oculus [Marketplace]
                Hands for VR: SciFi - For HTC Vive and Oculus Touch [Marketplace]

                Comment


                • #9
                  Hi. can any of you guys show some videos of the perception neuron working? or give a short review? I am interested in this system but it is so new that there are not many reviews out there... how is the latency on the "live" set up... and do you have to do a lot of cleaning after you record something? Thanks!

                  Comment


                  • #10
                    catalejo, you can download the neuron software for free here: https://neuronmocap.com/downloads
                    it comes with several recordings that you can play back and test this template with.
                    VR Shooter Guns - Arcade Shooter for Vive
                    Magic Tavern - A Magic Shooting Gallery for Virtual Reality
                    Unreal Meetup Franken - Unreal Engine 4 Meetup
                    Hands for VR: Basic - For Vive and Oculus [Marketplace]
                    Hands for VR: SciFi - For HTC Vive and Oculus Touch [Marketplace]

                    Comment


                    • #11
                      Thanks btengelh, i will do so, but still a review or video samples from a user could be really useful for evaluating this system

                      Comment


                      • #12
                        well, it works like in the videos noitom posted
                        you just have to stay away from magnetic sources. e.g. I can't use it at my desk, because there is too much interference from whatever (a lot of drift).
                        in the next room, the PN works flawlessly
                        VR Shooter Guns - Arcade Shooter for Vive
                        Magic Tavern - A Magic Shooting Gallery for Virtual Reality
                        Unreal Meetup Franken - Unreal Engine 4 Meetup
                        Hands for VR: Basic - For Vive and Oculus [Marketplace]
                        Hands for VR: SciFi - For HTC Vive and Oculus Touch [Marketplace]

                        Comment


                        • #13
                          Hey there Cyxx !

                          I'm having a little difficulty getting my custom mesh to replicate the movement of the data stream. I'm using Blender btw
                          I followed the steps below , please do let me know if I'm on the right track and if I'm missing any steps that you may have followed yourself.

                          1- Export Neuron Mesh from Template as FBX.
                          2- Import into blender and replace Neuron Mesh with Custom Mesh, make sure that it's rigged.
                          3- Export Custom mesh as FBX (I have a feeling that my settings may be off)
                          4- Import Custom Mesh FBX into UE4, Retarget Nueron Mesh's AnimBP to Custom Mesh.
                          5- Go through New AnimBP resulting from Retarget, make sure that all the Transform(Modify)Bone nodes match my current skeleton's bones.
                          6- Drop Skeletal Mesh into scene,assign the new AnimBP to it. Then open the Level Blueprint and set the reference in there.

                          Thus far it works with the exception that the movement is wildly off. The character is often upside down with arms and legs backwards etc, moving but not in the right way.
                          I would be most grateful if you could share the blender workflow you followed for preparing your custom mesh, what FBX settings you used for export and possibly what version of blender you used.
                          Thanks in advance !

                          Best Regards
                          MrNexy

                          Comment


                          • #14
                            Hi MrNexy

                            you don't need to export the Neuron Mesh, I have published the Blender source file for free here:
                            https://github.com/Cyx69/TP_ThirdPer...er/Blender.zip

                            To add your mesh you can follow these steps in Blender:
                            - Delete the old Neuron (Makehuman) mesh
                            - Add your own mesh and tweak the mesh in that way that the armature fits into the mesh
                            - Parent the mesh to the existing armature with empty or automated weighted groups (Whatever you like to work with)
                            - Check all groups if they are present and correctly weighted
                            - Use my Blender export settings that I described in der Blender.zip
                            - Import the FBX to Unreal and choose the Neuron skeleton or create a new one if you prefer that

                            You could also manipulate the armature in Blender if you want and import it as a new skeleton in Unreal.
                            For instance the length of a bone, but then you have to switch the transform settings to 'Ignore' in the Animation graph for that bone.
                            But do not change the rotation of a bone and make sure that all Roll values for the bones are set to zero.
                            And if you use a new skeleton you may be have to check all 'Transform (Modify) Bone' nodes in the Anim graph if the 'Bone to Modify' setting is still correct.

                            Best regards
                            Heiko
                            Follow me on Twitter: @heikofink69 | Game: Crazy Max VR | Plugins: Perception Neuron Plugin and Template, Texture Merge | Demo: Blood Mine VR with PN

                            Comment


                            • #15
                              Thanks Heiko !

                              Thank does the trick, thanks for going the extra mile good sir.

                              -MrNexy

                              Comment

                              Working...
                              X