User Tag List

Page 1 of 4 123 ... LastLast
Results 1 to 40 of 143

Thread: Perception Neuron Template

  1. #1
    1

    Perception Neuron Template

    Hi MoCap Interests

    this is the discussion thread of my unofficial "Third Person C++ Perception Neuron" template and plugin which I have published for free under the MIT license on Github.
    Perception Neuron (TM) is a motion capture suit from Noitom financed by a Kickstarter campaign in 2014.
    Further details about the suit here: https://neuronmocap.com
    In the meantime also an official plugin version was released => Discussion thread.

    Please use this thread to discuss the technical details about this unofficial template and plugin.

    Download:
    Github Repository
    Latest template version: v0.5.4
    Latest plugin version: v0.3.0 (UE4.10) v0.3.0 (UE4.11) v0.3.0 (UE4.12) v0.3.1 (UE4.13) v0.3.2 (UE4.14) v0.3.3 (UE4.15)
    v0.3.4 (UE4.16)

    Features Template/Plugin:
    - Live connection over TCP to Axis Neuon player or any other BVH server
    - Playback of locally stored BVH files with stop, pause and rewind
    - Support for multiple instances and different sources (can be shared)
    - Easy to use custom skeleton support with T-Pose, A-Pose or Pose with negative Y axis
    (Just map bone names in event and anim graph. Since plugin v0.3.0 bone coordinate systems are mapped automatically)
    - No dependency to third party libraries and open source solution (MIT license)
    - Remove pelvis drift
    - In-Game and/or In-Editor animation recording (not part of free version, send me a PM if you are interested)

    Changes Template:
    v0.5.4:
    - Remove UE4.16 build warnings

    v0.5.3:
    - Update to Plugin v0.3.3

    v0.5.2:
    - Plugin enable Fix for UE4.14

    v0.5.1:
    - Compile Fixes for UE4.13

    v0.5.0:
    - Cleanup of all Animation BP and switch to new NeuronReadArray function which improves the support of custom skeletons.
    - Remove old custom bone mapping functions (are now deprecated)
    - Add function in Animation BP to remove pelvis drift on ground
    - Change project to UE 4.10
    - Update to Plugin v0.3.0

    v0.4.0:
    - Network and player controller actors now support manual configuration of reference skeleton instead using external BVH reference file.
    - To have an example how to add custom skeletons, the Mannequin now uses the new bone map feature of the plugin.
    For a "HowTo" please have a look into the Mannequin BP function "Init Bone Map".
    - Update to Plugin v0.2.0

    v0.3.0:
    - Source code restructure: C++ code is now placed in a plugin to be more project independend.
    This plugin can be used in any of your projects and there is no need to create a C++ project anymore.
    The binary plugin version 0.1.0 can be downloaded from here:
    https://github.com/Cyx69/TP_ThirdPersonNeuron/releases/download/v0.3.0/Perception_Neuron_Plugin_0.1.0.zip
    - Further comments and minor improvements

    v0.2.0:
    - Playback Controller added.
    Now you can directly play BVH files placed ín your content folder instead connecting to a BVH server.
    See new Blueprint nodes Play and Pause.
    To have an example Neuron character is now connected to a player controller instead to a network controller.
    - Neuron and TPP_Hero are now part of a character blueprint (To have an example for that)
    - Hips bone of TPP_Hero mesh is used to translate complete actor (capsule). Actor rotation is still a problem and is not connected.
    - Add support for standard BVH format
    - Add support for disabled displacement and/or enabled reference bone in motion line
    - Add DNS name support
    - Add Blender source file for Neuron character
    - Further comments and minor improvements

    v0.1.0: Initial version

    Changes Plugin:
    v0.3.4:
    - Remove UE4.16 build warnings

    v0.3.3:
    - Do not resolve HostName if HostName is already a valid IPv4 address

    v0.3.2:
    - Plugin enable Fix for UE4.14

    v0.3.1:
    - Compile Fixes for UE4.13

    v0.3.0:
    - Add better custom skeleton handling. No need to map bone coordinate systems by hand anymore, just map the bone names in anim and event graph.
    - Mark old neuron read functions as deprecated
    - Add BVH player rewind node
    - NeuronRead functions are now able to read the skeleton translation and rotation in component space
    - Compile pre build libraries with UE4.10
    - Further comments and minor improvements

    v0.2.0:
    - It is now possible to initialize Reference skeleton manually and initialization with external BVH reference file is optional.
    - Support all rotation orders and add missing XZY, YZX, ZYX rotations
    - Better support for custom skeletons: Bone coordinate system can now be mapped in blueprint.
    - Fix a synchronization bug when using Neuron format
    - Further comments and minor improvements

    v0.1.0: Initial version (based and included in template v0.3.0)


    Videos:
    Live connection between Perception Axis Player and Unreal Engine 4 (Part 2)
    Live connection between Perception Axis Player and Unreal Engine 4
    Perception Neuron in Unreal Engine 4 Video from galaxyexpress998
    Speech about Perception Neuron and this Plugin at Unreal Meetup #3 Franken (in german) and the used slides (in english)
    Recording Solution (not part of free version, send me a PM if you are interested)

    Applications/Games/Demos based on this Plugin:
    Blood Mine VR Demo (Video)

    Quick Installation Notes Template:
    - Download the template from above link
    - Unpack the ZIP archive to your UE4 template folder, for e.g. "C:\Program Files\Unreal Engine\4.9\Templates\TP_ThirdPersonNeuron".
    Make sure that the project files are placed and visible under "...\Templates\TP_ThirdPersonNeuron" after unzip.
    - Launch Unreal Editor and click on "New project".
    - Under C++ you should now see a template called "TP Neuron".
    - Create a new project with the above template selected.

    Quick Installation Notes Plugin:
    (Not needed if you use the template)
    - Download the plugin under: https://github.com/Cyx69/TP_ThirdPersonNeuron/releases
    - Copy the ZIP File to your closed Unreal project and unpack it directly in the main folder
    - You should see now a new directory "Plugins\PerceptionNeuron" and under "Content" two new folders "Neuron" and "NeuronPlugin"
    - Launch Unreal Editor
    - Under "Edit/Plugins/Project" you should see now a new plugin
    - Add the Neuron Blueprint to your level (See Neuron/Character/Mesh)
    - Add a Network or Player Controller Blueprint to your level (See NeuronPlugin/Blueprints)
    - Connect Neuron Blueprint and Controller with each other in your Level Blueprint (See also "Blueprint Installation Notes.pdf")
    - If you use a Network Controller change the Neuron Animation Blueprint accordingly

    Requirements:
    - Unreal Engine 4.10 or higher
    - MS Visual Studio 2015 (for template)
    - Axis Neuron SW Player (Free version is available here: https://neuronmocap.com/downloads)
    or alternatively you can use my free minimal BVH send server here: https://github.com/Cyx69/bvhsend/releases

    Configuration:
    To connect from the template/plugin to the Axis Neuron SW following settings should be done in the Axis Neuron SW:
    - Open File|Settings dialog
    - Activate under Broadcasting BVH and choose port 7001
    - Configure under Output Format:
    * Frequency reducing: 1
    * Block type: String
    * Rotation: YXZ
    * Activate 'Displacement'
    * Deactivate 'Reference'
    Per default the template/plugin connects to 'localhost' (127.0.0.1) and to port 7001.
    You can change these settings in the Controller Blueprint if you want to use a separate PC or other ports.
    And you can add further controllers if you want to retrieve motion data from additional suits or BVH server.

    Donate:
    If you find the template/plugin or parts of it useful, please support me and the further development and make a donation here:
    Donate

    Best regards
    Heiko Fink aka Cyx
    Last edited by Cyxx; 06-16-2017 at 01:05 PM. Reason: New release for UE4.15

  2. #2
    0
    Promoter
    Join Date
    Apr 2015
    Posts
    239
    This is an amazing gift to the community and exactly what I was waiting for in order to preview a motion in the engine. Can't wait to try it out. Thanks Cyx!

    From a workflow perspective though how do you see this fitting into an animation pipeline? In its current form is its purpose simply to preview motion or do you see potential in it to grab a raw sequence directly in UE4 and start using it in Persona? (I would be blown away if we could already do this). As I understand it, and I could be wrong, you'd still need to save out the motion from Axis Neuron, do some clean up (which might include skeletal translation as I think all the bones have different names in Axis Neuron), and import the fbx into UE4. Does that sound right?

    I'm still getting my head around the Axis Neuron -> UE4 workflow and would love to hear more views on it. Admittedly all I'm able to do at the moment is get some nice mocap in Axis Neuron and save it as fbx. I haven't the foggiest how to apply it properly to the UE4 mannequin. When I tried to import the fbx onto the mannequin control rig it was a mess. I did manage to make it less messy by renaming the skeleton from the fbx to match the mannequin but figured there must be a better way/process.
    Last edited by Naveed; 09-22-2015 at 09:31 AM.

  3. #3
    0
    @Naveed:
    My focus so far was to get a live connection between UE4 and Axis Player and use it later also in a VR environment.
    But I think it should be possible to store the values of the rotation and translation array for e.g. in a file and playback it later.
    Some small enhancements in the Event Graph of the Animation BP should do it.

    But may be there is a better way for your purpose if you have some Blender KnowHow:
    - Export your animation for instance in BVH format and import it in Blender. Apply a simple mesh to it, export it as FBX and import this to Unreal.
    - Then retarget your animation to the Mannquin skeleton like it is described here: https://docs.unrealengine.com/latest/INT/Engine/Animation/RetargetingDifferentSkeletons/index.html

  4. #4
    0
    Well Cyxx,

    I have to say that I'm super impressed by this

    I got everything up and running in short order and was able to test some basic data that I captured earlier this week with one of the suites we have.
    Other than some finger movement oddities it works very well. I'll get around to doing some live capture tests later tomorrow but first thing tomorrow I'll be sending you my donation. Great job ! You'll certainly be hearing more from me shortly

    MrNexy

  5. #5
    0
    Less then a month and I'll be receiving my kits...can't wait to test it out with UE4 livestreaming!!!

    Thanks again Heiko, this is great!

    Guys don't forget to donate, this is a marvelous gift

  6. #6
    0
    It sounds awesome! As soon as I've got my kits I will test and donate it.

    It's such a great thing to test your animations directly in your game environment.

    Game Design - Photogrammetry - Programming
    Tutorials · Twitter · Twitch


  7. #7
    0
    Thanks, will try it.

  8. #8
    0
    Heiko, this is awesome, thanks for sharing this. I've tested it with recordings and live streaming, works fantastically! Donation done, too

  9. #9
    0
    Samaritan
    Join Date
    Mar 2014
    Posts
    139
    Hi. can any of you guys show some videos of the perception neuron working? or give a short review? I am interested in this system but it is so new that there are not many reviews out there... how is the latency on the "live" set up... and do you have to do a lot of cleaning after you record something? Thanks!

  10. #10
    0
    catalejo, you can download the neuron software for free here: https://neuronmocap.com/downloads
    it comes with several recordings that you can play back and test this template with.

  11. #11
    0
    Samaritan
    Join Date
    Mar 2014
    Posts
    139
    Thanks btengelh, i will do so, but still a review or video samples from a user could be really useful for evaluating this system

  12. #12
    0
    well, it works like in the videos noitom posted
    you just have to stay away from magnetic sources. e.g. I can't use it at my desk, because there is too much interference from whatever (a lot of drift).
    in the next room, the PN works flawlessly

  13. #13
    0
    Hey there Cyxx !

    I'm having a little difficulty getting my custom mesh to replicate the movement of the data stream. I'm using Blender btw
    I followed the steps below , please do let me know if I'm on the right track and if I'm missing any steps that you may have followed yourself.

    1- Export Neuron Mesh from Template as FBX.
    2- Import into blender and replace Neuron Mesh with Custom Mesh, make sure that it's rigged.
    3- Export Custom mesh as FBX (I have a feeling that my settings may be off)
    4- Import Custom Mesh FBX into UE4, Retarget Nueron Mesh's AnimBP to Custom Mesh.
    5- Go through New AnimBP resulting from Retarget, make sure that all the Transform(Modify)Bone nodes match my current skeleton's bones.
    6- Drop Skeletal Mesh into scene,assign the new AnimBP to it. Then open the Level Blueprint and set the reference in there.

    Thus far it works with the exception that the movement is wildly off. The character is often upside down with arms and legs backwards etc, moving but not in the right way.
    I would be most grateful if you could share the blender workflow you followed for preparing your custom mesh, what FBX settings you used for export and possibly what version of blender you used.
    Thanks in advance !

    Best Regards
    MrNexy

  14. #14
    0
    Hi MrNexy

    you don't need to export the Neuron Mesh, I have published the Blender source file for free here:
    https://github.com/Cyx69/TP_ThirdPersonNeuron/blob/master/Blender.zip

    To add your mesh you can follow these steps in Blender:
    - Delete the old Neuron (Makehuman) mesh
    - Add your own mesh and tweak the mesh in that way that the armature fits into the mesh
    - Parent the mesh to the existing armature with empty or automated weighted groups (Whatever you like to work with)
    - Check all groups if they are present and correctly weighted
    - Use my Blender export settings that I described in der Blender.zip
    - Import the FBX to Unreal and choose the Neuron skeleton or create a new one if you prefer that

    You could also manipulate the armature in Blender if you want and import it as a new skeleton in Unreal.
    For instance the length of a bone, but then you have to switch the transform settings to 'Ignore' in the Animation graph for that bone.
    But do not change the rotation of a bone and make sure that all Roll values for the bones are set to zero.
    And if you use a new skeleton you may be have to check all 'Transform (Modify) Bone' nodes in the Anim graph if the 'Bone to Modify' setting is still correct.

    Best regards
    Heiko

  15. #15
    0
    Thanks Heiko !

    Thank does the trick, thanks for going the extra mile good sir.

    -MrNexy

  16. #16
    0

    v0.2.0 released

    Changes since last version:

    - Playback Controller added.
    Now you can directly play BVH files placed ín your content folder instead connecting to a BVH server.
    See new Blueprint nodes Play and Pause.
    To have an example Neuron character is now connected to a player controller instead to a network controller.
    In other words: You can now play your recorded Neuron BVH animation files in Unreal.
    And you can also play animations created by other tools if the skeleton matches the reference skeleton (see axis.bvh file).
    - Neuron and TPP_Hero are now part of a character blueprint (To have an example for that)
    - Hips bone of TPP_Hero mesh is used to translate complete actor (capsule). Actor rotation is still a problem and is not connected.
    - Add support for standard BVH format
    - Add support for disabled displacement and/or enabled reference bone in motion line
    - Add DNS name support
    - Add Blender source file for Neuron character
    - Further comments and minor improvements

    Best wishes and have fun
    Heiko

  17. #17
    0
    Amazing...I have to wait until the end of October for the suits but this plugin is absolutely astonishing!

    Thanks again Heiko

    PS: Guys donate!

  18. #18
    0
    Yea the news for BATCH E sucks but hey what can we do...
    I'm fully supporting this project and will definitely contribute by donation.
    Hopefully I'll have my suit by mid Nov.
    Man it's crazy the time difference for delivering only a few hundred suits....
    Almost an entire year to place about 1500 orders...

    anyway cheers
    and congrats

  19. #19
    0
    Promoter
    Join Date
    Apr 2015
    Posts
    239
    Quote Originally Posted by Cyxx View Post
    In other words: You can now play your recorded Neuron BVH animation files in Unreal.
    Outstanding work! Would I be correct in saying that with this update UE4 can handle BVH file types as well as it can FBX?
    Cannot wait to try this when my neurons arrive.

  20. #20
    0
    @Naveed:
    With this plugin it is now possible to play BVH motion capture files directly in UE4 that are stored somewhere in your content folder.
    They are not directly imported like FBX files.
    And you will need a skeleton that is supported by the plugin.
    Currently supported are: Perception Neuron, TPP_Hero or Mannequin skeletons.
    For other skeletons types the plugin must be patched manually at the moment.

    If there is a need for other skeletons, please send me a PN or mail.
    Last edited by Cyxx; 10-06-2015 at 01:49 PM.

  21. #21
    0

    v0.3.0 released

    Changes since last version:

    - Source code restructure: C++ code is now placed in a plugin to be more project independend.
    This plugin can be used in any of your projects and there is no need to create a C++ project anymore.
    The binary plugin version 0.1.0 can be downloaded from here:
    https://github.com/Cyx69/TP_ThirdPersonNeuron/releases/download/v0.3.0/Perception_Neuron_Plugin_0.1.0.zip
    - Further comments and minor improvements

    A special thanks to Benedikt Engelhard for sponsoring the plugin work !

  22. #22
    0
    Howdy,

    I accidentally stumbled across your thread on the plugin you've made for the Perception Neuron. I have yet to buy one, and they don't seem to be forthcoming with information. I'm would definitely like to buy one, but I am 201cm, and is true with the rest of most things made in the world, I'm simply too big. Do you think this would be the case with the Neuron rig?

    As a side note, I am interested in using it more for cinematic animation which will be rendered with Unreal. Do you think this would be a good addition to my pipeline?

    Someone post they might get their rig in mid November!! That was posted months ago, and the Perception website says shipping in October. Anyone know what the reality is?

    Cheers,

    Sterling

  23. #23
    0
    Quote Originally Posted by SterlingY View Post
    Howdy,

    I accidentally stumbled across your thread on the plugin you've made for the Perception Neuron. I have yet to buy one, and they don't seem to be forthcoming with information. I'm would definitely like to buy one, but I am 201cm, and is true with the rest of most things made in the world, I'm simply too big. Do you think this would be the case with the Neuron rig?
    well, you have sonsers on your limbs, and the sensors are connected with cables. though, the cables should be long enough (they have this whirly stretch thingies - anyone knows what it's called?). so you should be fine, IMHO.


    Quote Originally Posted by SterlingY View Post
    As a side note, I am interested in using it more for cinematic animation which will be rendered with Unreal. Do you think this would be a good addition to my pipeline?
    absolutely.

    Quote Originally Posted by SterlingY View Post
    Someone post they might get their rig in mid November!! That was posted months ago, and the Perception website says shipping in October. Anyone know what the reality is?
    I was one of the early backers and got mine in mid-august. unfortunately, I didn't have much time to tinker with it extensively. but it works very well. you can actually test everything: just get there free "neuron axis player" and use it with this plugin. they provide example data with the player.

  24. #24
    0
    I'm in batch E and currently no one received any notification/update, but they reply to one of my email saying that the suits will be shipped in 2-3 weeks, means that mine "should" be on the way now or next week...roughly 10 days for the shipping from Hong Kong, so in theory the first week of novemeber the suits should arrive...hopefully...

  25. #25
    0
    Thanks guys. I think I'm going to go all in! I actually just received an email from a sales rep. He told me that if I ordered on today, I'd get it in November. Unfortunately, he also told me they are spec'd for people up to 6'5", and that any taller would be a stretch. While I do want to wear it, I'd probably work with other actors most of the time.

    On their website, they claim the base software can track two rigs at the same time while the pro version can track more and has other features. Any word on this?

    FYI, Reallusion, the makers of iClone, Character Creator, and 3DXchange will have a plugin for Perception shortly. I've used their products in my pipeline with Unreal and it is seamless and extremely painless. One of the guys at Reallusion also told me that they should have a full face tracking system working by January.

    -Sterling
    Last edited by SterlingY; 10-16-2015 at 12:09 AM.

  26. #26
    0
    Promoter
    Join Date
    Apr 2015
    Posts
    239
    Quote Originally Posted by SterlingY View Post
    Thanks guys. I think I'm going to go all in! I actually just received an email from a sales rep. He told me that if I ordered on today, I'd get it in November.
    I honestly wouldn't get your hopes up for a November delivery. I say this after reading the comments on their kickstarter and the fact that I made an order in July and still haven't received any communication regarding it. If you're super patient and in no rush then I'd say go for it as it is a fantastic piece of kit that you're going to have to wait for.

    Also worth adding if you hit up their forums you'll learn that UE4 integration is not a priority for them and the loose time frame given is 'within a year'. It is for this reason @Cyxx is a legend for putting this integration together for us.

  27. #27
    0
    Hey Heiko!

    Quick question: will the plugin work even on a packaged project? or it will work only inside the editor/simulate mode?
    I guess that you still need ( of course ) to have Axis player running, but it would be interesting to know if, for example, I want to package my game for and android phone but still be able to send the mocap data to the game via wifi.

    PS: Mid-End November I will finally be able to get the suits!!!!!! FINALLYYYYYY!!!!!

  28. #28
    0
    Hi Nicolas
    the plugin works also in a packaged project.
    But so far I tested packaging only for Win64 and currently I ship with the binary plugin only the Win64 libraries.
    I don't know if the plugin works on Android at all.
    Otherwise the source code itself should be OS independent.

    Best regards
    Heiko

  29. #29
    0
    Thanks for the reply Heiko.
    I guess that the best solution would be to "convert" it or be able to stream the mocap data inside the packaged project on Android over wifi...will try that

  30. #30
    0
    Very interesting stuff; I'll sub to this thread to learn

  31. #31
    0
    Since the suits will arrive shortly I made this useful script

    PN animation to Maya Retargeting

  32. #32
    0

    v0.4.0 released

    Changes since last version:

    Template v0.4.0 changes:
    - Network and player controller actors now support manual configuration of reference skeleton instead using external BVH reference file.
    - To have an example how to add custom skeletons, the Mannequin now uses the new bone map feature of the plugin.
    For a "HowTo" please have a look into the Mannequin BP function "Init Bone Map".
    - Update to Plugin v0.2.0

    Plugin v0.2.0 changes:
    - It is now possible to initialize Reference skeleton manually and initialization with external BVH reference file is optional.
    - Support all rotation orders and add missing XZY, YZX, ZYX rotations
    - Better support for custom skeletons: Bone coordinate system can now be mapped in blueprint.
    - Fix a synchronization bug when using Neuron format
    - Further comments and minor improvements

  33. #33
    0
    Promoter
    Join Date
    Apr 2015
    Posts
    239
    Still waiting for my neurons but there is some amazing work going on here. Cannot wait to get stuck in!

  34. #34
    0
    Dude, congrats for the Unreal Grants!!!
    You really deserve them

  35. #35
    0
    Promoter
    Join Date
    Apr 2015
    Posts
    239
    Quote Originally Posted by Nicolas3D View Post
    Dude, congrats for the Unreal Grants!!!
    You really deserve them
    https://www.unrealengine.com/blog/epic-games-awards-150000-in-unreal-dev-grants

    Fantastic news! Really excited to see what you're able to do in future. Must be a great feeling to know that your efforts have been rewarded by Epic. Extremely happy for you.

  36. #36
    0
    This is incredible news. I had been told that my rig would arrive within the next few weeks. Yesterday I received an email that it won't be here until the first quarter of next year. There are 2553 people in front of me. The Neuron website says the same thing today as it did when I ordered mine, which is pre-orders and new orders begin shipping in November. It is extremely misleading. In a long back and forth email exchange, the final explanation was that they are in the middles of switching shipping agents. Added he noted that the Chinese don't do a very good job at translations.

    I only mention the above so you don't do what I did and get your hopes up or (in my case) plan some pipeline and delivery of a project, based on using the Perception neuron. If anyone has ordered a Neuron rig within the last few months and has it in their hands, I'd love to know about it.

    All of that said, I am looking forward to my mocap rig, and can't wait to be able to use the tools provide here.

    Cheers,

    Sterling
    Run with the flock, end up a lamb chop!

    Unreal 4 in 30 Days - An Unreal Experience: https://youtu.be/DV5l7d4b9Cs

  37. #37
    0
    @All: Thank you for the congratulations and compliments. I am still very excited and totally overwhelmed!
    And of course I will continue developing this plugin/template further.
    If you have enhancement wishes just post it here on the forum and I will see what I can do.

  38. #38
    0
    What I would like to see is the ability to send data from the suit without the Axis player...I know that the guys from PN said that they're going to "unlock" that features in the future, but at this point I would love to be able to send the data to UE4/Unity without the software...also because I think that it'll be quicker to send the data, meaning less lag

    I strongly suggest you, since I remember Tobias asked you to contact him about the plugin, to directly contact them and work on the "unlock" feature to be available in short time.

    Thanks again for the plugin

  39. #39
    0
    Howdy,

    I finally received my Neuron rig. I'm really excited to try out your plugin, but when I put it in my project folder, I am told that it may have to be recompiled, and that's as far as I get. I've tried it with 4.9.2. Do I need an older version of UE?

    -Sterling
    Run with the flock, end up a lamb chop!

    Unreal 4 in 30 Days - An Unreal Experience: https://youtu.be/DV5l7d4b9Cs

  40. #40
    0
    I've been having the same problem.

Page 1 of 4 123 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •