Perception Neuron Template

Hi MoCap Interests

this is the discussion thread of my unofficial “Third Person C++ Perception Neuron” template and plugin which I have published for free under the MIT license on Github.
Perception Neuron ™ is a motion capture suit from Noitom financed by a Kickstarter campaign in 2014.
Further details about the suit here: https://neuronmocap.com
In the meantime also an official plugin version was released => Discussion thread.

Please use this thread to discuss the technical details about this unofficial template and plugin.

Download:
Github Repository
Latest template version: v0.5.4
Latest plugin version:
v0.3.0 (UE4.10)
v0.3.0 (UE4.11)
v0.3.0 (UE4.12)
v0.3.1 (UE4.13)
v0.3.2 (UE4.14)
v0.3.3 (UE4.15)
v0.3.4 (UE4.16)
v0.3.4 (UE4.17)
v0.3.4 (UE4.18)

Features Template/Plugin:

  • Live connection over TCP to Axis Neuon player or any other BVH server
  • Playback of locally stored BVH files with stop, pause and rewind
  • Support for multiple instances and different sources (can be shared)
  • Easy to use custom skeleton support with T-Pose, A-Pose or Pose with negative Y axis
    (Just map bone names in event and anim graph. Since plugin v0.3.0 bone coordinate systems are mapped automatically)
  • No dependency to third party libraries and open source solution (MIT license)
  • Remove pelvis drift
  • In-Game and/or In-Editor animation recording (not part of free version, send me a PM if you are interested)

Changes Template:
v0.5.4:

  • Remove UE4.16 build warnings

v0.5.3:

  • Update to Plugin v0.3.3

v0.5.2:

  • Plugin enable Fix for UE4.14

v0.5.1:

  • Compile Fixes for UE4.13

v0.5.0:

  • Cleanup of all Animation BP and switch to new NeuronReadArray function which improves the support of custom skeletons.
  • Remove old custom bone mapping functions (are now deprecated)
  • Add function in Animation BP to remove pelvis drift on ground
  • Change project to UE 4.10
  • Update to Plugin v0.3.0

v0.4.0:

  • Network and player controller actors now support manual configuration of reference skeleton instead using external BVH reference file.
  • To have an example how to add custom skeletons, the Mannequin now uses the new bone map feature of the plugin.
    For a “HowTo” please have a look into the Mannequin BP function “Init Bone Map”.
  • Update to Plugin v0.2.0

v0.3.0:

  • Source code restructure: C++ code is now placed in a plugin to be more project independend.
    This plugin can be used in any of your projects and there is no need to create a C++ project anymore.
    The binary plugin version 0.1.0 can be downloaded from here:
    https://github.com/Cyx69/TP_ThirdPer…ugin_0.1.0.zip
  • Further comments and minor improvements

v0.2.0:

  • Playback Controller added.
    Now you can directly play BVH files placed ín your content folder instead connecting to a BVH server.
    See new Blueprint nodes Play and Pause.
    To have an example Neuron character is now connected to a player controller instead to a network controller.
  • Neuron and TPP_Hero are now part of a character blueprint (To have an example for that)
  • Hips bone of TPP_Hero mesh is used to translate complete actor (capsule). Actor rotation is still a problem and is not connected.
  • Add support for standard BVH format
  • Add support for disabled displacement and/or enabled reference bone in motion line
  • Add DNS name support
  • Add Blender source file for Neuron character
  • Further comments and minor improvements

v0.1.0: Initial version

Changes Plugin:
v0.3.4:

  • Remove UE4.16 build warnings

v0.3.3:

  • Do not resolve HostName if HostName is already a valid IPv4 address

v0.3.2:

  • Plugin enable Fix for UE4.14

v0.3.1:

  • Compile Fixes for UE4.13

v0.3.0:

  • Add better custom skeleton handling. No need to map bone coordinate systems by hand anymore, just map the bone names in anim and event graph.
  • Mark old neuron read functions as deprecated
  • Add BVH player rewind node
  • NeuronRead functions are now able to read the skeleton translation and rotation in component space
  • Compile pre build libraries with UE4.10
  • Further comments and minor improvements

v0.2.0:

  • It is now possible to initialize Reference skeleton manually and initialization with external BVH reference file is optional.
  • Support all rotation orders and add missing XZY, YZX, ZYX rotations
  • Better support for custom skeletons: Bone coordinate system can now be mapped in blueprint.
  • Fix a synchronization bug when using Neuron format
  • Further comments and minor improvements

v0.1.0: Initial version (based and included in template v0.3.0)

Videos:
Live connection between Perception Axis Player and Unreal Engine 4 (Part 2)
Live connection between Perception Axis Player and Unreal Engine 4
Perception Neuron in Unreal Engine 4 Video from galaxyexpress998
Speech about Perception Neuron and this Plugin at Unreal Meetup #3 Franken (in german) and the used slides (in english)
Recording Solution (not part of free version, send me a PM if you are interested)

Applications/Games/Demos based on this Plugin:
Blood Mine VR Demo (Video)

Quick Installation Notes Template:

  • Download the template from above link
  • Unpack the ZIP archive to your UE4 template folder, for e.g. “C:\Program Files\Unreal Engine\4.9\Templates\TP_ThirdPersonNeuron”.
    Make sure that the project files are placed and visible under “…\Templates\TP_ThirdPersonNeuron” after unzip.
  • Launch Unreal Editor and click on “New project”.
  • Under C++ you should now see a template called “TP Neuron”.
  • Create a new project with the above template selected.

Quick Installation Notes Plugin:
(Not needed if you use the template)

  • Download the plugin under: Releases · Cyx69/TP_ThirdPersonNeuron · GitHub
  • Copy the ZIP File to your closed Unreal project and unpack it directly in the main folder
  • You should see now a new directory “Plugins\PerceptionNeuron” and under “Content” two new folders “Neuron” and “NeuronPlugin”
  • Launch Unreal Editor
  • Under “Edit/Plugins/Project” you should see now a new plugin
  • Add the Neuron Blueprint to your level (See Neuron/Character/Mesh)
  • Add a Network or Player Controller Blueprint to your level (See NeuronPlugin/Blueprints)
  • Connect Neuron Blueprint and Controller with each other in your Level Blueprint (See also “Blueprint Installation Notes.pdf”)
  • If you use a Network Controller change the Neuron Animation Blueprint accordingly

Requirements:

Configuration:
To connect from the template/plugin to the Axis Neuron SW following settings should be done in the Axis Neuron SW:

  • Open File|Settings dialog
  • Activate under Broadcasting BVH and choose port 7001
  • Configure under Output Format:
  • Frequency reducing: 1
  • Block type: String
  • Rotation: YXZ
  • Activate ‘Displacement’
  • Deactivate ‘Reference’
    Per default the template/plugin connects to ‘localhost’ (127.0.0.1) and to port 7001.
    You can change these settings in the Controller Blueprint if you want to use a separate PC or other ports.
    And you can add further controllers if you want to retrieve motion data from additional suits or BVH server.

Donate:
If you find the template/plugin or parts of it useful, please support me and the further development and make a donation here:
Donate

Best regards
Heiko Fink aka Cyx

This is an amazing gift to the community and exactly what I was waiting for in order to preview a motion in the engine. Can’t wait to try it out. Thanks Cyx!

From a workflow perspective though how do you see this fitting into an animation pipeline? In its current form is its purpose simply to preview motion or do you see potential in it to grab a raw sequence directly in UE4 and start using it in Persona? (I would be blown away if we could already do this). As I understand it, and I could be wrong, you’d still need to save out the motion from Axis Neuron, do some clean up (which might include skeletal translation as I think all the bones have different names in Axis Neuron), and import the fbx into UE4. Does that sound right?

I’m still getting my head around the Axis Neuron -> UE4 workflow and would love to hear more views on it. Admittedly all I’m able to do at the moment is get some nice mocap in Axis Neuron and save it as fbx. I haven’t the foggiest how to apply it properly to the UE4 mannequin. When I tried to import the fbx onto the mannequin control rig it was a mess. I did manage to make it less messy by renaming the skeleton from the fbx to match the mannequin but figured there must be a better way/process.

@Naveed:
My focus so far was to get a live connection between UE4 and Axis Player and use it later also in a VR environment.
But I think it should be possible to store the values of the rotation and translation array for e.g. in a file and playback it later.
Some small enhancements in the Event Graph of the Animation BP should do it.

But may be there is a better way for your purpose if you have some Blender KnowHow:

Well Cyxx,

I have to say that I’m super impressed by this :slight_smile:

I got everything up and running in short order and was able to test some basic data that I captured earlier this week with one of the suites we have.
Other than some finger movement oddities it works very well. I’ll get around to doing some live capture tests later tomorrow but first thing tomorrow I’ll be sending you my donation. Great job ! You’ll certainly be hearing more from me shortly :slight_smile:

MrNexy

Less then a month and I’ll be receiving my kits…can’t wait to test it out with UE4 livestreaming!!!

Thanks again Heiko, this is great!

Guys don’t forget to donate, this is a marvelous gift :slight_smile:

It sounds awesome! As soon as I’ve got my kits I will test and donate it.

It’s such a great thing to test your animations directly in your game environment.

Thanks, will try it. :slight_smile:

Heiko, this is awesome, thanks for sharing this. I’ve tested it with recordings and live streaming, works fantastically! Donation done, too :slight_smile:

Hi. can any of you guys show some videos of the perception neuron working? or give a short review? I am interested in this system but it is so new that there are not many reviews out there… how is the latency on the “live” set up… and do you have to do a lot of cleaning after you record something? Thanks!

catalejo, you can download the neuron software for free here: Downloads | Perception Neuron Motion Capture
it comes with several recordings that you can play back and test this template with.

Thanks btengelh, i will do so, but still a review or video samples from a user could be really useful for evaluating this system :slight_smile:

well, it works like in the videos noitom posted :slight_smile:
you just have to stay away from magnetic sources. e.g. I can’t use it at my desk, because there is too much interference from whatever (a lot of drift).
in the next room, the PN works flawlessly :slight_smile:

Hey there Cyxx !

I’m having a little difficulty getting my custom mesh to replicate the movement of the data stream. I’m using Blender btw :slight_smile:
I followed the steps below , please do let me know if I’m on the right track and if I’m missing any steps that you may have followed yourself.

1- Export Neuron Mesh from Template as FBX.
2- Import into blender and replace Neuron Mesh with Custom Mesh, make sure that it’s rigged.
3- Export Custom mesh as FBX (I have a feeling that my settings may be off)
4- Import Custom Mesh FBX into UE4, Retarget Nueron Mesh’s AnimBP to Custom Mesh.
5- Go through New AnimBP resulting from Retarget, make sure that all the Transform(Modify)Bone nodes match my current skeleton’s bones.
6- Drop Skeletal Mesh into scene,assign the new AnimBP to it. Then open the Level Blueprint and set the reference in there.

Thus far it works with the exception that the movement is wildly off. The character is often upside down with arms and legs backwards etc, moving but not in the right way.
I would be most grateful if you could share the blender workflow you followed for preparing your custom mesh, what FBX settings you used for export and possibly what version of blender you used.
Thanks in advance !

Best Regards
MrNexy

Hi MrNexy

you don’t need to export the Neuron Mesh, I have published the Blender source file for free here:

To add your mesh you can follow these steps in Blender:

  • Delete the old Neuron (Makehuman) mesh
  • Add your own mesh and tweak the mesh in that way that the armature fits into the mesh
  • Parent the mesh to the existing armature with empty or automated weighted groups (Whatever you like to work with)
  • Check all groups if they are present and correctly weighted
  • Use my Blender export settings that I described in der Blender.zip
  • Import the FBX to Unreal and choose the Neuron skeleton or create a new one if you prefer that

You could also manipulate the armature in Blender if you want and import it as a new skeleton in Unreal.
For instance the length of a bone, but then you have to switch the transform settings to ‘Ignore’ in the Animation graph for that bone.
But do not change the rotation of a bone and make sure that all Roll values for the bones are set to zero.
And if you use a new skeleton you may be have to check all ‘Transform (Modify) Bone’ nodes in the Anim graph if the ‘Bone to Modify’ setting is still correct.

Best regards
Heiko

Thanks Heiko !

Thank does the trick, thanks for going the extra mile good sir.

-MrNexy

v0.2.0 released

Changes since last version:

  • Playback Controller added.
    Now you can directly play BVH files placed ín your content folder instead connecting to a BVH server.
    See new Blueprint nodes Play and Pause.
    To have an example Neuron character is now connected to a player controller instead to a network controller.
    **In other words: You can now play your recorded Neuron BVH animation files in Unreal. **
    And you can also play animations created by other tools if the skeleton matches the reference skeleton (see axis.bvh file).
  • Neuron and TPP_Hero are now part of a character blueprint (To have an example for that)
  • Hips bone of TPP_Hero mesh is used to translate complete actor (capsule). Actor rotation is still a problem and is not connected.
  • Add support for standard BVH format
  • Add support for disabled displacement and/or enabled reference bone in motion line
  • Add DNS name support
  • Add Blender source file for Neuron character
  • Further comments and minor improvements

Best wishes and have fun
Heiko

Amazing…I have to wait until the end of October for the suits but this plugin is absolutely astonishing!

Thanks again Heiko :slight_smile:

PS: Guys donate!

Yea the news for BATCH E sucks but hey what can we do… :cool:
I’m fully supporting this project and will definitely contribute by donation.
Hopefully I’ll have my suit by mid Nov.
Man it’s crazy the time difference for delivering only a few hundred suits…
Almost an entire year to place about 1500 orders…
:confused:
anyway cheers
and congrats

Outstanding work! Would I be correct in saying that with this update UE4 can handle BVH file types as well as it can FBX?
Cannot wait to try this when my neurons arrive.

@Naveed:
With this plugin it is now possible to play BVH motion capture files directly in UE4 that are stored somewhere in your content folder.
They are not directly imported like FBX files.
And you will need a skeleton that is supported by the plugin.
Currently supported are: Perception Neuron, TPP_Hero or Mannequin skeletons.
For other skeletons types the plugin must be patched manually at the moment.

If there is a need for other skeletons, please send me a PN or mail.