Full-body motion capture

Development Programming & Scripting Blueprint
, , , ,
1

Hello,
me and my Team are currently working on a bachelor project where we want to try full body capture using Optitrack so we can move a skeleton in Unreal. We have a full body suit with markers on it and several cameras to record our movement. We get the position and the orientation of the markers in our blueprint. Now we tried to move the bones of the ue4 mannequin and it looks like this:

As you guys can see, our skeleton does not stay connected (except for the arm) and we do not know why.
We were wondering if any of you ever worked on motion capturing in unreal
and can give us some tips how on to do it in a better way. Or how to fix our current issue with the Skelton (see prev vid). Currently we are using the Transform (modify) Bone function to move bones. Is there a better way to do this so that for example connected bones change their rotation when we move a bone?
Our blueprint and our Skeleton are in the attached Folder.

2

We also made a forum post asking for some general advise [here][1]

[1]:

3

Hi,

Im writing my Thesis too and have a similar problem. Your problem looks like, you dont have connected your upperLeg right. So your knees are moving the right way, but your upperLeg dont. Have you already fixed this problem?

Can you help me with figuring out, how to send the data from motive to unreal 4? Now I just have the newest Plugin from optitrack to unreal, but it just supports rigid bodys, and no full skeletons.

thanks and best

4

hi michael,

I’m one of the guys who worked with wolff. For our project we told motive to send the skeleton as rigid bodys and then did some modifications to the plugin. Some new structs and also other stuff. I’m not really what exactly was changed since that was some other guys job. At the end we got the 21 bones Motive sends you in an array. The IDs keep changing, but the order in which they are send is always the same. The location data we got was pretty good, but the orientation was sometimes of by 90 degrees since the position Motive conciders 0/0/0 is different from what Unreal concinders 0/0/0 for that bone. So we added some offsets to every bone. What you see in the video is a gimbal lock. We avoided it by using Quats instead of Rotators. Only derictly before we apply the data to the Mannequin we go back to using an Rotator. Our project is due on May 31th. I will ask the guy who oversees us if I can upload it on an public git repository, after we are finished.

5

Hi Zartrocs,

happy to see a response!

This is a pretty good way to receive the missing dataparts from motive. I also had a look into the cpp and h files from the optitrack natnet plugin. It would be more than great, if you or the other guys could tello me, what they did, besauce I’m running out of time with my thesis and mocap is kind of a core theme of it. The IKinema Plugin is doing exactly what I need, but they dont give Students a free or cheep license. They want have about 420 pouns a Month.

So, if you could send me any kind of information, it would help me alot and, of course if you and your team allow me to use anything, you would be officially referenced in my thesis. I dont want to steel anything :DD

6

P.S.:
Hab grade gesehen ihr seid aus Bremen?! Bin von der HSD in Düsseldorf. Es wäre echt geil wenn wir in weiteren Kontakt kommen würden. Meine Master Thesis befasst sich mit der Fähigkeit Unreal 4 und Motioncapture als Previsualisierungstool verwenden zu können. Da würde ich gerne ein funktionierendes MoCap, wie in dem Video hier, vorzeigen und Aufnehmen können:

7

-removed- (Edit: 06.08.2019:
Our implementation was very bad even back then, since we refused to use the SDK, and is totally outdated at this point in time. Please don’t use this in any way or form, if you want to keep your sanity. Skeletons should now be supported by the official plugin.)

Das ist unser fertiges Projekt. In der readme sollte eigentlich alles stehen was du über die settings und die Klassen wissen musst. Unser Code liegt in source/P1 und im Plugin Ordner. P1 war unser Teilprojekt, das sich mit OptiTrack befasst hat. Es gab auch noch einige andere Teilprojekte, die sind aber bei dem was ich dir verlinkt habe nicht enthalten. Bei IKinema hatten wir uns auch mal die Testversion geholt und mit unserem verglichen. Ist natürlich einfacher zu bedienen und man kann leicht beliebige Skelette reinladen, aber bei dem Preis ist das ja auch das mindeste.

8

Hi,

vielen Dank für die Antwort, ich guck es mir direkt mal an. Ich schreibe mal in Englisch weiter.

How do you compile/import the Optitrack PLugin? When I copy the Optitrack Folder in my Project-Plugins folder, I got an compiler error. Even when I try to recompile the whole project folder, it throws errors.

Thanks so much for you reply. That gives me hope, that there is way to do it.

Best,

9

We only had to do this once and that was some months ago, so I’m not 100% sure what we did. Have you tried adding the plugin to your projectname.Build.cs? For me the dependencies look like this:

PublicDependencyModuleNames.AddRange(new string[] {
            "Core",
            "CoreUObject",
            "Engine",
            "InputCore",
            "OptitrackNatnet",
            //"Json",
            //"JsonUtilities",
            //"UMG",
	    //"XmlParser"
	    //"HydraPlugin",
	    //"HeadMountedDisplay",
	    //"PhysX",
	    //"APEX",
        });

Maybe try that, then delete the Binaries and Intermediate folder of both your plugin and project and build again. We often had problems with old binaries after we made some bigger changes so deleting them and rebuilding was our go to solution. If that doesn’t work you will have to figure out what your errors actually mean. :smiley:

10

Ok, I got it. I had to add some Header files to the Optitrack files. Now it compiles and I can change the code.

Now I have to test my Plugin and got a huge problem: I think the data comming from Motive isnt complete! The rigid Body data from the several bones are missing! I changed any kind of streaming-setting into Motive, but no bonedata. I am checking it with the Sample program from the NatNet SDK “WinFormTestApp.exe”. This program should show me all available data in the stream, as you can see in the Dokumentation of it: https://optitrack.com/public/documents/natnet-api-user-guide-2.10.0.pdf

At Page 8 you can see exactly the data I need, but I cant pick it up. Do you do something special in Motive to stream the rigid body data of the bones? Which version of Motive and NatNet bzw. optitrack-Plugin do you use?

Very Best,

11

Hey,

I did alotta work yet. The problem was, that the sample file of NatNet didnt catch the data, though they was send. Now all the data is in UE4, and I “just” have to order and apply them :slight_smile:

Thanks a lot for the help!!!

12

“stream rigid bodies”, “stream skeletons” and “skeletons as rigid bodies” need to be true. Everything else false. We are also using Unicast.

Our plugin is based on the newest official release (1.5.0). If our plugin compiles for you, you should be able to use the “Receive Motive” blueprint function. This should give you an array of all the data coming in. Everything we changed in the plugin was to make that function work. Everything that comes after that is done by our RMocapComponent class and not the plugin. The original plugin had some code that got rid of all the skeleton data, but we commented it out so that shouldn’t happen anymore.

We never used the testing program, since it just instantly worked for us. Are you getting no data at all, or is it just the skeleton data that is missing?

13

The order should be like we described in our readme. When applying the data the location data should be ok. The rotation should be wrong but you can just look into our code for the offsets. Keep Quats and gimbal lock in mind while doing so. (see first post)

14

Hi,

everything is working now. Thanks again for your help. When there is a nice Video to show, I will link it up here.

Best,

15

Hey :))

Did you have problems reading the data into blueprint?? After a while, I think about 7 to 25 minutes, my program crashes, because of an Copy Error. Thomething is reading at position 00000000000000000.

I saw, that you directly write into 21 rotators in the c++ file, instead of writing into an array.

Best,

16

A ok thanks.

I now write the data, as you did, into FRotator Variables, and read them later in BP. This works for about 17 Houres :DD

Deos it have a reason, that you doesnt take the “sequence recorder” from UE4, to record the incomming motions?

PS: three weeks to go now for my thesis :expressionless: If your Thesis is online somewhere, I would reference it. Otherwise I could perhaps take the GIthub Page to reference.

Best,

17

no idea. we never had animations longer than 2 minutes.

18

I’m not the one who did the recording part. The guy who did it probably didn’t know about the “sequence recorder”. I just know that his goal was to simulate the incoming Motive data as close as possible.
The guy who owns the repo I linked u to reverted my commits. (no idea why)

-removed- (Edit: 06.08.2019:
Our implementation was very bad even back then, since we refused to use the SDK, and is totally outdated at this point in time. Please don’t use this in any way or form, if you want to keep your sanity. Skeletons should now be supported by the official plugin.)

This is the repo we had to use for our final commit and grading. But I have no idea for how long it will be up.

http://www.robcog.org/

This is the side of the "big"project. For our bachelor project we had to add additinal functionality to it.

19

your youtube result looks great. I would like to get something like this going at my university. as far as I can tell the code for the natnet skeleton plugin is not public in the links above, is that right?

20

Hi Zartrocs and Wolff,

I am also trying to do some research for the same thing. We use Motive 1.10 and regularly stream into MotionBuilder, but we’re trying to do some new stuff in UE4. Would you be able to share the repository again, or perhaps in a private message? It would be greatly appreciated.

I find I’m a little stuck with understanding exactly how Motive is sending that skeleton data as rigid bodies, and then from there how youre reading the data into UE4 blueprints? I looked through the rar files that Wolff included, but am still having a bit of trouble understanding where youre getting, modifying, and converting the Motive data to be read into the blueprint system.

Thanks!!!