Kinect 4 Windows v2.0 Plugin

plangton · January 5, 2015, 12:26pm

can we do background removal with this plugin ?
most preferably with the blueprint section

lion032 · January 5, 2015, 8:05pm

[=plangton;201181]
can we do background removal with this plugin ?
most preferably with the blueprint section
[/]

Initial framework added.

anonymous_user_c074ad13 · January 6, 2015, 12:12am

Lion, you gave us blueprints for the basic bone rotations awhile back, but they do not work as is. What changes need to be made for this blueprint to work? In particular, I am having issues pulling in any data from my Kinect using those blueprints (no signal is sent when I debug).

anonymous_user_c074ad13 · January 7, 2015, 4:00am

Does anyone know how to get the plugin to continuously pull in data from the Kinect? I have got the character seemingly setup correctly when I debug it, however once it is setup I do not seem to be getting any signal from Kinect Body Events.

The important parts of my event graph are given below:

The bone rotations are essentially done the same way as lion’s demo, however I can not seem to get the initial event signal to go to the “Rotate Bones” function.

anonymous_user_ecb5cc49 · January 7, 2015, 5:50pm

[=;202057]
Does anyone know how to get the plugin to continuously pull in data from the Kinect? I have got the character seemingly setup correctly when I debug it, however once it is setup I do not seem to be getting any signal from Kinect Body Events.

The important parts of my event graph are given below:

The bone rotations are essentially done the same way as lion’s demo, however I can not seem to get the initial event signal to go to the “Rotate Bones” function.
[/]

You need to make sure your skeleton is being detected and that the events are getting bound. Those seemed to be my 2 largest issues.

I have been having a hard time getting my skeleton detected if I PIE so I have been choosing the standalone option when testing kinect stuff. Give that a try and make sure the kinect event manager is valid before use. This is helpful for debug. There are some odd order of operations things when working in the editor that cause some sync issues it seems. Like the fact that objects are already constructed before you start play when playing in editor.

Wilson89 · January 7, 2015, 9:50pm

No longer working in 4.6

[=RuBa1987;176760]
The other way you can do this is via an anim instance. Create a new anim instance and in the even graph do something like this:

Then go to the anim graph, create a state machine and all those basics (or you can do it in the root of the anim graph, I just don’t like doing that) and do something like this:

Again, you will need to fiddle around with the rotations and such but this is the general idea (or at least the approach I am taking).
[/]

Hi Lion/RuBa1987,

First off, first post here so wanna say thanks to all the devs on this. Was planning to do this myself but found this plugin before i really got set into things (and have come to understand i really did not realise how much of an undertaking it would have been ), awesome job.

Anyway i was following this method of doing the avatar controlling and am noticing something strange. Since lion posted recently with 4.6 on github i decided to move to it as 4.6 brings alot of now features for the engine itself. For some reason however, using the exact same blueprint from 4.5 to 4.6, it has stopped working. The KinectBodyEvent and NewSkeletonDetectedEvent no longer trigger in 4.6. Are you coming across this issue aswell?

Also, what would you recommend to do for trying to debug things like this. I would normally add a print log between events to see where things are hitting but it seems like Kinect Event Manager isnt being triggered in this case.

Thanks,

anonymous_user_c074ad13 · January 8, 2015, 9:36pm

[=RuBa1987;202344]
You need to make sure your skeleton is being detected and that the events are getting bound. Those seemed to be my 2 largest issues.

I have been having a hard time getting my skeleton detected if I PIE so I have been choosing the standalone option when testing kinect stuff. Give that a try and make sure the kinect event manager is valid before use. This is helpful for debug. There are some odd order of operations things when working in the editor that cause some sync issues it seems. Like the fact that objects are already constructed before you start play when playing in editor.
[/]

It seems to have fixed itself somehow…maybe my Kinect was just in a weird place or something before not sure, thanks for the tip though. Now I’m just working on trying to get the plugin to actually move the character in-game. The Kinect body event signal is passing through fine now, but there is no movement of my character. I’m thinking it’s either an assignment issue (character is not properly being assigned the skeleton), or there is a problem with the rotation blueprints (which there shouldn’t be since it is essentially the same as lion’s demo blueprints).

Let me know if you have any ideas that can save me some time. I will post my progress here in the hopes that people can speed up their own work with this plugin. I will be working on this a bit in the next few days so hopefully I can at least come close to finishing up.

skitten · January 15, 2015, 7:06am

I’m pretty new to Unreal so struggling to understand exactly how this should be wired up. I have the plugin built (in 4.5) and it appears to be working - the Kinect comes on when I run the editor and I see various kinect nodes in the right click menu in the graph editor.

But I don’t understand how to get the events into my graphs - for example in RuBa1987’s example graphs where does the NewSkeletonDetectedEvent come from? I understand a little about custom events and binding but I don’t see how anything from the plugin actually gets into the graph here - can anyone give some clues to someone who’s familiar with DAG based systems (Maya etc) but not Unreal?

Mosel3y · January 15, 2015, 10:09am

@ Do you plan on doing an update for 4.7 when it’s released?

Also, will you making a standalone plugin which doesn’t require custom engine build?

anonymous_user_c074ad13 · January 16, 2015, 5:57am

I have the elbows working and managed to get the Kinect to work in standalone mode. However, when I introduce the hands and wrists the tracking becomes extremely unreliable (everything starts randomly moving around). Unless you came to an epiphany recently Ruba, I think I will also be doing retargeting of the other upper body joints, but instead based off of the elbows. I like the idea of using the elbows more than the wrists since they seem to be more reliable, and they are in the middle of the arm which may make things a bit easier.

ConorBuller · January 21, 2015, 5:39pm

Hey all, so I’ve been using the plugin for a project of mine and I’m not looking to do anything crazy, just pull out some body data and create some simple gestures to be recognised in game. I’ve been using the 1.6 version of this plugin and everything has compiled great and the editor is up and running. The problem I have is setting up the components of the plugin to get the actual data. I can’t seem to figure it out in my own project so I tried importing and converting the last working demo, but it seems to be a bit outdated at this point. I’m getting an error in the KinectAvatar blueprint relating to an unexpected node type “K2Node…” at the ForEach loop. The video streams are working great, but the skeletal data is broken due to seemingly this one problem.

Is there any way that someone who has experience with the plugin could put together a more up to date example project, or even a detailed guide on where to start blueprint wise?

Thanks to Lion and others for making this possible at all, you saved me from having to use Unity or revert to the Kinect 1.

Mosel3y · January 22, 2015, 9:57am

This works for me in 4.6. Basically with what was in the demo. I get warnings as well, but they don’t cause any apparent problems. You don’t need the interp stuff obviously, I was just trying to smooth it a bit.

The BoneNames array should contain the bones of your skeleton and match with the kinect bones.

anonymous_user_f7e02a51 · January 27, 2015, 2:20pm

Hi All,

just getting to this and not finding the github project, is there somewhere that I need to go to get the plugin? I’ve ordered my kinect2 and have the oculus set up.

Thanks

Mosel3y · January 29, 2015, 9:16am

https://github.com/lion03/UE4-K4W-Plugin/tree/KinectV2-4.6

You need to be logged in and have your account linked to your UE4 account.

anonymous_user_65c7a3fe · February 5, 2015, 10:54am

Thank you for the great work.
I have successfully built with these two branches:
KinectV2Plugin-4.5
KinectV2-4.6
How ever the events from all Kinect interfaces (Body, Color, Depth) are not fired.
However if I use “Kinect Event Manager” and then bind it to “RawBodyFrameEvent” or “ColorFrameEvent” it works as expected. Binding it to “KinectBodyEvent” will not work though (the event will not be fired).
If I use KinectV2Plugin-4.5, I see the “Add custom event dispatcher” after I bind the events to “Kinect Event Manager”, but in KinectV2-4.6, the “Add custom event dispatcher” is not there, however if i migrate my asset from 4.5 to 4.6, it will work without problem.
Can you explain a little? I am confused!
Many thanks, !

Darshan_Shankar · February 12, 2015, 6:51pm

[=Mosel3y;212876]
https://github.com/lion03/UE4-K4W-Plugin/tree/KinectV2-4.6

You need to be logged in and have your account linked to your UE4 account.
[/]

I’m still struggling with this. I’ve added my github account to my UE profile, I’m logged into github & UE, but I’m unable to access that link. Is that a private repo?

edit: nevermind, the Github email invitation to join the Epic org was buried my inbox. Accept that and then you can access that repo.

anonymous_user_c074ad13 · February 28, 2015, 9:23pm

[=Mosel3y;209607]
This works for me in 4.6. Basically with what was in the demo. I get warnings as well, but they don’t cause any apparent problems. You don’t need the interp stuff obviously, I was just trying to smooth it a bit.

The BoneNames array should contain the bones of your skeleton and match with the kinect bones.
[/]

I have the implementation working with an implementation similar to yours, however the unit conversions are not correct. Would you be willing to share how you did your unit conversions?

I tried it with lion’s unit conversions which did not work and a few other configurations that I have tried myself with no luck as of yet.

Any help with this would be greatly appreciated.

Also, I have not attempted to do any of the interp stuff but I would be interested in hearing what values you are using for that so I have a good starting point if you are willing to share those numbers.

Mosel3y · March 2, 2015, 10:03am

[=;232155]
I have the implementation working with an implementation similar to yours, however the unit conversions are not correct. Would you be willing to share how you did your unit conversions?

I tried it with lion’s unit conversions which did not work and a few other configurations that I have tried myself with no luck as of yet.

Any help with this would be greatly appreciated.

Also, I have not attempted to do any of the interp stuff but I would be interested in hearing what values you are using for that so I have a good starting point if you are willing to share those numbers.
[/]

I haven’t changed to conversion macro,

I did have to change the values of the “Rotate Vector” for different bones, this was mostly trial and error for me though. And different "Make Rot From ". It probably depends on your skeleton.
For it to work, it’s probably best you have a similar hierarchy to the default skeleton. I don’t understand how it works even, as I said it was just trial and error, but it does work.

My interp speed is 6.

anonymous_user_f9867368 · March 3, 2015, 7:33pm

[=Mosel3y;205997]
@ Do you plan on doing an update for 4.7 when it’s released?
Also, will you making a standalone plugin which doesn’t require custom engine build?
[/]

The 4.6 already works as project plugin. You don’t need a custom engine.

I tried to recompile for 4.7 and I get this :

“LogWindows : error : Windows GetLastError: The operation completed successfully. (0)
1>Error : Failed to generate code for Plugins47Editor - error code: CrashOrAssert (3)
1> UnrealHeaderTool failed for target ‘Plugins47Editor’ (platform: Win64, module info: C:\Users\bzysberg\Documents\Unreal Projects\Plugins47\Intermediate\Build\Win64\Plugins47Editor\Development\UnrealHeaderTool.manifest).
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.MakeFile.Targets(38,5): error MSB3073: The command ““C:\Program Files\Unreal Engine\4.7\Engine\Build\BatchFiles\Build.bat” Plugins47Editor Win64 Development “C:\Users\bzysberg\Documents\Unreal Projects\Plugins47\Plugins47.uproject” -rocket” exited with code -1.”

I looked everywhere and nobody has any cues.

If Microsoft hadn’t drop Kinect that fast, maybe people would have used and Epic would have made an officiel Kinect plugin instead of Leap Motion…

lion032 · March 3, 2015, 7:42pm

I hope to get to it this weekend.