Kinect 4 Windows v2.0 Plugin

[=RuBa1987;187802]
I just tried this link and it’s working fine for me: https://github.com/lion03/UE4-K4W-Plugin/tree/KinectV2Plugin-4.5
[/]

Whoops, turns out I had linked my UE account to the wrong github account. After linking it to the correct one it all works, thanks!

Edit:
I compiled the editor successfully, the plugin is present and enabled in the plugin window, my kinect turns on, and I get no errors aside from the ones listed below, but I get no response from the video streams from the demo editor project. The prebuilt project works fine. Thoughts?


Error /Game/DemoRoom/Materials/M_DemoWallPattern_NoText : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Metal
Error /Game/DemoRoom/Materials/M_DemoWall_Inst : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Rubber
Error /Game/DemoRoom/Materials/M_DemoWall_Inst_3 : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Metal
Error /Game/DemoRoom/Materials/M_DemoRoomTiles_Inst : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Concrete
Error /Game/DemoRoom/Materials/M_DemoWallPattern_Inst : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Metal

Hey all, I’ve got the plugin in 4.5.0, and it’s all working in-game.
However, when I package and try run the packaged game I get the following error:

Has anyone else got this and know of a solution?

EDIT: I do have the SDK installed obviously

[=Mosel3y;188261]
Hey all, I’ve got the plugin in 4.5.0, and it’s all working in-game.
However, when I package and try run the packaged game I get the following error:


Has anyone else got this and know of a solution?

EDIT: I do have the SDK installed obviously
[/]

Have you done this?:
9)Copy Kinect20.VisualGestureBuilder.dll from C:\Program Files\Microsoft SDKs\Kinect\v2.0-PublicPreview1409\Redist\VGB\x64 into Engine\Binaries\Win64

[=nekocat;188298]
Have you done this?:
9)Copy Kinect20.VisualGestureBuilder.dll from C:\Program Files\Microsoft SDKs\Kinect\v2.0-PublicPreview1409\Redist\VGB\x64 into Engine\Binaries\Win64
[/]

Either that or maybe try putting on your path. I haven’t ran into this personally, that’s just a guess.

[=RuBa1987;187207]
I did have some weirdness around creating the event bindings. I’m not sure why this is but, as with most things, there is more than one way to do it. I noticed I had this problem when I made the event to bind to myself. I can’t remember everything exactly right now as I’m not in front of my dev box but when you do the binding I think there are two options (maybe “bind” and “assign”) try both and the one that creates the event binding for you (draws the little red line and make a new event node) is the one you want to use. I have no idea why this happens but it was yelling at me for a similar thing.

Can you describe the issue you are having with the bone rotations?

Also, I wouldn’t recommend using the transforms when you break the kinect body. I know it says that it’s deprecated but use the break the gives you the bones and use the info on the bones. That seems to be only reliable way I can get things to work. For some reason the transform data isn’t the same as the rest making it very hard to work with. I’m not sure if the mappings are different or if something is off with the rotation but it is just too difficult to work with and I haven’t had the time to look into it.

That being said, IMHO, the ideal way to do the animations here would be to set up an IK system rather than an FK system. Meaning the only bone in an arm that you would track would be the hand (or wrist) and retarget the elbow and shoulder based on that. It would be a similar feel to how A.R.T. works when using the hand IK control. Then you could also do it based on location rather than the rotation which I feel is easier to work with in general. This would give you a couple of big wins. The first being that you would be tracking less bones which means less noise on the input. The second is that it’s less data to manage. This isn’t so big but it’s nice to have to manage less data from an API.

The big downside to using the kinect bones over the transforms is that you will need to write your own filter. The one euro filter that is there only works for transforms. This isn’t all that bad and it gives you a bit more control over the filter. I redid the one euro in BP for the kinect bones and I’m getting decent results. I did it a bit differently though. I have one filter per parameter. Meaning if I am using a filter on the shoulder rotation I have a separate filters for yaw, pitch, and roll. This lets me filter each piece of data independently as each piece has its own noise and therefore needs its own filter. This also lets me set the filter params per piece of data. This also makes it extremely tedious but with some reasonable defaults and the option to override or ignore each filter it’s not that bad.

Just some food for thought.
[/]

I have been working in Unreal Engine for about a couple months now, so excuse my lack of knowledge in some areas. I’m still in fairly early stages of my development for this part of a project that I’m working on. I was trying to get the demo working since we are not looking to do any incredibly detailed design (just going to get upper body movement working with no microphone input). The demo seemed to fulfill my requirements for my project, but I can not get the demo to work as is. I tried adding the event manager in the event graph as you did in the blueprint you showed which gave me those errors I mentioned earlier linking the event bindings. I only saw one option so I will give that a shot when I have a chance, but I have some exams to write in the next couple of weeks so I may or may not have time. I will give updates here when I have tried your suggestion with the event bindings which will hopefully solve my issues.

To answer your question Ruba, my issue getting the demo to work is that the events won’t fire off at all (but they are missing the event bindings in the demo). Perhaps it is supposed to be tested in some other manner, but I figured I would add the event bindings as you suggested but I was stuck on getting those to compile so I could re-test the demo.

Anyone got this working with Unreal 4.5.1? I think I followed the installation correctly (1st time with Unreal) but I can’t build the VS solution. Are these errors related to using a 4.5.0 plugin with 4.5.1 or did I do the installation incorrectly? Thanks

[=nekocat;188298]
Have you done this?:
9)Copy Kinect20.VisualGestureBuilder.dll from C:\Program Files\Microsoft SDKs\Kinect\v2.0-PublicPreview1409\Redist\VGB\x64 into Engine\Binaries\Win64
[/]

Yep, I have done that.

[=RuBa1987]
Either that or maybe try putting on your path. I haven’t ran into this personally, that’s just a guess.

[/]

Sorry, not sure what you mean, is there a place I can set a path to it?

I just put the Kinect20.VisualGestureBuilder.dll with the packaged game at WindowsNoEditor\ProjectName\Binaries\Win64\ and it’s working for now

[=Mosel3y;189973]
Yep, I have done that.

Sorry, not sure what you mean, is there a place I can set a path to it?
[/]

It sounds like you got it so don’t try adding it to your path. My guess would be that you need to set something in your build/package settings but I’m not sure what it is.

[=;188755]
Anyone got this working with Unreal 4.5.1? I think I followed the installation correctly (1st time with Unreal) but I can’t build the VS solution. Are these errors related to using a 4.5.0 plugin with 4.5.1 or did I do the installation incorrectly? Thanks


[/]

I’m working in 4.5.1 and things are working just fine on my side.

Just a couple dumb things to check.

1 - Make sure you have the correct version of the plugin installed
2 - Make sure you have the dll copied over
3 - Make sure you are putting the plugin in the engine and not in your project

In theory there should rarely be an issue with using anything within 4.x.* with something else from 4.x.* provided that x is constant. This is semantic versioning. In general this means that if the first digit changes (the 4) then there has been a major change. This can be likened to a new version of windows getting release (like windows 7 vs windows 8). The second digit changing (the 5 in this case) indicates potential breaking changes due to API changes. The last digit increments that typically indicates smaller internal changes, meaning no API/breaking changes just internal changes.

[=nekocat;187823]
Whoops, turns out I had linked my UE account to the wrong github account. After linking it to the correct one it all works, thanks!

Edit:
I compiled the editor successfully, the plugin is present and enabled in the plugin window, my kinect turns on, and I get no errors aside from the ones listed below, but I get no response from the video streams from the demo editor project. The prebuilt project works fine. Thoughts?


Error /Game/DemoRoom/Materials/M_DemoWallPattern_NoText : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Metal
Error /Game/DemoRoom/Materials/M_DemoWall_Inst : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Rubber
Error /Game/DemoRoom/Materials/M_DemoWall_Inst_3 : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Metal
Error /Game/DemoRoom/Materials/M_DemoRoomTiles_Inst : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Concrete
Error /Game/DemoRoom/Materials/M_DemoWallPattern_Inst : Can't find file for asset. /Game/ExampleContent/Audio/Physical_Materials/Metal

[/]

I’ve narrowed the issue down to the frame events not outputting anything in the blueprints. Any clues as to what might be going wrong?

[=nekocat;191003]
I’ve narrowed the issue down to the frame events not outputting anything in the blueprints. Any clues as to what might be going wrong?
[/]

Are they not getting fired or are they not outputting data?

Make sure you bound the skeleton to an actual player number on the skeleton detected event as well. There will be a little drop down in the assign skeleton node that you can use to pick the player number.

I placed breakpoints on the New Color Frame event, as well as the other events, but it doesn’t look like any of the streams are being fired.

Can we do background removal with the already available blueprints ?

Hi guys and gals, I’m a late comer. New to ue, but I already have the new Kinect v2 and the SDK. I got the plugin for Unity. Where to I go to download the plugin for ue4? I appreciate any help.

[=;194965]
Hi guys and gals, I’m a late comer. New to ue, but I already have the new Kinect v2 and the SDK. I got the plugin for Unity. Where to I go to download the plugin for ue4? I appreciate any help.
[/]

this should do it for you. Just grab the correct version. You’re probably going to want to go back to page 4 or 5 and find the tutorial on installing it and stuff since it’s an engine plugin and you’re new to UE.

https://github.com/lion03/UE4-K4W-Plugin/tree/KinectV2Plugin-4.4

Hi there…

So where can I expect Plugin for XBOX 360 Kinect!

Has anyone gotten lion’s demo to work? If so, what changes need to be made?

Hey all!!!

It’s been awhile :slight_smile:

Sorry for that, as some of you know my day job is robotics engineer at Zuta labs (the mini robotic printer) and I’ve been working almost 24/7 for the past month and a half or so preparing the CES demo for the company (BTW we won a CES Innovation award (that didn’t help with the pressure at all :P)).

Now I am back with a bit more the time and I have a lot of slack to pickup.

I will update the plugin as soon as I can :slight_smile:

Hey!

4.6 update is coming soon (I am testing it right now :)) also I will release another demo soon afterwards that will include motion capture example.