User Tag List

Page 1 of 11 123 ... LastLast
Results 1 to 40 of 424

Thread: Kinect 4 Windows v2.0 Plugin

  1. #1
    0

    Kinect 4 Windows v2.0 Plugin



    Important! Read First!
    Disclaimer:
    The Kinect for Windows V2 hardware and software described here are preliminary and subject to change.

    Compatibility:

    1) This plugin is designed for Kinect for Windows V2 and can be compiled only by participants of Microsoft's Kinect4Windows Developer preview program with properly installed SDK.
    2) It is not backward compatible to the Kinect 4 Windows V1 and XBOX 360 Kinect!

    Kinect 4 Windows V2 SDK:
    1) This plugin does not include the Kinect 4 Windows V2 SDK and K4WLib assumes that you have it installed.
    2) Please do not ask for the SDK because I cannot redistribute it (see Compatibility note 1).


    Hello all!

    So the moment I found out that UE4 is out and went over the code I knew that the potential for serious Kinect game is amazing!
    Fortunately I participate in the developer preview program for the windows version of the Xbox One Kinect
    So I started hacking at it!

    For now I was able to hook into the input device system and open a connection to the sensor from inside the UE.

    Repos:

    4.1

    4.2



    Demos:

    Prebuilt

    Editor Project

    UPDATES:

    Fully functional

    Work in progress

    Does not function


    Upcoming update:

    1) No more custom engine build!
    The plugin framework is now based on IInputDeviceModule that means that it is now a true plugin.


    2) One editor to rule them all!
    From the next update if you want Kinect in your game you don't need to go into C++ at all!
    That means that no more recompiling and overriding functions all you'll need to receive Kinect events is to add kinect listener inrerface to any blueprint you want and you will get instantly a Kinect event in that blueprint!


    3) If the kinect can recognize track hand states why can't I see it on my model?
    Well the update will include an animation node adding the ability to blend animations with the kinect controlled avatar!
    So now if you close or open your hand the model will too!


    4) Who said that if you stand in one spot in your room you should stay in the same place in game?
    The update will include body joystick! Want to go back and forward just lean forward or back! If you want to strafe just lean to the sides!
    And if you want to turn? No problem! Use your shoulders as a steering wheel!


    17/5/14

    Added color,depth and infrared stream interfaces

    Latest update

    Last edited by lion032; 05-21-2014 at 08:08 AM.

  2. #2
    0
    Nice, I hope that with the source code being available that Kinect will be able to have a high level of integration.

    Hopeing for the ability to use Kinect 2.0 + Oculus Rift

  3. #3
    0
    Quote Originally Posted by lion032 View Post
    Hello all!

    For now I was able to hook into the input device system and open a connection to the sensor from inside the UE.

    Hi,
    That was pretty quick work! I too am part of the Kinect V2 preview program and am also using the Kinect for use with the Oculus rift. I've had great results in Unity, but it looks like it's time to port to Unreal.

    Can you tell me the steps you too to get access to the device from within Unreal Engine (or maybe have any sample code)? I've got some ideas (maybe a plugin), but I'm not sure where to start.

  4. #4
    0
    I'm setting up a github repo I'll share my code there.

    https://github.com/lion03/UnrealEngine
    Last edited by lion032; 03-21-2014 at 05:36 PM.

  5. #5
    0
    This is relevant to my interests.

    Thank you.

    If only I had a Kinect2 for Windows...
    Rule#21: Be polite, be professional, but have a plan to kill everyone you meet.


  6. #6
    0
    I am implementing it with the option of easy addition of other NUI sensors.

  7. #7
    0
    I expect to get my Kinect 2 sensor soon, and will be very interested in the integration so keep us posted please

  8. #8
    0
    Ok, some progress reports

    I have decided to go around the input system and add my own due to the limitations of the input system.
    Also it took me a while but I finally figured out how the threading system works and hopefully the data pooling from the sensor would be in it's own thread.

  9. #9
    0
    Quote Originally Posted by lion032 View Post
    Ok, some progress reports

    I have decided to go around the input system and add my own due to the limitations of the input system.
    Also it took me a while but I finally figured out how the threading system works and hopefully the data pooling from the sensor would be in it's own thread.
    Hey, thanks for posting your work so far with the Kinect plugin. It's been a great help.

    I am also looking at how the threading works now, do you have any samples/advice on how to get the sensor polling running in it's own thread?

  10. #10
    0
    I am actually looking into it as we speak.
    Trying to figure out why the heck the thread I opened doesn't die on shutdown

  11. #11
    0
    Quote Originally Posted by lion032 View Post
    Hello all!

    So the moment I found out that UE4 is out and went over the code I knew that the potential for serious Kinect game is amazing!
    Fortunately I participate in the developer preview program for the windows version of the Xbox One Kinect
    So I started hacking at it!

    For now I was able to hook into the input device system and open a connection to the sensor from inside the UE.

    More updates coming soon!
    Fantastic, keeps us informed

  12. #12
    0
    So!

    Big update!

    I am almost done! The only thing that is left is to write the dispatcher system.

    Hopefully It will be ready in the next few days!

  13. #13
    0
    Big like on that

  14. #14
    0
    want a bigger like?
    About a minute ago I got it to work

    Managed to automatically initiate the sensor in it's own thread.
    Build event dispatching system that is automatically manages listeners.
    And the most important thing is that a game character is actually receives body tracking data.

    the code is still really dirty but if you want I can share it

  15. #15
    0
    News is supposedly the Windows version of Kinect 2.0 isn't far from being released.

  16. #16
    0
    Quote Originally Posted by lion032 View Post
    want a bigger like?
    About a minute ago I got it to work

    Managed to automatically initiate the sensor in it's own thread.
    Build event dispatching system that is automatically manages listeners.
    And the most important thing is that a game character is actually receives body tracking data.

    the code is still really dirty but if you want I can share it
    I'd love to play around with it, but have to wait 2 more weeks for my Kinect 2 kit The UK went out of stock immediately, and I wasnt quick enough.

  17. #17
    0
    Quote Originally Posted by SiggiG View Post
    I'd love to play around with it, but have to wait 2 more weeks for my Kinect 2 kit The UK went out of stock immediately, and I wasnt quick enough.
    well I am from Israel and I had to get my sensor from the US

  18. #18
    0
    I've just moved over from Unity and was going to look at adding Kinect v2 support. Looks like you're making great progress. Would love to see skeletal tracking

  19. #19
    0
    I hope I'll make some progress towards skeletal tracking in the next few days

  20. #20
    0
    Hey,

    How goes your progress? I should be getting my Kinect kit next week and looking forward to playing with it in UE4

  21. #21
    0
    So excited for this. A simple Kinect tie in with UE is something I've been wanting for awhile. Unity makes it so easy.

    Lion your github repo is 404.

  22. #22
    0
    how bad is the response time?

  23. #23
    0
    Quote Originally Posted by JohnF30 View Post
    how bad is the response time?
    I think the NDA prevents testers from discussing specifics of the kit.


    Got my kit today, looking forward to doing some tests of my own

  24. #24
    0
    Hello all!

    Sorry I couldn't update this thread earlier but things at my day job are kinda crazy for the last few weeks.
    (my day job -> https://www.kickstarter.com/projects/1686304142/the-mini-mobile-robotic-printer)

    I made few advances but they are really preliminary and in need for more work.

    I hope to resume my work on the plugin ASAP

  25. #25
    0
    This is extremely cool!

    But I am wondering if you have any idea of whether this will play nice with the positional tracking added to the Oculus Rift DK2? I'm concerned that the systems will confuse each other. I am really hoping to see a VR experience in which the hands can be used to interact with the game world and their positions reproduced accurately. From what I've read I'm not sure the Leap Motion is the right tool for the job, as I am more interested in the hand position over a large area than in the precise position of individual fingers.

  26. #26
    0
    Samaritan
    Join Date
    Apr 2014
    Posts
    141
    This has huge potential - especially when used with Oculus Rift, if you can capture your own movements and realise them in a 3D world, and free yourself from using keyboard/mouse/joypad - that is true VR.

    p.s. I dont think using a hand gesture like that would be very good for firing a weapon though - players will want speed and accuracy - and thats something only a trigger or button can provide. :/

  27. #27
    0
    That is the main reason I am doing it.

    I have a game in mind

  28. #28
    0
    Leap doesn't have enough range for motion control. If you want to combine tracking there shouldn't be a problem, you just have to setup how your character skeleton interprets the information and applies it.

  29. #29
    0
    Quote Originally Posted by lion032 View Post
    I'm setting up a github repo I'll share my code there.
    https://github.com/lion03/UnrealEngine
    Hi lion032, thanks for sharing your code! Is this Github page still up? I tried to access it today and only found a 404 page.

  30. #30
    0
    It's not, mainly because I changed my approach to the problem and actually im already working on retargeting a mesh.
    If there is someone that can help me to retarget the kinect coordinate system to the unreal coordinate system it will speed things up a lot.

  31. #31
    0
    Quote Originally Posted by lion032 View Post
    It's not, mainly because I changed my approach to the problem and actually im already working on retargeting a mesh.
    If there is someone that can help me to retarget the kinect coordinate system to the unreal coordinate system it will speed things up a lot.
    We would be willing to help out, we have some experience doing this with the first version of the Kinect. Perhaps we could collaborate?
    We'd also be interested to see the first approach, if you still have the code around, even if it isn't very complete.

  32. #32
    0
    Can you share your approach? I'm essentially doing the same thing, but making sure to take an approach generic enough for other trackers. I've already got most of it working follow essentially this approach. However, I was trying to keep things purely plugin based (custom actors and controller wrapped up in a plugin) and not changing the main engine, but really the proper architectural fit is to use the standard input stack and modify the GenericApplicationMessangeHandler class to include an OnPoseAction event with a TArray of FRotators as the parameters. Or perhaps a named bone -> FRotator map.

  33. #33
    0
    I use a simple listener/dispatcher pattern.
    the kinect runs on it's own thread and basicly just dispatches processed kinect to listeners.
    I was unsuccessful in adding it in to the GenericApplicationMessangeHandler without altering the engine itself.

  34. #34
    0
    Yea there are a lot of places it touches GenericMessageHandler, UPlayerInput, PlayerController, etc. especially for a module that is pretty windows specific at the moment. I don't want to go fussing around and diverge so far from the source until we get a more generic data path from Epic. Something that can work for any skeletal mesh style input (Kinect 1/2, PrioVR, etc.). Once I get a couple trackers working in a non-generic way I'll post my thoughts in the feature request forums.

  35. #35
    0


    so I really want to upload the code but i have some trouble with github :\

    also after thinking it over i decided to add additional module to the engine itself to add support for custom hardware input.

  36. #36
    0
    so you should be able to get my code from my github now

    github.com/lion03/UnrealEngien

    fork it because I made changes to the engine itself.

  37. #37
    0
    I built your version of UE4 and have enabled the Kinectv2 plugin.
    How do you set up your character?

    Or if you have an example project with it running that would be great
    Perform live gigs in virtual reality.
    Want to get involved? Please get in touch.

  38. #38
    0
    I haven't had the chance to properly write an actor class yet.
    But you can inherit a blueprint from AKiniectListenerActor and you will have access to OnBodyReciev event but for now it is abit useless because I haven't written the conversation from FBodyFrame to FTransform.

  39. #39
    0
    Sorry I didn't read everything, I just watched the video :P did you code or use blueprints?

  40. #40
    0
    Hi Lion32,

    I don't know how I've missed this thread, but I am quite impressed with your progress thus far! Are you planning on providing your plugin for Github or the marketplace? I can't wait to see more of what you come up with!
    Adam Davis | Marketplace Support | Epic Games
    How to report a bug? | Installation & Setup issues? | Answerhub Bug Reports | Twitter

Page 1 of 11 123 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •