Announcement

Collapse
No announcement yet.

[Plugin] Leap Motion - Event Driven

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • replied
    Originally posted by getnamo View Post
    Just had a look at the commit, that's a beefy commit, great work!

    I should have a chance to play with it this weekend, always wanted a better understanding of setting up IK/animation in c++ and I think this is a great resource to brush up on that department.


    Long term thoughts:
    In the end I think convenience setups such as rigged characters should be abstracted away from the hardware plugins themselves. A full 'abstract skeleton' target could be used that can take input from multiple devices and merge the information by the best data available. From that skeleton you would then extract the information and forward it to any output node e.g. a convenience rigged character. Then whether you use just floating rigged hands, poly models or a full body character you would have the same backend and you wouldn't need to change your rigging for new hardware inputs and you would now support mixed inputs. In my mind this is something akin to a Body State Plugin. You could further expand this by adding some smart class to handle skeleton re-targeting, which could automatically match various skeletal meshes to the abstract one and perhaps allow for almost a one click change to swap any rigged character to support a new skeletal mesh you want to replicate your movement. That would be the ideal setup in my mind.

    The hard part of this is figuring out a way in which plugins such as this one can forward input automatically to a target that may not be there (unless something like this idea makes it into the engine and we can depend on it existing).
    We might see such an interface happen in the near future.
    There's quite a lot going on around VR, and I understand that teams like Epic have other priorities than body rigging for now.
    But it could make quite an interesting project during my thesis if I still have time after the original topic.

    Have you had the time to take a look at how I took care of the fingers bones?
    Code is still a bit messy since it was my first Unreal Engine coding session, but I could clean it up and comment more seriously if you have any use for it. Same for the general c++ implementation, if it could somehow be useful, we can discuss the architecture to integrate it to the master.

    By the way, have you played with the "image-hand" that Leap integrated to their unity3D asset?
    I didn't see it in their UE4 plugin.

    Leave a comment:


  • replied
    Originally posted by Deams View Post
    Finally had a bit of time between the exams to work on this.


    It seems that the only problem there for the leap motion plugin was the lack of the export macro on the classes or specific methods.
    Might be the same for the Zip plugin.



    It seems that best practices are fuzzy for many people on many topics...
    Hopefully, more will pop up in the documentation.

    In the meantime, I decided to keep going into the conversion to C++.
    In short, it works but needs to be finished.
    Most of the blueprints functionalities are now also available from C++ but can also be manipulated via blueprints. This means that you can create a game mode using directly the c++ LeapCharacter class in the settings. The VRController will come later.
    The only blueprints really necessary to conform with the "best practices" is the animation blueprint.
    And since I'm lazy, I preferred adding custom animation nodes than connecting each bones of each fingers to all the nodes... I have to say, I'm impressed by your patience to make it even all look nice in there. :P

    More information in the commit message: Commit details

    There's still some work to do, but it should allow both the blueprint and c++ ways to coexist at each level.
    Just had a look at the commit, that's a beefy commit, great work!

    I should have a chance to play with it this weekend, always wanted a better understanding of setting up IK/animation in c++ and I think this is a great resource to brush up on that department.


    Long term thoughts:
    In the end I think convenience setups such as rigged characters should be abstracted away from the hardware plugins themselves. A full 'abstract skeleton' target could be used that can take input from multiple devices and merge the information by the best data available. From that skeleton you would then extract the information and forward it to any output node e.g. a convenience rigged character. Then whether you use just floating rigged hands, poly models or a full body character you would have the same backend and you wouldn't need to change your rigging for new hardware inputs and you would now support mixed inputs. In my mind this is something akin to a Body State Plugin. You could further expand this by adding some smart class to handle skeleton re-targeting, which could automatically match various skeletal meshes to the abstract one and perhaps allow for almost a one click change to swap any rigged character to support a new skeletal mesh you want to replicate your movement. That would be the ideal setup in my mind.

    The hard part of this is figuring out a way in which plugins such as this one can forward input automatically to a target that may not be there (unless something like this idea makes it into the engine and we can depend on it existing).

    Leave a comment:


  • replied
    Finally had a bit of time between the exams to work on this.

    Originally posted by getnamo View Post
    I haven't touched C++ project binding to plugins in a while, and I've just recently hit the same problem actually with my zip plugin!
    It seems that the only problem there for the leap motion plugin was the lack of the export macro on the classes or specific methods.
    Might be the same for the Zip plugin.

    Originally posted by getnamo View Post
    I'm going to try your fixes and get back to you, but thanks a lot for the suggestions! If this compiles for you without problems, consider making a pull request to have the changes added to the master branch.

    Edit: Back in 4.0 I would copy the class implementation files for editor (where plugins are isolated in their own dlls) and remove them in packaged solutions (your dlls are collapsed into a monolithic build so it would link without issue). So this is much cleaner than that hack.

    According to plugin documentation what we're doing is essentially creating a hard bind, but I believe it falls under

    If you are creating a game plugin (not engine plugin), and you want to allow the game to statically link against one of your plugin modules. This breaks the concept of a plugin a little bit, but is often useful for plugins that want to declare new UObject types that game classes can inherit from or use directly. The engine itself has no dependencies on this plugin, but the game project code and content may very well have direct dependencies.

    I'm not 100% sure on best practices, but it does seem to be the most convenient and when you want to use leap motion specific methods in C++ there really isn't a practical way to not use this approach.
    It seems that best practices are fuzzy for many people on many topics...
    Hopefully, more will pop up in the documentation.

    In the meantime, I decided to keep going into the conversion to C++.
    In short, it works but needs to be finished.
    Most of the blueprints functionalities are now also available from C++ but can also be manipulated via blueprints. This means that you can create a game mode using directly the c++ LeapCharacter class in the settings. The VRController will come later.
    The only blueprints really necessary to conform with the "best practices" is the animation blueprint.
    And since I'm lazy, I preferred adding custom animation nodes than connecting each bones of each fingers to all the nodes... I have to say, I'm impressed by your patience to make it even all look nice in there. :P

    More information in the commit message: Commit details

    There's still some work to do, but it should allow both the blueprint and c++ ways to coexist at each level.

    Leave a comment:


  • replied
    Originally posted by Deams View Post

    ...

    Is it intended?
    I haven't touched C++ project binding to plugins in a while, and I've just recently hit the same problem actually with my zip plugin!

    I'm going to try your fixes and get back to you, but thanks a lot for the suggestions! If this compiles for you without problems, consider making a pull request to have the changes added to the master branch.


    Edit: Back in 4.0 I would copy the class implementation files for editor (where plugins are isolated in their own dlls) and remove them in packaged solutions (your dlls are collapsed into a monolithic build so it would link without issue). So this is much cleaner than that hack.

    According to plugin documentation what we're doing is essentially creating a hard bind, but I believe it falls under

    If you are creating a game plugin (not engine plugin), and you want to allow the game to statically link against one of your plugin modules. This breaks the concept of a plugin a little bit, but is often useful for plugins that want to declare new UObject types that game classes can inherit from or use directly. The engine itself has no dependencies on this plugin, but the game project code and content may very well have direct dependencies.

    I'm not 100% sure on best practices, but it does seem to be the most convenient and when you want to use leap motion specific methods in C++ there really isn't a practical way to not use this approach.
    Last edited by getnamo; 01-08-2016, 04:57 PM.

    Leave a comment:


  • replied
    Originally posted by getnamo View Post
    1) Attach a component to your pawn/actor.

    2) Use the LeapEventInterface via multiple-inheritance
    I tried to add the LeapController component to a default character, but so far I can't managed to link the code.

    In the Char.h I include "LeapController.h" and define a pointer to the component.
    Code:
    class ULeapController* LeapController;
    In the Char.cpp, I instantiate the LeapController in the class constructor
    Code:
    ALeapCharacter::ALeapCharacter()
    {
    	PrimaryActorTick.bCanEverTick = true;
    
    	LeapController = CreateDefaultSubobject<ULeapController>(TEXT("LeapControllerComponent"));
    }
    But end up getting the following link error:
    Code:
    LeapCharacter.cpp.obj : error LNK2019: unresolved external symbol "private: static class UClass * __cdecl ULeapController::GetPrivateStaticClass(wchar_t const *)" (?GetPrivateStaticClass@ULeapController@@CAPEAVUClass@@PEB_W@Z) referenced in function "public: class ULeapController * __cdecl UObject::CreateDefaultSubobject<class ULeapController>(class FName,bool)" (??$CreateDefaultSubobject@VULeapController@@@UObject@@QEAAPEAVULeapController@@VFName@@_N@Z)
    Is there another step I'm forgetting about?

    Edit:
    It seems that adding the MinimalAPI tag for the ULeapController class fixes the compilation error to instantiate the component.
    Code:
    UCLASS(MinimalAPI, ClassGroup=Input, meta=(BlueprintSpawnableComponent))
    class ULeapController : public UActorComponent
    But the same kind of linking error happen when using a method of ULeapController.
    Example when using the EnableBackgroundTracking:
    Code:
    LeapCharacter.cpp.obj : error LNK2019: unresolved external symbol "public: void __cdecl ULeapController::EnableBackgroundTracking(bool)" (?EnableBackgroundTracking@ULeapController@@QEAAX_N@Z) referenced in function "public: __cdecl ALeapCharacter::ALeapCharacter(void)" (??0ALeapCharacter@@QEAA@XZ)
    Using the LEAPMOTION_API tag for the class definition of ULeapController does seem to export all methods.
    Code:
    class LEAPMOTION_API ULeapController : public UActorComponent
    Is it intended?
    Last edited by Deams; 01-08-2016, 04:42 PM. Reason: New data

    Leave a comment:


  • replied
    Thank you for the detailed answer!
    I will give it a try tonight and let you know how it goes.

    Leave a comment:


  • replied
    Originally posted by Deams View Post
    Hi Getnamo!

    First let me thank you for all the plugins you've been sharing on github.

    I was wondering if you had any idea on how I should approach translating your bp using only c++.
    I mean reproducing the logic of your blueprints such as RiggedCharacter or VRController in code.
    Been playing around with the plugin, but thought I would ask you directly if you'd thought about it already.

    Edit:
    It seems that using directly the plugin in c++ was not possible based on one of your earlier answer (#45).
    Is it still the case?

    Edit2:
    Based on the workaround from your previous answer, I duplicated and then modified the parent of LeapBasicRiggedCharacter for a c++ character.
    It does work and allow to have the c++ code of the character and the leap plugin character working in concert.
    But in such an implementation, how could we access the leap info such as the hand position from the code?
    Does it forces us to do everything from blueprints with the plugin?
    Glad my plugins can be of use!

    To access leap motion from C++, do the same thing you would do in blueprint but using the C++ methods. If you're confident you should probably use the custom approach as it will give you more direct control and flexibility which is what I'm assuming your C++ approach is about.

    1) Attach a component to your pawn/actor.

    2) Use the LeapEventInterface via multiple-inheritance

    Now you will receive the same events you would get in blueprints, all defined in the LeapEventInterface class. Then you can follow the same instructions found at the repository readme, pay attention for example to the Example 'Debug Hand' graph on how to get basic hand information.

    You have nearly complete access to the Leap API through the UE components which handle all orientation and unit conversion, but since you're in C++ you can also query the leap directly after including the necessary headers, e.g. using the pure Leap API Leap Controller
    Code:
    Leap::Controller leap;
    See my unreal ULeapController component class for use examples.


    On the topic of recreating the VRController, all it does is ensure that your camera is separated from your view controller rotation. You can follow this wiki for an example of how to do that in C++.


    All the convenience characters such as RiggedCharacter are a bit more complex. What they do is contain an actor called LeapAnimBodyConnector or its subclass, which handles all the leap-specific interaction and stores it in an AnimBody data structure class. Then in the Animation Blueprint of the skeletal mesh, it looks for the AnimBody and copies its current state to the skeletal mesh, blending in animation when information is not being sent. This means the animation blueprints and characters are agnostic to the input, and the only leap specific part is the LeapAnimBodyConnector.

    I haven't had the opportunity to explore how animation blueprints can be set in C++, if you find out how to do that, some guidance on it would be awesome and might help pave the way for easier skeletal mesh swapping in the future.
    Last edited by getnamo; 01-07-2016, 09:29 AM.

    Leave a comment:


  • replied
    Hi Getnamo!

    First let me thank you for all the plugins you've been sharing on github.

    I was wondering if you had any idea on how I should approach translating your bp using only c++.
    I mean reproducing the logic of your blueprints such as RiggedCharacter or VRController in code.
    Been playing around with the plugin, but thought I would ask you directly if you'd thought about it already.

    Edit:
    It seems that using directly the plugin in c++ was not possible based on one of your earlier answer (#45).
    Is it still the case?

    Edit2:
    Based on the workaround from your previous answer, I duplicated and then modified the parent of LeapBasicRiggedCharacter for a c++ character.
    It does work and allow to have the c++ code of the character and the leap plugin character working in concert.
    But in such an implementation, how could we access the leap info such as the hand position from the code?
    Does it forces us to do everything from blueprints with the plugin?
    Last edited by Deams; 01-06-2016, 09:41 PM.

    Leave a comment:


  • replied
    Originally posted by caesarj View Post
    Hi,

    I just tried 0.9.7 on MAC 10.10.4 + Leapmotion 2.3.1 + Unreal 4.9.2

    But always report "Plugin 'LeapMotion' failed to load because module 'LeapMotion' does not appear to be compatible with the current version of the engine. The plugin may need to be recompiled." after I copy binaries and plugin to the project folder.

    What is wrong?

    BR
    Caesar
    0.9.7 has mac binaries synced to 4.7. Use UE4.7 or use plugin v 0.9.9 with UE4.10. Optionally add code to your project and recompile as you have source included with the plugin.

    Leave a comment:


  • replied
    Hi,

    I just tried 0.9.7 on MAC 10.10.4 + Leapmotion 2.3.1 + Unreal 4.9.2

    But always report "Plugin 'LeapMotion' failed to load because module 'LeapMotion' does not appear to be compatible with the current version of the engine. The plugin may need to be recompiled." after I copy binaries and plugin to the project folder.

    What is wrong?

    BR
    Caesar

    Leave a comment:


  • replied
    Originally posted by artoculon View Post
    Hi getnamo;

    Thanks so much for your plugin. Real amazing you've been able to provide UE support unofficially. Love the echo hands, so sculptural.

    I've noticed that when using the collision characters, the tracking accuracy is much worse. Without collision, seems to function better. Is that just my subjective experience or have you noticed this as well?
    There may be a subjective difference betweeen the yellow character and the echo hands since the echo hands are closer to 1:1, but collision itself should not make a difference, it is literally a single toggle.

    BTW Echo hands are Leap assets that I've gotten permission to release with my plugin, so praise leap for the geometry


    Originally posted by tolstenko View Post
    Hello, Do you haveany update for Unreal 4.10.0? It shows this message:

    [ATTACH=CONFIG]66125[/ATTACH]

    In mean time I will revert to the 4.9.2

    I am using OSX El Caitan 10.11.2
    Update to 0.9.9
    -Updated Mac binaries to 4.10

    Note that mac binaries will be very rarely synced due to lack of time, so I highly urge a Mac user of the plugin to take the helm for keeping mac binaries in sync.

    Edit: updated this response with 0.9.9 change

    Mac is now synced to 4.10, but you have full source so you can compile the mac binary yourself at any time and make a pull request to include the binaries in the master branch for others.
    Last edited by getnamo; 11-20-2015, 05:20 PM.

    Leave a comment:


  • replied
    Update to 0.9.8
    -Updated binaries to UE4.10

    Image hands are still not ready atm.

    Let me know if any bugs creep up!

    Originally posted by polemos View Post
    getnamo is your plugin going to be built into one of the upcoming ue4 releases?
    Not that I know of :O

    Originally posted by knack View Post
    mine failed, but i don't known if its because the plugin or something wrong after upgrade to Visual 2015 since i update from 2014. Haven't time for check.

    edit:
    compiled but i got this error on load

    LogModuleManager:Warning: ModuleManager: Unable to load module 'E:/Documents/Unreal Projects/xVR2/Plugins/LeapMotion/Binaries/Win64/UE4Editor-LeapMotion.dll' because the file couldn't be loaded by the OS.

    edit2:
    something wrong using duplicate project, i backup my project instead of duplicate for preserve 4.9.2 files, and work fine.

    binary http://s000.tinyupload.com/index.php...23592589999963
    My version compiled fine without any code changes, have a go with the new release (0.9.8) to confirm if it works.
    Last edited by getnamo; 11-20-2015, 02:33 PM.

    Leave a comment:


  • replied
    Originally posted by getnamo View Post
    I'll get a 4.10 version compiled this weekend. There shouldn't be a need for code changes for 4.10 so you should be able to compile a version on your own if you want it earlier.
    getnamo is your plugin going to be built into one of the upcoming ue4 releases?

    Leave a comment:


  • replied
    mine failed, but i don't known if its because the plugin or something wrong after upgrade to Visual 2015 since i update from 2014. Haven't time for check.

    edit:
    compiled but i got this error on load

    LogModuleManager:Warning: ModuleManager: Unable to load module 'E:/Documents/Unreal Projects/xVR2/Plugins/LeapMotion/Binaries/Win64/UE4Editor-LeapMotion.dll' because the file couldn't be loaded by the OS.

    edit2:
    something wrong using duplicate project, i backup my project instead of duplicate for preserve 4.9.2 files, and work fine.

    binary http://s000.tinyupload.com/index.php...23592589999963
    Last edited by knack; 11-18-2015, 06:11 PM.

    Leave a comment:


  • replied
    I'll get a 4.10 version compiled this weekend. There shouldn't be a need for code changes for 4.10 so you should be able to compile a version on your own if you want it earlier.

    Leave a comment:

Working...
X