Announcement

Collapse
No announcement yet.

[Plugin] Leap Motion - Event Driven

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    mine failed, but i don't known if its because the plugin or something wrong after upgrade to Visual 2015 since i update from 2014. Haven't time for check.

    edit:
    compiled but i got this error on load

    LogModuleManager:Warning: ModuleManager: Unable to load module 'E:/Documents/Unreal Projects/xVR2/Plugins/LeapMotion/Binaries/Win64/UE4Editor-LeapMotion.dll' because the file couldn't be loaded by the OS.

    edit2:
    something wrong using duplicate project, i backup my project instead of duplicate for preserve 4.9.2 files, and work fine.

    binary http://s000.tinyupload.com/index.php...23592589999963
    Last edited by knack; 11-18-2015, 06:11 PM.
    pd: excuse my english.

    lfw/paid modeling, painting, texturing.

    Comment


      Originally posted by getnamo View Post
      I'll get a 4.10 version compiled this weekend. There shouldn't be a need for code changes for 4.10 so you should be able to compile a version on your own if you want it earlier.
      getnamo is your plugin going to be built into one of the upcoming ue4 releases?

      Comment


        Update to 0.9.8
        -Updated binaries to UE4.10

        Image hands are still not ready atm.

        Let me know if any bugs creep up!

        Originally posted by polemos View Post
        getnamo is your plugin going to be built into one of the upcoming ue4 releases?
        Not that I know of :O

        Originally posted by knack View Post
        mine failed, but i don't known if its because the plugin or something wrong after upgrade to Visual 2015 since i update from 2014. Haven't time for check.

        edit:
        compiled but i got this error on load

        LogModuleManager:Warning: ModuleManager: Unable to load module 'E:/Documents/Unreal Projects/xVR2/Plugins/LeapMotion/Binaries/Win64/UE4Editor-LeapMotion.dll' because the file couldn't be loaded by the OS.

        edit2:
        something wrong using duplicate project, i backup my project instead of duplicate for preserve 4.9.2 files, and work fine.

        binary http://s000.tinyupload.com/index.php...23592589999963
        My version compiled fine without any code changes, have a go with the new release (0.9.8) to confirm if it works.
        Last edited by getnamo; 11-20-2015, 02:33 PM.
        Plugins: GES - Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo

        Comment


          Originally posted by artoculon View Post
          Hi getnamo;

          Thanks so much for your plugin. Real amazing you've been able to provide UE support unofficially. Love the echo hands, so sculptural.

          I've noticed that when using the collision characters, the tracking accuracy is much worse. Without collision, seems to function better. Is that just my subjective experience or have you noticed this as well?
          There may be a subjective difference betweeen the yellow character and the echo hands since the echo hands are closer to 1:1, but collision itself should not make a difference, it is literally a single toggle.

          BTW Echo hands are Leap assets that I've gotten permission to release with my plugin, so praise leap for the geometry


          Originally posted by tolstenko View Post
          Hello, Do you haveany update for Unreal 4.10.0? It shows this message:

          [ATTACH=CONFIG]66125[/ATTACH]

          In mean time I will revert to the 4.9.2

          I am using OSX El Caitan 10.11.2
          Update to 0.9.9
          -Updated Mac binaries to 4.10

          Note that mac binaries will be very rarely synced due to lack of time, so I highly urge a Mac user of the plugin to take the helm for keeping mac binaries in sync.

          Edit: updated this response with 0.9.9 change

          Mac is now synced to 4.10, but you have full source so you can compile the mac binary yourself at any time and make a pull request to include the binaries in the master branch for others.
          Last edited by getnamo; 11-20-2015, 05:20 PM.
          Plugins: GES - Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo

          Comment


            Hi,

            I just tried 0.9.7 on MAC 10.10.4 + Leapmotion 2.3.1 + Unreal 4.9.2

            But always report "Plugin 'LeapMotion' failed to load because module 'LeapMotion' does not appear to be compatible with the current version of the engine. The plugin may need to be recompiled." after I copy binaries and plugin to the project folder.

            What is wrong?

            BR
            Caesar

            Comment


              Originally posted by caesarj View Post
              Hi,

              I just tried 0.9.7 on MAC 10.10.4 + Leapmotion 2.3.1 + Unreal 4.9.2

              But always report "Plugin 'LeapMotion' failed to load because module 'LeapMotion' does not appear to be compatible with the current version of the engine. The plugin may need to be recompiled." after I copy binaries and plugin to the project folder.

              What is wrong?

              BR
              Caesar
              0.9.7 has mac binaries synced to 4.7. Use UE4.7 or use plugin v 0.9.9 with UE4.10. Optionally add code to your project and recompile as you have source included with the plugin.
              Plugins: GES - Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo

              Comment


                Hi Getnamo!

                First let me thank you for all the plugins you've been sharing on github.

                I was wondering if you had any idea on how I should approach translating your bp using only c++.
                I mean reproducing the logic of your blueprints such as RiggedCharacter or VRController in code.
                Been playing around with the plugin, but thought I would ask you directly if you'd thought about it already.

                Edit:
                It seems that using directly the plugin in c++ was not possible based on one of your earlier answer (#45).
                Is it still the case?

                Edit2:
                Based on the workaround from your previous answer, I duplicated and then modified the parent of LeapBasicRiggedCharacter for a c++ character.
                It does work and allow to have the c++ code of the character and the leap plugin character working in concert.
                But in such an implementation, how could we access the leap info such as the hand position from the code?
                Does it forces us to do everything from blueprints with the plugin?
                Last edited by Deams; 01-06-2016, 09:41 PM.
                Released:
                [Plugin] Custom Gesture Recognition
                [Plugin][Example] Java in UE4 - Text extraction via Tika
                [Plugin] Cover Generator

                GitHub Link

                Comment


                  Originally posted by Deams View Post
                  Hi Getnamo!

                  First let me thank you for all the plugins you've been sharing on github.

                  I was wondering if you had any idea on how I should approach translating your bp using only c++.
                  I mean reproducing the logic of your blueprints such as RiggedCharacter or VRController in code.
                  Been playing around with the plugin, but thought I would ask you directly if you'd thought about it already.

                  Edit:
                  It seems that using directly the plugin in c++ was not possible based on one of your earlier answer (#45).
                  Is it still the case?

                  Edit2:
                  Based on the workaround from your previous answer, I duplicated and then modified the parent of LeapBasicRiggedCharacter for a c++ character.
                  It does work and allow to have the c++ code of the character and the leap plugin character working in concert.
                  But in such an implementation, how could we access the leap info such as the hand position from the code?
                  Does it forces us to do everything from blueprints with the plugin?
                  Glad my plugins can be of use!

                  To access leap motion from C++, do the same thing you would do in blueprint but using the C++ methods. If you're confident you should probably use the custom approach as it will give you more direct control and flexibility which is what I'm assuming your C++ approach is about.

                  1) Attach a component to your pawn/actor.

                  2) Use the LeapEventInterface via multiple-inheritance

                  Now you will receive the same events you would get in blueprints, all defined in the LeapEventInterface class. Then you can follow the same instructions found at the repository readme, pay attention for example to the Example 'Debug Hand' graph on how to get basic hand information.

                  You have nearly complete access to the Leap API through the UE components which handle all orientation and unit conversion, but since you're in C++ you can also query the leap directly after including the necessary headers, e.g. using the pure Leap API Leap Controller
                  Code:
                  Leap::Controller leap;
                  See my unreal ULeapController component class for use examples.


                  On the topic of recreating the VRController, all it does is ensure that your camera is separated from your view controller rotation. You can follow this wiki for an example of how to do that in C++.


                  All the convenience characters such as RiggedCharacter are a bit more complex. What they do is contain an actor called LeapAnimBodyConnector or its subclass, which handles all the leap-specific interaction and stores it in an AnimBody data structure class. Then in the Animation Blueprint of the skeletal mesh, it looks for the AnimBody and copies its current state to the skeletal mesh, blending in animation when information is not being sent. This means the animation blueprints and characters are agnostic to the input, and the only leap specific part is the LeapAnimBodyConnector.

                  I haven't had the opportunity to explore how animation blueprints can be set in C++, if you find out how to do that, some guidance on it would be awesome and might help pave the way for easier skeletal mesh swapping in the future.
                  Last edited by getnamo; 01-07-2016, 09:29 AM.
                  Plugins: GES - Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo

                  Comment


                    Thank you for the detailed answer!
                    I will give it a try tonight and let you know how it goes.
                    Released:
                    [Plugin] Custom Gesture Recognition
                    [Plugin][Example] Java in UE4 - Text extraction via Tika
                    [Plugin] Cover Generator

                    GitHub Link

                    Comment


                      Originally posted by getnamo View Post
                      1) Attach a component to your pawn/actor.

                      2) Use the LeapEventInterface via multiple-inheritance
                      I tried to add the LeapController component to a default character, but so far I can't managed to link the code.

                      In the Char.h I include "LeapController.h" and define a pointer to the component.
                      Code:
                      class ULeapController* LeapController;
                      In the Char.cpp, I instantiate the LeapController in the class constructor
                      Code:
                      ALeapCharacter::ALeapCharacter()
                      {
                      	PrimaryActorTick.bCanEverTick = true;
                      
                      	LeapController = CreateDefaultSubobject<ULeapController>(TEXT("LeapControllerComponent"));
                      }
                      But end up getting the following link error:
                      Code:
                      LeapCharacter.cpp.obj : error LNK2019: unresolved external symbol "private: static class UClass * __cdecl ULeapController::GetPrivateStaticClass(wchar_t const *)" (?GetPrivateStaticClass@ULeapController@@CAPEAVUClass@@PEB_W@Z) referenced in function "public: class ULeapController * __cdecl UObject::CreateDefaultSubobject<class ULeapController>(class FName,bool)" (??$CreateDefaultSubobject@VULeapController@@@UObject@@QEAAPEAVULeapController@@VFName@@_N@Z)
                      Is there another step I'm forgetting about?

                      Edit:
                      It seems that adding the MinimalAPI tag for the ULeapController class fixes the compilation error to instantiate the component.
                      Code:
                      UCLASS(MinimalAPI, ClassGroup=Input, meta=(BlueprintSpawnableComponent))
                      class ULeapController : public UActorComponent
                      But the same kind of linking error happen when using a method of ULeapController.
                      Example when using the EnableBackgroundTracking:
                      Code:
                      LeapCharacter.cpp.obj : error LNK2019: unresolved external symbol "public: void __cdecl ULeapController::EnableBackgroundTracking(bool)" (?EnableBackgroundTracking@ULeapController@@QEAAX_N@Z) referenced in function "public: __cdecl ALeapCharacter::ALeapCharacter(void)" (??0ALeapCharacter@@QEAA@XZ)
                      Using the LEAPMOTION_API tag for the class definition of ULeapController does seem to export all methods.
                      Code:
                      class LEAPMOTION_API ULeapController : public UActorComponent
                      Is it intended?
                      Last edited by Deams; 01-08-2016, 04:42 PM. Reason: New data
                      Released:
                      [Plugin] Custom Gesture Recognition
                      [Plugin][Example] Java in UE4 - Text extraction via Tika
                      [Plugin] Cover Generator

                      GitHub Link

                      Comment


                        Originally posted by Deams View Post

                        ...

                        Is it intended?
                        I haven't touched C++ project binding to plugins in a while, and I've just recently hit the same problem actually with my zip plugin!

                        I'm going to try your fixes and get back to you, but thanks a lot for the suggestions! If this compiles for you without problems, consider making a pull request to have the changes added to the master branch.


                        Edit: Back in 4.0 I would copy the class implementation files for editor (where plugins are isolated in their own dlls) and remove them in packaged solutions (your dlls are collapsed into a monolithic build so it would link without issue). So this is much cleaner than that hack.

                        According to plugin documentation what we're doing is essentially creating a hard bind, but I believe it falls under

                        If you are creating a game plugin (not engine plugin), and you want to allow the game to statically link against one of your plugin modules. This breaks the concept of a plugin a little bit, but is often useful for plugins that want to declare new UObject types that game classes can inherit from or use directly. The engine itself has no dependencies on this plugin, but the game project code and content may very well have direct dependencies.

                        I'm not 100% sure on best practices, but it does seem to be the most convenient and when you want to use leap motion specific methods in C++ there really isn't a practical way to not use this approach.
                        Last edited by getnamo; 01-08-2016, 04:57 PM.
                        Plugins: GES - Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo

                        Comment


                          Finally had a bit of time between the exams to work on this.

                          Originally posted by getnamo View Post
                          I haven't touched C++ project binding to plugins in a while, and I've just recently hit the same problem actually with my zip plugin!
                          It seems that the only problem there for the leap motion plugin was the lack of the export macro on the classes or specific methods.
                          Might be the same for the Zip plugin.

                          Originally posted by getnamo View Post
                          I'm going to try your fixes and get back to you, but thanks a lot for the suggestions! If this compiles for you without problems, consider making a pull request to have the changes added to the master branch.

                          Edit: Back in 4.0 I would copy the class implementation files for editor (where plugins are isolated in their own dlls) and remove them in packaged solutions (your dlls are collapsed into a monolithic build so it would link without issue). So this is much cleaner than that hack.

                          According to plugin documentation what we're doing is essentially creating a hard bind, but I believe it falls under

                          If you are creating a game plugin (not engine plugin), and you want to allow the game to statically link against one of your plugin modules. This breaks the concept of a plugin a little bit, but is often useful for plugins that want to declare new UObject types that game classes can inherit from or use directly. The engine itself has no dependencies on this plugin, but the game project code and content may very well have direct dependencies.

                          I'm not 100% sure on best practices, but it does seem to be the most convenient and when you want to use leap motion specific methods in C++ there really isn't a practical way to not use this approach.
                          It seems that best practices are fuzzy for many people on many topics...
                          Hopefully, more will pop up in the documentation.

                          In the meantime, I decided to keep going into the conversion to C++.
                          In short, it works but needs to be finished.
                          Most of the blueprints functionalities are now also available from C++ but can also be manipulated via blueprints. This means that you can create a game mode using directly the c++ LeapCharacter class in the settings. The VRController will come later.
                          The only blueprints really necessary to conform with the "best practices" is the animation blueprint.
                          And since I'm lazy, I preferred adding custom animation nodes than connecting each bones of each fingers to all the nodes... I have to say, I'm impressed by your patience to make it even all look nice in there. :P

                          More information in the commit message: Commit details

                          There's still some work to do, but it should allow both the blueprint and c++ ways to coexist at each level.
                          Released:
                          [Plugin] Custom Gesture Recognition
                          [Plugin][Example] Java in UE4 - Text extraction via Tika
                          [Plugin] Cover Generator

                          GitHub Link

                          Comment


                            Originally posted by Deams View Post
                            Finally had a bit of time between the exams to work on this.


                            It seems that the only problem there for the leap motion plugin was the lack of the export macro on the classes or specific methods.
                            Might be the same for the Zip plugin.



                            It seems that best practices are fuzzy for many people on many topics...
                            Hopefully, more will pop up in the documentation.

                            In the meantime, I decided to keep going into the conversion to C++.
                            In short, it works but needs to be finished.
                            Most of the blueprints functionalities are now also available from C++ but can also be manipulated via blueprints. This means that you can create a game mode using directly the c++ LeapCharacter class in the settings. The VRController will come later.
                            The only blueprints really necessary to conform with the "best practices" is the animation blueprint.
                            And since I'm lazy, I preferred adding custom animation nodes than connecting each bones of each fingers to all the nodes... I have to say, I'm impressed by your patience to make it even all look nice in there. :P

                            More information in the commit message: Commit details

                            There's still some work to do, but it should allow both the blueprint and c++ ways to coexist at each level.
                            Just had a look at the commit, that's a beefy commit, great work!

                            I should have a chance to play with it this weekend, always wanted a better understanding of setting up IK/animation in c++ and I think this is a great resource to brush up on that department.


                            Long term thoughts:
                            In the end I think convenience setups such as rigged characters should be abstracted away from the hardware plugins themselves. A full 'abstract skeleton' target could be used that can take input from multiple devices and merge the information by the best data available. From that skeleton you would then extract the information and forward it to any output node e.g. a convenience rigged character. Then whether you use just floating rigged hands, poly models or a full body character you would have the same backend and you wouldn't need to change your rigging for new hardware inputs and you would now support mixed inputs. In my mind this is something akin to a Body State Plugin. You could further expand this by adding some smart class to handle skeleton re-targeting, which could automatically match various skeletal meshes to the abstract one and perhaps allow for almost a one click change to swap any rigged character to support a new skeletal mesh you want to replicate your movement. That would be the ideal setup in my mind.

                            The hard part of this is figuring out a way in which plugins such as this one can forward input automatically to a target that may not be there (unless something like this idea makes it into the engine and we can depend on it existing).
                            Plugins: GES - Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo

                            Comment


                              Originally posted by getnamo View Post
                              Just had a look at the commit, that's a beefy commit, great work!

                              I should have a chance to play with it this weekend, always wanted a better understanding of setting up IK/animation in c++ and I think this is a great resource to brush up on that department.


                              Long term thoughts:
                              In the end I think convenience setups such as rigged characters should be abstracted away from the hardware plugins themselves. A full 'abstract skeleton' target could be used that can take input from multiple devices and merge the information by the best data available. From that skeleton you would then extract the information and forward it to any output node e.g. a convenience rigged character. Then whether you use just floating rigged hands, poly models or a full body character you would have the same backend and you wouldn't need to change your rigging for new hardware inputs and you would now support mixed inputs. In my mind this is something akin to a Body State Plugin. You could further expand this by adding some smart class to handle skeleton re-targeting, which could automatically match various skeletal meshes to the abstract one and perhaps allow for almost a one click change to swap any rigged character to support a new skeletal mesh you want to replicate your movement. That would be the ideal setup in my mind.

                              The hard part of this is figuring out a way in which plugins such as this one can forward input automatically to a target that may not be there (unless something like this idea makes it into the engine and we can depend on it existing).
                              We might see such an interface happen in the near future.
                              There's quite a lot going on around VR, and I understand that teams like Epic have other priorities than body rigging for now.
                              But it could make quite an interesting project during my thesis if I still have time after the original topic.

                              Have you had the time to take a look at how I took care of the fingers bones?
                              Code is still a bit messy since it was my first Unreal Engine coding session, but I could clean it up and comment more seriously if you have any use for it. Same for the general c++ implementation, if it could somehow be useful, we can discuss the architecture to integrate it to the master.

                              By the way, have you played with the "image-hand" that Leap integrated to their unity3D asset?
                              I didn't see it in their UE4 plugin.
                              Released:
                              [Plugin] Custom Gesture Recognition
                              [Plugin][Example] Java in UE4 - Text extraction via Tika
                              [Plugin] Cover Generator

                              GitHub Link

                              Comment


                                Originally posted by Deams View Post
                                We might see such an interface happen in the near future.
                                There's quite a lot going on around VR, and I understand that teams like Epic have other priorities than body rigging for now.
                                But it could make quite an interesting project during my thesis if I still have time after the original topic.

                                Have you had the time to take a look at how I took care of the fingers bones?
                                Code is still a bit messy since it was my first Unreal Engine coding session, but I could clean it up and comment more seriously if you have any use for it. Same for the general c++ implementation, if it could somehow be useful, we can discuss the architecture to integrate it to the master.
                                I like the way you handle as much as possible in C++ and your overall structure is good. I think C++ anim blueprint nodes are the way to go allowing for cleaner setups downstream if we can handle input diversity well (orientation only FK (e.g. multiple IMUs) vs positional tracking IK (e.g. motion controllers)). The plugin as it is, is due for a refactoring and I would love to incorporate these changes as well start the Body State separation so all input plugins could target the same animation chain. I'll pm you if you're interested in helping/carrying this further.

                                The end goal in my mind would be to allow a developer to select a skeletal mesh with humanoid like rig and just select e.g. map Body State to this skeleton and have the plugin do all the hard work of re-targeting if necessary and switch animation chain to handle the type data available (do we have orientation only? Hands only or full body? Mixed inputs? which one is more accurate for this frame?).


                                Originally posted by Deams View Post
                                By the way, have you played with the "image-hand" that Leap integrated to their unity3D asset?
                                I didn't see it in their UE4 plugin.
                                I have and this plugin's UE version is in the works as we speak. It's fairly close to being ready, but there are quite a few changes to get right and given the fully deferred renderer of UE4, it may not look exactly the same. One of the non obvious aspects of image hands is that you have to use the leap IPD (40mm) for the hand look and interact with the world as you would expect which makes everything feel a bit bigger.
                                Plugins: GES - Node.js - TensorFlow - Socket.io Client - ZipUtility - Leap Motion - Hydra - Myo

                                Comment

                                Working...
                                X