Announcement

Collapse
No announcement yet.

OVRLipSync Plugin for UE4

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    [PLUGIN] OVRLipSync Plugin for UE4

    OBSOLETE! USE THE OFFICIAL OCULUS VERSION HERE: https://developer.oculus.com/downloa...ipsync-unreal/

    OFFICIAL DOCUMENTATION: https://developer.oculus.com/documen...ipsync-unreal/

    --------------------------------


    Old thread..............




    Example Project + Plugin Repo:
    https://github.com/ChairGraveyard/ovrlipsync-example

    Plugin Only Repo:
    https://github.com/ChairGraveyard/ovrlipsync-ue4

    Updates

    Updated to windywang's version with fixed code and support for the image based visemes.



    After messing with PocketSphinx's phoneme recognition and not getting the results I wanted, I looked back into porting Oculus' LipSync plugin for Unity over to UE4, since the Unity plugin is just a wrapper for a DLL and some examples of how to use it.

    I'm happy to report that I've got a basic version of the OVRLipSync plugin working in UE4, and it's ready for people to use. I've gone ahead and made an example project to show how to use the UE4 version of the plugin (quite straightforward, see example images below). The project has an example mesh to see it in action, and should work out of the box.

    For those that just want a quick rundown on how to use it without downloading the example project, here are some screenshots of the VisemeGenerationActor derived blueprint class.

    VisemeGeneratActor event graph setup:



    SetMorphTargets function:



    MorphIdxToNames array:



    As long as the mesh has the appropriate morphs as listed above, it will work decently well. I'm sure there are things I'm doing wrong, and bug reports are welcome!
    Last edited by n00854180t; 10-06-2018, 03:02 PM.
    Storyteller - An immersive VR audiobook player

    Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

    #2
    Hey!

    Nice work!

    I downloaded the plugin and example file, but strangely she doesn't do anything, when I hit play on the example map. Did I miss something?
    Got a task for me? Then check this out:
    https://forums.unrealengine.com/comm...ple-skilllsets

    Main channel:
    https://www.youtube.com/channel/UCGC...o9s3YcA/videos

    Comment


      #3
      Originally posted by DarkGodsLair View Post
      Hey!

      Nice work!

      I downloaded the plugin and example file, but strangely she doesn't do anything, when I hit play on the example map. Did I miss something?
      Speak into your mic! :P
      Storyteller - An immersive VR audiobook player

      Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

      Comment


        #4
        I did try it, but it just won't work. I also deactivated all things besides my mic, which is an USB mic. Are there any other requirements, than the plugin?

        Would be nice if other users could check, if they have the same problem. I also tried standalone mode, new editor window and selected viewport. I'm running it on Windows 10.
        Got a task for me? Then check this out:
        https://forums.unrealengine.com/comm...ple-skilllsets

        Main channel:
        https://www.youtube.com/channel/UCGC...o9s3YcA/videos

        Comment


          #5
          Originally posted by DarkGodsLair View Post
          I did try it, but it just won't work. I also deactivated all things besides my mic, which is an USB mic. Are there any other requirements, than the plugin?

          Would be nice if other users could check, if they have the same problem. I also tried standalone mode, new editor window and selected viewport. I'm running it on Windows 10.
          Not sure then, I'll have to debug it.
          Storyteller - An immersive VR audiobook player

          Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

          Comment


            #6
            [MENTION=172129]DarkGodsLair[/MENTION] - DOH! Really really stupid of me, I know why it's not working! I just checked it out on my end and saw the error message "Failed to create audio device". That's when it hit me: Voice capture isn't enabled in the ini, doh.

            The repo has now been updated to fix this. If you don't want to re-download, you can simply go into the Config folder and edit DefaultEngine.ini and add these lines:

            Code:
            [Voice]
            bEnabled=true
            I should note that even though that will get it working, the results aren't really up to snuff with the official Unity version just yet. Basically, it's not entirely ready for production.
            Storyteller - An immersive VR audiobook player

            Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

            Comment


              #7
              Haha, I know exactly, how you feel. Sometimes I am so braindead that I even forget the simplest of things .

              Thanks for looking into it. Will test it right away. You stated already above that this is a really basic implementation, so I don't mind the limitations .

              EDIT: I can confirm that it is working .
              Last edited by DarkGodsLair; 06-09-2016, 09:51 PM.
              Got a task for me? Then check this out:
              https://forums.unrealengine.com/comm...ple-skilllsets

              Main channel:
              https://www.youtube.com/channel/UCGC...o9s3YcA/videos

              Comment


                #8
                There's something actually going wrong with it, despite it working right now - I think it's due to not utilizing the frame numbers/frame delays, which from my tests were returning weird numbers, so I just skipped it. There's also the issue that it doesn't go back to silence/closed mouth sometimes after speaking.

                Edit: Gotta be something else going on, the frame numbers I don't think are the main issue.
                Last edited by n00854180t; 06-09-2016, 10:27 PM.
                Storyteller - An immersive VR audiobook player

                Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

                Comment


                  #9
                  I kind of got the feeling that the mic input always triggers/activates the lip sync, so the motion of the mouth doesn't stop. Maybe imposing a mic input threshold would get rid of it, but I am mainly a blueprint guy, so take this with a grain of salt .
                  Got a task for me? Then check this out:
                  https://forums.unrealengine.com/comm...ple-skilllsets

                  Main channel:
                  https://www.youtube.com/channel/UCGC...o9s3YcA/videos

                  Comment


                    #10
                    Hey man. Great to see this come out
                    I`ll play around with this over the weekend.
                    If you are having accuracy issues, perhaps you could feed a constant audio recording through as input to both Unity/UE, to log timestamps.
                    At least then, you'd be able to visualize the differences easily. Just an idea.
                    Last edited by ShaneC; 06-10-2016, 01:35 AM.

                    Comment


                      #11
                      I think I know what's going on - I suspect it's to do with setting the main audiobuffer to a static size, and it's going over and reading through the random bytes that don't have data. I just have to test and upload the new version, probably tomorrow. Initially I was getting better results, more in keeping with the Unity version, so I think reverting that change should do the trick, we'll see.
                      Last edited by n00854180t; 06-10-2016, 02:42 AM.
                      Storyteller - An immersive VR audiobook player

                      Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

                      Comment


                        #12
                        Pretty cool stuff!

                        Thanks for sharing

                        Edit: I was thinking to adapt the Kite Boy in order to be used as a test subject for the plugin and share it once the morphs are done. Is there a sample mesh for Unity available? Just to have a look at the morph setup and avoid weird results
                        Last edited by Enter Reality; 06-10-2016, 03:10 AM.
                        ENTER REALITY

                        VR Solutions

                        Contact us for more informations

                        Comment


                          #13
                          There's a sample mesh in the example project right now (Oculus' one from their original Unity plugin). On the Unity asset store, I know the Morph3D characters have the appropriate morphs already set up, but you need to go in and delete the secondary skeleton for the eyes before UE4 will import the FBX files for those (there are free "lite" versions of those characters).
                          Storyteller - An immersive VR audiobook player

                          Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

                          Comment


                            #14
                            hi, does this work if I want to use animations instead of morph targets?
                            Follow me on Twitter!
                            Developer of Elium - Prison Escape
                            Localized IBL for UE4

                            Comment


                              #15
                              It just spits out an array of viseme values - how to use them is up to you, animations can certainly work.
                              Storyteller - An immersive VR audiobook player

                              Dungeon Survival - WIP First person dungeon crawler with a focus on survival and environmental gameplay ala roguelikes

                              Comment

                              Working...
                              X