Announcement

Collapse
No announcement yet.

Subtitles-based Lip Sync

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    [SUPPORT] Subtitles-based Lip Sync

    The plugin is using text information (subtitles) to generate lips sync animation for characters in real-time. Audio envelope value is used to detect silent intervals and to pause animation in this intervals and adjust future animation speed.

    Important Notes.
    1. The plugin requres new experimental audio engine (need to be enabled menually in the engine's WindowsEngine.ini or MacEngine.ini config file, details here).
    2. Only supports animation based on morph targets (blend spaces), but support of animation curves is planned in the future.
    3. Animation quality isn't as good as lipsync animation from FaceFX, to check - try executable demo.
    4. It also requires manual adjustments depending on audio assets (if they're clean or noisy, loud or quiet) - please read documentation. I'm going to improve it in a future.
    Documentation: https://drive.google.com/file/d/1GKX...JMDdzoqkk/view
    Executable Demo: https://drive.google.com/open?id=1mu...0qMOZisnDUN2O6
    Video tutorial: https://www.youtube.com/watch?v=MWsNb4kOaws
    Last edited by YuriNK; 06-24-2018, 09:28 AM.
    Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
    And random stuff at Youtube

    #2
    Great job! It is interesting to see how it works on good quality audio and morph targets!)) Is there a possibility to save this animation to anim asset?
    Regards,
    Alexander Shatalov

    Tools Programmer
    Currently working on: Allright Rig (UE4 Plugin)

    Comment


      #3
      Originally posted by aleks_allright View Post
      Is there a possibility to save this animation to anim asset?
      Interesting question. I'll try and let you know.
      Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
      And random stuff at Youtube

      Comment


        #4
        No, I can't capture morph targets animation.
        Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
        And random stuff at Youtube

        Comment


          #5
          I am still trying to get CC2 characters to work with this system - which would be a major bonus. The phonemes in CC2 are as follows, but when in UE4 the morphtargets look quite different.

          The full pipeline is described here: https://www.youtube.com/watch?v=SzXmNTFzXxU
          Last edited by Macw0lf; 04-24-2018, 04:10 AM.

          Comment


            #6
            Macw0lf
            I believe, phonemes in iClone are simply composite presets for morph targets. There is no problem for me to add a support of composite presets, but there are few problems:
            1) plugin doesn't know how lips should move for every single phoneme, and hard-coded solution specially for Character Creator would decrease flexibility for devs
            2) even if I'd like to add a special support for CC: we don't know exact morph targets presets used in iClone phonemes
            3) animating multiple morph targets in realtime is bad for perfomance

            I'll add support for composite presets anyway, but that's all. In theory, you'll be able to use my plugin with this morph targets. On practice - unlikely, because you don't know how iClone mix its morph targets to get this phonemes. The best solution IMO is to export all phonemes from iClone to fbx as an animation, then import it in a modelling software (3ds max), collapse separate meshes for each phoneme from the animation and attach this meshes as an additional morph targets to the main skeletal mesh. And, of course, remove all lips morph targets which a unncesasary for emotions.

            Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
            And random stuff at Youtube

            Comment


              #7
              Thanks, that sounds like a better plan. I was worried about loss of accuracy using the iClone morph targets.... so let me see if I can make headway with your plan.....

              Comment


                #8
                Version 1.0.2

                - composite morph targets (using multiple morph targets for one phoneme)
                - built-in emotions system (by tags in subtitles)
                Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
                And random stuff at Youtube

                Comment


                  #9
                  Hey there Yuri,

                  What engine is used to produce the phoneme mapping from the audio input?

                  Thanks!

                  Comment


                    #10
                    Originally posted by calembendell View Post
                    Hey there Yuri,

                    What engine is used to produce the phoneme mapping from the audio input?

                    Thanks!
                    No-no, there is no audio recognition engine. The plugin just 'reads' provided subtitles.
                    Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
                    And random stuff at Youtube

                    Comment


                      #11
                      Hi Yuri, I think the lip sync animation in your demo is brilliant

                      Are you planning on adding support for bone based face rigs?

                      (I'm currently using daz 3d characters with a blender based rigify body rig and pitchipoy face rig).

                      Also, could you post a youtube showing the workflow involved in creating these animations with your plugin?

                      Comment


                        #12
                        Hello, I have purchased your plugin, but how to do microphone audio recording in real time,Can you provide the project files in the video?

                        Comment


                          #13
                          Originally posted by ljz0713 View Post
                          Hello, I have purchased your plugin, but how to do microphone audio recording in real time,Can you provide the project files in the video?
                          It doesn't work with microphone input, because it requires subtitles to work. You should try to refund, probably.
                          For real-time lipsync with microphone try Oculus plugin (https://forums.unrealengine.com/comm...plugin-for-ue4)
                          Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
                          And random stuff at Youtube

                          Comment


                            #14
                            Originally posted by ioFlow Studios View Post
                            Hi Yuri, I think the lip sync animation in your demo is brilliant
                            Are you planning on adding support for bone based face rigs?
                            (I'm currently using daz 3d characters with a blender based rigify body rig and pitchipoy face rig).
                            Also, could you post a youtube showing the workflow involved in creating these animations with your plugin?
                            1. Bones - not soon.
                            2. Yes, I'll make a video in a few days.
                            Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
                            And random stuff at Youtube

                            Comment


                              #15
                              Originally posted by ioFlow Studios View Post
                              Also, could you post a youtube showing the workflow involved in creating these animations with your plugin?
                              Tutorial: https://www.youtube.com/watch?v=MWsNb4kOaws
                              I use Daz3D-based head (Genesis 3), but with morph targets.
                              Marketplace: Vive Mocap Kit / VR IK Body Solver / Subtitles to LipSync / Ultimate Dialogue System with Character Animation
                              And random stuff at Youtube

                              Comment

                              Working...
                              X