Lipsync Tool

Distribution UE Marketplace
1

Hi all,

It’s time to give something back to the community which I’m glad to see is always willing to help everyone very quickly and efficiently, so this is my small contribution to the developers who wants to add something to their game.

So, what’s all this about?

Introducing the Lipsync Tool, and as you may already guess, it’s a tool that allows you to create lipsync animation for your characters!

Not to be confused with the way other lipsync tools work, meaning that this tool do not analyze the audio/text and gives you phonemes, instead this tool gives you a library of ( english ) animations, which are basically spoken words ( more or less 58000 animations ) for you to apply to your character.

How does it work?

You need to create an Animation Montage where you basically drag&drop the words/animations you need and string them accordingly to create whatever sentence you want, based on the audio you have, and since you can also add audio to the Notifies you have a realtime audio/video feedback of the character moving his/her lips while talking.
What you might need to tweak is the play rate of each animation, so that the word is spoken slower/faster, and it’ll match properly with the audio.

It is a one click solution? What do I need to do to make it work with my character?

This system is heavily inspired by the tool used in the good old Source Engine, meaning that you need to create a set of 19 mouth shapes ( using Maya/Max/Whatever ) which corresponds to a set of phonemes needed for the animation to be applied to your character.
In the test I did I’m using Blend Shapes for the phonemes, but I’m planning to also support joints in the future.
A custom Blueprint will be given so that the only thing you need to do inside UE4 will be to add the reference to your character so that you can use the animations.

How much it’ll cost?

5$? 900$? 1500$? It’ll be completely FREE! Why? Because will still be rough around the edges and lots of tweaks could be made to this tool, but that will require coding, which is something I’m not familiar at all, so that’s why I’ll release this tool completely free.
However if you’ll find this tool usefull you’ll have the possibility to donate whatever amount you want, since I’ll be using that money for hardware upgrade and other custom solutions for UE4 :slight_smile:

Room for improvement

As said above, a customized UI inside Persona would be great, with the ability to:

  • Filter all the words you want to use
  • Possibly drag&drop all of them in the anim montage so that they are layered accordingly
  • Something else for sure :smiley:

This will require someone who can code that, so I’ll leave to the community to refine this tool in order to be more user-friendly :wink:

When it’ll be available?

I’m planning to release the tool probably around the end of the year or early January ( if everything goes according to plan ).

If you want to know more, have any questions feel free to ask :slight_smile:

Cheers

Nicolas

2

Very nice that you release it for free. I use Blender and shapekeys too and love it.

3

YES! I would love this.

Edit: Would we get access to the raw files or no?

That said - If you can make it anything like The Phoneme Editor in the Source engine that would be a god send.

4

Sounds awesome Nicolas! I’m definitely interested.

5

You mean the FBX animations from Maya? yep, I think I’m going to include those as a separate download, but I don’t know how much help they can be since I’m using a custom rig for the lipsync animations in Maya…but I guess that I can also include the morph target animation curves of the single phonemes as an alternative, so that they can be tweaked :wink:
Regarding the audio I’m not 100% sure if I can also upload the single clips because they’re from an online text-to-speech converter, and since I’ll be accepting donation I think that I might have issue with copyrights so I think I need to ask…or directly give you the link of the translator I use and you can get the audio there :wink:

For the phoneme tool itself, well, that require coding, understanding of the Microsoft Sapi, patience and time, and I don’t have those! That’s why this will be a free release for the developers, mostly because isn’t a one click solution tool, but more of a “sentence builder”…well, in the end is exactly like creating an animation montage with a bit of tweaking on the animation themself :slight_smile:

@All: Next week I’ll upload a video showing how the system will work and then I’ll start building the words library and export them, and that is the most time consuming part :slight_smile:

6

Interested :wink:

7

Sounds interesting!

8

Very cool.
When you build, could you build these into, to make own stuff?

563×696 88.8 KB

9

So its like, i pass the string and it would animate face?
Sounds amazing!

10

I do the same thing now, but i use blender + papagayo and a timelinebased setup, but that is so complicated.
It would be nice to pin some notifiers onto a cue/audio file or similar, to drive the shape keys directly, like you do in papagayo.
I do not like to prebake the words in blender and export them seperate, into ue.
I could use it for basic stuff, like this.

11

@Luftbauch: Sorry, no request, but I guess that using blueprints you can combine the existing morph targets to create the ones you linked, and use them as you wish :wink:

@CriErr: As long as your character have those morph targets, which allow the sharing of those animations, you drag&drop the animation on the Montage in sequence, and that will create the lipsync animation, similar to what you see here

12

Little breakdown on how the system will work in UE4

If you have any questions please let me know

13

I am definetely interested in lipsync tool

14

Sneak Peek 2!

15

A better way is definitely needed for lip syncing in Unreal.

Not sure which approach will suit me best for my project. I want to have multiple characters holding conversations and film it in Unreal 4. Think sports talk show, in studio and “live” on scene.

The main reason Unreal is all the info out there, the assets already available, etc, and especially the new Sequencer. But I would have a TON of dialog to animate on a weekly basis, and so far, it looks like what’s available is just too much work.

I’ve done some testing with MakeHuman characters in Blender using Quick Talk and Papagayo. I’ve managed to get the animations adjusted and working decent in Blender, so technically, it should work in Unreal. But the work involved to get to the point of getting characters into Unreal is just too much for any kind of regular effort. A one off movie is one thing, but a regular show? I might need to rethink things, especially since my background is in 3d modeling and texturing static stuff and 2d art! I’m no coder at all, and even working with Unreal animations is new to me.

My other limitation is typical indie; no cash, free/open source or go home. The intent of the project is to generate some funds maybe through ad space once I have a product, but I have no working capital up front.

Basically, I want to have characters talking in Unreal and film it. Dare I ask, how hard can that be?