Any NPC - Character Customization Tool for UE4 - Featuring shareable Lipsync

Community Showcase Game Development
1

Hi guys,

Here is a sneak peek of the tool I’m currently working on.

Any NPC is a tool developed using Blueprints which feature the ability to create and customize a humanoid character.

Is completely Blend Shapes based and currently it features a couple of different body types ( more to come ).

Also one of the most usefull features ( especially for RPGs ) is the ability to share lipsync animation between different characters, which saves tons of time.

The lipsync is driven by 38 blend shapes based on the IPA phonemes setup.
The lipsync itself is not baked on the character skeleton but its driven externally.

What you see is the first build/test of this tool, also the lipsync itself is not 100% correct ( 10 minutes cleanup is more then enough ).

PS: The base mesh is the one from the plugin Species from Exocortex, its used just for testing purposes; the lipsync animation is done externally via a separate rig, its NOT created inside UE4 ( just to be clear ).

If you have any questions feel free to post it here

More features will come soon ( female version, zombies, military, setup for Faceshift and Faceware facial animation and so on ) :slight_smile:

https://youtube.com/watch?v=3CLuZJFCC7Q

2

Very nice Morphing System. What’s the plan for applying clothing/armor?

3

Hi TechLord,
I’m thinking about that…it would be a matter of adding new morph to the existing ones and setting the visibility as well…
I’m also working on a custom GUI for the customization itself and a random crowd generation with paths, but I still need to take a look at that stuff, since I never had experience with crowds, but I’ll give it a try.

Probably in the near future there will be:

  • Female version
  • Zombies ( well, its the HOT stuff right now :smiley: )
  • Military
  • Medieval ( so armour )
  • Casual
  • Business
4

Hi Nicolas,

Recently viewed a video for Exocortex Species. The blending methodology appears to be similar to Daz3D Genesis. Using Target Morphs to customize a character’s base shape is a awesome form of customization, and I would like to see the technique applied to non-bipedal creatures, weapons, vehicles, furniture, trees, structures, etc.

Daz3D offers Wearables (Clothes and Accessories) to auto-fit their Genesis models no matter proportion, so I speculate the clothing is also using blendshapes/morph-targets. I’ve seen interchangeable clothing applied in two ways:1) overlay to the base body mesh, 2) interchangeable mesh replacement on a Modular Pawn. Perhaps Modular Parts with Morph-Target Deformation offers the most flexible solution?

The Modular Pawn is a popular method of creating customizable Characters, and I believe using morph-target will be equally as popular (sameeek4 is also developing a Character Customization System using Morph-Targets). I’m developing a method of procedural generation that can be used to customize entities. It would be nice to combine all of these techniques into a single tool.

Adding a GUI is a good call. Can you elaborate on what you mean by custom? No UMG? I’m personally taking an agnostic approach, providing only a interface to communicate to a UI for in-game customization. Overall, I think a tool such this needs an Asset Supply to be successful and I believe this is why DAZ3D Genesis is so popular with their massive catalogue of Wearables. I don’t know if you have intention of selling on the Marketplace, but, I see huge potential with providing Clothes and Accessories for Any NPC.

5

I’ve actually downloaded the Daz3D package and I will take a look at it in the near future, but yes, the tech is the same, is all blend shape based.

I’m testing a kind of autofit tool using Softimage ICE ( very similar to the Blueprint system in UE4 ) which allows for geometry manipulation easily without any coding, and for me is a huge advantage.
This tool will allow to easily transfer any kind of geometry ( for now human only ) from one to another so that I can create Blend Shapes in seconds rather then spending time modeling and adjusting everything.

For clothing I was looking at Marvelous designer and also Daz3D…since I would like to include also Apex clothing support I need to study a bit better what’s going on with it and see what could be the best solution for clothes.

I really would like to explore the differences between the modular pawn setup and a Blend Shapes setup, since I guess one would be more memory heavy then the other…in case I will re-write the Blueprint setup or find an alternative solution.

Daz3D would be a very good candidate because of the huge amount of stuff available, but if I remember correctly there is a fee to pay for game development ( 500$ ) and right now I cannot invest this money.
Sorry, my bad, the GUI will be done via UMG, I need to work a bit on the graphics and figure out how to setup a random NPC generator which will include beard, clothes, glasses, hats, hairstyle without ******** the npc itself, menas without having conflicting results.

I’m actually thinking to sell Any NPC as a tool on the marketplace once I figured out all the stuff mentioned above, but I already saw that the models available on the martekplace are quite expensive…and since this toll would solve quite lots of problems in terms of content creation I guess that I will propose this tool at 50$ more or less…also consider that I will include LODs as well, which done in Softimage takes 5 seconds ( god bless Softimage and thanks Autodesk for murder this fantastic software )

Also the Lipsync tool will be sold separately, but it can be easily intergrated into Any NPC, since the base Blend Shape is the same…main point for the Lipsync tool is to provide very cheap lipsync animation ( 1$ for each word ) which would be perfect for RPGs or open world games.
The tool is completely agnostic to the rig itself and it rely on Blend Shapes as well, and I’m expanding it so that it can be adapted to already available facial animation software, such as Faceshift, Faceware and so on.

GOLEM looks great! really nice project! what about skin weights? I’m curious to see what can be done with different parts :slight_smile:

Sameek4 is doing a great job, he’s a really talented guy and also his other tools ( road intersection tool ) looks fantastic!

6

I learn something new with each one of your post. Marvelous Designer looks amazing but not so sure about the Monthly Subscription $59.99/Annual Subscription $360.00. My Subscription Budget is maxed out and not a big fan of a Subscription for Tools. I’d rather pay the flat Lifetime $550.00. I see export options for obj and fbx. I foresee additional work involved to fit garments to base mesh and adding morph-targets, but, overall could be valuable in producing an infinite assortment of marketable Clothing Packs for Any NPC in the Marketplace. IMO, Any NPC would sell better in the Marketplace with a decent number of Clothing Extras available for purchase.

I’m not certain if there’s a standard for blendshapes/morph-targets when submitting to the Marketplace, but, there are considerations for the Skeletal Rig. Does the Any NPC BaseMesh use the Epic Biped Skeleton? By the way, I noticed in the Daz3D Morphing Vid all of the morphs are Biped, but, I could easily visualize Bipeds morphing into various Quadrupeds/Bird-like Creatures with additional blends extending vertebra and fore-limbs. Possibly an edge up for Any NPC on the competition:)

I have yet to use Daz3D Genesis (I need to do this to at some point)…from what I’ve seen in their videos the base mesh is singular and Morph Sets and Clothing are modular. That’s a pretty slick design and the Genesis SuperSuit is inspiration for other type of morphing clothing and accessories. Marvelous Designer + Daz3D SuperSuit = MightyMorphingPowerSuitDesigner? Add AutoFit + Apex Cloth support + In-game Editing/Customizing Interface and there’s a Tool worth a subscription…hmmm

In a Modular Pawn, separate meshes represent different shapes that are swapped out entirely. I fancy the idea of a hybrid system using modular morphing base mesh pieces. In theory, should allow greater flexibility and variation (ie: different morph sets per piece). The option to use a singular base should be available by default. Again, I’m looking for possible edge up for Any NPC on the competition.

Lip Syncing and facial articulation is sorta of an expectation for characters in next gen games. I’ve seen it used in several games with narrative not just RPGs. I would vote for its integration into Any NPC more so as Developer expectation. Could very well be a highlight selling point.

Thanks. A GOLEM Entity Body is the conglomeration of hundreds instanced static meshes. There is no skin.

7

Thanks for the comment

Daz3D could very be the base for all the Any NPC blends, but still the license for game development is not perfectly clear…I guess I’ll send an email directly to them explaining what I need to do and see the response…anyway the price is much better then the 3d scans of people, which are bloody expensive!

Yes, the skeletal rig is based on the ART plugin from Epic, and I intend to add some more features to the base mesh, such as proxy bones for clothes and fat physic movement ( inspired by the amazing Judith Simantov The Last of Us tech demonstration and the brand new Uncharted 4 tech demo, so there will be a couple of variations for the skeletal rig itself in order to use the proper one for each character…also breast physics ( ala Witcher 2 ) will be implemented :smiley:

Still lot of stuff to check and still some RnD is necessary, but the base layer is now done, so it’ll be just a matter of adding content and make it the best and more efficient way possible.

Facial animation is something which I really would like to work in the future, and for now lipsync is just the beginning simply because the results are pretty good and, as said, facial animation is quite expensive, so I will also add a complete expression set for the face ( blend shape based, but I already developed an additional system which uses bones as well ) in order to have proper facial expressions with amazing results…
Also I’m re-learning Zbrush ( had to switch from Softimage to Maya, its still painfull ) and currently the next demo will be way more polished and will show off also a very nice trick, which is Dynamic normal map based on expressions
The video looks like ****, but it gives the idea…also the fast way to achieve that is via Blend Shapes, but I also found a method to use it with traditional joint based facial rig.

Lots of stuff to do, lots of things to test and lots of stuff to learn…hopefully by combining all of those things something great will come out :slight_smile:

No skin on Golem? interesting approach, I’m really curious to see the progress on that :slight_smile:

8

I applaud you sir…absolutely wonderful

9

Nicolas I viewed several of your Youtubes. You obviously possess significant experience with 3D Modeling & Animation. These are Skillsets I desire to improve on, being a code-centric gamedev. My lack of 3D Characters for my FP Hack & Slash, and skill to create them myself is actually what inspired the GOLEM ECS. This discussion has also been inspirational and informative. I’m currently using Shapes Components: Box, Capsule, Sphere in the GOLEM Fragment Attachment process. The addition of Static Mesh and Morph-enabled Mesh components to this process would provided even greater variation.

I viewed the videos on the Last Of Us Tech and Uncharted Tech, and their techniques look incredibly complex (and intimidating). I hope there are easier ways to pull this level of realism in UE4 for much smaller teams. I have a vague understanding of proxy bones and how they can be applied to clothing from working with the UE4 Owen Skeletal Asset. Perhaps UDK’s workflow concept of a Master Skeleton is applicable?

My thoughts on Clothing for a Morph-enabled Entity is a 100% Apex-driven Cloth overlay that physically reacts to the Character’s base mesh surface. In fact, the more I think about this subject, the more I see value in a in-game garment/wearables auto-fit & customization system. In a majority of games, humanbeings are often clothed, there is rarely a base mesh displayed. In some cases creatures are wearing armor etc. Such as system could be applied to any type of character, no matter the construction methodology used.

10

Thanks for the kind words TechLord :slight_smile:

Well, not being able to code…well anything actually was a critical point to switch to UE4 in order to develop some of the ideas which without coding would be impossible for me to create.
To be honest the art and polish of the 3d content is something that, regarding gameplay, could be easily added later, simply because the highlight would be the Blueprint/coding itself which allows to do a specific thing, so if you want you could begin start learning Maya/Max to improve your skills, but for testing purpose I would storngly suggest to use free asset from the store or create something basic, you don’t need always shiny stuff to wow people :wink:

A genius on this forum ( Rama) create usually small gameplay videos with his new stuff, the art is “normal” for what he need to show and it does the job perfectly :slight_smile:

Regarding the Master Skeleton this is the way I’m achieving small variations on the skin and proxy bones themself, means that I will add a couple of extra bones in some areas ( breast, belly, biceps ) using the ART plugin, which is pretty handy and not so impossible to use…it’ll be just a matter of having proper skin weighting which can be adapted between the morph variation.
Also regadring the proxy bones, a good example of cloth behaviour can be seen inside the Matinee fighting sequence, if you export one of the main dudes you can see that the rig itself has proxy bones all over the cloth ( that kind of skirt, I cannot described it better :smiley: ) and I think that the guys at Epic used the nCloth plugin ( or Syflex ) inside Maya and bake the simulation results on the cloth itself, so basically the bones are completely passive and they’re driven by the simulation.

The only problem which I didn’t test right now regaring the Apex cloth is that you should have a proxy mesh which will behave as a collision for the cloth itself on the character, and as far as I remember those are not morphable, or better yet I don’t know how to access those meshes when the Skeletal rig is imported into UE4…is something that I need to clarify both in Maya and UE4 and I need to do some research on the forum itself, so eventually I’ll find what I’m looking for.

Quick update on Any NPC:

  • Added Scale for root bones in order to set character height ( from child to giant )
  • Starting UMG GUI menu :slight_smile:
  • Improving thelipsync and phoneme shapes in order to achieve better results
  • Adding expressions for Faceshift and Faceware setup ( this will take a bit longer )
  • Cloth setup with dynamics

New video in a couple of days

11

Small update:
Character scale while applying blend shapes

Optionally ( need to check some stuff, but 99.9% sure ) I’ll include the possibility to scale single bones also so that you can create a non human shape directly inside the Unreal Engine editor.

13

Hey Nicolas,

The morphing is very smooth. Its a little tough to gauge the scaling without a reference. Does scaling the single bones equate to –> morphing into various Quadrupeds/Bird-like Creatures with additional blends extending vertebra and fore-limbs?

14

Yep,

About the height scale I didn’t have the time to setup the UMG, still learning how to make it work as I want :wink:

Anyway yes, since you have the ability to access bones transforms you’ll be able to scale/rotate what you want, of course I need to expose those values for the user to modify them, but overall if you want to create a warewolf/quadruped or non human you’ll be able to do that.
Alternatively, or as an additional option, I’ll probably include a different skeletal rig ( probably more then one ) for particular cases ( tail for quadrupeds, ponytail for hair, bpnes for chains for armor, and so on ).

15

Another small update showing the blend shapes in action controlled by the user.

The main reason why I didn’t create a UMG menu with sliders ( at least for now ) is that UMG works only when you’re in Play mode, so it becomes annoying to remember every time to hit the play button so you can modify the character in the scene.
Using this method you’re able to modify the character in both “non-play” mode and in play mode :slight_smile:

PS: Female base character coming soon :slight_smile:

16

WIP of the female version.

The Blueprint will be the same as the male version since its already setup, but it will include more features…

Next up, UMG customization slider/setup

Female_BS.JPG
Female_BS.JPG919×850 58.6 KB

17

Female base version added!

UMG next :slight_smile:

18

isn’t this a little too overkill for most games though? it’d certainly look good but a top-down RPG would be unplayable on a lower spec PC. heck, even in a first person shooter using LODs for characters in the distance would be too restrictive (i.e. useless) if you make everything Apex-cloth dependant.

Nicolas: I also wonder what your plans are for cloth. using morph targets is nice and all, but requiring to make all morph targets for every piece of clothing and armor means the amount of work you have to do will grow exponentially.
and the same will happen with hairstyles and facial hair once you add those

19

@Chosker. In retrospect, Apex is a process hog, perhaps it would not be used real-time, but at design-time to generate mesh shape/contour and deformation information for animation playback. The use of optimizations such as Level Of Detail, PBR Materials, etc would be applicable. What I visualize is a in-game Marvelous Designer-ish Editing System and with latest processing power, I believe its totally feasible.

@Nicolas, looking good. I’m also learning/designing a GUI with the UMG Tutorial. The Tutorial doesn’t mention Sliders:( I desire to use them for modular morphing meshes to assemble melee weapons, furniture, doors, and props. I’ll most likely simulate a slider with mouse scroll wheel. BTW, I have no idea how to get a morph-able skeletal mesh into UE4 and control the morph. Do you have any Tutorials on this?

20

Hi Chosker, if you take a look at the free downloadable “fight scene” from the launcher you can see that they’re not using apex cloth for simulation, but rather joint chains, I guess simulated into Maya ( with nCloth maybe? ) and the results are baked into the animation…this is an approach that can be used, but it requires to create dynamic joint chains which I’m not sure are so cheap on performance…and also you would need different skeletal rig setup in order to better simulate cloth behaviour, so I think I’m going with Apex…but I guess that its better to show directly instead of talking about it :slight_smile:
Right now I’m studying UMG, so hopefully I’ll sort everything out in the next video including a small cloth demo :wink:

Its true that the more content I have the more the amount of work will grow, but I’m using a functional workflow which allows me to build a mesh ( shirt for example ) for the base mesh of Any NPC, and then using some simple scripts I’m not literally modeling the morphs for the shirt mesh, but its kind-of automatically created.

Hi Techlord, the only tutorials available on UMG sliders I’ve found are related to color change and screen resolution, HorusHeretic have a pretty explanatory tutorials on UMG, I strongly suggest you to take a look at them :slight_smile:

Regarding your specific question, currently I’m filtering the morphs using arrays and setting the morph target by exposing the morph values on the details panel ( check my last video, you can see that I’m changing the values from 0 to 1 ); this is great for testing without having a UMG menu.
From what I’ve understood the slider creation is quite straight forward ( again, take a look at HorusHeretic channel ) and then you have to Cast it to the player, choosing “OnValueChange”, so that you basically “link” the slider to a specific morph value…take it with a bit of salt because I still haven’t tried it, so I might be wrong since I’m working on other stuff right now :slight_smile:

21

Very nice!

I am really looking forward to this :smiley: