Hi Unreal.
I am looking for a project lead to cede a new concept of facial capture in exchange for unlimited lifetime access to this tool if a savvy tech can make this proof of concept work. I am willing to cede all rights to you, so you can exploit this as your own venture.
Why? I write screenplays, I don’t have the inclination to develop tech products. I want a tool to make cg movies faster without facial capture. I want the actual actors face in a cg world.
I want a tool that indies can use for free up to two actors, one director and one DP in realtime. Your market might be charging a small fee to get access to directing 3 or more actors in realtime.
This is a new kind of facial capture with a regular webcam, but with almost no facial rigging. I’ll give you an idea of the vision.
Lets for now call it : Paint Face - Rough Concept Screenshot below.
Imagine directing your actors over a webcam on a virtual film set in Unreal. How? Send them a small realtime multiplayer build of a specific scene that hooks into their webcam. The engine tracks only six facial parameters, face tilt, face yaw, face turn, face forward/back, jaw and brow. Later add more things, but for now those parameters will do for now.
Question is how to get facial data from this? By creating a MATERIAL of the actors webcam feed placed onto the face of the cg actor in the scene.
How would this be used? A multiplayer unreal scene say of 3 actors and 1 director is sent to the participants. The actors can see the cg counterpart of each other, but can’t see the director. The director can place markers in the scene and mark out curves and lines for an actor to follow, and everyone can talk via streaming audio in realtime. The director can make himself visible when needed, so the actors can see him between takes
When it comes to recording, each station records a local copy of the mocap data that uploads to the director later, besides the realtime mocap data stream during multiplayer streaming. This way the two streams can be compared later and any holes filled due to latency issues.
Usage case: remote webcam rehearsals, auditions, short films.
Perk: If pulled off right, the nuances of acting come through because it is the webcam feed of a REAL LIFE actors face, not a cg face.
Other features: After market led array for flat lighting of actors, so an actors face can be cg lit and blended into the scene with shaders at a later stage.
Issues: getting key features of the real actor onto a simplified cg head, such as nose, brow, jaw.
Possible libraries that can handle some of the heavy work: Perhaps OpenTrack can track the head movement. Head movement could be blended with keyboard mouse movements, example, “W” pressed to move forward, mouse for BODY DIRECTION, but as Character turns his head, the CG character starts to look in direction of turn. If head keeps looking one way (ZERO = HEAD FULLY FORWARD), it starts affecting body turn, until heads stops (RETURNS TO ZERO).
Where markers are placed, when character hits a mark, a gui icon goes green, so he does not have to look at the floor. ETC. Drop me a pm for more info. I want to direct this as I would direct real actors in one of my films.
If you got a heart for the end-user and want to go full GNU GPL on this project, You are going to win me over. I don’t care much for the money. I do not want this tool to be released for the BIG STUDIO SHARKS who put VFX companies out for business. I do want the underdog film makers to have a cool tool they can just plug and play and start directing with any actor behind a webcam from anywhere in the world.
If you need clarity, pm!
Drop me a message to start the discussion. You heard it here first on Unreal Forums.
Regards
Human Being, Screenwriter, Indie CG Filmmaker
[
]()
FAQ
Q: Why would anyone want to build something so stupid?
A: Because it will run any any stupid generic webcam without CUDA, which means the numbers of actors you can access this way is as big as anyone who has a moderately powerful laptop, webcam and internet connection. Next question.
Q: Why would anyone want to build something so stupid?
A: How many actors can afford an iphone X? Dumb question, next.
Q: Why would anyone want to build something so stupid?
A: You’re not a film maker, you’re a programmer or a vfx artist? Next question.
Q: Why would anyone want to build something so stupid?
A: Actor access via webcam. Some film makers can tell a story this way. Next question.
Q: Why would anyone want to build something so stupid?
A: Covid-19. Next question.
Q: Why would anyone want to build something so stupid?
A. Me write stories, the then produce stories, then direct stories. Me prefer to work with real actors, but me locked in my house because of teensy weensy virus.
Q: Why would anyone want to build something so stupid?
A: Barrier to Entry is minimized. Next question.
Q: Why would anyone want to build something so stupid?
A: I will NEVER own an iphone X. Not by choice, they are just too effing expensive.
Q: Why would anyone want to build something so stupid?
A: I want ONE copy of my face firmly planted in reality. I do NOT want a copy of my face on Apples servers. That question was REALLY REALLY dumb.