The Android app doesn’t handle client disconnection perfectly, you might have to restart the app if you don’t get any incoming data.
The project requires the free ‘TCP Socket plugin’
Biggest limitation right now is the ability to convert the incoming data into blend weights. It can be fairly complex and advanced to put together a robust system to generate this data. Currently i’m simply interpolation between 0.0 and 1.0 based on the average distance vertice to vertice of a subset of vertices defined per blendshape, between the current position of a vertice and its blendshape Max position.
Unreal Engine Project to read this stream of data, generate blend weights, and hook it into LiveLink:
Hello Unreal Community,
Opening a new topic regarding my recent effort (24H literally) to get Android Face Capture going. Goal is to create an android alternative to the Apple FaceAR solution, with MetaHumans compatibility .
I am starting from scratch and everything is gonna be open source, starting with the Android App which send ArCore face data to Unreal Engine trough TCP (free TCP Socket Plugin from Unreal).
This is what i have so far (recorded galaxy s9):
quote: "wow wow wow wow wow wow"
Hi, thank you for the detailed explanation, there are two arrays in the AndroidBlendShape asset, Vertices to Control and Vertices Max Mvt, both have hundreds of elements. I think it’s quite difficult to fill these arrays by hand, so I am wondering whether you have any utilities, thank you.
On the default demo map there’s a blueprint representing the face points indices, which is actually a blueprint allowing the extraction of relevant indices:
From the blueprint viewport, go in top view/lit mode and adjust the inclusion box extent/position (the extent, not the scaling), to select the indices you want to extract (i.e points around left eyes).
Once your box is placed go into the construction script, select the blendshape you want to send the extracted indices to, toggle the safety boolean off and compile the blueprint.
It does not work. When I start the game, it does nothing.
Do I have to do anything other than what’s in the video and set the prot and id?
I would so love to make it work! Its is so cool.
It is imperative that you continue to work on this.
I hope the text is good, I used a translator.