# Face capture with Android - Metahuman - Download Links | Free | Open Source | Demo **URL:** \ **Category:** Character & Animation\ **Tags:** unreal-engine, metahuman\ **Created:** [June 11, 2021, 4:03pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927 "2021-06-11T16:03:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 11, 2021, 4:03pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/1 "2021-06-11T16:03:07Z")
[![](https://d3kjluh73b9h9o.cloudfront.net/original/4X/5/c/5/5c524eca630f338db3a10347269d606d9713704a.jpeg "Unreal Engine: MetaHumans with Android Smartphone | Free | Open Source | Setup | Download links") ](https://www.youtube.com/watch?v=PGcCtvRrS1M) Update: Current bugs and limitations: - The Android app doesn’t handle client disconnection perfectly, you might have to restart the app if you don’t get any incoming data. - The project requires the free ‘TCP Socket plugin’ - Biggest limitation right now is the ability to convert the incoming data into blend weights. It can be fairly complex and advanced to put together a robust system to generate this data. Currently i’m simply interpolation between 0.0 and 1.0 based on the average distance vertice to vertice of a subset of vertices defined per blendshape, between the current position of a vertice and its blendshape Max position. You can download both the Apk and the Unreal Project here: [https://drive.google.com/drive/folders/1xmO57xO6RTyviZEkmkYsE2YY4AAzxlsp?usp=sharing](https://drive.google.com/drive/folders/1xmO57xO6RTyviZEkmkYsE2YY4AAzxlsp?usp=sharing) Android app to stream Face data over Network: > **[GitHub - MaximeDup/AndroidArcoreFacesStreaming: This App is sending Face...](https://github.com/MaximeDup/AndroidArcoreFacesStreaming)** > > This App is sending Face capture data over network, built around the latest Android Arcore SDK. - GitHub - MaximeDup/AndroidArcoreFacesStreaming: This App is sending Face capture data over network,... Unreal Engine Project to read this stream of data, generate blend weights, and hook it into LiveLink: > **[GitHub - MaximeDup/FaceCaptureUnrealEngine: Performance capture for Unreal...](https://github.com/MaximeDup/FaceCaptureUnrealEngine)** > > Performance capture for Unreal Engine using Android smartphones. Rely on AndroidArcoreFacesStreaming - GitHub - MaximeDup/FaceCaptureUnrealEngine: Performance capture for Unreal Engine using Androi... [![](https://d3kjluh73b9h9o.cloudfront.net/original/4X/d/3/d/d3d290bbcb02a3e28d965e1ee14d93021e16c4d8.jpeg "Unreal Engine : Live Calibration | Face Capture | Android Smartphone") ](https://www.youtube.com/watch?v=MMxHCuSodTo) ```auto Hello Unreal Community, Opening a new topic regarding my recent effort (24H literally) to get Android Face Capture going. Goal is to create an android alternative to the Apple FaceAR solution, with MetaHumans compatibility . I am starting from scratch and everything is gonna be open source, starting with the Android App which send ArCore face data to Unreal Engine trough TCP (free TCP Socket Plugin from Unreal). This is what i have so far (recorded galaxy s9): quote: "wow wow wow wow wow wow" https://www.youtube.com/watch?v=_kLu96hfot0 https://i.imgur.com/w20bTLD.png ``` ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 11, 2021, 11:30pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/2 "2021-06-11T23:30:05Z")
Got the head bone movement imported, time to put the groundwork for blendshapes! Now that the whole android part is usable enough and send the data i need i am back in known territories. [![](https://d3kjluh73b9h9o.cloudfront.net/original/4X/8/8/4/884136837abb793268491e1b8072ec2c718826c9.jpeg "Unreal Engine 4 : Android face Capture to MetaHuman | ARCore | WIP 2") ](https://www.youtube.com/watch?v=Ci2H7FRKzCk) ---
**Author:** ![fengkan](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@fengkan](https://forums.unrealengine.com/u/fengkan)\ **Post date:** [June 12, 2021, 4:00am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/3 "2021-06-12T04:00:31Z")
we need this! ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 13, 2021, 3:55pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/4 "2021-06-13T15:55:35Z")
Everything is working Just, Fine ! [![]( " - YouTube") ](https://www.youtube.com/watch?v=lIaCEaQ4otw) ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 14, 2021, 12:35am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/5 "2021-06-14T00:35:50Z")
You can download both the Apk and the Unreal Project here: [https://drive.google.com/drive/folders/1xmO57xO6RTyviZEkmkYsE2YY4AAzxlsp?usp=sharing](https://drive.google.com/drive/folders/1xmO57xO6RTyviZEkmkYsE2YY4AAzxlsp?usp=sharing) Android app to stream Face data over Network: > **[GitHub - MaximeDup/AndroidArcoreFacesStreaming: This App is sending Face...](https://github.com/MaximeDup/AndroidArcoreFacesStreaming)** > > This App is sending Face capture data over network, built around the latest Android Arcore SDK. - GitHub - MaximeDup/AndroidArcoreFacesStreaming: This App is sending Face capture data over network,... Unreal Engine Project to read this stream of data, generate blend weights, and hook it into LiveLink: > **[GitHub - MaximeDup/FaceCaptureUnrealEngine: Performance capture for Unreal...](https://github.com/MaximeDup/FaceCaptureUnrealEngine)** > > Performance capture for Unreal Engine using Android smartphones. Rely on AndroidArcoreFacesStreaming - GitHub - MaximeDup/FaceCaptureUnrealEngine: Performance capture for Unreal Engine using Androi... [![](https://d3kjluh73b9h9o.cloudfront.net/original/4X/d/3/d/d3d290bbcb02a3e28d965e1ee14d93021e16c4d8.jpeg "Unreal Engine : Live Calibration | Face Capture | Android Smartphone") ](https://www.youtube.com/watch?v=MMxHCuSodTo) ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 14, 2021, 2:20am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/6 "2021-06-14T02:20:07Z")
Free and open source, updated the various links. [![](https://d3kjluh73b9h9o.cloudfront.net/original/4X/5/c/5/5c524eca630f338db3a10347269d606d9713704a.jpeg "Unreal Engine: MetaHumans with Android Smartphone | Free | Open Source | Setup | Download links") ](https://www.youtube.com/watch?v=PGcCtvRrS1M) ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 15, 2021, 3:56pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/7 "2021-06-15T15:56:32Z")
Updated the App presentation photo 😃 [![](https://i.imgur.com/uC77IqQl.jpg) ](https://i.imgur.com/uC77IqQl.jpg) ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 15, 2021, 4:30pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/8 "2021-06-15T16:30:28Z")
I did fix the demo project requirements: Initial version required 4.26.2-chaos built from source, whereas it now works with 4.26 downloaded from the Epic games Launcher. ---
**Author:** ![anonymous\_user\_143d58aa](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@anonymous\_user\_143d58aa](https://forums.unrealengine.com/u/anonymous_user_143d58aa)\ **Post date:** [June 17, 2021, 5:03am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/9 "2021-06-17T05:03:46Z")
Hi, the provided project file on google drive gives an error if using regular Unreal Engine 4.26.2 . Here is the error screen: ![error](https://d3kjluh73b9h9o.cloudfront.net/original/3X/1/4/1441e8d14429505070720ecf2da2138783549753.png) ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 17, 2021, 5:33am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/10 "2021-06-17T05:33:02Z")
Does the rebuild fails? You can also try the version available on github: > **[Release Original · MaximeDup/FaceCaptureUnrealEngine](https://github.com/MaximeDup/FaceCaptureUnrealEngine/releases/tag/1.0)** Downloading source code zip file ---
**Author:** ![anonymous\_user\_143d58aa](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@anonymous\_user\_143d58aa](https://forums.unrealengine.com/u/anonymous_user_143d58aa)\ **Post date:** [June 17, 2021, 6:06pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/11 "2021-06-17T18:06:02Z")
- I get “need manually source building error” after trying to rebuild with Google Drive version (as seen on the top of the picture): - Github version asks for Unreal Engine version and picking 4.26 gives the bottom error on the picture : ![aa](https://d3kjluh73b9h9o.cloudfront.net/original/3X/d/e/de66651e67fce4d12e526536f6233835252b726c.png) ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 17, 2021, 6:15pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/12 "2021-06-17T18:15:59Z")
This is a C++ project, you’re gonna need to have your Unreal Engine installation to be able to handle such project. (installing visual studio etc…) ---
**Author:** ![anonymous\_user\_143d58aa](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@anonymous\_user\_143d58aa](https://forums.unrealengine.com/u/anonymous_user_143d58aa)\ **Post date:** [June 17, 2021, 9:22pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/13 "2021-06-17T21:22:56Z")
Thanks, I rebuilt with VS2019 and configured all the steps on video. It works on my side now. I am looking forward for it be improved for more accurate results (such as blinks). Keep up the good work. ---
**Author:** ![fengkan](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@fengkan](https://forums.unrealengine.com/u/fengkan)\ **Post date:** [June 24, 2021, 3:33am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/14 "2021-06-24T03:33:41Z")
I always wanted to figure out if it’s possible to convert face mesh landmarks to blendshapes. It’s great to see your implementation. Thank you! In your project, you use the AndroidBlendshape assets to accomplish this, I am wondering how do you get the values in these AndroidBlendshape assets? Do you have utilities to generate these assets? ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 24, 2021, 9:41pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/15 "2021-06-24T21:41:03Z")
If you right click in the content browser you should see the category “FaceAndroid” which allow you to create 2 objects: - AndroidBlendShape : a blendshape location, a subset of relevant indices from the face, a position of face points when blendshape is maxed out. - AndroidBlenshapeWeights : Holds a map of blendweights computed at runtime purely based on aggregated distances from relevant face point runtime position compared to maxed out blendshape position. Using distances is really trivial, wouldn’t be surprised if machine learning was industry standard past this point to convert points to weights. I might give it a shot later if i find relevant data. ---
**Author:** ![fengkan](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@fengkan](https://forums.unrealengine.com/u/fengkan)\ **Post date:** [June 25, 2021, 2:52am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/16 "2021-06-25T02:52:16Z")
Hi, thank you for the detailed explanation, there are two arrays in the AndroidBlendShape asset, Vertices to Control and Vertices Max Mvt, both have hundreds of elements. I think it’s quite difficult to fill these arrays by hand, so I am wondering whether you have any utilities, thank you. ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [June 25, 2021, 3:03am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/17 "2021-06-25T03:03:33Z")
I. On the default demo map there’s a blueprint representing the face points indices, which is actually a blueprint allowing the extraction of relevant indices: From the blueprint viewport, go in top view/lit mode and adjust the inclusion box extent/position (the extent, not the scaling), to select the indices you want to extract (i.e points around left eyes). Once your box is placed go into the construction script, select the blendshape you want to send the extracted indices to, toggle the safety boolean off and compile the blueprint. You extraced : Vertices to Control ---
**Author:** ![fengkan](https://d3kjluh73b9h9o.cloudfront.net/original/4X/2/8/5/28502927f1d97ba1051b3b47ccfe17233bf41ef0.png) [@fengkan](https://forums.unrealengine.com/u/fengkan)\ **Post date:** [June 25, 2021, 2:18pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/18 "2021-06-25T14:18:18Z")
I see, that’s great, thank you for the reply. ---
**Author:** ![MaximeDupart](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/maximedupart/32/1194606_2.png) [@MaximeDupart](https://forums.unrealengine.com/u/MaximeDupart)\ **Post date:** [August 2, 2021, 12:29am UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/19 "2021-08-02T00:29:11Z")
If anyone is looking to implement some machine learning to the weight generation, i believe it could be a very rewarding application of it. Could be given to students too surely. ---
**Author:** ![Hansch](https://d1ap1mz92jnks1.cloudfront.net/user_avatar/forums.unrealengine.com/hansch/32/88708_2.png) [@Hansch](https://forums.unrealengine.com/u/Hansch)\ **Post date:** [August 6, 2021, 3:57pm UTC](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927/20 "2021-08-06T15:57:58Z")
It does not work. When I start the game, it does nothing. Do I have to do anything other than what’s in the video and set the prot and id? I would so love to make it work! Its is so cool. It is imperative that you continue to work on this. I hope the text is good, I used a translator. [Next page](https://forums.unrealengine.com/t/face-capture-with-android-metahuman-download-links-free-open-source-demo/234927.md?page=2)