Unreal VCam AR using Composure

Development Cinematics & Media Virtual Production
, ,
1

Greetings,

I have a request concerning my demonstrated feature in UE 5.7.3. What I need is to have a Unreal VCam open and feed it to unreal, at the same time, I really would like to have some way of projecting and masking the 3d scene in the Unreal VCam app (either Android or Apple). The thing is I cannot share two apps at the same time - one for a sort of NDI stream and the other for the Unreal VCam. That is the reason for using the two devices. Is there any way to avoid this - force Unreal VCam to send the image data to Unreal or make it a input device this way? Is there any chance we could get the source code for the Unreal VCam app to do this ourselves perhaps?

This solution is suitable in our usecase for both AR experience and InCamera VFX - masking green screens. However, is there a way to reduce it to one device only? Is there also a way to mask e.g. a green screen floor in the MainRenderLayer in Composure?

Thank you for your time inspecting this.

ue_ar_test.mp4(5.87 MB)

2

Steps to Reproduce
Log into a Unreal VCam app on your apple or android device, connect to the device (a scene has to have the VCam Actor in it)

Create a media player with a video streaming texture, put ANOTHER camera device into the media player as an input

Use Composure to align the layers and wanted objects

  • Turn off main layer, create a composite layer and filter objects you want to see in the final shot
  • Add a plate layer with the media texture video input

Align the devices cameras closely together and walk around

3

Hey Petr,

Yes we are looking into this but don’t have anything productized yet. The vcam source code is on Git already in engine/extras.

Engine/Extras/VirtualProduction/FlutterLiveLinkVCAM

4

Hey Petr, So if you have flutter installed and all of the dependencies via `flutter pub get` then you should be able to just do a `flutter build ios` to make a dev build that you can test. I know we don’t have good docs on this but the `flutter build` is the command that should do almost everything.

5

Hi Petr, Here are some instructions for removing Tentacle from your build. We do not distribute the tentacle SDK due to licensing restrictions.

The Tentacle SDK provides Bluetooth timecode sync with Tentacle Sync devices. It’s not distributed to licensees, so you need to strip it before building. There are 5

files to edit and 3 files to delete.

Files to Edit

1. pubspec.yaml — Remove the dependency

flutter_tentacle:

path: ../../Flutter/FlutterTentacle

2. lib/main.dart — Remove all Tentacle usage

a) Delete the import:

import ‘package:flutter_tentacle/tentacle.dart’;

b) In the shutdown() function, delete:

TentaclePlugin.shutdown();

c) In _VcamAppState.build(), delete the TentacleDeviceManager provider :

 Provider(

  create: (\_) \=\> TentacleDeviceManager(),

  dispose: (\_, provider) \=\> provider.dispose(),

 ),

d) In the same method, remove TentacleTimecodeSource registration:

   manager.registerSource(TentacleTimecodeSource());

e) In the localizationsDelegates list, delete (line 169):

    ...TentacleLocalizations.localizationsDelegates,

3. IOS/Podfile — Remove the Tentacle configuration

Delete the entire configure_flutter_tentacle function definition:

# Configure the Pods project with build information necessary for flutter_tentacle.

# … (entire block through `end`)

And remove its call in post_install:

configure_flutter_tentacle(installer)

4. assets/licenses/manifest.json — Remove the license entry

Find and delete this key-value pair:

“Tentacle SDK”: {“version”: “5.0.1”, “file”: “…”},

Files to Delete

1. ffigen_tentacle.yaml — FFI bindings config for the Tentacle C SDK

2. lib/tentacle/api/tentacle_ffi.g.dart — Auto-generated FFI bindings (and the lib/tentacle/ directory if empty after)

3. assets/licenses/text/2f46753e11435a8cd42019e72b05fc62 — Tentacle SDK license text

After All Changes

# Clean generated plugin registrations

flutter clean

# Re-resolve dependencies (this regenerates .flutter-plugins)

flutter pub get

# For iOS, re-install pods

cd IOS && pod install && cd ..

6

It looks like some files are not getting mirrored to github. I’ll follow-up with the team that manages to figure out why.

7

I talked to the GH admins and there are some filters that are preventing the files from getting copied. They are going to make an adjustment to the filters so that they get included. I’ll update you when that happens. It might take a couple of days.

8

I heard yesterday that the changes were imminent but I still don’t see the files on Github. I would expect soon but unfortunately “not yet”.

Jason

9

Hi Petr,

It does look like the filters have been updated. You need to make sure you run Setup.sh to ensure that all of the dependencies are installed. Larger artifacts are stored in a secondary system. I’ll grab the latest from github to confirm everything is as expected.

Jason

10

let me re-try this. IIRC there is a localization script you need to run. I’ll dig that up.

11

Hi,

I found there is a flutter 3.29 build issue. I’ve attached a patch file that should fix the error.

# 1. cd to the UnrealEngine root (the dir that contains Engine/, Plugins/, Samples/, etc.)

cd /path/to/UnrealEngine

# 2. Apply the patch

git apply Engine/Extras/VirtualProduction/FlutterLiveLinkVCAM/flutter_live_link_vcam_3.29_fixes.patch

# — or, if they’re not using git for these files —

patch -p1 < Engine/Extras/VirtualProduction/FlutterLiveLinkVCAM/flutter_live_link_vcam_3.29_fixes.patch

# 3. Regenerate the app_localizations.dart file

cd Engine/Extras/VirtualProduction/FlutterLiveLinkVCAM

flutter clean

flutter pub get

flutter gen-l10n

flutter build ios

1. Flutter 3.29 removed the synthetic flutter_gen package, so package:flutter_gen/gen_l10n/app_localizations.dart imports no longer resolve — the generated file

now lives under the project’s own lib/l10n/ directory.

2. Flutter 3.29 renamed the CardTheme constructor in ThemeData(cardTheme: …) to CardThemeData.

The gist that has the patch can be found here:

You’ll need to copy it into the patch file mentioned above. I’m not able to attach it directly to this case.

https://gist.github.com/jdw\-epicgames/7daefe1ad27cb7daeb7649fb126d2d69

12

Hi Petr,

I’m going to close this case. It sounds like you were able to move forward. We have plans to update to support newer versions of flutter and hopefully will ease the path for developers.

Jason

13

Hello Shaun.

At the moment I am just trying to compile the plugin myself via flutter on MacOS (for iOS only). Can you guide me onto how to do it?

I already ran into some problems regarding FlutterTentacle’s android dependencies when trying to do ‘flutter pub get’ - I disabled the dependencies since the only relevant ones for us are the ones for iOS. Later I found out that there is no ‘lib/main.dart’ for the plugin alone.

Disclaimer: I am a beginner in iOS dev unfortunately, sorry.

14

Hello Jason,

Thanks for your reply. I did try my best to do this; First off, I am doing this from the git latest release of UE - 5.7.4. What I do get exactly after your instructions is that there a is a missing ‘lib/dart.main’ in the solution. What’s more, I delved deeper into the dependencies and found out, that FlutterTentacle has this in its readme:

### Tentacle SDK

This project depends on the Tentacle SDK located in `Engine/Restricted/NotForLicensees/Source/ThirdParty/TentacleSDK`. Note that this SDK is not

distributed to Unreal Engine licensees. If you have access to it, place it in that location to build this package.

Could that be the main cause of the problem? The app itself looks more like a wrapper than a “whole” application when I inspected the code.

15

Hello Jason,

Thanks for the guide, however, the files you are talking about are completely missing in the git repo, namely

lib/main.dart - there is no lib/ folder at all in the root folder and no main.dart file,

IOS/Podfile is missing,

the entry at step 4. you mentioned is not there,

assets/licenses/text/ folder does not exist.

To make sure we are on the same page I have attached the whole source of this directory that is available from github release branch. If I am supposed to checkout a different branch, please tell me which one.

Thanks for your help so far,

Petr

16

Perfect, happy to hear that. Our goal is specified in the beginning of this thread.

Thanks for your help so far!

Petr

17

Hello Jason,

is there any progress on this matter?

Thanks Petr

18

Hello Jason,

I noticed on the github’s ue5-main branch, there appeared to be new files. However, these are still not ready for the app’s compilation.

lib/l10n/app_en.arb is missing

the podfile in iOS folder is missing

If I am supposed to somehow generate some of these files, I will be very happy if you can guide me how to do so.

Thank you in advance for your cooperation and your patiance,

Petr

19

Hello Jason,

I just want to check on if you were able to compile the git version; we did not previously have the network multicast capability allowed, which was mandatory for the app to be build.

Now, I still I am unfortunately still not able to compile mostly due to AppLocalization errors as seen in the attached file. I tried building the original with flutter tentacle, that did not work ofc, then I tried (with a clean install to make sure) with following your instructions, both with flutter build ios and building via XCode, unfortunately, nothing worked. I do not know if I am missing something, but the

import 'package:flutter_gen/gen_l10n/app_localizations.dart';in main.dart does not seem to be working properly, as if it cannot find the generated package. I tried to do

import 'l10n/app_localizations.dart';instead but that one did not unfortunately build either with the same AppLocalization errors.

If you succeeded in building yourself, I would appreciate any help as I feel this is getting to a close end soon.

Thanks for your patience with me,

Petr

20

Hello,

thank you, this in fact in the end worked. However, it is still required to pull straight from the git, as there are missing dependencies from the setup.sh script in the beginning if you download just the zip file.

Much thanks for your help!

Petr