Compositing (transparent BG) video or image sequences in Unreal

[toontitan]

There seems to be very little information on how to achieve this in unreal. There's plenty of demos out there and some tutorials that are years old.

Can anyone gide me as to how i can bring in either a transparent background video, or making a chromakey video for compositing it within unreal engine? I have also tried with image sequences seems its rather cpu intensive, must be doing something wrong.

What I'm trying to do is use Unreal Engine for film-making, in particular compositing 2D sequences within a 3D environment similar to what Disney and Dream works was doing in the early to late 90's. Heres a test clip I did with iClone but I'm really looking to leverage engines lighting and rendering which is a million times better than iClone.

https://www.youtube.com/watch?v=AQSmekpH9e8

[Greg.Corson]

Sounds like you want to use Composure (part of Unreal) See the link here https://docs.unrealengine.com/en-US/...ure/index.html

I have some examples of real time cg and live video backgrounds here https://www.youtube.com/channel/UC9O...lUTzAF-gDeBaaA

Composure also does chroma-key and can handle multi layer compositing so you can have some CG objects in front of the live video and others behind it.

[toontitan]

This doesn't seem to work for me. Having a hard time reproducing it based on that documentation.

I also came across something called ImagePlate for unreal which also seems to have an insane "cicky dicky" approach to getting anything done. But i also haven't figured out how to either make the chromakey transparent or just importing video that's already transparent (By clicky dicky i mean you have to click create all sorts of what seems to be arbitrary nodes and settings to get anything done. You spend so much time fiddling with settings and nodes that it feels you spend more time doing that than actual creatinve work. Is did something that will have to be done every time for every single elements that bring in or can it be done once and just dragged and drooped onto something else or saved and applied as a set of premade settings etc?

Any heee, does any one have any suggestions, examples, easy to follow instructions.


Lets say I have a video of a character doing a walk cycle, and I want to put that in the scene. How do I bring it in and make it happen?

Or I have an animated sequence of a 2d character talking to another character that is in a a separate video that will be placed in a separate layer for depth. How can the backgrounds be removed from these videos, film it in a way that the environment blends (shadows casting, lighting etc. I'm really determined to make this work, there's gotta be a an easy way.


Here is a couple of samples of what I'm working on. These tests were done in iClone where i basically just drag and drop my video file into the program and drop whatever scenery objects I want and pretty much its done. (Yes I know, why i don't i just use iClone, if it's that easy. well because 1. I really like the rendering look and feel of unreal, and 2. The assets market place seems to have more of what I need available for my project, iClones, with my limited 3d knowledge is a pain in the *** for creating my own content). I know that if i can get this done in Unreal, I will be able to make my film in a way that its gonna look a million times better than iclone, and also the variety of assets readily available for me to use from the market place will allow me to finish this at record pace.

iClone test 1:
https://www.youtube.com/watch?v=AQSmekpH9e8

iClone Test 2:
https://youtu.be/8hM8eAUhod8

PLEASE HELP!

>>> https://www.youtube.com/watch?v=0jGT_ZMPhd8

[toontitan]

Im still looking to figure this out. Does any one have any info?

[commedieu]

Toon, how are you making your toons?

However you're making them. Render them out with an alpha channel of the character. Which means your character has a fully white solid shape of their animation. To use as amask.

Put them on a CARD in unreal or a flat plane with uvs on it.

https://forums.unrealengine.com/deve...channel-to-ue4


So hopefully you can spit out your 2d animation as an image sequence, a PNG sequence, or something that supports alpha channels. Place that in the midst of your scene... see if you can get that going from there.

You ever checkout GREASE PENCIL from blender? Might be a cool tool for you as well.

This is assuming a character is just say, walking down a hallway. You do a walk cycle, and are moving the camera toward them. Complex animations with the camera is going to require you blocking out the animation with an animated camera first to match perspective, and all that jazz.

The method im talking about is jus tputting out a pre-animated thing on a card with a mask around it. So that the background of the character is transparent.

[toontitan]

Hi, thanks for replying Commedieu.

Yes i have checked out grease pencil but it seems like its still stuck on a 100 year old workflow. Requires the drawing and manual manipulation of vector lines which quite honestly its a million times easier to just draw with paper and pencil. I'm not really that much of a traditionalist to want to deal with that particular workflow. (i think this is one of the resons people feel 2d is dead, cause they keep doing the same stuff from 100 years ago and don't really attempt to innovate the workflow itself.)

What I'm doing is that I'm creating hand drawn characters that are rigged in a way that the body parts are mapped against a 3d skeletal system. This allows me to create the performances using motion capture. The end result is a 2d animated character that is driven through mocap. See demo here... https://www.youtube.com/watch?v=W67KkMthqAk

I was able to use the same technique in the batman missions web-series ( https://www.youtube.com/watch?v=ie6LsLUH3Bk ) except that instead of drawing the characters, we used photographs of the toys, but its pretty much the same thing.

Anyhee, in terms of producing the animation, this part i have it licked. I am able to export both as video and as PNG sequences if needed. I'm pretty sure i could also generate an alpha matte. I tried with PNG sequences in Blender but it causes the system to run really slow. I guess it needs more memory for that. Yes i'm doing more than just walk cycles. Doing entire shots where characters will be performing. Video seems to be a more reasonable approach but i cant seem to be able to bring in video with transparent BG. i have also attempted with a green BG for chromakey removal but haven't been able to figure that out either. I'm also thinking that if i go with straight PNG, which im not opposed to doing, that i will also next need to deal with syncronizing sound.

I have seen many video demos of people using video for filmmaking, and have managed to get the background removed from the green screen footage. But there just doent seem to be a single tutorial out there that shows how its done.

[toontitan]

I tried following that tutorial you posted, the user interface in Unreal has changed a lot from 2015, none off the features shown in that post seem to correlate to anything in the current version of unreal.Pretty much loses me after the art where the two videos are imported into Unreal. the that shots line up with what unreal does.

Any have any info on how that tutorial is pulled of in version 4.23?

[toontitan]

These guys have obviously managed to figure out a way to composite green-screen footage an unreal. I'm assuming they have some sort of real time feed going in from camera to the engine, but shouldn't the same thing be possible with video that's already been recorded?

https://www.youtube.com/watch?v=wcwx-qu68-g

I mean whatever these guys are doing is basically what I'm trying to do. The only difference is that my footage is cartoons and not live actors.

[toontitan]

BY the way i found in the plugin section something called a webm player enabled. I'm assuming this is a player for webm videos (which support transparency). I have not been able to figure out a way bring in that payer into the system and make use of it.

I did run in to some tuts about unity supporting webm, ave it a try and it was ridiculously easy. there's gotta be a way to do the same thing in this engine.

[Greg.Corson]

Hi Toontitan,

I've been using composure for compositing 3d rendered in unreal with a live feed from a camera with a greenscreen. It sounds like this is pretty much what you want. I have inserted cg characters into the real world and also inserted real world people into CG worlds. This supports multiple layers so you can have some layers appear in front of others. Take a look at my channel here https://www.youtube.com/user/GregCorson and see if anything I'm doing looks like what you need.

It sounds like what you want to do is setup composure with a media plate for your 2d animated video and one or more cg layers for the stuff rendered by unreal. The 2d video could contain a chroma key background (which is what I'm doing) that composure can use to remove the background.

On my channel there are references to a github that has my unreal project in it, however right now I wouldn't recommend you look at that one. It has some really big assets in it and is about 3.5gig. I just figured out a much better way to do the compositing and camera tracking and will try to get a new project up this week. This time I'll just use the 3rd person content so the project will be MUCH smaller and easier to figure out. So you might want to wait till I post that.

Anyway, I'm assuming in your examples the animated character is coming from a video and the background will be 3d from unreal...if that's the case then Composure is what you need to put them together.

[toontitan]

Yes, that's exactly what I'm trying to do. My main action will in the form of some kind of image pate within the game engines cg environment. I will go through your channel and see what I can spot. Sounds like you understand exactly what I'm trying to do.

[toontitan]

I'm going nuts trying to figure it out. what the composure guide says and what actually happens are two different things.
Here's a link to a test file, any chance you can download it and see if you can get it to work on your end? When ever i do it, i end up with a white screen. Have no idea what could be happening.

https://www.dropbox.com/s/opz7f8kfqe...plate.mp4?dl=0
https://www.dropbox.com/s/opz7f8kfqe...plate.mp4?dl=1

[MRohan]

Hi Guys,

I am also trying the same thing, but Its showing only a single image of keyed out footage including foreground and background cg element. I am following the documentation given by UE4, But its very inconsistent as to what is said & what actually happens.

After following exact steps in the doc, I am still unable to get output as expected. As mentioned, process seems to be working fine for single image but not for a video with green screen. This is happening for both viewport display & Image sequence output. I am on UE4.22.

Looks like toontitan & I, both are in a same boat

Best.

[Greg.Corson]

Ok, I'm just wondering, did you guys try this guide https://docs.unrealengine.com/en-US/...art/index.html and if so, can you tell me at what step things started behaving strangely? I ask because I pretty much went through that guide and don't recall having problems. I'm working on a virtual production sample right now, hopefully I can get it posted in the next couple of days, maybe that will help.

[toontitan]

For me when loading the movie. It turns white. ( Sometimes black). But 90% of the time it's white.

Yes there's lot of demos out there but not a single tutorial.

I posted a link to a green screen videoclip earlier, I invite you to download it and try to get it working. If you follow the tutorial listed you'll get the same results as me. White video.