There seems to be very little information on how to achieve this in unreal. There’s plenty of demos out there and some tutorials that are years old.
Can anyone gide me as to how i can bring in either a transparent background video, or making a chromakey video for compositing it within unreal engine? I have also tried with image sequences seems its rather cpu intensive, must be doing something wrong.
What I’m trying to do is use Unreal Engine for film-making, in particular compositing 2D sequences within a 3D environment similar to what Disney and Dream works was doing in the early to late 90’s. Heres a test clip I did with iClone but I’m really looking to leverage engines lighting and rendering which is a million times better than iClone.
This doesn’t seem to work for me. Having a hard time reproducing it based on that documentation.
I also came across something called ImagePlate for unreal which also seems to have an insane “cicky dicky” approach to getting anything done. But i also haven’t figured out how to either make the chromakey transparent or just importing video that’s already transparent (By clicky dicky i mean you have to click create all sorts of what seems to be arbitrary nodes and settings to get anything done. You spend so much time fiddling with settings and nodes that it feels you spend more time doing that than actual creatinve work. Is did something that will have to be done every time for every single elements that bring in or can it be done once and just dragged and drooped onto something else or saved and applied as a set of premade settings etc?
Any heee, does any one have any suggestions, examples, easy to follow instructions.
Lets say I have a video of a character doing a walk cycle, and I want to put that in the scene. How do I bring it in and make it happen?
Or I have an animated sequence of a 2d character talking to another character that is in a a separate video that will be placed in a separate layer for depth. How can the backgrounds be removed from these videos, film it in a way that the environment blends (shadows casting, lighting etc. I’m really determined to make this work, there’s gotta be a an easy way.
Here is a couple of samples of what I’m working on. These tests were done in iClone where i basically just drag and drop my video file into the program and drop whatever scenery objects I want and pretty much its done. (Yes I know, why i don’t i just use iClone, if it’s that easy. well because 1. I really like the rendering look and feel of unreal, and 2. The assets market place seems to have more of what I need available for my project, iClones, with my limited 3d knowledge is a pain in the *** for creating my own content). I know that if i can get this done in Unreal, I will be able to make my film in a way that its gonna look a million times better than iclone, and also the variety of assets readily available for me to use from the market place will allow me to finish this at record pace.
However you’re making them. Render them out with an alpha channel of the character. Which means your character has a fully white solid shape of their animation. To use as amask.
Put them on a CARD in unreal or a flat plane with uvs on it.
So hopefully you can spit out your 2d animation as an image sequence, a PNG sequence, or something that supports alpha channels. Place that in the midst of your scene… see if you can get that going from there.
You ever checkout GREASE PENCIL from blender? Might be a cool tool for you as well.
This is assuming a character is just say, walking down a hallway. You do a walk cycle, and are moving the camera toward them. Complex animations with the camera is going to require you blocking out the animation with an animated camera first to match perspective, and all that jazz.
The method im talking about is jus tputting out a pre-animated thing on a card with a mask around it. So that the background of the character is transparent.
Yes i have checked out grease pencil but it seems like its still stuck on a 100 year old workflow. Requires the drawing and manual manipulation of vector lines which quite honestly its a million times easier to just draw with paper and pencil. I’m not really that much of a traditionalist to want to deal with that particular workflow. (i think this is one of the resons people feel 2d is dead, cause they keep doing the same stuff from 100 years ago and don’t really attempt to innovate the workflow itself.)
What I’m doing is that I’m creating hand drawn characters that are rigged in a way that the body parts are mapped against a 3d skeletal system. This allows me to create the performances using motion capture. The end result is a 2d animated character that is driven through mocap. See demo here… 2D character animation incorporating Mocap. - YouTube
Anyhee, in terms of producing the animation, this part i have it licked. I am able to export both as video and as PNG sequences if needed. I’m pretty sure i could also generate an alpha matte. I tried with PNG sequences in Blender but it causes the system to run really slow. I guess it needs more memory for that. Yes i’m doing more than just walk cycles. Doing entire shots where characters will be performing. Video seems to be a more reasonable approach but i cant seem to be able to bring in video with transparent BG. i have also attempted with a green BG for chromakey removal but haven’t been able to figure that out either. I’m also thinking that if i go with straight PNG, which im not opposed to doing, that i will also next need to deal with syncronizing sound.
I have seen many video demos of people using video for filmmaking, and have managed to get the background removed from the green screen footage. But there just doent seem to be a single tutorial out there that shows how its done.
I tried following that tutorial you posted, the user interface in Unreal has changed a lot from 2015, none off the features shown in that post seem to correlate to anything in the current version of unreal.Pretty much loses me after the art where the two videos are imported into Unreal. the that shots line up with what unreal does.
Any have any info on how that tutorial is pulled of in version 4.23?
These guys have obviously managed to figure out a way to composite green-screen footage an unreal. I’m assuming they have some sort of real time feed going in from camera to the engine, but shouldn’t the same thing be possible with video that’s already been recorded?
I mean whatever these guys are doing is basically what I’m trying to do. The only difference is that my footage is cartoons and not live actors.
BY the way i found in the plugin section something called a webm player enabled. I’m assuming this is a player for webm videos (which support transparency). I have not been able to figure out a way bring in that payer into the system and make use of it.
I did run in to some tuts about unity supporting webm, ave it a try and it was ridiculously easy. there’s gotta be a way to do the same thing in this engine.
I’ve been using composure for compositing 3d rendered in unreal with a live feed from a camera with a greenscreen. It sounds like this is pretty much what you want. I have inserted cg characters into the real world and also inserted real world people into CG worlds. This supports multiple layers so you can have some layers appear in front of others. Take a look at my channel here https://www.youtube.com/user/GregCorson and see if anything I’m doing looks like what you need.
It sounds like what you want to do is setup composure with a media plate for your 2d animated video and one or more cg layers for the stuff rendered by unreal. The 2d video could contain a chroma key background (which is what I’m doing) that composure can use to remove the background.
On my channel there are references to a github that has my unreal project in it, however right now I wouldn’t recommend you look at that one. It has some really big assets in it and is about 3.5gig. I just figured out a much better way to do the compositing and camera tracking and will try to get a new project up this week. This time I’ll just use the 3rd person content so the project will be MUCH smaller and easier to figure out. So you might want to wait till I post that.
Anyway, I’m assuming in your examples the animated character is coming from a video and the background will be 3d from unreal…if that’s the case then Composure is what you need to put them together.
Yes, that’s exactly what I’m trying to do. My main action will in the form of some kind of image pate within the game engines cg environment. I will go through your channel and see what I can spot. Sounds like you understand exactly what I’m trying to do.
I’m going nuts trying to figure it out. what the composure guide says and what actually happens are two different things.
Here’s a link to a test file, any chance you can download it and see if you can get it to work on your end? When ever i do it, i end up with a white screen. Have no idea what could be happening.
I am also trying the same thing, but Its showing only a single image of keyed out footage including foreground and background cg element. I am following the documentation given by UE4, But its very inconsistent as to what is said & what actually happens.
After following exact steps in the doc, I am still unable to get output as expected. As mentioned, process seems to be working fine for single image but not for a video with green screen. This is happening for both viewport display & Image sequence output. I am on UE4.22.
Looks like @toontitan & I, both are in a same boat
Ok, I’m just wondering, did you guys try this guide Real-Time Compositing Quick Start | Unreal Engine Documentation and if so, can you tell me at what step things started behaving strangely? I ask because I pretty much went through that guide and don’t recall having problems. I’m working on a virtual production sample right now, hopefully I can get it posted in the next couple of days, maybe that will help.
For me when loading the movie. It turns white. ( Sometimes black). But 90% of the time it’s white.
Yes there’s lot of demos out there but not a single tutorial.
I posted a link to a green screen videoclip earlier, I invite you to download it and try to get it working. If you follow the tutorial listed you’ll get the same results as me. White video.
Hello!
already three in one boat!
I also really need to be able to use the png sequence.
Keying in the unreal is certainly good, but let’s leave it to the after-effect!
There is a PNG sequence. How to place it simply on a plane so that it scrolls there? or ok, let it be a mask as a separate sequence for transparency!
I’m such a new beginner that I don’t even know how to make a media texture!
watched some lesson for in the old version, how to do it in the new? I do not understand!
I am glad to see that I’m not the only one alone in this. I hope we can find a solution because for people doing film and cinematography this particular workflow could be a game changer.
I know that compositing is probably an easier thing to do in after effects but the reason that is not an option at this time is because the game engine itself allows for direct interaction between the objects so the video element can be made to look like it’s part of the same world with no motion tracking required perfect shadow catching and overall a cohesive look in the render between the background the elements and the lights and things of that nature.
If this can be sorted out then all of the steps that are usually required in after effects to make a decent composition can be eliminatedBy just having everything within the same scene in real time.
Not to mention for people making video games I’m sure they would have additional options for using video elements as part of their scenery and things of that nature that go beyond just TV screens and stuff.
If you want to see my new sample I’ve posted it here GitHub - MiloMindbender/UE4VirtualProduction: An example Unreal Engine Virtual Production Project< there is NO documentation yet but I am working on it. The project is WAY smaller (13mb instead of 4gb) than my previous demo so it will be easy to download, it also uses very few shaders so the editor will start up quickly. The downside is that it’s not a photo realistic scene, but you can always use the Virtual Production setup in this project on another map.
This is using live camera input from an AJA Kona-HDMI video card, but you can substitute video from a webcam or from a movie file with a bit of work.
