On our environments for virtual production stages we use various “playback” elements in our environments such as fog, steam, fire etc… these are all short looping video frames that are meant to be used as a quick and inexpensive way to add effects to an environment.
This is different from plate playback where we have hi res footage meant to be played back across the entire wall, which the tiled EXR workflow is for.
How we have been doing this for the past 5 years is with paper2d flipbooks - a feature not longer in active development, and has some pain points.
Paper2D Flipbooks cannot be used as media sources in rect lights, a feature that Megalights enables (Textured area lights) https://youtu.be/7q1rI25tOUM?t=27
You cannot animate them with sequencer, but outside of sequencer there is some limited ability to play/pause/go to via blueprint
You bring them in as individual PNG or EXR sequences and have to make the sprites/flipbooks. The Graphics cards we use in virtual production have a lot of VRAM that does allow us to load the entire frames into memory for playback… but there’s a limit of course.
We’ve done some tests to see about the viability of using the media framework as a replacement system for element playback.
While these individual clips are short (only a couple seconds long) we need multiple different elements playing back simultaneously as seen in the video above, attempting to do this with MP4 or prores files has not only format limitations but performance issues.
Converting the footage into bink video files seems to reduce the filesize by alot when compared to a EXR/PNG seqeunce
However, Bink media sources cannot be loaded into the media plate actor, it only works with file or image media sources. The media framework has no way of loading a bink file - obviously these are two separate systems in the engine.
Creating a bink player in blueprint does work and therefore bink files will playback on an nDisplay cluster, but the content does not stay in sync, nor is there any way to control playback outside of blueprint nodes.
A couple of items I can suggest. First, you can animate flipbooks in Sequencer. This is done using a parameter in the material that is then exposed in Sequencer. I’ll send an email showing the details on this.
Hey Matthew, I have tried downloading here, but it’s giving me an error. In the meantime, I have shared a folder on Box that you should be able to upload the files to.
I just wanted to follow up on this ticket. The ticket owner was unfortunately laid off last week, so I wanted to check in to see if I can close out the ticket or if you need further assistance.
Thank you for the quick response. I will go ahead and close out the ticket then. Please feel free to reach out to us anytime if you have any more questions.
Hi Michael, I’ve created a project here where I’ve migrated in a bunch of paper2d flipbooks into an empty project. This has examples of paper2d flipbook we use in environments.
You can see one of the issues being the file size of the project is 10GB when unzipped.
When loading up environments we always use r.streaming.fullyloadusedtextures 1to make sure all the flipbooks stream in rather than waiting on the streaming pool to load them all.
Tim and a developer were able to look into some paper2D flipbook assets I sent along and confirmed the upcoming OpenAV codec support in 5.8 will address all the needs I was asking about with Bink.