Download

UE4 + Nvidia Ansel + 360 Video Capturing

Hi guys,

I am trying to take a 4K 360 30fps stereo video from UE4 projects (cinematic scenes made with Matinee or Sequencer).

I tried Stereo Panorama Tool plugin (Kite & Lighting), but it failed to render properly some problematic parts of some scenes and it’s claimed (by other developers) that it works not good with full scene reflection.

There’s a tool developed by Nvidia called Ansel that can take in game 360 stereo image screenshot. Also there’s a plugin for UE4 that integrates this tool using AnselSDK.

One of the problems is that Ansel can be activated (capturing itself, its session can be activated from code) only by pressing on Ansel UI button (whole Ansel UI injected into the game itself once the session is activated). The only way that I have found to automate the process is to inject Windows keyboard events (mouse events cause rotation of the camera sometimes and instability of the captured images) using Windows API (Native, Winform), AutoIt, etc. (all the tools, techniques are free, didn’t try commercial products) in order to press on required buttons by TABing (menu navigation) with proper time stamps.

The problem with this approach is that the Ansel UI which is injected into the game (UE4 standalone mode) doesn’t catch these event sometimes which causes a lot of problems (missed frames and wrong selection in the menu).

Another problem with Ansel is that if it runs for a long time (depending on a machine), it crashes UE4 itself. So the process should be started again and continue from the point of fail. For this problem I didn’t find a proper solution as well, I would like to use it as Kite & Lighting.

Hope you have some thoughts :wink:

Thanks!

I use Ansel with scripted keyboard events and it works great rendering 3000 stereo 8k 360 frames. Never had a problem with crashing. I use the macro program JitBit.

Could you please explain your process in a bit more detail? I’m interested in how you get frame accurate capture.

Hi, I’m also interested in knowing more about your process! I’ve been struggling with 360 video capture using the UE SP Tool and automatizing Ansel seems like the best option…

Hi, I’m also interested in knowing more about how you did! I’ve been struggling to capture 360 video with UE SP Tool and automatizing Ansel seems like the best solution so far…

Hey gang. Just found this thread recently and gave JitBit a try with Ansel. I’m able to spit out an image sequence but I’m not sure if it’s exactly capturing each frame at 30 FPS. I suspect it isn’t.

The problem as I see it is that all the cinematic nodes are in values of “float”. Because of that, one second of animation in a 30 fps speed is 1/30 which is (.0333333333333333333…). I think we need whole numbers some now as this infinite float value looks like it’ll let a lot of error creep in as we render longer image sequences.

Attached is the level blueprint I am using to advance the game one frame then pause it at a rate of 30 fps. I’ve tried this with and without forcing the game to an FPS of 30 and I’m not sure if that’s helping or not. Any help on playing the game then pausing it, one frame at a time in a 30 fps time frame, would be much appreciated.

Thanks everyone who has been contributing to this discussion so far. I put together a short teaser video of my (working) workflow here, expanding upon everything in this thread:

And will do my best to put together a full tutorial at some point.

hi man !

can you explain how u did?
it’s amazing!

This is a game changer, any more tips or info you can provide about how you are setting this up would be greatly appreciated!

It’s kind of insane how many emails I’ve gotten about this workflow over the last month. Here you go guys:

If you make something cool, please share it with me on twitter @iBrews :slight_smile:

@iBrews Have implemented as described in the video but it doesnt seem like the macro wants to fire off Ansel…I can see the other parts of it running through the motions but it wont start up ansel after I have done the first manual step. Is there a special window configuration or something we need to make sure is always on top for it to work?

@Warrior9VR hmmm don’t think so. I’m guessing you were able to use Ansel manually. Make sure your Macro has these checkboxes set up (I think the ControlSend is the key one here) :

@iBrews ah yes that would definitely have been the issue! Got it working now, very cool. One thing I noticed though is that my camera seems to shift during the capture process and as such i am getting some multiple exposure effects going on. Trying to figure it out now

testing across different levels and indeed the double image im getting is a result of the camera animation being paused seemingly between frames.

Is your frame-rate in the sequence the same as your frame-rate in the blueprint? Sounds like this would happen if your sequence framerate is 30 and your blueprint framerate is 60.

I have the blueprint and sequence set to 30 fps, as well as the constant frame rate set to 30 in the options

Not sure what’s causing this, but a few ideas based on threads like this: (Stereo Panoramic Movie Capture Skipping Frames - VR and AR Development - Unreal Engine Forums)
– try using a normal camera or a pawn and not a cinematic camera
– create an Unreal Engine shortcut on your desktop and launch Unreal Engine with the following options (this assumed 4.18 installed on your C drive) “C:\Program Files\Epic Games\UE_4.18\Engine\Binaries\Win64\UE4Editor.exe” -usefixedtimestep -fps=30 -notexturestreaming
– try doing a test export in an empty project to see if there might be something specific to your project settings

Changing it to a pawn for capturing made the difference! So folks, dont use a cinematic camera! Thanks for your assisstance @iBrews

Have been experimenting with the ansel rendering the past few days and now I am running out of video memory…assuming this is caused by the density of the scene I am trying to render. Any tips or work-arounds for this?

Why does NVidia not expose their UI to us? Relying on AHK is really unwieldy.