UE4 + Nvidia + 360 Video Capturing

[Roberto_De_Ioris]

@anonymous_user_03000d7e

The plugin doesn’t work properly. I have a sequencer scene that is 600 frames long. I tried to record the scene with this plugin, but It shoots only around 290 images instead of 600…

Hi, please open an issue on github with the whole procedure you followed.

[Roberto_De_Ioris]

@ax448

Nice!
There are a lot of things I’m curious about:

How does it work in practice:
When you execute the ‘Start Capture’ node, it looks like it does the open/select type/select res/render/close. loop that you can do once you’ve launched once. Correct?
Is that hardcoded, or is there a way to adjust that? After all, it’s just a matter of how many up/down/left/right inputs you send. It would be great if we could just select the type and res from a dropdown.
(Or honestly, even just select how many left/right directional inputs we want.)(I do both 360 and 360Pano stuff pretty interchangably, and at different resolutions. )

And for the Start Capture node, does it just run for 1 capture, so that we can set it up to trigger again afterwards? Or does it somehow loop by itself?
(It does have an outgoing executor, will that trigger once the window closes again?)

And does it trigger a new frame rendering the moment the previous frame is done rendering, or once it’s done processing/saving?
Because I noticed a big issue there with , if you finish a rendering while the previous frame is still being processed/saved, the older frame is often discarded.
So I had to add a small delay between finishing a render and starting a new one to avoid that.

It does seem super promising though! And not needing an external program like pulover’s macro makes it more practical and less error prone.

Why do you use the mouse/cursor to press the snap button, by the way? It can be triggered with keyboard inputs just fine normally. Or is this something specific to how your branch of it functions?

Hi, the plugin re-trigger itself after each capture, it then advances the world tick, wait a bit for the rendering thread to stabilize, and then make another capture (and so on)

[GrayAxen]

@Roberto_De_Ioris

Hi, the plugin re-trigger itself after each capture, it then advances the world tick, wait a bit for the rendering thread to stabilize, and then make another capture (and so on)

Interesting, I’ll take a look at it then. If I get it to work in my projects it would save me a bit of hassle.

[raptoreagle]

@Roberto_De_Ioris

Hi, just to inform you that i have released a plugin for automating 360 captures via :

hello i just went through the thread… your plugin will be really helpful.

only problem is i don’t understand how to set it up and run it. The “how to use it” instructions don’t really explain anything to someone like myself. I will appreciate it if you could make a video showing how to set it up. I’m sure it’ll be useful for others as well.

thank you

[Ivan_Elizarov]

Good day!
I wanted to report that a new method has been created to create 2D, 180, 360 , mono and stereo videos. It allows you to quickly get the desired result.
If someone is interested, a link to the discussion and a link to the product.
https://forums.unrealengine.com/unreal-engine/marketplace/1595263-camera-360-system-blueprints

[anonymous_user_975308551 (anonymous_user_97530855)]

Going insane trying to get this to work with macro. Already downloaded iBrews tutorial to be sure that I wasn’t doing anything wrong but it’s all the same. My problem is no matter what I do i can’t seem to have any macro program register the “space” button to actually snap the shot. Other keys it will register. I have even tried to hardcode the location on a mouse and have it click it but the same story. I’m using 4.22 and launching the game in stand alone to get working.

Have you guys made this approach work with 4.22?

[Jenova723]

@anonymous_user_975308551

. My problem is no matter what I do i can’t seem to have any macro program register the “space” button to actually snap the shot. Other keys it will register. I have even tried to hardcode the location on a mouse and have it click it but the same story. I’m using 4.22 and launching the game in stand alone to get working.

Are you using the GeForce Experience Overlay or do you have it disabled? I have it enabled under my Geforce Experience Settings.

Also, I couldn’t get a good method to dish out renders with any macro, so i made this python script to press spacebar after each capture, automatically :

import socket
import pynput
import time  

UDP_IP = "127.0.0.1"
UDP_PORT = 5005  

sock = socket.socket(socket.AF_INET, # Internet
                       socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))  

from pynput.keyboard import Key, Controller
keyboard = Controller()
FrameCount = 0
TotalFrames = input("Enter total number of frames to capture")
print(TotalFrames)  

while FrameCount < int(TotalFrames):     data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes    
if data.decode("utf-8") == "Capture":        
  [INDENT=2]print(FrameCount)        
time.sleep(1.5)        
keyboard.press(Key.space)        
keyboard.release(Key.space)        
FrameCount += 1    [/INDENT]
  else:        
  [INDENT=2]print("Incorrect string received")[/INDENT]
  

And the PlayerCameraManager Blueprint :

As you can see i’m recieving a UDP message from unreal to simulate the spacebar press (which I got working with the udp plugin and Extended BP Library)

But after using this method I was encountering a lot of jittering in the video being rendered so I had to set r.PhotographySettleFrames to 30 (yes, it takes a lot more time) but it seemed to have solved the problem, but now producing a new problem, my skeleton mesh animations jitter a lot more.

Does anyone have a better method to capture? I’m almost about to give up and live with the jitters

[raptoreagle]

@Roberto_De_Ioris

Hi, just to inform you that i have released a plugin for automating 360 captures via :

Have you tried this?

[Jenova723]

@raptoreagle

Have you tried this?

Yes, but that plugin doesnt seem to be enabling Anti-Aliasing, I have also opened a ticket on the github page for that as well. I currently am using the python script to automate but another issue has popped up where the camera doesnt move along with the player because i never end the session, so for scenes where i need to move the camera I use the old jitbit macro method.

[y_mike1]

You can try Surreal Capture (https://www.surrealcapture.com), it is world first and only UE4 360 degree video capture software, it can plug into UE4 games and capture realtime 360 degree video from it.

[raptoreagle]

shame epic wont embed 360 capture in the engine

[raptoreagle]

for some reason the plugin is forcing my game resolution to 720p this makes capture slower than it should. i’m trying to force 4k but i’m not sure what i’m doing wrong…

[Jenova723]

@y_mike1

You can try Surreal Capture (https://www.surrealcapture.com), it is world first and only UE4 360 degree video capture software, it can plug into UE4 games and capture realtime 360 degree video from it.

1. It is not the first
2. It is not the only
3. I will give it a try and hope all post processing works correctly, will give feedback soon as i test.

Edit: It says it currently only supports fortnite…

[ssmohona]

Hi, I was trying to create a new post but can’t, so I am posting here. I am rendering HDR stereo using . I tried render images with HUD, I have set r.Photography.AutoPostprocess 0. but the rendered image is not showing HUD. How can I get rid of this?

[Davidaurecon]

Hi I have been using this method of capturing 360 video with the plug in and the macros and it worked really well !
until recently with the latest Nvidia drivers update it seems that is not working anymore in the Engine ! I can’t bring up the interface anymore.
this folder is gone as well : C:\Program Files\NVIDIA Corporation\ . This is extremely disappointing that they broke something that was working fine. Anybody found a workaround for that ?

[raptoreagle]

@Davidaurecon

Hi I have been using this method of capturing 360 video with the plug in and the macros and it worked really well !
until recently with the latest Nvidia drivers update it seems that is not working anymore in the Engine ! I can’t bring up the interface anymore.
this folder is gone as well : C:\Program Files\NVIDIA Corporation\ . This is extremely disappointing that they broke something that was working fine. Anybody found a workaround for that ?

uninstall the gforce experience and install an older one that worked or the one you used last.

[Davidaurecon]

@raptoreagle

uninstall the gforce experience and install an older one that worked or the one you used last.

Thanks mate.

Actually i found an even better work around, if we disable in-game overlay in GeForce experience the old plug in is back ! NVCameraConfiguration.exe is still missing tho.

[y_mike1]

@Jenova723

Edit: It says it currently only supports fortnite…

Surreal Capture software was improved quite a lot in the past two months. It supports to capture 360 degree video directly from a lot of more UE4 games now. Here is the youtuble channel that demonstrate the sample 360 degree video:

Surreal Capture

Here is the link to the Surreal Capture website: https://www.surrealcapture.com

[anonymous_user_de7201af]

@Davidaurecon

Thanks mate.

Actually i found an even better work around, if we disable in-game overlay in GeForce experience the old plug in is back ! NVCameraConfiguration.exe is still missing tho.

Davidaurecon I could kiss you For some reason ALT F2 wasn’t working but with your post that fixed all of that! Thanks bigtime.

[UE4_Archviz]

@ax448

Yup! (Sorry for the double post, but I figured this warranted a new one. )

Pulover’s Macro Creator can actually just check for an image every so often. Meaning no more setting a generous timer to avoid problems.
Right now it works like this.

1: Start up process
2: Manually open , and close it again. ( So that the next time it opens, it starts with the ‘done’ button highlighted. )
3: Script runs, sets settings to 360 stereo + desired resolution.
4: It starts the render.
5: pauses for a second
6: starts checking if the ‘Snap’ button is green again, once a second. (It’s greyed out while rendering.)
(It does this using the image/pixel search macro function. Allowing it to check a region of the screen for a specific image. like the snap button being green.)
7: once the render is complete, and the snap button is green. It waits for a few seconds.
8: Then closes , and presses tab, to make time progress by a frame.
9: then it loops.

The only thing I need to wait for now is step 7, when the render is done and where processes + saves the rendered image.
Because if seems to slow down the processing a lot if you’re rendering something at the same time (understandable.), and if render 2 is done rendering before render 1 is done processing, it’ll just throw out 1. On my pc, reading from and saving to a normal 2.5 ssd, it seems to take approx 4-5sec to save a 360 8x8k jpg. Massively longer for an .exr.
While if I render while it’s processing, it’ll easily take 10-15 seconds, aka long enough for the next render to interrupt it.

With 4x4k images I only need to wait around a second before progressing to the next render.

Still, that’s massively more flexible than the old way where you had to account for how long it might take to render.
Now it’s just waiting for the render to complete, then waiting a few seconds for the image to process, then on to the next one.
(It doesn’t matter if the render takes 5 seconds or 500 seconds.)

Image: how it’s set up.
At instruction 50: Space to start rendering.
51: wait a couple seconds, since there’s no point in checking if it’s done rendering at <10 seconds.
52: starts checking for the green snap button / the render being done, once every second.
53-61: After render, wait for a bit, close window. End of macro, so that it can loop once it’s done.

nice idea. can u make video on how u manage all stuffs, specially that image/pixel search macro function in process. and i have one big doubt. do this process support cinecamera.