Im searching for a way to implement foveated rendering into UE4. I hope you can help me. Every idea is welcome.
What is foveated rendering?
In short terms: You use an eye-tracking device and you render your point of interest with high detail settings. Other parts with be rendered with lower details.
You can save performance by a factor of 5-6 on a fullHD desktop and you will see no difference from a normal fullHD rendering. (if you use a higher resolution or fov you will save more)
As clever as it looks, It doesn’t really mimic anything that happens in the real-world. Objects don’t lose detail when you look away from them, and your eyes will automatically do this when you look at a screen anyway and focus on a specific area of it. Also, what you may gain in overall memory requirements you would lose in memory speed anyway. Passing high, low and everything-in-between-detailed versions of everything in your scene every frame is going to take time, a lot of it. Alongside that, you have to do the actual calculation itself.
Screen-Space based effects would suffer massively too I’m betting. If you did want to implement it though, this would at least require a dedicated rendering path of it’s own, and not something you could integrate easily at all, and would require diving very very deep into engine source. How and where you would really apply this and at what level I don’t know, this seems more like a proof-of-concept than anything that’s really use-able, especially in a game environment where things tend to change very quickly.
You could fake the resulting effect itself with DOF Post-Process, but that wouldn’t improve performance. However, you can get an idea of how it would actually look. When I look at that reference image, I just feel like I’m looking through something. My eyes automatically blur everything around my focal point, I don’t need additional help with that
TheJamsh: I believe the technique is meant to draw the “unfocused” areas only in low-fidelity to save up processing power from the rendering of those areas that your eyes won’t even notice, thus improving performance
actually it reminds me a bit of HiAlgo Boost, which was available as a plugin for Skyrim.
what it did is that in case of low framerate with a lot of camera movement and/or stuff moving on the screen, it would dynamically reduce the game’s resolution, assuming that stuff happens so fast your ever won’t notice the change. aside a few postprocess conflicts it worked quite well if it was configured not to be too drastic
In the OP however he talked about using Eye-Tracking software, so I figure that area of the screen would change every frame. Sounds like a monster undertaking one way or the other though.
I don’t think there’s any reason why UE4 couldn’t be extended to support foveated rendering - but of course you need cameras capable of fast enough eye tracking, which is the real problem.
It might take a good amount of work to make it render the view 3-4 times to small buffers, but there shouldn’t be any reason why it can’t do it at all.
@TheJamsh - foveated rendering certainly works if you have eye tracking cameras fast enough to keep up with the eyes’ saccades. There is a lot of research with testing that shows how to make it work, the key being to have cameras that are fast enough that the fovea region only ever sees the high resolution “spot” in the middle as it moves around.
In addition to research there are commercial systems out there for high density data rendering (i.e., stuff like medical imaging, where you need to navigate a 3D mesh with way too many verts in real time). But those are supported by specialized monitors with fast eye tracking cams.
The paper from microsoft shows that with a fast eye-tracking device (300Hz) and a resolution of 1080p you can have a 4-5x performance boost and you see no difference from a normal rendering image! Its massive. With a 4K resolution you will have an even higher performance boost.
Your Eye can only see max. 2 Degree of your field of view. This is your focus point which will be rendered with high resolution. The Eye can move up to 900 Degree per Second and while it moves you see nothing.
Foveated Rendering is possible and it will be important for the Oculus Rift and other HMDs, because you will never render 4K at 120fps. (i have an oculus rift with integrated eye-tracking)
I have the hardware my problem is the software side. The Problem is that i need to change the renderer…
If I can setup cameras with different resolutions and different quality settings, i can render 3 images and try to blend them together.
UE4 has the option to do something similar for VR, where it renders the distorted peripheral pixels at a lower resolution than the central image. I would start looking at that implementation if you want to extend it to include eye tracking. I wouldn’t expect it to be integrated by Epic any time soon though, considering how niche the application is due to the need for specialized hardware in addition to really high framerates and resolutions before it becomes useful.
Foveated rendering is only a niche application due to the lack of consumer eye tracking hardware. It’s only a matter of time before VR headsets come with eye tracking, and foveated rendering for VR can really help reduce system requirements.
Well VR desperately needs foveated rendering… the GPUs of today are not powerful enough to properly handle it without some serious optimizations and compromises.
The eye tracking tech is not included in the Oculus by default as far as I know though is it? it looks like they are using their own tech in that demo and you need to upgrade your kit through them… and it could be super expensive.
if you have a Vive Pro Eye headset then you can do Eye-tracked Foveated Rendering with Unreal Engine.
At the moment you need to use a custom modified version of Unreal Engine of versions 4.21 or 4.23 but as HTC Vive we are working together with Nvidia and Epic Games towards an integration so eventually you will only need a plugin to enable this.
Although this is now a 7 years old thread on foveated rendering, in this years the progress of foveated rendering is very less. While the earlier foveated rendering was focused on polygon simplification, the latest trend moved toward pixel shader, and ray-tracing. I would like to mention, still proper tools and software support is inadequate.
