NVIDIA GameWorks Integration

[LexCurtis]

I tried to incorporate WaveWorks into latest 4.21 version by studing 4.19.0 version with waveworks integrated. By using VS 2017 I searched entire solution for WaveWorks start - end comments. Copied every bit of code from waveworks 4.19 into 4.21. Got every *.Build.cs file right, run GenerateProjectFiles got warnings which I fixed, although they were just warnings. Learned that integration is real mess. What I get at the end is entire day spent of staring at stupid source files and compiler unable to find include files. What am I doing wrong? 0lento, what kind of magic are you using to integrate any part of GameWorks?

[Alexey.Panteleev]

Also, looking at your image again, I dont have soft shadows on the walls and ground plane (not even if I scale the box smaller). Any idea what could be the issue? With what VXGI version have you created your image if I may ask?

I used the latest 4.21 + VXGI 2.0.1.
Have you tried switching the lighting modes after hitting Play? Use keys 1,2,3 and M - as hinted in the on-screen text.

[Alexey.Panteleev]

I tried to incorporate WaveWorks into latest 4.21 version by studing 4.19.0 version with waveworks integrated. By using VS 2017 I searched entire solution for WaveWorks start - end comments. Copied every bit of code from waveworks 4.19 into 4.21. Got every *.Build.cs file right, run GenerateProjectFiles got warnings which I fixed, although they were just warnings. Learned that integration is real mess. What I get at the end is entire day spent of staring at stupid source files and compiler unable to find include files. What am I doing wrong? 0lento, what kind of magic are you using to integrate any part of GameWorks?

git merge, I guess?
That's how I usually update the VXGI integration to a new engine version. Create a new branch from the existing integration, merge the new engine release into it, resolve conflicts, fix whatever is broken.

[DriftlessAI]

Then something else will break I assume, because that line just checks if the compile succeeded.

Can someone experiencing this texture limit issue share a simple project that would reproduce it? Thanks!

A marketplace project that reproduces this issue is the Third Person Shooter Kit. It will compile over 10,000 shaders, then fail with only a few hundred remaining. I'm guessing that commenting out that line #122 skips compiling the bad material. I haven't been able to notice any impact from the change while testing the project yet.

I'm not sure how to tell which material is causing triggering the error.

If you have any tips on that, I'd be willing to check it out on my end.

[DriftlessAI]

I tried to incorporate WaveWorks into latest 4.21 version by studing 4.19.0 version with waveworks integrated. By using VS 2017 I searched entire solution for WaveWorks start - end comments. Copied every bit of code from waveworks 4.19 into 4.21. Got every *.Build.cs file right, run GenerateProjectFiles got warnings which I fixed, although they were just warnings. Learned that integration is real mess. What I get at the end is entire day spent of staring at stupid source files and compiler unable to find include files. What am I doing wrong? 0lento, what kind of magic are you using to integrate any part of GameWorks?

Integrating the 4.19 branch of WaveWorks into 4.21 is much more difficult than integrating it into the other 4.19 gameworks branches . There are significant changes that need to be made to handle the refactoring in BasePassRendering.h TranslucentRendering.cpp and other files.

[Leo Rakes]

git merge, I guess?
That's how I usually update the VXGI integration to a new engine version. Create a new branch from the existing integration, merge the new engine release into it, resolve conflicts, fix whatever is broken.

Hello, compilation error is one of the problems, it seems that many people have encountered this problem.

Secondly, the VXGI voxelized material in 4.21 will occupy four more maps, which leads to many materials that use a lot of maps beyond the upper limit of 16 and become unavailable. It seems that there is no problem in previous versions.

And there are about 500-1000 shaders that don't seem to be cached, which are always recompiled every time a project is opened?

[0lento]

0lento, what kind of magic are you using to integrate any part of GameWorks?

I always start with "git merge --squash branch-to-merge", I favor squashing always for the initial merge as it's easier to solve the conflicts at once, I can merge or cherry pick individual commits later on.

On bigger engine changes, like you get from 4.19 to 4.21, what often happens is that some directories move around or some contents get internally moved into other files in the engine. These are always a bit tricky but when you know what to look out for, you can for example do diffs for the puzzling stock ue4 files from 4.19 to 4.21 in this case. For things that get into different files, there usually are only small amount of such things and in these cases you fix the issues on the moved content usually fastest by manually arranging the files.

If you are having issues on figuring out what all the merged tech is changing, it's often also beneficial to have clean one commit merge for the branch made for the engine version you are starting the merge with. You basically make an extra branch with clean stock ue4 release as basis, like in this case 4.19.2-release and then merge squash WaveWorks-4.19 to it so you can examine all changes it did to these files easily from single commit (there will never be any conflicts to solve when you do this step, so it's pretty trivial). Do note that it's fine to merge from 4.19.0 based WaveWorks on top of 4.19.2 as you will not get UE4 side changes on your commit in that case, git is smart enough to skip irrelevant parts. But if you accidentally merge like 4.19.2 Waveworks on top of 4.19.0-release you get also all stock UE 4.19.0->4.19.2 changes on top of the things you wanted to merge and it messes up the commit totally so never do it this way. I usually favor the latest release on the major version for this exact reason, it's pretty foolproof.

Often engine API functions / macros change a lot. While some are trivial to figure out, sometimes you have to do some detective work and see how Epic changed their own systems. Basically in cases like these, I try to look a function/class from the old engine that uses the same thing that throws compilation errors on newer engine, then I check how that same stock ue4 file looks on newer ue4 version and make similar changes to the actual merge branch if possible. This is REALLY common way for me to solve almost all build.cs errors and warnings but it's also required on many systems that I don't fully understand (UE4 codebase is huge and you'll never understand all of it even if you try, not to talk about GameWorks techs internals as they only have integration, not the full source code).

You also often have to fix bunch of include files as these structures evolve a lot from release to another. Often the directory fixes are easier to make by simply adding the needed include paths on relevant Build.cs files but this varies per case basis.

Sometimes there are some ENGINE_API etc macros that expose the function across that specific module that aren't exposed but GameWorks rely on them being exposed so you add those macros in right places. Sometimes you have to search online (forums/answers) or seek wisdom on Unreal Slackers on Discord to find other peoples experiences on the new UE4 version or to get help on compilation issues you still couldn't solve on your own. There are always some common fails for people who port their old work to the new engine and solutions they came up with. I could go on with this list as there's always new things to solve in a different way on each new release. To get some idea what needed to be done on 4.19->4.21, just take a look at commits starting with "4.21 fix" from here.

The work amount varies a lot based on the engine side changes. Some techs drop in quickly, some an require many days of work (mainly because UE4 compilation is so slow).

[0lento]

Alexey.Panteleev since you are around, I'm going to ask if you know what's the status for 4.21 Flex atm? Is it something we can expect from Nvidia?

I did take a look at porting it myself but it would require a lot of physics side changes (4.21 basically hid old physx calls and now you have to operate through new wrappers and interfaces. I'd rather leave this specific changes to people who designed the integration to fix, just in case you guys want to change something along the way now that things work differently.

[Alexey.Panteleev]

Hello, compilation error is one of the problems, it seems that many people have encountered this problem.

Secondly, the VXGI voxelized material in 4.21 will occupy four more maps, which leads to many materials that use a lot of maps beyond the upper limit of 16 and become unavailable. It seems that there is no problem in previous versions.

And there are about 500-1000 shaders that don't seem to be cached, which are always recompiled every time a project is opened?

I reproduced the compile error problem with the Elemental demo, and fixed it. Here's the commit: https://github.com/NvPhysX/UnrealEng...ec5e702667646b
Epic is declaring samplers in their shaders very generously, although there is only 16 of them available on DX11...

The uncached shaders could be caused by those compile errors actually. Please try the latest version.

[Alexey.Panteleev]

Alexey.Panteleev since you are around, I'm going to ask if you know what's the status for 4.21 Flex atm? Is it something we can expect from Nvidia?

I did take a look at porting it myself but it would require a lot of physics side changes (4.21 basically hid old physx calls and now you have to operate through new wrappers and interfaces. I'd rather leave this specific changes to people who designed the integration to fix, just in case you guys want to change something along the way now that things work differently.

I don't think there's any work being done in that direction currently.

[LexCurtis]

Thank you guys for answering my questions!

[FatmirVaraku]

There's no publicly available official integration between the two. For now Nvidia only shares these techs separately and for things to interact, they'd need to distribute them as merged repo and make the needed changes for HairWorks. Also I doubt having HairWorks working on VXGI is any kind of priority for them

Thank Olento

[Leo Rakes]

I reproduced the compile error problem with the Elemental demo, and fixed it. Here's the commit: https://github.com/NvPhysX/UnrealEng...ec5e702667646b
Epic is declaring samplers in their shaders very generously, although there is only 16 of them available on DX11...

The uncached shaders could be caused by those compile errors actually. Please try the latest version.

Thank you for the repairs. I downloaded and tested them. Now everything seems to be working. VXGI can finally run normally in version 4.21. Thank you for your work.

[enoes]

What is the VXGI compositing mode command ??

[porckchop]

Wow still nothing on 4.21