So I am starting to familiarize myself with the (new) tools in UE5 and now that I finally have compatible hardware, one of those tools would be raytracing. I followed a bunch of tutorials, and managed to get raytracing to work for at least opaque objects and closed translucent surfaces in what I believe is the latest version of UE5 (the one where the material settings demand you to actually use the refraction input again rather than the weird way of uding the specular). When the scene is not too demanding, the results are pretty cool.
However, when I try to apply raytracing to a bigger environment (a natural one to be more precise), everything basically breaks, and I am wondering how I should even approach a raytrace setup on something like this.
In the picture I have attached, the upper image shows the scene without raytracing enabled, the lower picture shows what happens when I turn translucency from raster to raytracing. In the corner image I have included an extra screenshot where I applied what should have been raytraced water, as well as show off just how broken the shadows are now with raytraced translucency on. The rocks mostly affected are meshes with Nanite enabled.
As for materials for the water, I have tried both the normal setup (the one with forward shading) and one with a thin translucent shading model. Neither gives anything resembling refraction, or even reflection. The trees, which uses masked blending and a two sided foliage shading model, and the cloud platforms, which use subsurface scattering, also turned ugly. The Nanite meshes get this weird shading like they suddenly started wearing camo or something. With this many things breaking at the same time, I am wondering what I did wrong. I have tried fiddling with various settings, but without any positive results.
So, does anyone have any pointers here?