Painting material in nanite messes with my landscape height

Hi, I have an issue with my material painting. When I paint my material on a flat surface all those weird bumps appear and heights are messed up everywhere:

When I go to sculpting, one click makes all the area suddenly go to what it’s supposed to be:

But when I’m building/rebuilding the data or saving it switches back to those bumps. I follow the tutorial but apparently messed something up with displacement/height settings or layer painting. I have no idea what this is however.