Okay, the only way I’ve managed to make it work was by switching Blend Mode to be Blend Mask, then creating a Blend Mask on the Skeleton (there’s a small cog in the right top corner of the skeleton editor, you can add it there, and set all bones you want to be affected to 1, while the rest should be at zero).
Then you go back to your Layered Blend Per Bone, and specify the mask you’ve created in the Blend Masks section.
Additionally, when porting my project from UE4 to UE5 apparently the slots didn’t transfer properly. I had a custom Slot Group, with a custom slot. When the project was transferred, it created the slot, but not the slot Group (so, it was under the default one). And that screwed everything up. So, my recommendation is to also double check the naming of slots and slot groups.
It only took me 6 hours of my life and one crying session in the bathroom to figure this one out, thanks Unreal xD