Ok, I figured it out. As of 4.24, the following flag was changed to false.
[Upgrade] bLegacyPublicIncludePaths = false => Omits subfolders from public include paths to reduce compiler command line length. (Previously: true).
TheKaosSpectrum was pretty much right. However you also need to include the module in your path:
Example:
Pre 4.24:
#include "Abilities/Core/AbilityComponent.h"
Post 4.24:
#include "Game/Abilities/Core/AbilityComponent.h"