This is my take.
It’s all about c++ types and how the computer lays out objects sequencially in memory.
YourCustomGameMode is 100% identical to AGameModeBase in memory, before you added your new stuff. Just beyond that limit in memory is your stuff.
A pointer in C++ contains information about the object type it can point to, and its memory address. an AActor-pointer can point to the right place of an ACharacter object. But you can only access its AActor-part. But it’s the right place in memory.
Epic Games defined GetGameMode to deliver the memory address to an AGameModeBase object. Because your GameMode is based on AGameModeBase, it can point to the right place in memory, because that portion fits the definition completely. But you can’t access your new stuff through that pointer, because it is only allowed to access the specific memory layout of AGameModeBase.
Casting is a just a process of saying, “Look beyond the limits of this object in memory, and see if it fits with the new stuff”. If it does, you can use that part. Where your stuff is. Or where ACharacters stuff is beyond APawn.
I think of it more in terms of Casting Molds you use to cast iron. You’re taking the object you are getting and trying to fit it into another mold.