It’s not that simple. There’s casting where is creates no harm whatsoever and is super useful; and there’s unnecessary casting that loads object into memory without you realising it ← that’s the one to be avoided. And it’s not about casting per se, it’s about it creating hard references.
In order to access the save game data, you must load it anyway, so just cast. If you do not want to cast, use an interface. But it’d be more of a convenience thing, to be honest. Also, look into soft references.