How can I create custom categories in GameUserSettings.ini?

It’s not, the error that you’re getting is because hard references (raw pointers and TObjectPtr) are not supported as config properties.

Those examples actually prove exactly the opposite. Those are completely separate classes, not object instances nested inside a game user settings. If they were nested, you’d see the values in a way that would be similar to structures. Not with separate sections.

One possible setup:
A UMasterGameSettings derived from UGameUserSettings. This would be the class that is configured in DefaultEngine.ini.
Other settings classes are derived from UGameUserSettings. Not referenced by DefaultEngine.ini.
UMasterGameSettings has object references (not config) to the other settings classes.
In the constructor, UMasterGameSettings creates default subobjects for each of the object references.
You would then have to override various virtual functions on UMasterGameSettings (ApplySettings, SaveSettings, etc) and call the matching function of the subobjects.

But I don’t know if that’s really any better than just using structure properties that are just marked as config and then just work. But it’s all about tradeoffs.