If I change something in the config/*.ini files, do I have to recompile?
Or are the ini-files only read at runtime?
Those are default config which is used on first run of your game, they are run time ofcorse. Compilation config is in build scripts and UBT config in engine directory somewhere in Saved