Ok, I feel kind of stupid now. My way of creating dedicated server was totally wrong. I made the normal build (from engine, Package) and then build the server code from Visual Studio and pasted it into WindowsNoEditor/Binaries/Win64 folder. Of course the content was cooked for the client …
For anyone else strugling with this, the solution is very simple: package a game specifically for the dedicated server and do not use that paste trick (unless you are just testing things out). Here is some reference:
How_to_package_your_game_with_commands
In my case, I am building the source as well as packaging for the server and client at the same time: this can be achieved using something like:
path_to_UE4_code\Engine\Build\BatchFiles\RunUAT.bat -ScriptsForProject=D:/pathtoproject.uproject BuildCookRun -nocompileeditor -nop4 -project=D:/pathtoproject.uproject -cook -stage -archive -archivedirectory=pathwhere toplacebuild -server -package -clientconfig=Development -serverconfig=Development -ue4exe=UE4Editor-Cmd.exe -prereqs -targetplatform=Win64 -build -CrashReporter -utf8output -compile
The server size went down from 700 MB of RAM to 110 MB + it loads up in less than a second.
Thanks for the help, that StipFlags put me on the right track.