When to run on server and when on client?

Apparently almost everything except cosmetic effects such as gun flashes needs to be spawned on the server to prevent cheating.

However there are still things that I don’t understand at a logical level

Basically making crucial things happen on server functions instead of client helps to prevent cheating. But if cheater can call a function on client then why can he call a function marked Server? E.g. He can call a function marked server give health which gives health to player , basically he’s calling it because its possible to call it so he isnt doing anything illegal directly.
Why would you mark anything crucial as Server function if the client (Hacker) can call it anyway? **

Checking authority does safeguard things but without it what do server only functions achieve in terms of security?

EDIT: as of cosmetic effects one can even draw a radar or overlay indicating enemy player positions (Known as aim botting) and technically he isn’t violating any server rules since client can do anything on its machine, is using 3rd party anti cheat systems like punkbuster the only way to stop those? or are there other things

Well, this is what _Validate function is for. You can perform all sorts of sanity checks of the input so that theoretical ‘hacker’ can’t add a million health points or set his maximum speed to infinity or whatever.

but he can simulate things like add health when he’s at 30% health to get to 90% health (assume a legit health pickup gives 60% health) , In short its not feasible to make like a hundred possible checks to find out whether the call was legit without unintentionally throwing off legit players mistaking them as cheaters.

Also how to you prevent client side only modifications that could mean cheating like overlays and radars that do not effect things on server

There is no way to prevent every possible misbehavior as long as your client is not a complete dummy-proxy that can not even call RPCs. But this would result in a very boring gameplay due to the huge amount of restrictions such behaviour would apply. Security is a very broad topic and you can find a lot of good articles on it on the web.

You save all players data and run through it every day (locally) by filtering for suspicious records then act accordingly.