Here is an video. Using mac sonoma, apple silicon. Smooth and flatten tool is broken.
It does look bugged 
How about you try a new, blank Unreal Engine project and test the landscape sculpting tools there? Let’s clear up if this issue is project-specific or a more general issue with Unreal Engine on your Mac. Also, try adjusting the engine’s scalability settings (weird thing to say, but maybe the tool is taxing too much on your Mac?). Go through the preferences related to landscape editing to see if adjustments there can help with the issue.
It could be an input issue, so do also check the mouse settings. UE might be interpreting the input incorrectly due to specific settings or sensitivities you are using. Or are you using a graphics tablet? Try using a different mouse or a tablet. It could also be a device specific issue … There’s so many possible reasons, so just try to narrow down the cause of the issue.