I just tried shooter game and effects cave with my Mac Pro (2008), and those works fine (I don’t expect 100fps or anything like that with 6 years old machine). My graphics card is Nvidia GTX670 (card is not officially supported by apple but it works fine in mavericks).
HD5870 is very old graphics card and I think it might cause some performance problems in your case. Btw, did you noticed that system requirements for UE4 requires HD6870 or higher?