Hi PsychicFreak,
I was able to reproduce this issue in UE4.
I am going to link you our lighting troubleshooting guide as well as a guide that gives a great, in-depth, explanation on how to set up lightmaps for UE4 and how they are intended to work.
Link to Troubleshoot :
Link to Light Maps,
There are a few things you can do to increase the calculation of your lightmaps.
1.) Min Lightmap Resolution,
2.) Engine Scalability Settings,
3.) Amount of Illumination on the object.
4.) Project Settings > Rendering > Generate Mesh Distance Fields
Have a look at the documentation and try and see if any of these are a viable solution to your project.
Thanks,