If you check at the BP_Ocean, in the parameter section called Ocean Manager, you will find the parameters which deals with landscape modulation. Those parameters will tell how the ocean material will behave to flatten the waves the more it approaches the terrain. Of course that if you are doing with a completely different landscape, the proper parameters for that should be feed, like Landscape (your current map) and the Heightmap Texture which should be an image of you height map obtained from the software you used to create your landscape (ie: WorldMachine). Also the Z for the landscape is used and I would tell you to notice it on the example map aswel.
Let me know about your findings after the hints above.