Try exporting using the Alembic format. FBX will only work with skinned meshes so using something like vertex animation will not export. But Alembic will exprot this type of animation data. Check out the following link for more information on how to use this.
This means that your mesh is an ngon and it needs to be triangulated to work in UE4. You can do this in 3Ds Max by putting the Turn to Poly modifer on your mesh and setting the max side(I think that is what it is called) from 4 to 3. This will make sure that all geo in your mesh is triangulated.