I should dig into the internal UE4 socket code to see what is going on exactly. It seems like the client side just assumes the connection has failed for no good reason.
I do wonder if this also happens when connecting to other servers, or when other clients connect to the server.
What happens when you make a websocket client (disable TLS) connect to your server? Does it handshake?