Alan Wake "Taken" effect

This is a combination of effects. Your idea of having distorted particles is good. Plus added black smoke effect spawned on the mesh itself. To get the distorted silhouette you could render the characters to the custom depth and use a distorted version of it post processing effect.

Tom Looman has a good article on custom depth: