This question is a bit old now, but here are some ideas and some info that will hopefully be helpful to someone.
The fresnel node mentioned in the original post should (at least partially) work, assuming that the character’s model isn’t flat. (It’s hard to tell due to it being a silhouette in the screenshot.) The fresnel node isolates triangles that are oriented obliquely from the camera’s perspective, which, for many models, are primarily the triangles along the outside edges (this is especially true for a sphere). Maybe that’s not the case for the ‘shadow’ character’s model, or maybe there are other factors that I’m not aware of.
In any case, if the fresnel node doesn’t provide usable results, you may be able to use a texture to indicate areas that should have reduced opacity. (Or perhaps even vertex colors, but I’m less familiar with that approach.) Unfortunately, this will only work as long as your model always faces the camera from roughly the same direction; if you need the model to rotate or be seen from arbitrary 3D angles, this effect will quickly fall apart.
One way or another, the trick is to inform the renderer which triangles should have reduced opacity, and which shouldn’t. This is tricky because the renderer has no concept of which triangles are ‘outer’ (near the ‘edge’ of the silhouette) and which are ‘inner’. Primarily, this is because each pixel is rendered independent of every other pixel — the model’s material shader has no clue that the model has multiple triangles, where those other triangles are, what the overall shape/silhouette is, and so on.
Post-process shaders are a special case, because they are to sample from previously rendered buffers. Edge detection algorithms are limited to being post-process effects because they need access to multiple pre-rendered pixels simultaneously to determine where sharp changes (edges) occur. Unfortunately, as stated above, post process shaders aren’t useful for every case, because they do their business after all the standard materials have already calculated their color values, including opacity.
One final suggestion for the original poster, or for anyone doing a similar smoke effect: whichever technique you use to isolate the ‘outer’ triangles, try playing with a procedural noise node (in the material graph) to create some animating smoky randomness in the opacity. It may help further sell the effect and help distract viewers’ eyes away from the model’s edges. (Or maybe not! Sometimes you can’t know whether something will work until you see it.)
Best of luck!
