How to prevent world position offsets in materials from distorting meshes?

Or just wait for the right person to pass by.

Technically no. You cannot use the same way for flowers.
practically yes… With the help of vertex paint and a ton of work.

You paint your flower mesh properly with at least 2 colors. One for what the wind affects that fades from white at the top to 0 at the bottom. Anything colored in 0 will not be affected by the wind.
And the other color to things that need to distort or move uniformly.

Within the material you can use both masks to isolate what would be affected by the WPO or not.
generally material solutions on flowers do Not look pretty. but if you combine the 2 techniques they can still be passable.

The theory is simple, if the object is painted with the second vertrx color you shift All the vertices relative to the normal between character position and themselves (absolute world position). You just translate those without any rotation (or very minimal)
Additionally, you can layer on the translation down the line using the other vertex colors thats used for wind.

When that fails or doesn’t give you the result you need, you can have a look at something like my plugin for foliage. Dynofoliage. It works with skeletal meshes and PHAT assets so it is definitely more accurate then the material method can be without Major work…