In my experience of working in VR, ambisonics are mainly useful for ambiences which don’t have to deal with the finer points of rotation, distance etc. like an air conditioning hum, or background traffic ambience etc. They behave oddly and I don’t think (though I could be wrong) that they’re supported in the way you’re trying to use them.
Mono files do spatialise really well, and you have to use mono for a point source that is then spatialised. The standard thing to do is simply to use a mono file, and then have it spatialised ambisonically within the engine.