Little update:
quartz clock works fine for me. The problem is for my use case its a bit limited and complicated to handle. (For example its not easy to determine where exactly in the song i am right now if i want to do some special effects on specific drops/parts)
I might have found a solution for the sequencer as well which seems to work.
In the settings of your audio set it to “retain on load”.
Then use the “prime sound for playback” node and choose the audio. From the sound loaded event you can start the sequencer and it should be in sync, even if you put the audio directly into sequencer.
I’m still a bit worried that it might not be 100% in sync all the time but i’ve sent simple test projects to some friends and the delay seems to be gone.
TLDR: set the sound to retain on load and then prime the sound before playing it.