Yes, there are limitations to what you can pre-bind as a delegate parameter for use later when it’s invoked. The underlying structure of the delegate (probably) requires all the data to be default constructible and assignable, two things you can’t do with a reference member.
In a way this also protects against naive implementations that would easily cause crashes by passing references to local variables who’s memory is no longer valid when the delegate is invoked. You can cause the same problem with a pointer type, but at least then you’re having to be explicit in a way that (hopefully) makes you think about the value that you’re binding to the delegate.