We’re actually working on updating our QA process as we speak! It definitely hasn’t been perfect in the past, and we’re working to buff it up.
One huge step up is getting a dedicated QA engineer in place with our VR team, so that issues can be identified and corrected much more rapidly. Historically, there have been a few things that have contributed to bugs like the ones you mention above:
Lack of Test Coverage:
We’ve admittedly had major gaps in the test coverage previously. This includes 4.8, and the bugs that you mention above are evidence of that. Thanks to our embedded engineer, we’ve gotten a nice long list of things to verify across all platforms this time around. We did the same thing with Blueprint tests not too long ago, and we’ve had great results internally with catching regressions and fatal errors much earlier. This should help make sure platform support is much more consistent.
This is one of the bigger issues we had in 4.8. The timing of hardware providers SDK releases doesn’t always match up well with our engine releases! This was doubly true for 4.8, and we ended up taking merges very late in the game in order to make sure we supported the latest SDKs and runtimes for the engine cycle. There is a limitation, due to binary compatibility, where we can’t change the outward facing APIs of any modules in hotfixes / QFEs / etc. So, we hedge ourselves against this by sometimes taking SDK merges very close to the release date. While we feel this is the right tradeoff, it can lead to issues like the tabbing and window one you mentioned. We need to get better about that, and weigh the pros and cons. At the very least, as you mention, we need to get quicker turnaround on fixing those issues as they’re discovered.
There were a few issues this release that were caused because, well, you’re dealing with pre-release software and hardware Not everything functions perfectly, because you’re dealing with the bleeding edge. Please be very aware of this as you update your runtimes, etc. Even minor revisions sometimes have API breaking changes. We try to keep on top of this as best as possible, but we don’t always get time to do a full verification pass. To that end, we’ve worked to deepen our relationship with Oculus, to make sure both of our teams are running QA regression passes against both releases. This has already borne fruit, so we have hope moving forward this will be a bit better.
In summary, you’re right! But hopefully the above lends some insight to how we’re trying to make it better for you guys. We’re trying to balance keeping everyone on the bleeding edge, and keeping everyone stable. I think we missed the calls a few times there, but I’m hopeful evolving our process will make that a less frequent occurance.