Does Epic have a QA process in place for VR?

The reason I ask this question is due to the state of the 0.6.0.x plugin as well as previous issues.

Im starting to get the impression that quality assurance for VR either does’nt exist or basically consists of slapping a HMD on with a single machine, looking around and going ‘yeah it still works’.

So my question is what sort of process does Epic currently have in place?

I’m a bit disappointed, too.

On the one hand, great that they are implementing SteamVR/Vive stuff now.

On the other hand: there are, what, like a hundred, Vive dev kits in the wild - and I have the feeling, they have already forgotten about Oculus and the 100 thousand DK2 dev kits…

On the third hand: I have to admit that every new Oculus SDK release gets worse than the one before (and quality dropped dramatically after the facebook deal). So - I don’t know if it is Epic’s fault at all…

Of course, things like this shouldn’t happen: VR Preview not working when Blueprint is open - XR Development - Unreal Engine Forums
Or be fixed ASAP.

Maybe we should point Epic’s staff to this site: https://www.unrealengine.com/vr-page
You can find it on the main page under “About” :stuck_out_tongue:

We’re actually working on updating our QA process as we speak! It definitely hasn’t been perfect in the past, and we’re working to buff it up.

One huge step up is getting a dedicated QA engineer in place with our VR team, so that issues can be identified and corrected much more rapidly. Historically, there have been a few things that have contributed to bugs like the ones you mention above:

Lack of Test Coverage:
We’ve admittedly had major gaps in the test coverage previously. This includes 4.8, and the bugs that you mention above are evidence of that. Thanks to our embedded engineer, we’ve gotten a nice long list of things to verify across all platforms this time around. We did the same thing with Blueprint tests not too long ago, and we’ve had great results internally with catching regressions and fatal errors much earlier. This should help make sure platform support is much more consistent.

Update Timings:
This is one of the bigger issues we had in 4.8. The timing of hardware providers SDK releases doesn’t always match up well with our engine releases! This was doubly true for 4.8, and we ended up taking merges very late in the game in order to make sure we supported the latest SDKs and runtimes for the engine cycle. There is a limitation, due to binary compatibility, where we can’t change the outward facing APIs of any modules in hotfixes / QFEs / etc. So, we hedge ourselves against this by sometimes taking SDK merges very close to the release date. While we feel this is the right tradeoff, it can lead to issues like the tabbing and window one you mentioned. We need to get better about that, and weigh the pros and cons. At the very least, as you mention, we need to get quicker turnaround on fixing those issues as they’re discovered.

Beta Software:
There were a few issues this release that were caused because, well, you’re dealing with pre-release software and hardware :slight_smile: Not everything functions perfectly, because you’re dealing with the bleeding edge. Please be very aware of this as you update your runtimes, etc. Even minor revisions sometimes have API breaking changes. We try to keep on top of this as best as possible, but we don’t always get time to do a full verification pass. To that end, we’ve worked to deepen our relationship with Oculus, to make sure both of our teams are running QA regression passes against both releases. This has already borne fruit, so we have hope moving forward this will be a bit better.

In summary, you’re right! But hopefully the above lends some insight to how we’re trying to make it better for you guys. We’re trying to balance keeping everyone on the bleeding edge, and keeping everyone stable. I think we missed the calls a few times there, but I’m hopeful evolving our process will make that a less frequent occurance.

Thank you Nick. This kind of profound and open answer is why I love Epic so much!
I’ll sleep a lot better tonight :slight_smile:

Just to add some context, from Oculus’ side:

https://forums.oculus.com/viewtopic.php?f=60&t=24683#p283597

https://forums.oculus.com/viewtopic.php?f=60&t=24843#p285050

I really hope that you guys at Epic and Oculus can somehow manage to align your work-flows a bit better in the future…

Quite frustrating to see improvements that have been asked and long awaited by the community finally being implemented by Oculus but then being rejected by Epic, or important bugs that have already been fixed pushed to a major new engine release instead of the next hotfix.

I know there’s a price to be working on the bleeding edge, but still… :stuck_out_tongue:

For the black screen issue, we’ve taken that integration. However, due to Epic’s policy on hotfixes, this one didn’t qualify for integration into a 4.8.x release. It will be in 4.9 though, along with the fixes for the BP tab blocking VR preview, which Mike Fricker on our side just fixed today.

As for the camera work, it wasn’t aligned with how we wanted it in the framework. With Oculus, we decided that people could pull it from their GitHub if they wanted, but that we wouldn’t integrate it into main. It would have required people to move to the Oculus solution for a version or two, and then break when we moved to the internally developed solution. Unfortunately, they announced their alternative before we had a chance to vet it, so people got excited for something that wasn’t ready yet :frowning: Again, apologies on that. We’ve now started regular engineering sync-ups bi-weekly with the Oculus SDK team engineers, so hopefully that will be a one-off. We’re much better aligned now with our priorities and workflow, so hopefully it prevents other false starts in the future.

Thanks for the reply Nick, that’s great to hear.

I know this is a complex integration process, especially when you factor all the other HMDs coming out.

Looking forward to 4.9, keep up the good work!

Yes, that’s my impression, too. Epic didn’t even feel the need to patch the broken VR preview (which is broken since the preview phase of 4.8) altough there is a fix available which basically just moves 2 lines in the source: [4.8] VR preview sometimes doesn't work, bad GPU-utilisation - XR Development - Epic Developer Community Forums

Instead, we are told to integrate this patch for ourself via a GIT cherry pick and then compile our own version of the engine to get the basic stuff working again (why can’t we just replace the broken plugin DLL?).

I really appreciate all your work in terms of VR but to be honest, ‘Epic’s policy on hotfixes’ should maybe be reconsidered when a obviously broken integration did qualify for release (as it did in case for the now-broken Leap Motion plugin) but the corresponding simple fix isn’t allowed to be released.

Yes, UE4 is complex as hell but you really can’t be serious to hold back essential bugfixes for weeks and weeks until the next major version (and then tellling people that a project-upgrade isn’t recommended at all). If you got working fixes for critical problems, please release them as soon as you can.

Sorry for sounding this negative but I just had to get this off my heart.

And yes, working with VR and cameras is still the usual nightmare, so I am very glad that this will change for the better soon (although it might be to late for our project).

Phew, sometimes the bleeding edge can really hurt. :wink:

When are we looking at getting 4.9? I have spent more time re-building and getting the project to work in the latest build from 4.4 i think. 4.8 was the last headache as i seem to spend more time trying to get everything working again before another build is released. I know its classed as beta, but for a dev - it would be great to have future builds so the user can upgrade aspects of the new build and choose to leave some in order to transition much smoother.

But apart from that, U4 is great :slight_smile:

vs.

(4.9 can’t even *detect * a HMD right now in a reliable way, instead it will always return TRUE if the HMD was enabled on editor start, even when explicitely starting a non-VR-PIE-session) and it looks like, Epic will be just keep it in this state for 4.9.

EDIT: Yep, 4.9 is out now, this was not fixed and there is ‘no timeframe’ to get isHeadmountedDisplayEnabled() working again.

So much for that.