Time for some uncomfortable criticism and a sweet *** solution

General Feedback & Requests
1

So this is how you guys seem to be doing things currently:

  1. Write feature
  2. Half-*** test it
  3. Release it without really knowing if it works or not
  4. Lose sleep(?)

But harken! There is a better way!

  1. Select feature you are about to add/improve/repair
  2. Write unit test for feature
  3. Add functionality to feature that passes unit test
  4. Add unit test to automated test list
  5. One day in future see that unit test is failing. Don’t release broken product.
  6. Release GOOD product.

This is how come your **** doesn’t work and why for you released three showstopping bugs for iOS in 4.19 that you still don’t know about and haven’t commented on.

I want to help you because bad developer culture is nobody’s fault. It just creeps in. But it is your fault when someone brings it up and you don’t fix it.

So let’s go over some examples. Right now you can package and launch a brand new iOS project but you can’t package and launch existing projects from 4.18. I honestly don’t know what the regression is, but I bet a unit test would help us figure it out! Here’s what you do:

  1. Go back to unit test you already wrote (good developer, you get a cookie)
  2. Expand test with new test criteria and validate that it now fails as per user report
  3. Fix feature
  4. Test passes
  5. Forum sings your praise and you sleep like a baby

Your unit test codebase should be as large if not larger than your actual codebase. Is that more work? Well, yeah, at the beginning. But once your framework grows colossal like UE’s is it starts saving you hours upon hours of developer time and frustration. It also requires little to no maintenance.

And the best part is it makes testing your feature changes a lot simpler because you sit down, adjust the test for the new output, adjust the feature, then run the test. If the test fails you don’t even bother loading the test build, you just fix it until it passes and THEN you perform user testing.

Unit tests even make you want to structure your methods properly as little bits of commandlets rather than monolithic functions that do too many things like what UE is full of right now.

So what’d we miss that we should write unit tests for today?

  1. Hot fix for metal debug library included in shipping build did not do what it was intended to do. At all. This was such a big deal that you should write a test for it even if it’s working in 4.19.
  2. Fix for UE packaging the wrong icon in iOS IPAs still has the wrong icon being packaged in iOS IPAs. This seems like a hard thing to miss unless you’re adhering to step 2: half-*** test it.
  3. iOS and Android Online Subsystems still don’t work in either fresh projects or old ones. There’s probably more to this but let’s find out by writing a unit test!

You know what’s super exciting? You can release your unit tests! Then when a user has a problem they can run the unit tests and tell you what the result was! Wowee!

Hey, your users might even update your unit test for you so that it catches the edge case and submit it as a pull request! And they’ll do it because you made it easy by writing the unit test in the first place! And you can validate that a pull request is any good by checking to see if they created or updated a unit test! Make them do it or they don’t get in! That’s called a virtuous circle!

2

So I guess the golden question now is…drum roll… do they have unit test in the first place?

3

A unit testing framework appears to be in the editor (for testing the editor apparently) but it seems to be mostly unused.

I see no evidence of any larger testing framework. There’s no reason not to include it with the engine distribution and all of the regressions seem to confirm that one isn’t in place.

4

Fortnite shipped to iOS and Android using 4.19, albeit a likely slightly modified version - what better kind of testing is there than that?

I find it extremely hard to believe Epic ships completely untested code, perhaps aside from minor modifications.

5

Unit testing, to be precise. One game doesn’t cover all test cases.

Once you’ve used it on a big project you realise you’d be insane not to have it.

See step 2 of the process I imagine that is there now: half-■■■ test it. Some testing obviously takes place but I was hit by four separate regressions this time around. I’m not the only one who can’t build for iOS. It’s clearly a specific configuration of settings and it’s clearly one they didn’t use in their one mobile game.

6

Generally speaking, every software company does testing, and yet, all software is swarmed with bugs. So why would Epic be any different?

7

Thank you for asking! More to the point, why shouldn’t they? Unit testing isn’t taboo!

Unit testing is not commonly implemented in games and C++ programs, for one good reason: it’s hard to define unit tests in that context so when you come from a purely game dev or academic background to work on a large game framework then unit testing may not be something that automatically comes to mind. But that’s where my point lies: unit testing in a framework is to die for.

Get into web applications and you find that the process I described above is becoming mandatory for a lot of development companies. And trust me, once you’ve used that development process you will want to evangelicize it. It’s so good. Lemme show you why.

I’m a big fan of Behaviour Driven Development as well. With BDD you write your new feature as a summary in English and it does some of the work of creating files, classes and methods for you. When it does that it also creates skeleton unit tests. This method recognises that the red tape of a microservice architecture is a pain in the *** and developers get lazy. So it does it for you.

You write down something like:


**Given **a **Component** has received an **Event**
And the **Event** is a **Post-Animation-Trigger**
When **NetworkReplication** is **True**
Then **Serialize **and **Replicate**

That text is then run through a tool that uses that language to create appropriately named and typed methods (such as UComponent::****SerializeAndReplicateOnPostAnimationTriggerEvent(), the important thing is that even without code you know what the method will do) inside the Component class (and creates the class and header files itself if they’re missing) and also creates unit tests for those methods in another file. The BDD text then sits there forever and if you need to change the code structure you just change the language and run the tool again. It keeps your code but moves the methods into your desired naming and arrangement.

It doesn’t write anything between the braces apart from maybe a return. That’s your job. But it does create a sane and consistent microservice oriented structure for you to write code into.

Your next step after running the skeletonizer tool is to write what you want to come back out of these methods into the unit tests, run them and note that it fails because you haven’t written any feature code yet. Then you write the feature code and run the unit tests until it passes. Then you move on to functional testing in the editor or in a game build. And that’s always the order in which you update this code: change the unit test if necessary, then change the feature until it passes the new unit test. If you find that you don’t need to change your unit test when you change the feature then that’s a sign that your unit test can be improved. We WANT unit tests to break when features change, because that’s what happens in the real world. Things break when features change.

To recap:

  1. Write behaviour statement
  2. Run BDD skeletonizer
  3. Fill in unit test to define what should pass
  4. Write feature to satisfy requirements
  5. Run unit tests to verify feature works and unit testing is correct

If while you’re writing the meat of your methods you find your code is doing too much you just write another bit of BDD text like above, let it generate the new methods you need and move some of your code into there. In this way you avoid creating a monolith. Since your original method is only a test for the event type and network-replication status of the component, putting serialization and replication in there would be monolithic. And that’s not good karma.

In the above example I’d make sure Serialize and Replicate were methods with their own BDD statements so that the unit test for our original statement doesn’t care what’s going on INSIDE the method. Only what comes out of it. And the unit tests for **Serialize **and **Replicate **cover those methods.

A serialization unit test for example would create a component, assign some known properties and then serialize it and compare it to a static piece of data from a previously successful serialization of those same properties. In this way if serialization changes unexpectedly for any reason the unit test fails and a developer is alerted that his last change messed something up.

Breaking up code like this is especially good for big old code bases. Refactoring old code is ****ing hard, but when you write BDD to replace it the whole thing just flows from the monolith into the microservices. Your monolithic method gets named properly and the guts of it replaced with calls to methods with better code separation, so for the rest of the codebase only one call gets renamed. But the feature itself is now much easier to understand and easier to unit test. Bits of it can be re-used as necessary instead of having two very similar monolithic functions (check out UE4’s Texture Import Factories for a very good example of what I mean). And unit tests continue to ensure that all of your use cases for those methods still work!

And here’s the truly beautiful part: continuous integration. Once you have CI set up your committed change is instantly tested, even before it’s been peer reviewed. If you get lazy and don’t write a unit test at all and you don’t run the tests, those skeleton unit tests automatically fail and after your commit your team leader brings you coffee and a hug and a wee chat about why unit tests are good. And after that you won’t ever get embarrassed by another developer during peer review ever again because all the obvious stuff is taken care of. And CI can also do things like run a chaos utility on your application and just try random things to see if it can cause an exception before your functional testers even get around to trying it.

So anyway, the very act of writing a BDD statement causes you to think about how your code is separated and ensures you write a unit test. The UE4 framework is a monolith that wants to become a more flexible system piece by piece - in fact in 4.17 and 4.18 Epic did a ton of updates in that direction. But as Epic starts modularizing UE4’s code base more they need to start using a unit testing framework, because the sooner they do the better it will be.

8

Problem is, it’s an engine - they have no idea what the vast majority of people are doing with it. 9 times out of 10 a lot of issues people run into with the engine are actually entirely down to the project. I’m sure Epic are clued up on Unit Testing and CI - any software developer worth their salt is, but perhaps right now it’s not feasible (or for all we know, maybe they are already doing it?)

The biggest barrier I can see tbh is that setting up unit tests takes a massive amount of time and money, and usually it’s a full time job for several people. With such large parts of the engine being under constant re-design, maintaining those tests and keeping them up to date probably costs a huge amount in maintenance too. If Epic were to say, hire a bunch of people who know the engine well enough to design these unit tests, it probably makes more sense to have those guys developing new software rather than verifying old software.

With such a massive community and Epic using the engine themselves, I don’t think it’s a huge necessity because we serve as the test bed. While I get that that’s not ideal for those of us who want to update to the latest features asap and have everything working without issue, I just don’t think that’s a possible reality - unless you have the budget for a huge team.

Who knows though. Maybe an Epic engineer will chime in here…

10

Yes that is what I am afraid of - the unit tests themselves probably will be bigger than the engine itself. The benefit is immense of course, but it can easily drive the focus away.

11

Interesting thread.
I have been a developer of 30 years now, always in the gaming industry. Maybe it’s because I come from a time before a phD was a requirement to write games (yes in my first company there was exactly one guy with an academic degree and he was resented because he got 300$ more than the rest of us and wasn’t doing anything different from what or how we did).
Anyway I have always looked with desire towards unit tests. I read about them, I saw the examples, I saw the frameworks and then I was always thinking how cool it must be. Write your test cases, define input and output parameters and the machine tells you when there is a bug.
I tried several times. In C++, in Ada and it all looked so clean and great.
However I began to realize that I was spending more time writing the test code than for the actual functionality. And as the use of OO programming progressed and became more common I had a hard time dealing with the class states. Hell, you define a beautiful input and output set but there are like a million state variables inside th class that your functions access and they are not part of the function signature.
And then I realized that it’s really hard to write automated tests that tell me whether an object is distorted on the screen or maybe appears in the wrong color or just looks awful.

I don’t think unit tests are a big thing in the gaming industry. My bosses paid me for games, not for testing frameworks. The general consent was that a game is written, released, has a lifetime of maybe 6 months when you’re lucky and then it goes in the trashcan. OK an engine is different but Epic mainly makes UE for their own games so I guess they mostly test it for their own stuff and anything we get out of it is a bonus.

My conclusion is it is futile to expect them to add another 50% developer time just to write some tests that need to be updated with every engine version anyway. It’s just not practical. We are their unit tests, we report errors. I guess that’s part of the reason they made UE available to the likes of us.

Anyway, it’s way better than what I was dealing with before. I take UE with it’s bugs anytime over the crap engines I had to use in the past

12

This is pretty much the reason why you have a microservice architecture, so each part is simple enough to test exhaustively. If your logic can’t be tested then you’re doing too much inside your methods.

I have to disagree that everyone knows about unit testing because if we were all clued up on unit testing we’d all be using it. I think most devs are familiar with the idea but it’s apparent that most of them have never used it. And even when everyone HAS used it, when it comes to big projects like UE4 it’s pretty clear that compromises have needed to be made in order to keep going forward at the desired pace, but like any big project now it’s paying for that.

If you listen to Sweeny talk about the history of the engine you realise that a lot of the fundamental ways of doing things in it predate things like BDD, so they simply weren’t a consideration and ever since someone has always been willing to say “but that’s a massive amount of time any money”. In short the engine is constantly going through a rolling refactor but key things like unit testing and a proper architecture overhaul are not being done.

Setting up unit tests is a natural side-benefit of refactoring your architecture, which is something Epic is already doing. They’ve been talking about moving to the standard library for years as well, so there’s a lot of wants and needs converging. And the nice part is this isn’t re-treading old ground, this is in fact “new software”. You simply start developing the right way going forward and bring the old stuff into line as it gets updated or retire it as it is deprecated.

13

What don’t you like about the bug report form? It feels a bit like a black hole but I’ve had engineers contact me after submitting bugs through it. Maybe it could publish submitted bugs somewhere, but Epic have always been shy about publishing issues they’re working on.

Yes, that’s one of the issues that spurred me to bring this up, exacerbated by the fact that the hotfix intended to fix it (and really only changed two lines of code) didn’t actually work. There’s an entire chain of failure that’s still going on related to packaging for iOS and some of us haven’t been able to get an update for our games out for over a month.

The thing is we can be constructive and proactive about it. We can look at what’s going wrong and solve it for the betterment of everyone.

14

They often are! But the actual amount of work required to create them during feature development isn’t additional time. It’s a different process of development. Instead of spending hours testing and debugging a monolithic function you simply break it down into simple units of work and write the tests before you write the code. Once the test is written all you have to do is write the code to satisfy that test. Then you use that method elsewhere.

There is a point at which you can no longer unit test your framework but it’s very, very far down the line and it’s effectively the outer layer of the program which is what most developers are best at testing themselves. Think about how many times you’ve tested a new feature only to have to figure out what’s wrong with something it’s calling. That’s hard work and a waste of time if there’s a better way to do it. Unit tests don’t cost money. Unit tests save money.

15

Yes! And this is why the natural partner to unit testing is a microservice architecture. I started talking about it because I realised that if you haven’t used unit tests before it’s difficult to understand how they’re actually effective, because an architectural change is always implied but often not stated.

By definition a good unit test is the test of a unit of work. If your work is doing more than a single unit then it’s starting to become a monolithic design, and worse it’s untestable. If something is untestable then it’s not good code.

Yes again. That situation is probably beyond the scope of unit testing, or at least it’s not a single unit of work. If you’re testing that your BSPs are correct you wouldn’t ask a program to look at them on screen. If you were testing if the camera FOV was correct you wouldn’t test it with a BSP. That’s a problem with multiple units of work in it.

The limit of what I’d test in that situation is whether the function for creating a BSP is returning data and whether that data validates, I wouldn’t test it much beyond that. And a camera FOV function is easy to test, you just look for the expected output based on the test input. Then when the thing looks wrong on screen you know that it’s **not **either of those methods causing the issue. Unit testing just saved you time. Unit testing is all about relieving developers from testing things that computers can test for you. Save your brain for the high level stuff.

It’s important to know the limitations of unit testing. You can’t test things like “does this tree feel real” in unit testing and that’s why unit testing in games isn’t what I’m proposing. This thread is all about unit testing inside of a game framework.

And that’s how it should be. Don’t bother unit testing a game unless you’ve developed a framework inside it, such as custom AI or some complicated game rules.

Again, I will never stop stressing that unit testing, when implemented correctly, saves time later on and doesn’t eat additional time during development. You spend five configuring your test and five minutes writing your feature versus spending five minutes writing your feature and five minutes testing it. And after that every time you test it, it’s free.

And best of all it’s not a whole-hog thing. They could start doing it today with every new feature they add.

16

It does get annoying that we never get a version of the engine where everything works. Something is always broken. Also, multi-monitor performance has been super poor since 4.17 and nobody cares.

18

There’s tons of “unit test” classes all over the engine source code.
Specially older more stable modules… But mostly they are all C++ functions only.

19

So a unit testing framework is there, it just needs to be used in a sane and committed way.

20

This pretty much sums up what I feel so far. If there is unit test deployed (and used), then easy bugs should not have slipped easily. TBH, they (Epic) are probably scrambling for ways to generate profit, and listening to us (and fixing all bugs) is not the high priority right now.

21

I believe this is the business model - triple A customers and investors pay for the company to run. We’re pretty lucky that we get UE4 as a free, excellent framework to use but it can still be vastly improved with a change in development culture. Otherwise the natural conclusion is that we should be using a framework whose business model is our market and go to Unity instead. Which would be a **** shame but it also feels like something that’s going to happen as people get fed up with the shortcomings. A complete move to Unity is being discussed in the dev group I am a part of.

22

Yes that is also true - when I first heard about UE4 charge only measly $19 per month subscription, I think wow… can they make it financially?? With all supports from UE4 experts/staffs…