UE4 Automation Testing

9 minute read.

Have you ev­er had a fea­ture in an Un­re­al En­gine game that you had to come back to again and again, be­cause it was a nest of bugs? This gives you this un­easy feel­ing of that part be­ing un­sta­ble, not re­al­ly done yet, you can nev­er re­lax… right?

What if you could guar­an­tee that part of your code works by test­ing it in a few sec­onds, rather than min­utes of man­u­al test­ing?

Au­to­mat­ed test­ing or “Au­to­ma­tion Tests” are the so­lu­tion to your prob­lem!

Au­to­ma­tion Tests

In non-game ap­pli­ca­tions test­ing is a vi­tal part of soft­ware projects. Au­to­ma­tion tests are small pro­gramms that test your code for cor­rect­ness. The tests can be used while de­vel­op­ing a fea­ture and en­sure it ac­tu­al­ly does what it is sup­posed to, or to en­sure some­thing still works af­ter you made changes to that code lat­er down the line (so called “re­gres­sion test­ing” 1). In game de­vel­op­ment it is less pop­u­lar, but equal­ly use­ful.

A test works by e.g. call­ing a sub­mit score func­tion and then get­ting all en­tries of the high­score list and mak­ing sure the new en­try ap­peared. Ba­si­cal­ly what you would prob­a­bly do man­u­al­ly oth­er­wise.

The “Au­to­ma­tion” in “Au­to­ma­tion Test­ing” means that you mere­ly have to click a but­ton and all your lit­tle test ap­pli­ca­tions run with­out futher man­u­al la­bor in­volved. This makes this su­per fast and easy to do, which makes you do it more of­ten.

Frame­work in UE4

For­tu­nate­ly Un­re­al En­gine has a frame­work for you that makes writ­ing au­to­ma­tion tests eas­i­er. It es­pe­cial­ly helps you with asyn­chro­neous calls, e.g. when telling the game to load a map and then need­ing to wait for it to be load­ed be­fore you can start test­ing with Ac­tors in­side that map.

Un­for­tu­nate­ly their doc­u­men­ta­tion on it is pret­ty out of date. In­stead I had to browse the Au­to­ma­tion frame­work’s source code and their en­gine tests to fig­ure out how to prop­er­ly use it.

Since I want to save you the time and pain, here is an up-to-date guide for you:

Test Plug­in Set­up

It is rec­om­mend­ed to keep your tests sep­a­rat­ed from the rest of your game (or plug­in) as at least a mod­ule or even as a plug­in. This is ben­e­fi­cial, since you can ex­clude the test code from the fi­nal game build due to the mod­ule’s "Type": "Developer" con­fig­u­ra­tion. Hav­ing it in a sep­a­rate plug­in would al­low you to even dis­able it man­u­al­ly, though I can­not come up with a re­al use case for that.

So go ahead and cre­ate a new fold­er in your projects Plugin/ di­rec­to­ry. We’ll call your game —I feel out­ra­geous­ly cre­ative to­day—YourGame for the sake of this tu­to­ri­al and there­fore the plug­in will be called YourGameTests.

The YourGameTests.uplugin looks like this:

{
    "FileVersion": 3,
    "Version": 1,
    "VersionName": "1.0.0",
    "FriendlyName": "Your Game Tests",
    "Description": "Automation tests for YourGame.",
    "Category": "Testing",
    "CreatedBy": "You",
    "CanContainContent": true,
    "Installed": false,
    "Modules": [
        {
            "Name": "YourGameTests",
            "Type": "Developer",
            "LoadingPhase": "Default"
        }
    ]
}

With that we’re ready to set up a Source/YourGameTests fold­er in there and add our mod­ule’s build file. YourGameTests.Build.cs should con­tain:

using UnrealBuildTool;

public class YourGameTests : ModuleRules
{
    public YourGameTests(ReadOnlyTargetRules Target) : base(Target)
    {
                PCHUsage = ModuleRules.PCHUsageMode.UseExplicitOrSharedPCHs;
                PrivateDependencyModuleNames.AddRange(new string[] {
                    "Core",
                    "Engine",
                    "CoreUObject",
                    "YourGame"
                });
    }
}

You add YourGame to the de­pen­den­cies to make its class­es and func­tions avail­able to the test.

Next, we add the mod­ule scaf­fold in YourGameTests.cpp and YourGameTests.h. While lat­ter can be emp­ty, the for­mer con­tains the fol­low­ing three lines:

#include "YourGameTests.h"
#include "Modules/ModuleManager.h"

IMPLEMENT_MODULE(FDefaultModuleImpl, YourGameTests);

Time to add our first lit­tle test!

Writ­ing a Test

Out of hab­bit, I put all my test sources in­to a Tests sub­fold­er. This would al­low me to merge it with the game mod­ule with­out mix­ing test and game sources, should I ev­er want to do that—prob­a­bly not—but oth­er than that, you can just aswell put your sources di­rect­ly next to the .Build.cs.

Add a source file for the tests with your name. E.g. MenuTest.cpp or MatchmakingTest.cpp or HighscoresTest.cpp etc. I will go with GameTest.cpp which is sup­posed to test gen­er­al func­tion­al­i­ty of the game. It will look like this:

#include "YourGameTests.h"

#include "Misc/AutomationTest.h"
#include "Tests/AutomationCommon.h"
#include "Engine.h"
#include "EngineUtils.h"

#include "YourGameModeBase.h"
#include "MyEssentialActor.h"

// Copy of the hidden method GetAnyGameWorld() in AutomationCommon.cpp.
// Marked as temporary there, hence, this one is temporary, too.
UWorld* GetTestWorld() {
    const TIndirectArray<FWorldContext>& WorldContexts = GEngine->GetWorldContexts();
    for (const FWorldContext& Context : WorldContexts) {
        if (((Context.WorldType == EWorldType::PIE) || (Context.WorldType == EWorldType::Game))
            && (Context.World() != nullptr)) {
            return Context.World();
        }
    }

    return nullptr;
}

IMPLEMENT_SIMPLE_AUTOMATION_TEST(FGameTest, "YourGame.Game",
    EAutomationTestFlags::Editor |
    EAutomationTestFlags::ClientContext |
    EAutomationTestFlags::ProductFilter)

bool FGameTest::RunTest(const FString& Parameters) {
    AutomationOpenMap(TEXT("/Game/Levels/StartupLevel"));

    UWorld* world = GetTestWorld();

    TestTrue("GameMode class is set correctly",
        world->GetAuthGameMode()->IsA<YourGameModeBase>());
    TestTrue("Essential actor is spawned",
        TActorIterator<AMyEssentialActor>(world));

    return true;
}

The GetTestWorld() func­tion is es­sen­tial to get some han­dle to the world. It’s hid­den in AutomationCommon.cpp, so I copied it out.

Al­ter­na­tive­ly to TestTrue, there are TestNotNull, TestEqual and so on as you would ex­pect in most test­ing fram­works (and most of which are su­per­flu­ous).

La­tent Com­mands

Some­times you will need to al­low your game to tick a cou­ple of times be­fore test­ing a re­sult. As­sume for ex­am­ple, you need wait for a ball fall­ing down on a pres­sure pad. You would check ev­ery tick whether the ball has reached a cer­tain Z lo­ca­tion yet, and on­ly if it did check whether the pres­sure pad has been trig­gered or some­thing.

(In this par­tic­u­lar case it’s prob­a­bly a bet­ter idea to just set the po­si­tion of the ball im­me­di­ate­ly which al­so speeds up the test.)

Sim­i­lar­ly for an­syn­chro­neous load­ing lev­els: you will have to wait un­til UE4 load­ed your lev­el to then start test­ing on it.

The Au­to­ma­tion Test­ing Frame­work pro­vides “La­tent Com­mands” to do this. They have an bool Up­date() method, which you over­ride to re­turn false to wait and true when you’re done. This means you can wait for some con­di­tion to hold true un­til you do your test­ing.

FMyCommand could look like this:

DEFINE_LATENT_AUTOMATION_COMMAND_ONE_PARAMETER(FMyCommand, FGameTest*, Test);
bool FMyCommand::Update() {
    if(!SomethingHasHappened) return false; // Try again later

    Test->TestTrue("SomethingWasCaused", SomethingWasCaused);
    return true; // Command completed
}

And to en­queue the la­tent com­mand, you use

ADD_LATENT_AUTOMATION_COMMAND(FMyCommand(this));

Note, that all test code af­ter that needs to al­so be added as la­tent au­to­ma­tion com­mand, be­cause oth­er­wise it is not guar­an­teed that this com­mand is com­plet­ed be­fore your code runs. This makes work­ing with these very te­dious.

this is a pa­ram­e­ter which al­lows call­ing Test*() func­tions from the au­to­ma­tion com­mand. You can pass any­thing else, al­so.

Be aware that the macros for more params are miss­ing the “S” in “PA­RAM­E­TERS”, it’s just DEFINE_LATENT_ATUOMATION_COMMAND_TWO_PARAMETER (i.e. doesn’t fol­low the DELCARE_DYNAMIC_DELEGATE_*Params con­ven­tion).

Ad­vanced Test­ing

For this blog post I de­cid­ed to pub­lish our “Ad­vancedTest­ing” plug­in on­to the Un­re­al En­gine Mar­ket­place. It al­lows you to spec­i­fy your la­tent au­to­ma­tion com­mands “in­line”, hence look­ing some­thing like this:

#include "AdvancedTesting.h"

// ...

bool FMyTest::RunTest(const FString& Parameters) {
    AddInlineLatentCommand([]() {
        return HasEventOccured;
    });

    AYourGameMode* myGameMode;
    AddInlineLatentCommand([](AYourGameMode* gm) {
        return gm->HasEventOccured();
    }, myGameMode);
}

It al­so has a cou­ple of oth­er neat util­i­ties.

If this ar­ti­cle was help­ful to you and want to sup­port my work, it is avail­able here. 2 Oth­er­wise, you can al­so grab it from the GitHub repos­i­to­ry, once it’s re­leased un­der an MIT li­cense, for free! (MIT li­cense al­lows you to use it com­mer­cial­ly, yes ;) )

1
A “re­gres­sion” is a bug in code that worked at some point in time, where­as a nor­mal bug is some­thing that nev­er worked ev­er.
2
Cur­rent­ly pend­ing ap­proval.

Writ­ten in 90 min­utes, ed­it­ed in 15 min­utes.