Test Environment Guide

Some of my plugins get their own test environments - including OmniToolbox. This page is covering the standardization measures taken for all of these environments to make sure they are all consistent. This page can also be used to help inspire your own test environments and their rules.

Style guide

• Each test environment must be a game feature plugin and be named with either the full name of the plugin or project its testing or its shorthand term (Inventory Framework Plugin -> IFP), followed with _TestingEnvironment

  • Game feature plugins feature everything we need to limit asset referencing and helps ensure all test content is disabled or not cooked in final builds.

  • Some tests might want to make temporary changes to the asset manager, or add components to actors for debugging the games normal gameplay maps in the middle of a session. Game Feature Plugins natively support this.

  • With the _TestingEnvironment suffix, it's possible to check in your shipping builds if any game features with that suffix exist. If so, then something went wrong with packaging.

• The environment can only have 1 map and it must be named with a M_ prefix, then named after the test environment plugin. For example, OmniToolbox's map is named M_OmniToolbox_TestingEnvironment

  • Making sure external maps are cooked for automation testing is painful and adding more test maps only forces automation tools to be updated. This rule makes sure that once the cook command has been setup, it doesn't have to be updated.

• Every map must have a camera actor with a tag DefaultCamera

  • UTestAssistantComponent can automatically assign all tests with no observation point to use this actor as its observation point.

• Every test must be located in loaction 0,0,0 (unless the test is testing something regarding elevation). The test map must have a flat plane that does not prevent actors at this location to fail to spawn, but still allow actors to walk without falling. This plane must also have a grid material with standard measurements.

  • With location set to 0 and the grid material makes it possible to identify if something in the scene is in the correct location through a screenshot or without opening a debugger.

  • I highly recommend the grid material from the Game Animation Sample.

• Every test must have atleast one tag.

  • This is to make the Automation RunTests console command easier to use.

• Every test goes into the Testers folder and then has its own folder. Even if the test is only 1 class.

• All assets inside these test environments must only reference assets inside of themselves or engine assets. Plugins found in the engine are not allowed to be referenced.

  • Most engine assets used are the simple primitive shapes, such as the default engine cube.

Setting up the Project Launcher

There's 2 ways to open the project launcher;

1. In the editor, go into Tools -> Project Launcher

2. Open file explorer and go to YourEngineInstallation\Engine\Binaries\Win64 and open UnrealFrontend.exe

I recommend the latter so the process is not tied to your editor, in case it crashes. It's also better to learn to not rely on the editor being open in case you ever build a home server or start using cloud services to automate your builds and tests.

Unfortunately, Unreal's support for blueprint-based automation tests are very limiting. Tests can only be discovered by maps that have been opened throughout the session and it seems to struggle with swapping maps.

Because of this, we have to create a launch profile for every test environment.

Creating a launch profile

1. Press the Create Launch Profile button

   1. I tend to name my launch profiles for test environments like so: Name of plugin being tested - Test tags (Full suite if all tests are being ran) - Configuration (DebugGame/Development.

   
2. Add this to the additional cook commands:

-Map=M_OmniToolbox_TestingEnvironment -package

• This will make sure the test environment map is packaged.

1. This step is optional, depending on how you want to use this profile. You can add commands to be executed, such as console commands to enable or disable certain features.

-ExecCmds=\"open M_OmniToolbox_TestingEnvironment\"

• Open the test environment map

-ExecCmds=\"OmniToolbox.Vanguard.GenerateSpreadsheets 1;\"

• Enable spreadsheet generation

-ExecCmds=\"OmniToolbox.Vanguard.OverrideIsAutomationTesting 1;\"

• When using Unreal Automation in the editor, there's a flag (GIsAutomationTesting) that is set to true. But when you start tests through a console command, this flag is not set to true. Vanguard relies on this flag, but due to an engine bug, we have to use our own flag when running through the project launcher.

-ExecCmds=\"Automation RunTests OmniToolbox;Quit\"

• Execute all automation tests with the OmniToolbox tag, then once they are finished, close the session.

  • Automation RunTests can also take specific names if you only want to run specific tests. It also accepts multiple tags

  • You can remove Quit if you don't want the client to automatically shut down.

  • Remember, this is a console command. So if you want to rerun a test during runtime, you can either use the Automation RunTests console command or open the Sessions Frontend -> Automation and run the tests through there.

My profiles usually end up looking something like this:

• I added stat fps and stat unitgraph for this profile, since I usually monitor it. But ideally, you should not have to monitor these tests.

When changing engine versions or reinstalling the engine, your profiles might get wiped.

To backup your profiles, go to EngineInstallation/Programs/UnrealFrontend/Profiles and backup the files found in there.

Common issues

• Could not find file ProjectDirectory\Binaries\Win64\ProjectNameEditor.target

  • To fix this, launch your project with your IDE and rebuild your project with the Development configuration assigned.

• DoesPackageExist called on PackageName that will always return false. Reason: Input '/SomeFile' is not a child of an existing mount point.

  • There seem to be many solutions to this posted online. For me, this was solved by disabling and enabling the game feature plugin

• Error_FailedToDeleteStagingDirectory

  • This usually occurs when another application is using a file inside the staging directory. This can happen when you have any of the spreadsheets or trace files open in a separate program.

    • If you use a cloud service, like OneDrive, pCloud or something else, those can sometimes lock files inside the staging folder.