Functional Tests

Due to dependency issues with the FunctionalTesting module, OmniToolbox can't make a C++ child of AFunctionalTest. Hence why we use an actor component instead.

Vanguard offers an actor component designed specifically to be placed on AFunctionalTest actors called TestAssistantComponent. It provides several utilities, such as:

• Generate a spreadsheet that displays what sub-tests were successful and what sub-tests failed.

This code will generate this spreadsheet (if the OmniToolbox.Vanguard.GenerateSpreadsheet console variable is set to true)

• When running a test, a folder with the current timestamp is created that can contain several files. One being a performance log (Do NOT use these numbers to drive your performance metrics if the session was in PIE. Only use these numbers for packaged builds).

The other files can be visual log files or spreadsheets generated from tests. If no file was generated, then that test did not want to generate a spreadsheet and did not use the visual logger.

When doing tests through the project launcher, these files can be found in Saved -> StagedBuilds -> Platform (for example; Windows) -> ProjectName -> Saved -> Automation -> Vanguard

Unreal Insights Integration

The assistant allows you to automatically begin and end a trace for the duration of a test. (Enable Automatically Start Insights Trace in the components settings)

When this is used, the assistant also automatically:

• Takes a screenshot when the WriteResults function encounters an error

• When a new spreadsheet row is added, we start a new region

• The trace file is moved from the profiling directory and into the Vanguard folder and renamed to the testers name.

You can drag and drop a .utrace file into Unreal Insights to open it

Stopping a trace can cause a spike in FPS.

Notes

• By default, all trace channels are disabled when making a new profile. You will want to familiarize yourself with them and enable the ones you want.

  • The more you enable, the larger the trace file and potentially more performance overhead.

  

Visual Logger Integration

The assistant allows you to automatically begin and end a Vislog file for the duration of a test. (Enable Automatically Record Vislog in the components settings)

This gives you a file you can open through the Visual Logger tool inside of the Unreal editor (Tools -> Debug -> Visual Logger

You can now see any Visual Logger shapes your test performed on the timeline.

• I recommend using OmniToolbox's DrawAndLog(Shape) functions to see the shapes both during the test and in the Visual Logger recording.

Performance requirements

Vanguard allows you to mark a test as failed depending on if a large enough hitch/lag spike was detected or if a target game thread time was not achieved. There's a few notes to take around this system as there are big plans in the future to expand this:

• You can find the settings to control this inside Editor Preferences -> Plugins -> Vanguard Testing Settings.

  • It's placed inside the editor preferences and not the project preferences because we want each computer to have its own settings.

• As of 0.3, the Scalability Perf Test Settings is not yet implemented. Use the settings above it.

  • This combined with the point above is aiming to achieve a test environment where you can have a computer with a bad CPU/GPU have appropriate settings for specific scalability settings. For example, a 5 year old cheap CPU might target 60 FPS on low scalability (index 0). Where as a modern expensive CPU might target 180 FPS on low scalability.

There's no timeline for when this feature will be implemented or how much the API will change.

The API might change in such a way where you list a CPU by its product name and give that specific CPU a target.

Do note that Epic *might* be working on something similar to this, but focusing specifically on performance. Depending on how this advances, this feature might get scrapped or focus only on CPU performance.