Skip to content
Framedash Docs
日本語 ja English en 简体中文 zh-cn 繁體中文 zh-tw 한국어 ko

Detect Unreal Engine 5 performance regressions in CI

To catch Unreal Engine 5 performance regressions before they merge, tag each build’s telemetry with a build_id, run a profiling scenario in CI, and gate the pipeline with framedash perf-diff. When the candidate build regresses beyond your threshold against a known-good baseline, the command exits non-zero and fails the job. This guide wires that up with the Framedash UE5 SDK (a C++ plugin for UE 5.3 and later).

For the concepts behind build tagging and the regression gate, see CI-integrated profiling.

1. Tag the build with your CI build ID

Section titled “1. Tag the build with your CI build ID”

Pass the CI commit or build identifier to the SDK as the build_id, so every event from that run is attributed to one build. Initialize from C++ and read FRAMEDASH_BUILD_ID (the variable framedash run-profile-test exports in step 3) so the telemetry’s build_id matches the value the gate waits on:

if (auto* Subsystem = GetGameInstance()->GetSubsystem<UFramedashSubsystem>())
{
    FString ApiKey = FPlatformMisc::GetEnvironmentVariable(TEXT("FRAMEDASH_API_KEY"));
    FString BuildId = FPlatformMisc::GetEnvironmentVariable(TEXT("FRAMEDASH_BUILD_ID"));
    // Pass an empty string for EndpointUrl to use the default value.
    Subsystem->InitializeTelemetry(ApiKey, TEXT(""), BuildId);
}

If you tag the build from a different variable (for example CI_COMMIT_SHA), pass that same value to run-profile-test with --build-id so both sides agree; otherwise the gate waits for a build_id that never ingests and times out. You can also set BuildId in DefaultGame.ini under [/Script/Framedash.FramedashSettings] if you prefer config over code. See the UE5 SDK guide for the full setup.

2. Tag the automated session

Section titled “2. Tag the automated session”

Call the automated-session API once in your test entry point. BeginAutomatedSessionFromEnvironment() reads the FRAMEDASH_BUILD_ID, FRAMEDASH_GIT_BRANCH, FRAMEDASH_GIT_COMMIT, and FRAMEDASH_TEST_SCENARIO variables that framedash run-profile-test exports, so every event carries the build and its branch, commit, and scenario with no per-event tagging code:

if (auto* Subsystem = GetGameInstance()->GetSubsystem<UFramedashSubsystem>())
{
    Subsystem->BeginAutomatedSessionFromEnvironment();
    // ... run the profiling scenario ...
    Subsystem->EndAutomatedSession();
}

3. Run the gate in one command

Section titled “3. Run the gate in one command”

framedash run-profile-test is the turnkey path: it exports the FRAMEDASH_* session variables, launches your profiling build, waits for the telemetry to ingest, then runs the perf-diff gate against a baseline.

Terminal window
framedash run-profile-test \
  --command "./Build/Game.exe -nullrhi -ExecCmds='Automation RunTest Perf'" \
  --scenario nightly --api-key-file ci-read.key \
  --baseline "$BASE_SHA" --threshold 5 --fail-on-regression

The gate reads telemetry with an analytics:read key, while the launched game sends telemetry with a separate events:write ingest key. Pass the gate key with --api-key-file so FRAMEDASH_API_KEY stays available as the game’s ingest key.

The command above uses three placeholders you must substitute:

  • ./Build/Game.exe: your packaged or headless game binary. For a UE5 build this is usually the UnrealEditor-Cmd.exe <Project>.uproject <MapPath> -game -nullrhi ... invocation from the headless run section, not a bare .exe. That -game run never exits on its own, so bind it with the timeout wrapper shown there (a PowerShell kill-after-delay on Windows, timeout / gtimeout on Linux/macOS) and treat the HTTP 2xx log line as the success signal. Replace it with your build’s actual launch command.
  • ci-read.key: a file holding an analytics:read API key that the gate uses to read telemetry. Create it from a CI secret, for example printf '%s' "$FRAMEDASH_READ_KEY" > ci-read.key.
  • $BASE_SHA: the commit or build ID of your known-good baseline to compare against (for example the SHA of main at the branch point). $GITHUB_SHA in step 4 is the candidate build under test.

To run a UE5 build headless in CI, build the editor target first, then launch the game with a null RHI. The UE5 SDK guide covers the exact UnrealEditor-Cmd.exe invocation and the log lines that confirm delivery.

4. Or drive the steps manually

Section titled “4. Or drive the steps manually”

If you want each step separate, list comparable build IDs, then compare the candidate against the baseline. framedash run-profile-test sets FRAMEDASH_BUILD_ID only for the command it launches, so when you drive the run yourself, export it with a build id of your choice (the step-1 initializer reads it) and reuse the same value for --candidate:

Terminal window
export FRAMEDASH_BUILD_ID="$GITHUB_SHA"
# ... launch your profiling run so the SDK tags telemetry with this build id ...


framedash builds --days 30


framedash perf-diff --baseline "$BASE_SHA" --candidate "$FRAMEDASH_BUILD_ID" \
  --threshold 5 --fail-on-regression

perf-diff compares frame time, memory, GPU time, map load time (load_time_ms), and disk I/O (io.read_bytes, io.read_time_ms, io.read_ops) as lower-is-better metrics. Map-load and io.* samples require UE5 SDK 0.1.6 or later. Restrict the gate to one metric with --metric, or to one map or platform with --map and --platform.

REST alternative

Section titled “REST alternative”

If you gate from a language without the CLI, the same comparison is available over REST. GET /v1/projects/{id}/builds lists build IDs, and GET /v1/projects/{id}/builds/compare compares two builds; it requires baseline and candidate and accepts days, mapId, platform, and fresh=1. See the API overview.

Next steps

Section titled “Next steps”
For AI agents: llms.txt