Finding out why a game is slow

The engine carries a flight recorder. While a game runs, it writes a small record of what every part of every frame did into a fixed ring in memory: which behaviours ran and for how long, how long physics, rendering and the frame itself took, and, when the run has GPU timestamps, how long each GPU pass took. Nothing is saved or sent unless something asks. When a frame is far slower than its neighbours the recorder saves the last few seconds to disk on its own, and everything below reads from that ring or from those saved captures.

This guide is for a person watching a game in the editor and for a tool or script asked "why is it slow?". They read the same numbers: the editor's Profiler tab and the query API draw from one layout, so a picture an agent is handed is the picture a person sees.

---

1. Turning it on and off

The recorder is on in every engine built with the default features, and it costs a few microseconds a frame: a span is one clock read and one 24-byte write into a ring that never allocates once it has filled. It is off in an export built without the profile feature, which holds no ring at all.

docs/PERFORMANCE.md, "Profiler overhead", holds the measured cost of leaving it on. It is written by a script, with the machine's load beside each number.

2. Spikes and captures

The recorder watches every frame. A frame is a spike when it is more than 2.5 times the median frame and more than 2 ms over it, or when it takes over 33 ms, or when it is over a scene's own frame budget. It waits 120 frames after a start before it judges anything, and it saves at most one capture every ten seconds, so one long hitch does not fill the disk.

A capture is a file named for what raised it, such as absolute-f300.argusprof, holding the last 180 or so frames with every span, and a header saying what the game was doing (the scene, the seed, draw and body counts, and the recipe to replay the run). The newest eight are kept. Next to it the recorder writes the same frames as a Chrome trace, absolute-f300.trace.json.

You can also save one by hand at any time: press Capture now in the editor, or POST /profile/capture.

Where they go: .argus/profile/ inside the project when the editor runs the game, or the folder named by ARGUS_PROFILE_DIR for a run started from the command line.

3. The editor's Profiler tab

Open it from Window > Profiler. It sits in the bottom drawer beside Activity, Jobs, Content, Output Log, History, Library and Animator (its short name on a crowded bar is "Prof"). With nothing running it says to press Play.

• The headline says what the recent frames are bound by, in one line: for example "Bound by CPU, script (64%), frame 5.7 ms, p95 6.0 ms", with the analyser's top finding after it. "Bound by GPU" names the pass. Software rendering can make the GPU side look larger than it would on a real card, see section 8.
• The frame-time strip has a column for every recent frame. Height is the frame's cost on a linear scale, and the 16.7 ms line is drawn over the bars, so an ordinary frame sits well below it and a hitch towers over it. A bar taller than the scale is cut off at the top with a white cap. The colour says what dominated the frame: script, physics, render or GPU. A spike has a red mark under its cap. Press a spike (or any bar) to select that frame; the strip stops scrolling so the frame stays where you pointed, and Follow puts the newest frame back.
• The timeline draws the selected frame's spans on one clock, with a row for each level of nesting: the frame, its phases, each behaviour call, and a GPU row of passes when the run has them. Press a behaviour's bar to select that entity in the viewport and the Outliner.
• The flame graph merges the shown frames by path, widest first, so you can see where the time went over the whole window.
• Compare with... picks a saved capture and draws a before strip and an after strip, the change in frame cost with its noise band, and the spans that changed by more than the noise: green for faster, red for slower.
• Source switches between the live game and any saved capture, so a spike can be read after the game has stopped.
• Heat (while playing) tints each entity's bounds in the viewport by what it costs, one metric at a time: script time per tick, triangles or contacts. Cold blue is cheap and hot yellow is costly, and an entity that cost nothing is not tinted. Triangles and draw counts turn on a few counters that cost something in a hot loop, so the tab turns them on while Heat is showing and off when it closes. Surfaces paints the meshes themselves in their heat colour and everything else dark grey. It works with the standard renderer only; the HD renderer shows the boxes, not the surfaces.
• The status bar shows Spikes N while a game runs and has hitched. Press it to open the tab.

The editor also records your input during Play, into session-<id>.replay beside the captures, so any capture from that run can be replayed later (section 6).

4. The live overlay

Run a game in a window with --stats (or press F3 in the game window). Under the frame-rate rows the overlay gains two more:

bound by CPU 64% | script-hot: patrol.update is 73% of CPU time
frames [..::-=+*#%@!..] max 36.2 ms

The first row says what the last frames are bound by and the top finding; the second is the last 48 frames as characters, . for cheap up to @ for costly, and ! for a spike. It is recomputed twice a second and costs nothing while the overlay is off.

5. The query API

A running game with --serve answers these. Every one is a plain HTTP request, and all but the two POSTs are reads.

A game with a reader attached pays to publish its state for it. serve.publish is that span on the main track, between frames, with the entity count as its argument. serve.publish.scripts, under it, is the walk of every behaviour's ctx.state, which only happens once something has read an entity (a GET /entity/..., an agent, the editor) and stays on after that. A timing run whose harness reads entities measures this cost too; a player's game has no reader.

explain is where to start. Each finding has an id such as script-hot (one behaviour is most of the CPU time), script-many (many small calls add up), render-cpu, draw-bound, physics-heavy, gpu-pass-hot, vsync (the frame is waiting for the display, so there is nothing to fix) or unexplained. It also lists what the window is not bound by, so an agent does not go and optimise something that costs 4% of the frame.

GPU time is reported only when the run has GPU timestamps. Without them it is null, never 0, and a finding about the GPU says so.

GPU timing keeps its own cost small: when timing every frame costs more than 50 µs of CPU, it times 1 frame in 4, and on a software adapter it starts there. That choice is made per run, so two runs of the same build can differ. A frame it did not time has no GPU spans, which means "not measured", not 0 ms. The Perfetto export's GPU row says which rate the run used ("timing every frame" or "timed 1 frame in 4"). For a measurement, such as an A/B of GPU time, start the engine with ARGUS_GPU_TIMING=every: every frame is timed for the whole run. ARGUS_GPU_TIMING=off is the other way round: the recorder keeps its CPU spans but nothing is GPU-timed, so a CPU measurement carries none of the timer's own cost (its timestamp writes and query resolve).

The standard renderer's encode spans carry counts in their two args (the Perfetto export's arg and arg2): render.encode.prepare the opaque pass's material-binding floor (the distinct built-in and raw materials and user-shader batch classes its draws used: the fewest material binds any draw order could make; against .main.opaque's bind-group sets, what sorting could save); render.encode.shadows the cascades' runs and draw calls; render.encode.main the draws that went in a multi-draw and how many multi-draw calls they took; render.encode.main.opaque its draw calls and per-draw bind-group sets; render.encode.main.scene its draw calls and how many were blended surfaces; render.encode.main.post and .layers their draw calls; render.encode.main.resolve the shadows' and the scene part's bind-group sets; render.encode.main.finish the frame's per-draw bind-group and pipeline sets, the state CommandEncoder::finish replays and tracks. render.instances.ism.cull carries the instanced meshes' visible cells; render.instances.ism.rows the groups drawn from kept rows (one per run of visible cells, Renderer::keep_rows) and the groups built from the frame's rows; render.submit.set_instances carries the most kept rows so far and the kept region's KiB on the GPU, and splits into .upload, .groups, .user_rows and .draw_order.

6. Checking a fix: the AI loop

The profile toolset holds profile.explain, profile.blame, profile.capture, profile.diff and profile.view. It is not in the default tool list, so an agent asks for it once with tools.enable (profile), or by calling any of its tools. The usual loop:

1. Ask why. profile.explain while the game runs. Read the first finding: its evidence names the behaviour or pass, and its confirm line says how to check the fix.
2. Save a baseline. profile.capture with action: save. Note the id it returns. The capture carries a recipe for replaying the run when the game was recording its input, which is always the case for Play in the editor.
3. Change the project. Edit the behaviour, the scene or the material.
4. Diff with replay. profile.diff with a set to the baseline id. It replays the baseline's recorded input against the edited project, three times by default, pools the runs and answers per-span changes with a noise band, and a verdict on each finding: confirmed (the saving showed up), partial, not confirmed, regressed, or inconclusive when the noise is larger than the change. A change that alters what the game does, not just how fast it does it, diverges from the recording, and the diff says so instead of pretending the two runs compare. Pass replay: false to compare against the live ring instead.
5. Look. profile.view with kind: flame or kind: heat returns a picture, held to the same size budget as any capture.

A capture with no replay recipe (a run that was not recording its input) can still be explained and compared against the live ring, but not replayed. Start such a run with --save-replay file.replay, or with ARGUS_PROFILE_DIR set, and captures from then on can be.

To read a capture without the toolset, use the routes in section 5 directly through play.query.

7. Perfetto

GET /profile/trace and profile.capture with action: export produce Chrome trace JSON, which the Perfetto viewer at ui.perfetto.dev opens directly, with one row per track (main thread, worker threads, GPU, present, load) and every span. From a saved capture on disk:

argus profile-trace .argus/profile/absolute-f300.argusprof

writes absolute-f300.argusprof.trace.json beside it (or where --out says). The export can hash script and scene names (redact) before a trace leaves the machine.

8. What is not measured

• GPU time on a software renderer. A CPU-based Vulkan driver (llvmpipe, lavapipe) runs the "GPU" on the same cores as the game, so its pass times include time stolen from the CPU side. Trust the shape, not the milliseconds, and expect a game that is CPU-bound on a real card to look GPU-bound there.
• GPU time without timestamps. A run whose device cannot time GPU passes reports GPU time as null and its frame cost as the CPU side alone.
• Per-entity GPU cost. GPU-driven passes do not attribute time to one entity. The HD renderer reports triangle share times pass time, marked as an estimate. Everything estimated says method: "estimate".
• HD surfaces. The heat map's Surfaces button paints meshes in the standard renderer only. The HD renderer does not read the view-mode flag, so there the map is the tinted bounds.
• Time inside the operating system. Waiting on the display, a driver stall, another program taking the CPU: they show as a slow frame without a span to blame. The vsync and unexplained findings say so.
• Load on the machine. Every measurement, including the overhead table, is affected by what else the machine is doing. The table records the load average beside each number for that reason.