A p95 number looks precise. By itself, it does not tell me whether two builds were tested under comparable conditions, whether the sample was complete, or whether a visible difference is large enough to matter.
PatchScope is my private Unity mobile build-comparison prototype. Its useful design choice is to treat a scripted run—not each frame inside that run—as the unit that gets paired and compared. The demo makes that pipeline easy to see, but its values are generated data. They are not measurements from a physical phone.
PATCHSCOPE / BUILD COMPARISON
One run is one comparison unit- 01 / MATCHPair comparable runsBASELINE
pair · scenario · scene · device · originCANDIDATEIncomplete and duplicate pair slots are excluded.
- 02 / CONTRASTKeep the paired changecandidate p95 − baseline p95baseline p95
Calculate one percentage change for each matched run pair.
- 03 / RESAMPLEBootstrap pair outcomes2,000resamples
A percentile interval summarizes uncertainty across run-level changes.
- 04 / DECIDEUse a practical guardrailDEFAULT THRESHOLD+5%
At least 8 pairs and 90% coverage per compared run are required.
The seeded example generates 20 pairs and deliberately raises candidate p95 by about 22% with noise. It is not a phone benchmark.
Reduce a run to a reviewable summary
The Unity recorder stores Time.unscaledDeltaTime in milliseconds for each Update. It targets 600 samples, sorts the recorded values at completion, and selects the p95 using the nearest upper-ranked sample index. It also records the proportion of samples above the configured frame budget, a managed-memory reading, accumulated GC allocation samples, build and scene identifiers, and the run’s pair ID.
The upload is one summary for one execution, not a stream of individual frame events. If a test calls CompleteRun early, the recorder still reports 600 expected samples beside the smaller observed count. The comparison path can then mark a pair inconclusive when any included run has less than the configured 90% coverage.
Some fields need careful interpretation. memory_mb comes from one GC.GetTotalMemory(false) reading at completion; it is not peak memory or process RSS. The scene-load duration is supplied to BeginRun by its caller, rather than measured automatically. If the ProfilerRecorder for GC allocation is unavailable, the current recorder leaves the allocation total at zero and does not send a separate availability flag. A zero should therefore be checked against recorder availability before it is interpreted as proof of no allocation.
Match runs before calculating a change
The comparison groups baseline and candidate summaries by pair ID, scenario, scene, device model, and data origin. A complete pair needs exactly one run for each build. Incomplete pairs and pair slots with duplicate runs are excluded instead of being silently averaged together.
For each valid pair, the code calculates the candidate-minus-baseline p95 change in milliseconds and as a percentage of that pair’s baseline. It then summarizes the pair-level percentage changes. This preserves the relationship between the two executions; it does not pretend that every frame sampled during either run is a separate independent build trial.
There is one matching detail I would tighten before using results from multiple platforms: platform is part of the upload contract, but it is not selected by the comparison query or included in the pairing key. The device model usually distinguishes hardware, but the code does not require the platform labels to match. I would add platform to the query, dashboard filter, and key, then test that a pair cannot combine runs from different platforms.
Bootstrap the paired changes
PatchScope resamples the per-pair percentage changes with replacement, recalculates their mean, and uses the 2.5th and 97.5th percentile values as a two-sided 95% interval. The default is 2,000 resamples. The seed comes from the selected builds and query, so the same data and filters produce a repeatable interval. For a concise reference to paired resampling, the SciPy bootstrap documentation describes using shared resampled indices to keep paired observations together; PatchScope applies that idea to its already-computed per-pair changes.
The interval is compared with a configured 5% practical p95 increase threshold. A result is REGRESSION only when the lower interval bound is above that threshold. It is NO_CLEAR_CHANGE when the upper bound is below it. If the interval crosses the threshold, or there are fewer than eight complete pairs, the result is INCONCLUSIVE.
That final state matters. “Not enough evidence to call a regression” is not the same as “the builds are equivalent.” Pair count, coverage, and the threshold are visible inputs to the decision, not hidden details behind a red or green badge. The defaults are demonstration settings; repeatability runs on the target device and scenario should inform any real release guardrail.
Keep transport retries separate from measurement identity
The recorder writes each batch to Unity’s persistent data path before trying to upload it. The API validates the schema and stores the batch plus new event IDs in one SQLite transaction before returning acceptance. Repeating the same batch ID with the same content returns the prior result; changing the content under that ID returns a conflict. Reusing an event ID with different content also rolls back the new batch.
The worker sends pending outbox rows to Redpanda and marks each row published after an acknowledgement. A crash between those two actions can publish the same event again. ClickHouse stores the event ID in a replacing table, and comparison queries use FINAL to read one current row per ID. That is retryable, at-least-once delivery with event-level deduplication in the read path—not a claim of end-to-end exactly-once processing.
What the demo and tests actually prove
The demo generator creates 20 synthetic matched pairs and multiplies the candidate p95 by about 1.22 before adding noise. A test checks that this deliberately injected pattern is classified as a regression and retains the synthetic_demo origin. It verifies the pipeline’s decision path; it does not benchmark a phone or establish that a real Unity build is 22% slower.
At the reviewed commit, python scripts/run_tests.py passed all 32 tests locally on Python 3.11. They cover contract validation, SQLite retries and conflicts, pair matching, coverage gates, deterministic bootstrap decisions, worker acknowledgements, and ClickHouse request formatting. They do not compile the C# recorder in Unity, exercise a physical device, or replace a full Docker Compose run.
The latest GitHub Actions run for the reviewed head was marked failed but exposed no job steps or log, so I cannot identify a cause. An earlier run for the same application and test source passed; later commits changed only README content and its image. I do not count the current remote run as a pass. The Compose services are also explicitly a one-machine demo stack without authentication, TLS, replication, or a production backup policy.
Before calling this a release gate, I would include platform in the match key, expose whether profiler counters are available, randomize and record baseline/candidate order, and repeat paired runs after a defined warm-up on the target device. I would then report the device, OS, Unity version, scenario, pair count, sample coverage, and interval alongside the result. That would turn a useful analysis prototype into evidence about a specific build and test setup.
PatchScope is a private portfolio MVP. Its seeded dashboard is synthetic, and the Unity recorder has not been validated on a physical device.