Same seeds, fresh tests: tuning combat balance with EncounterLab AI

An optimizer can find a profile that plays better, or it can find the one that got lucky. A table of winning settings cannot tell you which happened.

I built EncounterLab AI to make that distinction easier to inspect. It compares candidate combat profiles on the same training seeds, selects a candidate against an explicit objective, then checks the baseline and finalists on fresh seeds that did not influence selection.

The distinction matters because combat outcomes are stochastic. Enemy shots, player-policy choices and hit rolls can move a completion rate even when the settings stay fixed. If every candidate sees a different random sequence, the search may rank noise alongside the design changes.

Make “balanced” an explicit objective

The prototype evaluates five bounded profile parameters, including boss health, projectile speed, attack interval, enemy count and weapon damage. Its example objective targets a 65% completion rate and penalizes a spread between weapon completion rates when that gap exceeds 18 percentage points:

(clear rate − 0.65)² + 0.75 × max(0, weapon gap − 0.18)²

Here, weapon gap is the highest minus lowest completion rate across the tested weapons. Lower scores are closer to the configured target and less uneven across those weapons.

Those values are design assumptions for the example encounter, not a definition of fun or a recommendation for another game. The “players” in the reference simulator are three fixed bot policies: random, cautious and aggressive. That helps compare how profiles behave under a small set of controlled policies, but it says nothing about human skill, enjoyment or retention.

The search can use random sampling or optional Optuna/TPE. Both methods receive the same trial budget, replicate count and seed construction. The profile is validated before a run, and each trial records the full configuration and its hash so a result points back to the settings that produced it.

Compare like with like

For each replicate, the runner evaluates all three policies with all three weapons. That creates nine runs per profile per replicate. The seed key is derived from the experiment’s base seed, the training or held-out partition, the replicate, the policy and the weapon. The profile itself is intentionally left out of that key, so corresponding candidates begin from matching random seeds.

This is a form of common-random-number comparison: keep the random starting conditions paired while changing one candidate profile. It can make a difference between two settings easier to see when the random variation is shared.

There is a limit to how tightly these runs remain paired. The simulator uses one pseudorandom stream inside each run. If a profile changes the course or duration of a fight, the simulation may consume random values in a different order. A matching seed therefore gives a shared starting stream, not a guarantee that every later projectile or hit is an identical paired event. Separate streams for independent systems, or a pre-generated scenario tape, would make the pairing stronger.

Do not let selection grade itself

The search chooses the lowest-scoring candidate from its training runs. Reusing those same runs to announce that the winner is better would reward a candidate for fitting their particular random outcomes.

EncounterLab addresses this with a second partition. After search, it evaluates the baseline and the best candidate from each method on a fresh held-out seed bank. The held-out replicate count is at least 10 and at most 30; in the small run shown above, two requested training replicates produced ten held-out replicates. Each profile therefore received 90 held-out runs: ten replicates across the nine policy-and-weapon combinations.

That is a useful check for an obviously lucky training result, but it is not a large-scale generalization study. The seed partition changes; the simulator, policies and weapon set do not. A stronger follow-up would vary the scenarios and test policies too, then run the same comparison in the connected Unity build.

The report keeps run groups separate by method, train or held-out status, policy, weapon, measurement source, platform and build. It includes Wilson intervals for completion rates and bootstrap intervals for median duration and damage. Those intervals expose uncertainty in the recorded runs; they do not repair a small sample or make the chosen target a product truth.

What I verified

At private source snapshot a183422556f7, I ran the project’s 22 Python tests. They cover deterministic results for a fixed seed, profile validation, statistics, output safety, report escaping and run-record validation. I also ran a small random-search smoke test with two candidates and two training replicates. It generated 234 synthetic run records: 54 training records across the baseline and two candidates, plus 180 held-out records for the baseline and selected candidate.

I do not present the winning profile’s completion rate as evidence of better combat. The smoke run confirms that the search and reporting path executes; it is intentionally too small and too synthetic to justify a balance decision. The reference model is not a Unity scene, and the repository does not include a game-specific scene or trained player policy. The C# adapter only becomes gameplay evidence after it is connected to a real build’s balance targets, agent input path and game events.

Keep the designer in the loop

The useful output is a candidate profile and its evidence, not an automatic edit to production assets. Before using one, I would connect the adapter to the actual encounter, confirm that the test agent goes through normal movement, collision and weapon rules, and tag every run with the build revision. I would compare profiler recording on and off on the same target device because measurement can add overhead.

Then I would review the candidate with people. Bot completion and weapon symmetry are useful diagnostics, but neither measures whether an encounter feels tense, readable or fair. Human playtests should be reported separately from bot results, with their own consent and sample limitations.

The source repository is private, so this article describes the reviewed implementation without linking to code a reader cannot access. For another example of paired technical comparisons, read how I compare Unity build runs in PatchScope. For the shipped-project story behind my Unity work, see the development of Humanity’s Last Stand.

If you are building a game system that needs repeatable balance experiments, I’d be glad to compare approaches.

← All notes