spoopyghosty0--frameport/docs/HEURISTICS_EVAL.md

Evaluating patch heuristics

scripts/eval_heuristics.py compares engine.suggest(analysis, use_catalog=False) with the bundled catalog/games/*.yaml entries. It does not merge personal or remote catalog recipes into the benchmark. It supports the original dumps command, or a precomputed scripts/heuristics_snapshot.py snapshot:

.venv/bin/python scripts/eval_heuristics.py '<dumps folder>' --verbose
.venv/bin/python scripts/eval_heuristics.py --snapshot analyses.json --split all --json baseline.json
.venv/bin/python scripts/eval_heuristics.py --snapshot analyses.json --fit-report --out fit-report.md

Use a scratch FRAMEPORT_HOME, FRAMEPORT_NO_CATALOG_UPDATE=1, and an empty FRAMEPORT_WINDOWS_DATA for offline measurement. Snapshot mode reads no APKs or game data. Each snapshot read copies the source to a temporary file, then parses the copy, retrying incomplete JSON. To compare multiple splits against an actively growing snapshot, freeze one valid copy first and use that file for every run.

The default split is dev. split_packages({package: analysis_engine}) sorts packages by SHA-256 within Unity, Unreal, and Other strata, then holds out the nearest 20% (half rounded up). Pass only packages with both an analysis and a catalog entry. The same input set always yields the same split regardless of iteration order; adding packages can move the cutoff. Very small strata can have no holdout. Unsupported entries participate in splitting.

What is scored

Ground truth contains catalog choices and patch registry defaults, with no inherited heuristic adapter settings. Default expectations are determined per analysis, using each default patch's own detection in the same way as the engine's initial heuristic pass. Thus a conditional default that is not detected (e.g. LTW on a game without LTW) is not expected. Explicit catalog overport_remove and frame_remove entries remove expected defaults. An absent expected default is represented as remove:<patch id>; enabled defaults do not inflate patch scores.

Opt-in overport/frame/device/PC VR IDs, default removals, adapter values, device-file names, foveation parameters, and alternate-build patch IDs are compared. Alternate-build identities use alt:<patch id> so choosing one in a fallback is distinct from choosing it in the primary build. Device-file content is not compared. Adapter values must match: a wrong value contributes both an FP and an FN for adapter.<key>. Patch precision, recall and F1 are micro-averaged across games; an empty denominator is reported as zero. Exact-game accuracy also compares the unsupported status flag. Fields report matching games, including empty-field matches.

RUNTIME_ONLY lists choices excluded from the headline: use_alt, selected runtime adapter settings, Lepton environment variables, and config-sync settings. They have their own exact, field and patch metrics. Catalog entries with status unsupported are excluded from both headline and runtime supported-game metrics, and have separate metrics and per-game diffs. Converted analyses are included and marked. JSON contains all metrics per engine, all game signatures, split assignments, and missing/extra decisions.

Dated score: catalog gaps vs. detection errors

The strict score treats the catalog as complete, but older entries were verified before many patches existed, and a catalog recipe inherits every heuristic adapter setting and device patch (except device.text_input_window) without listing them (engine.suggest), so the verified build had those. The dated score (headline_dated, printed under "Dated") sets aside, on both sides:

• patches first committed after the entry's verified.date (the verifier couldn't have chosen them), and
• inherited adapter/device picks (inherited_decisions).

What remains are real disagreements: an extra on an entry verified after the patch existed, or a catalog choice the heuristics don't make. Gaps are counted per patch ("Catalog gaps by patch"). The dated score needs the intro date cache that --fit-report writes; without it only inherited picks are set aside.

Before tuning a rule to remove an "extra", check which kind it is: a gap on an old entry is a candidate for the catalog (fit report), not a detection error. Before adding a patch to a working entry, weigh its benefit (does it fix something the game hits?) against a rebuild prompt for everyone who installed it; adapter settings never need adding (inherited). Results 2026-10-15: strict 5/65 exact, F1 73.8%; dated 39/65, F1 88.6% (dev 90.5%, holdout 79.5%). Remaining misses are one-game fixes found at runtime (e.g. Vader's gl_error_clear, BlazeRush's GL settings).

An evaluation writes JSON even when its headline has differences, then exits 1 for compatibility with the original script. It exits 0 when exact, or after producing a fit report.

New patch fit

The fit report covers all catalog entries, independently of --split. Candidates come from the final heuristic recipe, preserving platform, flat-app, dependency, and 32-bit normalization rules. A candidate must also be opt-in, applicable and recommended by its own detect(). Entries already listing or explicitly removing the patch are excluded; a listed adapter key is excluded regardless of its value. This report proposes additions, not value edits.

Each candidate includes its reason, experimental/strict flags, first local git introduction date, catalog entry date (verified.date; updated only without one, since it also moves for unrelated edits), and whether the patch is newer. Unknown dates are marked unknown. Warnings come from the full catalog recipe plus that one candidate; they may include pre-existing conflicts. Missing-analysis entries appear as metadata-only rows. Intro dates, including unknown results, are cached in patch-intro-dates.json beside the snapshot (or dumps folder); subsequent runs only query uncached patch IDs.

tests/test_eval_heuristics.py uses synthetic data. The opt-in real-dump test in tests/test_games.py ratchets the supported all-split exact ratio and patch F1 against measured, rounded-down baseline constants. Machine-local .eval-home/ and .eval-results/ scratch outputs are ignored by git.