Add isolated Xbox link probe and repair private helper ownership

docs/xbox_link_probe_20260924.md

@@ -0,0 +1,87 @@
+# Xbox link probe (2026-09-24)
+
+The comparison build deliberately compiles each translation unit separately.
+`tools/link_probe.py` makes an **opt-in** Xbox PE link attempt using those same
+compiled objects. It does not change C sources, compiler flags, object files,
+`objdiff.json`, the default Ninja target, or any byte-matching credit. It never
+uses `/FORCE` or substitutes dummy functions. A successful PE link would still
+need conversion to an XBE and game assets before an xemu boot test.
+
+Run from a checkout whose `build/base` objects are current:
+
+```powershell
+python tools/link_probe.py --object-root . --output-dir build/link-probe
+```
+
+An SDK library directory and specific import/vendor libraries may be supplied
+with repeatable `--library-dir` and `--library` arguments. `--exclude-project`
+and `--exclude-source` are only for measuring an alternative link composition;
+`--replace-object` substitutes a separately compiled source candidate and
+`--extra-object` includes a non-manifest input. None changes the comparison
+manifest or matching status. Do not commit or distribute SDK files extracted
+for a local probe.
+
+The script makes two attempts. `full` honors object default-library directives
+and is the authoritative link outcome. `diagnostic` adds `/NODEFAULTLIB` to
+expose otherwise-hidden unresolved symbols. Both use `/SUBSYSTEM:XBOX`, the
+January `_mainCRTStartup` entry, and a response file. `summary.json` records
+the linker result, diagnostic counts, and before/after SHA-256 of every input
+object. A nonzero script exit means the full PE link did not succeed.
+
+## Current measurements
+
+At canonical `f6d00a8c`, all 621 manifest-backed non-missing objects exist and
+were passed to the linker. Their combined before/after digest matched.
+
+| Inputs | Libraries | Full-link result |
+| --- | --- | --- |
+| All 621 rebuilt objects | None | `LIBC.lib` unavailable; nine duplicate definitions. |
+| All 621 rebuilt objects | Local XDK 3911 library directory | 403 unresolved symbols; 33 duplicate definitions. |
+| 475 objects (Halo and reconstructed Bink; excluding reconstructed `libcmt`, `d3d8`, `xapilib`) | Local XDK 3911 library directory plus selected Xbox SDK libraries | 156 unresolved symbols; nine duplicate definitions. |
+
+An explicitly **hybrid**, non-publishable experiment selected 473 objects,
+substituted scratch-compiled versions of two recorded-animation TUs, supplied
+January's original data-only `linker_common.obj`, and used local later XDK/Bink
+libraries. It reached **four unique unresolved symbols**
+(`DirectSoundGetStreamVoiceStatus`, `DirectSoundStopStream`, `fast_ftol_C`,
+`main_crash`) and **four unique duplicate definitions**. It did not link.
+The original data object and later libraries are proof-of-concept inputs, not
+reconstructed source or matching credit.
+
+Adding January's original `dsstrmh/main.obj` (a vendor object containing the
+two absent DirectSound functions) reduced that experiment to **two unresolved
+symbols** (`fast_ftol_C`, `main_crash`) and the same four unique duplicate
+definitions. These two Halo functions are already subject to separate
+house-rule holds; they must not be silently replaced by dummy source or
+counted as matched. The four providers require source/ownership work, not
+`/FORCE:MULTIPLE`.
+
+A further genuine source repair removed `items.c`'s hand-written
+`object_get_type` copy and used the existing `objects.h` inline. Its 18/18
+functions stayed exact, leaving **three** unique provider conflicts:
+`cross_product2d`, `real_local_random`, and `plane2d_from_points`. The latter
+two do not have an evidence-backed zero-regression source fix yet; making
+them private merely to silence the linker would be unauthentic.
+
+The recorded-animation change itself is authentic and safe: January marks the
+five duplicated `_apply_*` functions static in both TUs. Making their
+prototypes/definitions static eliminated those five duplicate providers; both
+scratch-compiled objects retained every exact function (14/14 and 13/13).
+
+The no-default-library diagnostic on all 621 objects reports 478 unique
+unresolved externals and the same nine duplicate definitions. The nine include
+`_object_get_type`, `_real_local_random`, `_plane2d_from_points`, five
+`recorded_animation_playback` helpers, and `_cross_product2d`. This is a real
+link audit, **not** a bootable game or a byte-match regression. XDK 3911 is
+newer than the January target, so its libraries are a diagnostic candidate,
+not proof of an authentic final link.
+
+Next: recover/declare genuinely missing Halo COMMON globals, resolve duplicate
+providers through correct ownership and link composition, and locate compatible
+Bink/Xbox libraries. Only when the PE link succeeds should `imagebld` convert
+it to an XBE, followed by an XISO boot test. Neither the proprietary SDK
+libraries nor the original game assets belong in this repository.
+
+The locally extracted XDK 3911 `imagebld.exe` reports `/IN:<Xbox PE>` and
+`/OUT:<XBE>`; it has not yet been tried on a rebuilt image. Its newer SDK
+version remains a compatibility caveat.

source/cutscene/recorded_animation_playback.c

@@ -203,27 +203,27 @@ void recorded_animation_initialize_unit_control(
 	byte **stream,
 	byte unit_control_data_version);
 
-void apply_animation_state(
+static void apply_animation_state(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
 	byte const **playback_stream);
-void apply_aiming_speed(
+static void apply_aiming_speed(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
 	byte const **playback_stream);
-void apply_control_flags(
+static void apply_control_flags(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
 	byte const **playback_stream);
-void apply_weapon_index(
+static void apply_weapon_index(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
 	byte const **playback_stream);
-void apply_throttle(
+static void apply_throttle(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
@@ -453,7 +453,7 @@ void byte_swap_recording_stream(
 
 /* ---------- private code */
 
-void apply_animation_state(
+static void apply_animation_state(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
@@ -471,7 +471,7 @@ void apply_animation_state(
 	return;
 }
 
-void apply_aiming_speed(
+static void apply_aiming_speed(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
@@ -489,7 +489,7 @@ void apply_aiming_speed(
 	return;
 }
 
-void apply_control_flags(
+static void apply_control_flags(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
@@ -507,7 +507,7 @@ void apply_control_flags(
 	return;
 }
 
-void apply_weapon_index(
+static void apply_weapon_index(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,
@@ -525,7 +525,7 @@ void apply_weapon_index(
 	return;
 }
 
-void apply_throttle(
+static void apply_throttle(
 	struct animation_playback_controller *animation_state,
 	struct recorded_unit_control *control,
 	struct animation_event_header const *header,

source/cutscene/recorded_animation_playback_v1.c

@@ -213,23 +213,23 @@ void recorded_animation_initialize_unit_control(
 	byte **stream,
 	byte unit_control_data_version);
 
-void apply_animation_state(
+static void apply_animation_state(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *event,
 	byte const **playback_stream);
-void apply_aiming_speed(
+static void apply_aiming_speed(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *event,
 	byte const **playback_stream);
-void apply_control_flags(
+static void apply_control_flags(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *event,
 	byte const **playback_stream);
-void apply_weapon_index(
+static void apply_weapon_index(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *event,
 	byte const **playback_stream);
-void apply_throttle(
+static void apply_throttle(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *event,
 	byte const **playback_stream);
@@ -420,7 +420,7 @@ void byte_swap_recording_stream_v1(
 
 /* ---------- private code */
 
-void apply_animation_state(
+static void apply_animation_state(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *anim_event_v1,
 	byte const **playback_stream)
@@ -439,7 +439,7 @@ void apply_animation_state(
 	return;
 }
 
-void apply_aiming_speed(
+static void apply_aiming_speed(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *anim_event_v1,
 	byte const **playback_stream)
@@ -458,7 +458,7 @@ void apply_aiming_speed(
 	return;
 }
 
-void apply_control_flags(
+static void apply_control_flags(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *anim_event_v1,
 	byte const **playback_stream)
@@ -477,7 +477,7 @@ void apply_control_flags(
 	return;
 }
 
-void apply_weapon_index(
+static void apply_weapon_index(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *anim_event_v1,
 	byte const **playback_stream)
@@ -496,7 +496,7 @@ void apply_weapon_index(
 	return;
 }
 
-void apply_throttle(
+static void apply_throttle(
 	struct recorded_unit_control *control,
 	struct recorded_animation_event_v1 const *anim_event_v1,
 	byte const **playback_stream)

source/items/items.c

@@ -58,7 +58,6 @@ symbols in this file:
 
 /* ---------- headers */
 
-#define object_get_type items_object_get_type_inline
 #include "cseries.h"
 
 #include "cseries/profile.h"
@@ -77,7 +76,6 @@ symbols in this file:
 #include "scenario/scenario.h"
 #include "sound/game_sound.h"
 #include "units/units.h"
-#undef object_get_type
 
 /* ---------- constants */
 
@@ -111,12 +109,6 @@ real const item_maximum_impact_velocity = 0.1f;
 
 /* ---------- public code */
 
-short object_get_type(
-	long object_index)
-{
-	return object_header_get(object_index)->type;
-}
-
 void items_initialize(
 	void)
 {

tools/link_probe.py

@@ -0,0 +1,216 @@
+#!/usr/bin/env python3
+"""Opt-in Xbox PE link attempt using the already-compiled comparison objects.
+
+This is deliberately separate from configure.py and the byte-matching graph.
+It never edits source, object files, objdiff configuration, or matching reports.
+The diagnostic pass suppresses unavailable default libraries so the linker can
+report unresolved symbols; neither pass uses /FORCE or claims to make an XBE.
+"""
+
+import argparse
+import hashlib
+import json
+import re
+import subprocess
+from pathlib import Path
+
+
+DIAGNOSTIC_CODES = ("LNK2001", "LNK2005", "LNK2019", "LNK1104", "LNK1181")
+
+
+def manifest_objects(
+    root: Path,
+    excluded_projects: set[str] | None = None,
+    excluded_sources: set[str] | None = None,
+    replacements: dict[str, Path] | None = None,
+) -> tuple[list[Path], list[dict[str, str]]]:
+    config = json.loads((root / "config" / "config.json").read_text(encoding="utf-8"))
+    objects: list[Path] = []
+    missing: list[dict[str, str]] = []
+    for project in config["projects"]:
+        if excluded_projects and project["name"] in excluded_projects:
+            continue
+        for entry in project["objects"]:
+            if entry["status"] == "MISSING" or (excluded_sources and entry["name"] in excluded_sources):
+                continue
+            path = (replacements or {}).get(entry["name"])
+            if path is None:
+                path = root / "build" / "base" / Path(entry["name"]).with_suffix(".obj")
+            if path.is_file():
+                objects.append(path.resolve())
+            else:
+                missing.append({"project": project["name"], "source": entry["name"], "object": str(path)})
+    return objects, missing
+
+
+def object_digest(objects: list[Path]) -> str:
+    digest = hashlib.sha256()
+    for path in objects:
+        digest.update(str(path).encode("utf-8"))
+        digest.update(b"\0")
+        with path.open("rb") as stream:
+            for block in iter(lambda: stream.read(1024 * 1024), b""):
+                digest.update(block)
+    return digest.hexdigest()
+
+
+def diagnostic_summary(output: str) -> dict[str, object]:
+    lines = output.splitlines()
+    counts = {code: sum(code in line for line in lines) for code in DIAGNOSTIC_CODES}
+    samples = [line for line in lines if re.search(r"\bLNK\d{4}\b", line)]
+    unresolved = set()
+    duplicates = set()
+    for line in lines:
+        match = re.search(r"unresolved external symbol (.*?)(?: referenced in function|$)", line)
+        if match:
+            unresolved.add(match.group(1))
+        duplicate = re.search(r"LNK2005: (.*?) already defined", line)
+        if duplicate:
+            duplicates.add(duplicate.group(1))
+    summary = re.search(r"LNK1120: (\d+) unresolved externals", output)
+    return {
+        "counts": counts,
+        "unique_unresolved_seen": len(unresolved),
+        "unique_duplicate_definitions": len(duplicates),
+        "duplicate_symbols": sorted(duplicates),
+        "linker_unresolved_count": int(summary.group(1)) if summary else None,
+        "first_diagnostics": samples[:30],
+    }
+
+
+def run_link(
+    linker: Path,
+    objects: list[Path],
+    extra_objects: list[Path],
+    output_dir: Path,
+    label: str,
+    library_dirs: list[Path],
+    libraries: list[Path],
+    suppress_default_libraries: bool,
+) -> dict[str, object]:
+    image = output_dir / f"{label}.exe"
+    map_file = output_dir / f"{label}.map"
+    response = output_dir / f"{label}.rsp"
+    args = [
+        "/NOLOGO",
+        "/MACHINE:X86",
+        "/SUBSYSTEM:XBOX",
+        "/ENTRY:mainCRTStartup",
+        "/INCREMENTAL:NO",
+        "/OPT:NOREF",
+        f"/OUT:{image}",
+        f"/MAP:{map_file}",
+    ]
+    if suppress_default_libraries:
+        args.append("/NODEFAULTLIB")
+    args.extend(f"/LIBPATH:{path}" for path in library_dirs)
+    args.extend(str(path) for path in objects)
+    args.extend(str(path) for path in extra_objects)
+    args.extend(str(path) for path in libraries)
+    response.write_text("\n".join(f'"{arg}"' for arg in args) + "\n", encoding="ascii")
+    completed = subprocess.run(
+        [str(linker), f"@{response}"],
+        cwd=output_dir,
+        capture_output=True,
+        text=True,
+        encoding="utf-8",
+        errors="replace",
+        check=False,
+    )
+    output = completed.stdout + completed.stderr
+    (output_dir / f"{label}.log").write_text(output, encoding="utf-8")
+    return {
+        "exit_code": completed.returncode,
+        "image_exists": image.is_file(),
+        "response": str(response),
+        "log": str(output_dir / f"{label}.log"),
+        **diagnostic_summary(output),
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--object-root", type=Path, default=Path.cwd())
+    parser.add_argument("--output-dir", type=Path, default=Path("build/link-probe"))
+    parser.add_argument("--linker", type=Path)
+    parser.add_argument("--library-dir", type=Path, action="append", default=[])
+    parser.add_argument("--library", type=Path, action="append", default=[])
+    parser.add_argument("--extra-object", type=Path, action="append", default=[])
+    parser.add_argument("--replace-object", action="append", default=[], metavar="SOURCE=OBJECT")
+    parser.add_argument("--exclude-project", action="append", default=[])
+    parser.add_argument("--exclude-source", action="append", default=[])
+    args = parser.parse_args()
+
+    root = args.object_root.resolve()
+    output_dir = args.output_dir.resolve()
+    linker = (args.linker or root / "xbox/bin/vc7/Link.Exe").resolve()
+    if not linker.is_file():
+        parser.error(f"linker not found: {linker}")
+    if output_dir == root or output_dir.is_relative_to(root / "build/base") or output_dir.is_relative_to(root / "build/split"):
+        parser.error("output must not overwrite the repository or comparison objects")
+    replacements = {}
+    for definition in args.replace_object:
+        source, separator, object_path = definition.partition("=")
+        if not separator or not source or not object_path:
+            parser.error(f"invalid replacement: {definition}")
+        if source in replacements:
+            parser.error(f"duplicate replacement: {source}")
+        replacements[source] = Path(object_path).resolve()
+    config = json.loads((root / "config" / "config.json").read_text(encoding="utf-8"))
+    manifest_names = {entry["name"] for project in config["projects"] for entry in project["objects"] if entry["status"] != "MISSING"}
+    unknown_replacements = set(replacements) - manifest_names
+    if unknown_replacements:
+        parser.error(f"unknown manifest sources: {sorted(unknown_replacements)}")
+    unknown_exclusions = set(args.exclude_source) - manifest_names
+    if unknown_exclusions:
+        parser.error(f"unknown manifest sources: {sorted(unknown_exclusions)}")
+    if set(replacements) & set(args.exclude_source):
+        parser.error("a source cannot be both replaced and excluded")
+    objects, missing = manifest_objects(root, set(args.exclude_project), set(args.exclude_source), replacements)
+    if missing:
+        print(f"Cannot link: {len(missing)} manifest objects have not been compiled.")
+        for entry in missing[:20]:
+            print(entry["object"])
+        return 2
+    library_dirs = [path.resolve() for path in args.library_dir]
+    libraries = [path.resolve() for path in args.library]
+    extra_objects = [path.resolve() for path in args.extra_object]
+    for library_dir in library_dirs:
+        if not library_dir.is_dir():
+            parser.error(f"library directory not found: {library_dir}")
+    for library in libraries:
+        if not library.is_file():
+            parser.error(f"library not found: {library}")
+    for extra_object in extra_objects:
+        if not extra_object.is_file():
+            parser.error(f"extra object not found: {extra_object}")
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    before = object_digest(objects + extra_objects)
+    results = {
+        "object_root": str(root),
+        "object_count": len(objects),
+        "excluded_projects": args.exclude_project,
+        "excluded_sources": args.exclude_source,
+        "extra_objects": [str(path) for path in extra_objects],
+        "replaced_objects": {key: str(value) for key, value in replacements.items()},
+        "object_sha256_before": before,
+        "missing_manifest_objects": missing,
+        "full": run_link(linker, objects, extra_objects, output_dir, "full", library_dirs, libraries, False),
+        "diagnostic": run_link(linker, objects, extra_objects, output_dir, "diagnostic", library_dirs, libraries, True),
+    }
+    results["object_sha256_after"] = object_digest(objects + extra_objects)
+    results["input_objects_unchanged"] = before == results["object_sha256_after"]
+    (output_dir / "summary.json").write_text(json.dumps(results, indent=2) + "\n", encoding="utf-8")
+    print(f"Inputs: {len(objects)} objects; unchanged: {results['input_objects_unchanged']}")
+    for label in ("full", "diagnostic"):
+        result = results[label]
+        print(f"{label}: exit {result['exit_code']}; {result['counts']}; log: {result['log']}")
+    print(f"Summary: {output_dir / 'summary.json'}")
+    if not results["input_objects_unchanged"]:
+        return 3
+    return 0 if results["full"]["exit_code"] == 0 else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tools/test_link_probe.py

@@ -0,0 +1,64 @@
+import json
+
+from tools.link_probe import diagnostic_summary, manifest_objects
+
+
+def test_manifest_excludes_missing_units_and_requires_compiled_inputs(tmp_path):
+    config = tmp_path / "config"
+    config.mkdir()
+    (config / "config.json").write_text(
+        json.dumps(
+            {
+                "projects": [
+                    {
+                        "name": "halo",
+                        "objects": [
+                            {"name": "source/ready.c", "status": "NonMatching"},
+                            {"name": "source/unbuilt.c", "status": "Matching"},
+                            {"name": "source/synthetic.c", "status": "MISSING"},
+                        ],
+                    }
+                ]
+            }
+        ),
+        encoding="utf-8",
+    )
+    ready = tmp_path / "build" / "base" / "source" / "ready.obj"
+    ready.parent.mkdir(parents=True)
+    ready.write_bytes(b"object")
+
+    objects, missing = manifest_objects(tmp_path)
+
+    assert objects == [ready.resolve()]
+    assert [entry["source"] for entry in missing] == ["source/unbuilt.c"]
+
+    replacement = tmp_path / "replacement.obj"
+    replacement.write_bytes(b"new object")
+    objects, missing = manifest_objects(
+        tmp_path,
+        excluded_sources={"source/unbuilt.c"},
+        replacements={"source/ready.c": replacement},
+    )
+    assert objects == [replacement.resolve()]
+    assert not missing
+
+
+def test_diagnostic_summary_distinguishes_references_from_unique_symbols():
+    output = "\n".join(
+        [
+            "a.obj : error LNK2019: unresolved external symbol _missing referenced in function _a",
+            "b.obj : error LNK2001: unresolved external symbol _missing",
+            "c.obj : error LNK2005: _duplicate already defined in d.obj",
+            "LINK : fatal error LNK1120: 1 unresolved externals",
+        ]
+    )
+
+    summary = diagnostic_summary(output)
+
+    assert summary["counts"]["LNK2019"] == 1
+    assert summary["counts"]["LNK2001"] == 1
+    assert summary["counts"]["LNK2005"] == 1
+    assert summary["unique_unresolved_seen"] == 1
+    assert summary["unique_duplicate_definitions"] == 1
+    assert summary["duplicate_symbols"] == ["_duplicate"]
+    assert summary["linker_unresolved_count"] == 1