diff --git a/docs/xbox_link_probe_20260924.md b/docs/xbox_link_probe_20260924.md new file mode 100644 index 00000000..06146319 --- /dev/null +++ b/docs/xbox_link_probe_20260924.md @@ -0,0 +1,87 @@ +# Xbox link probe (2026-09-24) + +The comparison build deliberately compiles each translation unit separately. +`tools/link_probe.py` makes an **opt-in** Xbox PE link attempt using those same +compiled objects. It does not change C sources, compiler flags, object files, +`objdiff.json`, the default Ninja target, or any byte-matching credit. It never +uses `/FORCE` or substitutes dummy functions. A successful PE link would still +need conversion to an XBE and game assets before an xemu boot test. + +Run from a checkout whose `build/base` objects are current: + +```powershell +python tools/link_probe.py --object-root . --output-dir build/link-probe +``` + +An SDK library directory and specific import/vendor libraries may be supplied +with repeatable `--library-dir` and `--library` arguments. `--exclude-project` +and `--exclude-source` are only for measuring an alternative link composition; +`--replace-object` substitutes a separately compiled source candidate and +`--extra-object` includes a non-manifest input. None changes the comparison +manifest or matching status. Do not commit or distribute SDK files extracted +for a local probe. + +The script makes two attempts. `full` honors object default-library directives +and is the authoritative link outcome. `diagnostic` adds `/NODEFAULTLIB` to +expose otherwise-hidden unresolved symbols. Both use `/SUBSYSTEM:XBOX`, the +January `_mainCRTStartup` entry, and a response file. `summary.json` records +the linker result, diagnostic counts, and before/after SHA-256 of every input +object. A nonzero script exit means the full PE link did not succeed. + +## Current measurements + +At canonical `f6d00a8c`, all 621 manifest-backed non-missing objects exist and +were passed to the linker. Their combined before/after digest matched. + +| Inputs | Libraries | Full-link result | +| --- | --- | --- | +| All 621 rebuilt objects | None | `LIBC.lib` unavailable; nine duplicate definitions. | +| All 621 rebuilt objects | Local XDK 3911 library directory | 403 unresolved symbols; 33 duplicate definitions. | +| 475 objects (Halo and reconstructed Bink; excluding reconstructed `libcmt`, `d3d8`, `xapilib`) | Local XDK 3911 library directory plus selected Xbox SDK libraries | 156 unresolved symbols; nine duplicate definitions. | + +An explicitly **hybrid**, non-publishable experiment selected 473 objects, +substituted scratch-compiled versions of two recorded-animation TUs, supplied +January's original data-only `linker_common.obj`, and used local later XDK/Bink +libraries. It reached **four unique unresolved symbols** +(`DirectSoundGetStreamVoiceStatus`, `DirectSoundStopStream`, `fast_ftol_C`, +`main_crash`) and **four unique duplicate definitions**. It did not link. +The original data object and later libraries are proof-of-concept inputs, not +reconstructed source or matching credit. + +Adding January's original `dsstrmh/main.obj` (a vendor object containing the +two absent DirectSound functions) reduced that experiment to **two unresolved +symbols** (`fast_ftol_C`, `main_crash`) and the same four unique duplicate +definitions. These two Halo functions are already subject to separate +house-rule holds; they must not be silently replaced by dummy source or +counted as matched. The four providers require source/ownership work, not +`/FORCE:MULTIPLE`. + +A further genuine source repair removed `items.c`'s hand-written +`object_get_type` copy and used the existing `objects.h` inline. Its 18/18 +functions stayed exact, leaving **three** unique provider conflicts: +`cross_product2d`, `real_local_random`, and `plane2d_from_points`. The latter +two do not have an evidence-backed zero-regression source fix yet; making +them private merely to silence the linker would be unauthentic. + +The recorded-animation change itself is authentic and safe: January marks the +five duplicated `_apply_*` functions static in both TUs. Making their +prototypes/definitions static eliminated those five duplicate providers; both +scratch-compiled objects retained every exact function (14/14 and 13/13). + +The no-default-library diagnostic on all 621 objects reports 478 unique +unresolved externals and the same nine duplicate definitions. The nine include +`_object_get_type`, `_real_local_random`, `_plane2d_from_points`, five +`recorded_animation_playback` helpers, and `_cross_product2d`. This is a real +link audit, **not** a bootable game or a byte-match regression. XDK 3911 is +newer than the January target, so its libraries are a diagnostic candidate, +not proof of an authentic final link. + +Next: recover/declare genuinely missing Halo COMMON globals, resolve duplicate +providers through correct ownership and link composition, and locate compatible +Bink/Xbox libraries. Only when the PE link succeeds should `imagebld` convert +it to an XBE, followed by an XISO boot test. Neither the proprietary SDK +libraries nor the original game assets belong in this repository. + +The locally extracted XDK 3911 `imagebld.exe` reports `/IN:` and +`/OUT:`; it has not yet been tried on a rebuilt image. Its newer SDK +version remains a compatibility caveat. diff --git a/source/cutscene/recorded_animation_playback.c b/source/cutscene/recorded_animation_playback.c index 318e505c..e093607a 100644 --- a/source/cutscene/recorded_animation_playback.c +++ b/source/cutscene/recorded_animation_playback.c @@ -203,27 +203,27 @@ void recorded_animation_initialize_unit_control( byte **stream, byte unit_control_data_version); -void apply_animation_state( +static void apply_animation_state( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, byte const **playback_stream); -void apply_aiming_speed( +static void apply_aiming_speed( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, byte const **playback_stream); -void apply_control_flags( +static void apply_control_flags( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, byte const **playback_stream); -void apply_weapon_index( +static void apply_weapon_index( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, byte const **playback_stream); -void apply_throttle( +static void apply_throttle( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, @@ -453,7 +453,7 @@ void byte_swap_recording_stream( /* ---------- private code */ -void apply_animation_state( +static void apply_animation_state( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, @@ -471,7 +471,7 @@ void apply_animation_state( return; } -void apply_aiming_speed( +static void apply_aiming_speed( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, @@ -489,7 +489,7 @@ void apply_aiming_speed( return; } -void apply_control_flags( +static void apply_control_flags( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, @@ -507,7 +507,7 @@ void apply_control_flags( return; } -void apply_weapon_index( +static void apply_weapon_index( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, @@ -525,7 +525,7 @@ void apply_weapon_index( return; } -void apply_throttle( +static void apply_throttle( struct animation_playback_controller *animation_state, struct recorded_unit_control *control, struct animation_event_header const *header, diff --git a/source/cutscene/recorded_animation_playback_v1.c b/source/cutscene/recorded_animation_playback_v1.c index a95e585e..c30a75dd 100644 --- a/source/cutscene/recorded_animation_playback_v1.c +++ b/source/cutscene/recorded_animation_playback_v1.c @@ -213,23 +213,23 @@ void recorded_animation_initialize_unit_control( byte **stream, byte unit_control_data_version); -void apply_animation_state( +static void apply_animation_state( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *event, byte const **playback_stream); -void apply_aiming_speed( +static void apply_aiming_speed( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *event, byte const **playback_stream); -void apply_control_flags( +static void apply_control_flags( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *event, byte const **playback_stream); -void apply_weapon_index( +static void apply_weapon_index( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *event, byte const **playback_stream); -void apply_throttle( +static void apply_throttle( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *event, byte const **playback_stream); @@ -420,7 +420,7 @@ void byte_swap_recording_stream_v1( /* ---------- private code */ -void apply_animation_state( +static void apply_animation_state( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *anim_event_v1, byte const **playback_stream) @@ -439,7 +439,7 @@ void apply_animation_state( return; } -void apply_aiming_speed( +static void apply_aiming_speed( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *anim_event_v1, byte const **playback_stream) @@ -458,7 +458,7 @@ void apply_aiming_speed( return; } -void apply_control_flags( +static void apply_control_flags( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *anim_event_v1, byte const **playback_stream) @@ -477,7 +477,7 @@ void apply_control_flags( return; } -void apply_weapon_index( +static void apply_weapon_index( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *anim_event_v1, byte const **playback_stream) @@ -496,7 +496,7 @@ void apply_weapon_index( return; } -void apply_throttle( +static void apply_throttle( struct recorded_unit_control *control, struct recorded_animation_event_v1 const *anim_event_v1, byte const **playback_stream) diff --git a/source/items/items.c b/source/items/items.c index 80597d71..74ef687c 100644 --- a/source/items/items.c +++ b/source/items/items.c @@ -58,7 +58,6 @@ symbols in this file: /* ---------- headers */ -#define object_get_type items_object_get_type_inline #include "cseries.h" #include "cseries/profile.h" @@ -77,7 +76,6 @@ symbols in this file: #include "scenario/scenario.h" #include "sound/game_sound.h" #include "units/units.h" -#undef object_get_type /* ---------- constants */ @@ -111,12 +109,6 @@ real const item_maximum_impact_velocity = 0.1f; /* ---------- public code */ -short object_get_type( - long object_index) -{ - return object_header_get(object_index)->type; -} - void items_initialize( void) { diff --git a/tools/link_probe.py b/tools/link_probe.py new file mode 100644 index 00000000..e76b0438 --- /dev/null +++ b/tools/link_probe.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +"""Opt-in Xbox PE link attempt using the already-compiled comparison objects. + +This is deliberately separate from configure.py and the byte-matching graph. +It never edits source, object files, objdiff configuration, or matching reports. +The diagnostic pass suppresses unavailable default libraries so the linker can +report unresolved symbols; neither pass uses /FORCE or claims to make an XBE. +""" + +import argparse +import hashlib +import json +import re +import subprocess +from pathlib import Path + + +DIAGNOSTIC_CODES = ("LNK2001", "LNK2005", "LNK2019", "LNK1104", "LNK1181") + + +def manifest_objects( + root: Path, + excluded_projects: set[str] | None = None, + excluded_sources: set[str] | None = None, + replacements: dict[str, Path] | None = None, +) -> tuple[list[Path], list[dict[str, str]]]: + config = json.loads((root / "config" / "config.json").read_text(encoding="utf-8")) + objects: list[Path] = [] + missing: list[dict[str, str]] = [] + for project in config["projects"]: + if excluded_projects and project["name"] in excluded_projects: + continue + for entry in project["objects"]: + if entry["status"] == "MISSING" or (excluded_sources and entry["name"] in excluded_sources): + continue + path = (replacements or {}).get(entry["name"]) + if path is None: + path = root / "build" / "base" / Path(entry["name"]).with_suffix(".obj") + if path.is_file(): + objects.append(path.resolve()) + else: + missing.append({"project": project["name"], "source": entry["name"], "object": str(path)}) + return objects, missing + + +def object_digest(objects: list[Path]) -> str: + digest = hashlib.sha256() + for path in objects: + digest.update(str(path).encode("utf-8")) + digest.update(b"\0") + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def diagnostic_summary(output: str) -> dict[str, object]: + lines = output.splitlines() + counts = {code: sum(code in line for line in lines) for code in DIAGNOSTIC_CODES} + samples = [line for line in lines if re.search(r"\bLNK\d{4}\b", line)] + unresolved = set() + duplicates = set() + for line in lines: + match = re.search(r"unresolved external symbol (.*?)(?: referenced in function|$)", line) + if match: + unresolved.add(match.group(1)) + duplicate = re.search(r"LNK2005: (.*?) already defined", line) + if duplicate: + duplicates.add(duplicate.group(1)) + summary = re.search(r"LNK1120: (\d+) unresolved externals", output) + return { + "counts": counts, + "unique_unresolved_seen": len(unresolved), + "unique_duplicate_definitions": len(duplicates), + "duplicate_symbols": sorted(duplicates), + "linker_unresolved_count": int(summary.group(1)) if summary else None, + "first_diagnostics": samples[:30], + } + + +def run_link( + linker: Path, + objects: list[Path], + extra_objects: list[Path], + output_dir: Path, + label: str, + library_dirs: list[Path], + libraries: list[Path], + suppress_default_libraries: bool, +) -> dict[str, object]: + image = output_dir / f"{label}.exe" + map_file = output_dir / f"{label}.map" + response = output_dir / f"{label}.rsp" + args = [ + "/NOLOGO", + "/MACHINE:X86", + "/SUBSYSTEM:XBOX", + "/ENTRY:mainCRTStartup", + "/INCREMENTAL:NO", + "/OPT:NOREF", + f"/OUT:{image}", + f"/MAP:{map_file}", + ] + if suppress_default_libraries: + args.append("/NODEFAULTLIB") + args.extend(f"/LIBPATH:{path}" for path in library_dirs) + args.extend(str(path) for path in objects) + args.extend(str(path) for path in extra_objects) + args.extend(str(path) for path in libraries) + response.write_text("\n".join(f'"{arg}"' for arg in args) + "\n", encoding="ascii") + completed = subprocess.run( + [str(linker), f"@{response}"], + cwd=output_dir, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + check=False, + ) + output = completed.stdout + completed.stderr + (output_dir / f"{label}.log").write_text(output, encoding="utf-8") + return { + "exit_code": completed.returncode, + "image_exists": image.is_file(), + "response": str(response), + "log": str(output_dir / f"{label}.log"), + **diagnostic_summary(output), + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--object-root", type=Path, default=Path.cwd()) + parser.add_argument("--output-dir", type=Path, default=Path("build/link-probe")) + parser.add_argument("--linker", type=Path) + parser.add_argument("--library-dir", type=Path, action="append", default=[]) + parser.add_argument("--library", type=Path, action="append", default=[]) + parser.add_argument("--extra-object", type=Path, action="append", default=[]) + parser.add_argument("--replace-object", action="append", default=[], metavar="SOURCE=OBJECT") + parser.add_argument("--exclude-project", action="append", default=[]) + parser.add_argument("--exclude-source", action="append", default=[]) + args = parser.parse_args() + + root = args.object_root.resolve() + output_dir = args.output_dir.resolve() + linker = (args.linker or root / "xbox/bin/vc7/Link.Exe").resolve() + if not linker.is_file(): + parser.error(f"linker not found: {linker}") + if output_dir == root or output_dir.is_relative_to(root / "build/base") or output_dir.is_relative_to(root / "build/split"): + parser.error("output must not overwrite the repository or comparison objects") + replacements = {} + for definition in args.replace_object: + source, separator, object_path = definition.partition("=") + if not separator or not source or not object_path: + parser.error(f"invalid replacement: {definition}") + if source in replacements: + parser.error(f"duplicate replacement: {source}") + replacements[source] = Path(object_path).resolve() + config = json.loads((root / "config" / "config.json").read_text(encoding="utf-8")) + manifest_names = {entry["name"] for project in config["projects"] for entry in project["objects"] if entry["status"] != "MISSING"} + unknown_replacements = set(replacements) - manifest_names + if unknown_replacements: + parser.error(f"unknown manifest sources: {sorted(unknown_replacements)}") + unknown_exclusions = set(args.exclude_source) - manifest_names + if unknown_exclusions: + parser.error(f"unknown manifest sources: {sorted(unknown_exclusions)}") + if set(replacements) & set(args.exclude_source): + parser.error("a source cannot be both replaced and excluded") + objects, missing = manifest_objects(root, set(args.exclude_project), set(args.exclude_source), replacements) + if missing: + print(f"Cannot link: {len(missing)} manifest objects have not been compiled.") + for entry in missing[:20]: + print(entry["object"]) + return 2 + library_dirs = [path.resolve() for path in args.library_dir] + libraries = [path.resolve() for path in args.library] + extra_objects = [path.resolve() for path in args.extra_object] + for library_dir in library_dirs: + if not library_dir.is_dir(): + parser.error(f"library directory not found: {library_dir}") + for library in libraries: + if not library.is_file(): + parser.error(f"library not found: {library}") + for extra_object in extra_objects: + if not extra_object.is_file(): + parser.error(f"extra object not found: {extra_object}") + output_dir.mkdir(parents=True, exist_ok=True) + + before = object_digest(objects + extra_objects) + results = { + "object_root": str(root), + "object_count": len(objects), + "excluded_projects": args.exclude_project, + "excluded_sources": args.exclude_source, + "extra_objects": [str(path) for path in extra_objects], + "replaced_objects": {key: str(value) for key, value in replacements.items()}, + "object_sha256_before": before, + "missing_manifest_objects": missing, + "full": run_link(linker, objects, extra_objects, output_dir, "full", library_dirs, libraries, False), + "diagnostic": run_link(linker, objects, extra_objects, output_dir, "diagnostic", library_dirs, libraries, True), + } + results["object_sha256_after"] = object_digest(objects + extra_objects) + results["input_objects_unchanged"] = before == results["object_sha256_after"] + (output_dir / "summary.json").write_text(json.dumps(results, indent=2) + "\n", encoding="utf-8") + print(f"Inputs: {len(objects)} objects; unchanged: {results['input_objects_unchanged']}") + for label in ("full", "diagnostic"): + result = results[label] + print(f"{label}: exit {result['exit_code']}; {result['counts']}; log: {result['log']}") + print(f"Summary: {output_dir / 'summary.json'}") + if not results["input_objects_unchanged"]: + return 3 + return 0 if results["full"]["exit_code"] == 0 else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/test_link_probe.py b/tools/test_link_probe.py new file mode 100644 index 00000000..6f8772a1 --- /dev/null +++ b/tools/test_link_probe.py @@ -0,0 +1,64 @@ +import json + +from tools.link_probe import diagnostic_summary, manifest_objects + + +def test_manifest_excludes_missing_units_and_requires_compiled_inputs(tmp_path): + config = tmp_path / "config" + config.mkdir() + (config / "config.json").write_text( + json.dumps( + { + "projects": [ + { + "name": "halo", + "objects": [ + {"name": "source/ready.c", "status": "NonMatching"}, + {"name": "source/unbuilt.c", "status": "Matching"}, + {"name": "source/synthetic.c", "status": "MISSING"}, + ], + } + ] + } + ), + encoding="utf-8", + ) + ready = tmp_path / "build" / "base" / "source" / "ready.obj" + ready.parent.mkdir(parents=True) + ready.write_bytes(b"object") + + objects, missing = manifest_objects(tmp_path) + + assert objects == [ready.resolve()] + assert [entry["source"] for entry in missing] == ["source/unbuilt.c"] + + replacement = tmp_path / "replacement.obj" + replacement.write_bytes(b"new object") + objects, missing = manifest_objects( + tmp_path, + excluded_sources={"source/unbuilt.c"}, + replacements={"source/ready.c": replacement}, + ) + assert objects == [replacement.resolve()] + assert not missing + + +def test_diagnostic_summary_distinguishes_references_from_unique_symbols(): + output = "\n".join( + [ + "a.obj : error LNK2019: unresolved external symbol _missing referenced in function _a", + "b.obj : error LNK2001: unresolved external symbol _missing", + "c.obj : error LNK2005: _duplicate already defined in d.obj", + "LINK : fatal error LNK1120: 1 unresolved externals", + ] + ) + + summary = diagnostic_summary(output) + + assert summary["counts"]["LNK2019"] == 1 + assert summary["counts"]["LNK2001"] == 1 + assert summary["counts"]["LNK2005"] == 1 + assert summary["unique_unresolved_seen"] == 1 + assert summary["unique_duplicate_definitions"] == 1 + assert summary["duplicate_symbols"] == ["_duplicate"] + assert summary["linker_unresolved_count"] == 1