* Build: check out scripts with LF line endings
* Build: replace the build scripts with Kazarov
* Port: rename PC_OVERRIDE to Kazarov's PORT_OVERRIDE
* Build: build the PS2 executables through Kazarov's CMake modules
* Scripts: remove unused tooling
* Build: update Kazarov
* Build: update Kazarov
* Build: run the port without a disc when its data exists, reject the old REGION variable, and lock the shared Windows dependencies
* Build: update Kazarov
* Port: serialise game data extraction across configurations
* Build: update Kazarov
* Port: hold a shared data lock while the game runs
* Port: refuse to start on data another configuration re-extracted
* Build: update Kazarov
Game text can come from JSON files, with the disc's own message files as the
fallback, and message text can be drawn from a TrueType font so it is sharp at
any window size. Nothing from the disc is added to the repository.
- The files are laid out as Minecraft's language files are: one flat JSON file
per language (en_gb.json, fr_fr.json...), each a "dotted.key": "text" object
with the same keys in every language. A key is the message file's key (its
path with the language and the .mes taken out, slashes made dots) and the
message id: "dun.script.d01.d01.12".
- localize.{hpp,cpp} reads <exe>/lang and <save>/lang and lays the text over a
message file as the game loads it (LoadFile2, LoadFileBG and the pack lookup
GetPackFile). A missing message, file, bad JSON or a text that does not fit
keeps the disc's text and says so once on stderr. patches/*.json make small
edits to the disc's text (replace pairs, or whole text, guarded by a hash).
- darkcloud --export-text DIR, run once automatically for a save folder with
no export, writes one file per language from the user's own data (855
message files on the PAL disc, each reading back to the disc's codes), and
the hashes patches check. The export is gitignored.
- The Options screen's own words are looked up by key too
(options.<setting>.label, .help, .choice.<n>, options.page.<name>), with the
English in the code as the fallback; port/lang/ ships French, German,
Italian and Spanish for them, copied beside the executable by the build.
- lang/font.ttf draws message text with stb_truetype at the screen's
resolution, in the game's own cells; letters the font lacks (accents,
cedilla, inverted marks, oe) are built from the font's own glyphs.
- A Language row on the Game page and game.language in config.json skip the
start-up language screen; changing it reloads the town or title once Options
closes so images, menus and messages all change together.
- scripts/port/compare_text.py compares the languages message by message.
- docs/LOCALIZATION.md describes the formats and limits.
Japanese is not decoded: its text is glyph indices into the game's own font,
so ja_jp.json keeps them as {N} codes, which still round-trip.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The ' of 1'000 or 0xff'ff opened a character literal, whose masking hid
the declaration's commas and semicolons up to the next '. A number is
now taken whole first and left as it is. A macro that expands to a
comma or a semicolon in a replaced definition is documented as
unsupported.
Telling a template argument list from a comparison or a shift kept
letting a second name through, stripped unseen. Angle brackets are no
longer counted: a declaration with a comma outside (), [] and {} stops
the list and asks for a type alias or a split.
An expression such as 1 << 2 in a grouped declaration's initialiser was
taken for a template argument list, which hid the comma before the next
name. Angle brackets now count only before the initialiser.
pc_override.py took the first name of a grouped declaration such as
int (*fp)() = nullptr, (*keep)() = nullptr; and stripped the whole
statement, so keep was lost without a word. A tagged one now stops the
list with its line; in ps2/src one is left whole and the check names the
override it does not find. An uninitialised pointer-to-function variable
is read too.
pc_override.py took a variable whose head held parentheses for a
declaration. A pointer to a function, or an array of them, carries its
name inside its declarator, as memcard.cpp's SaveMenuFunc does.
* Tag port/src's replacements with PC_OVERRIDE
PC_OVERRIDE goes in front of each of the 323 definitions in port/src that
replace one of ps2/src's: 312 functions, main and ten globals of the title
sequence. port.h defines the macro as nothing, so nothing is compiled
differently yet.
* Build the port from ps2/src without the definitions port/src overrides
scripts/port/pc_override.py lists the names port/src's PC_OVERRIDE
definitions link under and writes a copy of each ps2/src unit without its
definitions of those names; the port compiles the copies. A replaced
function is left as its declaration, a member function is removed and a
variable becomes an extern declaration. ps2/src itself is not touched.
Each name then has one definition on every platform, so the merged object,
llvm-objcopy --weaken, -fsemantic-interposition, the interposition check
and tools/weaken go.
scripts/port/truncations.py reads each unit as ps2/src has it, so the
replaced bodies stay in docs/port/truncations.md, and it refuses a build
that is not a Debug one: without semantic interposition an optimised build
inlines functions whose bodies the link then drops.
* Describe PC_OVERRIDE in the port's documentation
The PAL release sped the game up to play its 60 Hz timing at 50: motion
keys and SetMotionSpeed by 6/5, the opening's camera step, fades, cues,
dance and footstep windows, beam and title-sprite speeds, Jinn's and Ruby's
shot windows, and the loading screen's logo holds. Those blocks are now
#ifdef PAL_TIMING, which the PS2 PAL build defines beside PAL, so both
releases compile as before (MWCC emits identical code and data for every
changed unit, PAL and NTSC). The port defines PAL alone: it gets PAL's code
with NTSC's timing, its own copies of those functions take the NTSC
values, and the tick rate defaults to 60 a second.
Co-authored-by: Thomas Cantwell <thomascantwell@Thomass-Mac-mini.local>
The audit lists no round trip that can run, so the crutches that kept the rest alive go:
darkcloud is linked -pie -z separate-code (objects are PIC throughout), the arenas are ordinary
mappings with no MAP_32BIT, and --high-arenas, DC_HIGH_ARENAS, ArenaMemoryIsLow,
PortAssertLow and macOS's -pagezero_size option are removed. A debug build stops when no arena
lies above 4 GiB, so a mapping that drifts low cannot hide a truncation. The --defsym aliases
and EditGaijiTbl's assembler alias work unchanged under PIE.
DrawPartsCursor compares the preview frame's low 32 bits without casting the pointer, which the
audit read as a round trip. truncations.py gains --check, which fails while
docs/port/truncations.md is stale or lists a round trip that can run.
bits64_arenas_high checks the arenas, the image and a function lie above 4 GiB and that the
arena alignment helpers still round correctly there. docs/PC.md lists every retail copy and
why; docs/MACOS_PLAN.md marks the Rosetta fallback unnecessary.
scripts/port/truncations.py runs libclang over the port's compile
commands and lists every cast of a pointer to a narrower integer in
src/ps2 and src/port, explicit or narrowed after a cast to a full-width
integer: its function and whether the linked darkcloud runs it (port,
replaced, retail, dead), where the pointer points (image, stack, arena,
field, ...), and whether the value comes back as a pointer, following
integer assignments, arguments and returns across units by USR. Its
count of explicit casts matches clang's -Wpointer-to-int-cast exactly.
docs/port/truncations.md is its output. arm64 macOS cannot keep any of
these values alive, so the round trips that can run are the list of
replacements it still needs.
scripts/host/mesa-macos.sh builds mesa-26.2.4, the newest tag found that
carries KosmicKrisp (it first ships in 26.0), with kosmickrisp and swrast
into a prefix and prints the VK_DRIVER_FILES lines for both. Reruns reuse a
prefix holding the tag's build.
The macos job runs on macos-26, since KosmicKrisp is built on Metal 4: it
builds Mesa through the script under a cache keyed by tag, builds the port
with the macos-arm64 preset, runs ctest on lavapipe, and runs darkcloud on
KosmicKrisp headless and in a window with vulkaninfo summaries, all kept as
artifacts. The KosmicKrisp steps report without failing the job until the
first green run. The Ubuntu job is unchanged.
src/common becomes src/ps2 and holds only the game's code as MWCC compiles
it: every DC_PC guard is gone, leaving those units as they were before the
port. src/pc becomes src/port.
The root CMakeLists.txt only picks the platform. The PS2 build moves to
src/ps2/CMakeLists.txt (with its modules in src/ps2/cmake), the port's to
src/port/CMakeLists.txt.
The port copies each src/ps2 unit into the build tree with its MWCC inline
assembly turned into PS2_ASM() stubs (src/port/cmake/StubAsm.cmake), so the
hand-written stubs for assembly functions are no longer needed. A unit that
needs MWCC-only names supplied gets src/port/shims/<unit>.h; mathutil's
renames the runtime's std::exception apart from the host's. __dl's
declaration gains the throw() its definition has.
src/ps2 now compiles with -Wall rather than -w. Retail's uninitialised
reads and missing returns stay in the source -- initialising any of them
changes what MWCC emits -- and the port defines them instead with
-ftrivial-auto-var-init=zero and -fno-strict-return.
* Name the discs by region and put every build input and output under a region directory
The disc images are rom/Dark Cloud (NTSC).iso and rom/Dark Cloud (PAL).iso;
each region's checksums and extracted files live under rom/ntsc and rom/pal.
scripts/build/region.py turns REGION (NTSC by default, or PAL) into the paths
every script and CMake use: config/<region> holds the split configuration,
symbol lists, fixups and linker script, splat writes asm/<region>, and CMake
builds into build/<region>. The overlay headers and pads are splat pad
segments, so no assets/ tree is written, and the unused config/reference_asm
and reference_asm.py are gone. overlay_private.sh places a private-repository
file still kept under rom/extracted/ in rom/ntsc/extracted/.
* Carry PR 40's PAL source differences; NTSC unchanged
The PAL prototype's small divergences sit inline under #ifdef PAL in 30
units; common.h gains the per-region screen geometry and GS Y offset,
clsmes.hpp the message window width and ClsMes size. The twenty PAL-only
functions stand as stubs at their retail positions, dataread's four
tree-building functions are NTSC-only, and PAL compiler-state pragma
blocks sit beside NTSC's. config/pal/splice.txt lists the 213 functions
whose PAL bytes are not yet expressed in C. Every unit compiles with
-DPAL; every NTSC object is byte-identical.
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
* Generate the PAL split configuration from NTSC's
scripts/build/splat_config.py writes a release's symbol lists from its own
executable and carries NTSC's yaml, linker script and object fixups over to
another release by symbol and relocation pairing: the overlay entry points by
the retail name they note, _gp from .reginfo, @ORDER comments and the dun text
start, and function-local statics through the function that owns them. The
carried files are written once and maintained by hand; --check compares the
symbol lists and, on NTSC, proves the carry is the identity. region.paths()
names any release's files.
config/pal holds the PAL prototype's yaml, symbol lists, linker script, fixups
and expression-node overrides. NTSC unchanged.
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
* Supply the PAL prototype's undecompiled functions from INCLUDE_ASM markers
Each PAL function not yet in C is `#ifdef PAL INCLUDE_ASM("asm/pal/nonmatchings/
<unit>", <symbol>)` beside its NTSC body (207 markers); include_asm.h and the
mwccgap marker path are back for PAL, with the name counter pinned where a
marker stands so later compiler constants keep their names. PAL-differing data
is C under #ifdef PAL in the owning units. datacheck.py reads a relocation's
target section from the REL entry's info field. docs/PAL.md describes the PAL
build. NTSC is byte-identical; PAL links with 120 functions and 2 data bytes
still to reconcile.
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
* Carry PR 40's PAL data and layout; every PAL image verifies
The PAL linker script places EditGaijiTbl at 0x0025f02c, aligns _dun_end to
128, drops dngmessageman's small-data line (PAL's LanguageStr is in
hitvalue.cpp) and aligns editmapscript's .sbss to 16 over retail's unnamed
pad at 0x002a9f3c. title/op_c loads the Chara__3 entries NTSC does in
OpC_InitProcess2/5; battlemenu loads wepstatus and reloads DrawAallWeapon's
textures from BtlMenuExtBlock under PAL.
datacheck.py resolves retail's name__N spellings (Names), scores each voted
anchor on its own layout and resolves a pointer word by its exact name.
PAL: SCUS_971.11 3048 perfect / 154 asm, TITLE.BIN 126 / 44, DUN.BIN 52 / 9,
data perfect. NTSC unchanged.
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
* PAL: 29 main-executable functions in C from PR 40; NTSC unchanged
battle_globals, battlemenu (9), memcard (2), menu_dungeon, shot_freefuncs,
clsmes, dataread (2), edit, editloop, fireomni, frame, mglib, texture (3),
stockitem, chararead, dungeonmap and spritetable express their PAL
differences under #ifdef PAL instead of INCLUDE_ASM markers; editloop's PAL
argument-flag indices follow. mglib.hpp declares PAL's mgTopVRAM and
mgZBufferAdr.
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
* PAL: 10 title-overlay functions in C from PR 40; NTSC unchanged
op_a (2), op_b, op_c (2), op_d, opening, rushmovi (2) and title express their
PAL differences under #ifdef PAL. A string retail shares with a function
still supplied by a marker is reached through PAL_POOLED, so the compiled
function does not emit a second copy.
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
* PAL: dun, gamepad, eastking, menu_draw and collisiondata functions in C from PR 40
dun/gameloop (2), gamepad (3), eastking (2), menu_draw (2) and collisiondata
express their PAL differences under #ifdef PAL; dun/gameloop takes PR 40's
PAL argument-flag block, InitEastKingEvent returns int under PAL, and the
PAL expression-node overrides carry PR 40's five entries. The PAL fixups
follow every unit's renumbered statics. All three PAL images verify with
158 functions still supplied by markers; NTSC unchanged.
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
* Make decompile.sh's m2c context parse again
m2ctx.py spells a C++ template-id as a plain identifier, and now declares each
one it spells as an opaque struct, so CDataAlloc<1, 1690000> no longer breaks
m2c's C parser. decompile.sh generates a missing context with the container's
venv Python, which has libclang's bindings.
* Work on the PAL version
---------
Co-authored-by: Alessandro Suha <plarpoon@zandark.eu>
- scripts/build/retail.py reads retail's bytes from the disc images; literals.py
binds float loads and postprocess_object.py matches rodata_exports from them
instead of from splat's dumps.
- Function-local statics are paired with retail's names by a new `statics`
fixup rather than by reading each unit's reference assembly; the unused
data_runs machinery is gone.
- The VU microprograms, the frame's DMA chain, the overlay table and the
literal pool are data units in C++ (vutext.cpp, vudata.cpp, literals.cpp);
the overlays' entry points and dun's text start are linker-script symbols.
src/handwritten/ and the last INCLUDE_ASM (and include_asm.h) are gone.
Only crt0.s and the SDK/runtime units under asm/lib are still assembled.
With asm/ holding nothing but lib, the build links byte-identical images.
The 170 constants the markers spliced from retail's assembly are defined in
their units at the markers' places: strings as named char arrays declared in
the owner's header and renamed to retail's @N by the owner's `symbols` fixup,
and mathutil's std::exception/std::bad_exception type information as
MWTypeInfo/MWBaseClass records. Strings other units pass as `char *` stay
`char` and are placed in .rodata explicitly; edit's villager and command names
are const.
postprocess_object.py looks a renamed constant's alignment up by its retail
name, which is what keeps each string in its retail slot now that the padding
no longer travels with it. The INCLUDE_RODATA macro and the machinery that read
it are gone. Every image stays byte-identical.
npcharacter.cpp emits __vt__12CNPCharacter now that CNPCharacter derives
from CCharacter as retail's vtable says (its overrides are Step,
ShadowStep, Draw, DrawShadow, PickUpPoly and Initialize); mathutil.cpp
carries the runtime's std::exception and std::bad_exception vtables, whose
type-information slot binds to sdr_main's RTTI record rather than
mathutil's own copy, so they are written as data; the linker script
defines __static_init_end.
With no dump part left to carve, CMakeLists.txt loses carve_part,
carve_dump and the residual-part lists, scripts/build/carve_part.py and
filter_asm_symbols.py go, and the checked-in asm/data/main/parts are
deleted. 221 file-scope `extern` lines that repeated a declaration an
included header already makes are removed.
SCUS_971.11: 3186 perfect, 0 asm, 0 unmatched; data perfect; every named
data symbol at retail's address. TITLE.BIN and DUN.BIN OK.
Every .data/.sdata/.sbss/.bss datum that a carved whole-section dump part
still supplied is defined by its owning C++ unit, in retail address order,
so the linker script places one run per unit per section instead of
interleaving unit runs with dump parts. All 20 static initialisers
(__sinit_*.cpp) and their .ctor entries are generated by their units; the
linker script defines __static_init. Retail's data-only units come back as
src/menuetc.cpp and src/editpartsdata.cpp.
The library units' .data/.bss come from the split: config/main.yaml gives
each one its subsegments, newlib's data-only impure.c is a unit of its own,
and the linker-script symbol _gp no longer gets a label inside libmc's bss.
crt0.s carries __data_start and _args. The literal pool, which this
toolchain cannot regenerate, is src/handwritten/literals.s.
DUN.BIN's remaining .ddata, its __sinit_gameloop.cpp and its main-image
small data come from gameloop.cpp.
Left as dump parts: __vt__12CNPCharacter and the runtime's std::exception
vtables with __static_init_end.
scripts/build/datacheck.py lays a unit's data out the way MWLD will and
compares it with retail's addresses and bytes; layout.py takes
`// @ORDER <obj> <addr>` for a unit without code.
SCUS_971.11: 3186 perfect, 0 asm, 0 unmatched; data perfect; every named
data symbol at retail's address. TITLE.BIN and DUN.BIN OK.
Whether MWCC reuses a float literal's load across a store through an
unknown address depends on a byte it reads 0x1A past the literal's value
(0x00432F50). Retail's whole-program compile left that byte nonzero for
0.01f, so it reloads; a standalone compile of weaponelement.cpp does not.
`#pragma literal_reload <bits>` makes statefix answer "may be overwritten"
for the named literals, and weaponelement.cpp names 0.01f.
SCUS_971.11: 3181 perfect, 4 asm, 1 unmatched (DrawItemBox); data perfect.
TITLE and DUN OK.
- AnimeDataInit and PresetSmallItemNo_Get are promoted. ITEM_PUT_SET and
ItemPutListPtr move to dungeonparts.hpp, and dungeonparts.cpp pins its
argument reads so setCollisionData keeps retail's argument order now that
PresetSmallItemNo_Get is compiled ahead of it.
- crt0 is hand-written assembly in src/crt0.s. read_units lets a yaml
asmtu unit take a src/<unit>.s source, CMake assembles it in place of the
split assembly, and objdiff scores it against asm/crt0.s. src/crt0.c is
gone.
SCUS_971.11: 3180 perfect, 4 asm, 2 unmatched (Step_Fire and DrawItemBox,
both failing before this change); data perfect. TITLE and DUN OK.
EdMoveChara is written from retail and matches byte for byte. Its .data
templates and .sbss statics now come from ebattle.cpp.
Typework:
- ED_MOVE_CHARA_INFO: camera is the CCameraFollow; view_camera, acted,
fishing and motion_time_before/after are named.
- CMainChara's tail is a MoveCheckInfo, with landed and ground_found.
- CNPCharacter::talk_target and CSaveData::mardan_garayan_caught are named.
- EdVillager, EdVillagerInfo, EdSearchNearNPC, EdTalkModeInit, EdInitHashigo,
EdMenuLoop and FishLineStep are declared in headers.
Also:
- Recovered the previous session's uncommitted work: rewritten drafts for
menu_dungeon, gameutil and others, and CheckItemVolume promoted.
- CWater::CreateVUData is rewritten from retail. It is still behind
NON_MATCHING: same instructions as retail, registers still differ.
Retail's exception table is empty, so mathutil's object drops the tables its exceptions-on runtime functions emit; postprocess_object.py gains a remove_sections fixup for that.
Every per-character value is read up front and the item is dispatched through a switch. promote.py no longer lets a draft's block run across an earlier standalone guard.
Its zero arguments come from two locals, which is what puts them ahead of
the frame load, and it prints LoadModel's copy of the overflow message.
quicktu also finds a function whose name carries a template argument.
asm/ is splat's output and is now generated on every machine rather than
checked in. The exception is asm/data/main/parts, the hand-edited residual
data dumps, which cannot be regenerated and stay committed. The one
hand-written file moves to src/handwritten.
Without the disc image, the three binaries splat reads can be copied into
rom/extracted/iso from a private repository that mirrors this tree's layout,
with scripts/host/overlay_private.sh. They are checked against
rom/extracted.sha256 before every split.
The progress workflow does exactly that: it checks out the repository named
by the PRIVATE_REPOSITORY variable with the PRIVATE_REPO_SSH_KEY deploy key,
and skips when either is missing, as it is for pull requests from forks.
The reverse-engineering notes under re/ai are no longer tracked.
dataread's .rodata holds two runs: @234..@468, reached only by the
archive readers, and @187..@255 on the padded 0x00299180, reached first
by InitCloth. Its .data agrees -- CurrentDir and LoadFile2's templates
end at 0x002517A0, where the cloth command tables begin.
So the tail from 0x0013F9C0 is its own object: InitCloth, the nine
cloth-description commands and the parser they share. It takes the unit
name counter that was pinned for it, 187, and dataread gets its own, 234.
Adds tu_rodata_runs.py, which reads those runs out of every unit and
names the first function that reaches each.
The header work from the annotation pass: field names, signatures and
prose for actionseq, the effects, the shot helpers, editloop3 and the
dungeon game loop, plus the four headless scripts that apply, verify,
export and inventory the annotations.
MWCC numbers a unit's invented names from one in parse order, so within a
unit they never go backwards. `battle_globals` ended on @1574 and then
ran @15, @17, @24, @32, @34, @50-52 across seven functions --
`runerror`, `stkoverflow`, `chk_int`, `is_true`, `divby0error`,
`modby0error` and `print` -- and `runscript` picked up at @68 and carried
on. One ascending run, split by a boundary drawn 0x280 bytes too late.
Their constants say the same: @15 through @52 sit at 0x29F770-0x29F803,
immediately below @68 at 0x29F810 where `runscript`'s rodata subsegment
was said to start. The run is contiguous; the boundary was not.
The text boundary moves to 0x23D6C0 and the rodata one to 0x29F770, the
seven move to the head of `runscript.cpp`, and its name counter starts at
15 rather than 68. All seven still match, and the image's data is still
byte-identical.
`tu_boundaries.py` reads the numbering and `tu_attribution.py` asks which
unit's constants each function reaches; together they are what found
this. Across the whole image they now agree with every boundary drawn.
Dylan Carr