diff --git a/AGENTS.md b/AGENTS.md index 20a6e0fb..5e5a277b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -190,10 +190,18 @@ otherwise. ### Third-party code and licences -The repository is CC0. Vendored code goes in `port/third_party/`, -copied unchanged, with a README naming the upstream, the version or commit, -the checksum and what uses it. GPL code (OpenSauce, Reclaimer) may be read -as documentation of file formats, never copied. +The repository is `AGPL-3.0-or-later` (`LICENSE`); the commits before the +change, and the releases built from them, stay CC0 (`LICENSES/CC0-1.0.txt`). +The README's "Licence" section says what each part is under. Vendored code +goes in `port/third_party/`, copied unchanged, with a README naming +the upstream, the version or commit, the licence, the checksum and what uses +it. Its licence must let it be combined with the AGPL 3.0: MIT, BSD (without +an advertising clause), ISC, zlib, Apache 2.0, MPL 2.0, LGPL or GPL 3.0 or +later, public domain. GPL-2.0-only code and the 4-clause BSD licence cannot +be. Code taken from another project, GPL 3.0 code included, keeps its +notices and its licence, and says where it came from. +The decompilation's upstream, punpckhdq/halo, is AGPL 3.0 from 17 October +2026; bnunu/halo-1 is CC0. ### Commits and pull requests diff --git a/LICENSE b/LICENSE new file mode 100644 index 00000000..be3f7b28 --- /dev/null +++ b/LICENSE @@ -0,0 +1,661 @@ + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +. diff --git a/LICENSE.md b/LICENSES/CC0-1.0.txt similarity index 100% rename from LICENSE.md rename to LICENSES/CC0-1.0.txt diff --git a/README.md b/README.md index 9af23c55..1648e52e 100644 --- a/README.md +++ b/README.md @@ -202,3 +202,34 @@ Enter `adb pull` to copy the recording parts to a computer. A profiling build plays with normal builds. It does not use `--pgo=train`. + +## Licence + +OpenCE is free software: you can give it to others and change it under the +terms of the GNU Affero General Public License as published by the Free +Software Foundation, version 3 of the License or (at your option) any later +version (`AGPL-3.0-or-later`). It comes with no warranty. The full text is in +[LICENSE](LICENSE). + +- **From the change.** The AGPL applies from the commit that made the change, + and to each release built from that commit or a later one. Earlier + releases, and the source at every earlier commit, were released under + [CC0 1.0](LICENSES/CC0-1.0.txt) and stay that way. +- **What OpenCE builds on.** The decompilation of + [punpckhdq/halo](https://github.com/punpckhdq/halo) is under the AGPL 3.0 + from 17 October 2026, and under CC0 1.0 before that date. + [bnunu/halo-1](https://github.com/bnunu/halo-1) is under CC0 1.0. +- **Contributions** are under `AGPL-3.0-or-later`. A change taken from a + project under GPL 3.0 keeps that licence and its notices; the GPL 3.0 and + the AGPL 3.0 let the two be combined (section 13 of each). +- **Hosts and servers.** If you change the game and let other persons play + with your version over a network (for example, as the host of an internet + game, or on a web site), the AGPL asks you to offer them the source of + your version (section 13). The game as released has its source here, and + each release's source is its tag (`build-`). +- **Third-party code, fonts and art** keep their own licences, which are + next to them (`port/third_party`, `port/assets/fonts`, + `port/assets/icons`, `port/assets/text_filter`). The release downloads and + the web site's `licenses.txt` carry their notices. +- **Halo is not covered.** The licence covers the work of OpenCE's authors. + It gives no rights in Halo, its maps or anything else that Microsoft owns. diff --git a/docs/custom_edition_caches.md b/docs/custom_edition_caches.md index 070a5efb..76a91cea 100644 --- a/docs/custom_edition_caches.md +++ b/docs/custom_edition_caches.md @@ -493,9 +493,9 @@ file the same as the host's; with `network.map_download` off, or a host with but their model parts have its 0x68-byte layout and their shader types its numbering, which makes them a check on what the conversion produces. -OpenSauce and Reclaimer are GPL-3.0 and this repository is CC0: no code from -either was copied. They were read as documentation of the file formats, and -every layout used was then checked against the sample maps. +No code from OpenSauce or Reclaimer (both GPL-3.0) was copied: they were +read as documentation of the file formats, and every layout used was then +checked against the sample maps. ### Documented facts (source, checked against the sample maps) diff --git a/port/android/app/src/main/java/com/halo/decomp/XisoExtractor.java b/port/android/app/src/main/java/com/halo/decomp/XisoExtractor.java index 95ce9372..31688036 100644 --- a/port/android/app/src/main/java/com/halo/decomp/XisoExtractor.java +++ b/port/android/app/src/main/java/com/halo/decomp/XisoExtractor.java @@ -4,94 +4,87 @@ import java.io.File; import java.io.FileOutputStream; import java.io.IOException; import java.nio.ByteBuffer; -import java.nio.ByteOrder; import java.nio.channels.FileChannel; import java.nio.charset.StandardCharsets; import java.util.ArrayList; +import java.util.BitSet; import java.util.List; /** - * Copies the maps folder out of an Xbox disc image (an "xiso", or a whole - * disc's ".iso"), as the desktop games do (port/linux/src/xiso.c, which this - * follows). + * Copies the maps folder out of the player's Xbox disc image of Halo, for + * LauncherActivity, as the desktop games do (port/linux/src/xiso.c; the web + * page's is port/web/site/xiso-worker.js). The image is a document the + * player picked, read through its FileChannel at absolute positions. * - * The image's file system is XDVDFS, read as extract-xiso does - * (https://github.com/XboxDev/extract-xiso, extract-xiso.c, whose format - * handling this follows; its license is below and in - * port/third_party/extract-xiso/LICENSE.TXT): 2048-byte sectors; a volume - * descriptor at 0x10000 that starts and ends with "MICROSOFT*XBOX*MEDIA" and - * gives the root directory's sector and size; and directories whose entries - * form a binary tree (each entry: left and right subtree offsets in 4-byte - * units, the start sector, the size, attributes, the name's length and the - * name, 4-byte aligned). Images made from a whole disc put the game partition - * further in, at one of the offsets below. + * Written from a description of the Xbox's disc file system, XDVDFS + * (xboxdevwiki's XDVDFS page, https://xboxdevwiki.net/XDVDFS, and the format + * facts OpenCE's readers rely on). What this class relies on: * - * The files are written to maps.partial first and moved into maps once they - * all are, so an interrupted extraction never leaves maps/ui.map behind. + * - Sectors are 2048 bytes and integers little-endian. Sector numbers count + * from the start of the game partition: an image of the partition alone + * has it at 0, an image of a whole disc further in (0x0FD90000, 0x02080000 + * or 0x18300000 for XGD2, XGD3 and XGD1 discs). They are tried in that + * order. + * - The volume descriptor is the partition's sector 32 (at 0x10000). It + * holds "MICROSOFT*XBOX*MEDIA" at its start and again at 0x7EC, and the + * root directory table's start sector (u32 at 0x14) and size in bytes (u32 + * at 0x18). + * - A directory table is a binary tree of entries whose root is the entry at + * offset 0. An entry is a left and a right subtree link (u16 each, in + * 4-byte units from the table's start, 0 for none), the start sector and + * the size of the file or of the subdirectory's table (u32 each), the + * attributes (u8, 0x10 for a directory), the name's length (u8) and the + * name. Unused bytes of a table are 0xFF, so a left link of 0xFFFF is + * padding, not an entry. + * - A file is one contiguous run of bytes from the start of its sector. * - * Some parts of this code are copyright in@fishtank.com. This product - * includes software developed by in <in@fishtank.com>. + * The image is the player's file and nothing read from it is trusted. Only + * the root's table and the maps folder's are read, each whole and at most + * 4 MB; the walk of a table visits each offset once, goes at most 64 links + * deep and looks at most 4096 entries; and a file is written only under a + * plain name that no file before it had (ignoring case). * - * Copyright (c) 2003 in <in@fishtank.com> - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * 3. All advertising materials mentioning features or use of this software - * must display the following acknowledgement: - * - * This product includes software developed by in <in@fishtank.com>. - * - * 4. Neither the name of "in" nor the email address "in@fishtank.com" - * may be used to endorse or promote products derived from this software - * without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED `AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES - * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND - * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * AUTHOR OR ANY CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; - * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, - * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR - * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF - * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * The app starts the game once maps/ui.map exists, and makes the maps folder + * itself, so the copy is written to maps.partial (emptied first of what an + * earlier attempt left) and its files moved into maps once all are + * complete, ui.map last. */ final class XisoExtractor { private static final int SECTOR_SIZE = 2048; - private static final long VOLUME_DESCRIPTOR_OFFSET = 0x10000; - private static final int ENTRY_HEADER_SIZE = 14; + private static final int DESCRIPTOR_OFFSET = 0x10000; + private static final byte[] MAGIC = "MICROSOFT*XBOX*MEDIA".getBytes(StandardCharsets.ISO_8859_1); + private static final int SECOND_MAGIC_OFFSET = 0x7EC; + private static final int ENTRY_FIXED_SIZE = 14; private static final int ATTRIBUTE_DIRECTORY = 0x10; - /* directory tables are a few sectors; anything much larger is not one */ - private static final long MAXIMUM_DIRECTORY_SIZE = 4 << 20; - private static final int MAXIMUM_FILES = 256; - private static final int COPY_BUFFER_SIZE = 1 << 20; - private static final byte[] VOLUME_MAGIC = "MICROSOFT*XBOX*MEDIA".getBytes(StandardCharsets.US_ASCII); - - /* where the game partition starts: a plain image, and whole-disc images - (extract-xiso's GLOBAL, XGD3 and XGD1 offsets) */ - private static final long[] PARTITION_OFFSETS = { 0, 0x0FD90000L, 0x02080000L, 0x18300000L }; + private static final int PADDING_LINK = 0xFFFF; + /** where the game partition may start, in the order they are tried */ + private static final long[] PARTITION_STARTS = { + 0x00000000L, // an image of the game partition alone + 0x0FD90000L, // a whole XGD2 disc + 0x02080000L, // a whole XGD3 disc + 0x18300000L, // a whole XGD1 disc (the original Xbox's) + }; + /** directory tables are a few sectors; anything much larger is not one */ + private static final int MAXIMUM_TABLE_SIZE = 4 * 1024 * 1024; + private static final int MAXIMUM_DEPTH = 64; + private static final int MAXIMUM_VISITS = 4096; + private static final int MAXIMUM_ENTRIES = 256; + private static final int COPY_CHUNK = 1024 * 1024; + private static final String PARTIAL_DIRECTORY = "maps.partial"; + /** Told as the copy goes: the file being copied, and the bytes copied of all the files'. */ interface Progress { void report(String file, long done, long total); } - /** a reason the player can act on */ + /** A reason the player can act on: its message is shown as it is. */ static final class ExtractException extends IOException { ExtractException(String message) { super(message); } } + /** a file or folder found in a directory table */ private static final class Entry { final String name; final long sector; @@ -104,187 +97,282 @@ final class XisoExtractor { } } - private final FileChannel image; - private long partition; - - private XisoExtractor(FileChannel image) { - this.image = image; + private XisoExtractor() { } - /** copies the image's maps folder to destination/maps */ + /** Copies the image's maps folder to destination/maps. */ static void extractMaps(FileChannel image, File destination, Progress progress) throws IOException { - new XisoExtractor(image).extract(destination, progress); - } + long imageSize = image.size(); + long partition = -1; + long rootSector = 0, rootSize = 0; + ByteBuffer descriptor = ByteBuffer.allocate(SECTOR_SIZE); - private void readAt(long offset, ByteBuffer buffer) throws IOException { - while (buffer.hasRemaining()) { - int count = image.read(buffer, offset); - - if (count <= 0) - throw new ExtractException("Could not read the disc image (is it complete?)."); - offset += count; - } - buffer.flip(); - } - - private static boolean magicAt(ByteBuffer buffer, int offset) { - for (int index = 0; index < VOLUME_MAGIC.length; index++) { - if (buffer.get(offset + index) != VOLUME_MAGIC[index]) - return false; - } - return true; - } - - /** whether a name is one to write in the maps folder: no path, no "." or - * "..", and only printable ASCII that every file system takes as a plain - * name (a disc's maps folder holds .map files and loading.tga) */ - static boolean isPlainName(String name) { - if (name.isEmpty() || name.equals(".") || name.equals("..")) - return false; - for (int index = 0; index < name.length(); index++) { - char c = name.charAt(index); - if (c < 0x20 || c > 0x7E || "/\\:*?\"<>|".indexOf(c) >= 0) - return false; - } - return true; - } - - /** the partition's volume descriptor: the root directory's sector and size */ - private long[] findVolume() throws IOException { - for (long offset : PARTITION_OFFSETS) { - ByteBuffer descriptor = ByteBuffer.allocate(SECTOR_SIZE).order(ByteOrder.LITTLE_ENDIAN); - - try { - readAt(offset + VOLUME_DESCRIPTOR_OFFSET, descriptor); - } catch (IOException e) { + for (long start : PARTITION_STARTS) { + if (start + DESCRIPTOR_OFFSET + SECTOR_SIZE > imageSize) continue; - } - if (!magicAt(descriptor, 0) || !magicAt(descriptor, 0x7EC)) + descriptor.clear(); + if (!readFully(image, descriptor, start + DESCRIPTOR_OFFSET)) continue; - partition = offset; - return new long[] { - descriptor.getInt(20) & 0xFFFFFFFFL, descriptor.getInt(24) & 0xFFFFFFFFL, - }; + byte[] bytes = descriptor.array(); + if (!hasMagic(bytes, 0) || !hasMagic(bytes, SECOND_MAGIC_OFFSET)) + continue; + partition = start; + rootSector = u32(bytes, 0x14); + rootSize = u32(bytes, 0x18); + break; } - throw new ExtractException("This is not an Xbox disc image."); - } + if (partition < 0) + throw new ExtractException("This is not an Xbox disc image."); - /** a directory's table, read whole */ - private ByteBuffer readDirectory(long sector, long size, String what) throws IOException { - if (size <= 0 || size > MAXIMUM_DIRECTORY_SIZE) - throw new ExtractException(what); - ByteBuffer table = ByteBuffer.allocate((int) size).order(ByteOrder.LITTLE_ENDIAN); - readAt(partition + sector * SECTOR_SIZE, table); - return table; - } - - /** - * collects the entries of the subtree at offset (4-byte units), files or - * directories as asked - */ - private static void walk(ByteBuffer table, long offset, int depth, boolean directories, List out, - int[] visited) { - offset *= 4; - /* a malformed table must not loop or run off its end */ - if (depth > 64 || ++visited[0] > 4096 || offset + ENTRY_HEADER_SIZE > table.limit()) - return; - int at = (int) offset; - int left = table.getShort(at) & 0xFFFF; - int right = table.getShort(at + 2) & 0xFFFF; - /* 0xFFFF: padding, an empty directory */ - if (left == 0xFFFF) - return; - int nameLength = table.get(at + 13) & 0xFF; - if (left != 0) - walk(table, left, depth + 1, directories, out, visited); - boolean isDirectory = (table.get(at + 12) & ATTRIBUTE_DIRECTORY) != 0; - if (at + ENTRY_HEADER_SIZE + nameLength <= table.limit() && nameLength > 0 && isDirectory == directories - && out.size() < MAXIMUM_FILES) { - byte[] bytes = new byte[nameLength]; - for (int index = 0; index < nameLength; index++) - bytes[index] = table.get(at + ENTRY_HEADER_SIZE + index); - String name = new String(bytes, StandardCharsets.ISO_8859_1); - /* (as extract-xiso refuses them: no name may leave the folder, and - none but a plain file name is written) */ - if (isPlainName(name)) - out.add(new Entry(name, table.getInt(at + 4) & 0xFFFFFFFFL, table.getInt(at + 8) & 0xFFFFFFFFL)); - } - if (right != 0) - walk(table, right, depth + 1, directories, out, visited); - } - - private void extract(File destination, Progress progress) throws IOException { - long[] root = findVolume(); - - /* the root's maps folder */ - ByteBuffer table = readDirectory(root[0], root[1], "The disc image's file system is damaged."); - List directories = new ArrayList<>(); - walk(table, 0, 0, true, directories, new int[1]); + // the root's maps folder + byte[] root = readTable(image, partition, rootSector, rootSize, "The disc image's file system is damaged."); Entry maps = null; - for (Entry entry : directories) { - if (entry.name.equalsIgnoreCase("maps")) + for (Entry entry : walkTable(root, true)) { + if (entry.name.equalsIgnoreCase("maps")) { maps = entry; + break; + } } if (maps == null) throw new ExtractException("The disc image has no maps folder: it is not a Halo disc."); - /* its files */ - table = readDirectory(maps.sector, maps.size, "The disc image's maps folder is damaged."); - List files = new ArrayList<>(); - walk(table, 0, 0, false, files, new int[1]); - long total = 0; - boolean hasUi = false; + // its files + byte[] table = readTable(image, partition, maps.sector, maps.size, "The disc image's maps folder is damaged."); + List files = walkTable(table, false); + Entry userInterface = null; for (Entry file : files) { - total += file.size; - hasUi |= file.name.equalsIgnoreCase("ui.map"); + if (file.name.equalsIgnoreCase("ui.map")) + userInterface = file; } - if (!hasUi) + if (userInterface == null) throw new ExtractException("The disc image's maps folder has no ui.map: it is not a Halo disc."); - - File partial = new File(destination, "maps.partial"); - File finished = new File(destination, "maps"); - if (!partial.isDirectory() && !partial.mkdirs()) - throw new ExtractException("Could not create " + partial + "."); - ByteBuffer buffer = ByteBuffer.allocateDirect(COPY_BUFFER_SIZE); - long done = 0; + // (an image that ends before a file does is refused before anything + // is written; an empty file's sector need not be anywhere) + long total = 0; for (Entry file : files) { - File path = new File(partial, file.name); - long offset = partition + file.sector * SECTOR_SIZE; - long remaining = file.size; - - try (FileOutputStream out = new FileOutputStream(path)) { - FileChannel output = out.getChannel(); - - while (remaining > 0) { - buffer.clear(); - buffer.limit((int) Math.min(remaining, COPY_BUFFER_SIZE)); - readAt(offset, buffer); - int count = buffer.remaining(); - while (buffer.hasRemaining()) - output.write(buffer); - offset += count; - remaining -= count; - done += count; - progress.report(file.name, done, total); - } - } catch (ExtractException e) { - throw new ExtractException("Could not read " + file.name + " from the disc image (is it complete?)."); - } catch (IOException e) { - throw new ExtractException("Could not write " + path + " (is the storage full?)."); - } + if (file.size > 0 && partition + file.sector * SECTOR_SIZE + file.size > imageSize) + throw new ExtractException(readFailure(file)); + total += file.size; } - /* (the maps folder may be there, empty: the app makes it for adb) */ - if (!finished.isDirectory() && !finished.mkdirs()) - throw new ExtractException("Could not create " + finished + "."); - for (Entry file : files) { - File from = new File(partial, file.name); - File to = new File(finished, file.name); + // the copy, to maps.partial + File partial = new File(destination, PARTIAL_DIRECTORY); + if (partial.isDirectory()) { + File[] leftovers = partial.listFiles(); + if (leftovers != null) { + for (File leftover : leftovers) + leftover.delete(); + } + } else { + partial.delete(); + partial.mkdirs(); + } + if (!partial.isDirectory()) + throw new ExtractException("Could not create " + partial.getPath() + "."); + ByteBuffer buffer = ByteBuffer.allocate(COPY_CHUNK); + long done = 0; + for (Entry file : files) + done = copyFile(image, partition, file, new File(partial, file.name), buffer, done, total, progress); - to.delete(); - if (!from.renameTo(to)) - throw new ExtractException("Could not move " + file.name + " into " + finished + "."); + // then into maps, ui.map (the app's sign that the data is there) last + File mapsFolder = new File(destination, "maps"); + if (!mapsFolder.isDirectory() && !mapsFolder.mkdirs() && !mapsFolder.isDirectory()) + throw new ExtractException("Could not create " + mapsFolder.getPath() + "."); + List order = new ArrayList<>(files); + order.remove(userInterface); + order.add(userInterface); + for (Entry file : order) { + File target = new File(mapsFolder, file.name); + target.delete(); + if (!new File(partial, file.name).renameTo(target)) + throw new ExtractException("Could not move " + file.name + " into " + mapsFolder.getPath() + "."); } partial.delete(); } + + /** No path, no . or .., and only printable ASCII that every file system takes as a plain name. */ + static boolean isPlainName(String name) { + if (name == null || name.isEmpty() || name.equals(".") || name.equals("..")) + return false; + for (int index = 0; index < name.length(); index++) { + char character = name.charAt(index); + + if (character < 0x20 || character > 0x7E || "/\\:*?\"<>|".indexOf(character) >= 0) + return false; + } + return true; + } + + /* ---------- the image */ + + private static long u16(byte[] bytes, int offset) { + return (bytes[offset] & 0xFF) | (bytes[offset + 1] & 0xFF) << 8; + } + + private static long u32(byte[] bytes, int offset) { + return u16(bytes, offset) | u16(bytes, offset + 2) << 16; + } + + private static boolean hasMagic(byte[] bytes, int offset) { + for (int index = 0; index < MAGIC.length; index++) { + if (bytes[offset + index] != MAGIC[index]) + return false; + } + return true; + } + + /** Fills buffer from the image at position; false if the image ends first. */ + private static boolean readFully(FileChannel image, ByteBuffer buffer, long position) throws IOException { + while (buffer.hasRemaining()) { + int count = image.read(buffer, position); + + if (count <= 0) + return false; + position += count; + } + return true; + } + + private static byte[] readTable(FileChannel image, long partition, long sector, long size, String damaged) + throws IOException { + if (size == 0 || size > MAXIMUM_TABLE_SIZE) + throw new ExtractException(damaged); + ByteBuffer table = ByteBuffer.allocate((int) size); + if (!readFully(image, table, partition + sector * SECTOR_SIZE)) + throw new ExtractException("Could not read the disc image (is it complete?)."); + return table.array(); + } + + /* ---------- directory tables */ + + private static final class Walk { + final byte[] table; + final boolean directories; + final BitSet visited = new BitSet(); + final List entries = new ArrayList<>(); + int visits; + boolean stopped; + + Walk(byte[] table, boolean directories) { + this.table = table; + this.directories = directories; + } + } + + /** The folders (or the files) of a table with plain names, in the tree's order. */ + private static List walkTable(byte[] table, boolean directories) { + Walk walk = new Walk(table, directories); + + walkEntry(walk, 0, 0); + return walk.entries; + } + + /** + * The subtree whose root entry is at offset: its left subtree, the + * entry, its right subtree. A link to padding, past the table or back to + * an entry already visited ends that branch. + */ + private static void walkEntry(Walk walk, int offset, int depth) { + byte[] table = walk.table; + + if (walk.stopped || depth > MAXIMUM_DEPTH) + return; + if (offset >= table.length || table.length - offset < ENTRY_FIXED_SIZE) + return; + int left = (int) u16(table, offset); + int right = (int) u16(table, offset + 2); + if (left == PADDING_LINK || walk.visited.get(offset / 4)) + return; + walk.visited.set(offset / 4); + if (walk.visits >= MAXIMUM_VISITS) { + walk.stopped = true; + return; + } + walk.visits++; + + if (left != 0) + walkEntry(walk, left * 4, depth + 1); + int nameLength = table[offset + 13] & 0xFF; + if (nameLength != 0 && nameLength <= table.length - offset - ENTRY_FIXED_SIZE) + collect(walk, offset, nameLength); + if (right != 0) + walkEntry(walk, right * 4, depth + 1); + } + + private static void collect(Walk walk, int offset, int nameLength) { + byte[] table = walk.table; + boolean directory = (table[offset + 12] & ATTRIBUTE_DIRECTORY) != 0; + + if (directory != walk.directories || walk.entries.size() >= MAXIMUM_ENTRIES) + return; + String name = new String(table, offset + ENTRY_FIXED_SIZE, nameLength, StandardCharsets.ISO_8859_1); + if (!isPlainName(name)) + return; + // (the destination's file system may ignore case) + for (Entry entry : walk.entries) { + if (entry.name.equalsIgnoreCase(name)) + return; + } + walk.entries.add(new Entry(name, u32(table, offset + 4), u32(table, offset + 8))); + } + + /* ---------- the copy */ + + private static String readFailure(Entry file) { + return "Could not read " + file.name + " from the disc image (is it complete?)."; + } + + /** Copies one file of the image to output; returns the bytes copied of all the files so far. */ + private static long copyFile(FileChannel image, long partition, Entry file, File output, ByteBuffer buffer, + long done, long total, Progress progress) throws IOException { + String writeFailure = "Could not write " + output.getPath() + " (is the storage full?)."; + long position = partition + file.sector * SECTOR_SIZE; + long remaining = file.size; + FileOutputStream stream = null; + + try { + stream = new FileOutputStream(output); + FileChannel channel = stream.getChannel(); + while (remaining > 0) { + int count = (int) Math.min(remaining, COPY_CHUNK); + boolean read; + + buffer.clear(); + buffer.limit(count); + try { + read = readFully(image, buffer, position); + } catch (IOException exception) { + read = false; + } + if (!read) + throw new ExtractException(readFailure(file)); + buffer.flip(); + while (buffer.hasRemaining()) { + if (channel.write(buffer) <= 0) + throw new ExtractException(writeFailure); + } + position += count; + remaining -= count; + done += count; + if (progress != null) + progress.report(file.name, done, total); + } + FileOutputStream closing = stream; + stream = null; + closing.close(); + } catch (ExtractException exception) { + throw exception; + } catch (IOException exception) { + throw new ExtractException(writeFailure); + } finally { + if (stream != null) { + try { + stream.close(); + } catch (IOException ignored) { + // (the copy failed already) + } + } + } + return done; + } } diff --git a/port/linux/game/cache_file_formats.c b/port/linux/game/cache_file_formats.c index 9f856c6e..8a691c7c 100644 --- a/port/linux/game/cache_file_formats.c +++ b/port/linux/game/cache_file_formats.c @@ -14,11 +14,11 @@ Sources for the layouts, cited per structure below: - BlamLib, in the same archive: the font and unicode string list layouts. - Reclaimer (GPL-3.0, Gravemind2401; the Reclaimer-master archive): how bitmap tags and pixel data are found in bitmaps.map. -No code from either project is used: they are documentation here, and this -repository is CC0. Facts that neither source states were established by -reading the sample Custom Edition maps; the comments below say so where they -are used ("observed", "every map examined"), and -docs/custom_edition_caches.md lists each with its evidence. +No code from either project is used: they are documentation here. Facts +that neither source states were established by reading the sample Custom +Edition maps; the comments below say so where they are used ("observed", +"every map examined"), and docs/custom_edition_caches.md lists each with its +evidence. */ /* ---------- headers */ diff --git a/port/linux/src/updater.c b/port/linux/src/updater.c index c8357594..7a7b4f3d 100644 --- a/port/linux/src/updater.c +++ b/port/linux/src/updater.c @@ -521,13 +521,15 @@ static void updater_update(void) } } -/* the files a previous update left behind */ +/* the files a previous update left behind, and the notice of extract-xiso, +which releases carried until the disc image readers were written again */ static void updater_clean_up(void) { static const char *const names[] = { "halo.old", "halo.exe.old", "SDL3.dll.old", "libSDL3.so.0.old", "SDL3-LICENSE.txt.old", - "extract-xiso-LICENSE.txt.old", "mbedtls-LICENSE.txt.old", + "extract-xiso-LICENSE.txt.old", "mbedtls-LICENSE.txt.old", "LICENSE.txt.old", + "extract-xiso-LICENSE.txt", }; char path[1200]; size_t index; diff --git a/port/linux/src/xiso.c b/port/linux/src/xiso.c index 84fae910..e2097950 100644 --- a/port/linux/src/xiso.c +++ b/port/linux/src/xiso.c @@ -1,307 +1,408 @@ /* XISO.C -Copies the maps folder out of an Xbox disc image (an "xiso"), for the -desktop ports' first start without game data (sdl_platform.c). +The maps folder out of an Xbox disc image, for the desktop games (Linux and +Windows). When no data root has a maps folder, sdl_platform.c asks the player +for their disc image of Halo and this copies the image's maps folder next to +the executable. (The Android app's copy of this is XisoExtractor.java, the +web page's xiso-worker.js.) -The image's file system is XDVDFS, read as extract-xiso does -(https://github.com/XboxDev/extract-xiso, extract-xiso.c, whose format -handling this follows; its license is below and in -port/third_party/extract-xiso/LICENSE.TXT): 2048-byte sectors; a -volume descriptor at 0x10000 that starts and ends with -"MICROSOFT*XBOX*MEDIA" and gives the root directory's sector and size; and -directories whose entries form a binary tree (each entry: left and right -subtree offsets in 4-byte units, the start sector, the size, attributes, the -name's length and the name, 4-byte aligned). Images made from a whole disc -put the game partition further in, at one of the offsets below. +Written from a description of the Xbox's disc file system, XDVDFS +(xboxdevwiki's XDVDFS page, https://xboxdevwiki.net/XDVDFS, and the format +facts OpenCE's readers rely on). What this file relies on: -The files are written to /maps.partial first, which becomes -/maps once they all are, so an interrupted extraction never -leaves a maps folder that looks complete. +- Sectors are 2048 bytes and integers little-endian. Sector numbers count + from the start of the game partition: an image of the partition alone has + it at 0, an image of a whole disc further in (0x0FD90000, 0x02080000 or + 0x18300000 for XGD2, XGD3 and XGD1 discs). They are tried in that order. +- The volume descriptor is the partition's sector 32 (at 0x10000). It holds + "MICROSOFT*XBOX*MEDIA" at its start and again at 0x7EC, and the root + directory table's start sector (u32 at 0x14) and size in bytes (u32 at + 0x18). +- A directory table is a binary tree of entries whose root is the entry at + offset 0. An entry is a left and a right subtree link (u16 each, in 4-byte + units from the table's start, 0 for none), the start sector and the size + of the file or of the subdirectory's table (u32 each), the attributes (u8, + 0x10 for a directory), the name's length (u8) and the name. Unused bytes of + a table are 0xFF, so a left link of 0xFFFF is padding, not an entry. +- A file is one contiguous run of bytes from the start of its sector. -Some parts of this code are copyright in@fishtank.com. This product -includes software developed by in . +The image is the player's file and nothing read from it is trusted. Only the +root's table and the maps folder's are read, each whole and at most 4 MB; +the walk of a table visits each offset once, goes at most 64 links deep and +looks at most 4096 entries; and a file is written only under a plain name +that no file before it had (ignoring case). - * Copyright (c) 2003 in - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * 3. All advertising materials mentioning features or use of this software - * must display the following acknowledgement: - * - * This product includes software developed by in . - * - * 4. Neither the name of "in" nor the email address "in@fishtank.com" - * may be used to endorse or promote products derived from this software - * without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED `AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES - * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND - * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * AUTHOR OR ANY CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; - * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, - * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR - * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF - * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +The game takes a data root as having data once it has a maps folder, so the +copy is written to maps.partial (emptied first of what an earlier attempt +left) and given the name maps only when every file is complete. */ -/* (the desktop ports only: the Android app and the web page import the game -data themselves) */ -#if !defined(HALO_ANDROID) && !defined(HALO_WEB) - #include "platform.h" -#include "posix.h" #include "xiso.h" +#if !defined(HALO_ANDROID) && !defined(HALO_WEB) + +#include "posix.h" + +#include #include +#include #include #include #include #include +/* (images are larger than 4 GB; Windows has neither flag) */ #ifndef O_LARGEFILE #define O_LARGEFILE 0 #endif -#ifndef O_CLOEXEC -#define O_CLOEXEC 0 -#endif #ifndef O_NOFOLLOW #define O_NOFOLLOW 0 #endif -enum +#define XISO_SECTOR_SIZE 2048 +#define XISO_DESCRIPTOR_OFFSET 0x10000 +#define XISO_MAGIC "MICROSOFT*XBOX*MEDIA" +#define XISO_MAGIC_LENGTH 20 +#define XISO_SECOND_MAGIC_OFFSET 0x7EC +#define XISO_ENTRY_FIXED_SIZE 14 +#define XISO_ATTRIBUTE_DIRECTORY 0x10 +#define XISO_PADDING_LINK 0xFFFF + +/* directory tables are a few sectors; anything much larger is not one */ +#define XISO_MAXIMUM_TABLE_SIZE (4 * 1024 * 1024) +#define XISO_MAXIMUM_DEPTH 64 +#define XISO_MAXIMUM_VISITS 4096 +#define XISO_MAXIMUM_ENTRIES 256 +#define XISO_MAXIMUM_PATH 4096 +#define XISO_COPY_CHUNK (1024 * 1024) + +#define XISO_PARTIAL_DIRECTORY "maps.partial" + +/* where the game partition may start, in the order they are tried */ +static const unsigned long xiso_partition_starts[] = { - SECTOR_SIZE = 2048, - VOLUME_DESCRIPTOR_OFFSET = 0x10000, - ENTRY_HEADER_SIZE = 14, - ATTRIBUTE_DIRECTORY = 0x10, - /* directory tables are a few sectors; anything much larger is not one */ - MAXIMUM_DIRECTORY_SIZE = 4 << 20, - MAXIMUM_FILES = 256, - COPY_BUFFER_SIZE = 1 << 20, -}; - -static const char volume_magic[] = "MICROSOFT*XBOX*MEDIA"; - -/* where the game partition starts: a plain image, and whole-disc images -(extract-xiso's GLOBAL, XGD3 and XGD1 offsets) */ -static const unsigned long long partition_offsets[] = { 0, 0x0FD90000ull, 0x02080000ull, 0x18300000ull }; - -struct xiso_file -{ - char name[256]; - unsigned long sector; - unsigned long size; + 0x00000000, /* an image of the game partition alone */ + 0x0FD90000, /* a whole XGD2 disc */ + 0x02080000, /* a whole XGD3 disc */ + 0x18300000, /* a whole XGD1 disc (the original Xbox's) */ }; struct xiso_image { int descriptor; unsigned long long partition; - char *error; - int error_size; + /* the image's size in bytes, or ~0 when it could not be found */ + unsigned long long size; }; -static int name_is_plain(const char *name); - -static unsigned long read_u32(const unsigned char *bytes) +/* a file or folder found in a directory table */ +struct xiso_entry { - return (unsigned long)bytes[0] | (unsigned long)bytes[1] << 8 | (unsigned long)bytes[2] << 16 | - (unsigned long)bytes[3] << 24; + char name[256]; + unsigned long sector; + unsigned long size; +}; + +struct xiso_walk +{ + const unsigned char *table; + unsigned long size; + /* a bit for each 4-byte offset of the table, set once it is visited */ + unsigned char *visited; + int visits; + int stopped; + /* whether folders are collected (or else files) */ + int directories; + struct xiso_entry *entries; + int entry_count; +}; + +/* ---------- errors */ + +static void xiso_error(char *error, int error_size, const char *format, ...) +{ + va_list arguments; + + if (!error || error_size <= 0) + return; + va_start(arguments, format); + vsnprintf(error, (size_t)error_size, format, arguments); + va_end(arguments); + error[error_size - 1] = 0; } -static int fail(struct xiso_image *image, const char *format, const char *detail) +/* ---------- names */ + +static int xiso_lower(int character) { - snprintf(image->error, (size_t)image->error_size, format, detail ? detail : ""); - return 0; + return character >= 'A' && character <= 'Z' ? character - 'A' + 'a' : character; } -/* reads size bytes at offset (from the start of the file) */ -static int read_at(struct xiso_image *image, unsigned long long offset, void *buffer, unsigned long size) +/* whether a name of length bytes is name (a C string), ignoring ASCII case */ +static int xiso_name_is(const char *bytes, int length, const char *name) { - posix_ulong low, high; - unsigned char *cursor = buffer; + int index; - if (posix_seek(image->descriptor, (posix_long)(offset & 0xFFFFFFFFu), (posix_long)(offset >> 32), SEEK_SET, - &low, &high) != 0) - { + if ((int)strlen(name) != length) return 0; - } - while (size > 0) + for (index = 0; index < length; index++) { - long count = (long)read(image->descriptor, cursor, size > COPY_BUFFER_SIZE ? COPY_BUFFER_SIZE : size); - - if (count <= 0) + if (xiso_lower((unsigned char)bytes[index]) != xiso_lower((unsigned char)name[index])) return 0; - cursor += count; - size -= (unsigned long)count; } return 1; } -/* the partition's volume descriptor: the root directory's sector and size */ -static int find_volume(struct xiso_image *image, unsigned long *root_sector, unsigned long *root_size) +/* no path, no . or .., and only printable ASCII that every desktop file +system takes as a plain name */ +static int xiso_plain_name(const unsigned char *name, int length) { int index; - for (index = 0; index < (int)(sizeof(partition_offsets) / sizeof(*partition_offsets)); index++) + if (length <= 0 || (length == 1 && name[0] == '.') || (length == 2 && name[0] == '.' && name[1] == '.')) + return 0; + for (index = 0; index < length; index++) { - unsigned char descriptor[SECTOR_SIZE]; - unsigned long long offset = partition_offsets[index] + VOLUME_DESCRIPTOR_OFFSET; - - if (!read_at(image, offset, descriptor, sizeof(descriptor))) - continue; - if (memcmp(descriptor, volume_magic, 20) || memcmp(descriptor + 0x7EC, volume_magic, 20)) - continue; - image->partition = partition_offsets[index]; - *root_sector = read_u32(descriptor + 20); - *root_size = read_u32(descriptor + 24); - return 1; + if (name[index] < 0x20 || name[index] > 0x7E || strchr("/\\:*?\"<>|", name[index])) + return 0; } - return fail(image, "This is not an Xbox disc image.%s", NULL); + return 1; } -/* a directory's table, read whole */ -static unsigned char *read_directory(struct xiso_image *image, unsigned long sector, unsigned long size) +/* ---------- the image */ + +static unsigned long xiso_u16(const unsigned char *bytes) +{ + return (unsigned long)bytes[0] | (unsigned long)bytes[1] << 8; +} + +static unsigned long xiso_u32(const unsigned char *bytes) +{ + return xiso_u16(bytes) | xiso_u16(bytes + 2) << 16; +} + +/* reads size bytes at offset (from the start of the image file); returns +nonzero if all of them were read */ +static int xiso_read(struct xiso_image *image, unsigned long long offset, void *buffer, unsigned long size) +{ + posix_ulong position_low, position_high; + unsigned long count = 0; + + if (posix_seek(image->descriptor, (posix_long)(offset & 0xFFFFFFFFUL), (posix_long)(offset >> 32), SEEK_SET, + &position_low, &position_high) != 0) + { + return 0; + } + while (count < size) + { + int result = (int)read(image->descriptor, (char *)buffer + count, (unsigned int)(size - count)); + + if (result < 0 && errno == EINTR) + continue; + if (result <= 0) + return 0; + count += (unsigned long)result; + } + return 1; +} + +/* the byte offset of a sector of the game partition */ +static unsigned long long xiso_sector_offset(struct xiso_image *image, unsigned long sector) +{ + return image->partition + (unsigned long long)sector * XISO_SECTOR_SIZE; +} + +/* finds the game partition: the first start whose volume descriptor has +both magics. Returns the root table's sector and size */ +static int xiso_find_volume(struct xiso_image *image, unsigned long *root_sector, unsigned long *root_size) +{ + unsigned char descriptor[XISO_SECTOR_SIZE]; + int index; + + for (index = 0; index < (int)(sizeof(xiso_partition_starts) / sizeof(xiso_partition_starts[0])); index++) + { + unsigned long long start = xiso_partition_starts[index]; + + if (!xiso_read(image, start + XISO_DESCRIPTOR_OFFSET, descriptor, sizeof(descriptor)) || + memcmp(descriptor, XISO_MAGIC, XISO_MAGIC_LENGTH) != 0 || + memcmp(descriptor + XISO_SECOND_MAGIC_OFFSET, XISO_MAGIC, XISO_MAGIC_LENGTH) != 0) + { + continue; + } + image->partition = start; + *root_sector = xiso_u32(descriptor + 0x14); + *root_size = xiso_u32(descriptor + 0x18); + return 1; + } + return 0; +} + +/* reads a directory table whole into a new allocation; returns NULL if it +is empty, too large or cannot be read, setting *out_of_memory when that is +the reason */ +static unsigned char *xiso_read_table(struct xiso_image *image, unsigned long sector, unsigned long size, + int *out_of_memory) { unsigned char *table; - if (!size || size > MAXIMUM_DIRECTORY_SIZE) + *out_of_memory = 0; + if (size == 0 || size > XISO_MAXIMUM_TABLE_SIZE) return NULL; table = malloc(size); - if (table && !read_at(image, image->partition + (unsigned long long)sector * SECTOR_SIZE, table, size)) + if (!table) + { + *out_of_memory = 1; + return NULL; + } + if (!xiso_read(image, xiso_sector_offset(image, sector), table, size)) { free(table); - table = NULL; + return NULL; } return table; } -struct directory_walk -{ - const unsigned char *table; - unsigned long size; - struct xiso_file *entries; - int entry_count; - int maximum_count; - int visited; - int directories; -}; +/* ---------- directory tables */ -/* collects the entries of the subtree at offset (4-byte units), files or -directories as asked */ -static void walk_directory(struct directory_walk *walk, unsigned long offset, int depth) +static void xiso_walk_collect(struct xiso_walk *walk, const unsigned char *entry) +{ + struct xiso_entry *collected; + int name_length = entry[13]; + int index; + + if (((entry[12] & XISO_ATTRIBUTE_DIRECTORY) != 0) != (walk->directories != 0)) + return; + if (walk->entry_count >= XISO_MAXIMUM_ENTRIES || !xiso_plain_name(entry + XISO_ENTRY_FIXED_SIZE, name_length)) + return; + /* (the destination's file system may ignore case) */ + for (index = 0; index < walk->entry_count; index++) + { + if (xiso_name_is((const char *)entry + XISO_ENTRY_FIXED_SIZE, name_length, walk->entries[index].name)) + return; + } + collected = &walk->entries[walk->entry_count++]; + memcpy(collected->name, entry + XISO_ENTRY_FIXED_SIZE, (size_t)name_length); + collected->name[name_length] = 0; + collected->sector = xiso_u32(entry + 4); + collected->size = xiso_u32(entry + 8); +} + +/* the subtree whose root entry is at offset: its left subtree, the entry, +its right subtree. A link to padding, past the table or back to an entry +already visited ends that branch */ +static void xiso_walk_entry(struct xiso_walk *walk, unsigned long offset, int depth) { const unsigned char *entry; - unsigned long left, right; - int name_length; + unsigned long left, right, name_length; - offset *= 4; - /* a malformed table must not loop or run off its end */ - if (depth > 64 || ++walk->visited > 4096 || offset + ENTRY_HEADER_SIZE > walk->size) + if (walk->stopped || depth > XISO_MAXIMUM_DEPTH) + return; + if (offset >= walk->size || walk->size - offset < XISO_ENTRY_FIXED_SIZE) return; entry = walk->table + offset; - left = (unsigned long)entry[0] | (unsigned long)entry[1] << 8; - right = (unsigned long)entry[2] | (unsigned long)entry[3] << 8; - /* 0xFFFF: padding, an empty directory */ - if (left == 0xFFFF) + left = xiso_u16(entry); + right = xiso_u16(entry + 2); + if (left == XISO_PADDING_LINK) return; - name_length = entry[13]; + /* (links are in 4-byte units, so every offset is a multiple of 4) */ + if (walk->visited[offset >> 5] & (1 << ((offset >> 2) & 7))) + return; + walk->visited[offset >> 5] |= (unsigned char)(1 << ((offset >> 2) & 7)); + if (walk->visits >= XISO_MAXIMUM_VISITS) + { + walk->stopped = 1; + return; + } + walk->visits++; + if (left) - walk_directory(walk, left, depth + 1); - if (offset + ENTRY_HEADER_SIZE + (unsigned long)name_length <= walk->size && name_length > 0 && - !(entry[12] & ATTRIBUTE_DIRECTORY) == !walk->directories && walk->entry_count < walk->maximum_count) - { - struct xiso_file *file = &walk->entries[walk->entry_count]; - - memcpy(file->name, entry + ENTRY_HEADER_SIZE, (size_t)name_length); - file->name[name_length] = 0; - file->sector = read_u32(entry + 4); - file->size = read_u32(entry + 8); - /* (as extract-xiso refuses them: no name may leave the folder, and - none but a plain file name is written) */ - if (name_is_plain(file->name)) - { - walk->entry_count++; - } - } + xiso_walk_entry(walk, left * 4, depth + 1); + name_length = entry[13]; + if (name_length && name_length <= walk->size - offset - XISO_ENTRY_FIXED_SIZE) + xiso_walk_collect(walk, entry); if (right) - walk_directory(walk, right, depth + 1); + xiso_walk_entry(walk, right * 4, depth + 1); } -/* whether a name is one to write in the maps folder: no path, no "." or -"..", and only printable ASCII that every desktop file system takes as a -plain name (a disc's maps folder holds .map files and loading.tga) */ -static int name_is_plain(const char *name) +/* collects the folders (or the files) of a table, with plain names, into +entries (XISO_MAXIMUM_ENTRIES of them); returns how many, or -1 when out of +memory */ +static int xiso_walk_table(const unsigned char *table, unsigned long size, int directories, + struct xiso_entry *entries) { - const char *at; + struct xiso_walk walk; - if (!*name || !strcmp(name, ".") || !strcmp(name, "..")) - return 0; - for (at = name; *at; at++) + memset(&walk, 0, sizeof(walk)); + walk.table = table; + walk.size = size; + walk.directories = directories; + walk.entries = entries; + walk.visited = calloc(size / 32 + 1, 1); + if (!walk.visited) + return -1; + xiso_walk_entry(&walk, 0, 0); + free(walk.visited); + return walk.entry_count; +} + +/* ---------- the copy */ + +/* removes what an earlier, interrupted copy left in the folder at path */ +static void xiso_empty_directory(const char *path, char *file_path, size_t file_path_size) +{ + void *directory = posix_directory_open(path); + char name[260]; + + if (!directory) + return; + while (posix_directory_next(directory, name, sizeof(name))) { - unsigned char c = (unsigned char)*at; + int length = snprintf(file_path, file_path_size, "%s/%s", path, name); - if (c < 0x20 || c > 0x7e || strchr("/\\:*?\"<>|", c)) - return 0; + if (length > 0 && (size_t)length < file_path_size) + remove(file_path); } - return 1; + posix_directory_close(directory); } -static int names_match(const char *a, const char *b) +/* copies one file of the image to path, adding to *done as it goes */ +static int xiso_copy_file(struct xiso_image *image, const struct xiso_entry *entry, const char *path, + unsigned char *buffer, unsigned long long *done, unsigned long long total, xiso_progress_proc progress, + void *context, char *error, int error_size) { - for (; *a && *b; a++, b++) - { - char x = *a >= 'A' && *a <= 'Z' ? (char)(*a - 'A' + 'a') : *a; - char y = *b >= 'A' && *b <= 'Z' ? (char)(*b - 'A' + 'a') : *b; - - if (x != y) - return 0; - } - return !*a && !*b; -} - -static int copy_file(struct xiso_image *image, const struct xiso_file *file, const char *path, - unsigned char *buffer, unsigned long long *done, unsigned long long total, - xiso_progress_proc progress, void *context) -{ - unsigned long long offset = image->partition + (unsigned long long)file->sector * SECTOR_SIZE; - unsigned long remaining = file->size; - /* (not through a link left in the folder) */ + unsigned long long offset = xiso_sector_offset(image, entry->sector); + unsigned long remaining = entry->size; int output = open(path, O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC | O_NOFOLLOW, 0644); if (output < 0) - return fail(image, "Could not write %s.", path); - while (remaining > 0) { - unsigned long count = remaining > COPY_BUFFER_SIZE ? COPY_BUFFER_SIZE : remaining; + xiso_error(error, error_size, "Could not write %s.", path); + return 0; + } + while (remaining) + { + unsigned long count = remaining < XISO_COPY_CHUNK ? remaining : XISO_COPY_CHUNK; unsigned long written = 0; - if (!read_at(image, offset, buffer, count)) + if (!xiso_read(image, offset, buffer, count)) { close(output); - return fail(image, "Could not read %s from the disc image (is it complete?).", file->name); + xiso_error(error, error_size, "Could not read %s from the disc image (is it complete?).", entry->name); + return 0; } while (written < count) { - long result = (long)write(output, buffer + written, count - written); + int result = (int)write(output, buffer + written, (unsigned int)(count - written)); + if (result < 0 && errno == EINTR) + continue; if (result <= 0) { close(output); - return fail(image, "Could not write %s (is the disk full?).", path); + xiso_error(error, error_size, "Could not write %s (is the disk full?).", path); + return 0; } written += (unsigned long)result; } @@ -309,109 +410,166 @@ static int copy_file(struct xiso_image *image, const struct xiso_file *file, con remaining -= count; *done += count; if (progress) - progress(context, file->name, *done, total); + progress(context, entry->name, *done, total); } if (close(output) != 0) - return fail(image, "Could not write %s (is the disk full?).", path); + { + xiso_error(error, error_size, "Could not write %s (is the disk full?).", path); + return 0; + } return 1; } +/* ---------- public code */ + int xiso_extract_maps(const char *image_path, const char *destination, xiso_progress_proc progress, void *context, char *error, int error_size) { struct xiso_image image; - struct xiso_file *entries = NULL; + struct xiso_entry *entries = NULL; unsigned char *table = NULL; unsigned char *buffer = NULL; - unsigned long root_sector, root_size; - unsigned long long total = 0, done = 0; - char partial[1024], final[1024], path[1300]; - int result = 0; - int index; + char *partial_path = NULL, *file_path = NULL, *maps_path = NULL; + unsigned long root_sector, root_size, maps_sector = 0, maps_size = 0; + unsigned long long done = 0, total = 0; + posix_ulong size_low, size_high; + int count, index, found, out_of_memory, succeeded = 0; - image.error = error; - image.error_size = error_size; + memset(&image, 0, sizeof(image)); image.descriptor = open(image_path, O_RDONLY | O_LARGEFILE | O_CLOEXEC); if (image.descriptor < 0) - return fail(&image, "Could not open %s.", image_path); - entries = calloc(MAXIMUM_FILES, sizeof(*entries)); - buffer = malloc(COPY_BUFFER_SIZE); - if (!entries || !buffer) { - fail(&image, "Out of memory.%s", NULL); + xiso_error(error, error_size, "Could not open %s.", image_path); + return 0; + } + image.size = ~0ULL; + if (posix_seek(image.descriptor, 0, 0, SEEK_END, &size_low, &size_high) == 0) + image.size = (unsigned long long)size_high << 32 | size_low; + + entries = malloc(sizeof(*entries) * XISO_MAXIMUM_ENTRIES); + buffer = malloc(XISO_COPY_CHUNK); + partial_path = malloc(XISO_MAXIMUM_PATH); + file_path = malloc(XISO_MAXIMUM_PATH); + maps_path = malloc(XISO_MAXIMUM_PATH); + if (!entries || !buffer || !partial_path || !file_path || !maps_path) + { + xiso_error(error, error_size, "Out of memory."); goto done; } - if (!find_volume(&image, &root_sector, &root_size)) + + if (!xiso_find_volume(&image, &root_sector, &root_size)) + { + xiso_error(error, error_size, "This is not an Xbox disc image."); goto done; + } /* the root's maps folder */ - table = read_directory(&image, root_sector, root_size); + table = xiso_read_table(&image, root_sector, root_size, &out_of_memory); if (!table) { - fail(&image, "The disc image's file system is damaged.%s", NULL); + xiso_error(error, error_size, out_of_memory ? "Out of memory." : "The disc image's file system is damaged."); goto done; } + count = xiso_walk_table(table, root_size, 1, entries); + free(table); + table = NULL; + if (count < 0) { - struct directory_walk walk = { table, root_size, entries, 0, MAXIMUM_FILES, 0, 1 }; - - walk_directory(&walk, 0, 0); - for (index = 0; index < walk.entry_count && !names_match(entries[index].name, "maps"); index++) - ; - if (index == walk.entry_count) + xiso_error(error, error_size, "Out of memory."); + goto done; + } + for (found = 0, index = 0; index < count && !found; index++) + { + if (xiso_name_is(entries[index].name, (int)strlen(entries[index].name), "maps")) { - fail(&image, "The disc image has no maps folder: it is not a Halo disc.%s", NULL); - goto done; + maps_sector = entries[index].sector; + maps_size = entries[index].size; + found = 1; } - free(table); - table = read_directory(&image, entries[index].sector, entries[index].size); - if (!table) - { - fail(&image, "The disc image's maps folder is damaged.%s", NULL); - goto done; - } - root_size = entries[index].size; + } + if (!found) + { + xiso_error(error, error_size, "The disc image has no maps folder: it is not a Halo disc."); + goto done; } /* its files */ + table = xiso_read_table(&image, maps_sector, maps_size, &out_of_memory); + if (!table) { - struct directory_walk walk = { table, root_size, entries, 0, MAXIMUM_FILES, 0, 0 }; - int has_ui = 0; - - walk_directory(&walk, 0, 0); - for (index = 0; index < walk.entry_count; index++) + xiso_error(error, error_size, out_of_memory ? "Out of memory." : "The disc image's maps folder is damaged."); + goto done; + } + count = xiso_walk_table(table, maps_size, 0, entries); + free(table); + table = NULL; + if (count < 0) + { + xiso_error(error, error_size, "Out of memory."); + goto done; + } + for (found = 0, index = 0; index < count && !found; index++) + found = xiso_name_is(entries[index].name, (int)strlen(entries[index].name), "ui.map"); + if (!found) + { + xiso_error(error, error_size, "The disc image's maps folder has no ui.map: it is not a Halo disc."); + goto done; + } + /* (an image that ends before a file does is refused before anything is + written; an empty file's sector need not be anywhere) */ + for (index = 0; index < count; index++) + { + if (entries[index].size && + xiso_sector_offset(&image, entries[index].sector) + entries[index].size > image.size) { - total += entries[index].size; - has_ui |= names_match(entries[index].name, "ui.map"); - } - if (!has_ui) - { - fail(&image, "The disc image's maps folder has no ui.map: it is not a Halo disc.%s", NULL); + xiso_error(error, error_size, "Could not read %s from the disc image (is it complete?).", + entries[index].name); goto done; } - snprintf(partial, sizeof(partial), "%s/maps.partial", destination); - snprintf(final, sizeof(final), "%s/maps", destination); - posix_make_directory(partial); - for (index = 0; index < walk.entry_count; index++) + total += entries[index].size; + } + + /* the copy, to maps.partial */ + if (snprintf(partial_path, XISO_MAXIMUM_PATH, "%s/" XISO_PARTIAL_DIRECTORY, destination) >= XISO_MAXIMUM_PATH || + snprintf(maps_path, XISO_MAXIMUM_PATH, "%s/maps", destination) >= XISO_MAXIMUM_PATH) + { + xiso_error(error, error_size, "Could not create %s/maps.", destination); + goto done; + } + /* (an existing one is reused, emptied; a failure to make it shows as a + failure to write the first file) */ + posix_make_directory(partial_path); + xiso_empty_directory(partial_path, file_path, XISO_MAXIMUM_PATH); + for (index = 0; index < count; index++) + { + platform_log("extracting maps/%s (%lu bytes)", entries[index].name, entries[index].size); + if (snprintf(file_path, XISO_MAXIMUM_PATH, "%s/%s", partial_path, entries[index].name) >= XISO_MAXIMUM_PATH) { - snprintf(path, sizeof(path), "%s/%s", partial, entries[index].name); - platform_log("extracting maps/%s (%lu bytes)", entries[index].name, entries[index].size); - if (!copy_file(&image, &entries[index], path, buffer, &done, total, progress, context)) - goto done; + xiso_error(error, error_size, "Could not write %s/%s.", partial_path, entries[index].name); + goto done; } - if (rename(partial, final) != 0) + if (!xiso_copy_file(&image, &entries[index], file_path, buffer, &done, total, progress, context, + error, error_size)) { - fail(&image, "Could not create %s.", final); goto done; } } - result = 1; + if (rename(partial_path, maps_path) != 0) + { + xiso_error(error, error_size, "Could not create %s.", maps_path); + goto done; + } + succeeded = 1; done: - close(image.descriptor); free(table); - free(entries); + free(maps_path); + free(file_path); + free(partial_path); free(buffer); - return result; + free(entries); + close(image.descriptor); + return succeeded; } #endif diff --git a/port/third_party/extract-xiso/LICENSE.TXT b/port/third_party/extract-xiso/LICENSE.TXT deleted file mode 100644 index b89be76a..00000000 --- a/port/third_party/extract-xiso/LICENSE.TXT +++ /dev/null @@ -1,75 +0,0 @@ -This is an xdvdfs (xbox iso) file creation/extraction utility for linux/darwin/ -freebsd. - -If you want to port it to your OS it shouldn't be too hard, just make sure your -fseek/ftell takes 64-bit file offsets. - -I wrote this code completely from scratch today because xbiso (the one on -sourceforge) keeps seg-faulting and the code is buggy. I got useful -documentation on the xdvdfs file system from http://xbox-linux.sourceforge.net. - -I will say thanks though to the author of xbiso, it was a good start and got me -thinking! - - -Regarding licensing: - -I think the GPL sucks! (it stands for Generosity Poor License) - -My open-source code is really *FREE* so you can do whatever you want with it, as -long as 1) you don't claim that you wrote my code and 2) you retain a notice -that some parts of the code are copyright in@fishtank.com and 3) you understand -there are no warranties. I only guarantee that it will take up disk space! - -If you want to help out with this project it would be welcome, just email me at -in@fishtank.com. - -This code is copyright in@fishtank.com and is licensed under a slightly -modifified version of the Berkeley Software License, which follows: - -/* - * Copyright (c) 2003 in - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * 3. All advertising materials mentioning features or use of this software - * must display the following acknowledgement: - * - * This product includes software developed by in . - * - * 4. Neither the name of "in" nor the email address "in@fishtank.com" - * may be used to endorse or promote products derived from this software - * without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED `AS IS' AND ANY EXPRESS OR IMPLIED WARRANTIES - * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND - * FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * AUTHOR OR ANY CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; - * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, - * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR - * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF - * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - */ - -Please send bug reports to me at in@fishtank.com - -To compile, run 'make' in the source directory. - -NOTE: You need to be running GNU make and /bin/bash must be executable - -Enjoy! - -in version 1.0 -March 10, 2003 diff --git a/port/third_party/extract-xiso/README.md b/port/third_party/extract-xiso/README.md deleted file mode 100644 index 8b61fcd5..00000000 --- a/port/third_party/extract-xiso/README.md +++ /dev/null @@ -1,16 +0,0 @@ -# extract-xiso - -An Xbox disc image (xdvdfs) extraction and creation tool by in -, under a modified BSD license (see `LICENSE.TXT`, copied -unchanged from upstream). - -Upstream: https://github.com/XboxDev/extract-xiso, commit -3f5b62cfe68f000b0e3c8a30104973f3a297948e. - -None of its code is built as it is: `port/linux/src/xiso.c`, which copies -the maps folder out of a disc image when a desktop port starts without game -data, reads the format as `extract-xiso.c` does and carries its notice. The -desktop builds' artifacts include `LICENSE.TXT` as -`extract-xiso-LICENSE.txt`. - -This product includes software developed by in . diff --git a/port/web/licenses.py b/port/web/licenses.py index e8ef775c..98c01d33 100644 --- a/port/web/licenses.py +++ b/port/web/licenses.py @@ -1,7 +1,7 @@ -"""Writes the web site's licenses.txt (tools/web_build.py): the notices of -the code and fonts that halo.wasm and the page carry, which their licences -ask copies to include, as the other builds' releases carry them -(tools/ci_build.py). +"""Writes the web site's licenses.txt (tools/web_build.py): OpenCE's licence, +and the notices of the code and fonts that halo.wasm and the page carry, +which their licences ask copies to include, as the other builds' releases +carry them (tools/ci_build.py). python port/web/licenses.py OUTPUT SDL3-LICENSE """ @@ -21,7 +21,6 @@ NOTICES = [ ("tomlc17 (config.toml, mod.toml, load_order.toml)", "port/third_party/tomlc17/LICENSE"), ("musl (the maths functions)", "port/third_party/musl-math/COPYRIGHT"), ("stb_vorbis (Custom Edition maps' and mods' sounds)", "port/third_party/stb_vorbis/LICENSE"), - ("extract-xiso (how the disc image is read)", "port/third_party/extract-xiso/LICENSE.TXT"), ("Overpass (the menus' text and the page's)", "port/assets/fonts/Overpass-OFL.txt"), ("OpenCE (the title font, from Newtown)", "port/assets/fonts/OpenCE-OFL.txt"), ("Newtown", "port/assets/fonts/Newtown-LICENSE.txt"), @@ -33,12 +32,17 @@ def main() -> None: output, sdl_license = Path(sys.argv[1]), Path(sys.argv[2]) parts = [ "OpenCE for the web\n\n" - "OpenCE's own code is CC0 (https://github.com/OpenCommunityEdition/OpenCE). The game data\n" - "comes from the player's own disc. It carries the following, under their licences:\n", + "OpenCE is free software under the GNU Affero General Public License, version 3 or (at\n" + "your option) any later version; its text is at the end of this file. Its source is at\n" + "https://github.com/OpenCommunityEdition/OpenCE, and each release's at its tag. The game\n" + "data comes from the player's own disc. OpenCE carries the following, under their\n" + "licences:\n", f"\n==================== SDL 3 (windows, input and audio)\n\n{sdl_license.read_text(encoding='utf-8')}", ] for title, path in NOTICES: parts.append(f"\n==================== {title}\n\n{(ROOT / path).read_text(encoding='utf-8')}") + parts.append(f"\n==================== OpenCE (GNU Affero General Public License)\n\n" + f"{(ROOT / 'LICENSE').read_text(encoding='utf-8')}") output.write_text("".join(parts), encoding="utf-8") diff --git a/port/web/site/xiso-worker.js b/port/web/site/xiso-worker.js index 4aaaa49f..49e973fe 100644 --- a/port/web/site/xiso-worker.js +++ b/port/web/site/xiso-worker.js @@ -1,211 +1,245 @@ /* XISO-WORKER.JS -Copies the game's maps folder into the site's Origin Private File System, -where the game finds it (/data/maps: port/web/src/web_main.c), from the -player's own Xbox disc image of Halo: Combat Evolved (an "xiso", .iso), as -port/linux/src/xiso.c does for the desktop ports, or from a maps folder -already copied out of one. It runs in a Worker, which can write with -synchronous OPFS access handles (Safari has no OPFS writable streams), and -reads the image a slice at a time: an image is 4 to 7 GB, and nothing of it -is kept but the maps. +Copies the game data into the site's Origin Private File System, where the +game reads it (/data/maps: port/web/src/web_main.c): the maps folder of the +player's Xbox disc image of Halo: Combat Evolved, or the files of a maps +folder the player chose. It runs in a Worker, which can write with +synchronous OPFS access handles (Safari has no OPFS writable streams). Only +the maps and loading.tga are kept, and each map's header is checked as the +game checks it, so that a disc the game cannot play is refused here. -The image's file system is XDVDFS, read as extract-xiso does -(port/third_party/extract-xiso): 2048-byte sectors; a volume descriptor at -0x10000 that starts and ends with "MICROSOFT*XBOX*MEDIA" and gives the root -directory's sector and size; directories whose entries form a binary tree -(left and right subtree offsets in 4-byte units, the start sector, the size, -attributes, the name's length and the name). Images of a whole disc put the -game partition further in, at one of PARTITION_OFFSETS. The image is the -player's file, so it is untrusted: every offset and size is checked against -the image and the directory it is read from, and a directory's walk is -bounded. +Written from a description of the Xbox's disc file system, XDVDFS +(xboxdevwiki's XDVDFS page, https://xboxdevwiki.net/XDVDFS, and the format +facts OpenCE's readers rely on). What this file relies on: -Each map is checked as the game checks it when it loads it (cache_files.c): -"head" and "foot" signatures, the Xbox's cache version, and the build of a -retail disc (01.10.12.2276) or of this decompilation's (01.01.14.2342). +- Sectors are 2048 bytes and integers little-endian. Sector numbers count + from the start of the game partition: an image of the partition alone has + it at 0, an image of a whole disc further in (0x0FD90000, 0x02080000 or + 0x18300000 for XGD2, XGD3 and XGD1 discs). They are tried in that order. +- The volume descriptor is the partition's sector 32 (at 0x10000). It holds + "MICROSOFT*XBOX*MEDIA" at its start and again at 0x7EC, and the root + directory table's start sector (u32 at 0x14) and size in bytes (u32 at + 0x18). +- A directory table is a binary tree of entries whose root is the entry at + offset 0. An entry is a left and a right subtree link (u16 each, in 4-byte + units from the table's start, 0 for none), the start sector and the size + of the file or of the subdirectory's table (u32 each), the attributes (u8, + 0x10 for a directory), the name's length (u8) and the name. Unused bytes of + a table are 0xFF, so a left link of 0xFFFF is padding, not an entry. +- A file is one contiguous run of bytes from the start of its sector. -Messages from the page: { type: 'image', file } or { type: 'folder', files } -(a maps folder's File objects). To the page: { type: 'progress', file, done, -total }, then { type: 'done', files, bytes } or { type: 'error', message }. +The image is the player's file (4 to 7 GB) and nothing read from it is +trusted: it is read a slice at a time, only the root's table and the maps +folder's (each at most 1 MB), and the walk of a table visits each offset +once, goes at most 64 links deep and looks at most 4096 entries. + +Messages from the page: { type: 'image', file } (the disc image) or +{ type: 'folder', files } (a chosen folder's File objects). To the page: +{ type: 'progress', file, done, total }, then { type: 'done', files, bytes } +or { type: 'error', message }. A new copy first removes the old maps folder +and writes maps/.complete ({"files":[...],"bytes":n}) last: the page takes +the data as ready only once that marker is there. */ 'use strict'; const SECTOR_SIZE = 2048; -const VOLUME_DESCRIPTOR_OFFSET = 0x10000; -const PARTITION_OFFSETS = [0, 0x0FD90000, 0x02080000, 0x18300000]; +const DESCRIPTOR_OFFSET = 0x10000; const MAGIC = 'MICROSOFT*XBOX*MEDIA'; -const ENTRY_HEADER_SIZE = 14; +const SECOND_MAGIC_OFFSET = 0x7EC; +const ENTRY_FIXED_SIZE = 14; const ATTRIBUTE_DIRECTORY = 0x10; -const MAXIMUM_DIRECTORY_SIZE = 1024 * 1024; -const MAXIMUM_ENTRIES = 4096; +const PADDING_LINK = 0xFFFF; +// where the game partition may start, in the order they are tried: an image +// of the partition alone, then whole XGD2, XGD3 and XGD1 discs +const PARTITION_STARTS = [0x00000000, 0x0FD90000, 0x02080000, 0x18300000]; +// directory tables are a few sectors; anything much larger is not one +const MAXIMUM_TABLE_SIZE = 1024 * 1024; const MAXIMUM_DEPTH = 64; -const COPY_CHUNK = 4 * 1024 * 1024; +const MAXIMUM_ENTRIES = 4096; const MAP_HEADER_SIZE = 0x800; -const CACHE_VERSION = 5; -const BUILDS = ['01.10.12.2276', '01.01.14.2342']; -// written last: the page starts the game only when it is there -const COMPLETE_MARKER = '.complete'; +const MAP_BUILDS = ['01.10.12.2276', '01.01.14.2342']; +const COPY_CHUNK = 4 * 1024 * 1024; -async function readAt(file, offset, size) { - const end = Math.min(offset + size, file.size); - if (offset < 0 || offset >= end) return new Uint8Array(0); - return new Uint8Array(await file.slice(offset, end).arrayBuffer()); +// ---------- names + +function asciiLower(text) { + return text.replace(/[A-Z]/g, (letter) => String.fromCharCode(letter.charCodeAt(0) + 32)); } -function text(bytes, start, length) { - let result = ''; - for (let i = 0; i < length; i++) result += String.fromCharCode(bytes[start + i]); - return result; +function sameName(a, b) { + return asciiLower(a) === asciiLower(b); } -function u16(bytes, at) { - return bytes[at] | bytes[at + 1] << 8; +// A name a file of the maps folder may have: no path, nothing hidden. +function plainName(name) { + return typeof name === 'string' && /^[A-Za-z0-9_. -]{1,64}$/.test(name) && !name.startsWith('.'); } -function u32(bytes, at) { - return (bytes[at] | bytes[at + 1] << 8 | bytes[at + 2] << 16 | bytes[at + 3] << 24) >>> 0; +function latin1(bytes) { + let text = ''; + for (const byte of bytes) text += String.fromCharCode(byte); + return text; } +// ---------- the image + +async function readBytes(file, start, end) { + return new Uint8Array(await file.slice(start, end).arrayBuffer()); +} + +// The game partition: { partition, rootSector, rootSize }, from the first +// start whose volume descriptor has both magics; null if none has. async function findVolume(file) { - for (const partition of PARTITION_OFFSETS) { - const descriptor = await readAt(file, partition + VOLUME_DESCRIPTOR_OFFSET, SECTOR_SIZE); - if (descriptor.length < SECTOR_SIZE) continue; - if (text(descriptor, 0, 20) === MAGIC && text(descriptor, 0x7EC, 20) === MAGIC) { - return { partition, rootSector: u32(descriptor, 20), rootSize: u32(descriptor, 24) }; + for (const partition of PARTITION_STARTS) { + const start = partition + DESCRIPTOR_OFFSET; + if (start + SECTOR_SIZE > file.size) continue; + const descriptor = await readBytes(file, start, start + SECTOR_SIZE); + if (descriptor.length !== SECTOR_SIZE) continue; + if (latin1(descriptor.subarray(0, MAGIC.length)) !== MAGIC || + latin1(descriptor.subarray(SECOND_MAGIC_OFFSET, SECOND_MAGIC_OFFSET + MAGIC.length)) !== MAGIC) { + continue; } + const view = new DataView(descriptor.buffer, descriptor.byteOffset, descriptor.byteLength); + return { partition, rootSector: view.getUint32(0x14, true), rootSize: view.getUint32(0x18, true) }; } return null; } -// a name a file of the maps folder may have: no path, nothing hidden -function plainName(name) { - return name.length > 0 && name.length <= 64 && /^[A-Za-z0-9_.\- ]+$/.test(name) && !name.startsWith('.'); -} - -// the entries of a directory table: { name, sector, size, directory } +// A directory table's entries, { name, sector, size, directory }, for every +// entry that has a name, in the tree's order (left subtree, the entry, right +// subtree). A link to padding, past the table or to an offset already +// visited ends that branch. function walkDirectory(table) { + const view = new DataView(table.buffer, table.byteOffset, table.byteLength); + const visited = new Set(); const entries = []; - const seen = new Set(); const walk = (offset, depth) => { - offset *= 4; - // (a cycle, a tree too deep or too wide, or an entry past the table: - // what is left of it is ignored) - if (depth > MAXIMUM_DEPTH || seen.has(offset) || seen.size >= MAXIMUM_ENTRIES || - offset + ENTRY_HEADER_SIZE > table.length) { - return; - } - seen.add(offset); - const left = u16(table, offset); - const right = u16(table, offset + 2); - // (padding at the end of a sector) - if (left === 0xFFFF) return; + if (depth > MAXIMUM_DEPTH || offset + ENTRY_FIXED_SIZE > table.length) return; + const left = view.getUint16(offset, true); + const right = view.getUint16(offset + 2, true); + if (left === PADDING_LINK || visited.has(offset) || visited.size >= MAXIMUM_ENTRIES) return; + visited.add(offset); + if (left) walk(left * 4, depth + 1); const nameLength = table[offset + 13]; - if (left) walk(left, depth + 1); - if (nameLength > 0 && offset + ENTRY_HEADER_SIZE + nameLength <= table.length) { + const nameStart = offset + ENTRY_FIXED_SIZE; + if (nameLength && nameStart + nameLength <= table.length) { entries.push({ - name: text(table, offset + ENTRY_HEADER_SIZE, nameLength), - sector: u32(table, offset + 4), - size: u32(table, offset + 8), + name: latin1(table.subarray(nameStart, nameStart + nameLength)), + sector: view.getUint32(offset + 4, true), + size: view.getUint32(offset + 8, true), directory: (table[offset + 12] & ATTRIBUTE_DIRECTORY) !== 0, }); } - if (right) walk(right, depth + 1); + if (right) walk(right * 4, depth + 1); }; walk(0, 0); return entries; } -async function readDirectory(file, volume, sector, size) { - if (!size || size > MAXIMUM_DIRECTORY_SIZE) return null; - const offset = volume.partition + sector * SECTOR_SIZE; - if (offset + size > file.size) return null; - const table = await readAt(file, offset, size); - return table.length === size ? table : null; +async function readTable(file, partition, sector, size, message) { + const start = partition + sector * SECTOR_SIZE; + if (size === 0 || size > MAXIMUM_TABLE_SIZE || start + size > file.size) throw new Error(message); + const table = await readBytes(file, start, start + size); + if (table.length !== size) throw new Error(message); + return table; } -// why a map's header is not one the game loads, or null +// ---------- the maps + +// Why the game would not load a map whose first bytes are header, or null. function mapProblem(header) { if (header.length < MAP_HEADER_SIZE) return 'it is too short'; - if (text(header, 0, 4) !== 'daeh' || text(header, 0x7FC, 4) !== 'toof') return 'it is not a map'; - if (u32(header, 4) !== CACHE_VERSION) return 'it is not an Xbox map (its cache version is ' + u32(header, 4) + ')'; - const build = text(header, 0x40, 32).replace(/\0.*$/s, ''); - if (!BUILDS.includes(build)) return 'its build, ' + JSON.stringify(build) + ', is not one the game loads'; + if (latin1(header.subarray(0, 4)) !== 'daeh' || latin1(header.subarray(0x7FC, 0x800)) !== 'toof') { + return 'it is not a map'; + } + const view = new DataView(header.buffer, header.byteOffset, header.byteLength); + const version = view.getUint32(4, true); + if (version !== 5) return `it is not an Xbox map (its cache version is ${version})`; + const field = header.subarray(0x40, 0x60); + const end = field.indexOf(0); + const build = latin1(end >= 0 ? field.subarray(0, end) : field); + if (!MAP_BUILDS.includes(build)) return `its build, ${JSON.stringify(build)}, is not one the game loads`; return null; } -// the files to copy, each { name, size, read(offset, size) }; the maps are -// checked, other files (loading.tga) copied as they are +// The files to copy, of those given ({ name, size, read }): the maps (each +// checked) and loading.tga, under plain names, the first of names that +// differ only in case; throws why they cannot be used. async function checkedFiles(candidates) { - const files = []; + const kept = []; for (const candidate of candidates) { - const lower = candidate.name.toLowerCase(); if (!plainName(candidate.name)) continue; + if (kept.some((file) => sameName(file.name, candidate.name))) continue; + const lower = asciiLower(candidate.name); if (lower.endsWith('.map')) { const problem = mapProblem(await candidate.read(0, MAP_HEADER_SIZE)); if (problem) throw new Error(`${candidate.name} cannot be used: ${problem}.`); - files.push(candidate); + kept.push(candidate); } else if (lower === 'loading.tga') { - files.push(candidate); + kept.push(candidate); } } - if (!files.some((file) => file.name.toLowerCase() === 'ui.map')) { + if (!kept.some((file) => sameName(file.name, 'ui.map'))) { throw new Error('There is no ui.map among the maps: this is not Halo: Combat Evolved for the Xbox.'); } - return files; + return kept; } +// The maps folder's files in a disc image, as { name, size, read(offset, +// size) }; throws why it cannot be used. async function imageFiles(file) { const volume = await findVolume(file); if (!volume) throw new Error('This file is not an Xbox disc image (no XDVDFS volume was found).'); - const root = await readDirectory(file, volume, volume.rootSector, volume.rootSize); - if (!root) throw new Error("The disc image's file system is damaged."); - const maps = walkDirectory(root).find((entry) => entry.directory && entry.name.toLowerCase() === 'maps'); + const { partition } = volume; + const root = await readTable(file, partition, volume.rootSector, volume.rootSize, + 'The disc image\'s file system is damaged.'); + const maps = walkDirectory(root).find((entry) => entry.directory && sameName(entry.name, 'maps')); if (!maps) throw new Error('The disc image has no maps folder: it is not a Halo disc.'); - const table = await readDirectory(file, volume, maps.sector, maps.size); - if (!table) throw new Error("The disc image's maps folder is damaged."); - return checkedFiles(walkDirectory(table).filter((entry) => !entry.directory).map((entry) => { - const offset = volume.partition + entry.sector * SECTOR_SIZE; - if (offset + entry.size > file.size) { + const table = await readTable(file, partition, maps.sector, maps.size, 'The disc image\'s maps folder is damaged.'); + const candidates = []; + for (const entry of walkDirectory(table)) { + if (entry.directory) continue; + const start = partition + entry.sector * SECTOR_SIZE; + // (an empty file's sector need not be anywhere) + if (entry.size && start + entry.size > file.size) { throw new Error(`The disc image ends before ${entry.name} does: is it complete?`); } - return { name: entry.name, size: entry.size, read: (at, size) => readAt(file, offset + at, Math.min(size, entry.size - at)) }; - })); + candidates.push({ + name: entry.name, + size: entry.size, + read: (offset, size) => { + const count = Math.max(0, Math.min(size, entry.size - offset)); + return readBytes(file, Math.min(start + offset, file.size), Math.min(start + offset + count, file.size)); + }, + }); + } + return checkedFiles(candidates); } +// The same, from the files of a chosen folder. async function folderFiles(list) { - return checkedFiles(list.map((file) => ({ name: file.name, size: file.size, read: (at, size) => readAt(file, at, size) }))); + return checkedFiles(list.map((file) => ({ + name: file.name, + size: file.size, + read: (offset, size) => readBytes(file, offset, offset + size), + }))); } -async function writeFile(directory, source, onChunk) { - const handle = await directory.getFileHandle(source.name, { create: true }); - const access = await handle.createSyncAccessHandle(); - try { - access.truncate(0); - let position = 0; - while (position < source.size) { - const count = Math.min(COPY_CHUNK, source.size - position); - const bytes = await source.read(position, count); - if (bytes.length !== count) throw new Error(`Could not read ${source.name} (is the file complete?).`); - let written = 0; - while (written < count) { - const result = access.write(bytes.subarray(written), { at: position + written }); - if (!result) throw new Error(`Could not write ${source.name}: is the device full?`); - written += result; - } - position += count; - onChunk(count); - } - access.flush(); - } finally { - access.close(); +// ---------- the copy + +function writeAll(access, bytes, position, name) { + let written = 0; + while (written < bytes.length) { + const result = access.write(bytes.subarray(written), { at: position + written }); + if (!result) throw new Error(`Could not write ${name}: is the device full?`); + written += result; } } async function copy(files) { - const total = files.reduce((sum, entry) => sum + entry.size, 0); - const storage = await navigator.storage.getDirectory(); + const total = files.reduce((sum, file) => sum + file.size, 0); if (navigator.storage.estimate) { const { quota, usage } = await navigator.storage.estimate(); if (quota && quota - usage < total) { @@ -213,34 +247,54 @@ async function copy(files) { `and the maps take ${Math.ceil(total / 1e6)} MB.`); } } - // a new copy replaces whatever an earlier, perhaps interrupted, one left + const storage = await navigator.storage.getDirectory(); + // (a new copy replaces whatever an earlier, perhaps interrupted, one left) await storage.removeEntry('maps', { recursive: true }).catch(() => {}); - const directory = await storage.getDirectoryHandle('maps', { create: true }); + const maps = await storage.getDirectoryHandle('maps', { create: true }); let done = 0; let lastReport = 0; - for (const entry of files) { - await writeFile(directory, entry, (count) => { - done += count; - const now = Date.now(); - if (now - lastReport > 100 || done === total) { - lastReport = now; - postMessage({ type: 'progress', file: entry.name, done, total }); + for (const file of files) { + const handle = await maps.getFileHandle(file.name, { create: true }); + const access = await handle.createSyncAccessHandle(); + try { + access.truncate(0); + let position = 0; + while (position < file.size) { + const count = Math.min(COPY_CHUNK, file.size - position); + const bytes = await file.read(position, count); + if (bytes.length < count) throw new Error(`Could not read ${file.name} (is the file complete?).`); + writeAll(access, bytes.subarray(0, count), position, file.name); + position += count; + done += count; + const now = Date.now(); + if (now - lastReport > 100 || done === total) { + lastReport = now; + postMessage({ type: 'progress', file: file.name, done, total }); + } } - }); + access.flush(); + } finally { + access.close(); + } + } + // last: the sign that the copy is complete + const marker = await (await maps.getFileHandle('.complete', { create: true })).createSyncAccessHandle(); + try { + marker.truncate(0); + const text = new TextEncoder().encode(JSON.stringify({ files: files.map((file) => file.name), bytes: total })); + writeAll(marker, text, 0, '.complete'); + marker.flush(); + } finally { + marker.close(); } - const marker = await directory.getFileHandle(COMPLETE_MARKER, { create: true }); - const access = await marker.createSyncAccessHandle(); - access.truncate(0); - access.write(new TextEncoder().encode(JSON.stringify({ files: files.map((f) => f.name), bytes: total })), { at: 0 }); - access.flush(); - access.close(); return { files: files.length, bytes: total }; } if (typeof onmessage !== 'undefined' || typeof WorkerGlobalScope !== 'undefined') { onmessage = async (event) => { try { - const files = event.data.type === 'folder' ? await folderFiles(event.data.files) : await imageFiles(event.data.file); + const message = event.data; + const files = message.type === 'folder' ? await folderFiles(message.files) : await imageFiles(message.file); postMessage({ type: 'done', ...(await copy(files)) }); } catch (error) { postMessage({ type: 'error', message: error && error.message ? error.message : String(error) }); diff --git a/tools/ci_build.py b/tools/ci_build.py index a0f6335f..76f92cdd 100644 --- a/tools/ci_build.py +++ b/tools/ci_build.py @@ -117,10 +117,8 @@ def main() -> int: for pdb in [ROOT / "build/windows/halo.pdb", *sorted((ROOT / "build/windows/third_party").glob("SDL3-*/lib/x86/SDL3.pdb"))]: shutil.copy2(pdb, symbols) print(f"{pdb.relative_to(ROOT)} -> {symbols.relative_to(ROOT)}", flush=True) - # the disc image readers (port/linux/src/xiso.c, and the Android app's - # XisoExtractor.java) follow extract-xiso, whose license asks binaries - # to carry its notice - shutil.copy2(ROOT / "port/third_party/extract-xiso/LICENSE.TXT", dist / "extract-xiso-LICENSE.txt") + # OpenCE's own licence, the AGPL, which asks each copy to carry its text + shutil.copy2(ROOT / "LICENSE", dist / "LICENSE.txt") if args.platform != "web": # internet play's DTLS with browsers, and the Linux self-updater's TLS # (port/third_party/mbedtls), whose Apache license asks the same diff --git a/tools/web_build.py b/tools/web_build.py index fa166ffe..677fdeea 100644 --- a/tools/web_build.py +++ b/tools/web_build.py @@ -366,7 +366,7 @@ def generate_web_build(n: Writer, sln: Any) -> None: command=f"$python {PORT_DIR}/licenses.py $out {sdl_license}", description="WEB LICENSES $out", ) - n.build(outputs=licenses, rule="web_licenses", implicit=[PORT_DIR / "licenses.py", sdl_license]) + n.build(outputs=licenses, rule="web_licenses", implicit=[PORT_DIR / "licenses.py", Path("LICENSE"), sdl_license]) site_files.append(licenses) n.rule( name="web_version",