From d8abf2c4383ac45967a057dea07d970111907fb6 Mon Sep 17 00:00:00 2001 From: Toastconcern Date: Wed, 21 Oct 2026 12:26:16 -0500 Subject: [PATCH] Environment bundles (.fbundle): several .fenv files in one, each its own environment src/fenv_bundle.h: a ZIP of bundle.json and the .fenv files stored as-is, copied in pieces (zip64 from the start when 4 GB or more). Write, read the manifest, read one environment's fenv.json + thumbnail in place, unpack all or one (.part then rename, checksummed; never replaces a different file: "name 2"), validate. fenv-tool bundle / unbundle, and info lists a bundle's environments. The switcher unpacks bundles dropped into envs/ into their environments and deletes them (frame-switcher --unpack-bundles does it at once). The installers copy .fbundle files alongside .fenv; only a .fenv sets the one shown. tools/bundletest checks it against real environments (--big: past 4 GB). Co-Authored-By: Claude Opus 5.5 --- README.md | 4 + frame/Install Frame Home.bat | 4 +- frame/README-frame-home.txt | 15 +- frame/install-frame-home.ps1 | 14 +- frame/install-frame-home.sh | 12 +- src/fenv_bundle.h | 375 ++++++++++++++++++++++++++++++++ src/fenv_tool.cpp | 62 +++++- src/switcher.cpp | 37 +++- tools/bundletest/CMakeLists.txt | 21 ++ tools/bundletest/bundletest.cpp | 239 ++++++++++++++++++++ 10 files changed, 765 insertions(+), 18 deletions(-) create mode 100644 src/fenv_bundle.h create mode 100644 tools/bundletest/CMakeLists.txt create mode 100644 tools/bundletest/bundletest.cpp diff --git a/README.md b/README.md index 4a473c7..4099903 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,10 @@ largest homes may not fit. **Frame Home without Studio:** `FrameHome-installer.zip` installs it on the headset from any computer (or on the headset itself); see `frame/README-frame-home.txt` inside it. +**Environment bundles** (`.fbundle`): several `.fenv` environments in one file, to share or install together; each +stays its own environment. `fenv-tool bundle out.fbundle a.fenv b.fenv` makes one, `fenv-tool unbundle` takes it +apart, and on the headset the Environments panel unpacks one dropped into `~/frame-home/envs` (`src/fenv_bundle.h`). + ## Building Everything Studio needs is in `third_party/` (Vulkan is loaded at runtime). CMake 3.16+ and Ninja. diff --git a/frame/Install Frame Home.bat b/frame/Install Frame Home.bat index 8b08456..08110b3 100644 --- a/frame/Install Frame Home.bat +++ b/frame/Install Frame Home.bat @@ -1,6 +1,6 @@ @echo off -rem Installs Frame Home on the Steam Frame (no Frame Studio needed). Double-click it, or drop .fenv environments on -rem it to copy them to the headset too. Asks for the headset's address the first time (default: frame). +rem Installs Frame Home on the Steam Frame (no Frame Studio needed). Double-click it, or drop .fenv environments or +rem .fbundle bundles on it to copy them to the headset too. Asks for the headset's address the first time (default: frame). setlocal set "HOSTFILE=%APPDATA%\FrameStudio\frame-host.txt" set "FRAMEHOST=" diff --git a/frame/README-frame-home.txt b/frame/README-frame-home.txt index 2fa26ed..d1a4664 100644 --- a/frame/README-frame-home.txt +++ b/frame/README-frame-home.txt @@ -14,12 +14,15 @@ What you need Install from a Windows PC ------------------------- Double-click "Install Frame Home.bat" and type the headset's address (Enter keeps the one shown). -To copy environments to the headset at the same time, drop .fenv files onto "Install Frame Home.bat". +To copy environments to the headset at the same time, drop .fenv files (or .fbundle bundles) onto +"Install Frame Home.bat". (Needs Windows' OpenSSH Client, under Settings > Apps > Optional features, or Git for Windows.) Install from Linux or macOS --------------------------- - sh install-frame-home.sh --host steamos@frame [environment.fenv ...] + sh install-frame-home.sh --host steamos@frame [environment.fenv | bundle.fbundle ...] + +The first .fenv given is the one shown. Install on the headset itself ----------------------------- @@ -29,6 +32,14 @@ Copy this folder to the headset (Desktop Mode), open a terminal (Konsole) in it Put .fenv environments in ~/frame-home/envs; the Environments panel in the Steam menu lists them. +Bundles +------- +A .fbundle holds several environments in one file, to share or install together (Frame Studio or +"fenv-tool bundle" makes them). Copied into ~/frame-home/envs (the installers do that), it is unpacked there by the +Environments panel into its .fenv environments, each listed on its own, and then deleted. An environment already +there is kept: the same one is skipped, a different one of the same name gets the bundle's as "name 2". A bundle +never changes which environment is shown. + Uninstall --------- sh install-frame-home.sh --uninstall (Linux / macOS) diff --git a/frame/install-frame-home.ps1 b/frame/install-frame-home.ps1 index b3d1960..63703d0 100644 --- a/frame/install-frame-home.ps1 +++ b/frame/install-frame-home.ps1 @@ -2,11 +2,13 @@ # Frame > Install on Steam Frame. Needs ssh access to the headset (its developer key, or a password) and an ssh # client: Windows' own OpenSSH (Settings > Optional features) or Git for Windows'. # -# powershell -ExecutionPolicy Bypass -File install-frame-home.ps1 [-HostName HOST] [-Uninstall] [environment.fenv ...] +# powershell -ExecutionPolicy Bypass -File install-frame-home.ps1 [-HostName HOST] [-Uninstall] [environment.fenv | bundle.fbundle ...] # # HOST: the headset's ssh address (default "steamos@frame": an entry named frame in ~/.ssh/config, or the headset's # name on the network; an IP works too, e.g. steamos@192.168.1.50). Environments (.fenv) given are copied into its -# selector as well; the first is the one shown. -Uninstall gives the environment back to SteamVR (files stay). +# selector as well; the first .fenv is the one shown. Bundles (.fbundle) are copied as they are and the headset's +# Environments panel unpacks them into their environments (they never change the one shown). -Uninstall gives the +# environment back to SteamVR (files stay). param( [string]$HostName = "steamos@frame", [switch]$Uninstall, @@ -64,9 +66,11 @@ if ($Uninstall) { } & $ssh $HostName "sed -i 's/\r`$//' frame-home/.install/frame/install.sh; sh frame-home/.install/frame/install.sh" if ($LASTEXITCODE -ne 0) { throw "The installer on the headset failed (see above)" } - if ($Environments.Count -gt 0 -and (Test-Path $Environments[0])) { # the first is the one shown - $first = Frame-Name $Environments[0] - & $ssh $HostName "printf '%s\n' '$first' > frame-home/current" + # The first environment given is the one shown (a bundle's are only known once the headset unpacks it: skipped). + $first = $Environments | Where-Object { $_ -match "\.fenv$" } | Select-Object -First 1 + if ($first -and (Test-Path $first)) { + $name = Frame-Name $first + & $ssh $HostName "printf '%s\n' '$name' > frame-home/current" } } Say "done" diff --git a/frame/install-frame-home.sh b/frame/install-frame-home.sh index 163b2fd..22ec6d5 100755 --- a/frame/install-frame-home.sh +++ b/frame/install-frame-home.sh @@ -2,11 +2,13 @@ # Installs Frame Home on a Steam Frame from a Linux or macOS computer, without Frame Studio: the same steps as # Studio's Frame > Install on Steam Frame. Needs ssh access to the headset (its developer key, or a password). # -# sh install-frame-home.sh [--host HOST] [--uninstall] [environment.fenv ...] +# sh install-frame-home.sh [--host HOST] [--uninstall] [environment.fenv | bundle.fbundle ...] # # HOST: the headset's ssh address (default "steamos@frame": an entry named frame in ~/.ssh/config, or the # headset's name on the network; an IP works too, e.g. steamos@192.168.1.50). Environments (.fenv) given are -# copied into its selector as well. --uninstall gives the environment back to SteamVR (files stay). +# copied into its selector as well; the first .fenv is the one shown. Bundles (.fbundle) are copied as they are and +# the headset's Environments panel unpacks them into their environments (they never change the one shown). +# --uninstall gives the environment back to SteamVR (files stay). set -eu HERE=$(cd "$(dirname "$0")" && pwd) PAYLOAD="$HERE/payload" @@ -17,7 +19,7 @@ while [ $# -gt 0 ]; do case "$1" in --host) HOST=$2; shift 2 ;; --uninstall) MODE=uninstall; shift ;; - -h|--help) sed -n '2,10p' "$0"; exit 0 ;; + -h|--help) sed -n '2,11p' "$0"; exit 0 ;; *) ENVS="$ENVS $1"; shift ;; esac @@ -44,8 +46,8 @@ if [ "$MODE" = install ]; then scp -q "$f" "$HOST:frame-home/envs/$name.part" && ssh "$HOST" "mv -f frame-home/envs/$name.part frame-home/envs/$name" done ssh "$HOST" "sed -i 's/\r\$//' frame-home/.install/frame/install.sh; sh frame-home/.install/frame/install.sh" - # the first environment given is the one shown - FIRST=$(printf '%s\n' "$ENVS" | sed -n '/./{p;q;}') + # the first environment given is the one shown (a bundle's are only known once the headset unpacks it: skipped) + FIRST=$(printf '%s\n' "$ENVS" | grep -i '\.fenv$' | sed -n '1p') if [ -n "$FIRST" ] && [ -f "$FIRST" ]; then name=$(basename "$FIRST" | tr 'A-Z ' 'a-z-' | tr -cd 'a-z0-9._-') ssh "$HOST" "printf '%s\n' '$name' > frame-home/current" diff --git a/src/fenv_bundle.h b/src/fenv_bundle.h new file mode 100644 index 0000000..19a3106 --- /dev/null +++ b/src/fenv_bundle.h @@ -0,0 +1,375 @@ +#pragma once +// .fbundle — several Frame environments in one file, to share or install together. Each stays its own environment: +// a bundle is only the carrier. It is a ZIP holding bundle.json and the .fenv files as they are: +// +// { "format": "fbundle", "version": 1, "name": "Meta homes", +// "environments": [ { "file": "environments/haven.fenv", "name": "Haven + Calming", "size": 283115520 } ] } +// +// The .fenv members are STORED, not recompressed (their content is compressed already), and copied file to zip and +// zip to file in pieces, so a bundle of any size never passes through memory. A bundle of 4 GB or more is written as +// zip64 from the start (miniz only switches before a file that starts past 4 GB, not for one that ends past it, and +// then can't finish the archive). A member being stored means its bytes sit in the bundle as one run, so a member's +// own fenv.json and thumbnail are read in place, without unpacking anything. +// +// The Steam Frame's switcher unpacks a bundle dropped into ~/frame-home/envs into its environments (switcher.cpp +// unpackBundles); Frame Studio and fenv-tool write, list and unpack them. +#include "fenv.h" + +#include +#include + +struct FenvBundleEntry { + std::string file; // member path in the bundle: "environments/.fenv" + std::string name; // the environment's display name (its fenv.json "name") + uint64_t size = 0; // bytes +}; + +struct FenvBundleManifest { + std::string name; + std::vector environments; +}; + +// progress(bytesDone, bytesTotal), called as the environments' bytes are copied. +using FenvBundleProgress = std::function; + +inline bool isFenvBundlePath(const std::string& p) { + return p.size() > 8 && (p.compare(p.size() - 8, 8, ".fbundle") == 0 || p.compare(p.size() - 8, 8, ".FBUNDLE") == 0); +} + +namespace fenv_bundle_detail { + +constexpr const char* kManifestName = "bundle.json"; + +inline std::string manifestJson(const FenvBundleManifest& m) { + std::string j = "{\n \"format\": \"fbundle\",\n \"version\": 1,\n \"name\": " + fenv_detail::quote(m.name) + ",\n \"environments\": ["; + for (size_t i = 0; i < m.environments.size(); ++i) { + const FenvBundleEntry& e = m.environments[i]; + j += std::string(i ? "," : "") + "\n { \"file\": " + fenv_detail::quote(e.file) + ", \"name\": " + fenv_detail::quote(e.name) + + ", \"size\": " + std::to_string(e.size) + " }"; + } + return j + "\n ]\n}\n"; +} + +inline bool parseManifest(const std::string& json, FenvBundleManifest& m) { + try { + const tinyjson::Value doc = tinyjson::parse(json); + if (!doc.isObject() || doc["format"].asString() != "fbundle") return false; + m = FenvBundleManifest{}; + m.name = doc["name"].asString(); + for (size_t i = 0; i < doc["environments"].size(); ++i) { + const auto& v = doc["environments"][i]; + FenvBundleEntry e; + e.file = v["file"].asString(); + e.name = v["name"].asString(); + e.size = static_cast(std::max(0, v["size"].asInt())); + if (fenv_detail::safeMember(e.file) && isFenvPath(e.file)) m.environments.push_back(e); + } + return !m.environments.empty(); + } catch (...) { + return false; + } +} + +// An opened bundle: the ZIP and its manifest; closes the ZIP when it goes. +struct OpenBundle { + mz_zip_archive zip = {}; + bool open = false; + FenvBundleManifest manifest; + + bool load(const std::string& path, std::string& error) { + if (!mz_zip_reader_init_file(&zip, path.c_str(), 0)) { error = "can't open " + path; return false; } + open = true; + const std::vector j = fenv_detail::readMember(zip, kManifestName, uint64_t(1) << 20); + if (j.empty() || !parseManifest(std::string(j.begin(), j.end()), manifest)) { error = "not a Frame environment bundle (.fbundle): " + path; return false; } + return true; + } + // The member's index and stat, refusing anything that isn't stored whole (a damaged or foreign bundle). + bool member(const FenvBundleEntry& e, mz_uint& index, mz_zip_archive_file_stat& st, std::string& error) { + const int i = mz_zip_reader_locate_file(&zip, e.file.c_str(), nullptr, 0); + if (i < 0 || !mz_zip_reader_file_stat(&zip, static_cast(i), &st)) { error = "the bundle has no " + e.file; return false; } + if (st.m_method != 0 || st.m_comp_size != st.m_uncomp_size || st.m_is_encrypted) { error = e.file + " isn't stored as-is in the bundle"; return false; } + index = static_cast(i); + return true; + } + ~OpenBundle() { if (open) mz_zip_reader_end(&zip); } +}; + +// A file read at a fixed offset: one stored member of the bundle, read as a ZIP of its own. +struct FileSlice { + std::ifstream file; + uint64_t start = 0; + static size_t read(void* opaque, mz_uint64 offset, void* buffer, size_t n) { + FileSlice& s = *static_cast(opaque); + s.file.clear(); + s.file.seekg(static_cast(s.start + offset)); + s.file.read(static_cast(buffer), static_cast(n)); + return static_cast(s.file.gcount()); + } +}; + +// Where a stored member's bytes begin: after its local header (30 bytes, then its name and extra field, whose +// lengths the local header holds; they may differ from the central directory's). +inline bool memberDataOffset(const std::string& bundlePath, const mz_zip_archive_file_stat& st, uint64_t& offset) { + std::ifstream f(std::filesystem::u8path(bundlePath), std::ios::binary); + unsigned char header[30]; + f.seekg(static_cast(st.m_local_header_ofs)); + if (!f.read(reinterpret_cast(header), sizeof header)) return false; + if (header[0] != 'P' || header[1] != 'K' || header[2] != 3 || header[3] != 4) return false; + const unsigned nameLength = header[26] | (header[27] << 8), extraLength = header[28] | (header[29] << 8); + offset = st.m_local_header_ofs + 30 + nameLength + extraLength; + return true; +} + +// Reads a .fenv's own file from disk, piece by piece, as the ZIP writer asks for it; counts the bytes for progress. +struct SourceReader { + std::ifstream file; + uint64_t* done = nullptr; + uint64_t total = 0; + const FenvBundleProgress* progress = nullptr; + static size_t read(void* opaque, mz_uint64 offset, void* buffer, size_t n) { + SourceReader& r = *static_cast(opaque); + r.file.clear(); // (after the last piece: reads 0, the writer's sign that it's all there) + r.file.seekg(static_cast(offset)); + r.file.read(static_cast(buffer), static_cast(n)); + const size_t got = static_cast(r.file.gcount()); + *r.done += got; + if (*r.progress) (*r.progress)(*r.done, r.total); + return got; + } +}; + +// Where a member's bytes go as they come out of the bundle: into `out` (when open), or compared against a file already +// there (`compare` open: equal means that environment is installed already); checksummed either way but the last. +struct MemberSink { + std::ofstream out; + std::ifstream compare; + bool same = true; + mz_uint32 crc = MZ_CRC32_INIT; + uint64_t* done = nullptr; // with progress: bytes so far, of `total` + uint64_t total = 0; + const FenvBundleProgress* progress = nullptr; + static size_t write(void* opaque, mz_uint64, const void* buffer, size_t n) { + MemberSink& s = *static_cast(opaque); + if (s.compare.is_open()) { + char theirs[65536]; + for (size_t at = 0; at < n && s.same;) { + const size_t piece = std::min(n - at, sizeof theirs); + s.same = static_cast(s.compare.read(theirs, static_cast(piece))) && + std::memcmp(theirs, static_cast(buffer) + at, piece) == 0; + at += piece; + } + return s.same ? n : 0; // (a difference stops the extraction: the answer is known) + } + if (s.out.is_open() && !s.out.write(static_cast(buffer), static_cast(n))) return 0; + s.crc = static_cast(mz_crc32(s.crc, static_cast(buffer), n)); + if (s.done) { + *s.done += n; + if (s.progress && *s.progress) (*s.progress)(*s.done, s.total); + } + return n; + } +}; + +// "haven.fenv", "haven 2.fenv", "haven 3.fenv"...: the first name `taken` doesn't refuse. +inline std::string freeName(const std::string& stem, const std::function& taken) { + std::string name = stem + ".fenv"; + for (int n = 2; taken(name); ++n) name = stem + " " + std::to_string(n) + ".fenv"; + return name; +} + +inline std::string lowercase(std::string s) { + for (char& c : s) if (c >= 'A' && c <= 'Z') c = static_cast(c - 'A' + 'a'); + return s; +} + +} // namespace fenv_bundle_detail + +// Packs the .fenv files into one bundle at outPath (written as outPath + ".part", renamed when whole). Every file +// must be a readable Frame environment; two of the same file name become "name" and "name 2". `name` is the bundle's +// own name (empty: outPath's file name). +inline bool fenvBundleWrite(const std::string& outPath, const std::vector& fenvPaths, const std::string& name, std::string& error, + const FenvBundleProgress& progress = {}) { + namespace fs = std::filesystem; + using namespace fenv_bundle_detail; + if (fenvPaths.empty()) { error = "no environments to bundle"; return false; } + FenvBundleManifest manifest; + manifest.name = name.empty() ? fs::u8path(outPath).stem().u8string() : name; + std::vector takenLowercase; // member names compared without case: they must unpack apart on Windows too + uint64_t total = 0; + for (const std::string& path : fenvPaths) { + std::error_code ec; + FenvManifest m; + if (!isFenvPath(path)) { error = "not a Frame environment (.fenv): " + path; return false; } + const uint64_t size = fs::file_size(fs::u8path(path), ec); + if (ec || !fenvReadManifest(path, m)) { error = "can't read the environment " + path; return false; } + FenvBundleEntry e; + const std::string member = freeName(fs::u8path(path).stem().u8string(), [&](const std::string& n) { + return std::find(takenLowercase.begin(), takenLowercase.end(), lowercase(n)) != takenLowercase.end(); + }); + takenLowercase.push_back(lowercase(member)); + e.file = "environments/" + member; + e.name = m.name.empty() ? fs::u8path(path).stem().u8string() : m.name; + e.size = size; + total += size; + manifest.environments.push_back(e); + } + const std::string tmp = outPath + ".part", json = manifestJson(manifest); + const uint64_t headroom = json.size() + (uint64_t(1) << 20); // names, headers, the central directory + const mz_uint zip64 = total + headroom >= 0xFFFFFFFFull ? MZ_ZIP_FLAG_WRITE_ZIP64 : 0; + mz_zip_archive zip = {}; + if (!mz_zip_writer_init_file_v2(&zip, tmp.c_str(), 0, zip64)) { error = "can't write " + outPath; return false; } + bool ok = mz_zip_writer_add_mem(&zip, kManifestName, json.data(), json.size(), MZ_DEFAULT_COMPRESSION); + if (!ok) error = "couldn't write " + outPath; + uint64_t done = 0; + const MZ_TIME_T now = std::time(nullptr); + for (size_t i = 0; ok && i < fenvPaths.size(); ++i) { + const FenvBundleEntry& e = manifest.environments[i]; + SourceReader reader; + reader.file.open(fs::u8path(fenvPaths[i]), std::ios::binary); + reader.done = &done; + reader.total = total; + reader.progress = &progress; + const uint64_t before = done; + ok = reader.file.is_open() && + mz_zip_writer_add_read_buf_callback(&zip, e.file.c_str(), &SourceReader::read, &reader, e.size, &now, nullptr, 0, MZ_NO_COMPRESSION, + nullptr, 0, nullptr, 0) && + done - before == e.size; // (a file that changed size while it was copied) + if (!ok) error = "couldn't add " + fenvPaths[i] + " to the bundle"; + } + ok = ok && mz_zip_writer_finalize_archive(&zip); + if (!ok && error.empty()) error = "couldn't finish " + outPath; + if (!ok) error += std::string(" (") + mz_zip_get_error_string(mz_zip_get_last_error(&zip)) + ")"; + mz_zip_writer_end(&zip); + std::error_code ec; + if (!ok) { fs::remove(fs::u8path(tmp), ec); return false; } + fs::rename(fs::u8path(tmp), fs::u8path(outPath), ec); + if (ec) { fs::remove(fs::u8path(outPath), ec); fs::rename(fs::u8path(tmp), fs::u8path(outPath), ec); } + if (ec) { fs::remove(fs::u8path(tmp), ec); error = "couldn't replace " + outPath + ": " + ec.message(); return false; } + return true; +} + +// Reads bundle.json alone. +inline bool fenvBundleReadManifest(const std::string& path, FenvBundleManifest& manifest, std::string* error = nullptr) { + fenv_bundle_detail::OpenBundle bundle; + std::string why; + if (!bundle.load(path, why)) { if (error) *error = why; return false; } + manifest = bundle.manifest; + return true; +} + +// One environment's own manifest (and, if asked, its thumbnail bytes) read in place inside the bundle. `memberFile` +// is its "file" from the bundle's manifest. +inline bool fenvBundleReadEnvironment(const std::string& bundlePath, const std::string& memberFile, FenvManifest& manifest, + std::vector* thumbnail = nullptr, std::string* error = nullptr) { + using namespace fenv_bundle_detail; + std::string why; + const auto fail = [&](const std::string& w) { if (error) *error = w; return false; }; + OpenBundle bundle; + if (!bundle.load(bundlePath, why)) return fail(why); + const auto it = std::find_if(bundle.manifest.environments.begin(), bundle.manifest.environments.end(), + [&](const FenvBundleEntry& e) { return e.file == memberFile; }); + if (it == bundle.manifest.environments.end()) return fail("the bundle lists no " + memberFile); + mz_uint index = 0; + mz_zip_archive_file_stat st; + if (!bundle.member(*it, index, st, why)) return fail(why); + FileSlice slice; + if (!memberDataOffset(bundlePath, st, slice.start)) return fail("damaged bundle: " + bundlePath); + slice.file.open(std::filesystem::u8path(bundlePath), std::ios::binary); + mz_zip_archive inner = {}; + inner.m_pRead = &FileSlice::read; + inner.m_pIO_opaque = &slice; + if (!slice.file.is_open() || !mz_zip_reader_init(&inner, st.m_uncomp_size, 0)) return fail(memberFile + " in the bundle isn't a Frame environment"); + const std::vector json = fenv_detail::readMember(inner, "fenv.json"); + bool ok = !json.empty() && fenvParseManifest(std::string(json.begin(), json.end()), manifest); + if (ok && thumbnail) *thumbnail = manifest.thumbnail.empty() ? std::vector() : fenv_detail::readMember(inner, manifest.thumbnail.c_str()); + mz_zip_reader_end(&inner); + return ok || fail(memberFile + " in the bundle isn't a Frame environment"); +} + +// Unpacks the bundle's environments into outDir as separate .fenv files (`onlyMember`: just that one, by its "file"). +// Each is written as ".part" and renamed when whole and checked against the bundle's checksum. A file already +// there of the same name is never replaced: if it is the same environment byte for byte it counts as unpacked, +// otherwise this one becomes " 2.fenv". `unpacked` gets the environments' paths, in the bundle's order. +inline bool fenvBundleExtract(const std::string& bundlePath, const std::string& outDir, std::vector& unpacked, std::string& error, + const std::string& onlyMember = {}, const FenvBundleProgress& progress = {}) { + namespace fs = std::filesystem; + using namespace fenv_bundle_detail; + unpacked.clear(); + OpenBundle bundle; + if (!bundle.load(bundlePath, error)) return false; + std::vector wanted; + for (const FenvBundleEntry& e : bundle.manifest.environments) + if (onlyMember.empty() || e.file == onlyMember) wanted.push_back(e); + if (wanted.empty()) { error = "the bundle lists no " + onlyMember; return false; } + uint64_t total = 0, done = 0; + for (const FenvBundleEntry& e : wanted) total += e.size; + std::error_code ec; + const fs::path dir = fs::u8path(outDir); + fs::create_directories(dir, ec); + for (const FenvBundleEntry& e : wanted) { + mz_uint index = 0; + mz_zip_archive_file_stat st; + if (!bundle.member(e, index, st, error)) return false; + const std::string stem = fs::u8path(e.file).stem().u8string(); + if (stem.empty() || stem[0] == '.') { error = "the bundle names an environment \"" + e.file + "\""; return false; } + // The first free name, or the file that already is this environment. + std::string existing; + const std::string name = freeName(stem, [&](const std::string& candidate) { + const fs::path p = dir / fs::u8path(candidate); + if (!fs::exists(p, ec)) return false; + if (existing.empty() && fs::is_regular_file(p, ec) && fs::file_size(p, ec) == st.m_uncomp_size) { + MemberSink check; + check.compare.open(p, std::ios::binary); + if (check.compare.is_open() && mz_zip_reader_extract_to_callback(&bundle.zip, index, &MemberSink::write, &check, 0) && check.same) { + existing = candidate; + return false; // stop here: this one is it + } + } + return true; + }); + const fs::path target = dir / fs::u8path(name); + if (!existing.empty()) { + done += e.size; + if (progress) progress(done, total); + unpacked.push_back(target.u8string()); + continue; + } + const fs::path part = dir / fs::u8path(name + ".part"); + MemberSink sink; + sink.out.open(part, std::ios::binary | std::ios::trunc); + sink.done = &done; + sink.total = total; + sink.progress = &progress; + bool ok = sink.out.is_open() && mz_zip_reader_extract_to_callback(&bundle.zip, index, &MemberSink::write, &sink, 0); + sink.out.close(); + ok = ok && static_cast(sink.out) && sink.crc == st.m_crc32; + if (!ok) { fs::remove(part, ec); error = "couldn't unpack " + e.file + " from " + bundlePath + " (damaged, or the disk is full)"; return false; } + fs::rename(part, target, ec); + if (ec) { fs::remove(part, ec); error = "couldn't write " + target.u8string() + ": " + ec.message(); return false; } + unpacked.push_back(target.u8string()); + } + return true; +} + +// Checks a bundle without unpacking it: its manifest, each environment present, stored whole and of the size listed, +// with a readable fenv.json of its own; with `checkData`, each one's bytes against its checksum too (reads them all). +inline bool fenvBundleValidate(const std::string& path, std::string& error, bool checkData = true) { + using namespace fenv_bundle_detail; + OpenBundle bundle; + if (!bundle.load(path, error)) return false; + for (const FenvBundleEntry& e : bundle.manifest.environments) { + mz_uint index = 0; + mz_zip_archive_file_stat st; + if (!bundle.member(e, index, st, error)) return false; + if (st.m_uncomp_size != e.size) { error = e.file + " is " + std::to_string(st.m_uncomp_size) + " bytes, the bundle says " + std::to_string(e.size); return false; } + FenvManifest m; + if (!fenvBundleReadEnvironment(path, e.file, m, nullptr, &error)) return false; + if (!checkData) continue; + MemberSink sum; // (nothing open: checksums only) + if (!mz_zip_reader_extract_to_callback(&bundle.zip, index, &MemberSink::write, &sum, 0) || sum.crc != st.m_crc32) { + error = e.file + " in the bundle is damaged (checksum)"; + return false; + } + } + return true; +} diff --git a/src/fenv_tool.cpp b/src/fenv_tool.cpp index a4d1dd7..66b6493 100644 --- a/src/fenv_tool.cpp +++ b/src/fenv_tool.cpp @@ -6,7 +6,11 @@ // inputs: a home APK (HSL, Clay .opa, glTF .ovrscene), a footprint + its vista, models (.glb .gltf .fbx // .obj). Every home is baked (scene.bin: opens several times faster; a Clay / glTF home's animation goes in as // data, anim_bake.h). --keep-apks stores the source APKs as well. -// fenv-tool info file.fenv +// fenv-tool info a bundle: its environments, their names and sizes +// fenv-tool bundle out.fbundle [--name NAME] file.fenv... several environments in one file, to share or install +// together (fenv_bundle.h); each stays its own environment +// fenv-tool unbundle file.fbundle outdir the bundle's environments as .fenv files (one there already is kept: +// the same one is skipped, a different one makes this "name 2.fenv") // fenv-tool convert outdir ... one .fenv per input ("a.apk+b.apk" pairs a footprint with a vista) // fenv-tool upgrade file.fenv... re-bake a .fenv's scene with this build, everything else kept (one that // loads from its homes is baked: its APKs, models and world.json give way) @@ -16,6 +20,7 @@ // the world's best settings on this headset (walked by frame-home --bench, paced at the display rate), stored in // the .fenv (or .settings.json); prints them as JSON last #include "fenv.h" +#include "fenv_bundle.h" #include "home_report.h" #include "tex_mips.h" #include "scene_bake.h" @@ -128,7 +133,46 @@ int pack(const std::string& out, const std::vector& inputs, std::st return 0; } +// A bundle's environments: name, member file and size, each read from the bundle in place. +int bundleInfo(const std::string& path) { + FenvBundleManifest bundle; + std::string err; + if (!fenvBundleReadManifest(path, bundle, &err)) { std::fprintf(stderr, "fenv-tool: %s\n", err.c_str()); return 1; } + std::printf("%s\n bundle: %s, %zu environment%s\n", path.c_str(), bundle.name.c_str(), bundle.environments.size(), bundle.environments.size() == 1 ? "" : "s"); + int rc = 0; + for (const FenvBundleEntry& e : bundle.environments) { + FenvManifest m; + std::vector thumb; + const bool readable = fenvBundleReadEnvironment(path, e.file, m, &thumb, &err); + std::printf(" %-36s %8.1f MB %s%s\n", e.name.c_str(), e.size / 1048576.0, e.file.c_str(), + !readable ? " (damaged)" : thumb.empty() ? "" : " (thumbnail)"); + if (!readable) rc = 1; + } + return rc; +} + +int bundle(const std::string& out, const std::vector& inputs, const std::string& name) { + std::string err; + int shown = -10; + const bool ok = fenvBundleWrite(out, inputs, name, err, [&](uint64_t done, uint64_t total) { + const int percent = total ? static_cast(done * 100 / total) : 100; + if (percent / 10 != shown / 10) { shown = percent; std::fprintf(stderr, "\r %d%%", percent); } + }); + std::fprintf(stderr, "\r \r"); + if (!ok) { std::fprintf(stderr, "fenv-tool: %s\n", err.c_str()); return 1; } + return bundleInfo(out); +} + +int unbundle(const std::string& path, const std::string& outDir) { + std::vector unpacked; + std::string err; + if (!fenvBundleExtract(path, outDir, unpacked, err)) { std::fprintf(stderr, "fenv-tool: %s\n", err.c_str()); return 1; } + for (const std::string& f : unpacked) std::printf("%s\n", f.c_str()); + return 0; +} + int info(const std::string& path) { + if (isFenvBundlePath(path)) return bundleInfo(path); FenvManifest m; std::vector thumb; if (!fenvReadManifest(path, m, &thumb)) { std::fprintf(stderr, "fenv-tool: not a .fenv: %s\n", path.c_str()); return 1; } @@ -392,6 +436,17 @@ int tune(const std::string& home, const std::string& benchArg, int seconds, doub int main(int argc, char** argv) { if (argc >= 3 && !std::strcmp(argv[1], "info")) return info(argv[2]); + if (argc >= 4 && !std::strcmp(argv[1], "bundle")) { + std::string name; + std::vector inputs; + for (int i = 3; i < argc; ++i) { + if (!std::strcmp(argv[i], "--name") && i + 1 < argc) name = argv[++i]; + else inputs.push_back(argv[i]); + } + if (!isFenvBundlePath(argv[2])) { std::fprintf(stderr, "fenv-tool bundle: the bundle's name must end in .fbundle: %s\n", argv[2]); return 2; } + return bundle(argv[2], inputs, name); + } + if (argc >= 4 && !std::strcmp(argv[1], "unbundle")) return unbundle(argv[2], argv[3]); if (argc >= 3 && !std::strcmp(argv[1], "upgrade")) { for (int i = 2; i < argc; ++i) if (!std::strcmp(argv[i], "--no-optimize")) g_optimize = false; int rc = 0; @@ -467,7 +522,10 @@ int main(int argc, char** argv) { #endif std::fprintf(stderr, "usage:\n fenv-tool pack out.fenv [--name NAME] [--edits edits.json] [--thumb thumb.png] [--keep-apks] [--fast] ...\n" - " fenv-tool info file.fenv\n fenv-tool convert outdir ...\n" + " fenv-tool info \n" + " fenv-tool bundle out.fbundle [--name NAME] file.fenv... several environments in one file\n" + " fenv-tool unbundle file.fbundle outdir its environments as .fenv files\n" + " fenv-tool convert outdir ...\n" " fenv-tool upgrade file.fenv... re-bake with this build (keeps world, edits, sound, picture)\n" " fenv-tool report [--json] ... what it costs and what's wrong with it\n" " fenv-tool settings [JSON | --clear] a world's own settings, shown or set\n" diff --git a/src/switcher.cpp b/src/switcher.cpp index 2e98eda..1f2f9be 100644 --- a/src/switcher.cpp +++ b/src/switcher.cpp @@ -1,6 +1,7 @@ // frame-switcher — the Steam Frame's environment switcher: a SteamVR dashboard panel listing every environment // in ~/frame-home/envs (Frame environments .fenv, home APKs, and footprint + vista pairs from pairs.txt) with -// thumbnails. Point and click one: ~/frame-home/current changes and Frame Home switches to it live. +// thumbnails. Point and click one: ~/frame-home/current changes and Frame Home switches to it live. A bundle of +// environments (.fbundle) put there is unpacked into its .fenv files, which the panel then lists one by one. // // frame-switcher the dashboard panel (SteamVR starts it: see --install) // frame-switcher --install register with SteamVR and start it with SteamVR from now on @@ -8,8 +9,10 @@ // frame-switcher --set NAME switch (NAME: the file name, or a number from --list) // frame-switcher --thumbs render missing thumbnails (frame-home --snapshot) into envs/.thumbs // frame-switcher --delete NAME delete an environment (like the panel's delete button; NAME as for --set) +// frame-switcher --unpack-bundles unpack the bundles (.fbundle) in envs now (the panel does it by itself) // frame-switcher --test-delete DIR self-check of deleting, in a fake root made in DIR (must not exist yet) #include "fenv.h" +#include "fenv_bundle.h" #include "overlay_texture.h" #include "overlays.h" #include "perf_settings.h" @@ -36,6 +39,7 @@ #include #include #include +#include #include #include #include @@ -1433,8 +1437,35 @@ std::string statsLine() { volatile sig_atomic_t g_quit = 0; +// Bundles (.fbundle, fenv_bundle.h) in envs/ become the environments they carry: each is unpacked beside them as +// its own .fenv (an environment already there is kept: the same one is skipped, a different one of the same name +// makes this one " 2.fenv"), then the bundle is deleted, as everything it holds is in envs/ now. One that +// can't be unpacked (damaged, or still being copied in) stays and is tried again once its size or time changes. +void unpackBundles() { + static std::set failed; // "||