Frame Home as the real SteamVR environment on a PC (Windows too), not a home app

On the Steam Frame a layer inside SteamVR's compositor draws the home where it would draw Aurora, so the home is the
environment itself: at boot, during game loads, behind the Steam menu, with the controllers and passthrough, and it
steps aside for a game the instant it opens. On a PC this was faked with SteamVR's home-app slot and a flat 360 photo
in the gaps, which launched and handed off like an app and showed a picture, not the live home. This does it properly
on any PC that runs SteamVR, the Frame's way:

- Windows: the compositor is Direct3D 11, so the layer is a proxy d3d11.dll beside the compositor (layer/
  frame_home_layer_d3d11.cpp, built from the .def that forwards every other export to the real d3d11.dll, kept as
  fh_d3d11real.dll). It wraps the compositor's device, finds Aurora's draws by their shader bytecode and draws the
  home over the sky, leaving the floor and stars off; everything else passes straight through. Pure pass-through in
  any process that isn't vrcompositor.
- The home's frames cross from Frame Home (Vulkan) to the compositor (D3D11): the layer owns kSlots shared textures
  and hands Frame Home their NT handles over a named pipe; Frame Home imports them (VK_KHR_external_memory_win32) and
  renders into them. No cross-API lock — a slot is published only once the GPU has finished it and the four rotate,
  as on Linux. The shared State stays Frame Home's.
- The Environments panel runs frame-home --system and keeps it alive (one at a time, a pid file across panel
  restarts), the same systemEnv path as the Frame. The installer places/removes the proxy and marks the layer with a
  root/layer folder, in place of the old home-app registration and background pictures.
- Safe like the Vulkan layer: no home, or one stale over half a second, and the compositor draws Aurora as usual;
  the layer turns itself off after two unhealthy compositor starts, and root/layer/off turns it off by hand.

Linux keeps its Vulkan layer (unchanged). Verified without a headset on the null driver: the proxy loads and wraps
the compositor without crashing, recognises Aurora's shaders, and the frame hand-over completes (Frame Home connects,
imports the textures, renders). The compositor only draws the environment while presenting to a headset, so the final
on-headset image is the one step the null driver can't exercise.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/env_export.h

@@ -5,22 +5,176 @@
 #include "hsl_renderer.h"
 
 #ifdef _WIN32
-// A PC has no compositor layer: Frame Home is SteamVR's scene app there, so system mode never starts (init() fails).
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN
+#endif
+#include <windows.h>
+
+#include <chrono>
+#include <cstdio>
+#include <cstring>
+
+// Windows system mode: Frame Home renders the home into the compositor's own Direct3D 11 textures (see env_share.h).
+// Frame Home owns the shared State (a file mapping) and hands the layer its handle; the layer owns the textures and
+// hands Frame Home theirs. createImages() does that hand-over over the layer's named pipe and imports the textures
+// into the renderer; after that it mirrors the Linux EnvExport.
 class EnvExport {
 public:
-    bool init() { return false; }
-    envshare::State* state() { return &state_; }
-    void release(HslRenderer&) {}
-    bool connected() const { return false; }
-    bool createImages(HslRenderer&) { return false; }
-    bool connect() { return false; }
-    void poll() {}
-    int beginFrame() { return 0; }
-    void publish(int, const float[2][16], const float[2][3]) {}
-    uint32_t refusedFlags() const { return 0; }
-    double sinceSampled() const { return 1e9; }
+    ~EnvExport() {
+        closeTextures();
+        if (pipe_ != INVALID_HANDLE_VALUE) ::CloseHandle(pipe_);
+        if (state_) ::UnmapViewOfFile(state_);
+        if (stateMap_) ::CloseHandle(stateMap_);
+    }
+
+    bool init() {
+        stateMap_ = ::CreateFileMappingW(INVALID_HANDLE_VALUE, nullptr, PAGE_READWRITE, 0, sizeof(envshare::State), nullptr);
+        if (!stateMap_) return false;
+        void* p = ::MapViewOfFile(stateMap_, FILE_MAP_ALL_ACCESS, 0, 0, sizeof(envshare::State));
+        if (!p) return false;
+        state_ = new (p) envshare::State();
+        state_->show = 0;
+        return true;
+    }
+    envshare::State* state() { return state_; }
+
+    // (Re)negotiates the compositor's textures at the renderer's current size and imports them. Called at startup
+    // (where it waits a little for the layer to come up) and on every resize. True once the home can be drawn into
+    // the compositor's textures.
+    bool createImages(HslRenderer& r) {
+        r_ = &r;
+        const bool first = !everImported_;
+        const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(first ? 3 : 0);
+        do {
+            if (negotiate(r)) return true;
+            if (first) ::Sleep(200);
+        } while (first && std::chrono::steady_clock::now() < deadline);
+        return false;
+    }
+    bool connected() const { return pipe_ != INVALID_HANDLE_VALUE && imported_; }
+
+    // Keeps the hand-over alive: reconnects and re-imports after the compositor restarts (as the Linux connect()
+    // re-sends its Hello). Cheap when already connected.
+    bool connect() {
+        if (connected()) return true;
+        if (!r_) return false;
+        return negotiate(*r_);
+    }
+
+    // Is the layer still there? (the pipe breaks when the compositor goes away)
+    void poll() {
+        if (pipe_ == INVALID_HANDLE_VALUE) return;
+        DWORD avail = 0;
+        if (!::PeekNamedPipe(pipe_, nullptr, 0, nullptr, &avail, nullptr)) {
+            const DWORD e = ::GetLastError();
+            if (e == ERROR_BROKEN_PIPE || e == ERROR_PIPE_NOT_CONNECTED || e == ERROR_INVALID_HANDLE) {
+                std::fprintf(stderr, "[frame-home] the compositor went away: reconnecting\n");
+                dropPipe();
+            }
+        }
+    }
+
+    int beginFrame() { const int k = static_cast<int>(next_); next_ = (next_ + 1) % envshare::kSlots; return k; }
+
+    void publish(int k, const float viewProj[2][16], const float eyePos[2][3]) {
+        envshare::Slot& s = state_->slot[k];
+        s.frame = ++frame_;
+        s.generation = generation_;
+        std::memcpy(s.viewProj, viewProj, sizeof s.viewProj);
+        for (int e = 0; e < 2; ++e) { for (int c = 0; c < 3; ++c) s.eyePos[e][c] = eyePos[e][c]; s.eyePos[e][3] = 1; }
+        state_->published.store(frame_);
+        state_->latest.store(static_cast<uint32_t>(k), std::memory_order_release);
+        state_->publishNs.store(envshare::nowNs());
+    }
+
+    uint32_t refusedFlags() const { return 0; }   // (Windows frames carry no Vulkan create flags for the layer to refuse)
+
+    double sinceSampled() const {
+        const uint64_t s = state_->sampledNs.load();
+        return s ? (envshare::nowNs() - s) / 1e9 : 1e9;
+    }
+
+    void release(HslRenderer&) { dropPipe(); }
+
 private:
-    envshare::State state_;
+    void closeTextures() { for (HANDLE& h : texture_) if (h) { ::CloseHandle(h); h = nullptr; } }
+    void dropPipe() {
+        if (pipe_ != INVALID_HANDLE_VALUE) { ::CloseHandle(pipe_); pipe_ = INVALID_HANDLE_VALUE; }
+        imported_ = false;
+    }
+
+    // Opens the pipe if need be, asks the layer for textures at this size, imports them. Throttled to once a second
+    // while it can't connect. On success the renderer holds the imported images and connected() is true.
+    bool negotiate(HslRenderer& r) {
+        const auto now = std::chrono::steady_clock::now();
+        if (pipe_ == INVALID_HANDLE_VALUE) {
+            if (now - lastTry_ < std::chrono::seconds(1)) return false;
+            lastTry_ = now;
+            pipe_ = ::CreateFileA(envshare::kPipeName, GENERIC_READ | GENERIC_WRITE, 0, nullptr, OPEN_EXISTING, 0, nullptr);
+            if (pipe_ == INVALID_HANDLE_VALUE) {
+                static bool said = false;
+                if (!said) { said = true; std::fprintf(stderr, "[frame-home] waiting for the compositor layer (no pipe yet, error %lu)\n", ::GetLastError()); }
+                return false;
+            }
+            DWORD mode = PIPE_READMODE_MESSAGE;
+            ::SetNamedPipeHandleState(pipe_, &mode, nullptr, nullptr);
+        }
+        // The State mapping, duplicated into the layer's (pipe server's) process so it can map it.
+        DWORD serverPid = 0;
+        HANDLE serverProc = nullptr, dupState = nullptr;
+        if (!::GetNamedPipeServerProcessId(pipe_, &serverPid) ||
+            !(serverProc = ::OpenProcess(PROCESS_DUP_HANDLE, FALSE, serverPid)) ||
+            !::DuplicateHandle(::GetCurrentProcess(), stateMap_, serverProc, &dupState, 0, FALSE, DUPLICATE_SAME_ACCESS)) {
+            if (serverProc) ::CloseHandle(serverProc);
+            dropPipe();
+            return false;
+        }
+        ::CloseHandle(serverProc);
+
+        envshare::WinRequest req;
+        req.width = r.width();
+        req.height = r.height();
+        req.format = 87;   // DXGI_FORMAT_R8G8B8A8_UNORM_SRGB
+        req.pid = ::GetCurrentProcessId();
+        req.generation = generation_ + 1;
+        req.state = reinterpret_cast<uint64_t>(dupState);
+        req.stateSize = static_cast<uint32_t>(sizeof(envshare::State));
+        DWORD wrote = 0;
+        envshare::WinReply reply = {};
+        DWORD got = 0;
+        if (!::WriteFile(pipe_, &req, sizeof req, &wrote, nullptr) || wrote != sizeof req ||
+            !::ReadFile(pipe_, &reply, sizeof reply, &got, nullptr) || got != sizeof reply ||
+            reply.magic != envshare::kMagic || !reply.ok || reply.generation != req.generation) {
+            std::fprintf(stderr, "[frame-home] the layer didn't hand over textures (wrote %lu read %lu ok %u gen %u/%u)\n",
+                         wrote, got, reply.ok, reply.generation, req.generation);
+            dropPipe();
+            return false;   // (the layer now owns dupState: it closes it, or drops it when the pipe breaks)
+        }
+        // The compositor's adapter: Frame Home must be on it, or the import silently reads another GPU's memory.
+        if (const uint8_t* luid = r.deviceLUID(); std::memcmp(luid, reply.deviceLUID, 8) != 0) {
+            static bool said = false;
+            if (!said) { said = true; std::fprintf(stderr, "[frame-home] the compositor draws on a different GPU than Frame Home: the home can't be shared with it\n"); }
+            dropPipe();
+            return false;
+        }
+        closeTextures();
+        for (uint32_t i = 0; i < envshare::kSlots; ++i) texture_[i] = reinterpret_cast<HANDLE>(static_cast<uintptr_t>(reply.texture[i]));
+        if (!r.importWin32Images(envshare::kSlots, reply.texture, req.width, req.height)) { dropPipe(); return false; }
+        generation_ = req.generation;
+        imported_ = everImported_ = true;
+        std::fprintf(stderr, "[frame-home] the compositor's textures are ours now: Frame Home is the environment (%ux%u per eye)\n", req.width, req.height);
+        return true;
+    }
+
+    envshare::State* state_ = nullptr;
+    HANDLE stateMap_ = nullptr;
+    HANDLE pipe_ = INVALID_HANDLE_VALUE;
+    HANDLE texture_[envshare::kSlots] = {};
+    HslRenderer* r_ = nullptr;
+    uint32_t next_ = 0, generation_ = 0;
+    uint64_t frame_ = 0;
+    bool imported_ = false, everImported_ = false;
+    std::chrono::steady_clock::time_point lastTry_;
 };
 #else
 #include <sys/mman.h>