hands: a hand fitted as the other side can't bend its fingers backward

A left hand's skeleton could flip to a right hand's mirror image and stay
that way: with the knuckles allowed to bend 35 degrees backward, the mirror
follows half-curled fingers well enough, so curling them showed the fingers
bending back. Live on 2026-10-19 the left hand was fitted as a right one in
41% of frames, knuckles at -18 degrees.

A fit as the other side than the model's call now goes back to the model's
side as soon as its four knuckles bend backward on average (below -10
degrees), and a mirror that needs that never wins the side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -35,12 +35,14 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
 
    It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/Frametop/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
 
-   The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.4 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
+   The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.2 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
 
    The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks along the way whether to install each optional part: voice dictation (yes by default; it downloads about 320 MB of speech models), hand tracking (experimental, yes by default), in a release the [Hand Recorder](https://frametop.github.io/frametop/hand-recorder) (yes by default; it records your hands for Frametop's open hand dataset), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), in a release the Eye Recorder (yes by default, with our eye tracker; it records your eyes for Frametop's eye tracking dataset), and the Bluetooth fixes, then whether to restart SteamVR. The stable version asks only about gaze mode, the eye tracker, and the Bluetooth fixes. Hand tracking, the Hand Recorder, the eye tracker, and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
 
 After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings. SteamOS's own single-screen desktop is still there, as Native Desktop in the same list.
 
+The first time the desktop starts, Frametop Tour opens: a page at a time on the screens, the keyboard shortcuts, eye and hand tracking, spotwatch, floating windows, pins, profiles, and pausing for VR games, with buttons to try each one. Closing it, skipping it, or finishing it keeps it from opening again. It's in the application menu, in spotwatch, in SteamVR's Launch a program list, and in Frametop Input Settings' menu for whenever you want it again, and `tour/ft-tour --reset` has it open at the desktop's next start.
+
 If you work in the desktop for long stretches, or leave the headset on a stand, open Frametop Display Settings → Power. Turn on Stay awake while plugged in: by default Steam puts the Frame to sleep after an hour without input, even while it charges. And choose when the displays turn off while the headset isn't used. SteamVR turns them off a few seconds after you take the headset off, but a stand or mount that covers the proximity sensor inside it makes the headset seem worn, and its displays stay on all night.
 
 ### Add a Bluetooth mouse or keyboard
@@ -68,7 +70,7 @@ If you work in the desktop for long stretches, or leave the headset on a stand,
 | Meta+Shift+H in the desktop | Hides or shows all screens (also in the menu as Hide/Show Screens, and mappable). The Visibility & pins tab of Frametop Display Settings can instead show them only with the dashboard open, or while you look at your wrist |
 | Tap Meta, on any keyboard | Opens the Steam menu in the SteamVR dashboard, or closes the dashboard, wherever you are. The desktop's launcher is still on the taskbar and Alt+F1. Change it in Frametop Input Settings (Keyboard page), where any key combination or modifier tap can do a Frametop or Steam action, open a profile, or run a command of your own |
 | Switch a screen to Hidden (Frametop Display Settings, Visibility & pins → Screens shown) | Hides just that screen until you switch it back, whatever the other visibility settings say; new windows that would open on it float instead |
-| Play a VR game | Frametop pauses so the game gets the headset to itself (see [Pause for VR games](#pause-for-vr-games)): the screens hide and your controllers stay in the game. Click both thumbsticks together twice to bring Frametop back. With the automatic pause off, the screens still hide, and the SteamVR dashboard or Meta+Shift+H shows them; to keep them visible over games, change During VR games on the Visibility & pins tab |
+| Play a VR game | Frametop pauses so the game gets the headset to itself (see [Pause for VR games](#pause-for-vr-games)): the screens hide and your controllers stay in the game. Click both thumbsticks together twice to bring Frametop back (with one controller, click its thumbstick twice). With the automatic pause off, the screens still hide, and the SteamVR dashboard or Meta+Shift+H shows them; to keep them visible over games, change During VR games on the Visibility & pins tab |
 
 Restarting the desktop (Restart desktop in Frametop Display Settings) closes its windows, but background work you started in it, such as servers, tmux sessions, or builds, keeps running.
 
@@ -92,6 +94,7 @@ Any desktop window can float in VR as a panel of its own. It stays a window of t
 | Click the float button, left of Close in a window's title bar | The same. Float in VR is also in every window's menu (Alt+F3). Apps that draw their own title bar, like Chromium and Electron apps, don't have the button: use Meta+Shift+F |
 | Right-click an app in the Application Launcher (or the taskbar) and pick Launch as Standalone | Starts the app with its window floating, where that app last floated, or in front of you the first time. From a terminal: `float/ft-float launch org.kde.dolphin`, or `float/ft-float run <command>` |
 | Meta+scroll over a floating window | Scales it up or down |
+| Set Apps from the SteamVR dashboard to Open as Frametop floating windows (Frametop Display Settings, Visibility & pins), then pick an app in the dashboard's Launch a program list | The app opens in the Frametop desktop, floating, instead of as a SteamVR window. When the desktop isn't running, Frametop is paused, or all floating windows are in use, it opens as a SteamVR window as before |
 
 ### Profiles
 
@@ -125,7 +128,7 @@ Frametop pauses while a VR game runs, so the game gets the headset's CPU and GPU
 
 | Do this | To get this |
 | --- | --- |
-| Click both thumbsticks together, twice | Pauses Frametop, or brings it back, in a game or not. You hear a short sound. The game sees the clicks too |
+| Click both thumbsticks together, twice | Pauses Frametop, or brings it back, in a game or not. You hear a short sound. The game sees the clicks too. With only one controller connected, click its thumbstick twice |
 | Start a VR game | Frametop pauses, and comes back 5 seconds after the game ends. Bring it back during the game, and it stays on until that game ends |
 | Map Pause/resume Frametop to a mouse button, key combination, or controller button | The same, from that button (Frametop Input Settings) |
 
@@ -256,6 +259,7 @@ A Plasma session runs nested inside ft-screens (`screens/`), a small Wayland com
 | `float/` | Floating windows: ft-floatd and the KWin script that float a desktop window in VR. |
 | `decoration/` | The desktop's window decoration: Breeze's look plus the float button. |
 | `input/` | The input relay (Bluetooth mice and keyboards, button maps). |
+| `keyboard/` | Frametop's keyboard: Full Keyboard for Steam Frame by TaiKeid, on the screens and in place of Steam's keyboard. See [keyboard/FRAMETOP.md](keyboard/FRAMETOP.md). |
 | `pointer/` | The 3D mouse: SteamVR driver, helper service, and a probe tool. |
 | `power/` | ft-powerd: turns the displays off while the headset isn't used. |
 | `voice/` | Frametop Voice: dictation and wake words, recognised on the headset; the input relay types the text. See [voice/README.md](voice/README.md). |
@@ -310,7 +314,11 @@ Programs built in the `dev` container link against its glibc, which is newer tha
 
 ## License
 
-MIT. See [LICENSE](LICENSE). The rendered hands' 3D models in `screens/assets/hands` (see its [README](screens/assets/hands/README.md); each folder has its licence) are from:
+MIT. See [LICENSE](LICENSE).
+
+Frametop's keyboard is [Full Keyboard for Steam Frame](https://github.com/TaiKeid/steam-frame-full-keyboard) by TaiKeid and contributors (MIT, Copyright (c) 2026 TaiKeid and contributors), in `keyboard/` with Frametop's changes. [keyboard/FRAMETOP.md](keyboard/FRAMETOP.md) lists what Frametop changed, and [keyboard/third_party/README.md](keyboard/third_party/README.md) the parts it includes.
+
+The rendered hands' 3D models in `screens/assets/hands` (see its [README](screens/assets/hands/README.md); each folder has its licence) are from:
 
 - the W3C Immersive Web Working Group's WebXR Input Profiles, `generic-hand` (MIT, Copyright (c) 2019 Amazon), the default;
 - Babylon.js's hand meshes by Patrick Ryan, https://github.com/BabylonJS/Assets (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/), unchanged;

display-settings/ft_display_settings.py

@@ -11,8 +11,10 @@ the dev container:
   - Visibility (ft-screens): when the screens show (always, only with the SteamVR
     dashboard open, while you look at a controller, or only when toggled), the wrist
     angle within which a pinned screen shows, pinning each screen to a wrist or your
-    head, and whether the desktop's notifications go to the Steam session
-    (NOTIFY_FORWARD, session/ft-notifyfwd; its state from its log).
+    head, whether the desktop's notifications go to the Steam session
+    (NOTIFY_FORWARD, session/ft-notifyfwd; its state from its log), and whether apps from
+    the SteamVR dashboard's Launch a program list open as floating windows (DASHBOARD_APPS,
+    float/ft-dashroute).
   - Layout: a preset (curved or flat, rows, distance, gap, height) or a named layout
     saved from where the screens are, with a preview; arrange now; save the current
     arrangement under a name; rename and delete; arrange automatically when the
@@ -64,6 +66,7 @@ FT_POWERD = "\0ft_powerd"
 STEAM_SLEEP_AC_DEFAULT = 3600
 SCALES = [0.75, 1.0, 1.25, 4 / 3, 1.5, 1.75, 2.0]
 NOTIFY_FORWARD = ("auto", "on", "off")  # session/ft-notifyfwd; frametop.conf's NOTIFY_FORWARD
+DASHBOARD_APPS = ("steamvr", "frametop")  # float/ft-dashroute; frametop.conf's DASHBOARD_APPS
 NOTIFYFWD_LOG = "/tmp/frametop-notifyfwd.log"  # its last line is its state (/tmp is the host's)
 ROTATIONS = [("normal", "Landscape"), ("left", "Portrait"), ("right", "Portrait (flipped)")]
 
@@ -138,7 +141,10 @@ class Backend(QObject):
             except ValueError:
                 return default
         notify = conf.get("NOTIFY_FORWARD", "auto")
-        return {"screens": num("SCREENS", 2, int), "width": num("WIDTH", 1920, int),
+        dash = conf.get("DASHBOARD_APPS", "steamvr")
+        floats = num("FLOAT_SLOTS", 16, int)
+        return {"dashboardApps": dash if dash in DASHBOARD_APPS else "steamvr", "floatSlots": floats,
+                "screens": num("SCREENS", 2, int), "width": num("WIDTH", 1920, int),
                 "height": num("HEIGHT", 1080, int), "physWidth": num("PHYS_WIDTH", 1.6),
                 "notifyForward": notify if notify in NOTIFY_FORWARD else "off"}
 
@@ -428,6 +434,22 @@ class Backend(QObject):
             write_conf_value("NOTIFY_FORWARD", mode)
             self.changed.emit()
 
+    # --- apps from the SteamVR dashboard's Launch a program list (float/ft-dashroute) ---
+    @Property(str, notify=changed)
+    def dashboardApps(self):
+        return self._conf()["dashboardApps"]
+
+    @Property(int, notify=changed)
+    def floatSlots(self):
+        return max(0, min(16, self._conf()["floatSlots"]))
+
+    @Slot(str)
+    def setDashboardApps(self, mode):
+        """ft-dashroute and ft-floatd read it as they go: no restart."""
+        if mode in DASHBOARD_APPS:
+            write_conf_value("DASHBOARD_APPS", mode)
+            self.changed.emit()
+
     # --- ft-screens: visibility and pinning ---
     @Property("QVariantMap", notify=changed)
     def visibility(self):

display-settings/main.qml

@@ -868,15 +868,21 @@ Kirigami.ApplicationWindow {
 
                     Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Screens shown" }
 
+                    // Counts as models, here and in Pinned screens: a new list rebuilds the rows, and
+                    // FormLayout puts rebuilt rows after the ones below (a report: pinning a screen
+                    // moved the wrist slider and All screens up between Screen 2 and Screen 3).
                     Repeater {
-                        model: backend.screensShown
+                        model: backend.screensShown.length
                         delegate: Controls.Switch {
-                            required property var modelData
+                            id: shownSwitch
                             required property int index
                             Kirigami.FormData.label: "Screen " + (index + 1) + ":"
-                            text: modelData ? "Shown" : "Hidden"
-                            checked: modelData
-                            onToggled: backend.setScreenShown(index, checked)
+                            text: checked ? "Shown" : "Hidden"
+                            checked: backend.screensShown[index]
+                            onToggled: {
+                                backend.setScreenShown(shownSwitch.index, checked)
+                                checked = Qt.binding(() => backend.screensShown[shownSwitch.index])
+                            }
                         }
                     }
                     Controls.Label {
@@ -890,10 +896,11 @@ Kirigami.ApplicationWindow {
                     Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Pinned screens" }
 
                     Repeater {
-                        model: backend.pins
+                        model: backend.pins.length
                         delegate: Controls.ComboBox {
-                            required property var modelData
+                            id: pinBox
                             required property int index
+                            readonly property int pinIndex: Math.max(0, ["none", "left", "right", "head"].indexOf(backend.pins[index]))
                             Kirigami.FormData.label: "Screen " + (index + 1) + ":"
                             model: [
                                 { text: "In the room", value: "none" },
@@ -903,8 +910,13 @@ Kirigami.ApplicationWindow {
                             ]
                             textRole: "text"
                             valueRole: "value"
-                            currentIndex: Math.max(0, ["none", "left", "right", "head"].indexOf(modelData))
-                            onActivated: backend.pin(String(index + 1), currentValue)
+                            currentIndex: pinIndex
+                            // pinBox.index: activated's own argument is also called index (the
+                            // chosen row), so a bare index sent "On your head" to screen 4.
+                            onActivated: {
+                                backend.pin(String(pinBox.index + 1), currentValue)
+                                currentIndex = Qt.binding(() => pinBox.pinIndex)  // what ft-screens has, if the pin failed
+                            }
                         }
                     }
                     RowLayout {
@@ -1011,6 +1023,39 @@ Kirigami.ApplicationWindow {
                             onClicked: restartDialog.open()
                         }
                     }
+
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Apps from the SteamVR dashboard" }
+
+                    Repeater {
+                        model: [
+                            { value: "steamvr", text: "Open as SteamVR windows", help: "Apps picked in the dashboard's Launch a program list open as SteamVR's own windows, as without Frametop." },
+                            { value: "frametop", text: "Open as Frametop floating windows", help: "They open in the Frametop desktop instead, floating in front of you (where that app last floated), so drag and drop, the clipboard, spotwatch and profiles work with them. When the desktop isn't running, Frametop is paused, or all floating windows are in use, they open as SteamVR windows. Steam, SteamVR and Frametop's own layout entries always stay SteamVR's." }
+                        ]
+                        delegate: ColumnLayout {
+                            required property var modelData
+                            spacing: 0
+                            Controls.RadioButton {
+                                text: modelData.text
+                                enabled: backend.floatSlots > 0
+                                checked: backend.dashboardApps === modelData.value
+                                onToggled: if (checked) backend.setDashboardApps(modelData.value)
+                            }
+                            Controls.Label {
+                                text: modelData.help
+                                opacity: 0.7
+                                font: Kirigami.Theme.smallFont
+                                leftPadding: Kirigami.Units.gridUnit * 1.6
+                                wrapMode: Text.Wrap
+                                Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                            }
+                        }
+                    }
+                    Controls.Label {
+                        visible: backend.floatSlots === 0
+                        text: "Floating windows are off (FLOAT_SLOTS=0 in ~/.config/frametop.conf), so these open in SteamVR."
+                        wrapMode: Text.Wrap
+                        Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    }
                 }
             }
         }

docs/design.md

@@ -144,7 +144,7 @@ A few overlays need special handling:
 
 Head follow is experimental and off by default. It works, but it's only lightly tested, and the feel is mostly a matter of its settings; polishing it is left open. With it on (`POINTER_FOLLOW=1`, or a mouse button mapped to Head follow on/off), the cursor rides on a reference direction, where you were facing when your head last settled, and keeps its offset from it. The mouse can put the cursor anywhere up to `POINTER_FOLLOW_REACH` (70 degrees) from the reference, a corner of your view included. While your head stays within `POINTER_LEASH_DEG` of the reference, nothing moves on its own. Once your head has been past the leash for `POINTER_LEASH_DELAY` (0.2 s, so a glance out and back doesn't count), the reference eases to where you're facing (time constant `POINTER_LEASH_RETURN`, 0.2 s), never falling further behind than the leash, and the cursor ends up back where it was in your view. Then it waits for the leash again. Two earlier versions didn't work out. Moving the reference only while your head pulled at the end of the leash left it up to the leash off after you turned back, and getting it centred again meant overshooting with your head. Easing it toward your facing all the time moved the cursor on every small head movement. A leash of 0 makes the reference your facing direction, so the cursor is locked to your view, and mouse movement shifts it within the view. Head roll is ignored, so tilting your head doesn't swing the cursor around. While the left button is held the cursor stays put in the room, so your head can't nudge a click or a drag. When you let go, it carries on from where it is instead of jumping.
 
-Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: the session now starts an AT-SPI registry, but apps still need to expose useful accessibility trees (and may need restarting), and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still up to the click; a rightLine truncated
+Gaze mode is experimental and off by default (`POINTER_GAZE=1`, the Gaze page of Frametop Input Settings, `gaze/ft-gazectl on`, or a mouse button or key combination mapped to Gaze pointer on/off). It's MAGIC pointing (Zhai, Morimoto and Ihde, 1999): the pointer goes where you look, and the mouse does the last bit. The gaze service (`gaze/ft-gazed`) sends the helper the corrected gaze at 90 Hz (from one eye while the tracker has lost the other), and while the gaze has the pointer, the cursor ray is that gaze from the eye. The pointer is aimed at the gaze each frame, not steered toward it, so nothing can pile up. An earlier try in the gaze probe steered the pointer with relative moves, and lost it when the pointer went idle or a controller had the laser. By default (`POINTER_GAZE_MOUSE_MOVE=held`, the Gaze page's Mouse movement switch) moving the mouse does nothing while the gaze has the pointer: it moves the pointer only while a button is held, as a correction. A bumped or drifting mouse can't pull the pointer off what you're looking at, and every mouse move is a correction, so the lessons aren't polluted by mouse moves to somewhere else (they used to be kept out by an 8 degree limit, which also dropped real corrections when the tracker was further off). With the gaze stale for a second, in a game, or with the headset off, the mouse moves the pointer as usual; with `free`, moving the mouse takes the pointer from the gaze. A left press while the gaze has the pointer isn't sent at once: the pointer stops where the gaze put it, you drag it onto what you meant with the button still down (panels only see it hover), and the release clicks there. Clicking at once clicked wherever the gaze was, often the wrong thing, before you could correct it. The drag is the correction. Snapping the pointer onto buttons and links is deferred: the session now starts an AT-SPI registry, but apps still need to expose useful accessibility trees (and may need restarting), and it makes Chromium and Electron apps use more CPU. A press held still for `POINTER_GAZE_HOLD` (0.5 s) becomes a real press, so drags still work: hold, then move. The right button works the same way, with the right click on the release, and pressing it while the left press is held back starts a drag where the pointer is, like Meta+J then Meta+K. That drag lasts while either button (or key) is held, so a second right press, or a second Meta+K, is free to pan and tilt the panel being dragged; with the keyboard, the head turns it. Outside games the pointer then stays: the mouse going idle doesn't release it. A moving controller still releases it, as without gaze. Gaze mode is a mouse and keyboard feature: Steam reads the Frame controllers itself, outside SteamVR's bindings, so controller clicks at the gaze kept knocking SteamVR out of laser mode (see `docs/gaze-controllers.md`). Keyboard clicks (Meta+J, Meta+K) hold the dot still in your view while the keys are down, so the head, not the mouse, does the last bit; a quick tap clicks where the dot was at the press, since the head moves as you hit the keys. The relay hides Meta from the desktop as soon as such a combination fires, because KWin takes Meta with a mouse button as a window move or resize, which swallowed the clicks. The dot shows all the time by default. With `POINTER_GAZE_DOT=moving` it shows only while the mouse moves it (`POINTER_GAZE_SHOW`), while a press is held, and as a pulse for each click; otherwise it's transparent, so the laser still lands on it. Looking more than `POINTER_GAZE_RETAKE` (5 degrees) away from it, with the mouse still, gives it back, so small eye movements around the pointer don't pull it off what you're doing. A mouse nudge before a click whose correction is within `POINTER_GAZE_NUDGE_MAX` (55 degrees, half of what the headset shows across) is sent to the gaze service as a lesson: you were looking at where you clicked when the mouse took over, so the nudge is the eye tracker's error there. Using it is what calibrates it. A one-dot check in a panel fixed to the headset tops that up when the headset goes on, when our tracker thinks it moved, and when a correction is past that limit (the tracker is far off, so a click there isn't trusted as a lesson), and the full calibration and the headset fit check run in the same panel, so everything a user does to calibrate happens in one place in the headset; the gaze probe, a fullscreen GTK app, is the development tool. The limit was 8 degrees, which dropped every correction while our tracker was 12 off. Its dots sit at known directions from the headset, so the panel needs no screen geometry. The quick check's dot takes the gaze once it has held still, so what the tracker says doesn't have to be close for the capture to work. The full calibration's and the five-dot check's dots wait for a click while you look at the dot (a left click, Enter, an index pinch or Meta+J), because a steady gaze isn't always on the dot, and take the gaze held still uLine truncated
 
 Replacing a loaded driver's files, as re-running the installer used to do, leaves SteamVR honoring the virtual controller's hand role but not its laser claim: the dashboard pointer stays unassigned until SteamVR restarts. The driver installer now leaves an unchanged driver in place.
 
@@ -178,6 +178,8 @@ Frametop's keyboard opens by itself for a text field on the desktop. The apps ru
 
 The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, as the old `@frametop_keys` could), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU into one of three shared buffers when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
 
+Full Keyboard for Steam Frame (TaiKeid's, `keyboard/`) replaces that panel when it runs: it has real layouts, input methods and themes, and it already replaces Steam's keyboard in Steam's fields. It types through gamescope (libei), which never sees the nested KWin, so in Frametop's mode (`--frametop`) its desktop typing goes through the input relay instead, which already types Frametop Voice's text there and knows where typing goes. The relay trusts it the way it trusts Voice: only this user's datagrams, and only from the address that said hello, since the control socket is abstract and anyone can send to it. Full Keyboard in turn takes word only from the relay's own name, and sends the desktop's typing only while that word is fresh, so a stopped relay never leaves keys addressed to a name another process could bind. Keys go as key presses on ft-screens' seat, as the old panel's did, so they work in every app and nothing typed leaves Frametop's processes; composed text goes through the desktop's layout like dictation's. ft-screens stays the one place that decides when a keyboard opens: while Full Keyboard says it runs, ft-screens passes its `vrkeyboard` commands to it through the relay, and opens its own panel otherwise.
+
 The Frame controllers can be mapped like mouse buttons, but they aren't input devices on the host: they reach SteamVR over the headset's own radio, and no evdev or hidraw node exists for them. So only a SteamVR client can read them. Overlay apps normally get controller input only while they have input focus, which a background helper never has. SteamVR's experimental global action set priority (`steamvr/globalActionSetPriority`, "Enable global input from overlays") lets an overlay's action set receive input anyway, and takes the inputs it binds from the scene app. Binding every button would take them all from games, so the helper's action manifest puts each button in an action set of its own, and it activates only the sets of mapped buttons. The mapping itself stays in the relay, which does the action, so mice and controllers share one list of actions.
 
 ## The desktop session
@@ -206,6 +208,14 @@ Notifications sent through the notification portal (`org.freedesktop.portal.Noti
 
 Run the isolated checks on the host with `/usr/bin/python3 session/test/test_notifyfwd.py`. Two private buses stand in for the desktop's and the user bus, with a fake Plasma (the watcher API, Inhibit) on one and a fake notification server on the other. They check forwarding with the hints' types, actions coming back, dismissed and expired notifications, replacements and closes mapping ids, `auto` skipping `simple_notif_daemon` until another server replaces it and giving Plasma its popups back when none is left, registering again after a Plasma restart, exiting with the desktop's bus, and doing nothing outside the Frametop desktop.
 
+### Title bar buttons
+
+Firefox and Chromium (and GTK apps) draw their own title bar, and they showed only a Close button in the desktop while showing minimize, maximize, and close when started from the SteamVR dashboard. Both take the buttons from GTK's decoration layout (`gtk-decoration-layout`), and GTK reads it from a different place in each. From the dashboard, the app runs in gamescope as an X11 client; GTK reads XSETTINGS there, gamescope has no XSETTINGS manager, and GTK's built-in default, `menu:minimize,maximize,close`, applies. In the desktop, the app is a Wayland client of KWin, and a Flatpak app reads the layout from the Settings portal, as `org.gnome.desktop.wm.preferences` `button-layout`. On Plasma, `kde-gtk-config` copies KWin's button order there and `xdg-desktop-portal-gtk` serves it (`kde-portals.conf` asks for Settings from `kde;gtk;`). SteamOS ships neither, and the KDE backend doesn't serve that key, so GTK fell back to GSettings in the sandbox and got GNOME's default: `appmenu:close` in Freedesktop runtimes, `menu:close` in GNOME 51. Neither the menu nor the app menu is drawn there, which left Close.
+
+`session/ft-gtksettings` is a Settings portal backend (`org.freedesktop.impl.portal.Settings`, version 1) that serves only that key. It makes the layout from kwinrc's `ButtonsOnLeft` and `ButtonsOnRight` (KDecoration's defaults `MS` and `HIAX`), mapping the letters as `kde-gtk-config` does (`M` icon, `I` minimize, `A` maximize, `X` close; the rest have no GTK button), watches kwinrc and emits `SettingChanged` when the result changes, and writes the same value to the desktop's own GSettings (its dconf database is under `~/.config/frametop`), which GTK apps outside Flatpak read. xdg-desktop-portal 1.18.4 reads backends from a single folder, `XDG_DESKTOP_PORTAL_DIR` or `/usr/share/xdg-desktop-portal/portals`, and, with the variable set, reads `portals.conf` from that folder only (the user config folders are skipped). So the session script builds one in its runtime folder: links to the system's `.portal` files, `frametop.portal`, and a copy of `kde-portals.conf` with Settings from `kde;frametop;`. The portal merges `ReadAll` across backends and takes `Read` from the first that has the key, so KDE's appearance settings come through unchanged. The helper's D-Bus service file is in `$XDG_RUNTIME_DIR/frametop/share/dbus-1/services`, a folder the script puts on `XDG_DATA_DIRS`, so only the desktop's bus can start it.
+
+Run the isolated checks on the host with `/usr/bin/python3 session/test/test_gtksettings.py`: a private bus, a temporary kwinrc, and GSettings' memory backend. They check the default and a configured order, the `ReadAll` filter, `NotFound` for other keys, the signal on a change and none on a save that changes nothing, the version property, and exiting with the bus.
+
 ### Other session behavior
 
 Steam, not systemd, suspends the Frame: after `system_idle_suspend_ac_sec` (an hour by default) without input on AC power, it logs `Switching to power state: k_ESystemPowerState_Sleep` and suspends, even while charging. It's a Steam setting (Settings → Power → When Plugged In and Idle → Sleep after), which the Stay awake while plugged in switch in Frametop Display Settings sets to Never. SteamVR's standby, which turns the displays off when the headset comes off, is separate; see below.

docs/reference.md

@@ -32,6 +32,8 @@ kwriteconfig6 --file ~/.config/frametop/kdeglobals --group KDE --key AnimationDu
 
 The desktop's notifications can show outside it, in the Steam session. Plasma there runs on the desktop's own D-Bus, so its popups only appear on the desktop's screens. `session/ft-notifyfwd` (Plasma's autostart; log: `/tmp/frametop-notifyfwd.log`) hands each one to the notification server on the user bus, the one the Steam session's apps use, for example a VR notification server that draws them as panels. Its buttons work: a click there reaches the app as if made in Plasma, and dismissing it there closes it in Plasma. While it forwards, Plasma's own popups are off (do not disturb, held by "frametop"), and Plasma's notification history still has every one. `NOTIFY_FORWARD` in `~/.config/frametop.conf` (or Notifications on the Visibility & pins tab of Frametop Display Settings) sets it: `auto` (the default) forwards only when the user bus's server isn't SteamOS's own `steam_notif_daemon`, which hands notifications to a Steam client that can't show them on the Frame, so with stock SteamOS nothing changes; `on` forwards to any server, SteamOS's too, where they're lost (it's in the file only; the settings app shows it while it's set); `off` keeps them in the desktop. Restart the desktop after a change. Notifications Flatpak apps send through the notification portal go along too. KDE's inline reply shows as a plain Reply button there; see [design.md](design.md#notifications-outside-the-desktop).
 
+Apps that draw their own title bar, like Firefox and Chromium, show the same buttons as KWin's title bars (minimize, maximize, and close unless changed in System Settings → Window Decorations). `session/ft-gtksettings` hands them KWin's button order through the Settings portal; D-Bus starts it in this desktop (log: `/tmp/frametop-gtksettings.log`). Frametop's float button has no GTK counterpart, so those apps use the key or the window menu to float (see [Floating windows](#floating-windows)), or KWin's title bar instead of their own: Use system title bar and borders in Chromium's tab strip menu, Title Bar in Firefox's Customize Toolbar. See [design.md](design.md#title-bar-buttons).
+
 Three of the system's autostart programs don't start in this desktop: Discover's update notifier (`org.kde.discover.notifier`), which starts Discover to check for updates, IBus (`ibus`), which can't reach the desktop's apps because KWin's input method is `input/ft-textinput`, and Steam (`steam`), which is already running. The session script puts copies with `Hidden=true` in `~/.config/frametop/autostart` once (marked in `frametoprc`; Steam's was added later and is hidden once on desktops that already had the other two), and skips a name you already have a file for. Delete a copy to start that program again.
 
 Settings are in two files, and Frametop Display Settings edits both. The screens (resolution, width in metres, scale, curve, which one has the taskbar) and their layout are in `~/.config/frametop-layout.json`. The backend, remote desktop, and pointer settings are in `~/.config/frametop.conf`; `session/frametop.conf.example` lists every key.
@@ -74,6 +76,13 @@ Input from the lasers reaches KWin through ft-screens' own seat. Keys come from
 
 Frametop's keyboard opens by itself when a text field on the desktop gets focus, and stays open until its Close key, a layout reset, or a mapped button closes it (or, with Keep it open off in Frametop Input Settings, until the text field loses focus). While the Steam menu (the dashboard) or Steam's own keyboard is up, it steps aside, and it comes back where it was when they're gone; one asked for meanwhile appears then. In the "only with the dashboard" visibility mode, the dashboard doesn't count. It doesn't open without a head pose (the headset in standby). It's a panel of keys (a US laptop layout, with Esc where Caps Lock would be, arrows, and a Close key) that ft-screens shows 0.7 m in front of you and below your eyes, facing you. It stays where it opened, and its grab bar (the pill along the top) moves it like a screen's. Type on it with a controller's laser or the 3D mouse. Shift, Ctrl and Alt latch for the next key, and a held key repeats. KWin starts `input/ft-textinput` as the desktop's input method, and KWin activates it whenever the focused app turns on text input for a field. It tells the relay (`textfield 1` or `0`), the relay decides by the Keyboard setting in Frametop Input Settings, and ft-screens opens the keyboard for the screen that has keyboard focus (`vrkeyboard show`, `hide`, or `toggle` from a mapped button). Its keys reach the focused screen as key presses, so it works in every app, but only apps that use Wayland text input (Qt, GTK, Firefox) open it by themselves; Chromium, Electron and X11 apps need the button. The session drops the `QT_IM_MODULE=xim` and `GTK_IM_MODULE=xim` that the gamescope session sets, or Qt and GTK apps wouldn't use Wayland text input either.
 
+Frametop's keyboard is Full Keyboard for Steam Frame, by TaiKeid (`keyboard/`, MIT; [keyboard/FRAMETOP.md](../keyboard/FRAMETOP.md)), when it runs: the user service `frametop-keyboard.service` starts it with SteamVR (`framekeyboard --frametop`). It has full layouts, languages and input methods, themes, and replaces Steam's own keyboard. The same things open and close it as the panel above: while it runs, ft-screens passes its `vrkeyboard` commands to it through the input relay, and the panel above is the fallback for when it isn't running. It shows over the screens with the dashboard closed while Frametop asked for it, and in the dashboard as always. Its typing goes where typing goes: on the desktop, the relay types it on ft-screens' seat (keys by their place, so pick the desktop's layout in it too; text the desktop's layout has no key for, like Chinese or Japanese, isn't typed there yet), and Steam and games get it through gamescope as before. A click on its panel leaves typing where it was. Its settings are in `~/.config/frametop-keyboard`.
+
+```
+keyboard/run.sh status | restart | log [lines] | install | uninstall
+FULL_KEYBOARD_TESTS=1 keyboard/build.sh     # build it with its tests
+```
+
 KWin's nested backend doesn't undo a screen's scale on pointer input, so ft-screens divides panel positions (in pixels) by it. `ft-layout` sends it each screen's scale as KWin reports it (`scale N s`) whenever it applies scales: at desktop start and from Frametop Display Settings. A scale changed only in Plasma's own display settings is put back to the Frametop layout's the next time `ft-layout` runs.
 
 ft-screens listens for datagrams on the abstract socket `@ft_screens` and replies to the sender:
@@ -81,7 +90,7 @@ ft-screens listens for datagrams on the abstract socket `@ft_screens` and replie
 ```
 place N x y z yaw pitch roll     width N metres          curve N radius|on|off
 pin N|all left|right|head [matrix]    unpin N|all        size N w h
-get N    screens    head    state    key code value    scale N s    vrkeyboard show|hide|toggle|close
+get N    screens    head    state    key code value    scale N s    vrkeyboard show|hide|toggle|close    fullkb 0|1
 visibility always|dashboard|gesture|toggle    wrist degrees    gesture left|right degrees
 hide | show | toggle    controllers always|outside_games|dashboard    ingames hide|visible    pause on|off|state
 conceal N|all    reveal N|all    concealed    cutouts on|off|state    cutouts predict on|off    cutouts lead ms
@@ -110,7 +119,8 @@ The relay also owns the volume keys, on every device that has them, the headset'
 desktops.sh relay install     # enable it (starts with the next reboot or SteamVR start)
 desktops.sh relay status | log | uninstall
 input/input-relay.py --no-grab   # try it without taking devices from SteamVR
-input/test/keys-test.py          # key combinations, modifier taps, and what reaches the desktop, against fake devices (safe next to the live relay)
+input/test/keys-test.py          # key combinations, modifier taps, dictation's and Full Keyboard's typing, and what reaches the desktop, against fake devices (safe next to the live relay)
+input/test/roles-test.py         # each node's role, Keyboard + 3D pointer included, offline
 steam/ft-steam menu              # what Open Steam menu does; ft-steam check: Steam's UI still has the calls
 ```
 
@@ -137,18 +147,18 @@ pointer/driver/install.sh probe     # devices, hand roles, activity, charging, w
 pointer/probe/build.sh              # probes: vrprobe, lasertest (who has the laser), ctlnoise (controller noise)
 ```
 
-The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_CONTROL_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`), `GAZE_EYE` (the eye bias), and `GAZE_FIT_WATCH` (the put-on check and the slip note, off by default: [gaze/README.md](../gaze/README.md), "Headset fit"). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its three again when the file changes).
+The pointer settings are in `~/.config/frametop.conf`: `POINTER_SENSITIVITY`, `POINTER_IDLE`, `POINTER_WAKE_COUNTS`, `POINTER_CONTROLLER_PICKUP`, `POINTER_DISTANCE`, `POINTER_CURSOR_DEG`, `POINTER_ORIGIN_FRACTION`, `POINTER_ORIGIN_MARGIN`, `POINTER_SCENE_RADIUS`, `POINTER_EDGE_REACH`, `POINTER_LASER_WIDTH`, `POINTER_IGNORE`, the head follow settings `POINTER_FOLLOW`, `POINTER_LEASH_DEG`, `POINTER_LEASH_DELAY`, `POINTER_LEASH_RETURN`, and `POINTER_FOLLOW_REACH`, and the gaze mode settings `POINTER_GAZE`, `POINTER_GAZE_RETAKE`, `POINTER_GAZE_NUDGE_MAX`, `POINTER_GAZE_HOLD`, `POINTER_GAZE_CONTROL_HOLD`, `POINTER_GAZE_DOT`, `POINTER_GAZE_SHOW`, `POINTER_GAZE_MOUSE`, `POINTER_GAZE_MOUSE_MOVE`, and the keyboard clicks' `POINTER_HEAD_DEADZONE` and `POINTER_KEY_TAP`, and the gaze service's `GAZE_TRACKER` (`auto`, the default: our own eye tracker when it's installed, else SteamVR's; or `own` or `steam`), `GAZE_EYE` (the eye bias), `GAZE_FIT_WATCH` (the put-on check and the slip note, off by default: [gaze/README.md](../gaze/README.md), "Headset fit"), and `GAZE_FUSION` (our tracker's gaze mixed with SteamVR's, off by default: [gaze/README.md](../gaze/README.md), "Gaze pointer"), and our eye tracker's `EYES_SIZE_GUARD` (`both`, the default, `long`, or `off`: [gaze/README.md](../gaze/README.md), "Our own eye tracker"). The example config explains each. Frametop Input Settings changes them live; after editing the file by hand, restart the relay or the helper (the gaze service reads its four again when the file changes; ft-eyes reads `EYES_SIZE_GUARD` when it starts).
 
 ## Frametop Input Settings
 
-A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has ten pages:
+A Kirigami app with a Python backend, in the Plasma menu under Settings. It runs in the `dev` container and talks to the relay over its control socket, `@frametop_relay`. It has ten pages, and Frametop Tour at the end of its menu:
 
-- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role. Forget drops everything saved for a device.
+- Devices lists every USB and Bluetooth mouse and keyboard, with a light that flashes when the device is used. Each device gets a role: 3D pointer (grabbed, drives the pointer; the default for anything with a mouse), Pass through (grabbed only while typing goes to the desktop; the default for keyboards, whose key combinations work everywhere), Keyboard + 3D pointer (for a keyboard with a touchpad, offered for devices that have both: its touchpad drives the pointer and its keys pass through, as if they were two devices), or Ignore. A device is identified by its Bluetooth address, or its USB ids and name, so all of its input nodes share one role, except with Keyboard + 3D pointer. That one is never a default: the Swiftpoint Z3 also has a mouse and a keyboard node, and its keyboard node is its extra buttons. Forget drops everything saved for a device.
 - Buttons maps a pointer device's buttons. Choose Capture a button, press the button or key, then pick an action: a click, back, scroll, toggle dashboard, recenter, pointer on or off, head follow on or off, gaze pointer on or off, gaze precision, gaze drag, gaze quick check, faster or slower, reset the screen layout, hide or show the screens, open or close the keyboard, float a window in VR or put it back, put all floating windows back, pause or resume Frametop ([Pausing for VR games](#pausing-for-vr-games)), Open profile NAME (one per profile, [profiles.md](profiles.md)), pass the key through, or nothing. Devices with saved mappings are listed even while they're asleep.
 - Controllers maps the Frame controllers' buttons (every button but the system button) to the same actions, except passing a key through and the gaze actions: gaze mode is a mouse and keyboard feature ([gaze-controllers.md](gaze-controllers.md)). Capture a button and press it on a controller, or pick it from the list. The controllers aren't input devices on the host; only SteamVR sees them. So the pointer helper reads them with SteamVR input (`pointer/helper/vrbuttons.h`, `pointer/helper/actions/`) and sends presses to the relay (`vrbtn right/a 1`), which does the mapped action. The helper only takes the buttons that are mapped (the relay tells it with `vrbind`), at an overlay-global priority, and only while no game (scene application) runs, so games keep every button; with In games on (`controller_in_games`), a mapped button is taken from games too. That needs SteamVR's "Enable global input from overlays (Experimental)" setting (`steamvr/globalActionSetPriority`), which the page's Global input switch turns on and off. Mappings are saved as `controller_buttons` in `~/.config/frametop-input.json`.
 - Game optimization has the pause for VR games: its state with Pause now or Resume, whether VR games pause Frametop by themselves, the controller gesture (one or two buttons, pressed once or twice; one button always takes two presses), what happens to the desktop, and the sound. They're saved as `pause_auto`, `pause_gesture`, `pause_desktop`, and `pause_sound` in `~/.config/frametop-input.json`. See [Pausing for VR games](#pausing-for-vr-games).
 - Keyboard sets when Frametop's keyboard opens: whenever a text field is selected; only while no pass-through keyboard is connected (the default; keyboards other programs make through uinput don't count); only with a mouse or controller button mapped to Open/close keyboard; or never, which turns the button off too. Keep it open (on by default, `vr_keyboard_persist`) leaves it open after the text field loses focus. The mode is saved as `vr_keyboard` in `~/.config/frametop-input.json`, and the page lists the keyboards that count as connected. Its Key combinations section maps modifiers plus a key, or one modifier tapped on its own, on any keyboard, to any action but passing a key through or nothing, or to Run a command…: a command line the input relay runs with `sh -c` when you press the keys (`command:CMD`). The command runs as the relay's user service, outside the desktop's session, with `layout/`, `float/` and `steam/` on its `PATH` (so `ft-layout use Work` or `ft-float launch org.kde.dolphin` work as they are), and its output goes to the relay's journal. The gaze clicks (Gaze left click and Gaze right click) only go on key combinations. The defaults are a Meta tap (open the Steam menu, or close the dashboard), Meta+J (gaze left click), Meta+K (gaze right click), Meta+Shift+F (float window in VR or put it back), and Meta+Alt+Tab and Meta+Alt+Shift+Tab (spin the panels: every screen and floating window turns about your head, so the next one on the right or left comes to the front; ft-screens' `spin next|prev|<degrees>`); remove them or add others there. With Frametop Voice installed, a Right Ctrl tap starts and stops dictation too. A tap is a press with no other modifier held, then a release with no other key, mouse button, or scroll in between; a bound one sends the desktop F24 before the release, so Plasma's launcher doesn't open on it. Tapping a left modifier binds either side's; tapping a right one (Right Ctrl) binds that key alone. The combination's last key isn't typed, and the modifiers still reach the app; while typing goes to Steam rather than the desktop, keyboards aren't grabbed, so Steam or the game sees the keys too. They're saved as `key_bindings` in the same file; a file with its own list, even an empty one, gets no defaults.
-- Voice has Frametop Voice's settings (see Voice dictation): on or off, the dictation keys (set on the Keyboard page), the controller buttons and their mode, typing as you talk, sounds, and the wake word and its phrases. They're saved in `~/.config/frametop-voice.json`, and the service applies them within 2 seconds.
+- Voice has Frametop Voice's settings (see Voice dictation): on or off, the dictation keys (set on the Keyboard page) and whether they're tapped or held, the controller buttons and their mode, the language and whisper model, typing as you talk, sounds, and the wake word and its phrases. They're saved in `~/.config/frametop-voice.json`, and the service applies them within 2 seconds.
 - Pointer has a Head follow switch and sliders for the pointer settings, which apply immediately, and a Recenter button.
 - Ignored panels lists the SteamVR overlays that are showing, grouped by app (the first two parts of the overlay key, such as `sasaken.frame-perf-overlay`), from the pointer helper (`overlays`). Tick a panel, or Ignore the whole app, and the pointer passes through it to what's behind. It's for panels you only look at, like a performance overlay that follows your view. The list is saved as `POINTER_IGNORE` in `~/.config/frametop.conf`: comma-separated overlay keys, where a shell pattern like `vendor.app*` covers a whole app, including panels it opens later. The helper reloads at once. Frametop's own screens aren't listed, and entries for apps that aren't open are listed below, to remove.
 - Gaze has the gaze pointer switch (on now and from now on; a mapped button toggles it until the helper restarts), what the mouse's left button and movement do, the gaze dot, the eye tracker and eye bias, the gaze mode sliders, the gaze service's state (headset, samples per second, how often the tracker is losing each eye, the calibration, the nudges learned), and Quick check, Calibrate, and Check headset fit (each in a panel in the headset), Reload calibration, and Forget nudges. The gaze probe, a development tool, is in the page's overflow menu.
@@ -191,11 +201,12 @@ The layout is stored relative to your head when it's applied. `/run/user/<uid>/f
 
 ## Frametop Remote Displays
 
-Other computers' monitors as Frametop screens (ft-screens only), streamed from Vibepollo or Sunshine with Moonlight's protocol. Frametop Remote Displays (`remote-displays/`, also opened by the Remote displays button on Display Settings' Screens page) finds Vibepollo and Sunshine computers on the network, signs in to one with its Web UI login (it keeps a narrow API token, not the password, and pins the host's certificate), shows whether it answers, and adds its displays: its monitors, or a virtual one at any size. A computer with a Steam Link dongle on the Frame's hotspot streams over it (Connection: auto, network or dongle only); the others use the network. Each display has a Connected switch (and the host one for all of its displays), Shown, its stream's resolution, frame rate and bitrate, and its width in VR. On Windows, a monitor switches to its stream's resolution while it streams (or its closest mode with the same shape), and back when the stream ends. That takes Frametop's build `frametop-2.0.0-2` or newer; `frametop-2.0.0-1` scaled the monitor's own resolution into the stream. A disconnected display keeps its settings, and within the same desktop run, its place. In VR each one is a panel with a screen's controls, and profiles keep where they are. See [remote-displays.md](remote-displays.md).
+Other computers' monitors as Frametop screens (ft-screens only), streamed from Vibepollo or Sunshine with Moonlight's protocol. Frametop Remote Displays (`remote-displays/`, also opened by the Remote displays button on Display Settings' Screens page) finds Vibepollo and Sunshine computers on the network, signs in to one with its Web UI login (it keeps a narrow API token, not the password, and pins the host's certificate), shows whether it answers, and adds its displays: its monitors, or a virtual one at any size. A computer with a Steam Link dongle on the Frame's hotspot streams over it (Connection: auto, network or dongle only); the others use the network. Each display has a Connected switch (and the host one for all of its displays), Shown, its stream's resolution, frame rate and bitrate, and its width in VR. On Windows, a monitor switches to its stream's resolution while it streams (or its closest mode with the same shape), and back when the stream ends. That takes Frametop's build `frametop-2.0.0-2` or newer; `frametop-2.0.0-1` scaled the monitor's own resolution into the stream. A Windows host with Frametop's build can also get the Frame's microphone (Send my microphone on its card, off by default): apps on the PC record it as "Microphone (Steam Streaming Microphone)" while its displays stream. A disconnected display keeps its settings, and within the same desktop run, its place. In VR each one is a panel with a screen's controls, and profiles keep where they are. See [remote-displays.md](remote-displays.md).
 
 ```
 layout/ft-layout remote list                 # the remote displays and their streams' state
 layout/ft-layout remote connect|disconnect ID...
+layout/ft-layout remote host NAME mic=on|off # send it the Frame's microphone (streams pick it up within 3 s)
 remote-displays/install.sh                   # its menu entry
 ```
 
@@ -209,6 +220,8 @@ A desktop window can float in VR as a panel of its own, away from the screens. M
 
 `float/ft-floatd` does this. It runs inside the desktop's Plasma session (log: `/tmp/frametop-floatd.log`), loads the KWin script `float/frametop-float.js`, and moves each floating window to a spare KWin output of its own, which ft-screens shows as a panel cropped to the window. The settings are in `~/.config/frametop.conf`: `FLOAT_SLOTS` is how many windows can float at once (16, the most; 0 turns floating off; restart the desktop after a change), and `FLOAT_MARGIN` the pixels around each window on its output, so menus have room past its edges (300). Each app's last floating place, size, and scale are kept in `~/.config/frametop-float.json` by desktop file name, relative to the primary screen, so they move with the screens. `FT_FLOAT_DEBUG=1` in ft-floatd's environment logs every event from the KWin script.
 
+Apps from the SteamVR dashboard: with `DASHBOARD_APPS=frametop` (Frametop Display Settings, Visibility & pins, "Apps from the SteamVR dashboard"; the default is `steamvr`), an app picked in the dashboard's Launch a program list opens floating in the desktop, as Launch as Standalone does, not as a SteamVR window. The list is Steam's (its VR menu), and picking an app runs `SteamClient.Apps.LaunchNonSteamApp(cmdline)` in Steam's UI. `float/ft-dashroute`, which the session starts next to ft-floatd (log: `/tmp/frametop-dashroute.log`), wraps that call over Steam's DevTools port, 127.0.0.1:8080 (`float/dash-route.js`), and hands the command line to ft-floatd (`dashlaunch`). ft-floatd finds the app's desktop file by its command (Steam's command line is the Exec line without its field codes) and launches it floating, or answers that it stays SteamVR's: when the setting is off, Frametop is paused, every floating window is in use (counting launches still waiting for their window), or it's Steam, SteamVR, a desktop, or one of Frametop's layout entries (`STEAMVR_ONLY`). Then Steam's own launch runs after all. The wrapper only hands launches over while ft-dashroute's heartbeat is fresh (every 3 seconds; stale after 10), so with the desktop closed every launch is Steam's own, and ft-dashroute turns it off as it exits. The setting applies at once. `float/ft-dashroute status` shows whether the wrapper is in; `scripts/update-check.py` checks that Steam's UI still launches that way.
+
 With floating on, the session gives the desktop's windows Frametop's own decoration (`decoration/`): Breeze's look plus the float button. Apps that draw their own title bar, like Chromium and Electron apps, don't have the button. `decoration/apply.sh` puts a changed copy into the running desktop, and `decoration/apply.sh --off` goes back to Breeze until the next desktop start.
 
 ```
@@ -222,6 +235,7 @@ float/ft-float list                   # the spare outputs and what floats on the
 float/ft-float launch firefox action=new-private-window   # one of the app's own actions, floating
 float/ft-float launch org.kde.dolphin screen=2            # its first window docked on screen 2
 float/ft-float activate ID            # make a window the active one, unminimized
+float/ft-float dashlaunch CMDLINE     # what ft-dashroute sends for a dashboard launch: ok, or steamvr REASON
 ```
 
 ## Spotwatch
@@ -235,6 +249,16 @@ spotwatch/test/spotwatch-test.py   # the ranking and the search against a fake f
 spotwatch/tools/render.py OUT.png   # the panel's picture, without a headset
 ```
 
+## Frametop Tour
+
+`tour/ft-tour` (in the app menu, spotwatch, and SteamVR's Launch a program list as Frametop Tour, and in Frametop Input Settings' menu) walks through Frametop a page at a time: the screens and the 3D mouse, the key combinations the relay has now (`key_bindings`, as Frametop Input Settings shows them) and macros, gaze mode with Meta+J and Meta+K, hand tracking with and without gaze, spotwatch, floating windows (with the Apps from the SteamVR dashboard choice), pinning, profiles, pausing for VR games, and the rest. Each page has buttons that try things (spotwatch, a spin, a calibration or quick check) or open their settings page (`FT_INPUT_PAGE`, `FT_DISPLAY_PAGE`); the gaze and hands pages have a button to practise clicking on. A part that isn't installed says how to install it. A Kirigami app with a Python backend (`tour/ft_tour.py`), in the container like Frametop Input Settings; `FT_TOUR_PAGE=ID` opens it on a page.
+
+The desktop's autostart runs `ft-tour --first-run` (`frametop-tour.desktop`, written by `session/frametop-session.sh`; log: `/tmp/frametop-tour.log`), which opens it only while `~/.local/state/frametop/tour.json` is missing or from an older tour (`TOUR_VERSION`). Closing the tour, Skip the tour, and Finish write that file, unless Show this tour when the desktop starts is ticked on its last page. `ft-tour --reset` removes it.
+
+```
+tour/test/tour-test.py   # the flag, the wrapper's --first-run and --reset, and the pages for each set of installed parts (offline)
+```
+
 ## Displays off and sleep
 
 SteamVR turns the displays off a few seconds after the headset's proximity sensor says it came off. A stand or display mount that covers the sensor makes the headset seem worn, so its displays stay on, and Steam, which then counts someone as present, never puts it to sleep either.
@@ -269,7 +293,7 @@ Resuming starts again only what pausing stopped, and plays a second sound. The p
 
 Ways to pause and resume:
 
-- The controller gesture, by default both thumbsticks clicked together twice: both go down within 0.3 seconds of each other, and the second time within 0.7 seconds of the first. The relay reads it from vrserver's web socket (`input/vrws.py`), which works whatever has input focus and takes nothing from the game, so the game sees the clicks too. The Game optimization page changes it: one or two of the buttons the Controllers page lists, pressed once or twice (one button always takes two), or none.
+- The controller gesture, by default both thumbsticks clicked together twice: both go down within 0.3 seconds of each other, and the second time within 0.7 seconds of the first. Once it goes off, it's ignored for 3 seconds, so a repeat or a triple press doesn't toggle Frametop straight back. The relay reads it from vrserver's web socket (`input/vrws.py`), which works whatever has input focus and takes nothing from the game, so the game sees the clicks too. With only one controller connected, a gesture with a button on each controller is that controller's button alone, clicked twice (`one_controller`), so the default works with one controller too. The Game optimization page changes it: one or two of the buttons the Controllers page lists, pressed once or twice (one button always takes two), or none.
 - The Pause/resume Frametop action, on a mouse button, a key combination, or a controller button (outside games, like every mapped controller button).
 - VR games, with Pause while a VR game runs on (`pause_auto`, the default). The pointer helper tells the relay when a scene app starts and ends (`vrgame 1|0`, repeated every 5 seconds). A game starting pauses Frametop. A pause that starts while a game runs ends 5 seconds after the game does, unless another game starts first. Resumed during a game, Frametop stays on until that game ends. A pause that starts outside a game lasts until you resume. Flatscreen games aren't scene apps, so they don't pause it.
 - From a terminal or a script:
@@ -282,15 +306,15 @@ input/test/pause-test.py           # the gesture and the automatic pause, offlin
 input/test/pause-buttons-test.py   # a click held into a pause comes up, offline
 ```
 
-The state outlives a relay restart, in `/run/user/UID/frametop-pause.json`. A SteamVR restart while paused starts the gaze service with it, and the relay stops it again when the pointer helper comes back.
+The state outlives a relay restart, in `/run/user/UID/frametop-pause.json`. A desktop that starts while paused (a restart, an update) is told so the first time it reports to the relay, so it hides too instead of showing with the 3D mouse off. A SteamVR restart while paused starts the gaze service with it, and the relay stops it again when the pointer helper comes back.
 
 ## Voice dictation
 
-Frametop Voice (`voice/`, [voice/README.md](../voice/README.md)) types what you say where you type, recognised on the headset; nothing leaves it. `voice/ft-voiced` runs in Frametop's container as `frametop-voice.service` (installed by `voice/run.sh install`, which `install.sh` offers, yes by default), starting at login. It records with `pw-record` and transcribes live with Moonshine on the CPU; `frametop-voice-whisper.service` (whisper.cpp on the GPU) runs only while the wake word is on, for its commands. Its models, about 320 MB, are downloaded on install into `~/.local/share/frametop/voice/models` (`voice/models.sh`: fixed URLs, checked by sha256).
+Frametop Voice (`voice/`, [voice/README.md](../voice/README.md)) types what you say where you type, recognised on the headset; nothing leaves it. `voice/ft-voiced` runs in Frametop's container as `frametop-voice.service` (installed by `voice/run.sh install`, which `install.sh` offers, yes by default), starting at login. It records with `pw-record` and transcribes English live with Moonshine on the CPU; `frametop-voice-whisper.service` (whisper.cpp on the GPU, `voice/whisper-run` loads the model the settings choose) runs only while something uses it: wake word commands, or a language other than English. Its models, about 320 MB, are downloaded on install into `~/.local/share/frametop/voice/models` (`voice/models.sh`: fixed URLs, checked by sha256).
 
-The input relay does the keyboard side. Dictation's keys are key combinations with the actions `dictate` (a press starts or stops) and `dictate_hold` (records while held); with Frametop Voice installed, the default key combinations add a Right Ctrl tap (`"97": "dictate"`, Right Ctrl alone: a right-hand modifier's own code is that key's tap only). Like every tap, it doesn't count with another modifier held, or with a key, click or scroll in between. The relay sends `toggle|start|stop <source>` to `@frametop_voice`, and dictation works while Frametop is paused. ft-voiced sends its text back as `type TEXT` on `@frametop_relay`; the relay types it, one key every 4 ms on a US layout, to ft-screens while typing goes to the desktop, otherwise on its virtual keyboard for Steam and games, and holds a character back while a modifier is held. Both sockets take these only from their own user. Frame controller buttons (left d-pad down, held, by default) are read by ft-voiced itself from vrserver's web socket (`input/vrws.py`), so games still get them.
+The input relay does the keyboard side. Dictation's keys are key combinations with the actions `dictate` (a press starts or stops) and `dictate_hold` (records while held; a lone modifier held to talk is kept from the desktop until another key makes it a shortcut, which drops that recording with `cancel keyboard quiet`); with Frametop Voice installed, the default key combinations add a Right Ctrl tap (`"97": "dictate"`, Right Ctrl alone: a right-hand modifier's own code is that key's tap only). Like every tap, it doesn't count with another modifier held, or with a key, click or scroll in between. The relay sends `toggle|start|stop|cancel <source> [quiet]` to `@frametop_voice`, and dictation works while Frametop is paused. ft-voiced sends its text back as `type TEXT` on `@frametop_relay`; the relay types it, one key every 4 ms, to ft-screens on the desktop's keyboard layout (`~/.config/frametop/kxkbrc`, through libxkbcommon) while typing goes to the desktop, otherwise on its virtual keyboard with a US layout for Steam and games, and holds a character back while a modifier is held. Both sockets take these only from their own user. Frame controller buttons (left d-pad down, held, by default) are read by ft-voiced itself from vrserver's web socket (`input/vrws.py`), so games still get them.
 
-Frametop Input Settings' Voice page turns it on and off and sets the controller buttons and their mode (hold or press to toggle), typing as you talk, sounds, and the wake word (off by default) and its phrases, in `~/.config/frametop-voice.json`; the keys are on its Keyboard page. Wake word commands that run a program, and advanced settings, are edited in that file by hand (voice/README.md).
+Frametop Input Settings' Voice page turns it on and off and sets whether the dictation keys are tapped or held, the controller buttons and their mode (hold or press to toggle), the language and whisper model (one not yet there is downloaded, `voice/models.sh model NAME`), typing as you talk, sounds, and the wake word (off by default) and its phrases, in `~/.config/frametop-voice.json`; the keys are on its Keyboard page. Wake word commands that run a program, and advanced settings, are edited in that file by hand (voice/README.md).
 
 ```
 voice/build.sh && voice/run.sh install
@@ -308,7 +332,7 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke
 - Meta+J left-clicks and Meta+K right-clicks where you look. A quick tap clicks where the dot was at the press. Hold instead, and the dot stays put in your view: turn your head until it's on what you meant, and let go to click there. Held still for `POINTER_GAZE_HOLD` (0.5 s), the press becomes a real one, and your head drags. Meta+K with Meta+J held presses where the dot is now, to drag from there, and a second Meta+K during that drag (a double Meta+K) pans and tilts what you're dragging while it's held.
 - The mouse's buttons work the same way, with the mouse steering instead of your head (`POINTER_GAZE_MOUSE=precision`, the default): the right button with the left held starts a drag, and a double right click pans and tilts what you're dragging. With `POINTER_GAZE_MOUSE_MOVE=held`, the default, the mouse only corrects: while the gaze has the pointer, moving it does nothing unless a button is held. `free` lets the mouse take the pointer any time. With the gaze stale for a second, in a game, or with the headset off, the mouse works as usual.
 - A correction before a click teaches the gaze service the tracker's error there. A correction bigger than `POINTER_GAZE_NUDGE_MAX` (55 degrees) isn't learned; it opens a quick check instead.
-- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it.
+- Calibration and checks run in a panel fixed to the headset (`gaze/panel/ft-gazepanel`, which the gaze service runs), from the Gaze page: Quick check is one dot, and also opens when you put the headset on. Calibrate is three rounds of dots, dark to bright; look at each dot and left click, press Enter, pinch (index to thumb, with hand tracking) or press Meta+J to take it. Check headset fit shows, live, how well the tracker sees each eye. A right click or Meta+K closes the panel. Gaze mode on without a calibration opens Calibrate by itself, as soon as your eyes are seen. If gaze mode is on but can't follow your eyes yet (no calibration, the calibration can't open, the gaze service not running), the Gaze page says why under the Gaze pointer switch, and `gaze/ft-gazectl on` notes it.
 
 [gaze/README.md](../gaze/README.md) has the details, our own eye tracker, and the gaze probe, a development tool.
 

docs/remote-displays.md

@@ -476,6 +476,14 @@ The user decided remote displays are Frametop displays, with the same controls a
 - Disconnect, connect: the panel came back where it was, at its width. Connect in a new run with the headset off: placed from screen 1's anchor.
 - Windows rescales a window that moves between displays with different scaling, so the point you held moves on the window. A second drag from the same spot can land on the address bar instead of the title bar. That's Windows, not the routing.
 
+**The Frame's microphone on the host (2026-10-18):**
+- Send my microphone on a host's card (`"mic": true` in its layout entry, `ft-layout remote host NAME mic=on|off`, off by default) sends the Frame's microphone to it. The stream that plays the host's sound (the audio lock's holder) records PipeWire's default source, which on the Frame runs through echo cancelling, so a host gets it once however many displays stream. It reads the setting every 3 s, so switching needs no restart. Its 10 s report adds `mic N packets/s`.
+- Moonlight has no microphone. `stream/moonlight-mic.patch` adds `LiSendMicrophoneOpusData` to moonlight-common-c: packet type 0x3003 with `[sequence (2 bytes, big-endian), channel 1, flags 0]` then the Opus packet, the layout of Vibepollo's draft PR #169. It refuses (-1) when the control stream isn't encrypted, so the microphone never goes in the clear. Mono, 48 kHz, 20 ms packets, 64 kbit/s, Opus VOIP with in-band FEC.
+- ENet's throttle drops unreliable packets whenever round trips wobble. Over Wi-Fi beside three video streams it dropped 27% of them. The microphone now goes reliable, on ENet channel 0x07 of its own (moonlight-common-c uses 0x00-0x06, and 0x10 up for controllers), so a resent packet holds up only the microphone. Loss went to 0.
+- The host side is in our fork (`frametop/mic`, `src/frametop_mic*.cpp`). It takes the packets only on an encrypted control stream from a client allowed keyboard input, one client at a time (another waits and tries every 2 s). It decodes them on the control thread, hides up to 5 lost packets in a row (the last from FEC) and starts over after longer gaps, and plays them on a thread of its own. On Windows that's WASAPI into "Speakers (Steam Streaming Microphone)", which apps record as "Microphone (Steam Streaming Microphone)". The driver comes with Steam; the build installs `SteamStreamingMicrophone.inf` once if it's missing and puts the default devices back after. That capture endpoint adds +30 dB at Windows' default level (its range goes to +30), 32 times too loud, so the build plays that much quieter and follows the level when it changes.
+- On Linux the fork makes a "Frametop Microphone" source (`module-pipe-source`), but only when Sunshine runs as the desktop user. Vibepollo's packaged Linux service runs as its own user and reaches the desktop's sound only through its root session broker (`pactl`, `parec`), which has no operation for this. There it logs that once and does nothing. A broker operation would fix it, once a Linux host can test it.
+- Tested on the test PC with a 440 Hz test tone from the Frame instead of the microphone: every packet arrived encrypted, and a recorder on "Microphone (Steam Streaming Microphone)" heard the level sent (RMS 0.345 sent, 0.345 recorded), with no gaps. The Linux source was tested here on PipeWire. Not tested yet: a real voice in a real app, by ear, in the headset.
+
 **Still to do:**
 - A headset test of all of it in the live desktop:
   - the row above the screens;

docs/spotwatch.md

@@ -19,7 +19,7 @@ Spotwatch works while the Frametop desktop is running and Frametop isn't paused
 
 - Searching open windows by title, Frametop's own actions ("Reset layout"), and profiles (decision 4).
 - Moving the caret within the query. Typing and Backspace work at the end only.
-- A setting for its distance and size (`--distance` and `--width` on ft-spotpanel; 1 m and 0.56 m).
+- A setting for its distance and size (`--distance` and `--width` on ft-spotpanel; 0.75 m and 0.42 m, `spotwatch/reserve.h`).
 
 ## Decisions
 
@@ -28,7 +28,7 @@ These were settled with the user on 2026-10-10. The numbers may be cited in the
 | # | Question | Decision |
 |---|---|---|
 | 1 | The look | A rounded glass pill with a results card under it. The glass is faux, because SteamVR's overlays have alpha but no backdrop blur: a translucent dark tint, a soft sheen from the top, a bright rim fading down the edges, and a shadow outside. Real blur would mean reading the panels back every frame on the Frame's GPU |
-| 2 | Where it opens | Fixed in the room, centred where the head points (not the eye gaze), at 1 m, upright, tilted no more than 60° up or down. The card hangs below the pill, so the pill stays put as results change |
+| 2 | Where it opens | Fixed in the room, centred where the head points (not the eye gaze), at 0.75 m, upright, tilted no more than 60° up or down. The card hangs below the pill, so the pill stays put as results change. That spot is reserved (2026-10-17): the lazy susan's front slot stays at least 0.15 m behind it, and the search draws over any panel nearer than it |
 | 3 | An app with several windows | One row per app. Enter goes to its most recently active window, and Shift+Enter or a right click lists the others with the menu items |
 | 4 | What it searches | Apps and remote displays only. Hidden remote displays are left out, and disconnected ones show with Connect |
 | 5 | Disconnected remote displays | Enter connects the display (`ft-layout remote connect`), waits for its panel, then spins to it |

float/dash-route.js

@@ -0,0 +1,34 @@
+// Put into Steam's SharedJSContext by float/ft-dashroute. The SteamVR dashboard's "Launch a
+// program" list (Steam's VR menu) runs SteamClient.Apps.LaunchNonSteamApp(cmdline) when an app
+// is picked. This wraps it: while ft-dashroute is there and the setting is on, the command line
+// goes to ft-dashroute (its CDP binding, window.__ftDashLaunch), which floats the app in the
+// Frametop desktop, or, if it can't, runs Steam's own launch after all (state.original). With
+// ft-dashroute gone (no heartbeat for STALE_MS) every launch is Steam's own again.
+(() => {
+    const VERSION = 1;
+    const STALE_MS = 10000;
+    const apps = SteamClient.Apps;
+    const state = window.__ftDashRoute || (window.__ftDashRoute = {});
+    if (!state.original || !apps.LaunchNonSteamApp.__ftDashRoute) {
+        state.original = apps.LaunchNonSteamApp;  // Steam's own, not a wrapper from before
+    }
+    state.version = VERSION;
+    state.alive = Date.now();
+    state.on = true;
+    const wrapper = function (cmdline, ...rest) {
+        const s = window.__ftDashRoute;
+        if (s && s.on && Date.now() - s.alive < STALE_MS && typeof cmdline === "string"
+            && typeof window.__ftDashLaunch === "function") {
+            try {
+                window.__ftDashLaunch(JSON.stringify({ cmdline }));
+                return;
+            } catch (e) {
+                // the binding went with its DevTools session: Steam's own launch below
+            }
+        }
+        return state.original.call(apps, cmdline, ...rest);
+    };
+    wrapper.__ftDashRoute = true;
+    apps.LaunchNonSteamApp = wrapper;
+    return "dashboard apps open in Frametop (v" + VERSION + ")";
+})()

float/ft-dashroute

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""ft-dashroute: apps picked in the SteamVR dashboard's Launch a program list open as Frametop
+floating windows, not as SteamVR windows of their own (DASHBOARD_APPS=frametop in
+~/.config/frametop.conf: Frametop Display Settings, Visibility & pins tab, "Apps from the
+SteamVR dashboard"). The default, DASHBOARD_APPS=steamvr, leaves them to SteamVR.
+
+The list is in Steam's own UI (its VR menu), and picking an app runs
+SteamClient.Apps.LaunchNonSteamApp(cmdline) in Steam's SharedJSContext, which Frametop reaches
+over the Chrome DevTools Protocol on 127.0.0.1:8080 (steam/steamui.py). With the setting on,
+this puts dash-route.js there: it wraps that call and hands the command line to this program
+(a CDP binding, Runtime.bindingCalled), which asks ft-floatd ("dashlaunch CMDLINE",
+@frametop_float). ft-floatd starts the app with its first window floating, or answers why it
+should stay SteamVR's (the setting is off, Frametop is paused, it's Steam or one of Frametop's
+own commands, or every floating window is in use), and then Steam's own launch runs after all.
+The wrapper only sends launches here while this program's heartbeat (every HEARTBEAT seconds)
+is fresh, so a launch is Steam's own whenever the desktop isn't running.
+
+Runs inside the Frametop desktop's session, next to ft-floatd (its autostart entry, written by
+session/frametop-session.sh when windows can float). It looks at the setting every HEARTBEAT
+seconds, so a change applies at once; with it off, it doesn't touch Steam at all.
+
+  ft-dashroute           keep the wrapper in Steam's UI while the setting is on
+  ft-dashroute status    the setting, and whether the wrapper is in
+"""
+import json
+import os
+import re
+import select
+import signal
+import socket
+import sys
+import time
+import urllib.request
+
+HERE = os.path.dirname(os.path.realpath(__file__))
+sys.path.insert(0, os.path.join(HERE, "..", "steam"))
+sys.path.insert(0, os.path.join(HERE, "..", "layout"))
+import steamui  # noqa: E402  (Frametop's small CDP client)
+import ft_layout  # noqa: E402  (frametop.conf)
+
+PATCH = os.path.join(HERE, "dash-route.js")
+BINDING = "__ftDashLaunch"
+FLOAT = "\0" + os.environ.get("FT_FLOAT_SOCKET", "frametop_float")
+HEARTBEAT = 3.0  # seconds; the wrapper stops sending launches here after 10 s without one
+VERSION = int(re.search(r"const VERSION = (\d+);", open(PATCH).read()).group(1))
+BEAT = ("(() => { const s = window.__ftDashRoute; if (!s || s.version !== %d || !s.original) return 'missing';"
+        " s.alive = Date.now(); s.on = true; return 'ok'; })()" % VERSION)
+OFF = "(() => { const s = window.__ftDashRoute; if (s) s.on = false; return 'off'; })()"
+STATUS = ("JSON.stringify(window.__ftDashRoute ? {version: window.__ftDashRoute.version, on: window.__ftDashRoute.on,"
+          " age: Date.now() - window.__ftDashRoute.alive} : null)")
+
+
+def log(*args):
+    print(time.strftime("%H:%M:%S"), *args, flush=True)
+
+
+def wanted():
+    return ft_layout.read_conf().get("DASHBOARD_APPS", "steamvr") == "frametop"
+
+
+def shared_context():
+    """The SharedJSContext's debugger URL, or None (Steam's UI isn't up)."""
+    try:
+        with urllib.request.urlopen(f"http://127.0.0.1:{steamui.CDP_PORT}/json", timeout=3) as r:
+            targets = json.load(r)
+    except (OSError, ValueError):
+        return None
+    return next((t.get("webSocketDebuggerUrl") for t in targets if t.get("title") == "SharedJSContext"), None)
+
+
+class Cdp:
+    """One DevTools session: calls by id, and the binding's calls as they come."""
+
+    def __init__(self, url):
+        self.ws = steamui._WebSocket(url)
+        self.next_id = 1
+        self.calls = []  # command lines the wrapper sent, not handled yet
+
+    def close(self):
+        self.ws.close()
+
+    def _message(self):
+        msg = json.loads(self.ws.recv())
+        if msg.get("method") == "Runtime.bindingCalled" and msg.get("params", {}).get("name") == BINDING:
+            try:
+                cmdline = json.loads(msg["params"].get("payload") or "{}").get("cmdline")
+            except ValueError:
+                cmdline = None
+            if isinstance(cmdline, str):
+                self.calls.append(cmdline)
+        return msg
+
+    def call(self, method, **params):
+        n, self.next_id = self.next_id, self.next_id + 1
+        self.ws.send(json.dumps({"id": n, "method": method, "params": params}))
+        while True:
+            msg = self._message()
+            if msg.get("id") == n:
+                return msg.get("result", {})
+
+    def evaluate(self, expression):
+        result = self.call("Runtime.evaluate", expression=expression, returnByValue=True)
+        if "exceptionDetails" in result:
+            d = result["exceptionDetails"]
+            raise steamui.SteamUnreachable(d.get("exception", {}).get("description") or d.get("text", "error"))
+        return result.get("result", {}).get("value")
+
+    def wait(self, seconds):
+        """Read what comes for up to `seconds`, then return the launches that came."""
+        end = time.monotonic() + seconds
+        while not self.calls:
+            left = end - time.monotonic()
+            if left <= 0:
+                break
+            if self.ws.buf or select.select([self.ws.sock], [], [], left)[0]:
+                self._message()
+        calls, self.calls = self.calls, []
+        return calls
+
+
+def ask_floatd(cmdline):
+    """ft-floatd's answer to "dashlaunch CMDLINE": "ok", or "steamvr REASON"."""
+    s = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    try:
+        s.bind("")
+        s.settimeout(5)
+        s.sendto(f"dashlaunch {cmdline}".encode(), FLOAT)
+        return s.recv(4096).decode(errors="replace")
+    except OSError as e:
+        return f"steamvr ft-floatd didn't answer ({e})"
+    finally:
+        s.close()
+
+
+def inject(cdp):
+    cdp.call("Runtime.addBinding", name=BINDING)
+    with open(PATCH) as f:
+        log(f"Steam's UI: {cdp.evaluate(f.read())}")
+
+
+def route(cdp, cmdline):
+    reply = ask_floatd(cmdline)
+    if reply.startswith("ok"):
+        log(f"floating: {cmdline}")
+        return
+    log(f"SteamVR window: {cmdline} ({reply.removeprefix('steamvr ').strip()})")
+    cdp.evaluate(f"window.__ftDashRoute.original.call(SteamClient.Apps, {json.dumps(cmdline)})")
+
+
+def session(url):
+    """Keep the wrapper in this page while the setting is on; returns when it's off."""
+    cdp = Cdp(url)
+    try:
+        inject(cdp)
+        while True:
+            for cmdline in cdp.wait(HEARTBEAT):
+                route(cdp, cmdline)
+            if not wanted():
+                cdp.evaluate(OFF)
+                log("setting off: dashboard apps open in SteamVR again")
+                return
+            if cdp.evaluate(BEAT) == "missing":  # Steam's UI reloaded
+                inject(cdp)
+    finally:
+        cdp.close()
+
+
+def status():
+    print(f"DASHBOARD_APPS={ft_layout.read_conf().get('DASHBOARD_APPS', 'steamvr')}")
+    url = shared_context()
+    if not url:
+        print("Steam's UI: not reachable on 127.0.0.1:8080")
+        return 0
+    try:
+        print(f"wrapper: {steamui.evaluate(STATUS)}")
+    except steamui.SteamUnreachable as e:
+        print(f"wrapper: unknown ({e})")
+    return 0
+
+
+def main():
+    if sys.argv[1:] == ["status"]:
+        return status()
+    if sys.argv[1:]:
+        sys.exit(__doc__)
+    signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
+    url = None
+    try:
+        while True:
+            if wanted():
+                url = shared_context()
+                if url:
+                    try:
+                        session(url)
+                        continue
+                    except (OSError, ValueError, steamui.SteamUnreachable) as e:
+                        log(f"Steam's UI went away ({e}); trying again")
+            time.sleep(HEARTBEAT)
+    finally:
+        if url:  # leaving with the wrapper in: Steam's own launch at once, not 10 s from now
+            try:
+                steamui.evaluate(OFF)
+            except steamui.SteamUnreachable:
+                pass
+
+
+if __name__ == "__main__":
+    sys.exit(main())

float/ft_floatd.py

@@ -15,6 +15,9 @@ command-line side). Replies go to the sender:
   launch APP.desktop [action=ACTION] [screen=N]   run ["argv", ...]
             (start an app, or one of its desktop file's actions, and float its first window;
             with screen=N its first window opens docked on screen N instead)
+  dashlaunch CMDLINE   (ft-dashroute: an app picked in the SteamVR dashboard's Launch a program
+            list, Steam's command line for it: launch it floating -> "ok", or "steamvr REASON"
+            when it should be SteamVR's own window after all; see dash_launch)
   windows   (-> JSON: the open apps' windows and where they are, for a profile; ft-layout save)
   spotlist  (-> JSON: every app window, its title, panel, and when it was last active; spotwatch)
   activate ID   (make that window KWin's active one, unminimized; spotwatch)
@@ -58,6 +61,8 @@ import json
 import math
 import os
 import re
+import shlex
+import shutil
 import socket
 import subprocess
 import sys
@@ -88,6 +93,12 @@ PLACES_PATH = os.path.expanduser("~/.config/frametop-float.json")  # each app's
 LAUNCH_SECONDS = 30      # a launched app's window must show up within this
 IN_FRONT = 2.0           # at most this far in front of you, for an app with no remembered place
 DEBUG = os.environ.get("FT_FLOAT_DEBUG") == "1"  # log every event from the script
+# The SteamVR dashboard's Launch a program list (float/ft-dashroute): these stay SteamVR's, since
+# they aren't apps with a window for the desktop: Steam, SteamVR, the desktops themselves, and
+# Frametop's layout commands (a profile's entry, Reset Screen Layout, Hide/Show Screens).
+STEAMVR_ONLY = {"steam", "vrurlhandler", "steamos-nested-desktop", "frametop-session.sh", "ft-layout",
+                "ft-layout-reset", "ft-hide-show"}
+PAUSE_PATH = f"/run/user/{os.getuid()}/frametop-pause.json"  # input/game_pause.py's state
 
 
 def whole(*scales):
@@ -239,6 +250,37 @@ def listable(ev):
     return ev.get("app", "") not in SKIP_APPS and not any(n.startswith("ksplash") for n in (ev.get("app", ""), ev.get("cls", "")))
 
 
+def exec_argv(info):
+    """A desktop file's command, without its field codes (%U and the like), as Steam lists it."""
+    try:
+        argv = shlex.split(info.get_commandline() or "")
+    except ValueError:
+        return []
+    return [a for a in argv if not re.fullmatch(r"%[a-zA-Z]", a)]
+
+
+def app_of(argv):
+    """The desktop file (its name, without .desktop) whose command is argv, or None. Steam's
+    command line for an app is its Exec line without the field codes, each argument quoted."""
+    if not argv:
+        return None
+    want = [shutil.which(argv[0]) or argv[0]] + argv[1:]
+    for info in sorted(Gio.AppInfo.get_all(), key=lambda i: not i.should_show()):  # menu entries first
+        got = exec_argv(info)
+        if got and [shutil.which(got[0]) or got[0]] + got[1:] == want:
+            return info.get_id().removesuffix(".desktop")
+    return None
+
+
+def paused():
+    """Frametop is paused for a VR game: its windows don't show."""
+    try:
+        with open(PAUSE_PATH) as f:
+            return json.load(f).get("paused") is True
+    except (OSError, ValueError, AttributeError):
+        return False
+
+
 def cmdline(pid):
     try:
         with open(f"/proc/{int(pid)}/cmdline", "rb") as f:
@@ -879,6 +921,30 @@ class Daemon:
             return self.took(waiting[0])
         return None
 
+    def dash_launch(self, text):
+        """An app picked in the SteamVR dashboard's Launch a program list (ft-dashroute):
+        launch it with its first window floating -> "ok". Or "steamvr REASON": SteamVR
+        opens it as its own window after all."""
+        if ft_layout.read_conf().get("DASHBOARD_APPS", "steamvr") != "frametop":
+            return "steamvr the setting is off"
+        try:
+            argv = shlex.split(text)
+        except ValueError:
+            argv = []
+        if not argv:
+            return "steamvr no command"
+        if os.path.basename(argv[0]) in STEAMVR_ONLY:
+            return f"steamvr {os.path.basename(argv[0])} is SteamVR's"
+        if paused():
+            return "steamvr Frametop is paused"
+        now = time.monotonic()
+        waiting = sum(1 for la in self.launches if la.dock is None and la.until > now)
+        if sum(1 for s in self.slots if s.window is None) - waiting <= 0:
+            return f"steamvr all {len(self.slots)} floating windows are in use"
+        app = app_of(argv)
+        reply = self.launch(app=app)[0] if app else self.launch(argv=argv)[0]
+        return "ok" if reply == "ok" else f"steamvr {reply}"
+
     def took(self, la):
         if not la.entries or len(la.entries) <= 1:
             self.launches.remove(la)
@@ -1147,6 +1213,8 @@ class Daemon:
             if not isinstance(argv, list) or not argv or not all(isinstance(a, str) for a in argv):
                 return "error run takes a JSON list"
             return self.launch(argv=argv)[0]
+        if cmd == "dashlaunch" and rest:
+            return self.dash_launch(text.split(None, 1)[1])
         if cmd in ("windows", "spotlist"):
             return self.capture(sender, cmd)
         if cmd == "activate" and len(rest) == 1:

gaze/README.md

@@ -30,6 +30,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
   - **Eye tracker:** our own (Own tracker: see "Our own eye tracker" below) or SteamVR's. The default, `GAZE_TRACKER=auto`, is ours when it's installed (its frame grabber, and ft-eyes' Python in the gaze service's checkout), else SteamVR's, and it switches when ours is installed or removed; picking one on the Gaze page sets it for good. The gaze service runs ours while it's the one in use. It keeps its own calibration: with Own tracker chosen, Calibrate on the Gaze page calibrates it. The gaze pointer's settings (hand back, nudges, hold to drag, the dot) are the pointer helper's, so they're the same with either.
   - **Eye bias:** Auto, Left, or Right. The gaze combines both eyes, each calibrated on its own, because their errors partly cancel: on 306 clicks with our tracker, the eyes' sideways errors were correlated -0.37, and both together were 0.65 degrees off (median) against 0.96 for the left eye alone and 1.11 for the right. So Left or Right leans instead of choosing: that eye counts twice as much as the other (0.03 degrees worse there toward the better eye, 0.13 toward the worse). Auto weights each eye by the inverse square of how far off it was at your last 20 nudges, once each eye has 5, and evenly before that. Each eye's miss is measured before that nudge teaches anything, so each is a fresh test. The calibration's own fit isn't used for this: on SteamVR's test of 2026-09-29, the calibration dots said the left eye was the better one, and new spots said the right. Either eye carries the gaze alone while the other is closed or lost.
   - **One eye:** SteamOS 0.4's Track Dominant Eye Only (SteamVR's settings, General, with Show advanced settings on; `steamvr.eyeTrackingDominantEyeOnly` with `steamvr.dominantEye`) makes SteamVR's tracker ignore the other eye. With SteamVR's tracker, Frametop then goes by that eye alone: the calibration and the checks wait only for it, the gaze is that eye's own reading once it's calibrated (SteamVR's combined gaze before, which follows that eye then), and the fit check shows the other eye as not tracked. The service reads the setting again when SteamVR's settings file changes. Not yet tried in the headset: what SteamVR's shared memory says about the ignored eye wasn't measured.
+  - **Fusion** (`GAZE_FUSION=on` in `~/.config/frametop.conf`, off by default, not on the Gaze page yet): with our tracker, the gaze takes in SteamVR's too. SteamVR's is its two eyes averaged, corrected by an affine fit (offset, gain, and tilt) on the same dots as our calibration: Calibrate with Own tracker saves it in `~/.local/state/frametop/gaze/fusion.json`, and `gaze/bench/compare.py --fit-fusion` can make one from bench runs. While ours sees both eyes, the gaze is half ours and half SteamVR's. While ours has one eye or none, SteamVR's carries it alone, and without SteamVR's eyes (closed, lost, or its one-eye setting) it's ours. The two trackers' errors partly cancel. On the gaze bench's two head-locked runs of 2026-10-17 (25 presses), each fitted on the other presses: ours alone was 2.01 degrees off (median; 6.47 at the 90th percentile), SteamVR's eyes averaged 2.67 (5.72), SteamVR's corrected 1.78 (4.26), and the fusion 1.51 (3.10). Fitted on the other run only, the fusion was 1.41 (4.59). The fit holds only while SteamVR's tracker reads as it did at the calibration, and SteamVR starts its eye model over each time the headset goes on, so it stays off until benches across wears say it holds.
 
   With SteamVR, each eye is its own reading (set 2), corrected by its calibration from the probe (the Left eye and Right eye sources) plus what the pointer has taught that eye since. On that test, the two eyes each calibrated and averaged were 1.70 degrees off (median; mean 1.62) against 1.72 (mean 1.84) for SteamVR's combined gaze with its calibration. A calibration from before the probe had the eyes as sources, or `--source`, uses the older path. That path runs on SteamVR's combined gaze (mmap set 1), corrected as a whole. When the tracker loses one eye (its variance for that eye jumps from about 0.001 to 0.02), the gaze comes from the other eye instead: that eye's own reading (set 2) plus what it usually reads against the combined gaze, learned while both eyes are seen, in 10 degree cells of where it looks. Set 1 keeps going on one eye too, but it holds the lost eye's yaw where it was, so the gaze moves half as far sideways as your eyes do. On a recording, one eye alone came out a median 0.8 degrees from both eyes' gaze over a steady look, a little more jittery.
 
@@ -38,7 +39,7 @@ Gaze as an input method for the whole desktop, without replacing anything of Ste
 - **The mouse only corrects** (the default; the Gaze page's Mouse movement switch, `POINTER_GAZE_MOUSE_MOVE=held`): while the gaze has the pointer, moving the mouse does nothing. The buttons work like Meta+J and Meta+K: press and hold one and the pointer stops where you look; move the mouse onto what you meant and let go to click there (a left or a right click). Held still for half a second, a press is a real one (to drag). Once you've moved, the left button alone only clicks: press the right one while still holding the left to start a drag there; it lasts while either button is held. Press the right one again (a double right click, the left still held) to pan and tilt what you're dragging, as a right press does during any drag. A bumped or drifting mouse can't pull the pointer away, and every mouse move is a correction, so the tracker only learns from real ones. With the gaze stale for a second (the tracker stopped, eyes lost), in a game, or with the headset off, the mouse moves the pointer as usual. `free` (the switch off) lets the mouse take the pointer any time.
 - **Keyboard clicks** (Meta+J left, Meta+K right; other key combinations on the Keyboard page of Input Settings): tap to click where you look. A quick tap (let go within 0.25 s, `POINTER_KEY_TAP`) clicks where the dot was when you pressed, whatever your head did, and tells the gaze service it was right there. Hold instead, and the dot stays put in your view: turn your head until it sits on what you meant, and let go to click there (the correction is a lesson, as with the mouse, under the same limit: past `POINTER_GAZE_NUDGE_MAX` it opens the quick check instead). Hold still for half a second to press for real, then turn your head to drag. With Meta+J held, Meta+K presses where the dot is now, so you can correct first and then drag; the drag lasts while either key is held. Meta+K during a Meta+J drag (again, after starting it with Meta+K: a double Meta+K) pans and tilts what you're dragging while it's held: turn your head to turn it.
 - **Learning from nudges:** if the mouse took the pointer from the gaze and moved it (0.2 degrees or more, and the correction within `POINTER_GAZE_NUDGE_MAX`: 55 degrees by default, half of the 109 the headset shows across, and 1 to 110; the same limit for mouse, keyboard, and pinch clicks) before you clicked, or you dragged a held press that far, you were nudging it onto what you looked at. The helper sends that as a lesson, from the raw gaze when the mouse took over to where you clicked, and ft-gazed learns it. So using it is what calibrates it. The raw gaze is one ft-gazed sent, so it also finds when that look was, and what each eye read then. With SteamVR, each eye learns its own error. With our tracker, the look goes to it as a click, like the probe's, and it relearns how the headset sits on your face. After the headset was off, your first nudge and click there resets that (the quick check's dot does the same). A correction past `POINTER_GAZE_NUDGE_MAX` isn't learned: the helper asks ft-gazed for the quick check instead ("recheck", after its 2-minute cooldown). Tested on our tracker's 409 clicks since its Sep 29 calibration: a one-dot check set from any one of them put the next 2 minutes' clicks within 15 degrees (99% within 4.2) and the next 10 minutes' within 25 (the far ones after the headset moved), so the check gets back well under it. The limit used to be 8 degrees, and live on 2026-10-01 our tracker was 12 off after the headset went on, so every correction was dropped. One lesson moves the whole correction by only a third of what it measured (more near where it was taken), since in the first live test one 6 degree lesson moved everything and put the next target 7 degrees off. `ft-gazectl status` shows the lessons, and `ft-gazectl forget` drops them.
-- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow: the middle, and 12 degrees left and right and 9 up and down, in a see-through panel 40 degrees wide (the quick check's 16 degree square left four of them off the panel, unseen). The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click or press Meta+J, and the gaze held still up to then is taken. A button mapped to gaze precision or gaze drag counts as the left click there (before 2026-10-09 the panel ignored it). Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). The dot's ring shows full while ft-eyes checks it; the service doesn't wait for that answer (before 2026-10-09 it did, up to 3 s a dot, so the pointer could come back over the panel, and a slow answer closed the calibration as if the headset came off). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
+- **Checks and calibration in the headset** (`gaze/gazecheck.py`, shown by `gaze/panel/ft-gazepanel`, a panel fixed to the headset that ft-gazed runs): a one-dot quick check opens when you put the headset on (SteamVR's tracker sees your eyes for 3 s after none for 3 s; its "HMD on" log line can't say, since it repeats every minute or so and can stay on for hours with nobody in the headset), when our tracker asks for a click (its "reseat", when the headset may sit differently), at most once every 2 minutes, and from Quick check on the Gaze page. Look at the dot: it takes your gaze once it has held still for 0.6 s (the steadiness counts, not where the tracker puts it, so it works however far off it is), or at once with a left click, Enter, an index pinch (with hand tracking) or Meta+J; a right click or Meta+K closes it, and ignoring it changes nothing. It also runs when a click's correction was past `POINTER_GAZE_NUDGE_MAX`. The dot is still and the ring fills in quarters, so the panel is drawn again only a few times per dot. If the first 3 lessons after it are still over 2 degrees off, five dots follow: the middle, and 12 degrees left and right and 9 up and down, in a see-through panel 40 degrees wide (the quick check's 16 degree square left four of them off the panel, unseen). The full calibration (Calibrate on the Gaze page, or by itself whenever gaze mode is on without one and your eyes are seen) is the probe's: three rounds, dark, medium and bright, of the middle and a ring around it, in a panel 64 degrees wide, with Frametop's screens hidden. Its dots (and the five-dot check's) wait for a click: look at the dot and left click, press Enter, pinch (index to thumb, with hand tracking) or press Meta+J, and the gaze held still up to then is taken. A button mapped to gaze precision or gaze drag counts as the left click there (before 2026-10-09 the panel ignored it). Enter works on a keyboard while typing goes to the desktop (the relay takes it while the panel is up, so it isn't typed; while typing goes to Steam, Steam sees it too). A pinch counts from its start, whether or not hand interaction is on, but not in a VR game, with the headset off, while typing, or at a controller. Our tracker's first calibration has no gaze to go on, since ft-eyes maps pupils to a gaze only once it has a calibration: it opens once SteamVR's tracker sees an eye and ft-eyes answers, and a click takes the 0.6 s up to it, as long as ft-eyes saw each pupil held still then (before 2026-10-05 it waited for a gaze, so a fresh install could never calibrate ours). The dot's ring shows full while ft-eyes checks it; the service doesn't wait for that answer (before 2026-10-09 it did, up to 3 s a dot, so the pointer could come back over the panel, and a slow answer closed the calibration as if the headset came off). A dot that isn't taken says why, on an orange line over the instructions: with SteamVR's tracker, what dropped most of that look's samples (an eye lost, a blink, the two eyes disagreeing); with ours, its reply (an eye seen in too few frames, or moving). A dot gets two tries, then it's skipped. A calibration left with under two thirds of its dots fails and names the most common reason, as the Gaze page does after it. A click that has taken nothing after 1.5 s says what it waits for: the gaze to hold still, or an eye tracker that isn't sending. Capturing whenever the gaze held still sometimes took a look that wasn't on the dot. The panel draws into three shared buffers SteamVR imported once, as Frametop's keyboard does: uploading each picture anew (SetOverlayRaw) flickered, and in one live test left the headset showing an old picture. Quitting it while there's still no calibration turns gaze mode off; turning it on again reopens it. One that closes otherwise unfinished (ignored for 2 minutes, too few dots) opens again after the headset comes off and on. Why gaze mode, on, can't work yet goes in the service's status as `checks.problem`, which the Gaze page shows under the Gaze pointer switch. For our tracker a check is a click and the calibration is its own (calib-point per dot); for SteamVR's, a check is a lesson for each eye and the calibration replaces calibration.json, and the lessons start over. Checks go to `checks.jsonl`.
 - Nothing writes to SteamVR, its eye tracker, or its files: ft-gaze maps the eye tracker's shared memory read-only. With no fresh gaze (a blink, the service stopped, the headset off), the pointer stays where it is, and the mouse works as always.
 - **Idle while the gaze isn't used:** ft-gaze and our own tracker run only while gaze mode is on and someone wears the headset (the pointer helper says both: SteamVR drops the headset's activity level as soon as it comes off), while a check or the calibration is open or asked for, or while the Gaze page of Frametop Input Settings is open (it renews a `wake` lease). 30 seconds after the last use they stop, and our frame grabber goes idle with our tracker. With our tracker, that saves over half a core: on 2026-10-02, with gaze mode off, ft-eyes took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. A check asked for while it idles starts the tracker and opens once it sends. When the gaze is used again, it takes a few seconds to come back, and our tracker's first click re-seats it, as after the headset was off: so the quick check opens when gaze mode comes on after the service idled, as it does when you put the headset on. `ft-gazectl status` says `"awake"`, and `"idle"` says why it isn't. A stand that covers the proximity sensor makes the headset seem worn, so with gaze mode on it doesn't idle there.
 
@@ -60,7 +61,7 @@ The tracker stops when the headset is off your head. SteamVR also calibrates gaz
 `gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
 
 - `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye, or 15 without `gaze/eye-rate.py on`) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). It also adds it to SteamOS's update keep list (`/etc/atomic-update.conf.d/frametop-eyegrab.conf`): an update deletes `/etc` files that list doesn't name, and without the grabber gaze mode falls back to SteamVR's tracker and its calibration. `scripts/doctor.sh` reports a grabber an update deleted.
-- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`. Instead of the pupil finder it can run an eye net (`eyes_net.py`): `EYES_MODEL` is `classic` (the default), `latest`, or a version from `gaze/tracker/models.json`, picked with its notes and benchmarks on Input Settings' Gaze page, which downloads it into `gaze/tracker/models/NAME` (`models/README.md`). Each net keeps its own calibration in `eyes/net-NAME/`.
+- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. A find much smaller than that eye's usual pupil (the median over the last 10 s) counts as not found, since it's usually a dark spot beside the pupil (`EYES_SIZE_GUARD`): with `both`, the default, under 0.75 of the other eye's ratio, so light shrinking both pupils gets through; with `long`, under 0.6 whatever the other eye does; `off` takes every find. On the bench runs of 2026-10-17, good finds were 0.84 of usual size or more 95% of the time, and the right camera's misfinds a median of 0.77. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`. Instead of the pupil finder it can run an eye net (`eyes_net.py`): `EYES_MODEL` is `classic` (the default), `latest`, or a version from `gaze/tracker/models.json`, picked with its notes and benchmarks on Input Settings' Gaze page, which downloads it into `gaze/tracker/models/NAME` (`models/README.md`). Each net keeps its own calibration in `eyes/net-NAME/`.
 - `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
 
 Ground rules, for anyone changing it:
@@ -72,7 +73,7 @@ Ground rules, for anyone changing it:
 
 ## Headset fit
 
-Check headset fit on the Gaze page opens it in the headset panel: a card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and the hints, live while you adjust the headset. A left click or Meta+J runs the guided check (dots, then looks down, up, left and right), and a right click or Meta+K closes it. The probe's Headset fit mode (`ft-gazeprobe --mode fit`, for development) has the same check with maps: it shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
+Check headset fit on the Gaze page opens it in the headset panel: a card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was seen) and the hints, live while you adjust the headset. A left click, Enter, a pinch or Meta+J runs the guided check (dots, then looks down, up, left and right), and a right click or Meta+K closes it. The probe's Headset fit mode (`ft-gazeprobe --mode fit`, for development) has the same check with maps: it shows, for each eye, whether the tracker has it, how open it is, and the tracker's confidence in it, and a map of where you looked coloured by how often it lost that eye there. Hints under the maps say which eye gets lost where, and what to try. Enter runs a guided check: dots around the screen, then looking down at the keyboard, up, left and right. R starts over. Adjust the headset while you watch it.
 
 **The fit watch** (`GAZE_FIT_WATCH=1` in `~/.config/frametop.conf`; off by default until it's been tried in the headset). The fit check keeps a good fit as yours: when its coach (`gaze/fitcoach.py`, from ft-eyes' pupils and glints, with the shipped reference) last read it as good while you looked straight ahead, closing the check saves that reading to `~/.local/state/frametop/gaze/fit-baseline.json`, with the IPD setting. The eye recorder's fitment step saves it too. Then, with the fit watch on:
 - **Put-on check:** when the headset goes on, the quick check first holds its dot for up to 4 s while the coach reads the fit against your saved one. If it sits the same, the dot is taken as usual. If it moved (5 degrees of gaze or more), the panel widens and says which way to move the headset, and a changed IPD over that, live, until it's back ("Good: it's back where it was"); then the dot is taken. A click takes the dot as the headset sits, and after 30 s it's taken anyway. It needs our tracker installed (ft-eyes runs for it with either tracker in use), and without a saved fit it's the plain quick check.

gaze/bench/README.md

@@ -0,0 +1,101 @@
+# Gaze Bench
+
+A test bench for gaze tracking. It shows dots in the places gaze tracking finds hard (far out
+toward the edges and corners, long holds, light changes, a moving head, after the headset moved).
+You look at each one and press the headset's button. Every press is recorded with everything the
+trackers saw, so gaze techniques can be compared afterwards on the same looks (`compare.py`),
+including ones that didn't run live.
+
+The dots show on a panel in the headset, 64 degrees across, that moves with your head: a dot is
+always at the same direction from your head, wherever you face. So every run asks the same of the
+eyes, and you never have to turn your head to find a target. Your eyes do all the looking.
+
+## Use
+
+1. Put the headset on and open **Frametop Gaze Bench** from the app menu (`gaze/bench/install.sh`
+   adds it), or run `gaze/bench/ft-gazebench` from Frametop's desktop. A window opens; its
+   status lines say whether the headset panel, the headset button and the trackers are ready.
+   Gaze mode can be on or off: the bench keeps the trackers running itself.
+2. Pick a scenario. The panel shows what to do; press the button on the headset's right side
+   to begin.
+3. For each dot: look at it and press the button. The dot turns green and the next one shows.
+   Where you looked is the dot, so only press while you're on it.
+   Two presses take the last one back. Holding the button ends the scenario.
+   As in the gaze calibration, a left click, Enter or an index pinch (index to thumb, with hand
+   tracking) marks too, and a right click or Meta+K ends: while the panel shows, the bench claims
+   the pointer helper's calibration panel (`calpanel 1 ft_gazebench.PID`), so the helper sends
+   those here and hides the pointer. Those marks are recorded `"by": "helper"`, at the time the
+   helper's answer came in (a pinch's a frame or two after it began).
+   The window's keys do the same: Space marks, Backspace takes the last one back, S skips a
+   dot, Esc ends.
+4. At the end, the panel and the window say how far our tracker and SteamVR's were off. A press
+   (or a click, Enter or a pinch) closes the panel. Then run `compare.py` on the runs.
+
+| Scenario | What it tests |
+|---|---|
+| Warm-up | nine dots around the middle |
+| Edges | sixteen dots along the panel's edges and in its corners, 24 to 37 degrees out |
+| Hold at the edges | edge dots you look at for 3 s before pressing (the tracker loses an eye out there) |
+| Out from the middle | twenty-four dots on eight spokes, at a third, two thirds and all the way out |
+| Head moving | nine dots; keep looking while you turn or nod your head, press while it moves |
+| Light changes | dark, bright and grey backgrounds: pupil size |
+| Small steps | twenty dots, each 2 to 4 degrees from the last |
+| After a nudge | lift the headset and set it back, then nine dots: recovery |
+| Long session | sixty dots anywhere at your own pace, a few seconds apart, about six minutes: drift and stale corrections |
+
+Scenarios live in `scenarios.json` (the generators are described at its top).
+
+## Compare
+
+```
+scripts/in-box /opt/frametop/venv/bin/python gaze/bench/compare.py ~/.local/share/frametop/gaze/bench
+```
+
+Give it runs, or the folder they're in. For every press it prints how far each technique was from
+where you looked: overall, by scenario, by where the dot was (edge, corner, inside), by how far
+from straight ahead, and by the time since the last nudge. `--per-click` adds a line per press;
+`--json OUT` writes the records. The techniques are described at the top of `compare.py`: what
+ran live (our tracker, SteamVR's combined gaze and its two eyes averaged, a 70/30 mix, SteamVR's
+corrected by the presses), SteamVR's eyes corrected by the run's `fusion.json` (copied into
+`start/` when the run begins) and the fusion with ours, the same fitted on the run's other presses
+(`-loo`) or on the other runs only (`-xrun`), and replays of our classic tracker on its recorded
+finds with changed rules (no glint jump, a jump only when both eyes see it, no lessons past the
+calibrated range, learning from every press, bridging a lost eye, the size guard's two rules or
+none). `fusion-loo:REPLAY` fuses a replay with SteamVR's.
+`--fit-fusion OUT` fits SteamVR's correction on all the given runs' presses and saves it as a
+`fusion.json` the gaze service reads (`~/.local/state/frametop/gaze/fusion.json`). The replay of the live rules should land close to what our tracker published (the
+"replay vs live" line); the variants are only as trustworthy as that.
+
+Runs from before 2026-10-17 showed their targets on a Frametop screen, with clicks. `compare.py`
+still reads them: their truth comes from ft-gaze's screen hits, and they have a "pointer"
+technique (where the gaze pointer was).
+
+A long session's run (gaze.jsonl and own.jsonl) takes a while to replay: run it through
+`frame-job` when the headset is in use.
+
+## What a run records
+
+`~/.local/share/frametop/gaze/bench/<time>-<scenario>/` (`benchrec.py` describes every file):
+the dots and the presses, the panel's log of what the dot did and when it reached the eyes,
+the headset's pose 250 times a second, ft-gaze's samples of every source except SteamVR's input
+action (the one that costs vrserver anything), our tracker's per-frame pupil and glint finds,
+and our tracker's and the gaze service's calibration and state at the start. With **Record the
+eye cameras** on (the default), both eye cameras too, as the Eye Recorder records them: about
+1.2 GB a minute.
+
+Eye images identify a person. They stay on the headset: the bench uploads nothing, and the runs
+are yours to delete (the folder above).
+
+## Files
+
+| File | |
+|---|---|
+| `gazebench.py`, `bench.qml` | the app and its window (PySide6, in Frametop's container); it runs its own `ft-gazepanel` |
+| `benchrec.py` | the recording |
+| `scenarios.json` | the scenarios |
+| `compare.py` | the comparison (numpy, OpenCV) |
+| `ft-gazebench`, `ft-gazebench.desktop`, `install.sh` | the launcher and its menu entry |
+| `tests/` | `test_bench.py` (runs with stand-ins for ft-gaze, the panel, the button and the services), `test_compare.py` (a made-up run with known errors) |
+
+Tests: `scripts/in-box python3 gaze/bench/tests/test_bench.py`, and
+`scripts/in-box /opt/frametop/venv/bin/python gaze/bench/tests/test_compare.py`.

gaze/bench/bench.qml

@@ -0,0 +1,188 @@
+// Frametop Gaze Bench's window (gazebench.py), one page at a time: "menu" picks a scenario,
+// "run" follows it (its targets show on the headset's panel, not here), "done" shows how far the
+// trackers were off. The keys work as the headset button does.
+import QtQuick
+import QtQuick.Controls
+import QtQuick.Layouts
+import QtQuick.Window
+
+Window {
+    id: win
+    title: "Frametop Gaze Bench"
+    width: 960
+    height: 720
+    visible: true
+    color: "#262626"
+
+    readonly property color ink: "#eeeeee"
+    readonly property color dim: "#9a9a9a"
+
+    Shortcut { sequence: "Esc"; onActivated: bench.stop() }
+    Shortcut { sequence: "Backspace"; onActivated: bench.undo() }
+    Shortcut { sequence: "S"; enabled: bench.page === "run"; onActivated: bench.skip() }
+    Shortcut { sequences: ["Space", "Return", "Enter"]; enabled: bench.page !== "menu"; onActivated: bench.markKey() }
+
+    // --- menu
+
+    Item {
+        anchors.fill: parent
+        visible: bench.page === "menu"
+
+        ColumnLayout {
+            anchors.centerIn: parent
+            width: Math.min(parent.width - 60, 900)
+            spacing: 12
+
+            Label {
+                text: "Gaze Bench"
+                font.pixelSize: 30
+                color: win.ink
+            }
+            Label {
+                Layout.fillWidth: true
+                wrapMode: Text.Wrap
+                color: win.dim
+                font.pixelSize: 15
+                text: "Each scenario shows dots one at a time on a panel in front of you in the headset. "
+                    + "It moves with your head, so your eyes do all the looking. Look at the dot "
+                    + "and press the button on the headset's right side. Every look is recorded, so different "
+                    + "gaze techniques can be compared on the same looks afterwards."
+            }
+            Label {
+                Layout.fillWidth: true
+                visible: bench.notice !== ""
+                wrapMode: Text.Wrap
+                color: "#ffb347"
+                font.pixelSize: 15
+                text: bench.notice
+            }
+            GridLayout {
+                Layout.fillWidth: true
+                Layout.topMargin: 6
+                columns: 2
+                columnSpacing: 12
+                rowSpacing: 8
+                Repeater {
+                    model: bench.scenarios
+                    delegate: Button {
+                        required property var modelData
+                        Layout.fillWidth: true
+                        Layout.preferredWidth: 1
+                        implicitHeight: 64
+                        focusPolicy: Qt.NoFocus
+                        onClicked: bench.start(modelData.id)
+                        contentItem: ColumnLayout {
+                            spacing: 2
+                            Label {
+                                text: modelData.name + "  (" + modelData.count + ")"
+                                font.pixelSize: 16
+                                font.bold: true
+                            }
+                            Label {
+                                Layout.fillWidth: true
+                                text: modelData.about
+                                wrapMode: Text.Wrap
+                                maximumLineCount: 2
+                                elide: Text.ElideRight
+                                font.pixelSize: 12
+                                opacity: 0.8
+                            }
+                        }
+                    }
+                }
+            }
+            CheckBox {
+                text: "Record the eye cameras too (about 1.2 GB a minute; they stay on this headset)"
+                checked: bench.images
+                focusPolicy: Qt.NoFocus
+                onToggled: bench.images = checked
+            }
+            Label {
+                Layout.fillWidth: true
+                wrapMode: Text.Wrap
+                color: win.dim
+                font.pixelSize: 13
+                text: bench.status
+            }
+            RowLayout {
+                Layout.topMargin: 4
+                Label {
+                    Layout.fillWidth: true
+                    wrapMode: Text.Wrap
+                    color: win.dim
+                    font.pixelSize: 13
+                    text: "The headset button: press = marked, two presses = back one, hold = end. A left click, "
+                        + "Enter or an index pinch marks too; a right click ends. "
+                        + "Here: Space marks, Backspace goes back one, S skips, Esc ends."
+                }
+                Button { text: "Quit"; focusPolicy: Qt.NoFocus; onClicked: Qt.quit() }
+            }
+        }
+    }
+
+    // --- a run (the dots are on the headset's panel)
+
+    ColumnLayout {
+        anchors.centerIn: parent
+        width: Math.min(parent.width - 60, 760)
+        visible: bench.page === "run"
+        spacing: 14
+        Label {
+            Layout.fillWidth: true
+            wrapMode: Text.Wrap
+            text: bench.progress
+            color: win.ink
+            font.pixelSize: 22
+        }
+        Label {
+            Layout.fillWidth: true
+            wrapMode: Text.Wrap
+            text: bench.message
+            color: win.dim
+            font.pixelSize: 16
+        }
+        Label {
+            Layout.fillWidth: true
+            text: bench.last
+            color: win.ink
+            font.pixelSize: 16
+        }
+        Label {
+            Layout.fillWidth: true
+            wrapMode: Text.Wrap
+            color: win.dim
+            font.pixelSize: 13
+            text: "The headset button: press = marked, two presses = back one, hold = end. A left click, "
+                + "Enter or an index pinch marks too; a right click ends. "
+                + "Here: Space marks, Backspace goes back one, S skips, Esc ends."
+        }
+        RowLayout {
+            spacing: 10
+            Button { text: "Back one"; focusPolicy: Qt.NoFocus; onClicked: bench.undo() }
+            Button { text: "Skip"; focusPolicy: Qt.NoFocus; onClicked: bench.skip() }
+            Button { text: "End"; focusPolicy: Qt.NoFocus; onClicked: bench.stop() }
+        }
+    }
+
+    // --- done
+
+    ColumnLayout {
+        anchors.centerIn: parent
+        width: Math.min(parent.width - 60, 860)
+        visible: bench.page === "done"
+        spacing: 16
+        Label {
+            Layout.fillWidth: true
+            wrapMode: Text.Wrap
+            text: bench.summary
+            color: win.ink
+            font.pixelSize: 18
+            lineHeight: 1.3
+        }
+        RowLayout {
+            spacing: 12
+            Button { text: "Again"; focusPolicy: Qt.NoFocus; onClicked: bench.again() }
+            Button { text: "Scenarios"; focusPolicy: Qt.NoFocus; onClicked: bench.stop() }
+        }
+    }
+}

gaze/bench/benchrec.py

@@ -0,0 +1,348 @@
+"""Gaze bench recording (README.md): everything a run needs to replay gaze techniques later.
+
+A run is one scenario played once. Its folder, under BASE:
+  run.json          the scenario, the panel, the settings, and the gaze service's and our
+                    tracker's status at the start and the end
+  trials.jsonl      a line per target: where it was, when it showed, and when you marked it
+                    (gazebench.py's own record: "click" is the mark, by the button or a key)
+  targets.jsonl     the panel's record of what the dot did, with when each change reached the
+  poses.jsonl       eyes; the headset's pose 250 times a second; SteamVR's IPD setting
+  ipd.jsonl         (ft-gazepanel's log, gaze/panel/ft-gazepanel.cpp)
+  gaze.jsonl        ft-gaze's samples, every source but SteamVR's input action (the one that
+                    calls into vrserver): SteamVR's eyes (mmap1, mmap2, left, right), our
+                    tracker's published gaze (own), the head ray, and where each lands on
+                    Frametop's screens, in pixels, with pixels per degree there (ft-gaze.cpp)
+  own.jsonl         our tracker's finds, a line per eye-camera frame (ft-eyes "finds start"):
+                    pupil, ellipse, glints, and each eye's gaze as it published it
+  lessons.jsonl     the gaze service's lesson log (pointer-lessons.jsonl) over the run: each
+                    nudge the pointer sent, its raw gaze and the true direction
+  eyes-clicks.jsonl our tracker's clicks over the run (clicks.jsonl): what each nudge taught it
+  start/            copies of our tracker's calibration, its dots and its state (the shift
+                    clicks taught it), and the gaze service's calibration, lessons and fusion
+                    (gazed-fusion.json), as they were when the run started, so a replay starts
+                    where the live tracker did
+  images/           both eye cameras, zstd chunks (gaze/rec/eyerec.py), unless images are off
+
+Eye images identify a person: they stay on this headset (and the maintainer's own training
+machines). Nothing here uploads anything.
+"""
+import json
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import threading
+import time
+from collections import deque
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+GAZE = HERE.parent
+
+# ft-gaze: this tree's build (gaze/build.sh), else the installed release's (a worktree with no build)
+FT_GAZES = [Path(p) for p in (os.environ.get("FT_GAZE"), GAZE / "build" / "ft-gaze",
+                              Path.home() / ".local/share/frametop/releases/current/gaze/build/ft-gaze") if p]
+BASE = Path(os.environ.get("FT_BENCH_BASE", Path.home() / ".local/share/frametop/gaze/bench"))
+STATE = Path.home() / ".local/state/frametop/gaze"
+EYES_STATE = STATE / "eyes"
+LESSON_LOG = STATE / "pointer-lessons.jsonl"
+CONF = Path.home() / ".config/frametop.conf"
+CONF_KEYS = ("GAZE_", "POINTER_GAZE", "EYES_MODEL")
+SOURCES = "mmap1,mmap2,left,right,own"
+EYES_SOCK = os.environ.get("FT_BENCH_EYES_SOCKET", "ft_eyes")     # (the tests' stand-ins)
+GAZED_SOCK = os.environ.get("FT_BENCH_GAZED_SOCKET", "ft_gazed")
+RECENT = 3.0        # s of parsed samples kept for the app's quick scores
+RESTART = 3.0       # s before ft-gaze is started again after it stopped (SteamVR restarted)
+KEEPALIVE = 2.0     # s between checks that our tracker is still logging finds
+LOG_SLACK = 3.0     # s either side of the run when copying the lesson logs
+
+
+def raw_now():
+    return time.clock_gettime(time.CLOCK_MONOTONIC_RAW)
+
+
+class Sock:
+    """A datagram client for one of the gaze services' abstract sockets (@ft_eyes, @ft_gazed)."""
+
+    def __init__(self, name):
+        self.name = "\0" + name
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
+        self.sock.bind("")
+
+    def ask(self, text, timeout=0.5):
+        """Send, and the reply (text) or None."""
+        self.sock.setblocking(False)
+        while True:  # drop late replies to earlier questions
+            try:
+                self.sock.recv(65536)
+            except OSError:
+                break
+        try:
+            self.sock.sendto(text.encode(), self.name)
+            self.sock.settimeout(timeout)
+            return self.sock.recv(1 << 20).decode(errors="replace")
+        except OSError:
+            return None
+
+    def json(self, text, timeout=0.5):
+        r = self.ask(text, timeout)
+        try:
+            return json.loads(r) if r else None
+        except ValueError:
+            return None
+
+    def close(self):
+        self.sock.close()
+
+
+class GazeStream:
+    """ft-gaze, for the app's life (it takes a moment to connect to SteamVR and the screens), and
+    again RESTART seconds after it stops: it quits when SteamVR does. Its lines go to the current
+    run's gaze.jsonl while one records, and the last RECENT seconds are kept parsed for the app's
+    quick scores and status."""
+
+    def __init__(self, prog=None, log=print):
+        self.prog = Path(prog) if prog else next((p for p in FT_GAZES if p.exists()), FT_GAZES[0])
+        self.log = log
+        self.proc = None
+        self.out = None
+        self.lock = threading.Lock()
+        self.recent = deque()
+        self.lines = 0
+        self.error = None
+        self.restarts = 0
+        self.stopping = threading.Event()
+        self.reader = None
+
+    def start(self):
+        if not self.prog.exists():
+            self.error = f"{self.prog} isn't built (gaze/build.sh)"
+            return self
+        self.reader = threading.Thread(target=self._run, name="ft-gaze", daemon=True)
+        self.reader.start()
+        return self
+
+    def _run(self):
+        while True:
+            with self.lock:
+                if self.stopping.is_set():
+                    return
+                self.proc = subprocess.Popen([str(self.prog), "--watch-stdin", "--sources", SOURCES],
+                                             stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+            before = self.lines
+            self._read()
+            code = self.proc.wait()
+            self.proc.stdout.close()
+            try:
+                self.proc.stdin.close()
+            except OSError:
+                pass
+            if self.stopping.is_set():
+                return
+            self.error = f"ft-gaze stopped (exit {code}); starting it again"
+            if self.lines > before or not self.restarts:   # not every few seconds while SteamVR is down
+                self.log(self.error)
+            self.restarts += 1
+            self.stopping.wait(RESTART)
+
+    def _read(self):
+        for line in self.proc.stdout:
+            with self.lock:
+                if self.out:
+                    self.out.write(line)
+            self.lines += 1
+            try:
+                s = json.loads(line)
+            except ValueError:
+                continue
+            self.error = None
+            with self.lock:
+                self.recent.append(s)
+                while self.recent and self.recent[0].get("t", 0) < s.get("t", 0) - RECENT:
+                    self.recent.popleft()
+
+    def to(self, path):
+        """Write the samples to path from now on (None: stop writing)."""
+        with self.lock:
+            if self.out:
+                self.out.close()
+            self.out = open(path, "ab") if path else None
+
+    def window(self, t0, t1):
+        with self.lock:
+            return [s for s in self.recent if t0 <= s.get("t", 0) <= t1]
+
+    def latest(self):
+        with self.lock:
+            return self.recent[-1] if self.recent else None
+
+    def stop(self):
+        self.to(None)
+        with self.lock:
+            self.stopping.set()
+            proc = self.proc
+        if proc:
+            try:
+                proc.stdin.close()
+            except OSError:
+                pass
+            try:
+                proc.wait(3)
+            except subprocess.TimeoutExpired:
+                proc.kill()
+        if self.reader:
+            self.reader.join(2)
+
+
+def conf_lines():
+    try:
+        return [l for l in CONF.read_text().splitlines() if l.startswith(CONF_KEYS)]
+    except OSError:
+        return []
+
+
+def eyes_state_dir(status):
+    """Our tracker's state folder for the model it runs (ft-eyes: net-NAME/ for a net)."""
+    model = (status or {}).get("model")
+    if model:
+        try:
+            name = json.loads((Path(model) / "model.json").read_text())["name"]
+        except (OSError, ValueError, KeyError):
+            name = Path(model).name
+        return EYES_STATE / ("net-" + str(name))
+    return EYES_STATE
+
+
+def last_time(path, before):
+    """The latest "time" (wall clock) in a JSON lines log before `before`, or None: the last nudge
+    before a run, for compare.py's time since the last nudge."""
+    last = None
+    try:
+        with open(path) as f:
+            for line in f:
+                try:
+                    tm = json.loads(line).get("time")
+                except ValueError:
+                    continue
+                if tm is not None and tm < before and "refused" not in line:
+                    last = tm if last is None else max(last, tm)
+    except OSError:
+        pass
+    return last
+
+
+def copy_rows(src, dst, t0, t1):
+    """The JSON lines of src whose "time" (wall clock) is within [t0, t1], appended to dst."""
+    n = 0
+    try:
+        with open(src) as f, open(dst, "a") as out:
+            for line in f:
+                try:
+                    tm = json.loads(line).get("time", 0)
+                except ValueError:
+                    continue
+                if t0 <= tm <= t1:
+                    out.write(line if line.endswith("\n") else line + "\n")
+                    n += 1
+    except OSError:
+        pass
+    return n
+
+
+class Run:
+    """One scenario played once: its folder, and what records into it (see the top)."""
+
+    def __init__(self, scenario, gaze, images=True, base=BASE, log=print, info=None):
+        self.scenario, self.gaze, self.images, self.log = scenario, gaze, images, log
+        stamp = time.strftime("%Y%m%d-%H%M%S")
+        self.dir = Path(base) / f"{stamp}-{scenario['id']}"
+        self.info = dict(info or {})
+        self.eyes = Sock(EYES_SOCK)
+        self.gazed = Sock(GAZED_SOCK)
+        self.writer = None
+        self.finds_on = False
+        self.checked = 0.0
+        self.trials = None
+
+    def status(self):
+        return {"eyes": self.eyes.json("status"), "gazed": self.gazed.json("status"), "t_raw": raw_now(),
+                "time": time.time()}
+
+    def start(self):
+        self.dir.mkdir(parents=True)
+        (self.dir / "start").mkdir()
+        st = self.status()
+        estate = eyes_state_dir(st["eyes"])
+        for name in ("calibration.json", "calibration-dots.jsonl", "state.json"):
+            if (estate / name).exists():
+                shutil.copy2(estate / name, self.dir / "start" / ("eyes-" + name))
+        for name in ("calibration.json", "pointer-lessons.json", "fusion.json"):
+            if (STATE / name).exists():
+                shutil.copy2(STATE / name, self.dir / "start" / ("gazed-" + name))
+        self.info.update(scenario=self.scenario, start=st, conf=conf_lines(), eyes_state=str(estate),
+                         images=self.images, sources=SOURCES, last_lesson=last_time(LESSON_LOG, st["time"]))
+        self.write_info()
+        self.trials = open(self.dir / "trials.jsonl", "a", buffering=1)
+        self.lines0 = self.gaze.lines
+        self.gaze.to(self.dir / "gaze.jsonl")
+        self.arm_finds()
+        if self.images:
+            try:
+                if str(GAZE / "rec") not in sys.path:
+                    sys.path.insert(0, str(GAZE / "rec"))
+                import eyerec
+                self.writer = eyerec.Writer(str(self.dir / "images"), log=self.log).start()
+                self.writer.record(True)
+            except Exception as e:  # no zstd, no share: record the rest
+                self.log(f"images: {e}")
+                self.writer = None
+                self.info["images_error"] = str(e)
+        return self
+
+    def arm_finds(self):
+        r = self.eyes.ask("finds start " + str(self.dir / "own.jsonl"))
+        self.finds_on = bool(r and r.startswith("ok"))
+        if not self.finds_on:
+            self.log(f"our tracker isn't logging finds ({r or 'no reply: is it running?'})")
+        return self.finds_on
+
+    def keepalive(self):
+        """Call now and then: our tracker may have restarted (or started), and dropped the log."""
+        if time.monotonic() - self.checked < KEEPALIVE:
+            return
+        self.checked = time.monotonic()
+        st = self.eyes.json("status")
+        if st is None:
+            self.finds_on = False
+        elif not self.finds_on:
+            self.arm_finds()
+
+    def trial(self, rec):
+        if self.trials:
+            self.trials.write(json.dumps(rec) + "\n")
+
+    def write_info(self):
+        tmp = self.dir / "run.json.tmp"
+        tmp.write_text(json.dumps(self.info, indent=1))
+        tmp.replace(self.dir / "run.json")
+
+    def stop(self, summary=None):
+        rows = self.eyes.ask("finds stop")
+        self.gaze.to(None)
+        if self.trials:
+            self.trials.close()
+            self.trials = None
+        if self.writer:
+            self.info["images_stats"] = self.writer.stop()
+        time.sleep(0.5)  # a nudge's lesson reaches the logs just after its click
+        st = self.status()
+        t0, t1 = self.info["start"]["time"] - LOG_SLACK, st["time"] + LOG_SLACK
+        n_lessons = copy_rows(LESSON_LOG, self.dir / "lessons.jsonl", t0, t1)
+        n_clicks = copy_rows(Path(self.info["eyes_state"]) / "clicks.jsonl", self.dir / "eyes-clicks.jsonl", t0, t1)
+        self.info.update(end=st, finds_rows=rows, lessons=n_lessons, eyes_clicks=n_clicks,
+                         gaze_lines=self.gaze.lines - self.lines0, summary=summary)
+        self.write_info()
+        self.eyes.close()
+        self.gazed.close()
+        return self.info

gaze/bench/compare.py

@@ -0,0 +1,670 @@
+#!/usr/bin/env python3
+"""Compare gaze techniques on gaze bench runs (README.md): how far each was from where you looked,
+at every click.
+
+  compare.py RUN [RUN...] [--per-click] [--json OUT] [--only a,b]
+
+RUN is a run's folder (benchrec.py), or a folder of them (all runs in it). A click is a mark:
+you looked at the target and pressed. Where you looked is the target, a head-relative direction
+on runs with the headset's panel (run.json "display": "panel", from 2026-10-17). The runs before
+showed targets on a Frametop screen: there it's the target's middle, or the click if it landed off
+the target, turned into a direction through ft-gaze's screen hits just before the click (the hit
+nearest that point, and its pixels per degree there: ft-gaze.cpp's HIT "j"). Each technique's
+gaze is its median over the same LOOK window (run.json), and its error the angle between the two.
+
+Techniques: what ran live, from gaze.jsonl:
+  own-live        our tracker's published gaze (ft-eyes, with its calibration and the shifts the
+                  nudges taught it)
+  steamvr         SteamVR's combined gaze (ft-gaze's mmap1, its own calibration)
+  steamvr-eyes    the mean of SteamVR's two eyes (left, right: set 2, no calibration of ours)
+  pointer         where the pointer was: a nudge's raw gaze (lessons.jsonl), else the click
+                  (screen runs only: on the panel there's no pointer)
+  fuse            70 % our tracker, 30 % SteamVR's eyes' mean (both live)
+  steamvr+clicks  SteamVR's, less the median error of the last SHIFT_KEEP clicks before
+With the fusion the run started with (start/gazed-fusion.json: gazecal.Fusion, fitted at our
+tracker's last calibration, or by --fit-fusion), what ft-gazed's GAZE_FUSION=on sends:
+  steamvr-affine  SteamVR's eyes' mean with its affine correction
+  steamvr-affine+clicks   ...plus the median of what the run's clicks before said was still off
+  fusion          per sample: ours (its eyes, weighted as ft-gazed does) mixed half and half with
+                  steamvr-affine where both our eyes are found, steamvr-affine alone where we
+                  have one, ours alone where SteamVR has no gaze; the look's median
+The same, fitted here on the other clicks (all runs given), for runs without one:
+  steamvr-affine-loo, fusion-loo     on every other click
+  ours-else-steam-loo                fusion-loo, but ours alone where both our eyes are found
+  steamvr-affine-xrun(+clicks), fusion-xrun(+clicks)   on the other runs' clicks only: a
+                  calibration from another sitting
+  fusion-loo:REPLAY   fusion-loo with our eyes from that replay (FUSE_REPLAYS, below), not
+                  live: a guard's worth with SteamVR covering the eyes it refuses
+Replays of our tracker on its finds (own.jsonl), from the calibration and the shift it had when
+the run started (start/), with ft-eyes' own rules (eyes_model.py), and nudges teaching it when
+they did live (lessons.jsonl):
+  own-replay      the live rules: check this against own-live (the "replay vs live" line)
+  no-jump         no glint jump: only clicks move the shift
+  both-jump       a glint jump counts only while both eyes see one
+  edge-guard      nudges past the calibration's dots (by EDGE_MARGIN) teach nothing
+  every-click     every click teaches it, not only the nudges
+  bridge          one eye lost: the other, plus how far it usually reads from the two together
+  guard-both      eyes_pupil.SizeGuard "both" over the finds (refused ones count as lost)
+  guard-long      SizeGuard "long"
+  no-guard        the finds the live SizeGuard refused (own.jsonl "guard": true) back in
+Each click is scored before anything it teaches. The breakdowns: by scenario, by where the target
+was (edge, corner or inside), by how far from straight ahead you looked, and by the time since
+the last nudge (run.json's last_lesson for the first).
+
+Needs numpy and (for the glint pairs) OpenCV: the tracker's Python, e.g.
+  scripts/in-box /opt/frametop/venv/bin/python gaze/bench/compare.py ~/.local/share/frametop/gaze/bench
+
+--fit-fusion OUT fits a gazecal.Fusion on all the clicks given (SteamVR's eyes' mean at each, and
+the target) and writes it to OUT, e.g. ~/.local/state/frametop/gaze/fusion.json for ft-gazed.
+"""
+import argparse
+import bisect
+import json
+import math
+import statistics
+import sys
+import time
+from collections import deque
+from pathlib import Path
+
+import numpy as np
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE.parent / "tracker"))
+sys.path.insert(0, str(HERE.parent))
+import eyes_model  # noqa: E402
+from gazecal import EyeWeights, Fusion  # noqa: E402
+
+LOOK = (0.35, 0.05)       # s before a click: as gazebench.py, for runs that don't say
+CLICK_BEFORE = 0.3        # ft-eyes: a click's frames are the 300 ms before it
+CLICK_MIN = 5             # ...at least this many
+FRESH = 0.03              # ft-eyes: an eye counts toward the output this long after its frame
+GAP = 3.0                 # ft-eyes: frames stopping this long is the headset off (a re-seat)
+HISTORY = 12.0            # ft-eyes: seconds of pupil positions kept per eye
+LESSON_MATCH = 1.5        # s between a click and its nudge's lesson
+EDGE_MARGIN = 2.0         # degrees past the calibration's outermost dots for edge-guard
+BRIDGE_TAU = 2.0          # s: bridge's average of each eye against the two together
+FUSE = 0.7                # fuse: our tracker's share
+SHIFT_KEEP = eyes_model.SHIFT_KEEP
+LIVE = ("own-live", "steamvr", "steamvr-eyes", "pointer", "fuse", "steamvr+clicks",
+        "steamvr-affine", "steamvr-affine+clicks", "fusion")
+CROSS = ("steamvr-affine-loo", "fusion-loo", "ours-else-steam-loo",
+         "steamvr-affine-xrun", "steamvr-affine-xrun+clicks", "fusion-xrun", "fusion-xrun+clicks")
+FUSE_REPLAYS = ("own-replay", "guard-both", "guard-long")  # fusion-loo:NAME, with that replay's eyes
+REPLAYS = {  # name: Replay options
+    "own-replay": {},
+    "no-jump": {"jump": None},
+    "both-jump": {"jump": "both"},
+    "edge-guard": {"guard": True},
+    "every-click": {"teach": "every"},
+    "bridge": {"bridge": True},
+    "guard-both": {"size": "both"},
+    "guard-long": {"size": "long"},
+    "no-guard": {"size": "none"},
+}
+ECC_BINS = ((0, 10, "<10°"), (10, 20, "10-20°"), (20, 30, "20-30°"), (30, 999, "30°+"))
+SINCE_BINS = ((0, 30, "<30 s"), (30, 120, "30-120 s"), (120, math.inf, "120 s+"))
+
+
+def jsonl(path):
+    try:
+        with open(path) as f:
+            for line in f:
+                try:
+                    yield json.loads(line)
+                except ValueError:
+                    continue
+    except FileNotFoundError:
+        return
+
+
+def unit(hy, hp):
+    """ft-gaze's head-relative yaw (+left), pitch (+up), degrees, as a unit vector (-Z ahead)."""
+    y, p = math.radians(hy), math.radians(hp)
+    return np.array([-math.sin(y) * math.cos(p), math.sin(p), -math.cos(y) * math.cos(p)])
+
+
+def angle(a, b):
+    if a is None or b is None:
+        return None
+    return math.degrees(math.acos(max(-1.0, min(1.0, float(unit(*a) @ unit(*b))))))
+
+
+def median2(rows):
+    rows = [r for r in rows if r is not None]
+    if not rows:
+        return None
+    return (statistics.median(r[0] for r in rows), statistics.median(r[1] for r in rows))
+
+
+def rays(s):
+    """A sample's rays with screen hits: (yaw, pitch, hit), head-relative."""
+    head = s.get("head") or {}
+    if head.get("hit"):
+        yield 0.0, 0.0, head["hit"]
+    for name, src in (s.get("src") or {}).items():
+        if src.get("hit") and "hy" in src:
+            yield src["hy"], src["hp"], src["hit"]
+        for e, h in zip(src.get("eyes") or (), src.get("ehit") or ()):
+            if e and h:
+                yield e[0], e[1], h
+
+
+class Run:
+    def __init__(self, d):
+        self.dir = Path(d)
+        self.info = json.loads((self.dir / "run.json").read_text())
+        st = self.info["start"]
+        self.wall = st["time"] - st["t_raw"]          # wall clock = raw + this
+        self.look = tuple(self.info.get("look") or LOOK)
+        self.scenario = self.info["scenario"]["id"]
+        self.samples = sorted(jsonl(self.dir / "gaze.jsonl"), key=lambda s: s.get("t", 0))
+        self.ts = [s.get("t", 0) for s in self.samples]
+        self.notes = []
+        self.replay_eyes = {}   # replay name: (times, [(left, right) eyes' gazes]) (replays())
+        self.lessons = sorted((r for r in jsonl(self.dir / "lessons.jsonl") if "refused" not in r),
+                              key=lambda r: r["time"])
+        self.panel = self.info.get("display") == "panel"
+        if self.panel:   # the targets' directions are in trials.jsonl
+            self.screen, self.sx, self.sy = None, 1.0, 1.0
+            self.trials = self.load_trials()
+            return
+        votes = {}
+        for s in self.samples:
+            hit = (s.get("head") or {}).get("hit")
+            if hit:
+                votes[hit["s"]] = votes.get(hit["s"], 0) + 1
+        self.screen = max(votes, key=votes.get) if votes else None
+        # ft-screens' pixels per window pixel: its size over the window's (gazebench.Bench.scale)
+        win, size = self.info.get("screen") or {}, (self.info.get("screens") or {}).get(str(self.screen))
+        if size and win.get("w") and win.get("h"):
+            self.sx, self.sy = size[0] / win["w"], size[1] / win["h"]
+        else:
+            self.sx = self.sy = win.get("dpr") or 1.0
+            self.notes.append("no screen sizes in run.json (a run before 2026-10-17): window pixels at Qt's "
+                              "pixel ratio, wrong where it isn't the desktop's scale")
+        self.trials = self.load_trials()
+
+    def load_trials(self):
+        """The trials that stand: the last click on each target, unless taken back or skipped. On a
+        screen run, each with the nudge's lesson that came with its click, if there was one."""
+        by_i = {}
+        for r in jsonl(self.dir / "trials.jsonl"):
+            if r.get("kind") == "trial":
+                by_i[r["i"]] = r
+            elif r.get("kind") == "undo":
+                by_i.pop(r["i"], None)
+        out = []
+        used = set()
+        for r in sorted(by_i.values(), key=lambda r: r["click"]["t"]):
+            if r.get("skipped") or "truth" not in r:
+                continue
+            r = dict(r)
+            if self.panel:
+                out.append(r)
+                continue
+            wall = r["click"]["t"] + self.wall
+            best = None
+            for k, ls in enumerate(self.lessons):
+                if k not in used and abs(ls["time"] - wall) <= LESSON_MATCH and (
+                        best is None or abs(ls["time"] - wall) < abs(self.lessons[best]["time"] - wall)):
+                    best = k
+            if best is not None:
+                used.add(best)
+                r["lesson"] = self.lessons[best]
+            out.append(r)
+        return out
+
+    def window(self, t0, t1):
+        return self.samples[bisect.bisect_left(self.ts, t0):bisect.bisect_right(self.ts, t1)]
+
+    def look_window(self, r):
+        """The click's look: before the press for a nudge (the lesson's look_t), else before the click."""
+        ls = r.get("lesson")
+        if ls and ls.get("look_t") is not None:
+            return ls["look_t"] - (self.look[0] - self.look[1]), ls["look_t"]
+        t = r["click"]["t"]
+        return t - self.look[0], t - self.look[1]
+
+    def direction(self, samples, px, py):
+        """Head-relative (yaw, pitch) of a point on the bench's screen (ft-screens' pixels): per
+        sample, from the hit nearest it, through that hit's pixels per degree; the median."""
+        out = []
+        for s in samples:
+            best = None
+            for hy, hp, h in rays(s):
+                if h.get("s") != self.screen or not h.get("j"):
+                    continue
+                d = math.hypot(h["x"] - px, h["y"] - py)
+                if best is None or d < best[0]:
+                    best = (d, hy, hp, h)
+            if best is None:
+                continue
+            _, hy, hp, h = best
+            j = h["j"]  # [dx/dhy, dy/dhy, dx/dhp, dy/dhp]
+            det = j[0] * j[3] - j[2] * j[1]
+            if abs(det) < 1e-9:
+                continue
+            ex, ey = px - h["x"], py - h["y"]
+            out.append((hy + (j[3] * ex - j[2] * ey) / det, hp + (-j[1] * ex + j[0] * ey) / det))
+        return median2(out)
+
+    def truth(self, r):
+        if "yaw" in r["truth"]:   # the panel's target
+            return r["truth"]["yaw"], r["truth"]["pitch"]
+        t0, t1 = self.look_window(r)
+        return self.direction(self.window(t0, t1), r["truth"]["x"] * self.sx, r["truth"]["y"] * self.sy)
+
+    def source(self, name, t0, t1):
+        rows = []
+        for s in self.window(t0, t1):
+            src = (s.get("src") or {}).get(name) or {}
+            if "hy" in src:
+                rows.append((src["hy"], src["hp"]))
+        return median2(rows)
+
+    def frames(self, size=None):
+        """Our tracker's finds, in time order: (t, cam, x, y, glint pair midpoint or None, gaze).
+        size: None, the live rules (the finds its SizeGuard refused are out); "none", those back
+        in; or a SizeGuard rule, run here over all of them instead."""
+        import eyes_pupil  # OpenCV
+        rows = sorted((r for r in jsonl(self.dir / "own.jsonl") if not r.get("lost") and "x" in r),
+                      key=lambda r: (r["t"], r["cam"]))
+        guard = eyes_pupil.SizeGuard(size) if size not in (None, "none") else None
+        out = []
+        for r in rows:
+            if size is None and r.get("guard"):
+                continue
+            if guard and not guard.keep(r["cam"], r["t"], r["a"]):
+                continue
+            pair = eyes_pupil.glint_pair({"x": r["x"], "y": r["y"], "glints": r.get("glints") or []})
+            out.append((r["t"], r["cam"], r["x"], r["y"], eyes_model.pair_mid(pair) if pair else None, r.get("gaze")))
+        return out
+
+    def fusion_inputs(self, t0, t1, replay=None):
+        """Per sample over [t0, t1]: (our eyes [left, right], SteamVR's left, right), for fusions.
+        Our eyes as published, or as a replay had them then (its newest output, if FRESH)."""
+        out = []
+        if replay:
+            rt, re = self.replay_eyes[replay]
+        for s in self.window(t0, t1):
+            src = s.get("src") or {}
+            if replay:
+                k = bisect.bisect_right(rt, s.get("t", 0)) - 1
+                eyes = list(re[k]) if k >= 0 and s.get("t", 0) - rt[k] < FRESH else [None, None]
+            else:
+                eyes = [tuple(e) if e else None for e in ((src.get("own") or {}).get("eyes") or [None, None])]
+            lr = [(src[n]["hy"], src[n]["hp"]) if "hy" in (src.get(n) or {}) else None for n in ("left", "right")]
+            out.append((eyes, lr[0], lr[1]))
+        return out
+
+
+class Replay:
+    """ft-eyes' classic tracker (Eye.feed, Tracker.command "click", the output loop) over a run's
+    finds, with variations."""
+
+    def __init__(self, run, frames, cal, shifts, status, clicks, jump="each", guard=False, bridge=False, box=None):
+        self.run, self.frames, self.cal = run, frames, cal
+        self.clicks = sorted(clicks, key=lambda c: c[0])  # (apply at, look t, (yaw, pitch))
+        self.jump, self.guard, self.bridge, self.box = jump, guard, bridge, box
+        self.shift = {}
+        for e in (0, 1):
+            sh = eyes_model.Shift.from_json(shifts[e]) if e in shifts else eyes_model.Shift()
+            st = ((status or {}).get("eyes") or {}).get(("right", "left")[e]) or {}
+            sh.reseated = bool(st.get("reseat", e not in shifts))
+            if st.get("jump") and jump is not None:
+                sh.jump = np.asarray(st.get("jump_px", (0, 0)), float)
+            self.shift[e] = sh
+        self.slip = {e: eyes_model.SlipTracker(cal, e, window=eyes_model.JUMP_WINDOW) for e in (0, 1)}
+        self.history = {0: deque(), 1: deque()}
+        self.gaze = {0: None, 1: None}
+        self.last_t = {0: None, 1: None}
+        self.off = {0: None, 1: None}
+        self.taught = self.skipped = 0
+        self.out_e = []   # with each output: the eyes' gazes in it, (left, right)
+
+    def value(self, e):
+        sh = self.shift[e]
+        if self.jump == "both" and not all(np.any(self.shift[o].jump) for o in (0, 1)):
+            return sh.base
+        return sh.value
+
+    def click(self, t, truth):
+        if self.guard and self.box and not (self.box[0] - EDGE_MARGIN <= truth[0] <= self.box[1] + EDGE_MARGIN
+                                            and self.box[2] - EDGE_MARGIN <= truth[1] <= self.box[3] + EDGE_MARGIN):
+            self.skipped += 1
+            return
+        both = all(np.any(self.shift[o].jump) for o in (0, 1))
+        for e in (0, 1):
+            rows = [h for h in self.history[e] if t - CLICK_BEFORE <= h[0] <= t]
+            if len(rows) < CLICK_MIN or not self.cal.has("where", e):
+                continue
+            pupil = np.median(np.array([h[1:3] for h in rows]), axis=0)
+            d = self.cal.click_shift(e, pupil, truth)
+            sh = self.shift[e]
+            # A jump in force (both-jump: in both eyes) means the headset moved: the older clicks are stale
+            if (both if self.jump == "both" else np.any(sh.jump)) or sh.reseated:
+                sh.meas, sh.reseated = [], False
+            sh.meas = (sh.meas + [np.asarray(d, float)])[-SHIFT_KEEP:]
+            sh.base = np.median(sh.meas, axis=0)
+            sh.jump, sh.held = np.zeros(2), None
+            if self.slip[e].value is not None:
+                sh.ref = np.asarray(self.slip[e].value, float)
+        self.taught += 1
+
+    def __call__(self):
+        """The output, (times, yaw, pitch) arrays, one per frame that changed it."""
+        out_t, out_g = [], []
+        ci = 0
+        last_out = None
+        for t, e, x, y, mid, _ in self.frames:
+            while ci < len(self.clicks) and self.clicks[ci][0] <= t:
+                self.click(*self.clicks[ci][1:])
+                ci += 1
+            if self.last_t[e] is not None and t - self.last_t[e] > GAP:
+                self.shift[e].reseat()
+            self.last_t[e] = t
+            h = self.history[e]
+            h.append((t, x, y))
+            while h[0][0] < t - HISTORY:
+                h.popleft()
+            if not self.cal.has("pupil", e):
+                continue
+            if mid is not None:
+                self.slip[e].add(t, (x, y), mid)
+            if self.jump is not None:
+                self.shift[e].glint(self.slip[e].get(t), t)
+            g = self.cal.gaze(e, x, y, self.value(e))
+            self.gaze[e] = (t, float(g[0]), float(g[1]))
+            latest = max(v[0] for v in self.gaze.values() if v)
+            use = {k: v[1:] for k, v in self.gaze.items() if v and latest - v[0] < FRESH}
+            out = self.cal.combine(use)
+            if self.bridge:
+                if len(use) == 2:
+                    a = 1.0 if last_out is None else 1 - math.exp(-(t - last_out) / BRIDGE_TAU)
+                    for k, v in use.items():
+                        d = np.asarray(out) - np.asarray(v)
+                        self.off[k] = d if self.off[k] is None else self.off[k] + a * (d - self.off[k])
+                elif len(use) == 1:
+                    (k, v), = use.items()
+                    if self.off[k] is not None:
+                        out = np.asarray(v) + self.off[k]
+            last_out = t
+            out_t.append(latest)
+            out_g.append((float(out[0]), float(out[1])))
+            self.out_e.append(tuple(None if k not in use else (float(use[k][0]), float(use[k][1])) for k in (1, 0)))
+        return out_t, out_g
+
+
+def calibration_box(run, cal):
+    """The calibration's dots' extent (yaw min, max, pitch min, max), from calibration-dots.jsonl."""
+    made = cal.info.get("made")
+    pts = [r["truth"] for r in jsonl(run.dir / "start" / "eyes-calibration-dots.jsonl") if r.get("made") == made]
+    if len(pts) < 4:
+        return (-20.0, 20.0, -20.0, 20.0)
+    return (min(p[0] for p in pts), max(p[0] for p in pts), min(p[1] for p in pts), max(p[1] for p in pts))
+
+
+def replays(run, truths, only):
+    """{name: function(t0, t1) -> (yaw, pitch) or None} for the replays, or {} with a note."""
+    want = [n for n in REPLAYS if not only or n in only]
+    if not want:
+        return {}
+    status = (run.info["start"] or {}).get("eyes")
+    if status is None:
+        run.notes.append("our tracker wasn't running at the start: no replays")
+        return {}
+    if status.get("feature", "pupil") != "pupil":
+        run.notes.append(f"our tracker ran {status.get('feature')}: replays are of the classic tracker only")
+        return {}
+    calp = run.dir / "start" / "eyes-calibration.json"
+    if not calp.exists():
+        run.notes.append("no calibration in start/: no replays")
+        return {}
+    cal = eyes_model.Calibration.load(calp)
+    shifts = {}
+    try:
+        st = json.loads((run.dir / "start" / "eyes-state.json").read_text())
+        if st.get("calibration") == cal.info.get("made"):
+            shifts = {int(k): v for k, v in st.get("shift", {}).items()}
+    except (OSError, ValueError):
+        pass
+    frames = {}
+    if not run.frames():
+        run.notes.append("no finds in own.jsonl: no replays")
+        return {}
+    nudges = [(ls["time"] - run.wall, ls["look_t"], tuple(ls["true"])) for ls in run.lessons
+              if ls.get("kind") == "own" and ls.get("look_t") is not None]
+    every = [(r["click"]["t"], r["click"]["t"] - run.look[1], truths[id(r)]) for r in run.trials
+             if truths.get(id(r)) is not None]
+    box = calibration_box(run, cal)
+    out = {}
+    for name in want:
+        opt = dict(REPLAYS[name])
+        teach = every if opt.pop("teach", None) == "every" else nudges
+        size = opt.pop("size", None)
+        if size not in frames:
+            frames[size] = run.frames(size)
+        replay = Replay(run, frames[size], cal, shifts, status, teach, box=box, **opt)
+        t, g = replay()
+        run.replay_eyes[name] = (t, replay.out_e)
+
+        def look(t0, t1, t=t, g=g):
+            return median2(g[bisect.bisect_left(t, t0):bisect.bisect_right(t, t1)])
+        out[name] = look
+    return out
+
+
+def fuse_look(inputs, weights, fusion, share=None, clicks=False):
+    """Fusion.combine per sample (fusion_inputs), ours combined as ft-gazed does; the look's median."""
+    out = []
+    for eyes, left, right in inputs:
+        own = weights.combine(eyes)
+        out.append(fusion.combine(own, sum(e is not None for e in eyes), fusion.steam_gaze(left, right, clicks), share))
+    return median2(out)
+
+
+def own_weights(run):
+    """ft-gazed's weights for our tracker's eyes when the run started (its lessons' misses)."""
+    try:
+        misses = json.loads((run.dir / "start" / "gazed-pointer-lessons.json").read_text())["misses"]["own"]
+    except (OSError, ValueError, KeyError, TypeError):
+        misses = None
+    return EyeWeights(((run.info["start"] or {}).get("gazed") or {}).get("eye_bias", "auto"), misses)
+
+
+def score(run, only=None):
+    """A record per click: where it was, and each technique's error (degrees, or None)."""
+    truths = {id(r): run.truth(r) for r in run.trials}
+    rep = replays(run, truths, only)
+    last_lesson = run.info.get("last_lesson")
+    weights = own_weights(run)
+    fusion = Fusion.load(run.dir / "start" / "gazed-fusion.json")
+    if fusion and not fusion.ready():
+        fusion = None
+    if fusion:
+        fusion.wear = run.dir.name
+    recs, steam_offsets = [], []
+    for r in run.trials:
+        truth = truths[id(r)]
+        t0, t1 = run.look_window(r)
+        own, steam = run.source("own", t0, t1), run.source("mmap1", t0, t1)
+        left, right = run.source("left", t0, t1), run.source("right", t0, t1)
+        mean = Fusion.mean(left, right)
+        inputs = run.fusion_inputs(t0, t1)
+        g = {"own-live": own, "steamvr": steam, "steamvr-eyes": mean}
+        ls = r.get("lesson")
+        if ls:
+            g["pointer"] = tuple(ls["raw"])
+        elif not run.panel:
+            c = r["click"]
+            g["pointer"] = run.direction(run.window(t0, t1), c["x"] * run.sx, c["y"] * run.sy)
+        g["fuse"] = (tuple(FUSE * a + (1 - FUSE) * b for a, b in zip(own, mean)) if own and mean else None)
+        if fusion:
+            g["steamvr-affine"] = fusion.steam_gaze(left, right, clicks=False)
+            g["steamvr-affine+clicks"] = fusion.steam_gaze(left, right)
+            g["fusion"] = fuse_look(inputs, weights, fusion)
+            if truth:
+                fusion.click(left, right, *truth)
+        if steam and steam_offsets:
+            keep = steam_offsets[-SHIFT_KEEP:]
+            g["steamvr+clicks"] = (steam[0] + statistics.median(o[0] for o in keep),
+                                   steam[1] + statistics.median(o[1] for o in keep))
+        else:
+            g["steamvr+clicks"] = steam
+        if steam and truth:
+            steam_offsets.append((truth[0] - steam[0], truth[1] - steam[1]))
+        for name, look in rep.items():
+            g[name] = look(t0, t1)
+        wall = r["click"]["t"] + run.wall
+        before = [ls_["time"] for ls_ in run.lessons if ls_["time"] < wall - LESSON_MATCH]
+        last = before[-1] if before else last_lesson
+        u, v = r["u"], r["v"]
+        edges = sum(1 for w in (u, v) if w in (0.0, 1.0))
+        rec = {"run": run.dir.name, "scenario": run.scenario, "i": r["i"], "u": u, "v": v,
+               "where": ("corner" if edges == 2 else "edge" if edges else "inside"),
+               "truth": truth, "ecc": math.hypot(*truth) if truth else None,
+               "since": None if last is None else wall - last, "nudged": bool(ls),
+               "err": {k: angle(val, truth) for k, val in g.items() if not only or k in only},
+               "_fz": {"left": left, "right": right, "mean": mean, "inputs": inputs, "weights": weights,
+                       "replays": {n: run.fusion_inputs(t0, t1, n) for n in FUSE_REPLAYS if n in run.replay_eyes}}}
+        if ls:
+            c = r["click"]
+            cd = run.direction(run.window(t0, t1), c["x"] * run.sx, c["y"] * run.sy)
+            rec["check"] = angle(cd, tuple(ls["true"]))
+        if g.get("own-replay") and own:
+            rec["replay_vs_live"] = angle(g["own-replay"], own)
+        recs.append(rec)
+    return recs
+
+
+def cross(recs, only=None):
+    """CROSS: SteamVR's affine and the fusion fitted on the other clicks (every other one: -loo;
+    the other runs' only: -xrun), each record's errors with them."""
+    want = [n for n in CROSS if not only or n in only]
+    have = [r for r in recs if r["truth"] and r["_fz"]["mean"]]
+    if not want or not have:
+        return
+    pts = {id(r): (*r["_fz"]["mean"], *r["truth"]) for r in have}
+    xrun = {}
+    for name in dict.fromkeys(r["run"] for r in recs):
+        xrun[name] = Fusion()
+        xrun[name].fit([pts[id(r)] for r in have if r["run"] != name])
+        xrun[name].wear = name
+    for r in recs:
+        fz, g = r["_fz"], {}
+        loo = Fusion()
+        loo.fit([pts[id(q)] for q in have if q is not r])
+        if loo.ready():
+            g["steamvr-affine-loo"] = loo.steam_gaze(fz["left"], fz["right"])
+            g["fusion-loo"] = fuse_look(fz["inputs"], fz["weights"], loo)
+            g["ours-else-steam-loo"] = fuse_look(fz["inputs"], fz["weights"], loo, share=1.0)
+            for name, inputs in fz["replays"].items():
+                g[f"fusion-loo:{name}"] = fuse_look(inputs, fz["weights"], loo)
+        x = xrun[r["run"]]
+        if x.ready():
+            g["steamvr-affine-xrun"] = x.steam_gaze(fz["left"], fz["right"], clicks=False)
+            g["steamvr-affine-xrun+clicks"] = x.steam_gaze(fz["left"], fz["right"])
+            g["fusion-xrun"] = fuse_look(fz["inputs"], fz["weights"], x)
+            g["fusion-xrun+clicks"] = fuse_look(fz["inputs"], fz["weights"], x, clicks=True)
+            if r["truth"]:
+                x.click(fz["left"], fz["right"], *r["truth"])
+        for k, v in g.items():
+            if not only or k in only:
+                r["err"][k] = angle(v, r["truth"])
+
+
+def fit_fusion(recs, out, runs):
+    """A Fusion fitted on all the clicks, written to out."""
+    pts = [(*r["_fz"]["mean"], *r["truth"]) for r in recs if r["truth"] and r["_fz"]["mean"]]
+    f = Fusion()
+    f.fit(pts, {"how": "bench", "runs": runs, "made": time.strftime("%Y-%m-%d %H:%M:%S")})
+    if not f.ready():
+        sys.exit(f"--fit-fusion: only {len(pts)} clicks with SteamVR's two eyes (need {Fusion.MIN})")
+    f.save(out)
+    print(f"\nfusion fitted on {len(pts)} clicks from {len(runs)} runs (SteamVR's affine leaves "
+          f"{statistics.median(f.misses()):.2f}° at the median on them): {out}")
+
+
+def bin_of(value, bins):
+    if value is None:
+        return "none" if bins is SINCE_BINS else "?"
+    return next((name for lo, hi, name in bins if lo <= value < hi), "?")
+
+
+def table(title, recs, key, names):
+    groups = {}
+    for r in recs:
+        groups.setdefault(key(r), []).append(r)
+    cols = list(groups)
+    if not cols:
+        return
+    w = max(len(n) for n in names) + 2
+    print(f"\n{title}: median degrees off (clicks)")
+    print(" " * w + "".join(f"{c:>13}" for c in cols))
+    for n in names:
+        cells = []
+        for c in cols:
+            errs = sorted(e for e in (r["err"].get(n) for r in groups[c]) if e is not None)
+            cells.append(f"{statistics.median(errs):6.2f} ({len(errs):3d})" if errs else f"{'-':>12}")
+        print(f"{n:<{w}}" + "".join(f"{x:>13}" for x in cells))
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    ap.add_argument("runs", nargs="+", help="run folders, or folders of them")
+    ap.add_argument("--per-click", action="store_true", help="a line per click")
+    ap.add_argument("--json", help="write the per-click records here")
+    ap.add_argument("--only", help="comma-separated techniques")
+    ap.add_argument("--fit-fusion", metavar="OUT", help="fit a gazecal.Fusion on all the clicks, write it here")
+    a = ap.parse_args()
+    only = set(a.only.split(",")) if a.only else None
+    dirs = []
+    for p in map(Path, a.runs):
+        dirs += [p] if (p / "run.json").exists() else sorted(q.parent for q in p.glob("*/run.json"))
+    recs = []
+    for d in dirs:
+        run = Run(d)
+        got = score(run, only)
+        recs += got
+        print(f"{d.name}: {len(got)} clicks, {sum(r['nudged'] for r in got)} nudged"
+              + "".join(f"; {n}" for n in run.notes))
+    if not recs:
+        sys.exit("no clicks")
+    cross(recs, only)
+    names = [n for n in LIVE + CROSS + tuple(f"fusion-loo:{n}" for n in FUSE_REPLAYS) + tuple(REPLAYS)
+             if any(r["err"].get(n) is not None for r in recs)]
+    print()
+    w = max(len(n) for n in names) + 2
+    print(f"{'':<{w}}{'median':>8}{'p90':>8}{'mean':>8}{'n':>6}")
+    for n in names:
+        errs = sorted(e for e in (r["err"].get(n) for r in recs) if e is not None)
+        p90 = errs[min(len(errs) - 1, int(0.9 * len(errs)))]
+        print(f"{n:<{w}}{statistics.median(errs):8.2f}{p90:8.2f}{statistics.fmean(errs):8.2f}{len(errs):6d}")
+    apart = sorted(r["replay_vs_live"] for r in recs if r.get("replay_vs_live") is not None)
+    if apart:
+        p90 = apart[min(len(apart) - 1, int(0.9 * len(apart)))]
+        print(f"\nreplay vs live: our tracker's replay is {statistics.median(apart):.2f}° from what it published, "
+              f"at the median ({p90:.2f}° p90, {len(apart)} clicks); the replays are only as good as this")
+    checks = [r["check"] for r in recs if r.get("check") is not None]
+    if checks:
+        print(f"screen to direction vs the pointer's own: {statistics.median(checks):.2f}° apart at the median "
+              f"({len(checks)} nudges)")
+    table("By scenario", recs, lambda r: r["scenario"], names)
+    table("By where the target was", recs, lambda r: r["where"], names)
+    table("By how far from straight ahead", recs, lambda r: bin_of(r["ecc"], ECC_BINS), names)
+    table("By time since the last nudge", recs, lambda r: bin_of(r["since"], SINCE_BINS), names)
+    if a.per_click:
+        print()
+        for r in recs:
+            print(f"{r['run']} #{r['i']:<3} {r['where']:<6} ecc {r['ecc'] or 0:5.1f}°"
+                  + (" nudged" if r["nudged"] else "       ")
+                  + "".join(f"  {n} {r['err'][n]:.2f}" for n in names if r["err"].get(n) is not None))
+    if a.fit_fusion:
+        fit_fusion(recs, a.fit_fusion, [d.name for d in dirs])
+    if a.json:
+        Path(a.json).write_text(json.dumps([{k: v for k, v in r.items() if k != "_fz"} for r in recs], indent=1))
+
+
+if __name__ == "__main__":
+    main()

gaze/bench/ft-gazebench

@@ -0,0 +1,18 @@
+#!/bin/bash
+# Launch the Frametop Gaze Bench from a Plasma session on the Frame host (its app menu entry,
+# gaze/bench/install.sh), as the eye recorder's gaze/rec/ft-eyerec: the app runs in Frametop's
+# container (PySide6), on the session's Wayland socket and bus; podman gets the real
+# XDG_RUNTIME_DIR and user bus.
+# Options go to gazebench.py: --base DIR (where runs go), --no-images, --no-button.
+here=$(cd "$(dirname "$(readlink -f "$0")")" && pwd)
+wl=${WAYLAND_DISPLAY:-wayland-0}
+case $wl in /*) ;; *) wl="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/$wl" ;; esac
+session_bus=${DBUS_SESSION_BUS_ADDRESS:-}
+export XDG_RUNTIME_DIR=/run/user/$(id -u)
+export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
+cd "$HOME" || exit 1
+exec "$here/../../scripts/in-box" env PATH="/opt/frametop/venv/bin:/opt/frametop/bin:$PATH:/usr/sbin:/sbin" \
+  WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
+  XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
+  QT_QPA_PLATFORM="wayland;xcb" ${FT_GAZE:+FT_GAZE="$FT_GAZE"} ${FT_BENCH_BASE:+FT_BENCH_BASE="$FT_BENCH_BASE"} \
+  python3 "$here/gazebench.py" "$@"

gaze/bench/ft-gazebench.desktop

@@ -0,0 +1,9 @@
+[Desktop Entry]
+Type=Application
+Name=Frametop Gaze Bench
+GenericName=Test gaze tracking on hard targets
+Comment=Look at dots in hard places on a panel in the headset and press its button, to compare gaze techniques
+Exec=@REPO@/gaze/bench/ft-gazebench
+Icon=view-visible
+Categories=Utility;
+Keywords=eyes;eye tracking;gaze;test;bench;frametop;

gaze/bench/gazebench.py

@@ -0,0 +1,851 @@
+#!/usr/bin/env python3
+"""Frametop Gaze Bench: targets in hard places for gaze tracking, and a recording of every look
+at them, so gaze techniques can be compared on the same looks afterwards (README.md).
+
+The targets show on a panel fixed to the headset, our own ft-gazepanel in the eye recorder's
+mode (gaze/rec/session.py): wherever the head turns, a target stays at the same direction from
+it, so every run asks the same of the eyes. Pick a scenario (scenarios.json) in the window; the
+panel shows one dot at a time. Look at the dot and press the button on the headset's right side:
+where you looked is the dot. Two presses take the last one back, a hold ends the scenario. The
+window's keys do the same: Space marks, Backspace takes back, S skips, Esc ends. As in Frametop's
+gaze calibration, a left click, Enter, an index pinch (hand tracking) or Meta+J marks too, and a
+right click or Meta+K ends: the bench claims the pointer helper's calibration panel while its
+panel shows (pointer/helper/calpanel.h), so the helper answers those here and hides the pointer.
+
+Each scenario played is a run, recorded by benchrec.py into BASE/<time>-<scenario>/, with the
+panel's own log of what the dot did and when it reached the eyes, and the headset's pose. After
+each mark the window works out how far our tracker and SteamVR were off (their median over the
+LOOK before the press) for the summary; compare.py replays every technique on the recording.
+
+Runs in Frametop's container (PySide6), launched by ft-gazebench. --base DIR, --no-images.
+"""
+import argparse
+import json
+import math
+import os
+import random
+import signal
+import socket
+import statistics
+import subprocess
+import sys
+import threading
+import time
+from pathlib import Path
+
+from PySide6.QtCore import Property, QObject, QSocketNotifier, QTimer, QUrl, Signal, Slot
+from PySide6.QtGui import QGuiApplication
+from PySide6.QtQml import QQmlApplicationEngine
+from PySide6.QtQuickControls2 import QQuickStyle
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+GAZE = os.path.dirname(HERE)
+REPO = os.path.dirname(GAZE)
+sys.path.insert(0, HERE)
+sys.path.insert(0, REPO)
+import benchrec  # noqa: E402
+from recorder.button import ButtonReader, find_button, read_input_devices  # noqa: E402
+from recorder.common import in_container  # noqa: E402
+
+PANEL_DEG, PANEL_ASPECT = 64.0, 0.75   # ft-gazepanel's rec panel (kFullDeg, 4:3), as gaze/rec/session.py
+MARGIN = 1.5            # degrees from the panel's edge to a target at 0 or 1, measured in the panel's plane
+                        # from straight ahead: about 1.1 out at the edges, where the dot's ring (0.75
+                        # out from its middle) still fits
+PANEL_SOCK = os.environ.get("FT_BENCH_PANEL_SOCKET", "ft_benchpanel")   # (the tests' stand-in)
+PANEL_OVERLAY = "frametop.benchpanel"
+HELPER_SOCK = os.environ.get("FT_BENCH_HELPER_SOCKET", "ft_pointer_helper")   # (the tests' stand-in)
+CLAIM_EVERY = 1.0       # s between claims of the helper's calibration panel (it lets one go after 3 s)
+PANELS = [Path(p) for p in (os.environ.get("FT_GAZEPANEL"), os.path.join(GAZE, "build", "ft-gazepanel"),
+                            Path.home() / ".local/share/frametop/releases/current/gaze/build/ft-gazepanel") if p]
+IN_BOX = os.path.join(REPO, "scripts", "in-box")
+PANEL_WAIT = 20.0       # s for a new panel to answer (SteamVR's overlays can take a while)
+LOOK = (0.4, 0.1)       # s before a press: the gaze samples a mark takes. It ends a little before the
+                        # press (a click's was 0.05 s): the push on the headset's side comes before it
+DONE_S = 0.25           # s a dot marked from the window's keys shows done before the next (the
+                        # button's shows done from its down, a double-press time before)
+WAKE = 60               # s: the gaze service's wake lease while a run records (Input Settings' Gaze
+                        # page does the same), so the trackers run whether gaze mode is on or not
+# SteamVR's is the mean of its two eyes (set 2's left and right): on the 2026-10-17 runs it was
+# nearer than its combined gaze (mmap1), and it's what ft-gazed's fusion corrects.
+QUICK_SOURCES = (("own", "Our tracker"), ("steam-eyes", "SteamVR"))
+
+
+def quick_gaze(src, key):
+    """A sample's (yaw, pitch) for a QUICK_SOURCES key, or None."""
+    if key == "steam-eyes":
+        left, right = src.get("left") or {}, src.get("right") or {}
+        if "hy" in left and "hy" in right:
+            return (left["hy"] + right["hy"]) / 2, (left["hp"] + right["hp"]) / 2
+        return None
+    g = src.get(key) or {}
+    return (g["hy"], g["hp"]) if "hy" in g and g.get("ok", 1) != 0 else None
+KEYS_START = "Press the button, click, Enter or pinch to start. Hold the button or right click to end."
+KEYS_END = "Press the button, click, Enter or pinch to close this."
+
+
+def boxed(argv):
+    """A program built in the dev container: run it there (scripts/in-box from the host)."""
+    return list(argv) if in_container() or not os.path.exists(IN_BOX) else [IN_BOX] + list(argv)
+
+
+def margins(margin=MARGIN):
+    """MARGIN as fractions of the panel across and down."""
+    half = math.tan(math.radians(PANEL_DEG / 2))
+    m = math.tan(math.radians(margin)) / (2 * half)
+    return m, m / PANEL_ASPECT
+
+
+def panel_direction(u, v, margin=MARGIN):
+    """The head-relative direction (yaw +left, pitch +up, degrees) of a target u across and v down
+    the panel, 0 and 1 being margin degrees in from its edges: ft-gazepanel's ToPixel backwards."""
+    half = math.tan(math.radians(PANEL_DEG / 2))
+    mx, my = margins(margin)
+    x, y = mx + u * (1 - 2 * mx), my + v * (1 - 2 * my)
+    yaw = math.atan((0.5 - x) * 2 * half)
+    pitch = math.atan((0.5 - y) * 2 * half * PANEL_ASPECT * math.cos(yaw))
+    return math.degrees(yaw), math.degrees(pitch)
+
+
+def panel_place(yaw, pitch, margin=MARGIN):
+    """panel_direction backwards: u, v for a direction."""
+    half = math.tan(math.radians(PANEL_DEG / 2))
+    mx, my = margins(margin)
+    y_r = math.radians(yaw)
+    x = 0.5 - math.tan(y_r) / (2 * half)
+    y = 0.5 - math.tan(math.radians(pitch)) / math.cos(y_r) / (2 * half) / PANEL_ASPECT
+    return (x - mx) / (1 - 2 * mx), (y - my) / (1 - 2 * my)
+
+
+def unit(yaw, pitch):
+    """ft-gaze's head-relative yaw (+left), pitch (+up), degrees, as a unit vector (-Z ahead)."""
+    y, p = math.radians(yaw), math.radians(pitch)
+    return (-math.sin(y) * math.cos(p), math.sin(p), -math.cos(y) * math.cos(p))
+
+
+def angle(a, b):
+    """Degrees between two (yaw, pitch) directions."""
+    d = sum(x * y for x, y in zip(unit(*a), unit(*b)))
+    return math.degrees(math.acos(max(-1.0, min(1.0, d))))
+
+
+def make_trials(sc, rng):
+    """The scenario's targets, in order: dicts with u, v (fractions of the panel across and down,
+    panel_direction), and maybe bg (the panel's brightness, 0 to 1) and dwell (s)."""
+    spec = sc["targets"]
+    pts = []
+    if "grid" in spec:
+        cols, rows = spec["grid"]
+        lo, hi = spec.get("span", (0.2, 0.8))
+        at = lambda k, n: lo + (hi - lo) * k / (n - 1) if n > 1 else 0.5  # noqa: E731
+        pts = [{"u": at(i, cols), "v": at(j, rows)} for j in range(rows) for i in range(cols)]
+    elif "edges" in spec:
+        n = spec["edges"]
+        seen = []
+        for k in range(n):
+            t = k / (n - 1)
+            for p in ((t, 0.0), (t, 1.0), (0.0, t), (1.0, t)):
+                if p not in seen:
+                    seen.append(p)
+        pts = [{"u": u, "v": v} for u, v in seen]
+    elif "rays" in spec:
+        dirs = spec.get("dirs", 8)
+        for f in spec["rays"]:
+            for k in range(dirs):
+                a = 2 * math.pi * k / dirs
+                c, s = math.cos(a), math.sin(a)
+                m = max(abs(c), abs(s))  # so 1 reaches the edge, or the corner on a diagonal
+                pts.append({"u": 0.5 + 0.5 * f * c / m, "v": 0.5 - 0.5 * f * s / m})
+    elif "random" in spec:
+        for _ in range(spec["random"]):
+            u, v = rng.random(), rng.random()
+            if rng.random() < spec.get("edge_share", 0.0):
+                if rng.random() < 0.5:
+                    u = float(round(u))
+                else:
+                    v = float(round(v))
+            pts.append({"u": u, "v": v})
+    elif "steps" in spec:
+        # A walk of small steps from the middle: each target lo to hi degrees from the last, within
+        # the box (degrees either side of straight ahead)
+        lo, hi = spec.get("size", (2.0, 4.0))
+        by, bp = spec.get("box", (12.0, 9.0))
+        at = (0.0, 0.0)
+        for _ in range(spec["steps"]):
+            for _ in range(100):
+                a, d = rng.uniform(0, 2 * math.pi), rng.uniform(lo, hi)
+                nxt = (at[0] + d * math.cos(a), at[1] + d * math.sin(a))
+                if abs(nxt[0]) <= by and abs(nxt[1]) <= bp:
+                    break
+            at = nxt
+            u, v = panel_place(*at)
+            pts.append({"u": u, "v": v})
+    if sc.get("shuffle"):
+        rng.shuffle(pts)
+    for i, p in enumerate(pts):
+        if sc.get("backgrounds"):
+            p["bg"] = sc["backgrounds"][i % len(sc["backgrounds"])]
+        if sc.get("dwell"):
+            p["dwell"] = sc["dwell"]
+    return pts
+
+
+def git_rev():
+    try:
+        return subprocess.run(["git", "-C", HERE, "describe", "--always", "--dirty"], capture_output=True,
+                              text=True, timeout=5).stdout.strip() or None
+    except (OSError, subprocess.TimeoutExpired):
+        return None
+
+
+class Panel:
+    """Our own ft-gazepanel in its rec mode (as gaze/rec/session.py runs one): 64 degrees across,
+    fixed to the headset, a solid background and one dot. Commands go to its control socket
+    without waiting; it answers each one, in order, and the answers are read later (errors
+    logged), or waited for by ask()."""
+
+    def __init__(self, prog=None, log=print, no_vr=False):
+        self.prog = Path(prog) if prog else next((p for p in PANELS if p.exists()), PANELS[0])
+        self.log, self.no_vr = log, no_vr
+        self.proc = None
+        self.ready = False
+        self.error = None
+        self.sent = {}
+        self.pending = 0     # commands sent and not answered yet
+        self.lock = threading.Lock()
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_CLOEXEC)
+        self.sock.bind("")
+        self.sock.setblocking(False)
+
+    def start(self):
+        """Start it and wait for its answer: call from a thread (SteamVR can take a while)."""
+        self.ready, self.error = False, None
+        if not self.prog.exists():
+            self.error = f"{self.prog} isn't there: run gaze/build.sh in the dev container"
+            return False
+        argv = [str(self.prog), "--watch-stdin", "--socket", PANEL_SOCK, "--overlay", PANEL_OVERLAY]
+        try:
+            self.proc = subprocess.Popen(boxed(argv + (["--no-vr"] if self.no_vr else [])), stdin=subprocess.PIPE,
+                                         stdout=subprocess.DEVNULL, start_new_session=True)
+        except OSError as e:
+            self.error = f"can't start {self.prog.name}: {e}"
+            return False
+        self.log(f"started {self.prog} (pid {self.proc.pid})")
+        end = time.monotonic() + PANEL_WAIT
+        while time.monotonic() < end:
+            if self.ask("ping", 0.3):
+                self.ready = True
+                return True
+            if self.proc.poll() is not None:
+                self.error = f"{self.prog.name} exited ({self.proc.returncode}): is SteamVR running?"
+                return False
+        self.error = f"{self.prog.name} doesn't answer"
+        self.stop()
+        return False
+
+    def alive(self):
+        return self.ready and self.proc is not None and self.proc.poll() is None
+
+    def _got(self, r):
+        self.pending = max(0, self.pending - 1)
+        if r.startswith("error"):
+            self.log(f"panel: {r}")
+
+    def _drain(self):
+        while True:
+            try:
+                r = self.sock.recv(65536).decode(errors="replace")
+            except OSError:
+                return
+            self._got(r)
+
+    def cmd(self, text):
+        with self.lock:
+            self._drain()
+            try:
+                self.sock.sendto(text.encode(), "\0" + PANEL_SOCK)
+                self.pending += 1
+            except OSError:
+                pass
+
+    def ask(self, text, timeout=0.5):
+        """Send, and its answer (or None): the one after those of the commands still unanswered."""
+        with self.lock:
+            self._drain()
+            try:
+                self.sock.sendto(text.encode(), "\0" + PANEL_SOCK)
+            except OSError:
+                return None
+            self.pending += 1
+            end = time.monotonic() + timeout
+            while (left := end - time.monotonic()) > 0:
+                try:
+                    self.sock.settimeout(left)
+                    r = self.sock.recv(65536).decode(errors="replace")
+                except OSError:
+                    break
+                finally:
+                    self.sock.setblocking(False)
+                last = self.pending == 1
+                self._got(r)
+                if last:
+                    return r
+            self.pending = 0   # a lost answer: count afresh
+            return None
+
+    def set(self, key, text):
+        """Send only if it changes that part of the panel."""
+        if self.sent.get(key) != text:
+            self.sent[key] = text
+            self.cmd(text)
+
+    def show(self, bg):
+        self.cmd("show rec")   # an empty panel
+        self.sent = {}
+        self.set("bg", f"bg {bg:.3f}")
+
+    def lines(self, title="", text="", keys="", step=""):
+        for key, value in (("title", title), ("text", text), ("keys", keys), ("step", step)):
+            self.set(key, f"{key} {value}" if value else key)
+
+    def dot(self, yaw, pitch, state, progress=None):
+        self.set("dot", f"dot {yaw:.4f} {pitch:.4f} {state}" + ("" if progress is None else f" {progress:.2f}"))
+
+    def close(self):
+        self.sock.close()
+
+    def stop(self):
+        p, self.proc, self.ready = self.proc, None, False
+        if not p:
+            return
+        try:
+            p.stdin.close()   # --watch-stdin: it quits
+            p.wait(5)
+        except (OSError, subprocess.TimeoutExpired):
+            try:
+                os.killpg(p.pid, signal.SIGTERM)
+                p.wait(3)
+            except (OSError, subprocess.TimeoutExpired):
+                p.kill()
+        self.log(f"panel stopped (exit {p.returncode})")
+
+
+def text_line(text):
+    """One line for the panel's socket: its own line breaks are "|"."""
+    return " ".join(text.replace("\n", "|").split())
+
+
+class Bench(QObject):
+    changed = Signal()
+    statusChanged = Signal()
+    # The headset button's events, from its reader's thread: "down", "press", "double", "hold",
+    # and the raw time of the press they're about (its down)
+    button = Signal(str, float)
+
+    def __init__(self, scenarios, base, images, gaze_prog=None, panel_prog=None, panel_no_vr=False, button=None):
+        super().__init__()
+        self.all = scenarios
+        self.base = base
+        self._images = images
+        self._page = "menu"
+        self._status = "Looking for the gaze service..."
+        self._progress = self._message = self._summary = self._last = self._notice = ""
+        self.bg = float(scenarios.get("background", 0.15))
+        self.gaze = benchrec.GazeStream(prog=gaze_prog, log=self.log).start()
+        self.gazed = benchrec.Sock(benchrec.GAZED_SOCK)
+        self.panel = Panel(panel_prog, self.log, panel_no_vr)
+        self.panel_starting = None
+        self.run = self.sc = None
+        self.stopping = None       # the last run's stop (it copies the logs), for shutdown to wait for
+        self.trials, self.i, self.results = [], 0, []
+        # A run's phase: intro (its message, until a press), target (a dot to look at), marked (its
+        # done shows), gap (no dot, a scenario's gap), end (the summary on the panel, until a press)
+        self.phase = None
+        self.target = None         # the dot showing: (yaw, pitch)
+        self.shown_t = self.ready_t = None
+        self.flashed = False       # the button's down showed this dot done before its press came
+        self.next_at = None        # monotonic: when the marked or gap phase ends
+        self.rev = git_rev()
+        self.button.connect(self.on_button)
+        self.down_t = None         # the reader's thread: its last down's raw time
+        self.reader = None
+        path = find_button(read_input_devices()) if button is None else button
+        if path:
+            self.reader = ButtonReader(path, self._gesture, self.log, on_down=self._down).start()
+        self.button_path = path or None
+        # The pointer helper's calibration panel (see the top): claimed as @ft_gazebench.PID while
+        # a run or its summary shows; its answers ("calaccept", "calquit") come back there.
+        self.claim_name = f"ft_gazebench.{os.getpid()}"
+        self.claim_at = None       # monotonic: the last claim; None while not claiming
+        self.helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK | socket.SOCK_CLOEXEC)
+        self.helper_note = None
+        try:
+            self.helper.bind("\0" + self.claim_name)
+            self.helper_note = QSocketNotifier(self.helper.fileno(), QSocketNotifier.Type.Read, self)
+            self.helper_note.activated.connect(self.from_helper)
+        except OSError as e:
+            self.log(f"pointer helper socket: {e} (clicks, Enter and pinches won't mark)")
+        self.timer = QTimer(self)
+        self.timer.timeout.connect(self.tick)
+        self.timer.start(50)
+        self.status_at = 0.0
+        self.wake_at = 0.0
+        self.status_busy = False
+        self.status_thread = None
+        self.start_panel()
+
+    def log(self, line):
+        print("ft-gazebench: " + line, file=sys.stderr, flush=True)
+
+    def start_panel(self):
+        if self.panel_starting and self.panel_starting.is_alive():
+            return
+        self.panel_starting = threading.Thread(target=self._start_panel, daemon=True)
+        self.panel_starting.start()
+
+    def _start_panel(self):
+        if not self.panel.start():
+            self.log(f"panel: {self.panel.error}")
+        self.refresh_status()
+
+    # --- properties for bench.qml
+
+    def _get(name):  # noqa: N805
+        return lambda self: getattr(self, name)
+
+    page = Property(str, _get("_page"), notify=changed)
+    progress = Property(str, _get("_progress"), notify=changed)
+    message = Property(str, _get("_message"), notify=changed)
+    summary = Property(str, _get("_summary"), notify=changed)
+    last = Property(str, _get("_last"), notify=changed)
+    notice = Property(str, _get("_notice"), notify=changed)
+    status = Property(str, _get("_status"), notify=statusChanged)
+
+    def _set_images(self, v):
+        self._images = bool(v)
+        self.changed.emit()
+
+    images = Property(bool, _get("_images"), _set_images, notify=changed)
+
+    @Property("QVariantList", constant=True)
+    def scenarios(self):
+        out = []
+        for sc in self.all["scenarios"]:
+            n = len(make_trials(sc, random.Random(0)))
+            out.append({"id": sc["id"], "name": sc["name"], "about": sc["about"], "count": n})
+        return out
+
+    # --- status (in a thread: the sockets wait for replies)
+
+    def refresh_status(self):
+        if self.status_busy:
+            return
+        self.status_busy = True
+        self.status_thread = threading.Thread(target=self._status_thread, daemon=True)
+        self.status_thread.start()
+
+    def _status_thread(self):
+        try:
+            self._status = self.status_text()
+        finally:
+            self.status_busy = False
+        self.statusChanged.emit()
+
+    def status_text(self):
+        lines = []
+        if self.panel.alive():
+            lines.append("Headset panel: ready")
+        elif self.panel_starting and self.panel_starting.is_alive():
+            lines.append("Headset panel: starting...")
+        else:
+            lines.append(f"Headset panel: not running ({self.panel.error or 'it stopped'}). Starting a scenario tries again.")
+        lines.append(f"Headset button: {self.button_path}" if self.button_path else
+                     "Headset button: not found. Click, press Enter or pinch, or use this window's keys (Space marks).")
+        g = self.gazed.json("status")
+        if g is None:
+            lines.append("The gaze service isn't answering (frametop-gaze.service).")
+        else:
+            tracker = "our tracker" if g.get("tracker") == "own" else "SteamVR's tracker"
+            checks = g.get("checks") or {}
+            lines.append(f"Gaze service: {tracker}" + ("; gaze mode is on (the bench doesn't need it)"
+                                                        if checks.get("gaze_mode") else ""))
+            if checks.get("headset_worn") is False:
+                lines.append("Nobody is wearing the headset.")
+            if g.get("tracker") == "own":
+                made = g.get("calibration_made")
+                lines.append(f"Our tracker: calibrated {made}" if made else "Our tracker: not calibrated yet")
+        s = self.gaze.latest()
+        if self.gaze.error:
+            lines.append(f"ft-gaze: {self.gaze.error}")
+        elif s is None or benchrec.raw_now() - s.get("t", 0) > 2:
+            lines.append("ft-gaze: no samples yet (is SteamVR running?)")
+        else:
+            src = s.get("src", {})
+            ok = [n for k, n in QUICK_SOURCES if quick_gaze(src, k)]
+            lines.append("Gaze now from: " + (", ".join(ok) or "nothing (eyes closed, or the headset off?)"))
+        lines.append("Eye images: " + ("recorded (they stay on this headset)" if self._images else "off"))
+        lines.append(f"Runs go to {self.base}")
+        return "\n".join(lines)
+
+    # --- the headset button (its thread: to the Qt thread through the signal)
+
+    def _down(self, t):
+        self.down_t = benchrec.raw_now() - (time.monotonic() - t)
+        self.button.emit("down", self.down_t)
+
+    def _gesture(self, gesture):
+        self.button.emit(gesture, self.down_t or benchrec.raw_now())
+
+    @Slot(str, float)
+    def on_button(self, what, t):
+        if not self.run and self.phase != "end":
+            return
+        if what == "down":
+            # Done at once, so the press feels taken; it counts when "press" comes (a double is undo)
+            if self.phase == "target" and self.ready_t is not None and t >= self.ready_t:
+                self.panel.dot(*self.target, "done")
+                self.flashed = True
+        elif what == "press":
+            if self.phase == "end":
+                self.close_panel()
+            elif self.phase == "intro":
+                self.begin()
+            else:
+                self.mark(t, "button")
+        elif what == "double":
+            self.undo()
+        elif what == "hold" and self.run:
+            self.log("headset button: hold (end)")
+            self.finish(early=True)
+
+    # --- the pointer helper's calibration panel: clicks, Enter and pinches
+
+    def claim(self):
+        """Claim the helper's panel while a run or its summary shows on ours, and let it go after."""
+        want = (self.run is not None or self.phase == "end") and self.panel.alive()
+        now = time.monotonic()
+        if want and (self.claim_at is None or now - self.claim_at >= CLAIM_EVERY):
+            self.claim_at = now
+            self.to_helper(f"calpanel 1 {self.claim_name}")
+        elif not want and self.claim_at is not None:
+            self.claim_at = None
+            self.to_helper(f"calpanel 0 {self.claim_name}")
+
+    def to_helper(self, text):
+        try:
+            self.helper.sendto(text.encode(), "\0" + HELPER_SOCK)
+        except OSError:
+            pass  # the helper isn't running (SteamVR isn't)
+
+    @Slot()
+    def from_helper(self):
+        while True:
+            try:
+                data = self.helper.recv(256)
+            except OSError:
+                return
+            t = benchrec.raw_now()
+            what = data.decode(errors="replace").strip()
+            if self.claim_at is None:
+                continue  # a late answer: the panel is gone
+            if what == "calaccept":     # a press: as the button's, at its arrival
+                if self.phase == "end":
+                    self.close_panel()
+                elif self.phase == "intro":
+                    self.begin()
+                elif self.run:
+                    self.mark(t, "helper")
+            elif what == "calquit":     # as holding the button
+                if self.run:
+                    self.log("pointer helper: quit (end)")
+                    self.finish(early=True)
+                else:
+                    self.close_panel()
+
+    # --- a run
+
+    @Slot(str)
+    def start(self, sid):
+        if self.run:
+            return
+        if not self.panel.alive():
+            self._notice = ("The headset panel isn't running" + (f": {self.panel.error}" if self.panel.error else "")
+                            + ". Trying again; start the scenario once it's ready.")
+            self.start_panel()
+            self.refresh_status()
+            self.changed.emit()
+            return
+        self.sc = next(s for s in self.all["scenarios"] if s["id"] == sid)
+        seed = self.sc.get("seed", random.randrange(1 << 30))
+        self.trials = make_trials(self.sc, random.Random(seed))
+        self.i, self.results = 0, []
+        info = {"display": "panel", "panel": {"deg": PANEL_DEG, "aspect": PANEL_ASPECT, "margin": MARGIN,
+                                              "prog": str(self.panel.prog)},
+                "seed": seed, "look": LOOK, "app": {"rev": self.rev}, "trials_planned": len(self.trials),
+                "button": self.button_path}
+        self.run = benchrec.Run(self.sc, self.gaze, images=self._images, base=self.base, log=self.log, info=info).start()
+        r = self.panel.ask(f"log start {self.run.dir}")
+        if not (r or "").startswith("ok"):
+            self.log(f"panel log: {r or 'no answer'}")
+            self.run.info["panel"]["log_error"] = r or "no answer"
+            self.run.write_info()
+        self._page, self._summary, self._last, self._notice = "run", "", "", ""
+        prompt = self.sc.get("prompt", self.all.get("prompt", "Look at the dot and press the button (or click, Enter or pinch)."))
+        intro = [self.sc["message"]] if self.sc.get("message") else []
+        self._message = "\n".join(intro + [prompt])
+        self.phase, self.target, self.next_at = "intro", None, None
+        self.panel.show(self.bg)
+        self.panel.dot(0, 0, "off")
+        self.panel.lines(self.sc["name"], text_line(self._message), KEYS_START, f"{len(self.trials)} targets")
+        self._progress = f"{self.sc['name']}: {len(self.trials)} targets. Press the headset button (or Space) to start."
+        self.run.trial({"kind": "message", "text": self._message, "t": benchrec.raw_now()})
+        self.wake_at = 0.0
+        self.claim()
+        self.changed.emit()
+
+    def begin(self):
+        """The intro read: the first target."""
+        self.run.trial({"kind": "message-done", "t": benchrec.raw_now()})
+        self.panel.lines()
+        self.show(self.i)
+
+    @Slot()
+    def again(self):
+        if self._page == "done" and self.sc:
+            self.start(self.sc["id"])
+
+    def show(self, i):
+        self.i = i
+        if i >= len(self.trials):
+            return self.finish()
+        p = self.trials[i]
+        self.target = panel_direction(p["u"], p["v"])
+        bg = float(p.get("bg", self.bg))
+        self.panel.set("bg", f"bg {bg:.3f}")
+        self.shown_t = benchrec.raw_now()
+        dwell = p.get("dwell")
+        self.ready_t = None if dwell else self.shown_t
+        self.panel.dot(*self.target, "capture" if dwell else "look", 0.0 if dwell else None)
+        self.phase, self.flashed, self.next_at = "target", False, None
+        self._progress = f"{self.sc['name']}: {i + 1} of {len(self.trials)}"
+        self.changed.emit()
+
+    def quick(self, t, target):
+        """How far our tracker and SteamVR were from the target: their median direction over the
+        LOOK before t, {source: {"yaw", "pitch", "n", "deg"}}."""
+        samples = self.gaze.window(t - LOOK[0], t - LOOK[1])
+        out = {}
+        for key, _ in QUICK_SOURCES:
+            rows = [g for g in (quick_gaze(s.get("src") or {}, key) for s in samples) if g]
+            if len(rows) < 3:
+                continue
+            g = (statistics.median(r[0] for r in rows), statistics.median(r[1] for r in rows))
+            out[key] = {"yaw": round(g[0], 2), "pitch": round(g[1], 2), "n": len(rows), "deg": round(angle(g, target), 2)}
+        return out
+
+    @Slot()
+    def markKey(self):
+        """Space in the window: a mark now."""
+        if self.phase == "intro":
+            self.begin()
+        elif self.phase == "end":
+            self.close_panel()
+        elif self.run:
+            self.mark(benchrec.raw_now(), "key")
+
+    def mark(self, t, by):
+        """You were looking at the dot at t (a press's down)."""
+        if self.phase != "target" or t < self.shown_t:
+            self.run.trial({"kind": "stray", "t": t, "by": by, "phase": self.phase})
+            return
+        if self.ready_t is None or t < self.ready_t:
+            self.run.trial({"kind": "early", "i": self.i, "t": t, "by": by})
+            return
+        p = self.trials[self.i]
+        yaw, pitch = self.target
+        rec = {"kind": "trial", "i": self.i, "u": p["u"], "v": p["v"], "yaw": round(yaw, 4), "pitch": round(pitch, 4),
+               "bg": float(p.get("bg", self.bg)), "shown_t": self.shown_t, "ready_t": self.ready_t,
+               "click": {"t": t, "by": by}, "truth": {"yaw": round(yaw, 4), "pitch": round(pitch, 4), "from": "target"}}
+        rec["quick"] = self.quick(t, self.target)
+        self.results.append(rec)
+        self.run.trial(rec)
+        self._last = "Last: " + ", ".join(f"{label} {rec['quick'][k]['deg']:.1f}° off" if k in rec["quick"] else
+                                          f"{label} no gaze" for k, label in QUICK_SOURCES)
+        self.panel.dot(yaw, pitch, "done")
+        self.after(0.0 if self.flashed else DONE_S)
+
+    def after(self, wait):
+        """The next target, after wait (and the scenario's gap)."""
+        self.i += 1
+        self.phase, self.next_at = "marked", time.monotonic() + wait
+        self.changed.emit()
+
+    @Slot()
+    def skip(self):
+        if self.phase == "target":
+            p = self.trials[self.i]
+            self.run.trial({"kind": "trial", "i": self.i, "u": p["u"], "v": p["v"], "yaw": round(self.target[0], 4),
+                            "pitch": round(self.target[1], 4), "shown_t": self.shown_t,
+                            "click": {"t": benchrec.raw_now(), "by": "key"}, "skipped": True})
+            self.panel.dot(*self.target, "off")
+            self.after(0.0)
+
+    @Slot()
+    def undo(self):
+        """Back to the last target (self.i is the one showing, or the next one while marked or in
+        a gap)."""
+        if not self.run or self.phase in ("intro", "end"):
+            return
+        back = self.i - 1
+        if back < 0:
+            if self.phase == "target" and self.flashed:   # the first dot's down: show it again
+                self.show(self.i)
+            return
+        if self.results and self.results[-1]["i"] == back:
+            self.results.pop()
+        self.run.trial({"kind": "undo", "i": back, "t": benchrec.raw_now()})
+        self._last = "Took the last one back"
+        self.show(back)
+
+    @Slot()
+    def stop(self):
+        if self.run:
+            self.finish(early=True)
+        elif self._page == "done":
+            self.close_panel()
+            self._page = "menu"
+            self.changed.emit()
+        else:
+            QGuiApplication.quit()
+
+    def finish(self, early=False, why=None):
+        summary = self.summarize(early, why)
+        self.phase, self.next_at = "end", None
+        self._page = "done"
+        self._summary = summary["text"]
+        self.changed.emit()
+        run, self.run = self.run, None
+        run.info["panel"]["log_rows"] = self.panel.ask("log stop") if self.panel.alive() else None
+        if self.panel.alive():
+            self.panel.dot(0, 0, "off")
+            self.panel.set("bg", f"bg {self.bg:.3f}")
+            self.panel.lines("Ended early" if early else "Done", text_line(summary["panel"]), KEYS_END)
+        self.stopping = threading.Thread(target=lambda: run.stop(summary), daemon=True)
+        self.stopping.start()
+
+    def close_panel(self):
+        if self.phase == "end":
+            self.phase = None
+            self.panel.cmd("hide")
+            self.claim()
+
+    def summarize(self, early, why=None):
+        marks = [r for r in self.results if not r.get("skipped")]
+        lines = [f"{self.sc['name']}: {len(marks)} marked" + (" (ended early)" if early else "")]
+        if why:
+            lines.append(why)
+        out = {"marks": len(marks), "early": early}
+        for key, label in QUICK_SOURCES:
+            errs = sorted(r["quick"][key]["deg"] for r in marks if key in r.get("quick", {}))
+            if errs:
+                med, worst = statistics.median(errs), errs[-1]
+                p90 = errs[min(len(errs) - 1, int(0.9 * len(errs)))]
+                out[key] = {"median": med, "p90": p90, "worst": worst, "n": len(errs)}
+                lines.append(f"{label}: {med:.1f}° off at the median, {p90:.1f}° at the 90th percentile, worst {worst:.1f}°")
+            else:
+                lines.append(f"{label}: no gaze at the marks")
+        out["panel"] = "\n".join(lines)
+        if self.run:
+            lines.append(f"Saved in {self.run.dir}")
+        out["text"] = "\n".join(lines)
+        return out
+
+    # --- timers
+
+    def tick(self):
+        now = time.monotonic()
+        if self._page != "run" and now - self.status_at > 2:
+            self.status_at = now
+            self.refresh_status()
+        self.claim()
+        if not self.run:
+            return
+        if now - self.wake_at > WAKE / 2:
+            self.wake_at = now
+            threading.Thread(target=lambda: self.gazed.ask(f"wake {WAKE}"), daemon=True).start()
+        self.run.keepalive()
+        if not self.panel.alive():
+            self.log("the panel stopped")
+            return self.finish(early=True, why="The headset panel stopped.")
+        if self.phase == "target" and self.ready_t is None:
+            dwell = self.trials[self.i]["dwell"]
+            done = min(1.0, (benchrec.raw_now() - self.shown_t) / dwell)
+            self.panel.dot(*self.target, "capture", round(done, 2))
+            if done >= 1.0:
+                self.ready_t = benchrec.raw_now()
+        if self.next_at is not None and now >= self.next_at:
+            gap = self.sc.get("gap")
+            if self.phase == "marked" and gap and self.i < len(self.trials):
+                self.panel.dot(*self.target, "off")
+                self.phase, self.next_at = "gap", now + random.uniform(*gap)
+            else:
+                self.show(self.i)
+
+    def shutdown(self):
+        self.timer.stop()
+        if self.run:
+            self.finish(early=True)
+        if self.stopping:
+            self.stopping.join(10)
+        if self.reader:
+            self.reader.stop()
+        if self.panel.alive():
+            self.panel.cmd("hide")
+        if self.claim_at is not None:
+            self.to_helper(f"calpanel 0 {self.claim_name}")
+            self.claim_at = None
+        if self.helper_note:
+            self.helper_note.setEnabled(False)
+        self.helper.close()
+        for t in (self.panel_starting, self.status_thread):
+            if t:
+                t.join(PANEL_WAIT + 1)
+        self.panel.stop()
+        self.panel.close()
+        self.gaze.stop()
+        self.gazed.close()
+
+
+def main():
+    ap = argparse.ArgumentParser(description="Frametop Gaze Bench")
+    ap.add_argument("--base", default=str(benchrec.BASE), help="where runs are recorded")
+    ap.add_argument("--no-images", action="store_true", help="don't record the eye cameras")
+    ap.add_argument("--scenarios", default=os.path.join(HERE, "scenarios.json"))
+    ap.add_argument("--ft-gaze", help="the ft-gaze to run (default: this tree's build, else the installed release's)")
+    ap.add_argument("--ft-gazepanel", help="the ft-gazepanel to run (default: this tree's build, else the release's)")
+    ap.add_argument("--panel-no-vr", action="store_true", help="the panel without SteamVR (its --no-vr)")
+    ap.add_argument("--no-button", action="store_true", help="don't read the headset button")
+    ap.add_argument("--style", default="Fusion", help="the Qt Quick Controls style")
+    a, qt_args = ap.parse_known_args()
+    app = QGuiApplication([sys.argv[0]] + qt_args)
+    app.setApplicationName("ft-gazebench")
+    app.setApplicationDisplayName("Frametop Gaze Bench")
+    app.setDesktopFileName("ft-gazebench")
+    QQuickStyle.setStyle(a.style)
+    with open(a.scenarios) as f:
+        scenarios = json.load(f)
+    bench = Bench(scenarios, a.base, not a.no_images, gaze_prog=a.ft_gaze, panel_prog=a.ft_gazepanel,
+                  panel_no_vr=a.panel_no_vr, button=False if a.no_button else None)
+    app.aboutToQuit.connect(bench.shutdown)
+    # A stop (the launcher's container doesn't pass one on, but kill does) ends a run properly
+    for sig in (signal.SIGTERM, signal.SIGINT):
+        signal.signal(sig, lambda *_: QGuiApplication.quit())
+    engine = QQmlApplicationEngine()
+    engine.rootContext().setContextProperty("bench", bench)
+    engine.load(QUrl.fromLocalFile(os.path.join(HERE, "bench.qml")))
+    if not engine.rootObjects():
+        sys.exit(1)
+    sys.exit(app.exec())
+
+
+if __name__ == "__main__":
+    main()

gaze/bench/install.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Add "Frametop Gaze Bench" to the app menu on a Frame with Frametop, for this tree (README.md).
+# It needs ft-gaze and ft-gazepanel (gaze/build.sh; a release brings them built, and a tree
+# without a build uses the installed release's) and, to record our tracker, ft-eyes
+# (gaze/tracker/install.sh).
+# Usage: gaze/bench/install.sh            install or update
+#        gaze/bench/install.sh uninstall  remove the menu entry. Runs stay where they are.
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
+. "$root/scripts/_env.sh"
+entry='~/.local/share/applications/frametop-gazebench.desktop'
+
+case ${1:-install} in
+  install)
+    fill_template "$root/gaze/bench/ft-gazebench.desktop" |
+      on_frame "mkdir -p ~/.local/share/applications && cat > $entry"
+    echo "Installed. On Frametop's desktop, open \"Frametop Gaze Bench\" from the app menu."
+    echo "Runs go to ~/.local/share/frametop/gaze/bench."
+    ;;
+  uninstall)
+    on_frame "rm -f $entry"
+    echo "Removed the menu entry. Runs are still in ~/.local/share/frametop/gaze/bench."
+    ;;
+  *) echo "usage: $0 [install|uninstall]" >&2; exit 2 ;;
+esac

gaze/bench/scenarios.json

@@ -0,0 +1,75 @@
+{
+  "version": 2,
+  "about": "The gaze bench's scenarios (README.md). Targets show on the headset's panel (gazebench.py), 64 degrees across (4:3), fixed to the head. They're fractions of the panel across (u) and down (v); 0 and 1 are 1 degree in from its edges: 31 degrees left or right, 24 up or down, the corners 37 out. Generators: grid [cols, rows] over span; edges N per edge, corners included; rays: fractions of the way from the middle to the edge along dirs directions; random N with edge_share of them on an edge; steps N, each size [lo, hi] degrees from the last, within box [yaw, pitch] degrees of the middle. background and backgrounds are the panel's brightness, 0 to 1.",
+  "background": 0.15,
+  "prompt": "Look at the dot and press the button (or click, Enter or pinch).",
+  "scenarios": [
+    {
+      "id": "warmup",
+      "name": "Warm-up",
+      "about": "Nine dots around the middle.",
+      "targets": {"grid": [3, 3], "span": [0.2, 0.8]},
+      "shuffle": true
+    },
+    {
+      "id": "edges",
+      "name": "Edges",
+      "about": "Sixteen dots along the panel's edges and in its corners, 24 to 37 degrees out.",
+      "targets": {"edges": 5},
+      "shuffle": true
+    },
+    {
+      "id": "hold",
+      "name": "Hold at the edges",
+      "about": "Look at an edge dot until its ring fills (3 seconds), then press: the tracker loses an eye out there.",
+      "prompt": "Keep looking at the dot. Press when its ring is full.",
+      "targets": {"edges": 3},
+      "dwell": 3.0,
+      "shuffle": true
+    },
+    {
+      "id": "spokes",
+      "name": "Out from the middle",
+      "about": "Twenty-four dots on eight spokes, a third, two thirds and all the way out.",
+      "targets": {"rays": [0.35, 0.7, 1.0], "dirs": 8},
+      "shuffle": true
+    },
+    {
+      "id": "head-moving",
+      "name": "Head moving",
+      "about": "Keep looking at the dot while you turn or nod your head slowly, and press while it moves.",
+      "prompt": "Keep looking at the dot while you turn or nod your head slowly. Press while your head is still moving.",
+      "targets": {"grid": [3, 3], "span": [0.2, 0.8]},
+      "shuffle": true
+    },
+    {
+      "id": "light",
+      "name": "Light changes",
+      "about": "The panel switches between dark, bright and grey: your pupils change size.",
+      "targets": {"grid": [4, 3], "span": [0.1, 0.9]},
+      "backgrounds": [0.0, 0.95, 0.42],
+      "shuffle": true
+    },
+    {
+      "id": "steps",
+      "name": "Small steps",
+      "about": "Twenty dots, each 2 to 4 degrees from the last, near the middle.",
+      "targets": {"steps": 20, "size": [2.0, 4.0], "box": [12.0, 9.0]}
+    },
+    {
+      "id": "reseat",
+      "name": "After a nudge",
+      "about": "Lift the headset a little and set it back, then nine dots: how fast the tracker recovers.",
+      "message": "First lift the headset a little off your face and set it back down.",
+      "targets": {"grid": [3, 3], "span": [0.15, 0.85]},
+      "shuffle": true
+    },
+    {
+      "id": "long",
+      "name": "Long session",
+      "about": "Sixty dots anywhere at your own pace, a few seconds apart, about six minutes: drift, slips and stale corrections.",
+      "targets": {"random": 60, "edge_share": 0.3},
+      "gap": [1.0, 5.0]
+    }
+  ]
+}

gaze/bench/tests/fake_ft_gaze.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""A stand-in for ft-gaze in the bench's tests: 90 lines a second, the head facing the middle of
+a 1000 x 800 screen (number 2) at 20 px per degree, our tracker 0.5 degrees right of straight
+ahead and SteamVR 1 degree up. Quits when stdin closes, as ft-gaze --watch-stdin, or after
+$FAKE_GAZE_QUIT_AFTER seconds."""
+import json
+import os
+import sys
+import threading
+import time
+
+PPD = 20.0
+
+
+def hit(hy, hp):
+    return {"s": 2, "x": 500 - PPD * hy, "y": 400 - PPD * hp, "j": [-PPD, 0.0, 0.0, -PPD], "dpp": 1 / PPD}
+
+
+def watch():
+    while sys.stdin.buffer.read(4096):
+        pass
+    os._exit(0)
+
+
+threading.Thread(target=watch, daemon=True).start()
+quit_after = float(os.environ.get("FAKE_GAZE_QUIT_AFTER", "inf"))   # s: then exit, as when SteamVR quits
+start = time.monotonic()
+n = 0
+while time.monotonic() - start < quit_after:
+    t = time.clock_gettime(time.CLOCK_MONOTONIC_RAW)
+    own = {"hy": -0.5, "hp": 0.0, "hit": hit(-0.5, 0.0), "eyes": [[-0.5, 0.0], [-0.5, 0.0]],
+           "ehit": [hit(-0.5, 0.0), hit(-0.5, 0.0)]}
+    s = {"t": t, "age": 5, "n": n, "head": {"yaw": 0, "pitch": 0, "hit": hit(0, 0)},
+         "src": {"action": {"ok": 0}, "mmap1": {"hy": 0.0, "hp": 1.0, "hit": hit(0.0, 1.0)}, "mmap2": {"ok": 0},
+                 "left": {"hy": 0.0, "hp": 1.0, "hit": hit(0.0, 1.0)}, "right": {"hy": 0.0, "hp": 1.0, "hit": hit(0.0, 1.0)},
+                 "own": own}, "eye": None}
+    try:
+        sys.stdout.write(json.dumps(s) + "\n")
+        sys.stdout.flush()
+    except BrokenPipeError:
+        break
+    n += 1
+    time.sleep(1 / 90)

gaze/bench/tests/fake_ft_gazepanel.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+"""A stand-in for ft-gazepanel in the bench's tests: its control socket (--socket NAME) answers
+"ok" to everything, "log start DIR" writes the three logs' files there, "log stop" answers
+"ok 3 100 1". Each command goes on a line of $FT_FAKE_PANEL_LOG. Quits when stdin closes, as
+ft-gazepanel --watch-stdin."""
+import os
+import socket
+import sys
+import threading
+from pathlib import Path
+
+
+def watch():
+    while sys.stdin.buffer.read(4096):
+        pass
+    os._exit(0)
+
+
+name = sys.argv[sys.argv.index("--socket") + 1]
+log = os.environ.get("FT_FAKE_PANEL_LOG")
+threading.Thread(target=watch, daemon=True).start()
+sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+sock.bind("\0" + name)
+while True:
+    data, addr = sock.recvfrom(65536)
+    text = data.decode()
+    if log:
+        with open(log, "a") as f:
+            f.write(text + "\n")
+    reply = "ok"
+    if text.startswith("log start "):
+        for f in ("targets.jsonl", "poses.jsonl", "ipd.jsonl"):
+            (Path(text[10:]) / f).write_text("{}\n")
+    elif text == "log stop":
+        reply = "ok 3 100 1"
+    if addr:
+        sock.sendto(reply.encode(), addr)

gaze/bench/tests/test_bench.py

@@ -0,0 +1,442 @@
+#!/usr/bin/env python3
+"""The gaze bench without a headset: the scenarios' targets, bench.qml against gazebench.Bench,
+and whole runs recorded with stand-ins for ft-gaze (fake_ft_gaze.py), the panel
+(fake_ft_gazepanel.py), the headset button (a FIFO), our tracker, the gaze service and the pointer
+helper (their sockets), into a temporary folder. Needs PySide6 (the dev container); the parts that need it are
+skipped without it.
+
+  scripts/in-box python3 gaze/bench/tests/test_bench.py
+"""
+import json
+import os
+import random
+import re
+import socket
+import sys
+import tempfile
+import threading
+import time
+import unittest
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+BENCH = HERE.parent
+sys.path.insert(0, str(BENCH))
+sys.path.insert(0, str(BENCH.parent.parent))
+os.environ["FT_BENCH_EYES_SOCKET"] = f"ft_eyes_benchtest_{os.getpid()}"
+os.environ["FT_BENCH_GAZED_SOCKET"] = f"ft_gazed_benchtest_{os.getpid()}"
+os.environ["FT_BENCH_PANEL_SOCKET"] = f"ft_benchpanel_test_{os.getpid()}"
+os.environ["FT_BENCH_HELPER_SOCKET"] = f"ft_pointer_helper_benchtest_{os.getpid()}"   # (not the live helper)
+os.environ.setdefault("QT_QPA_PLATFORM", "offscreen")
+import benchrec  # noqa: E402
+from recorder.button import EV_KEY, INPUT_EVENT, KEY_SELECT  # noqa: E402
+
+try:
+    from PySide6.QtCore import QUrl
+    from PySide6.QtGui import QGuiApplication
+    from PySide6.QtQml import QQmlApplicationEngine
+    from PySide6.QtQuickControls2 import QQuickStyle
+    import gazebench
+    APP = QGuiApplication.instance() or QGuiApplication([sys.argv[0]])
+    QQuickStyle.setStyle("Fusion")  # the app's
+except ImportError as e:
+    gazebench, APP, NO_QT = None, None, str(e)
+
+SCENARIOS = json.loads((BENCH / "scenarios.json").read_text())
+FAKE_GAZE = HERE / "fake_ft_gaze.py"
+FAKE_PANEL = HERE / "fake_ft_gazepanel.py"
+OWN, STEAMVR = (-0.5, 0.0), (0.0, 1.0)   # fake_ft_gaze.py's gaze, head-relative degrees
+
+
+class FakeService(threading.Thread):
+    """A datagram server on an abstract socket that answers with reply(text)."""
+
+    def __init__(self, name, reply):
+        super().__init__(daemon=True)
+        self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+        self.sock.bind("\0" + name)
+        self.reply, self.heard, self.done = reply, [], False
+        self.start()
+
+    def stop(self):
+        self.done = True
+        self.join(1)
+        self.sock.close()
+
+    def run(self):
+        self.sock.settimeout(0.05)
+        while not self.done:
+            try:
+                data, addr = self.sock.recvfrom(65536)
+            except TimeoutError:
+                continue
+            except OSError:
+                return
+            text = data.decode()
+            self.heard.append(text)
+            r = self.reply(text)
+            if r is not None and addr:
+                self.sock.sendto(r.encode(), addr)
+
+
+class GazeStreamTest(unittest.TestCase):
+    def test_restarts(self):
+        """ft-gaze quits with SteamVR: it's started again, and the samples go on."""
+        saved = benchrec.RESTART
+        benchrec.RESTART = 0.2
+        os.environ["FAKE_GAZE_QUIT_AFTER"] = "0.3"
+        logs = []
+        g = benchrec.GazeStream(prog=FAKE_GAZE, log=logs.append).start()
+        try:
+            end = time.monotonic() + 5
+            while g.restarts < 2 and time.monotonic() < end:
+                time.sleep(0.05)
+            self.assertGreaterEqual(g.restarts, 2, logs)
+            lines = g.lines
+            end = time.monotonic() + 3
+            while g.lines <= lines and time.monotonic() < end:
+                time.sleep(0.05)
+            self.assertGreater(g.lines, lines)
+            self.assertIn("starting it again", logs[0])
+        finally:
+            del os.environ["FAKE_GAZE_QUIT_AFTER"]
+            benchrec.RESTART = saved
+            g.stop()
+        self.assertFalse(g.reader.is_alive())
+
+
+class ScenarioTest(unittest.TestCase):
+    def setUp(self):
+        if gazebench is None:
+            self.skipTest(f"no PySide6: {NO_QT}")
+
+    def test_targets(self):
+        for sc in SCENARIOS["scenarios"]:
+            ts = gazebench.make_trials(sc, random.Random(1))
+            self.assertTrue(ts, sc["id"])
+            for p in ts:
+                self.assertTrue(-1e-9 <= p["u"] <= 1 + 1e-9 and -1e-9 <= p["v"] <= 1 + 1e-9, (sc["id"], p))
+            if sc["id"] == "edges":
+                self.assertEqual(len(ts), 16)
+                self.assertTrue(all(p["u"] in (0, 1) or p["v"] in (0, 1) for p in ts))
+            if sc["id"] == "hold":
+                self.assertTrue(all(p["dwell"] == 3.0 for p in ts))
+            if sc["id"] == "light":
+                self.assertEqual({p["bg"] for p in ts}, set(sc["backgrounds"]))
+            if sc["id"] == "steps":
+                dirs = [(0.0, 0.0)] + [gazebench.panel_direction(p["u"], p["v"]) for p in ts]
+                steps = [gazebench.angle(a, b) for a, b in zip(dirs, dirs[1:])]
+                self.assertTrue(all(1.9 < d < 4.1 for d in steps), steps)
+                self.assertTrue(all(abs(y) <= 12.01 and abs(p) <= 9.01 for y, p in dirs), dirs)
+        rays = gazebench.make_trials(next(s for s in SCENARIOS["scenarios"] if s["id"] == "spokes"), random.Random(1))
+        outer = [p for p in rays if p["u"] in (0, 1) or p["v"] in (0, 1)]
+        self.assertEqual(len(outer), 8)  # the 1.0 ring reaches the edges and corners
+
+    def test_panel_direction(self):
+        d = gazebench.panel_direction
+        self.assertEqual(d(0.5, 0.5), (0.0, 0.0))
+        self.assertGreater(d(0, 0.5)[0], 30)    # u 0: left (+yaw)
+        self.assertGreater(d(0.5, 0)[1], 20)    # v 0: up (+pitch)
+        for u, v in ((0, 0), (1, 1), (0.2, 0.7), (0.9, 0.1)):
+            back = gazebench.panel_place(*d(u, v))
+            self.assertAlmostEqual(back[0], u, places=9)
+            self.assertAlmostEqual(back[1], v, places=9)
+        # Every scenario's dot shows whole: its ring (0.75 degrees out) is on the panel
+        for sc in SCENARIOS["scenarios"]:
+            for p in gazebench.make_trials(sc, random.Random(2)):
+                yaw, pitch = d(p["u"], p["v"])
+                for dy, dp in ((0.75, 0), (-0.75, 0), (0, 0.75), (0, -0.75)):
+                    x, y = gazebench.panel_place(yaw + dy, pitch + dp, margin=0)
+                    self.assertTrue(0 <= x <= 1 and 0 <= y <= 1, (sc["id"], p, dy, dp))
+
+
+class QmlTest(unittest.TestCase):
+    """bench.qml's bench.name(...) calls are slots and its bench.name reads are properties or slots."""
+
+    def setUp(self):
+        if gazebench is None:
+            self.skipTest(f"no PySide6: {NO_QT}")
+        meta = gazebench.Bench.staticMetaObject
+        self.slots = {bytes(meta.method(i).name()).decode() for i in range(meta.methodCount())}
+        self.props = {meta.property(i).name() for i in range(meta.propertyCount())}
+        self.qml = (BENCH / "bench.qml").read_text()
+
+    def test_names(self):
+        called = set(re.findall(r"\bbench\.(\w+)\s*\(", self.qml))
+        self.assertEqual(sorted(called - self.slots), [])
+        read = set(re.findall(r"\bbench\.(\w+)\b(?!\s*\()", self.qml)) - {"qml"}
+        self.assertEqual(sorted(read - self.props - self.slots), [])
+
+
+class RunTest(unittest.TestCase):
+    """A warm-up played through with the button and the keys, recorded and summed up; a hold
+    scenario ended by holding the button; bench.qml loaded on it."""
+
+    def setUp(self):
+        if gazebench is None:
+            self.skipTest(f"no PySide6: {NO_QT}")
+        self.tmp = tempfile.TemporaryDirectory()
+        tmp = Path(self.tmp.name)
+        self.saved = {k: getattr(benchrec, k) for k in ("STATE", "EYES_STATE", "LESSON_LOG", "CONF")}
+        benchrec.STATE = tmp / "state"
+        benchrec.EYES_STATE = benchrec.STATE / "eyes"
+        benchrec.LESSON_LOG = benchrec.STATE / "pointer-lessons.jsonl"
+        benchrec.CONF = tmp / "frametop.conf"
+        benchrec.EYES_STATE.mkdir(parents=True)
+        (benchrec.EYES_STATE / "calibration.json").write_text('{"version": 1, "fits": []}')
+        benchrec.CONF.write_text("GAZE_TRACKER=own\nOTHER=1\n")
+        benchrec.LESSON_LOG.write_text(json.dumps({"time": time.time() - 100, "kind": "own"}) + "\n")
+        self.finds = []
+        self.panel_log = tmp / "panel.log"
+        os.environ["FT_FAKE_PANEL_LOG"] = str(self.panel_log)
+        self.fifo = tmp / "button"
+        os.mkfifo(self.fifo)
+        self.button = os.open(self.fifo, os.O_RDWR)   # (Linux: doesn't wait for the reader)
+
+        def eyes(text):
+            w = text.split()
+            if w[0] == "status":
+                return json.dumps({"calibration": {"made": "2026-10-17 10:00:00"}, "feature": "pupil", "model": None,
+                                   "eyes": {"right": {"shift": [0, 0], "clicks": 0, "jump": False, "jump_px": [0, 0],
+                                                      "reseat": False}}})
+            if w[:2] == ["finds", "start"]:
+                self.finds.append(w[2])
+                Path(w[2]).write_text(json.dumps({"t": 1.0, "cam": 0, "lost": True}) + "\n")
+                return "ok"
+            if w[:2] == ["finds", "stop"]:
+                return "ok 1"
+            return "fail"
+
+        def gazed(text):
+            if text == "status":
+                return json.dumps({"tracker": "own", "calibration_made": "2026-10-17 10:00:00",
+                                   "checks": {"gaze_mode": False, "headset_worn": True}})
+            return "ok"
+
+        self.eyes = FakeService(benchrec.EYES_SOCK, eyes)
+        self.gazed = FakeService(benchrec.GAZED_SOCK, gazed)
+        self.helper = FakeService(os.environ["FT_BENCH_HELPER_SOCKET"], lambda text: None)
+
+    def tearDown(self):
+        for k, v in self.saved.items():
+            setattr(benchrec, k, v)
+        self.eyes.stop()
+        self.gazed.stop()
+        self.helper.stop()
+        os.close(self.button)
+        self.tmp.cleanup()
+
+    def wait(self, cond, timeout=5.0):
+        end = time.monotonic() + timeout
+        while time.monotonic() < end:
+            APP.processEvents()
+            if cond():
+                return True
+            time.sleep(0.02)
+        return False
+
+    def key(self, down):
+        t = time.time()
+        os.write(self.button, INPUT_EVENT.pack(int(t), int(t % 1 * 1e6), EV_KEY, KEY_SELECT, 1 if down else 0))
+
+    def press(self, held=0.08):
+        self.key(True)
+        self.wait(lambda: False, held)
+        self.key(False)
+
+    def panel_said(self):
+        return self.panel_log.read_text().splitlines() if self.panel_log.exists() else []
+
+    def bench(self, base):
+        bench = gazebench.Bench(SCENARIOS, str(base), images=False, gaze_prog=FAKE_GAZE, panel_prog=FAKE_PANEL,
+                                button=str(self.fifo))
+        self.assertTrue(self.wait(lambda: bench.gaze.latest() is not None), "no samples from the fake ft-gaze")
+        self.assertTrue(self.wait(bench.panel.alive), bench.panel.error)
+        return bench
+
+    def test_run(self):
+        base = Path(self.tmp.name) / "runs"
+        bench = self.bench(base)
+        try:
+            self.assertTrue(self.wait(lambda: "Headset panel: ready" in bench.status), bench.status)
+            self.assertIn(str(self.fifo), bench.status)
+            bench.start("warmup")
+            self.assertEqual((bench.page, bench.phase), ("run", "intro"))
+            run = next(base.iterdir())
+            self.assertTrue(self.wait(lambda: "title Warm-up" in self.panel_said()), self.panel_said())
+            said = self.panel_said()
+            self.assertIn("show rec", said)
+            self.assertIn(f"log start {run}", said)
+            self.press()                                # the intro: a press starts it
+            self.assertTrue(self.wait(lambda: bench.phase == "target"))
+            n = len(bench.trials)
+            pressed_at = None
+            for k in range(n):
+                self.assertTrue(self.wait(lambda: bench.phase == "target"), (k, bench.phase))
+                self.wait(lambda: False, 0.45)          # samples in the look window
+                if k == 0:                               # the button marks it, at its down
+                    pressed_at = benchrec.raw_now()
+                    self.press()
+                    self.assertTrue(self.wait(lambda: len(bench.results) == 1))
+                    continue
+                if k == 2:                               # two presses: back to 1, marked again
+                    self.press()
+                    self.wait(lambda: False, 0.1)
+                    self.press()
+                    self.assertTrue(self.wait(lambda: bench.i == 1 and bench.phase == "target"), (bench.i, bench.phase))
+                    self.wait(lambda: False, 0.45)
+                    bench.markKey()
+                    self.assertTrue(self.wait(lambda: bench.phase == "target"))
+                    self.wait(lambda: False, 0.45)
+                if k == 4:
+                    bench.skip()
+                    continue
+                bench.markKey()
+            self.assertTrue(self.wait(lambda: bench.page == "done"))
+            self.assertEqual(bench.phase, "end")
+            self.assertIn("Our tracker", bench.summary)
+            self.assertIn("log stop", self.panel_said())
+            self.assertTrue(self.wait(lambda: "title Done" in self.panel_said()), self.panel_said()[-5:])
+            self.press()                                 # closes the panel
+            self.assertTrue(self.wait(lambda: self.panel_said()[-1] == "hide"))
+            self.assertTrue(self.wait(lambda: "end" in json.loads((run / "run.json").read_text())))
+        finally:
+            bench.shutdown()
+        info = json.loads((run / "run.json").read_text())
+        self.assertEqual(info["scenario"]["id"], "warmup")
+        self.assertEqual(info["display"], "panel")
+        self.assertEqual(info["panel"]["log_rows"], "ok 3 100 1")
+        self.assertEqual(info["button"], str(self.fifo))
+        self.assertEqual(info["conf"], ["GAZE_TRACKER=own"])
+        self.assertIsNotNone(info["last_lesson"])
+        self.assertGreater(info["gaze_lines"], 50)
+        self.assertEqual(info["finds_rows"], "ok 1")
+        self.assertTrue((run / "start" / "eyes-calibration.json").exists())
+        self.assertTrue((run / "targets.jsonl").exists())
+        self.assertEqual(self.finds, [str(run / "own.jsonl")])
+        rows = [json.loads(line) for line in (run / "trials.jsonl").read_text().splitlines()]
+        kinds = [r["kind"] for r in rows]
+        self.assertEqual(kinds[:2], ["message", "message-done"])
+        self.assertIn("undo", kinds)
+        trials = [r for r in rows if r["kind"] == "trial"]
+        self.assertEqual(sum(1 for r in trials if r.get("skipped")), 1)
+        by_button = [r for r in trials if r["click"]["by"] == "button"]
+        self.assertEqual(len(by_button), 1)
+        # The mark's time is the press's down, not when the press was sure (a double-press time later)
+        self.assertLess(abs(by_button[0]["click"]["t"] - pressed_at), 0.2)
+        for r in trials:
+            if r.get("skipped"):
+                continue
+            target = gazebench.panel_direction(r["u"], r["v"])
+            self.assertAlmostEqual(r["truth"]["yaw"], target[0], places=3)
+            self.assertAlmostEqual(r["truth"]["pitch"], target[1], places=3)
+            q = r["quick"]
+            self.assertAlmostEqual(q["own"]["deg"], gazebench.angle(OWN, target), places=1)
+            self.assertAlmostEqual(q["steam-eyes"]["deg"], gazebench.angle(STEAMVR, target), places=1)
+        # The button's down showed the dot done before the press counted
+        self.assertTrue(any(c.startswith("dot ") and c.endswith(" done") for c in self.panel_said()))
+        self.assertGreater(len((run / "gaze.jsonl").read_text().splitlines()), 50)
+
+    def answer(self, bench, text):
+        """The pointer helper's answer to the bench's panel claim."""
+        c = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+        c.sendto(text.encode(), "\0" + bench.claim_name)
+        c.close()
+
+    def test_helper(self):
+        """The pointer helper's calibration panel: claimed while a run or its summary shows, renewed;
+        its calaccept starts, marks (at its arrival) and closes, calquit ends; then it's let go."""
+        base = Path(self.tmp.name) / "runs"
+        bench = self.bench(base)
+        claim, release = f"calpanel 1 {bench.claim_name}", f"calpanel 0 {bench.claim_name}"
+        try:
+            self.wait(lambda: False, 0.2)
+            self.assertEqual(self.helper.heard, [], "no claim before a run")
+            bench.start("warmup")
+            self.assertTrue(self.wait(lambda: claim in self.helper.heard))
+            self.assertTrue(self.wait(lambda: self.helper.heard.count(claim) >= 2, 2.5), self.helper.heard)
+            self.answer(bench, "calaccept")             # the intro: starts it
+            self.assertTrue(self.wait(lambda: bench.phase == "target"))
+            self.wait(lambda: False, 0.45)
+            sent = benchrec.raw_now()
+            self.answer(bench, "calaccept")
+            self.assertTrue(self.wait(lambda: len(bench.results) == 1))
+            self.assertEqual(bench.results[0]["click"]["by"], "helper")
+            self.assertLess(abs(bench.results[0]["click"]["t"] - sent), 0.1)
+            self.answer(bench, "calquit")                # ends it
+            self.assertTrue(self.wait(lambda: bench.page == "done"))
+            self.assertIn("ended early", bench.summary)
+            self.assertNotIn(release, self.helper.heard, "the summary still shows")
+            self.answer(bench, "calaccept")             # closes the summary
+            self.assertTrue(self.wait(lambda: self.panel_said()[-1] == "hide"))
+            self.assertTrue(self.wait(lambda: self.helper.heard[-1] == release), self.helper.heard[-3:])
+            n = len(self.helper.heard)
+            self.answer(bench, "calaccept")             # late: nothing
+            self.wait(lambda: False, 1.3)
+            self.assertEqual(len(self.helper.heard), n, "no claim once the panel is gone")
+            self.assertIsNone(bench.run)
+            run = next(base.iterdir())
+            self.assertTrue(self.wait(lambda: "end" in json.loads((run / "run.json").read_text())))
+        finally:
+            bench.shutdown()
+
+    def test_hold(self):
+        """Hold at the edges: a press before the ring fills doesn't count; holding the button ends it."""
+        base = Path(self.tmp.name) / "runs"
+        bench = self.bench(base)
+        try:
+            bench.start("hold")
+            bench.markKey()
+            self.assertEqual(bench.phase, "target")
+            bench.markKey()                              # too soon
+            self.assertTrue(self.wait(lambda: any(re.fullmatch(r"dot \S+ \S+ capture 0\.\d+", c) for c in self.panel_said())))
+            self.key(True)
+            self.assertTrue(self.wait(lambda: bench.page == "done", 3))
+            self.key(False)
+            self.assertIn("ended early", bench.summary)
+            run = next(base.iterdir())
+            self.assertTrue(self.wait(lambda: "end" in json.loads((run / "run.json").read_text())))
+        finally:
+            bench.shutdown()
+        kinds = [json.loads(line)["kind"] for line in (run / "trials.jsonl").read_text().splitlines()]
+        self.assertIn("early", kinds)
+        self.assertNotIn("trial", kinds)
+
+    def test_no_panel(self):
+        bench = gazebench.Bench(SCENARIOS, str(Path(self.tmp.name) / "runs"), images=False, gaze_prog=FAKE_GAZE,
+                                panel_prog="/nonexistent/ft-gazepanel", button=False)
+        try:
+            self.wait(lambda: bench.panel.error is not None, 2)
+            bench.start("warmup")
+            self.assertEqual(bench.page, "menu")
+            self.assertIn("isn't running", bench.notice)
+            self.assertTrue(self.wait(lambda: "Headset panel: not running" in bench.status), bench.status)
+            self.assertIn("Headset button: not found", bench.status)
+        finally:
+            bench.shutdown()
+
+    def test_qml_loads(self):
+        bench = self.bench(Path(self.tmp.name) / "runs")
+        warnings = []
+        try:
+            engine = QQmlApplicationEngine()
+            engine.warnings.connect(lambda ws: warnings.extend(w.toString() for w in ws))
+            engine.rootContext().setContextProperty("bench", bench)
+            engine.load(QUrl.fromLocalFile(str(BENCH / "bench.qml")))
+            self.assertTrue(engine.rootObjects())
+            self.wait(lambda: False, 0.3)
+            bench.start("steps")
+            self.wait(lambda: False, 0.2)
+            bench.markKey()
+            self.wait(lambda: False, 0.2)
+            bench.stop()
+            self.wait(lambda: bench.page == "done", 1)
+            self.wait(lambda: False, 0.5)
+            self.assertEqual(warnings, [])
+            del engine
+        finally:
+            bench.shutdown()
+
+
+if __name__ == "__main__":
+    unittest.main()

gaze/bench/tests/test_compare.py

@@ -0,0 +1,291 @@
+#!/usr/bin/env python3
+"""compare.py on a made-up run with known errors: SteamVR 1 degree high (its eyes 1 degree either
+side of that), our live tracker 0.5 degrees left (both its eyes), and our tracker's finds made
+from a calibration, with the headset slipping 8 px (2 degrees) a third of the way in, so the
+replays' jump rules differ in a known way. Needs
+numpy and OpenCV (the tracker's Python):
+
+  scripts/in-box /opt/frametop/venv/bin/python gaze/bench/tests/test_compare.py
+"""
+import json
+import math
+import shutil
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE.parent))
+try:
+    import numpy as np
+    import cv2  # noqa: F401  (eyes_pupil's)
+    import compare
+    import eyes_model
+except ImportError as e:
+    compare, NO_DEPS = None, str(e)
+
+PPD = 20.0               # screen pixels per degree, as fake_ft_gaze.py
+K = 4.0                  # pupil pixels per degree
+T0, WALL = 1000.0, 1.79e9
+SLIP_AT, SLIP = 14.0, (8.0, 0.0)   # s after T0, px: the headset slips this much (both eyes)
+# (s after T0 of the click, yaw, pitch, nudged)
+TRIALS = [(3, 0, 0, False), (5, 10, 0, False), (7, -15, 5, True), (9, 5, -10, False), (11, 25, 0, False),
+          (13, -5, 8, False),
+          (16, 8, 3, False),                                  # 2 s after the slip: the jump isn't in yet
+          (24, -10, -5, False), (27, 12, 6, False), (30, 0, -12, False), (33, -18, 0, False)]
+
+
+def hit(hy, hp):
+    return {"s": 2, "x": 500 - PPD * hy, "y": 400 - PPD * hp, "j": [-PPD, 0.0, 0.0, -PPD], "dpp": 1 / PPD}
+
+
+def pupil(e, hy, hp, t):
+    """Where eye e's pupil is for a gaze, t s after T0."""
+    sx = SLIP[0] if t >= SLIP_AT else 0.0
+    return (256 + 10 * e + K * hy + sx, 200 - K * hp)
+
+
+def mid(e, t):
+    sx = SLIP[0] if t >= SLIP_AT else 0.0
+    return (256 + 10 * e + sx, 230.0)
+
+
+def looking(t):
+    """The target being looked at: the next one to be clicked."""
+    for tc, hy, hp, _ in TRIALS:
+        if t <= T0 + tc:
+            return hy, hp
+    return TRIALS[-1][1:3]
+
+
+def make_run(d):
+    (d / "start").mkdir(parents=True)
+    dots = [{"truth": (hy, hp), "eye": [{"pupil": np.array(pupil(e, hy, hp, 0)), "mid": np.array(mid(e, 0))}
+                                        for e in (0, 1)]}
+            for hy in (-20, -10, 0, 10, 20) for hp in (-15, -7, 0, 7, 15)]
+    cal = eyes_model.Calibration.fit(dots, {"made": "M", "feature": "pupil", "dots": len(dots)})
+    cal.save(d / "start" / "eyes-calibration.json")
+    with open(d / "start" / "eyes-calibration-dots.jsonl", "w") as f:
+        for k in dots:
+            f.write(json.dumps({"made": "M", "truth": k["truth"]}) + "\n")
+    blank = {"base": [0, 0], "ref": None, "meas": []}
+    (d / "start" / "eyes-state.json").write_text(json.dumps({"calibration": "M", "shift": {"0": blank, "1": blank}}))
+    eyes = {"right": {"reseat": False, "jump": False, "jump_px": [0, 0]},
+            "left": {"reseat": False, "jump": False, "jump_px": [0, 0]}}
+    (d / "run.json").write_text(json.dumps({
+        "scenario": {"id": "synthetic"}, "screen": {"w": 1000, "h": 800, "dpr": 1.0}, "look": [0.35, 0.05],
+        "start": {"time": WALL, "t_raw": T0, "eyes": {"feature": "pupil", "eyes": eyes}}, "last_lesson": WALL - 500}))
+    with open(d / "gaze.jsonl", "w") as g, open(d / "own.jsonl", "w") as o:
+        n = 0
+        while n / 90 < TRIALS[-1][0] + 1:
+            t = T0 + n / 90
+            hy, hp = looking(t)
+            src = {"mmap1": {"hy": hy, "hp": hp + 1, "hit": hit(hy, hp + 1)},
+                   "left": {"hy": hy + 1, "hp": hp + 1, "hit": hit(hy + 1, hp + 1)},
+                   "right": {"hy": hy - 1, "hp": hp + 1, "hit": hit(hy - 1, hp + 1)},
+                   "own": {"hy": hy + 0.5, "hp": hp, "hit": hit(hy + 0.5, hp),
+                           "eyes": [[hy + 0.5, hp], [hy + 0.5, hp]]}}
+            g.write(json.dumps({"t": t, "head": {"yaw": 0, "pitch": 0, "hit": hit(0, 0)}, "src": src}) + "\n")
+            for e in (0, 1):
+                px, py = pupil(e, hy, hp, t - T0)
+                mx, my = mid(e, t - T0)
+                o.write(json.dumps({"t": t + 0.001 * e, "cam": e, "x": px, "y": py, "a": 20, "b": 18, "angle": 0,
+                                    "glints": [[mx, my - 10], [mx, my + 10]]}) + "\n")
+            n += 1
+    with open(d / "trials.jsonl", "w") as f, open(d / "lessons.jsonl", "w") as ls:
+        for i, (tc, hy, hp, nudged) in enumerate(TRIALS):
+            x, y = 500 - PPD * hy, 400 - PPD * hp
+            f.write(json.dumps({"kind": "trial", "i": i, "u": x / 1000, "v": y / 800, "x": x, "y": y,
+                                "click": {"x": x + 3, "y": y, "t": T0 + tc, "button": "left"},
+                                "truth": {"x": x, "y": y, "from": "target"}}) + "\n")
+            if nudged:
+                ls.write(json.dumps({"time": WALL + tc + 0.02, "kind": "own", "raw": [hy + 0.5, hp], "true": [hy, hp],
+                                     "look_t": T0 + tc - 0.05}) + "\n")
+
+
+class CompareTest(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        if compare is None:
+            raise unittest.SkipTest(f"needs numpy and OpenCV: {NO_DEPS}")
+        cls.tmp = tempfile.TemporaryDirectory()
+        cls.dir = Path(cls.tmp.name) / "run"
+        make_run(cls.dir)
+        cls.bench_run = compare.Run(cls.dir)
+        cls.recs = compare.score(cls.bench_run)
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.tmp.cleanup()
+
+    def errs(self, name):
+        return [r["err"][name] for r in self.recs]
+
+    def test_trials_and_truth(self):
+        self.assertEqual(len(self.recs), len(TRIALS))
+        self.assertEqual(self.bench_run.screen, 2)
+        for r, (_, hy, hp, nudged) in zip(self.recs, TRIALS):
+            self.assertAlmostEqual(r["truth"][0], hy, places=6)
+            self.assertAlmostEqual(r["truth"][1], hp, places=6)
+            self.assertEqual(r["nudged"], nudged)
+        # 25 degrees left is the screen's left edge here (x 0)
+        self.assertEqual([r["where"] for r in self.recs], ["edge" if t[1] == 25 else "inside" for t in TRIALS])
+        self.assertAlmostEqual(self.recs[0]["since"], 3 + 500, delta=0.01)
+
+    def test_live(self):
+        for r, (_, hy, hp, _) in zip(self.recs, TRIALS):
+            self.assertAlmostEqual(r["err"]["steamvr"], compare.angle((hy, hp + 1), (hy, hp)), places=4)
+            self.assertAlmostEqual(r["err"]["steamvr-eyes"], r["err"]["steamvr"], places=6)
+            self.assertAlmostEqual(r["err"]["own-live"], compare.angle((hy + 0.5, hp), (hy, hp)), places=4)
+            fused = (hy + 0.35, hp + 0.3)
+            self.assertAlmostEqual(r["err"]["fuse"], compare.angle(fused, (hy, hp)), places=4)
+        # steamvr+clicks learns the 1 degree from the first click on
+        self.assertTrue(all(e < 1e-6 for e in self.errs("steamvr+clicks")[1:]))
+
+    def test_pointer(self):
+        for r, (_, hy, hp, nudged) in zip(self.recs, TRIALS):
+            want = compare.angle((hy + 0.5, hp), (hy, hp)) if nudged else 3 / PPD
+            self.assertAlmostEqual(r["err"]["pointer"], want, delta=0.02)
+            if nudged:
+                self.assertAlmostEqual(r["check"], 3 / PPD, delta=0.02)  # the click vs the lesson's truth
+
+    def test_replays(self):
+        before = [k for k, t in enumerate(TRIALS) if t[0] < SLIP_AT]
+        late = [k for k, t in enumerate(TRIALS) if t[0] > SLIP_AT + 8]
+        slip_deg = SLIP[0] / K
+        for name in ("own-replay", "no-jump", "both-jump", "edge-guard", "every-click", "bridge"):
+            e = self.errs(name)
+            self.assertTrue(all(e[k] < 0.05 for k in before), (name, e))
+        # The slip: the glint jump puts it right some seconds later; without it, it stays 2 degrees
+        # off until a click (there are none after it here); every click puts it right at once.
+        for name in ("own-replay", "both-jump", "edge-guard", "bridge"):
+            e = self.errs(name)
+            self.assertTrue(all(e[k] < 0.1 for k in late), (name, e))
+        nj = self.errs("no-jump")
+        self.assertTrue(all(abs(nj[k] - slip_deg) < 0.15 for k in late), nj)
+        ec = self.errs("every-click")
+        self.assertTrue(all(ec[k] < 0.1 for k in late), ec)
+        k16 = next(k for k, t in enumerate(TRIALS) if t[0] == 16)   # 2 s after: nothing has caught it yet
+        self.assertAlmostEqual(self.errs("own-replay")[k16], slip_deg, delta=0.15)
+        # The live gaze here is made up, 0.5 degrees from the replay's
+        for k in before:
+            self.assertAlmostEqual(self.recs[k]["replay_vs_live"], compare.angle((TRIALS[k][1] + 0.5, TRIALS[k][2]),
+                                                                                 TRIALS[k][1:3]), delta=0.05)
+
+    def test_edge_guard(self):
+        cal = eyes_model.Calibration.load(self.dir / "start" / "eyes-calibration.json")
+        box = compare.calibration_box(self.bench_run, cal)
+        self.assertEqual(box, (-20, 20, -15, 15))
+        clicks = [(T0 + 5, T0 + 4.9, (10.0, 0.0)), (T0 + 11, T0 + 10.9, (25.0, 0.0))]
+        rp = compare.Replay(self.bench_run, self.bench_run.frames(), cal, {}, None, clicks, guard=True, box=box)
+        rp()
+        self.assertEqual((rp.taught, rp.skipped), (1, 1))
+
+    def test_screen_scale(self):
+        """A window 800 x 640 on the 1000 x 800 screen with Qt's pixel ratio 2 (on the Frame: 2752 wide
+        on 3440 pixels): the window's pixels are scaled by the sizes in run.json, not by the ratio."""
+        self.assertIn("no screen sizes", " ".join(self.bench_run.notes))   # this run has none: the ratio (1)
+        d = Path(self.tmp.name) / "scaled"
+        d.mkdir()
+        for f in self.dir.iterdir():
+            if f.is_file():
+                (d / f.name).write_bytes(f.read_bytes())
+        (d / "start").symlink_to(self.dir / "start")
+        info = json.loads((d / "run.json").read_text())
+        info.update(screen={"w": 800, "h": 640, "dpr": 2.0}, screens={"2": [1000, 800]})
+        (d / "run.json").write_text(json.dumps(info))
+        rows = []
+        for line in (d / "trials.jsonl").read_text().splitlines():
+            r = json.loads(line)
+            for k in ("click", "truth"):
+                r[k]["x"], r[k]["y"] = 0.8 * r[k]["x"], 0.8 * r[k]["y"]
+            r["x"], r["y"] = r["truth"]["x"], r["truth"]["y"]
+            rows.append(json.dumps(r))
+        (d / "trials.jsonl").write_text("\n".join(rows) + "\n")
+        run = compare.Run(d)
+        self.assertEqual((run.sx, run.sy), (1.25, 1.25))
+        self.assertEqual(run.notes, [])
+        for r, (_, hy, hp, _) in zip(compare.score(run), TRIALS):
+            self.assertAlmostEqual(r["truth"][0], hy, places=6)
+            self.assertAlmostEqual(r["truth"][1], hp, places=6)
+
+    def test_panel_run(self):
+        """A run on the headset's panel: the truth is the target's direction (no screen), there's
+        no pointer, and the nudges' lessons (none would come from the bench) aren't matched."""
+        d = Path(self.tmp.name) / "panel"
+        d.mkdir()
+        for f in self.dir.iterdir():
+            if f.is_file():
+                (d / f.name).write_bytes(f.read_bytes())
+        (d / "start").symlink_to(self.dir / "start")
+        info = json.loads((d / "run.json").read_text())
+        del info["screen"]
+        info.update(display="panel", look=[0.4, 0.1])
+        (d / "run.json").write_text(json.dumps(info))
+        with open(d / "trials.jsonl", "w") as f:
+            for i, (tc, hy, hp, _) in enumerate(TRIALS):
+                f.write(json.dumps({"kind": "trial", "i": i, "u": 0.0 if hy == 25 else 0.5, "v": 0.5, "yaw": hy, "pitch": hp,
+                                    "click": {"t": T0 + tc, "by": "button"},
+                                    "truth": {"yaw": hy, "pitch": hp, "from": "target"}}) + "\n")
+        run = compare.Run(d)
+        self.assertTrue(run.panel)
+        self.assertEqual(run.notes, [])
+        recs = compare.score(run)
+        self.assertEqual(len(recs), len(TRIALS))
+        for r, (_, hy, hp, _) in zip(recs, TRIALS):
+            self.assertEqual(r["truth"], (hy, hp))
+            self.assertFalse(r["nudged"])
+            self.assertNotIn("pointer", r["err"])
+            self.assertAlmostEqual(r["err"]["steamvr"], compare.angle((hy, hp + 1), (hy, hp)), places=4)
+            self.assertAlmostEqual(r["err"]["own-live"], compare.angle((hy + 0.5, hp), (hy, hp)), places=4)
+        self.assertEqual([r["where"] for r in recs], ["edge" if t[1] == 25 else "inside" for t in TRIALS])
+
+    def test_fusions(self):
+        """Fitted on the other clicks, SteamVR's affine takes out its 1 degree; the fusion is
+        halfway between that and ours (both our eyes are found), ours-else-steam is ours."""
+        recs = compare.score(self.bench_run)
+        compare.cross(recs)
+        before = [k for k, t in enumerate(TRIALS) if t[0] < SLIP_AT]
+        for k, (r, (_, hy, hp, _)) in enumerate(zip(recs, TRIALS)):
+            self.assertLess(r["err"]["steamvr-affine-loo"], 1e-6)
+            self.assertAlmostEqual(r["err"]["fusion-loo"], compare.angle((hy + 0.25, hp), (hy, hp)), places=4)
+            self.assertAlmostEqual(r["err"]["ours-else-steam-loo"], compare.angle((hy + 0.5, hp), (hy, hp)), places=4)
+            self.assertNotIn("steamvr-affine-xrun", r["err"])   # one run: no other runs to fit on
+            if k in before:   # the replay's eyes are right until the slip: so is the fusion with them
+                self.assertLess(r["err"]["fusion-loo:own-replay"], 0.05)
+        # Steady pupil sizes: the size guards refuse nothing
+        for name in ("guard-both", "guard-long", "no-guard"):
+            self.assertEqual(self.errs(name), self.errs("own-replay"), name)
+
+    def test_fusion_from_start(self):
+        """A run started with a fusion (start/gazed-fusion.json), and --fit-fusion's."""
+        with tempfile.TemporaryDirectory() as tmp:
+            d = Path(tmp) / "fused"
+            shutil.copytree(self.dir, d)
+            f = compare.Fusion()
+            f.fit([(hy, hp + 1, hy, hp) for hy in (-20, 0, 20) for hp in (-10, 0, 10)], {"how": "test"})
+            f.save(d / "start" / "gazed-fusion.json")
+            recs = compare.score(compare.Run(d))
+            for r, (_, hy, hp, _) in zip(recs, TRIALS):
+                self.assertLess(r["err"]["steamvr-affine"], 1e-6)
+                self.assertLess(r["err"]["steamvr-affine+clicks"], 1e-6)
+                self.assertAlmostEqual(r["err"]["fusion"], compare.angle((hy + 0.25, hp), (hy, hp)), places=4)
+            out = Path(tmp) / "fusion.json"
+            compare.fit_fusion(recs, out, ["fused"])
+            g = compare.Fusion.load(out)
+            self.assertTrue(g.ready())
+            self.assertEqual((g.info["how"], g.info["dots"]), ("bench", len(TRIALS)))
+            dy, dp = g.steam.get(5.0, 5.0, g.MODE)
+            self.assertAlmostEqual(dy, 0.0, places=3)
+            self.assertAlmostEqual(dp, -1.0, places=2)
+
+    def test_main(self):
+        out = Path(self.tmp.name) / "recs.json"
+        sys.argv = ["compare.py", str(self.dir.parent), "--json", str(out)]
+        compare.main()
+        self.assertEqual(len(json.loads(out.read_text())), len(TRIALS))
+
+
+if __name__ == "__main__":
+    unittest.main()

gaze/build.sh

@@ -13,7 +13,8 @@ g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -
   -o build/ft-gaze ft-gaze.cpp $OPENVR_LIBS -lpthread
 stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
 [ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
-g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
+g++ -std=c++17 -O2 -ffunction-sections -fdata-sections -Wl,--gc-sections -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
   -o build/ft-gazepanel panel/ft-gazepanel.cpp $OPENVR_LIBS \
   $(pkg-config --libs gbm libdrm) -lpthread
+strip --strip-unneeded build/ft-gaze build/ft-gazepanel
 echo "built build/ft-gaze build/ft-gazepanel"'

gaze/ft-gaze.cpp

@@ -14,10 +14,12 @@
 // about 1.5 KB, 130 KB a second through podman's relay.
 //
 //   {"t":<sample time, CLOCK_MONOTONIC_RAW s>,"age":<ms old when read>,"n":<sample counter>,
-//    "head":{"yaw":..,"pitch":..,"hit":HIT},          head forward ray (for head nudging)
+//    "head":{"yaw":..,"pitch":..,"q":Q,"hit":HIT},    head forward ray (for head nudging)
 //    "src":{"action":SRC,"mmap1":SRC,"mmap2":SRC,"left":SRC,"right":SRC,"own":SRC},"eye":EYE}
-//   SRC = {"hy":..,"hp":..,"hit":HIT} or {"ok":0}    hy/hp: gaze direction relative to the
-//                                                     head, degrees (yaw +left, pitch +up)
+//   SRC = {"hy":..,"hp":..,"hq":Q,"hit":HIT} or {"ok":0}  hy/hp: gaze direction relative to
+//                                                     the head, degrees (yaw +left, pitch +up);
+//                                                     hq: the head pose it's relative to (the
+//                                                     one at the source's own sample time)
 //         mmap1 adds "open":[l,r] (probably eye openness, 0 in a blink) and "dist" (vergence
 //         distance, m); both mmap sets add "lr", the angle between the eyes (deg), which
 //         jumps when the tracker loses an eye, and "eyes":[[hy,hp],[hy,hp]], each eye's own
@@ -39,6 +41,10 @@
 //         variance (about 2e-5 on a clear view of the eye, rising as the lid or lashes get
 //         in the way), "new" whether it changed since the last sample (it freezes while the
 //         tracker can't see that eye, and in blinks). "eye" is null without the mmap.
+//   Q = [w,x,y,z]   a head pose's rotation as a unit quaternion (standing universe), so a
+//         head-relative direction can be turned into the room and back: the gaze service's
+//         fixation lock holds the gaze still in the room, where a look at a screen stays put
+//         while the head turns (gazecal.GazeLock).
 //   HIT = {"s":<screen>,"x":..,"y":..,"j":[dx/dhy,dy/dhy,dx/dhp,dy/dhp],"dpp":<deg per px>}
 //         or null. x, y are pixels on that screen; j is pixels per degree of head-relative
 //         yaw and pitch there, so a correction in degrees can be turned into pixels and back.
@@ -468,13 +474,38 @@ std::string HitJson(const std::vector<Screen> &screens, const vr::HmdMatrix34_t
     return buf;
 }
 
+// Q: a pose's rotation as a unit quaternion [w,x,y,z] (v' = R v, R the pose's 3x3 part).
+std::string QuatJson(const vr::HmdMatrix34_t &p) {
+    const auto &m = p.m;
+    double w, x, y, z;
+    const double tr = m[0][0] + m[1][1] + m[2][2];
+    if (tr > 0) {
+        const double s = 2 * std::sqrt(tr + 1);  // 4w
+        w = s / 4, x = (m[2][1] - m[1][2]) / s, y = (m[0][2] - m[2][0]) / s, z = (m[1][0] - m[0][1]) / s;
+    } else if (m[0][0] > m[1][1] && m[0][0] > m[2][2]) {
+        const double s = 2 * std::sqrt(1 + m[0][0] - m[1][1] - m[2][2]);  // 4x
+        w = (m[2][1] - m[1][2]) / s, x = s / 4, y = (m[0][1] + m[1][0]) / s, z = (m[0][2] + m[2][0]) / s;
+    } else if (m[1][1] > m[2][2]) {
+        const double s = 2 * std::sqrt(1 + m[1][1] - m[0][0] - m[2][2]);  // 4y
+        w = (m[0][2] - m[2][0]) / s, x = (m[0][1] + m[1][0]) / s, y = s / 4, z = (m[1][2] + m[2][1]) / s;
+    } else {
+        const double s = 2 * std::sqrt(1 + m[2][2] - m[0][0] - m[1][1]);  // 4z
+        w = (m[1][0] - m[0][1]) / s, x = (m[0][2] + m[2][0]) / s, y = (m[1][2] + m[2][1]) / s, z = s / 4;
+    }
+    const double n = std::sqrt(w * w + x * x + y * y + z * z);
+    if (n < 1e-9) return "[1,0,0,0]";
+    char buf[80];
+    std::snprintf(buf, sizeof buf, "[%.6f,%.6f,%.6f,%.6f]", w / n, x / n, y / n, z / n);
+    return buf;
+}
+
 std::string SrcJson(const std::vector<Screen> &screens, const vr::HmdMatrix34_t &head, Vec3 dHead,
                     const std::string &extra = "") {
     double yaw, pitch;
     Angles(Normalize(dHead), yaw, pitch);
     char buf[96];
     std::snprintf(buf, sizeof buf, "{\"hy\":%.4f,\"hp\":%.4f,", yaw, pitch);
-    return buf + extra + "\"hit\":" + HitJson(screens, head, yaw, pitch) + "}";
+    return buf + extra + "\"hq\":" + QuatJson(head) + ",\"hit\":" + HitJson(screens, head, yaw, pitch) + "}";
 }
 
 // Sources (--sources, "sources LIST" on stdin): a bit each.
@@ -798,11 +829,12 @@ int main(int argc, char **argv) {
             const Vec3 f = Rotate(headNow, {0, 0, -1});
             yaw = std::atan2(-f.x, -f.z) * 180 / M_PI;
             pitch = std::asin(std::clamp(f.y, -1.0, 1.0)) * 180 / M_PI;
-            std::printf("{\"t\":%.5f,\"age\":%.1f,\"n\":%u,\"head\":{\"yaw\":%.4f,\"pitch\":%.4f,\"hit\":%s},"
+            std::printf("{\"t\":%.5f,\"age\":%.1f,\"n\":%u,\"head\":{\"yaw\":%.4f,\"pitch\":%.4f,\"q\":%s,\"hit\":%s},"
                         "\"src\":{\"action\":%s,\"mmap1\":%s,\"mmap2\":%s,\"left\":%s,\"right\":%s,\"own\":%s},"
                         "\"eye\":%s}\n",
-                        s.t, (now - s.t) * 1000, s.n, yaw, pitch, HitJson(list, headNow, 0, 0).c_str(), action.c_str(),
-                        m1.c_str(), m2.c_str(), left.c_str(), right.c_str(), own.c_str(), eye.c_str());
+                        s.t, (now - s.t) * 1000, s.n, yaw, pitch, QuatJson(headNow).c_str(),
+                        HitJson(list, headNow, 0, 0).c_str(), action.c_str(), m1.c_str(), m2.c_str(), left.c_str(),
+                        right.c_str(), own.c_str(), eye.c_str());
             if (std::fflush(stdout) != 0) break;  // the reader went away
         }
 

gaze/ft-gazed

@@ -10,9 +10,15 @@ again when the file changes:
                            grabber, and ft-eyes' Python in this checkout), else SteamVR's; it
                            switches when ours is installed or removed.
   GAZE_EYE=auto|left|right the eye bias (gazecal.EyeWeights): auto weights each eye by how far
-                           off it was at your recent nudges; left or right counts that eye twice
-                           as much as the other. Either eye alone carries the gaze when the
-                           other isn't seen.
+                           off it was at your recent nudges (since our tracker's last
+                           calibration: a new one starts them over); left or right counts that
+                           eye twice as much as the other. Either eye alone carries the gaze
+                           when the other isn't seen.
+  GAZE_FUSION=off|on       with our tracker: SteamVR's gaze too (gazecal.Fusion, fusion.json).
+                           SteamVR's two eyes' mean, corrected by an affine fitted at our
+                           calibration's dots, is mixed half and half with ours where both our
+                           eyes are found, and used alone where ours has one or none. off (the
+                           default) until the bench says so (gaze/bench/compare.py's fusion).
 
 Runs ft-gaze (in the dev container), and for every eye tracker sample (90 Hz):
 
@@ -27,8 +33,15 @@ Runs ft-gaze (in the dev container), and for every eye tracker sample (90 Hz):
      ft-gazeprobe (calibration.json, reloaded when the probe changes it) plus what the
      pointer's corrections have taught that eye since (LiveCorrection, saved in
      pointer-lessons.json), then weighted by the eye bias. A lost or closed eye drops out.
-     Our own tracker keeps its own calibration, so its eyes are used as they come;
-  3. smooths it with a fixation lock (the running mean of the current fixation, 1 degree);
+     Our own tracker keeps its own calibration, so its eyes are used as they come, and with
+     GAZE_FUSION=on (and a fusion.json) mixed with SteamVR's (above). When one of its eyes
+     drops out or comes back, gazecal.EyeBridge eases the step between the two eyes' gaze
+     and the one eye's over about 0.3 s, instead of a jump of about a degree. SteamVR's eyes count
+     only when both are open and found, and not while SteamVR tracks one eye only;
+  3. smooths it with a fixation lock (the running mean of the current fixation, 1 degree), on
+     where the gaze points in the room, so a look at a screen stays put while the head turns
+     (gazecal.GazeLock: ft-gaze gives each sample's head pose). With our tracker, an eye whose
+     lids came down stays out for BLINK_SETTLE after they're up again;
   4. sends it to the pointer helper: "gz <yaw> <pitch> <raw yaw> <raw pitch>", head-relative
      degrees (yaw +left, pitch +up). The helper uses it only in gaze mode.
 
@@ -80,7 +93,8 @@ tracker's next click after it starts again re-seats it, as after the headset was
 gaze mode on after a while opens the quick check.
 
 ft-gaze prints only the sources this uses ("sources LIST" on its stdin, see wanted_sources):
-own and mmap1 with our tracker, left, right and mmap1 with SteamVR's eyes, and every source
+own and mmap1 with our tracker (and left, right with the fusion), left, right and mmap1 with
+SteamVR's eyes, and every source
 while a check or the calibration runs or is asked for, since those record them all. So
 SteamVR's gaze action, the one source that calls into vrserver, is read only then (or with
 --source action).
@@ -98,7 +112,7 @@ Control socket: abstract unix datagram "@ft_gazed":
                                    POINTER_GAZE_NUDGE_MAX, so the quick check (gazecheck.py)
   status                           reply: one JSON object
   forget                           drop what the lessons taught (the calibration stays)
-  reload                           read calibration.json and the settings again
+  reload                           read calibration.json, fusion.json and the settings again
   eyes <seconds>                   keep our own tracker running that much longer (at most 120),
                                    whatever GAZE_TRACKER says: the probe's lease. Reply: "ok"
   wake <seconds>                   keep the gaze running (not idle) that much longer (at most 120),
@@ -109,8 +123,8 @@ Control socket: abstract unix datagram "@ft_gazed":
   fivecheck                        the five-dot check now (it otherwise follows a quick check
                                    that didn't fix the tracker)
   calaccept | calquit              from the pointer helper while the panel is up: take this dot
-                                   now (a left click, Meta+J, or a gaze precision or gaze drag
-                                   press) | close it (a right click, Meta+K)
+                                   now (a left click, Enter, a pinch, Meta+J, or a gaze
+                                   precision or gaze drag press) | close it (a right click, Meta+K)
 
 Options: --source action|mmap1|mmap2 (the older one-source path with that source, whatever
 the settings say; set 2 was a little quieter in the probe, but loses the pointer whenever
@@ -134,8 +148,8 @@ from collections import deque
 from pathlib import Path
 
 sys.path.insert(0, str(Path(__file__).resolve().parent))
-from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correction, EyeFallback,  # noqa: E402
-                     EyeWeights, Fixation, LiveCorrection, SteamEyeLog, TrackedEye)
+from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, FUSION, MODELS, STATE, Correction, EyeBridge,  # noqa: E402
+                     EyeFallback, EyeWeights, Fusion, GazeLock, LiveCorrection, SteamEyeLog, TrackedEye)
 from gazecheck import Checks  # noqa: E402
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "models"))
@@ -172,6 +186,8 @@ IDLE_AFTER = 30.0  # seconds after the gaze was last in use before ft-gaze and o
 WAKE_SETTLE = 15.0  # seconds after waking that the tracker may take to start sending
 WAKE_MAX = 120.0
 KEYBOARD_PITCH = -20.0  # degrees: gaze under this, on no screen, is a look at the keyboard
+BLINK_SETTLE = 0.1  # seconds our tracker's eye stays out after its lids came down: the pupil the
+                    # lid cuts finds wrong for a few frames past SteamVR's openness (2026-10-17)
 
 
 class PointerLessons(LiveCorrection):
@@ -190,9 +206,15 @@ def log(msg):
     print(f"ft-gazed: {msg}", file=sys.stderr, flush=True)
 
 
+def venv_built(python):
+    """Is a build/venv's python there? It's for the container: in a release its link points at the
+    image's Python (/usr/bin/python3.14), which the host doesn't have, so the link itself counts."""
+    return os.path.lexists(python)
+
+
 def own_installed():
     """Is our own tracker installed: its frame grabber, and ft-eyes' Python here?"""
-    return all(p.exists() for p in EYEGRAB) and EYES_PYTHON.exists()
+    return all(p.exists() for p in EYEGRAB) and venv_built(EYES_PYTHON)
 
 
 def read_settings():
@@ -218,6 +240,18 @@ def read_settings():
     return tracker, setting, bias if bias in BIASES else "auto", nudge
 
 
+def read_fusion():
+    """GAZE_FUSION from frametop.conf: True for on (off, the default, for anything else)."""
+    try:
+        for line in CONF.read_text().splitlines():
+            k, _, v = line.split("#", 1)[0].partition("=")
+            if k.strip() == "GAZE_FUSION":
+                return v.strip().lower() == "on"
+    except OSError:
+        pass
+    return False
+
+
 def mtime(path):
     try:
         return path.stat().st_mtime
@@ -226,6 +260,8 @@ def mtime(path):
 
 
 class Service:
+    fusion_on, fusion = False, None   # (set in __init__; here too for tests that build one by parts)
+
     def __init__(self, source, verbose, to=POINTER):
         self.override, self.verbose, self.to = source, verbose, to
         self.source = source or "mmap1"  # the older path's source
@@ -237,14 +273,20 @@ class Service:
         self.cal_mtime = None
         self.lives = {name: PointerLessons() for name in SOURCES}
         self.weights = {t: EyeWeights(self.bias) for t in TRACKERS}
+        self.own_made = None  # when our tracker's calibration was made, as its misses knew it
         self.dirty = False
+        self.fusion_on = read_fusion()
+        self.fusion, self.fusion_mtime = None, None
+        self.load_fusion()
         self.load_calibration()
         self.load_lessons()
         self.steam = SteamEyeLog()
         self.steam.poll()
         self.refit()
-        self.fix = Fixation(radius=1.0)
+        self.fix = GazeLock(radius=1.0)
         self.opens = (deque(maxlen=90), deque(maxlen=90))  # left, right
+        self.low_until = [0.0, 0.0]  # sample time until which each eye counts as closed (BLINK_SETTLE)
+        self.bridge = EyeBridge()    # our tracker: no jump when an eye drops out or comes back
         self.vergence = deque(maxlen=90)
         self.fallback = EyeFallback()
         # SteamVR's tracker following one eye only (gazecal.tracked_eye): 0 left, 1 right, or None.
@@ -253,7 +295,8 @@ class Service:
         self.lost = [False, False]
         self.bad_at = [0.0, 0.0]    # sample time an eye was last lost or closed
         self.counts = {"samples": 0, "sent": 0, "blinks": 0, "one_eye": 0, "one_eye_used": 0, "lost_left": 0,
-                       "lost_right": 0, "looking_down": 0, "dropped": 0, "lessons_taken": 0, "refused": 0}
+                       "lost_right": 0, "looking_down": 0, "dropped": 0, "lessons_taken": 0, "refused": 0,
+                       "fused": 0, "steam_only": 0}
         self.last_sample = 0.0
         self.last = None
         self.last_kind = None
@@ -300,6 +343,11 @@ class Service:
         sides = SIDES if self.one_eye is None else (SIDES[self.one_eye],)
         return "eyes" if all(self.models[e].samples for e in sides) else "source"
 
+    @property
+    def fusing(self):
+        """Our tracker's gaze mixed with SteamVR's (GAZE_FUSION=on, a fitted fusion.json)."""
+        return self.kind == "own" and self.fusion_on and self.fusion is not None and self.fusion.ready()
+
     # --- Settings, calibration and lessons ---
 
     @staticmethod
@@ -336,6 +384,12 @@ class Service:
         self.conf_mtime = mtime(CONF)
         self.check_eyes_model()
         tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
+        fusion_on = read_fusion()
+        if fusion_on != self.fusion_on:
+            self.fusion_on = fusion_on
+            log(f"fusion {'on' if fusion_on else 'off'}" + ("" if not fusion_on or (self.fusion and self.fusion.ready())
+                                                            else f": no fitted {FUSION} yet (our tracker's calibration makes one)"))
+            self.fix.reset()
         if (tracker, bias) != (self.tracker, self.bias):
             auto = ""
             if self.tracker_setting == "auto":
@@ -349,6 +403,14 @@ class Service:
                 w.bias = bias
             self.fix.reset()
 
+    def load_fusion(self):
+        self.fusion_mtime = mtime(FUSION)
+        self.fusion = Fusion.load(FUSION)
+        if self.fusion:
+            log(f"fusion: SteamVR's affine from {self.fusion.info.get('dots', 0)} dots "
+                f"({self.fusion.info.get('how', '?')}, {self.fusion.info.get('made', '?')}), "
+                + ("on" if self.fusion_on else "off (GAZE_FUSION)"))
+
     def load_calibration(self):
         try:
             mt = CALIBRATION.stat().st_mtime
@@ -377,12 +439,14 @@ class Service:
         for t, misses in (d.get("misses") or {}).items():
             if t in self.weights:
                 self.weights[t] = EyeWeights(self.bias, misses)
+        self.own_made = d.get("own_made")
         log(", ".join(f"{n} {len(self.lives[n].samples)}" for n in (self.source,) + SIDES) + " lessons")
 
     def save_lessons(self):
         tmp = LESSONS.with_suffix(".tmp")
         tmp.write_text(json.dumps({"version": 2, "sources": {n: lv.samples for n, lv in self.lives.items() if lv.samples},
-                                   "misses": {t: w.misses for t, w in self.weights.items()}}))
+                                   "misses": {t: w.misses for t, w in self.weights.items()},
+                                   "own_made": self.own_made}))
         tmp.replace(LESSONS)
         self.dirty = False
 
@@ -528,7 +592,7 @@ class Service:
             return "all"
         kind = self.kind
         if kind == "own":
-            return "own,mmap1"
+            return "own,mmap1,left,right" if self.fusing else "own,mmap1"
         if kind == "eyes":
             return "left,right,mmap1"
         return ",".join(dict.fromkeys((self.source, "mmap1", "mmap2")))  # mmap2: each eye, for the fallback
@@ -594,7 +658,7 @@ class Service:
     def start_eyes(self):
         """Our own tracker, in the container, with build/venv's numpy and OpenCV. Like
         ft-gaze, it quits when its stdin closes."""
-        if not EYES_PYTHON.exists():
+        if not venv_built(EYES_PYTHON):
             log(f"ft-eyes isn't built: run {REPO}/gaze/tracker/build.sh")
             self.eyes_restart_at = time.monotonic() + 30
             return
@@ -725,7 +789,7 @@ class Service:
             if "hy" not in src:
                 return
             eyes = [tuple(e) if e else None for e in (src.get("eyes") or [None, None])]
-            hp, hit = src["hp"], src.get("hit")
+            hp, hit, then = src["hp"], src.get("hit"), src.get("hq")
         else:
             per = [s["src"].get(name) or {} for name in SIDES]
             eyes = [(p["hy"], p["hp"]) if "hy" in p else None for p in per]
@@ -733,6 +797,7 @@ class Service:
                 return
             one = self.one_eye
             hp, hit = (eyes[one] if one is not None and eyes[one] else next(e for e in eyes if e))[1], m1.get("hit")
+            then = next((p["hq"] for p in per if "hq" in p), None)
         self.counts["samples"] += 1
         self.last_sample = time.monotonic()
         down = hp < KEYBOARD_PITCH and not hit
@@ -744,11 +809,22 @@ class Service:
             # SteamVR judges only that eye's openness, and a blink closes both: ours still
             # sees the other, so it stays in.
             low[1 - self.one_eye] = low[self.one_eye]
+        if own:
+            for k in (0, 1):
+                if low[k]:
+                    self.low_until[k] = s["t"] + BLINK_SETTLE
+                elif s["t"] < self.low_until[k]:
+                    low[k] = True
         # Our tracker finds the pupils itself; SteamVR's openness still marks the blinks.
         bad = [eyes[k] is None or low[k] or (not own and self.lost[k]) for k in (0, 1)]
         if not own and self.one_eye is not None:
             bad[1 - self.one_eye] = True  # SteamVR ignores that eye
-        if all(bad):
+        steam = None
+        if own and self.fusing and self.one_eye is None:
+            per = [s["src"].get(name) or {} for name in SIDES]
+            steam = self.fusion.steam_gaze(*[(p["hy"], p["hp"]) if "hy" in p and not low[k] and not self.lost[k]
+                                             else None for k, p in enumerate(per)])
+        if all(bad) and steam is None:
             self.counts["blinks"] += 1
             return
         if any(bad):
@@ -762,8 +838,15 @@ class Service:
             for name, e in zip(SIDES, seen):
                 c = self.correction(name, *e) if e else None
                 corrected.append((e[0] + c[0], e[1] + c[1]) if e else None)
-        gy, gp = self.weights["own" if own else "steam"].combine(corrected)
-        fy, fp = self.fix(gy, gp, s["t"], 1.0)
+        g = self.weights["own" if own else "steam"].combine(corrected)
+        if own:
+            g = self.bridge(s["t"], seen, g)
+        if steam is not None:
+            n = sum(e is not None for e in seen)
+            self.counts["fused" if n == 2 else "steam_only"] += 1
+            g = self.fusion.combine(g, n, steam)
+        gy, gp = g
+        fy, fp = self.fix(gy, gp, s["t"], then, (s.get("head") or {}).get("q"))
         self.send(s["t"], fy, fp, fy, fp, seen)
 
     def on_source_sample(self, s):
@@ -796,7 +879,7 @@ class Service:
                     self.counts["dropped"] += 1
                     return
                 hy, hp = eyes[one]
-            fy, fp = self.fix(hy, hp, s["t"], 1.0)
+            fy, fp = self.fix(hy, hp, s["t"], src.get("hq"), (s.get("head") or {}).get("q"))
             cy, cp = self.correction(self.source, fy, fp)
             self.send(s["t"], fy + cy, fp + cp, fy, fp, None)
             return
@@ -830,8 +913,8 @@ class Service:
             if eyes and s["t"] - max(self.bad_at) > SETTLE:
                 for k in (0, 1):
                     self.fallback.update(k, eyes[k][0], eyes[k][1], hy, hp)
-        # The fixation lock works in degrees here (1 degree per "pixel").
-        fy, fp = self.fix(hy, hp, s["t"], 1.0)
+        # The fixation lock works in degrees here, on the gaze's direction in the room.
+        fy, fp = self.fix(hy, hp, s["t"], src.get("hq"), (s.get("head") or {}).get("q"))
         cy, cp = self.correction(self.source, fy, fp)
         self.send(s["t"], fy + cy, fp + cp, fy, fp, None)
 
@@ -893,6 +976,7 @@ class Service:
                     reply = "error bad seconds"
             elif words[:1] == ["reload"]:
                 self.load_settings()
+                self.load_fusion()
                 self.load_calibration()
                 self.refit()
                 reply = "ok"
@@ -916,9 +1000,29 @@ class Service:
                     self.own, self.own_at = json.loads(data), time.monotonic()
                 except ValueError:
                     pass
+                else:
+                    self.note_own_calibration()
             else:
                 log(f"our tracker: {data}")
 
+    def note_own_calibration(self):
+        """Our tracker's status names a calibration other than the one its eye misses were
+        measured on: they start over, and the eyes count alike until each has EyeWeights.MIN
+        again. The misses say how far each eye's old mapping was off, and a new calibration
+        replaces both mappings (it can drop an eye, or be someone else's). Live 2026-10-18 a
+        calibration's own dots fed the right eye misses of the right-only one before it, and the
+        balance after it was that one's. Whatever made the calibration: the panel, the probe."""
+        made = (self.own.get("calibration") or {}).get("made")
+        if not made or made == self.own_made:
+            return
+        w = self.weights["own"]
+        if self.own_made is not None and any(w.misses):
+            log(f"our tracker's calibration from {made}: the eye weights start over "
+                f"(were {', '.join(f'{v:.2f}' for v in w.weights())} from {', '.join(str(len(m)) for m in w.misses)} misses)")
+            self.weights["own"] = EyeWeights(self.bias)
+        self.own_made = made
+        self.dirty = True
+
     def status(self):
         kind = self.kind
         tracker = "own" if kind == "own" else "steam"
@@ -940,7 +1044,13 @@ class Service:
             cal = own.get("calibration") or {}
             st.update(calibration_samples=cal.get("dots", 0), calibration_made=cal.get("made"),
                       lessons=max(len(m) for m in w.misses), own_running=bool(own),
-                      own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()))
+                      own_reseat=any(e.get("reseat") for e in own.get("eyes", {}).values()),
+                      size_guard=own.get("size_guard"), size_guard_counts=own.get("size_guard_counts"),
+                      own_mapped=own.get("mapped"))
+        f = self.fusion
+        st["fusion"] = {"setting": "on" if self.fusion_on else "off", "in_use": self.fusing,
+                        "fitted": bool(f and f.ready()), "dots": f.info.get("dots", 0) if f else 0,
+                        "how": f.info.get("how") if f else None, "made": f.info.get("made") if f else None}
         st.update(awake=self.awake, idle=None if self.awake else self.idle_reason())
         st.update(eyes_process=self.eyes_proc is not None, eyegrab=EYES_CAMS.exists(),
                   tracker_setting=self.tracker_setting, own_installed=own_installed())
@@ -959,6 +1069,8 @@ class Service:
         if mtime(CALIBRATION) != self.cal_mtime:
             self.load_calibration()
             self.refit()
+        if mtime(FUSION) != self.fusion_mtime:
+            self.load_fusion()
         if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
                                               and own_installed() != (self.tracker == "own")):
             self.load_settings()

gaze/gazecal.py

@@ -98,6 +98,99 @@ class Fixation:
         return self.sum[0] / self.count, self.sum[1] / self.count
 
 
+def direction(yaw, pitch):
+    """A head-relative (yaw, pitch) in degrees as a unit vector, as ft-gaze's Direction: -Z
+    forward, positive yaw turns left (about +Y), positive pitch looks up."""
+    y, p = math.radians(yaw), math.radians(pitch)
+    return (-math.sin(y) * math.cos(p), math.sin(p), -math.cos(y) * math.cos(p))
+
+
+def angles(v):
+    """(yaw, pitch) in degrees of a direction, as ft-gaze's Angles."""
+    n = math.sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]) or 1.0
+    return math.degrees(math.atan2(-v[0], -v[2])), math.degrees(math.asin(max(-1.0, min(1.0, v[1] / n))))
+
+
+def rotate(q, v, inverse=False):
+    """v turned by the unit quaternion q = [w, x, y, z] (ft-gaze's "hq" and the head's "q": head
+    space into the room), or back into head space with inverse."""
+    w, x, y, z = q
+    if inverse:
+        x, y, z = -x, -y, -z
+    # v + 2w (u x v) + 2 u x (u x v), u = (x, y, z)
+    cx, cy, cz = y * v[2] - z * v[1], z * v[0] - x * v[2], x * v[1] - y * v[0]
+    dx, dy, dz = y * cz - z * cy, z * cx - x * cz, x * cy - y * cx
+    return (v[0] + 2 * (w * cx + dx), v[1] + 2 * (w * cy + dy), v[2] + 2 * (w * cz + dz))
+
+
+class GazeLock:
+    """The fixation lock (Fixation, above) on where the gaze points in the room.
+
+    A look at something that stays put (a screen) stays put in the room while the head turns:
+    the eyes turn the other way, so the head-relative gaze slides. Locked in head-relative
+    degrees, the pointer rode along with the head until the slide passed the radius, then
+    snapped back, a jump every few hundred milliseconds; replaying two practice captures, during
+    still looks with the head moving 5-30 degrees a second that was 4 jumps over 1 degree a
+    second, and 13 in the one second it turned faster. Locked in the room it's 1 to 2 (2026-10-17).
+
+    Each sample's gaze is turned into the room with the head pose of its own sample time
+    (`then`, ft-gaze's "hq"), locked there, and turned back into the head with the newest pose
+    (`now`, the head's "q"), the closest one to the pose the pointer helper draws it with.
+    Without poses (an older ft-gaze) it's the head-relative lock."""
+
+    def __init__(self, radius=1.0):
+        self.radius = radius
+        self.head = Fixation(radius)
+        self.reset()
+
+    def reset(self):
+        self.head.reset()
+        self.sum = [0.0, 0.0, 0.0]
+        self.count = 0
+        self.outside = []
+        self.last_t = None
+
+    def __call__(self, yaw, pitch, t, then=None, now=None):
+        """The locked gaze, head-relative (yaw, pitch) degrees, for a head-relative sample at t."""
+        if not then or not now:
+            if self.count:
+                self.reset()
+            return self.head(yaw, pitch, t, 1.0)
+        if self.last_t is None and self.head.count:
+            self.head.reset()
+        m = self.lock(rotate(then, direction(yaw, pitch)), t)
+        return angles(rotate(now, m, inverse=True))
+
+    @staticmethod
+    def mean(s):
+        n = math.sqrt(s[0] * s[0] + s[1] * s[1] + s[2] * s[2]) or 1.0
+        return (s[0] / n, s[1] / n, s[2] / n)
+
+    def lock(self, v, t):
+        """Fixation's rules on room directions (unit vectors)."""
+        if self.last_t is not None and (t <= self.last_t or t - self.last_t > 0.5):
+            self.reset()
+        self.last_t = t
+        if self.count:
+            m = self.mean(self.sum)
+            dot = max(-1.0, min(1.0, m[0] * v[0] + m[1] * v[1] + m[2] * v[2]))
+            if math.degrees(math.acos(dot)) > self.radius:
+                self.outside.append(v)
+                if len(self.outside) < 2:
+                    return m  # one stray sample: probably noise
+                self.sum = [sum(p[k] for p in self.outside) for k in range(3)]
+                self.count = len(self.outside)
+                self.outside = []
+                return self.mean(self.sum)
+        self.outside = []
+        if self.count >= 90:  # the last second or so: a slow drift still gets followed
+            self.sum = [s * 89 / 90 for s in self.sum]
+            self.count = 89
+        self.sum = [self.sum[k] + v[k] for k in range(3)]
+        self.count += 1
+        return self.mean(self.sum)
+
+
 MODELS = ["none", "offset", "affine", "affine+grid", "quadratic", "quadratic+grid"]
 DEFAULT_MODEL = "quadratic"
 
@@ -456,6 +549,42 @@ class TrackedEye:
         return self.eye
 
 
+class EyeBridge:
+    """Our tracker's eyes, each with its own calibration, miss in their own ways, so the gaze
+    jumped whenever one eye dropped out or came back (a median 0.3-0.85 degrees in the 10-16
+    edges round): enough to break the fixation lock. While both are seen, this averages each
+    eye's offset from their combined gaze over about LEARN seconds; with one eye, the gaze is
+    that eye plus its offset, fading out over about FADE seconds, since the offset belongs to
+    where the eyes looked then.
+
+    `__call__` takes and gives head-relative degrees (yaw, pitch)."""
+
+    LEARN = 0.3
+    FADE = 0.3
+
+    def __init__(self):
+        self.off = [None, None]   # per eye: (yaw, pitch) from it to the combined gaze
+        self.at = None            # sample time both were last seen
+
+    def __call__(self, t, eyes, combined):
+        """The gaze at sample time t (s), from `combined`, the eye-weighted gaze of `eyes`
+        [(yaw, pitch) or None, (yaw, pitch) or None]."""
+        seen = [k for k in (0, 1) if eyes[k] is not None]
+        if len(seen) == 2:
+            a = 1.0 if self.at is None else 1.0 - math.exp(-max(t - self.at, 0.0) / self.LEARN)
+            for k in seen:
+                d = (combined[0] - eyes[k][0], combined[1] - eyes[k][1])
+                o = self.off[k]
+                self.off[k] = d if o is None else (o[0] + a * (d[0] - o[0]), o[1] + a * (d[1] - o[1]))
+            self.at = t
+            return combined
+        if len(seen) == 1 and self.at is not None and self.off[seen[0]] is not None:
+            f = math.exp(-max(t - self.at, 0.0) / self.FADE)
+            o = self.off[seen[0]]
+            return combined[0] + f * o[0], combined[1] + f * o[1]
+        return combined
+
+
 class EyeFallback:
     """The gaze from one eye, while the tracker has lost the other.
 
@@ -546,7 +675,10 @@ class EyeWeights:
     alike until then. Each miss is measured before its lesson teaches anything, so each is a
     fresh test. On SteamVR's own test (2026-09-29) its calibration dots said the left eye was
     the better one and new spots said the right, so the misses come from lessons, not the fit.
-    An eye that isn't seen (None) drops out, and the other carries the gaze alone."""
+    An eye that isn't seen (None) drops out, and the other carries the gaze alone. Our
+    tracker's start over with each new calibration of it (ft-gazed's note_own_calibration).
+    Not from the new fit's per-eye spread: on 2026-10-18's replays (frame-eyes
+    one_eye_replay.py) spread and equal weights came out the same, 2.29 degrees median."""
 
     LEAN = 2.0
     KEEP = 20
@@ -589,6 +721,126 @@ class EyeWeights:
         return (sum(wk * e[0] for wk, e in zip(w, seen)) / total, sum(wk * e[1] for wk, e in zip(w, seen)) / total)
 
 
+FUSION = STATE / "fusion.json"
+
+
+class Fusion:
+    """Our tracker and SteamVR's together (ft-gazed's GAZE_FUSION=on). On two Gaze Bench runs
+    (2026-10-17, 25 clicks on a head-locked panel out to 37 degrees) the two failed on
+    different looks: ours at its misfinds and with an eye lost, SteamVR with a steady bend
+    across the view. Ours was 2.01 degrees off at the median (6.47 at the 90th percentile),
+    SteamVR's two eyes' mean 2.67 (5.72). An affine correction fitted on the other clicks took
+    SteamVR's to 1.78 (4.26); ours where both its eyes were found, mixed half and half with
+    that, and SteamVR's alone where ours had one eye, to 1.51 (3.10). Fitted on the other run
+    only, 1.41 (4.59) (gaze/bench/compare.py's steamvr-affine-loo, fusion-loo, fusion-xrun).
+
+    So: SteamVR's eyes' mean, corrected by an affine fitted at the calibration's dots (gazecheck
+    records SteamVR's eyes at each of ours, and fits this after ours: fusion.json); then ours
+    mixed in at SHARE where both our eyes are found, SteamVR's alone where we have one, and ours
+    alone where SteamVR has no gaze.
+
+    click() adds an offset on top: the median of what the clicks since the headset went on said
+    was still off (SteamVR starts its eye model over then, and the error may move). ft-gazed
+    doesn't feed it yet: within one wear on those runs it made SteamVR's worse (2.15 to 2.72 at
+    the median, fitted on the other run), and across wears it's untested (compare.py's +clicks)."""
+
+    MODE = "affine"
+    RIDGE = 0.05    # Correction.fit's: on the gain and tilt, so a lopsided set of dots can't bend it far
+    SHARE = 0.5     # our tracker's share when both its eyes are found
+    MIN = 5         # dots for a fit
+    KEEP = 5        # clicks the offset on top comes from
+    STALE = 8.0     # degrees: a click this far off is a look elsewhere, not SteamVR's error
+
+    def __init__(self):
+        self.steam = Correction()
+        self.info = {}
+        self.points = []
+        self.clicks = []   # (wear time, dy, dp): what was still off at a click
+        self.wear = None   # the headset's current wear (SteamEyeLog.worn): clicks from others don't count
+
+    def ready(self):
+        return self.steam.samples >= self.MIN
+
+    def fit(self, points, info=None):
+        """points: (yaw, pitch, true yaw, true pitch), SteamVR's eyes' mean at a dot and the dot."""
+        self.points = [list(map(float, p)) for p in points]
+        self.steam.fit([(hy, hp, ty - hy, tp - hp) for hy, hp, ty, tp in self.points], self.MODE, ridge=self.RIDGE)
+        self.info = dict(info or {}, dots=len(self.points), fitted_at=time.time())
+        self.clicks = []
+
+    def misses(self):
+        """How far off the affine leaves each of the points it was fitted on (degrees)."""
+        out = []
+        for hy, hp, ty, tp in self.points:
+            dy, dp = self.steam.get(hy, hp, self.MODE)
+            out.append(math.hypot(ty - hy - dy, tp - hp - dp))
+        return out
+
+    @staticmethod
+    def mean(left, right):
+        if not left or not right:
+            return None
+        return (left[0] + right[0]) / 2, (left[1] + right[1]) / 2
+
+    def offset(self):
+        keep = [c for c in self.clicks if c[0] == self.wear][-self.KEEP:]
+        if not keep:
+            return 0.0, 0.0
+        return statistics.median(c[1] for c in keep), statistics.median(c[2] for c in keep)
+
+    def steam_gaze(self, left, right, clicks=True):
+        """SteamVR's corrected gaze from its two eyes' readings, or None without both."""
+        m = self.mean(left, right)
+        if m is None or not self.ready():
+            return None
+        dy, dp = self.steam.get(m[0], m[1], self.MODE)
+        oy, op = self.offset() if clicks else (0.0, 0.0)
+        return m[0] + dy + oy, m[1] + dp + op
+
+    def click(self, left, right, thy, thp):
+        """A look at (thy, thp) with SteamVR's eyes there: how far off the correction still was
+        joins the offset on top. Returns that miss (degrees), or None."""
+        g = self.steam_gaze(left, right, clicks=False)
+        if g is None:
+            return None
+        dy, dp = thy - g[0], thp - g[1]
+        miss = math.hypot(dy, dp)
+        if miss <= self.STALE:
+            self.clicks = (self.clicks + [(self.wear, dy, dp)])[-4 * self.KEEP:]
+        return miss
+
+    def combine(self, own, own_eyes, steam, share=None):
+        """own: ours, combined from own_eyes of its eyes (0-2); steam: steam_gaze(). Either may be None."""
+        share = self.SHARE if share is None else share
+        if steam is None:
+            return own
+        if own is None or own_eyes < 2:
+            return steam
+        return share * own[0] + (1 - share) * steam[0], share * own[1] + (1 - share) * steam[1]
+
+    def to_json(self):
+        return {"version": 1, "steam": self.steam.to_json(), "mode": self.MODE, "points": self.points, **self.info}
+
+    def from_json(self, d):
+        self.steam.from_json(d.get("steam") or {})
+        self.points = d.get("points") or []
+        self.info = {k: v for k, v in d.items() if k not in ("version", "steam", "mode", "points")}
+        return self
+
+    @classmethod
+    def load(cls, path=FUSION):
+        try:
+            return cls().from_json(json.loads(Path(path).read_text()))
+        except (OSError, ValueError):
+            return None
+
+    def save(self, path=FUSION):
+        path = Path(path)
+        tmp = path.with_suffix(".tmp")
+        tmp.write_text(json.dumps(self.to_json(), indent=1))
+        tmp.replace(path)
+
+
 class SteamEyeLog:
     """Follows SteamVR's eye tracking log (read only) for what moves the raw gaze under a
     calibration.

gaze/gazecheck.py

@@ -30,7 +30,7 @@ directions: look at each one.
   fit     the headset fit check (on "fitcheck", Check headset fit on the Gaze page): live, a
           card per eye (tracked or lost, the tracker's signal, how much of the last 10 s it was
           seen) and hints, from the gaze probe's Headset fit (gaze/fitcheck.py), while you
-          adjust the headset. A left click or Meta+J runs its guided check (dots, then looks
+          adjust the headset. A left click, Enter, a pinch or Meta+J runs its guided check (dots, then looks
           down, up, left and right); a right click or Meta+K closes it, as does FIT_TIMEOUT.
           With our tracker installed, its first hint line is the fit coach's (gaze/fitcoach.py):
           where the headset sits, from where ft-eyes sees each pupil and glint (ft-eyes "fit",
@@ -46,7 +46,7 @@ again when the file changes) adds two things, with our tracker installed, its re
           you look at its dot, against your saved fit, and holds the dot meanwhile (at most
           PUTON_READ). If the headset sits more than fitcoach.MOVED_DEG from it, or the IPD
           setting changed, the panel widens (five's size) and says which way to move it, live,
-          until it's back; then the dot is taken as usual. A left click or Meta+J takes it as
+          until it's back; then the dot is taken as usual. A left click, Enter, a pinch or Meta+J takes it as
           it sits, and after PUTON_TIMEOUT it's taken anyway. So the click teaches our tracker
           the headset where you meant it to be, not where it landed.
   slip    while gaze mode is on with our tracker, ft-eyes' glint jump (the headset slipped
@@ -61,15 +61,17 @@ waits until the tracker sends, at most ft-gazed's WAKE_SETTLE; then it opens, or
 The quick check's dot captures itself: from CHECK_SETTLE after it shows (the eyes getting
 there), once the gaze has held within CHECK_SPREAD for CHECK_WINDOW (the probe's max spread and
 capture time). It's the gaze holding still that counts, not where the tracker puts it, so it
-works however far off the tracker is; a left click or Meta+J (the pointer helper's
-"calaccept") takes it now. The full calibration's and five's dots wait for that click: you
+works however far off the tracker is; a left click, Enter, an index pinch with hand tracking,
+or Meta+J (the pointer helper's "calaccept") takes it now. The full calibration's and five's dots wait for that click: you
 click when you're looking at the dot (the user asked for that: a steady gaze isn't always on
 the dot), and the gaze held still up to then is taken (ACCEPT_SPREAD). They wait as long as
 it takes, up to CLICK_IDLE. A dot not taken says why in the panel's note line (reject_reason:
 gazecal.steady_samples' drop counts for SteamVR's tracker, ft-eyes' reply for ours), as does a
 click with nothing taken after ACCEPT_WAIT, and a failed calibration names its most common
 reason there and in the status. A right click or Meta+K ("calquit") closes the panel. A press
-mapped to gaze precision or gaze drag counts as the left click (pointer/helper/calpanel.h). The
+mapped to gaze precision or gaze drag counts as the left click (pointer/helper/calpanel.h);
+Enter is the input relay's ("calkey 1" to the helper while the panel is up), and a pinch the
+helper's own, read from ft-hands whether or not hand interaction is on. The
 pointer hides meanwhile ("calpanel 1", renewed every second; the helper shows it again by itself
 when that stops).
 
@@ -77,8 +79,21 @@ Our tracker's first calibration: before it has one, ft-eyes publishes no gaze (i
 to a gaze only with a calibration), so there's no gaze to hold still. Its calibration runs
 anyway ("blind"): someone in the headset (SteamVR's tracker sees an eye) and ft-eyes answering
 are enough to start it, each dot stands in for the gaze, and a click takes the CHECK_WINDOW up
-to it. ft-eyes then checks that each pupil was seen and held still in that window (calib-point)
-and says why not. Without this, a fresh install could never calibrate our tracker.
+to it. ft-eyes then checks that the pupils were seen and held still in that window (calib-point:
+one eye is enough, and the dot is kept for that eye) and says why not. Without this, a fresh
+install could never calibrate our tracker. A full calibration goes blind the same way whenever
+ours sends no gaze for OWN_GONE: its calibration maps one eye, and that's the one you can't
+show it now. Live 2026-10-18 a calibration with the left eye closed mapped only the right, and
+the next one, with the right eye closed, took no dot: every click waited for a gaze that
+couldn't come ("Waiting: the eye tracker isn't sending a gaze").
+
+Looked away (our tracker's full calibration): where SteamVR saw both eyes at a dot, their mean more
+than LOOKED_AWAY from it, past their usual offset in this calibration (the median over the dots
+it took, from AWAY_OFFSET_MIN of them), and the dot isn't taken: "you looked away from the dot"
+(looked_away; ft-eyes isn't asked). Live 2026-10-18 the palm menu opened the Steam menu
+mid-calibration, and three dots were taken 17-35 degrees from where the user looked. A third
+dot in a row like that is SteamVR being off, not you: it's taken, and the check stops for that
+calibration (AWAY_GIVE_UP).
 
 The service doesn't wait for ft-eyes' answers to calib-point and calib-fit (ask_eyes): the dot
 shows its ring full, and further clicks do nothing until the answer comes, or its deadline
@@ -88,7 +103,10 @@ behind the panel), and an answer over EYES_GONE closed the calibration as the he
 
 What a capture teaches:
   our tracker   quick and five: a click ("click T YAW PITCH", like a pointer lesson); full:
-                calib-start, a calib-point for each dot, calib-fit (its calibration)
+                calib-start, a calib-point for each dot, calib-fit (its calibration); and
+                SteamVR's two eyes at each dot ft-eyes took, where SteamVR saw both: after the
+                fit, SteamVR's affine for ft-gazed's fusion (gazecal.Fusion) is fitted on
+                them and saved to fusion.json (kept as it was with under Fusion.MIN of them)
   SteamVR's     quick and five: a lesson for each eye, like the pointer's; full: each source's
                 calibration fitted from the dots (gazecal.Correction, the probe's way), saved
                 to calibration.json, and the lessons start over on top of it
@@ -110,7 +128,7 @@ from pathlib import Path
 
 import fitcoach
 from fitcheck import MIN_REGION, FitCheck, wrap
-from gazecal import DEFAULT_MODEL, EYE_LOST, STATE, steady_samples
+from gazecal import DEFAULT_MODEL, EYE_LOST, FUSION, STATE, Fusion, steady_samples
 
 REPO = Path(__file__).resolve().parents[1]
 PANEL_PROG = REPO / "gaze" / "build" / "ft-gazepanel"
@@ -128,6 +146,11 @@ CHECK_WINDOW = 0.6
 CHECK_SPREAD = 1.0     # degrees
 ACCEPT_SPREAD = 2.5    # degrees: a capture asked for (calaccept) takes this much
 CLICK_IDLE = 120.0     # seconds a dot of five or full waits for its click; then the check closes
+LOOKED_AWAY = 15.0     # degrees: SteamVR's eyes this far from a full calibration's dot (our tracker's), past
+                       # their usual offset in it, and the dot isn't taken: you looked away
+AWAY_OFFSET_MIN = 3    # dots taken with SteamVR's eyes before their median offset counts
+AWAY_GIVE_UP = 2       # dots in a row turned down as looked away: at a third, SteamVR is off, not you;
+                       # that one's taken and the check stops
 QUICK_TIMEOUT = 6.0
 QUICK_COOLDOWN = 120.0
 DON_DELAY = 3.0        # seconds of eyes after AWAY_MIN without: the headset went on
@@ -188,6 +211,7 @@ STEAM_REASONS = {"lost_left": ("left eye lost", "SteamVR lost your left eye"),
                  "vergence": ("eyes disagreed", "SteamVR's two eyes disagreed")}
 FIT_HINT = "check the headset fit"
 ACCEPT_WAIT = 1.5      # seconds after a click with no capture before the panel says what it waits for
+OWN_GONE = 0.5         # seconds without our tracker's gaze before a full calibration goes blind (see the top)
 
 
 def reject_reason(reply, why):
@@ -199,13 +223,18 @@ def reject_reason(reply, why):
         key = max(why, key=why.get)
         return *STEAM_REASONS[key], key.startswith("lost")
     words = reply.removeprefix("fail ").split()
-    # ft-eyes: "fail the left eye was seen in only 3 frames", "fail the left eye moved (4.2 px)"
+    # ft-eyes: "fail the left eye moved (4.2 px)", "fail neither eye was seen enough (right 3
+    # frames, left 0 frames)"; one eye seen is enough for a dot (and its answer is "ok ...")
+    if reply.startswith("fail neither eye was seen"):
+        return "eyes not seen", "our tracker saw neither eye well enough", True
     if len(words) >= 3 and words[0] == "the" and words[2] == "eye":
         eye = words[1]
         if "seen" in words:
             return f"{eye} eye not seen", f"our tracker saw your {eye} eye in only {words[-2]} frames", True
         if "moved" in words:
             return f"{eye} eye moved", f"your {eye} eye moved while you looked", False
+    if reply.startswith("looked away"):  # gazecheck's own (looked_away): SteamVR's eyes were elsewhere
+        return "looked away", "you looked away from the dot (by SteamVR's eyes)", False
     if not reply:
         return "no answer", "our tracker didn't answer", False
     if reply.startswith("our tracker isn't running"):
@@ -581,7 +610,8 @@ class Checks:
         self.check = {"kind": kind, "reason": reason, "own": own, "dots": check_dots(kind, own), "i": 0,
                       "started": now, "shown": now, "run": [], "accept": False, "done_at": None, "tries": 0,
                       "skipped": 0, "captured": 0, "points": {}, "reasons": {}, "fit_reasons": set(), "note": "",
-                      "blind": blind}
+                      "blind": blind, "fusion": [], "away_dots": [], "away_off": False, "steam_offsets": [],
+                      "own_at": now, "run_blind": blind, "went_blind": False}
         log(f"{kind} check: {reason}" + (" (our tracker's first: no gaze yet, so each click takes the look "
                                          "up to it)" if blind else ""))
         self.note_until = None   # the check's panel replaces the slip note
@@ -619,7 +649,7 @@ class Checks:
         self.to_helper("calpanel 1")
         self.to_panel("show fit")
         self.to_panel("title Headset fit: adjust the headset while you watch")
-        self.to_panel("text Left click or Meta+J: guided check · Right click or Meta+K: done")
+        self.to_panel("text Left click, Enter or pinch: guided check · Right click or Meta+K: done")
         return "ok"
 
     def new_coach(self):
@@ -849,7 +879,7 @@ class Checks:
                 drawn[k] = cmd
                 self.to_panel(cmd)
         if fit.have_eye_data and fit.samples < 3 * MIN_REGION:
-            hints = ["Look around slowly: up, down, left and right. Or left click (Meta+J) for a guided check."]
+            hints = ["Look around slowly: up, down, left and right. Or left click, press Enter or pinch for a guided check."]
         else:
             hints = fit.hints()
         hints = self.coach_lines() + hints
@@ -865,9 +895,9 @@ class Checks:
         if c["kind"] == "full":
             self.to_panel(f"bg {ROUND_BG[rnd]}")
             self.to_panel(f"title Gaze calibration: {ROUND_NAMES[rnd]} round, {rnd + 1} of 3")
-            self.to_panel("text Look at the dot and click (left click or Meta+J). Right click or Meta+K: stop")
+            self.to_panel("text Look at the dot, then left click, press Enter or pinch. Right click or Meta+K: stop")
         elif c["kind"] == "five":
-            self.to_panel(f"text Look at the dot and click ({c['i'] + 1} of {len(c['dots'])})")
+            self.to_panel(f"text Look at the dot, then click, Enter or pinch ({c['i'] + 1} of {len(c['dots'])})")
         else:
             self.to_panel("text Look at the dot")
         self.to_panel(f"dot {yaw:.3f} {pitch:.3f} look")
@@ -896,10 +926,21 @@ class Checks:
             return  # (asking: ft-eyes has this dot's look, or the fit)
         if c["own"]:
             src = s["src"].get("own") or {}
-            if c["blind"]:
-                # Our tracker's first calibration: no gaze yet, so the dot stands in for it (the
-                # gaze can't seem to move) and a click takes the look up to it. ft-eyes checks
-                # the pupils held still (see the top).
+            if "hy" in src:
+                c["own_at"] = self.sample_at
+            # Our tracker's first calibration: no gaze yet, so the dot stands in for it (the
+            # gaze can't seem to move) and a click takes the look up to it. ft-eyes checks
+            # the pupils held still (see the top). A full calibration does the same while ours
+            # sends no gaze: its calibration maps only the eye you've closed.
+            blind = c["blind"] or (c["kind"] == "full" and self.sample_at - c["own_at"] > OWN_GONE)
+            if blind != c["run_blind"]:
+                c["run"].clear()  # the dot standing in and our gaze aren't one look
+                c["run_blind"] = blind
+                if blind and not c["went_blind"]:
+                    c["went_blind"] = True
+                    log(f"{c['kind']} check: our tracker sends no gaze (its calibration's eyes aren't seen): "
+                        "each click takes the look up to it")
+            if blind:
                 yaw, pitch, _ = c["dots"][c["i"]]
                 src = {"hy": yaw, "hp": pitch}
         else:
@@ -960,6 +1001,21 @@ class Checks:
             miss = [math.hypot(yaw - e[0], pitch - e[1]) if e else None for e in eyes]
             rec.update(eyes=eyes, miss=miss)
             t0, t1 = samples[0]["t"], samples[-1]["t"]
+            if c["kind"] == "full" and svc.one_eye is None:
+                # SteamVR's eyes at the dot too, for the fusion (finish): each one's median over
+                # the steady samples, as its own calibration takes them.
+                steady = steady_samples(samples)
+                lr = []
+                for name in ("left", "right"):
+                    pts = [(smp["src"][name]["hy"], smp["src"][name]["hp"]) for smp in steady
+                           if "hy" in (smp["src"].get(name) or {})]
+                    lr.append((statistics.median(p[0] for p in pts), statistics.median(p[1] for p in pts))
+                              if len(pts) >= 15 else None)
+                if all(lr):
+                    rec["steam"] = lr
+                    if self.looked_away(rec):
+                        self.captured(rec, False)
+                        return
             if c["kind"] == "full":
                 # ft-eyes checks the pupils held still: its answer takes the dot or not (point_done).
                 # The ring shows it full meanwhile, so the click is seen to have landed.
@@ -1023,12 +1079,41 @@ class Checks:
                 svc.dirty = True
         self.captured(rec, ok)
 
+    def looked_away(self, rec):
+        """A full calibration's dot for our tracker, where SteamVR saw both eyes: whether they say
+        you looked away (LOOKED_AWAY past their usual offset in this calibration). The dot isn't
+        taken then (ft-eyes isn't asked): one taken there teaches our tracker the wrong place.
+        Past AWAY_GIVE_UP dots in a row turned down, it's SteamVR that's off: the check stops."""
+        c = self.check
+        (my, mp), (ty, tp) = Fusion.mean(*rec["steam"]), rec["true"]
+        dy, dp = ty - my, tp - mp
+        offs = c["steam_offsets"]
+        if len(offs) >= AWAY_OFFSET_MIN:
+            dy -= statistics.median(o[0] for o in offs)
+            dp -= statistics.median(o[1] for o in offs)
+        rec["away"] = round(math.hypot(dy, dp), 2)
+        away = rec["away"] > LOOKED_AWAY and not c["away_off"]
+        if away and c["i"] not in c["away_dots"] and len(c["away_dots"]) >= AWAY_GIVE_UP:
+            c["away_off"], away = True, False
+            log(f"full check: SteamVR's eyes {rec['away']:.0f} deg off at {AWAY_GIVE_UP + 1} dots in a row: "
+                "taken as SteamVR being off, the looked-away check is off for this calibration")
+        if not away:
+            c["away_dots"] = []
+            offs.append((ty - my, tp - mp))
+            return False
+        if c["i"] not in c["away_dots"]:
+            c["away_dots"].append(c["i"])
+        rec["reply"] = f"looked away ({rec['away']:.0f} deg by SteamVR)"
+        return True
+
     def point_done(self, rec, miss, reply):
         """ft-eyes' answer to a full calibration dot's calib-point ("" without one)."""
         rec["reply"] = reply
         ok = reply.startswith("ok")
         if ok:
             self.svc.weights["own"].add(miss)
+            if rec.get("steam"):
+                self.check["fusion"].append((*Fusion.mean(*rec["steam"]), *rec["true"]))
         self.captured(rec, ok)
 
     def captured(self, rec, ok):
@@ -1115,6 +1200,8 @@ class Checks:
         if c["own"]:
             def fitted(reply):
                 log(f"calibration ({c['captured']} of {n} dots): our tracker says {reply or 'nothing'}")
+                if reply.startswith("ok"):
+                    self.fit_fusion(c["fusion"])
                 self.close()
 
             self.to_panel("text Saving the calibration")
@@ -1139,6 +1226,22 @@ class Checks:
             log(f"calibration ({c['captured']} of {n} dots): {mode}, saved")
         self.close()
 
+    def fit_fusion(self, pts):
+        """SteamVR's affine for ft-gazed's fusion, from its eyes at the dots ours took."""
+        if len(pts) < Fusion.MIN:
+            log(f"fusion: SteamVR saw both eyes at only {len(pts)} of the dots; {FUSION.name} stays as it was")
+            return
+        f = Fusion()
+        f.fit(pts, {"how": "calibration", "made": time.strftime("%Y-%m-%d %H:%M:%S"), "wear": self.svc.steam.worn()})
+        try:
+            f.save(FUSION)
+        except OSError as e:
+            log(f"fusion: couldn't save {FUSION}: {e}")
+            return
+        self.svc.load_fusion()
+        log(f"fusion: SteamVR's affine from {len(pts)} dots, {statistics.median(f.misses()):.2f} deg left at the "
+            "median, saved")
+
     def close(self, why=None):
         if not self.check:
             return

gaze/panel/ft-gazepanel.cpp

@@ -644,8 +644,12 @@ struct PoseLog {
             next += kPosePeriodNs;
             const int64_t now = RawNs();
             if (next < now - kPosePeriodNs) next = now;
-            const timespec ts{time_t(next / 1'000'000'000), long(next % 1'000'000'000)};
-            clock_nanosleep(CLOCK_MONOTONIC_RAW, TIMER_ABSTIME, &ts, nullptr);
+            // Linux has no absolute sleeps on CLOCK_MONOTONIC_RAW (EINVAL at once: the loop spun at
+            // ~160k poses/s), so sleep the gap relative on CLOCK_MONOTONIC.
+            if (const int64_t wait = next - now; wait > 0) {
+                const timespec ts{time_t(wait / 1'000'000'000), long(wait % 1'000'000'000)};
+                clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, nullptr);
+            }
         }
     }
 };

gaze/rec/CONSENT.md

@@ -50,7 +50,7 @@ The recorder doesn't record sound, the headset's outward cameras, your name, you
 
 - Recordings stay on your headset, in `~/.local/share/frametop/eyes/contrib`. Nothing is uploaded automatically.
 - Before you share anything, you can look through every recording in the Review page. You can delete any stretch of a take, a whole take or a whole round. Export leaves out what you deleted.
-- You upload the export yourself, from the Upload page. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
+- You upload the export yourself: from the Upload page, or with Export and upload, which sends it once it's exported. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
 
 ## Where it goes, and who can read it
 

gaze/rec/DESIGN.md

@@ -93,7 +93,7 @@ The launch bounties:
 |---|---|---|
 | View edges | Dots at the panel's edges and corners: 30 degrees to the sides, 23 up and down, about 35 in the corners (the rec panel is 64 degrees across; every round's dots are checked against it) | Eyes turned far: the lid, the glints lost, the pupil foreshortened |
 | Blinks and lids | Blink on a beep, close both eyes on a beep and open them on the next (sounds.py: the eyes are shut, so words can't cue them), wink each eye, squint, eyes wide, half-closed, looking down with lids lowered | Closed and partly closed eyes; one eye missing |
-| Headset slips | Nudge the headset up, down, left, right, tilt it; then take it off and put it back on; dots after each | Headset shifting on the face: the put-on check's data |
+| Headset slips | Nudge the headset up, down, left, right, tilt it, tip its front down, pull it off the face, on a beep, holding it through dots, letting go on a low beep; then take it off and put it back on; dots after each | Headset shifting on the face: the put-on check's data |
 | Glasses | A 5x5 grid and the edges with glasses or inserts on (offered when the checklist says so) | Lens reflections, the glasses' frame in view |
 | Jumps and pursuit | A dot jumping between random spots (0.5 to 1.5 s each), then moving on circles, lines and a figure eight at two speeds | Saccades, overshoot, smooth pursuit, the tracker's lag |
 | Depth | Dots at 0.5 m, 1.5 m and 5 m in turn (the panel's distance changes) | Vergence; the eyes' angle with distance |
@@ -157,7 +157,7 @@ The export recompresses nothing (the chunks are zstd already), drops deleted ran
 
 ## Shared with the hand recorder
 
-Much of `hands/rec/` isn't about hands: the headset button reader, profile and consent, take bookkeeping (deleted ranges, redo, parts, clock samples), the Hugging Face login and upload, validate's manifest and checksum checks, and most of the window. These live in `recorder/` at the top of the tree, with the bounty board, and both recorders use them: `common.py`, `units.py`, `button.py`, `bounties.py`, `files.py` (ranges, checksums, sizes), `checks.py` (validate's report, SHA256SUMS, typed manifest lookups), `hub.py` (login, errors, upload records, the pull request flow), `age.py` (the export's encryption) and `window.py` (the window backend's shared half: consent, disk, session controls, export, login and upload; each recorder's `Backend` subclasses it). The hand recorder's modules keep their names on top of them, and its tests pass.
+Much of `hands/rec/` isn't about hands: the headset button reader, profile and consent, take bookkeeping (deleted ranges, redo, parts, clock samples), the Hugging Face login and upload, validate's manifest and checksum checks, and most of the window. These live in `recorder/` at the top of the tree, with the bounty board, and both recorders use them: `common.py`, `units.py`, `button.py`, `bounties.py`, `files.py` (ranges, checksums, sizes), `checks.py` (validate's report, SHA256SUMS, typed manifest lookups), `hub.py` (login, errors, upload records, the pull request flow), `age.py` (the export's encryption) and `window.py` (the window backend's shared half: consent, disk, session controls, export, login and upload; each recorder's `Backend` subclasses it). The hand recorder's modules keep their names on top of them, and its tests pass. The export queue, with Export and upload sending a round by itself once it's exported (2026-10-17), is `window.py`'s too, so both windows have it: `hands/rec/DESIGN.md` (Upload) has how it works.
 
 Uploads: `gaze/rec/hub.py` sends nothing to the real dataset until `CONSENT.md` and `UPLOAD.md` exist and aren't drafts. `FT_EYEREC_ALLOW_UPLOAD=1` opens rehearsals, and only to another dataset (`FT_EYEREC_DATASET`). Rehearsed 2026-10-14 against the private `DeeJanuz/frametop-eyes-test` with a round of random bytes: uploaded, downloaded, checked, and decrypted with both keys by age 1.1.1 on the 7i.
 

gaze/rec/ft_eyerec.py

@@ -337,6 +337,9 @@ class Backend(RecorderBackend):
     def _upload_values(self, session, path):
         return {"ROUND": session}
 
+    def _session_label(self, session):
+        return round_label(session)
+
     @Property(str, constant=True)
     def toolVersion(self):
         return takes.tool_version()
@@ -739,7 +742,7 @@ class Backend(RecorderBackend):
 
     @Slot(str)
     def deleteSession(self, rnd):
-        if self._active_guard(rnd):
+        if self._active_guard(rnd) or self._busy_guard(rnd):
             return
         try:
             self.store.delete_round(rnd)

gaze/rec/main.qml

@@ -96,6 +96,24 @@ Kirigami.ApplicationWindow {
         return n + " " + word + (n === 1 ? "" : "s")
     }
 
+    // A round's label, from its id (the Export page's queue).
+    function sessionLabel(id) {
+        for (const s of backend.sessions)
+            if (s.id === id)
+                return s.label
+        return id
+    }
+    // The round's export can upload by itself once done: logged in, and its dataset takes
+    // contributions (or a dry run).
+    function canUploadAfter(id) {
+        return backend.canUploadAfter && (backend.hubDryRun || backend.uploadAllowedFor(id))
+    }
+    // Waiting to export, or exporting now.
+    function exportPending(id) {
+        return backend.exportSessionId === id && backend.exporting
+               || backend.exportQueue.some(q => q.session === id)
+    }
+
     function show(page) {
         pageStack.clear()
         pageStack.push(page)
@@ -171,9 +189,11 @@ Kirigami.ApplicationWindow {
             close.accepted = false
             confirm.ask("Stop the round?", "A round is recording. Closing stops it; what's recorded so far is kept.",
                         () => { root.quitting = true; Qt.quit() })
-        } else if (backend.uploading && !quitting) {
+        } else if ((backend.uploading || backend.exporting || backend.uploadQueue.length > 0) && !quitting) {
             close.accepted = false
-            confirm.ask("Cancel the upload?", "An upload is running. Closing cancels it; you can start it again later.",
+            confirm.ask("Cancel the exports and uploads?",
+                        "An export or upload is running or waiting (the Export page lists them). Closing cancels "
+                        + "them all; an export cancelled keeps nothing. You can start them again later.",
                         () => { root.quitting = true; Qt.quit() })
         }
     }
@@ -887,9 +907,10 @@ Kirigami.ApplicationWindow {
                     opacity: 0.7
                     text: "The dots and instructions appear in the headset. "
                           + (sessionView.st.button ? "The button on the right side of the headset: next, pause or "
-                             + "resume; twice: record the last step again; hold: stop. " : "")
-                          + "While this window has focus: Space: next · P: pause or resume · R: record the last step "
-                          + "again · S: skip the rest of the take · Esc: stop. Stopping keeps what's recorded so far."
+                             + "resume; twice: redo; hold: stop. " : "")
+                          + "While this window has focus: Space: next · P: pause or resume · R: redo · S: skip the "
+                          + "rest of the take · Esc: stop. Redo during a step starts it again; before a step "
+                          + "starts, it goes back to the one before. Stopping keeps what's recorded so far."
                 }
                 RowLayout {
                     // No keyboard focus on the buttons, so Space always reaches the shortcut.
@@ -911,7 +932,8 @@ Kirigami.ApplicationWindow {
                         visible: backend.sessionActive
                         enabled: !!sessionView.st.can_redo
                         focusPolicy: Qt.NoFocus
-                        text: "Redo step"
+                        // At a ready screen it goes back to the step before; else this one again.
+                        text: sessionView.state === "ready" ? "Back to the last step" : "Redo this step"
                         icon.name: "edit-undo"
                         onClicked: backend.redo()
                     }
@@ -1310,6 +1332,12 @@ Kirigami.ApplicationWindow {
                         return s
                 return null
             }
+            // The chosen round is exporting, or waiting to.
+            readonly property bool pending: chosen !== null && root.exportPending(chosen.id)
+            readonly property bool canExport: chosen !== null && !pending && backend.exportProblem === "" && !chosen.active
+                                              && !(backend.uploading && backend.upload.session === chosen.id)
+                                              && backend.uploadQueue.indexOf(chosen.id) < 0
+            Component.onCompleted: if (backend.login.state === "unknown") backend.checkLogin()
 
             Timer {
                 interval: 5000
@@ -1329,6 +1357,15 @@ Kirigami.ApplicationWindow {
                     text: "Exporting keeps the processor busy for a minute or so, so VR may stutter a little until "
                           + "it's done. You can keep using the headset meanwhile."
                 }
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    visible: backend.uploadAfter.length + backend.uploadQueue.length > 0
+                    position: Kirigami.InlineMessage.Position.Header
+                    type: Kirigami.MessageType.Positive
+                    text: "You can take the headset off and leave it plugged in: it stays awake until the "
+                          + "exports and uploads below are done, and they show here how they went. Keep this "
+                          + "window open: closing it cancels them."
+                }
                 Kirigami.InlineMessage {
                     Layout.fillWidth: true
                     visible: backend.exportProblem !== ""
@@ -1345,7 +1382,6 @@ Kirigami.ApplicationWindow {
                     model: backend.sessions
                     textRole: "label"
                     valueRole: "id"
-                    enabled: !backend.exporting
                     Component.onCompleted: {
                         const i = indexOfValue(root.chosenSession)
                         currentIndex = i >= 0 ? i : (count > 0 ? 0 : -1)
@@ -1361,32 +1397,73 @@ Kirigami.ApplicationWindow {
                     Kirigami.FormData.label: "Export:"
                     visible: exportView.chosen !== null
                     text: !exportView.chosen ? ""
+                          : exportView.pending ? (backend.exportSessionId === exportView.chosen.id && backend.exporting
+                                                  ? "exporting now" : "waiting its turn, below")
                           : exportView.chosen.export_stale ? "changed since it was exported: export it again"
                           : exportView.chosen.exported ? "ready, " + exportView.chosen.exportText : "not exported yet"
                 }
+                // Exports run one at a time; one asked for meanwhile waits its turn. "Export and
+                // upload" sends it by itself once it's exported.
                 RowLayout {
                     Controls.Button {
-                        text: backend.exporting ? "Exporting…"
-                              : exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
+                        text: exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
                         icon.name: "document-export"
-                        enabled: exportView.chosen !== null && !backend.exporting && backend.exportProblem === ""
-                                 && !exportView.chosen.active
-                        onClicked: backend.exportSession(exportView.chosen.id, false)
+                        enabled: exportView.canExport
+                        onClicked: backend.queueExport(exportView.chosen.id, false, false)
                     }
+                    Controls.Button {
+                        text: "Export and upload"
+                        icon.name: "cloud-upload"
+                        enabled: exportView.canExport && root.canUploadAfter(exportView.chosen.id)
+                        onClicked: backend.queueExport(exportView.chosen.id, false, true)
+                    }
+                }
+                Controls.Label {
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    visible: exportView.chosen !== null && !backend.hubDryRun
+                             && (!backend.canUploadAfter || !backend.uploadAllowedFor(exportView.chosen.id))
+                    wrapMode: Text.Wrap
+                    opacity: 0.7
+                    text: !backend.canUploadAfter ? "To upload by itself once exported, log in on the Upload page first."
+                          : "Contributions aren't open yet, so it can't upload by itself."
+                }
+                Controls.Label {
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    visible: backend.exporting && exportView.chosen !== null && !exportView.pending
+                    wrapMode: Text.Wrap
+                    opacity: 0.7
+                    text: "Another export is running: this one waits its turn."
+                }
+
+                // The running export
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Exporting now"
+                    visible: backend.exporting
+                }
+                RowLayout {
+                    Kirigami.FormData.label: "Round:"
+                    visible: backend.exporting
+                    Controls.Label { text: root.sessionLabel(backend.exportSessionId) }
                     // Shows the moment Export is pressed: the first progress can take a few seconds.
                     Controls.BusyIndicator {
-                        visible: backend.exporting
-                        running: visible
+                        running: backend.exporting
                         implicitWidth: Kirigami.Units.gridUnit * 1.5
                         implicitHeight: implicitWidth
                     }
                     Controls.Button {
-                        visible: backend.exporting
                         text: "Cancel"
                         icon.name: "dialog-cancel"
                         onClicked: backend.cancelExport()
                     }
                 }
+                Controls.CheckBox {
+                    visible: backend.exporting
+                    text: "Upload it once exported"
+                    checked: backend.uploadAfter.indexOf(backend.exportSessionId) >= 0
+                    enabled: checked || root.canUploadAfter(backend.exportSessionId)
+                    onToggled: backend.setUploadAfter(backend.exportSessionId, checked)
+                }
                 // Drawn here: the desktop style's ProgressBar paints nothing without a QApplication.
                 Rectangle {
                     Kirigami.FormData.label: "Progress:"
@@ -1409,7 +1486,7 @@ Kirigami.ApplicationWindow {
                     text: backend.exportText
                 }
                 RowLayout {
-                    visible: exportView.chosen !== null && exportView.chosen.exported && !backend.exporting
+                    visible: exportView.chosen !== null && exportView.chosen.exported && !exportView.pending
                     Controls.Button {
                         text: "Upload"
                         icon.name: "go-next"
@@ -1418,6 +1495,8 @@ Kirigami.ApplicationWindow {
                     Controls.Button {
                         text: "Delete export"
                         icon.name: "edit-delete"
+                        enabled: !(backend.uploading && backend.upload.session === exportView.chosen.id)
+                                 && backend.uploadQueue.indexOf(exportView.chosen.id) < 0
                         onClicked: {
                             const rid = exportView.chosen.id
                             confirm.ask("Delete this export?", "The export is deleted; the round's recordings stay.",
@@ -1425,6 +1504,107 @@ Kirigami.ApplicationWindow {
                         }
                     }
                 }
+
+                // Waiting: exports, then uploads
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Waiting to export"
+                    visible: backend.exportQueue.length > 0
+                }
+                Repeater {
+                    model: backend.exportQueue
+                    delegate: RowLayout {
+                        required property var modelData
+                        required property int index
+                        Kirigami.FormData.label: (index + 1) + "."
+                        Controls.Label { text: root.sessionLabel(modelData.session) }
+                        Controls.CheckBox {
+                            text: "then upload"
+                            checked: modelData.upload
+                            enabled: checked || root.canUploadAfter(modelData.session)
+                            onToggled: backend.setUploadAfter(modelData.session, checked)
+                        }
+                        Controls.ToolButton {
+                            icon.name: "list-remove"
+                            text: "Remove"
+                            display: Controls.AbstractButton.IconOnly
+                            Controls.ToolTip.text: text
+                            Controls.ToolTip.visible: hovered
+                            onClicked: backend.unqueueExport(modelData.session)
+                        }
+                    }
+                }
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Uploading"
+                    visible: backend.uploading || backend.uploadQueue.length > 0
+                }
+                RowLayout {
+                    Kirigami.FormData.label: "Now:"
+                    visible: backend.uploading
+                    Controls.Label {
+                        text: root.sessionLabel(backend.upload.session || "") + (backend.upload.text ? ": " + backend.upload.text : "")
+                    }
+                    Controls.Button {
+                        text: "Show"
+                        icon.name: "go-next"
+                        flat: true
+                        onClicked: { root.chosenSession = backend.upload.session; root.show(uploadPage) }
+                    }
+                }
+                Repeater {
+                    model: backend.uploadQueue
+                    delegate: RowLayout {
+                        required property var modelData
+                        required property int index
+                        Kirigami.FormData.label: "Then:"
+                        Controls.Label { text: root.sessionLabel(modelData) }
+                        Controls.ToolButton {
+                            icon.name: "list-remove"
+                            text: "Don't upload it"
+                            display: Controls.AbstractButton.IconOnly
+                            Controls.ToolTip.text: text
+                            Controls.ToolTip.visible: hovered
+                            onClicked: backend.unqueueUpload(modelData)
+                        }
+                    }
+                }
+
+                // How they went, for whoever comes back to the headset
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Done"
+                    visible: backend.doneLog.length > 0
+                }
+                Repeater {
+                    model: backend.doneLog
+                    delegate: RowLayout {
+                        required property var modelData
+                        Kirigami.Icon {
+                            source: modelData.error ? "dialog-error" : "checkmark"
+                            implicitWidth: Kirigami.Units.iconSizes.small
+                            implicitHeight: implicitWidth
+                        }
+                        Controls.Label {
+                            Layout.maximumWidth: Kirigami.Units.gridUnit * 24
+                            wrapMode: Text.Wrap
+                            text: modelData.text
+                        }
+                        Controls.ToolButton {
+                            visible: modelData.link !== ""
+                            icon.name: "internet-services"
+                            text: "Open"
+                            onClicked: Qt.openUrlExternally(modelData.link)
+                        }
+                    }
+                }
+                Controls.Button {
+                    visible: backend.doneLog.length > 0
+                    text: "Clear"
+                    icon.name: "edit-clear-history"
+                    flat: true
+                    onClicked: backend.clearDoneLog()
+                }
             }
 
             footer: Controls.Label {
@@ -1433,8 +1613,9 @@ Kirigami.ApplicationWindow {
                 opacity: 0.7
                 text: "Export leaves out the ranges you deleted, encrypts the eye images and any health answers to "
                       + "the dataset's keys (only its maintainers can read them), and adds a manifest and checksums, "
-                      + "in " + backend.exportsDir + ". It runs at the lowest priority; the headset stays awake "
-                      + "until it's done. Nothing is uploaded until you press Upload on the Upload page."
+                      + "in " + backend.exportsDir + ". It runs at the lowest priority, one at a time; the headset "
+                      + "stays awake until they're done. Nothing is uploaded unless you press Export and upload here, "
+                      + "or Upload on the Upload page."
             }
         }
     }
@@ -1460,7 +1641,7 @@ Kirigami.ApplicationWindow {
             readonly property string blocked: {
                 if (session === "") return "Choose an export"
                 if (chosen && chosen.export_stale) return "Export the round again first"
-                if (backend.exporting) return "Wait for the export to finish"
+                if (root.exportPending(session)) return "Wait for its export to finish"
                 if (backend.uploading) return ""
                 if (backend.hubDryRun) return ""
                 if (!backend.uploadAllowed) return "Contributions aren't open yet"
@@ -1473,6 +1654,14 @@ Kirigami.ApplicationWindow {
             Connections {
                 target: backend
                 function onSessionsChanged() { uploadView.refreshInfo() }
+                // An upload started by itself (queued on the Export page): show it.
+                function onUploadChanged() {
+                    const i = backend.uploading ? exportBox.indexOfValue(backend.upload.session) : -1
+                    if (i >= 0 && i !== exportBox.currentIndex) {
+                        exportBox.currentIndex = i
+                        exportBox.keep = backend.upload.session
+                    }
+                }
             }
             Component.onCompleted: {
                 refreshInfo()
@@ -1553,8 +1742,19 @@ Kirigami.ApplicationWindow {
                             textRole: "label"
                             valueRole: "id"
                             enabled: !backend.uploading
-                            Component.onCompleted: currentIndex = Math.max(0, indexOfValue(root.chosenSession))
-                            onActivated: root.chosenSession = currentValue
+                            // An export that finishes meanwhile adds a row (newest first): stay on the
+                            // same export, the uploading one while an upload runs.
+                            property string keep: ""
+                            Component.onCompleted: {
+                                currentIndex = Math.max(0, indexOfValue(root.chosenSession))
+                                keep = currentValue || ""
+                            }
+                            onActivated: root.chosenSession = keep = currentValue
+                            onModelChanged: Qt.callLater(() => {
+                                const i = indexOfValue(backend.uploading ? backend.upload.session : keep)
+                                currentIndex = i >= 0 ? i : (count > 0 ? Math.max(0, currentIndex) : -1)
+                                keep = currentValue || ""
+                            })
                         }
                         Controls.Label {
                             opacity: 0.7
@@ -1667,6 +1867,14 @@ Kirigami.ApplicationWindow {
                               + "\">its page</a>. Upload checks the export, opens your pull request, then sends the files."
                         onLinkActivated: link => Qt.openUrlExternally(link)
                     }
+                    Controls.Label {
+                        Layout.fillWidth: true
+                        visible: backend.uploadQueue.length + backend.uploadAfter.length > 0
+                        wrapMode: Text.Wrap
+                        opacity: 0.7
+                        text: "Queued on the Export page to upload after this one, by themselves: "
+                              + root.plural(backend.uploadQueue.length + backend.uploadAfter.length, "export") + "."
+                    }
                     RowLayout {
                         Controls.Button {
                             text: backend.uploading ? "Uploading…" : backend.hubDryRun ? "Upload (dry run)" : "Upload"

gaze/rec/rounds/README.md

@@ -26,6 +26,32 @@ Every recorded second becomes training data, labelled by where the dot was. So:
 - **Rounds open the same way.** The recorder adds putting the headset on and the nine dots before every
   round. Don't repeat them. The group `dots` is theirs **(checked)**.
 
+## What the person sees and hears, step by step
+
+A round moves through the same states every step. Each one has its own way of telling the person what
+to do and when, and the step's text and cues have to fit it.
+
+| State | Recorded | What they do | How they know when |
+|---|---|---|---|
+| Ready screen | no | read the text, press Next | the title, the text, the keys line |
+| Countdown 3-2-1 | yes | look at the dot it shows | the numbers |
+| Eyes on dots | yes | look at each dot | its ring fills, then the dot moves; the bar along the bottom |
+| Eyes shut | yes | close, then open | beeps (below) |
+| Moving the headset | yes | push, slide or tilt it by hand, hold, let go | a cue word and a beep each way (below) |
+| Head or reading | yes | the one action the text says, until the bar runs out | the bar along the bottom |
+
+- **The ready screen says it all, the action first.** It's the only time they read: what to do, what
+  starts it ("when it says Push up and beeps"), and what ends it ("until its ring fills", "until the bar
+  runs out", "until it says Let go"). A step with no ready screen (`"ready": false`) can only add to that
+  in a few words, as they're doing something already.
+- **Nothing starts in the countdown.** It points the eyes at the first dot. A one-off action (a blink, a
+  push) starts on a cue word after it, never on "go" from the numbers.
+- **One-off actions get a cue word; ongoing ones get the bar.** Turning the head, reading or looking
+  around run until the bar runs out. A push, a blink or closing the eyes happens on its cue.
+- **Redo does a chain again.** R or two presses during a step start it again, from the first step of its
+  chain; at a ready screen they go back to the step before. Make a chain one thing to do over: a nudge,
+  the dots done holding it, and letting go.
+
 ## Cues the person can't see get a beep
 
 The panel can't cue someone whose eyes are shut. A cue that asks for that plays a beep
@@ -42,6 +68,26 @@ The panel can't cue someone whose eyes are shut. A cue that asks for that plays
 - **No voice.** Beeps only; generated speech sounds odd in the headset. A new kind of cue sound goes
   into `sounds.SOUNDS` and the tests.
 
+## Moving the headset by hand
+
+A hand on the headset can't press Next, and the moment it moves is what the data is about. So:
+
+- **A cue word and a beep start it:** a cue word starting with push, pull, slide, tilt, tip or nudge plays
+  `go` **(checked)**: `["Push up", 2.5, "go"]`. Say the direction in the word, the same way each time
+  ("to your left", "left side up").
+- **Eyes first:** a cue on the dot comes before it, so there's a still moment to compare with
+  **(checked)**: `["Look at the dot", 1.5]`.
+- **Letting go has its own cue and beep:** "Let go" plays `done` **(checked)** and shows 2 s or more
+  **(checked)**, long enough to put the headset back and look at the dot again.
+- **In the same chain:** the `done` cue comes before the next ready screen, a new group or a reseat
+  **(checked)**, so a redo does the push, the dots and the letting go again.
+- **Steps done while holding say so:** their text starts "Keep holding" **(checked)**.
+- **Say exactly how, in the ready screen's text:** where the hand goes (the same grip each time), which way
+  it moves, how far, what stays still ("Keep it level", "Its middle stays where it is"), and that the head
+  and eyes don't move. "Push the headset up" alone leaves people guessing.
+- **A few millimetres, one hand, gently.** Say it in the round's intro, with the kinds of move to come, and
+  that a little blur while it's moved is fine. Never press it into the face.
+
 ## Cue words are labels
 
 Training reads the cue words in `prompts.jsonl` (`frame-eyes/train/data/ingest.py`, `shut_cams`):

gaze/rec/rounds/slips.json

@@ -2,25 +2,35 @@
  "schema": 1,
  "id": "slips",
  "title": "Headset slips",
- "intro": "You'll nudge the headset on your face on purpose, then put it back. Use one hand, gently: a few millimetres is enough.",
+ "intro": "You'll move the headset on your face on purpose, seven ways: up, down, to your left, to your right, tilted, tipped forward and away from your face. Each time it's one hand on the front and a few millimetres, no more; each step says how. The picture may blur a little while it's moved: that's fine. Then you take it off and put it back on.",
  "steps": [
   {
    "id": "nudge-up",
    "kind": "prompt",
    "group": "nudges",
-   "text": "Keep looking at the dot and push the headset up a little. Hold it there.",
+   "text": "Look at the dot. When it says Push up and beeps:|Hold the front with one hand, thumb under the bottom edge, and push the headset straight up a few millimetres, so it sits higher on your face. Keep it level: both sides go up together.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
    "dot": [
     0,
     0
    ],
-   "seconds": 5
+   "cues": [
+    [
+     "Look at the dot",
+     1.5
+    ],
+    [
+     "Push up",
+     2.5,
+     "go"
+    ]
+   ]
   },
   {
    "id": "after-up",
    "kind": "dots",
    "group": "nudges",
    "ready": false,
-   "text": "Hold the headset there. Look at each dot until its ring fills. Keep your head still.",
+   "text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
    "dots": [
     [
      0,
@@ -46,22 +56,50 @@
    "hold": 1.0
   },
   {
-   "id": "nudge-down",
+   "id": "back-up",
    "kind": "prompt",
    "group": "nudges",
-   "text": "Let go, then pull the headset down a little. Hold it there.",
+   "ready": false,
+   "text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
    "dot": [
     0,
     0
    ],
-   "seconds": 5
+   "cues": [
+    [
+     "Let go, put it back",
+     3,
+     "done"
+    ]
+   ]
+  },
+  {
+   "id": "nudge-down",
+   "kind": "prompt",
+   "group": "nudges",
+   "text": "Look at the dot. When it says Pull down and beeps:|Hold the front with one hand, thumb under the bottom edge, and pull the headset straight down a few millimetres, the way a loose headset sags. Keep it level: both sides go down together.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Look at the dot",
+     1.5
+    ],
+    [
+     "Pull down",
+     2.5,
+     "go"
+    ]
+   ]
   },
   {
    "id": "after-down",
    "kind": "dots",
    "group": "nudges",
    "ready": false,
-   "text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
+   "text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
    "dots": [
     [
      0,
@@ -87,22 +125,50 @@
    "hold": 1.0
   },
   {
-   "id": "nudge-left",
+   "id": "back-down",
    "kind": "prompt",
    "group": "nudges",
-   "text": "Let go, then slide the headset a little to your left.",
+   "ready": false,
+   "text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
    "dot": [
     0,
     0
    ],
-   "seconds": 5
+   "cues": [
+    [
+     "Let go, put it back",
+     3,
+     "done"
+    ]
+   ]
+  },
+  {
+   "id": "nudge-left",
+   "kind": "prompt",
+   "group": "nudges",
+   "text": "Look at the dot. When it says Slide to your left and beeps:|Hold the front with one hand, thumb under the bottom edge, and slide the headset a few millimetres to your left, across your face. Keep it level and at the same height.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Look at the dot",
+     1.5
+    ],
+    [
+     "Slide to your left",
+     2.5,
+     "go"
+    ]
+   ]
   },
   {
    "id": "after-left",
    "kind": "dots",
    "group": "nudges",
    "ready": false,
-   "text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
+   "text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
    "dots": [
     [
      0,
@@ -128,22 +194,50 @@
    "hold": 1.0
   },
   {
-   "id": "nudge-right",
+   "id": "back-left",
    "kind": "prompt",
    "group": "nudges",
-   "text": "Let go, then slide it a little to your right.",
+   "ready": false,
+   "text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
    "dot": [
     0,
     0
    ],
-   "seconds": 5
+   "cues": [
+    [
+     "Let go, put it back",
+     3,
+     "done"
+    ]
+   ]
+  },
+  {
+   "id": "nudge-right",
+   "kind": "prompt",
+   "group": "nudges",
+   "text": "Look at the dot. When it says Slide to your right and beeps:|Hold the front with one hand, thumb under the bottom edge, and slide the headset a few millimetres to your right, across your face. Keep it level and at the same height.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Look at the dot",
+     1.5
+    ],
+    [
+     "Slide to your right",
+     2.5,
+     "go"
+    ]
+   ]
   },
   {
    "id": "after-right",
    "kind": "dots",
    "group": "nudges",
    "ready": false,
-   "text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
+   "text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
    "dots": [
     [
      0,
@@ -169,22 +263,50 @@
    "hold": 1.0
   },
   {
-   "id": "nudge-tilt",
+   "id": "back-right",
    "kind": "prompt",
    "group": "nudges",
-   "text": "Let go, then tilt the headset a little: one side up.",
+   "ready": false,
+   "text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
    "dot": [
     0,
     0
    ],
-   "seconds": 5
+   "cues": [
+    [
+     "Let go, put it back",
+     3,
+     "done"
+    ]
+   ]
+  },
+  {
+   "id": "nudge-tilt",
+   "kind": "prompt",
+   "group": "nudges",
+   "text": "Look at the dot. When it says Tilt: left side up and beeps:|Hold the front with one hand, thumb under the bottom edge, and turn your wrist so the headset's left side goes up a few millimetres and its right side down, so it sits crooked on your face. Its middle stays where it is.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Look at the dot",
+     1.5
+    ],
+    [
+     "Tilt: left side up",
+     2.5,
+     "go"
+    ]
+   ]
   },
   {
    "id": "after-tilt",
    "kind": "dots",
    "group": "nudges",
    "ready": false,
-   "text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
+   "text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
    "dots": [
     [
      0,
@@ -209,6 +331,162 @@
    ],
    "hold": 1.0
   },
+  {
+   "id": "back-tilt",
+   "kind": "prompt",
+   "group": "nudges",
+   "ready": false,
+   "text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Let go, put it back",
+     3,
+     "done"
+    ]
+   ]
+  },
+  {
+   "id": "nudge-tip",
+   "kind": "prompt",
+   "group": "nudges",
+   "text": "Look at the dot. When it says Tip the front down and beeps:|Hold the front with one hand, thumb under the bottom edge, and tip the front down a little, the way a heavy headset droops: its bottom edge goes down a few millimetres and its top stays against your forehead. Keep it level side to side.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Look at the dot",
+     1.5
+    ],
+    [
+     "Tip the front down",
+     2.5,
+     "go"
+    ]
+   ]
+  },
+  {
+   "id": "after-tip",
+   "kind": "dots",
+   "group": "nudges",
+   "ready": false,
+   "text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
+   "dots": [
+    [
+     0,
+     0
+    ],
+    [
+     15,
+     12
+    ],
+    [
+     -15,
+     -12
+    ],
+    [
+     15,
+     -12
+    ],
+    [
+     -15,
+     12
+    ]
+   ],
+   "hold": 1.0
+  },
+  {
+   "id": "back-tip",
+   "kind": "prompt",
+   "group": "nudges",
+   "ready": false,
+   "text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Let go, put it back",
+     3,
+     "done"
+    ]
+   ]
+  },
+  {
+   "id": "nudge-away",
+   "kind": "prompt",
+   "group": "nudges",
+   "text": "Look at the dot. When it says Pull away from your face and beeps:|Hold the front with one hand, thumb under the bottom edge, and pull the headset a few millimetres straight out, away from your face, so there's a small gap at your cheeks. Gently: the strap pulls back. Keep it level.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Look at the dot",
+     1.5
+    ],
+    [
+     "Pull away from your face",
+     2.5,
+     "go"
+    ]
+   ]
+  },
+  {
+   "id": "after-away",
+   "kind": "dots",
+   "group": "nudges",
+   "ready": false,
+   "text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
+   "dots": [
+    [
+     0,
+     0
+    ],
+    [
+     15,
+     12
+    ],
+    [
+     -15,
+     -12
+    ],
+    [
+     15,
+     -12
+    ],
+    [
+     -15,
+     12
+    ]
+   ],
+   "hold": 1.0
+  },
+  {
+   "id": "back-away",
+   "kind": "prompt",
+   "group": "nudges",
+   "ready": false,
+   "text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
+   "dot": [
+    0,
+    0
+   ],
+   "cues": [
+    [
+     "Let go, put it back",
+     3,
+     "done"
+    ]
+   ]
+  },
   {
    "id": "reseat",
    "kind": "reseat",

gaze/rec/session.py

@@ -24,9 +24,12 @@ steps are recorded; the frame writer pauses while a step waits. During the round
 looks again at the end of each centre dot, and a headset that moved since the fitment gets a
 note (fit.json "slips").
 
-Controls, as the hand recorder's: Next, pause, redo (the step running, or at a ready screen the
-one before; the range is marked with a "redo" event), skip (the rest of the take), stop. The
-headset button: press = next or pause, two presses = redo, hold = stop.
+Controls, as the hand recorder's: Next, pause, redo, skip (the rest of the take), stop. Redo in
+the countdown or the step starts it again; at a ready screen it goes back to the step before,
+into the take before too (Back). A step that follows on without a ready screen goes again with
+the one it follows. The range done before is marked with a "redo" event. The headset button:
+press = next or pause, two presses = redo, hold = stop. The panel's keys line says what redo
+does at that moment.
 
 Round scripts, rounds/<id>.json: {"schema": 1, "id", "title", "intro", "shift_check": true,
 "steps": [step]}. A step: {"id", "kind", "text" ("|" starts a line), "group" (default its id),
@@ -37,8 +40,9 @@ Round scripts, rounds/<id>.json: {"schema": 1, "id", "title", "intro", "shift_ch
   jumps    "count", "range": [yaw, pitch] (a box), "each": [shortest, longest] s
   prompt   "dot": [yaw, pitch] or null (none), and "seconds" or "cues": [[word, s]...] (shown
            big in turn; [word, s, sound] also plays a beep as it starts, for steps done with the
-           eyes shut: sound blink, close or open, sounds.py; rounds/README.md has the rules,
-           and load_round enforces them), "cue_distance": {word: metres}
+           eyes shut or moving the headset by hand: sound blink, close, open, go or done,
+           sounds.py; rounds/README.md has the rules, and load_round enforces them),
+           "cue_distance": {word: metres}
   light    "dot", "levels": [[background, s]...]
   world    "point": [x, y, z] (metres from the head when it starts, -z ahead), "seconds"
   reseat   not recorded: the text, Next, then a fit reading
@@ -121,9 +125,11 @@ READY_BUTTON = "Ready? Press the button on the right side of the headset"
 READY_BUTTON_MOUSE = "Ready? Press Space, click Next, or press the headset button"
 RESUME_HINT = "Paused. Resume: P in the Eye recorder window"
 RESUME_HINT_BUTTON = "Paused. Resume: press the headset button, or P in the Eye recorder window"
-KEYS = "Eye recorder window:  Space next  ·  P pause  ·  R redo  ·  S skip  ·  Esc stop"
-KEYS_BUTTON = ("Headset button: press for next or pause  ·  press twice to redo  ·  hold to stop  "
-               "·  Window: Space P R S Esc")
+# What R (two presses of the headset button) does now, for the panel's keys line (_keys): during
+# the countdown and the step, it starts again; at a ready screen, the step before goes again.
+REDO_RUNNING = "start this step again"
+REDO_BACK = "back to the last step"
+NOTHING_BACK = "Nothing to go back to from here"
 FIT_TITLE = "Headset fit"
 FIT_TEXT = "Look at the dot. Adjust the headset until this says good."
 FIT_ADJUST = "Adjust the headset until it says good"
@@ -216,11 +222,20 @@ def _dot(v, what):
     return d
 
 
+HAND_WORDS = ("push", "pull", "slide", "tilt", "tip", "nudge")   # a cue word starting so moves the headset
+HOLD_TEXT = "Keep holding"      # the text of a step done while holding the headset moved
+
+
 def cue_needs(word):
-    """The sound a cue must have because the person can't see the panel then (rounds/README.md):
-    "blink" for a blink, "close" for shutting both eyes, else None. Half-closed eyes and one eye
-    (a wink: the other one reads the panel) don't need one."""
+    """The sound a cue must have because the person can't see the panel then, or the timing is
+    the point (rounds/README.md): "blink" for a blink, "close" for shutting both eyes, "go" for
+    moving the headset by hand, "done" for letting go of it, else None. Half-closed eyes and one
+    eye (a wink: the other one reads the panel) don't need one."""
     w = word.lower()
+    if w.split()[:1] and w.split()[0].strip(":,") in HAND_WORDS:
+        return "go"
+    if "let go" in w:
+        return "done"
     if "half" in w or "left" in w or "right" in w:
         return None
     if "blink" in w:
@@ -232,7 +247,8 @@ def cue_needs(word):
 
 def _check_sounds(cues, sid):
     """Eyes-shut cues beep, and the cue after a "close" plays "open": with the eyes shut, the
-    words can't tell anyone when to open them."""
+    words can't tell anyone when to open them. A push on the headset ("go") comes after a cue on
+    the dot, and letting go ("done") shows long enough to settle."""
     for k, (word, _secs, *sound) in enumerate(cues):
         need = cue_needs(word)
         if need and sound != [need]:
@@ -240,6 +256,38 @@ def _check_sounds(cues, sid):
                              % (sid, word, need))
         if sound == ["close"] and (k + 1 == len(cues) or cues[k + 1][2:] != ["open"]):
             raise ValueError('%s: after "%s" comes a cue with the sound "open" (rounds/README.md)' % (sid, word))
+        if sound == ["go"] and k == 0:
+            raise ValueError('%s: "%s" comes after a cue on the dot, so the eyes are still first (rounds/README.md)'
+                             % (sid, word))
+        if sound == ["done"] and _secs < 2:
+            raise ValueError('%s: "%s" shows 2 s or more, so the eyes settle (rounds/README.md)' % (sid, word))
+
+
+def _check_hands(steps, path):
+    """A headset moved by hand is let go of in the same chain (the steps that follow on without a
+    ready screen), and the steps done while holding it say so: the person can't take a hand off
+    the headset to press Next, and a redo does the whole chain again."""
+    holding = None      # the step whose "go" cue moved the headset
+    for k, st in enumerate(steps):
+        if holding and (st["ready"] or st["group"] != holding["group"] or st["kind"] == "reseat"):
+            raise ValueError('%s: %s moves the headset; a "done" cue ("Let go...") follows in its chain '
+                             '(rounds/README.md)' % (path, holding["id"]))
+        sounds_ = [c[2] for c in st.get("cues") or [] if len(c) > 2]
+        if holding and not st["text"].startswith(HOLD_TEXT):
+            raise ValueError('%s: %s is done holding the headset; its text starts "%s" (rounds/README.md)'
+                             % (path, st["id"], HOLD_TEXT))
+        for snd in sounds_:
+            if snd == "go":
+                if holding:
+                    raise ValueError('%s: %s moves the headset again before letting go' % (path, st["id"]))
+                holding = st
+            elif snd == "done":
+                if not holding:
+                    raise ValueError('%s: %s lets go of a headset nothing moved' % (path, st["id"]))
+                holding = None
+    if holding:
+        raise ValueError('%s: %s moves the headset; a "done" cue ("Let go...") follows in its chain '
+                         '(rounds/README.md)' % (path, holding["id"]))
 
 
 def _step(raw, i):
@@ -356,6 +404,7 @@ def load_round(path):
         last = s["group"]
     if "dots" in groups:
         raise ValueError("%s: the group dots is the opener's" % path)
+    _check_hands(steps, path)
     rid = raw.get("id")
     return {"id": rid if isinstance(rid, str) else os.path.basename(path)[:-5], "title": str(raw.get("title", rid)),
             "intro": str(raw.get("intro", "")), "shift_check": raw.get("shift_check", True) is not False and
@@ -647,6 +696,7 @@ class Session:
         self._paused = self._paused_recording = False
         self._waiting = False
         self._redo_ok = False
+        self._lingering = False
         self._button_reader = None
         self._ipd = None
         self._step_t0 = None
@@ -897,7 +947,7 @@ class Session:
     def _finish(self, state, error=""):
         if self._take:
             try:
-                self._end_take("complete" if state == "done" else "stopped")
+                self._end_take("complete" if state == "done" or self._lingering else "stopped")
             except Exception:
                 self._log(traceback.format_exc())
         if self._session_json is not None:
@@ -963,6 +1013,9 @@ class Session:
             raise _Skip()
         if want["redo"] and self._redo_ok:
             raise _Redo()
+        if want["redo"] and self._waiting:
+            self._panel.set("note", "note " + NOTHING_BACK)
+            self._emit(note=NOTHING_BACK)
         if not want["pause"]:
             return False
         self._pause()
@@ -1064,9 +1117,21 @@ class Session:
         text = (READY_BUTTON_MOUSE if mouse_connected(self.input_devices) else READY_BUTTON) if button else READY_TEXT
         if action:
             self._panel.set("action", "action " + text)
-        self._panel.set("keys", "keys " + (KEYS_BUTTON if button else KEYS))
+        self._keys(back=self._redo_ok)
         self._status.update(ready_text=text, button=button)
 
+    def _keys(self, running=False, back=False):
+        """The panel's bottom line: the controls, saying what R does now (or leaving it out)."""
+        redo = REDO_RUNNING if running else REDO_BACK if back else ""
+        if self._button_ok():
+            parts = ["Headset button: press for " + ("pause" if running else "next")]
+            parts += ["press twice: " + redo] if redo else []
+            text = "  ·  ".join(parts + ["hold to stop", "Window: Space P R S Esc"])
+        else:
+            parts = ["Eye recorder window:  Space next", "P pause"] + (["R " + redo] if redo else [])
+            text = "  ·  ".join(parts + ["S skip", "Esc stop"])
+        self._panel.set("keys", "keys " + text)
+
     def _show(self, title, text):
         self._panel.set("title", "title " + clean_text(title))
         self._panel.set("text", "text " + clean_text(text))
@@ -1292,15 +1357,29 @@ class Session:
             self._log("take %s %s" % (t["id"], status))
 
     # --- the steps
+    def _chain_head(self, k):
+        """The first step of k's chain: the steps that follow on without a ready screen ("ready":
+        false) go again with the one they follow (a nudge and the dots done while holding it)."""
+        plan, h = self.plan, k
+        while h > 0 and not plan[h]["ready"] and plan[h - 1]["group"] == plan[k]["group"]:
+            h -= 1
+        return h
+
     def _steps(self):
         """The plan's steps. Each "ready" one (and one to do again) shows first and waits for
         Next, then a recorded countdown; the recording pauses after a step unless the next one
-        follows straight on. A new group starts a new take. R during a step: it starts again;
-        at a ready screen: the step before (in the same take) goes again."""
+        follows straight on. A new group starts a new take, once its first step's Next is
+        pressed: until then the take before stays open, so Back can reach its last step.
+
+        R during the countdown or the step: it starts again, from the start of its chain (a step
+        that follows on goes again with the one it follows). R at a ready screen (Back): the
+        step before goes again, with its chain. Either way the range done before is marked with
+        a "redo" event. Nothing goes back over a reseat: the headset sits differently after it."""
         plan = self.plan
-        done = []       # steps finished in this take: (index, id, from_ns, to_ns)
+        done = []       # steps finished in the open take: (index, id, from_ns, to_ns)
         again = set()
         k = 0
+        self._lingering = False   # the open take's steps are done; it ends at the next step's Next
         while k < len(plan):
             st = plan[k]
             where = "%s · step %d of %d" % (self._session_json["title"], k + 1, len(plan))
@@ -1310,23 +1389,31 @@ class Session:
                 if self._take:
                     self._end_take("complete")
                 done = []
+                self._lingering = False
                 self._reseat(st)
                 k += 1
                 continue
-            if self._take and self._take["group"] != st["group"]:
-                self._end_take("complete")
+            ready = st["ready"] or k in again
+            if self._take and self._take["group"] != st["group"] and not ready:
+                self._end_take("complete")      # follows on into a new group: no screen to go back from
                 done = []
+            self._lingering = bool(self._take and self._take["group"] != st["group"])
             started = self._step_t0 = None
             try:
-                if st["ready"] or k in again:
+                if ready:
                     self._redo_ok = bool(done)
                     self._ready(st, can_redo=bool(done))
                     self._await_next()
+                    if self._take and self._take["group"] != st["group"]:
+                        self._end_take("complete")
+                        done = []
+                    self._lingering = False
                     self._redo_ok = True
                     started = self._countdown(st)
                 else:
                     self._redo_ok = True
                     self._show(self._title(st), st["text"])
+                    self._keys(running=True)
                     self._emit(can_redo=True, prompt=st["text"].replace("|", "\n"))
                     self._step_t0 = started = self._event("step", id=st["id"], kind=st["kind"])
                 self._take["json"]["steps"].append(st["id"]) if st["id"] not in self._take["json"]["steps"] else None
@@ -1340,29 +1427,43 @@ class Session:
                 k += 1
             except _Redo:
                 self._redo_ok = False
-                if started is not None:
-                    self._event("redo", id=st["id"], **{"from": started, "to": mono_ns()})
-                    self._log("    redo %s" % st["id"])
+                if started is None:
+                    started = self._step_t0   # R in the countdown: its "ready" event
+                if started is not None:   # this step again, from the start of its chain
+                    h = self._chain_head(k)
+                    while done and done[-1][0] >= h:
+                        started = done.pop()[2]
+                    self._event("redo", id=plan[h]["id"], **{"from": started, "to": mono_ns()})
+                    self._log("    redo %s" % plan[h]["id"])
                     self._step_end()
-                    again.add(k)
-                elif done:
+                    k = h
+                    again = {x for x in again if x < k} | {k}
+                elif done:                # Back: the step before, with its chain
                     k, sid, a, b = done.pop()
+                    h = self._chain_head(k)
+                    while done and done[-1][0] >= h:
+                        k, sid, a, _ = done.pop()
                     self._event("redo", id=sid, **{"from": a, "to": b})
-                    self._log("    redo %s" % sid)
-                    again.add(k)
+                    self._log("    back to %s" % sid)
+                    self._lingering = False
+                    again = {x for x in again if x < k} | {k}
             except _Skip:
                 group = st["group"]
                 self._log("skipped the rest of %s" % group)
-                if self._take:
+                if self._take and self._take["group"] != group:
+                    self._end_take("complete")      # at the first step's ready screen: the take before is done
+                elif self._take:
                     self._step_end()
                     self._end_take("skipped")
                 done = []
+                self._lingering = False
                 while k < len(plan) and plan[k]["group"] == group and plan[k]["kind"] != "reseat":
                     k += 1
             finally:
                 self._redo_ok = False
         if self._take:
             self._end_take("complete")
+        self._lingering = False
 
     def _ready(self, st, can_redo):
         self._show(self._title(st), st["text"])
@@ -1394,6 +1495,9 @@ class Session:
             self._start_take(st["group"])
         t = self._step_t0 = self._event("ready", id=st["id"], seconds=COUNTDOWN_S)
         self._record(True)
+        self._keys(running=True)
+        self._panel.set("note", "note ")
+        self._emit(note="")
         first = self._first_dot(st)
         if first:
             self._panel.cmd("dot %.3f %.3f look" % tuple(first))

gaze/rec/sounds.py

@@ -1,11 +1,13 @@
 #!/usr/bin/env python3
 """The eye recorder's sound cues: plain beeps for steps done with the eyes shut, where the panel's
-words can't be seen. A round script names one as a cue's third field (session.py: "cues":
+words can't be seen, and for moving the headset by hand, where the timing matters. A round script names one as a cue's third field (session.py: "cues":
 [[word, seconds, sound]...]); it plays as the cue starts, and prompts.jsonl's "cue" line says which.
 
   blink   a short high beep: blink once
   close   a beep: close your eyes
   open    the same beep again: open them
+  go      a higher beep: move the headset now (the cue word says how)
+  done    a low, longer one: let go of it
 
 No voice, no sound files: the beeps are made here (a sine with a short fade, so it doesn't click)
 into the run folder, and played with PipeWire's pw-play on the default output (the headset's
@@ -31,6 +33,8 @@ SOUNDS = {
     "blink": (1000.0, 0.09),
     "close": (660.0, 0.18),
     "open": (660.0, 0.18),
+    "go": (880.0, 0.14),
+    "done": (440.0, 0.3),
 }
 FADE_S = 0.005
 

gaze/rec/tests/test_session.py

@@ -76,6 +76,24 @@ class RoundTest(unittest.TestCase):
         self.bad([{"id": "a", "kind": "prompt", "cues": [["Close both eyes", 2], ["Open", 2]]}], 'needs the sound "close"')
         self.bad([{"id": "a", "kind": "prompt", "cues": [["Shut your eyes", 2, "close"]]}], 'comes a cue with the sound "open"')
         self.bad([{"id": "a", "kind": "prompt", "cues": [["Close your eyes", 2, "close"], ["Open", 2]]}], '"open"')
+        # Moving the headset by hand
+        look = ["Look at the dot", 1.5]
+        self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2]]}], 'needs the sound "go"')
+        self.bad([{"id": "a", "kind": "prompt", "cues": [["Push up", 2, "go"], ["Let go", 2, "done"]]}],
+                 "the eyes are still first")
+        self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2, "go"], ["Let go", 1, "done"]]}], "2 s or more")
+        self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2, "go"]]}], 'a "done" cue')
+        self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2, "go"]]},
+                  {"id": "b", "kind": "prompt", "cues": [["Let go", 2, "done"]]}], 'a "done" cue')
+        self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Tilt: left side up", 2, "go"]]},
+                  {"id": "b", "kind": "dots", "ready": False, "group": "a", "dots": [[0, 0]], "text": "Look"},
+                  {"id": "c", "kind": "prompt", "ready": False, "group": "a", "text": "Keep holding",
+                   "cues": [["Let go", 2, "done"]]}], 'starts "Keep holding"')
+        session.load_round(write_round(self.tmp, [
+            {"id": "a", "kind": "prompt", "cues": [look, ["Slide to your left", 2, "go"]]},
+            {"id": "b", "kind": "dots", "ready": False, "group": "a", "dots": [[0, 0]], "text": "Keep holding it"},
+            {"id": "c", "kind": "prompt", "ready": False, "group": "a", "text": "Keep holding",
+             "cues": [["Let go, put it back", 2, "done"]]}]))
         for word in ("Close your left eye", "Half closed", "Squint", "Open"):
             session.load_round(write_round(self.tmp, [{"id": "a", "kind": "prompt", "cues": [[word, 1]]}]))
 
@@ -300,6 +318,116 @@ class DryRunTest(unittest.TestCase):
         self.assertLess(redo[0]["from"], redo[0]["to"])
         self.assertNotIn("bounties", profile, "a stopped round isn't counted")
 
+    # A round of two takes: a nudge with dots done while holding it (a chain), then one prompt.
+    TWO = [{"id": "nudge", "kind": "prompt", "group": "one", "dot": [0, 0], "seconds": 1, "text": "Push"},
+           {"id": "hold", "kind": "dots", "group": "one", "dots": [[0, 0], [5, 0]], "hold": 0.5, "ready": False,
+            "text": "Keep holding"},
+           {"id": "other", "kind": "prompt", "group": "two", "dot": [0, 0], "seconds": 1, "text": "Other"}]
+
+    def run_two(self, steer, **kw):
+        write_round(self.tmp, self.TWO)
+        return self.run_round("t", steer=steer, speed=10, next_after=0.6, rounds_dir=self.tmp, fit_coach=False,
+                              profile={"eyes_ipd": 0.063}, **kw)
+
+    @staticmethod
+    def at(s, step, state):
+        end = time.monotonic() + 30
+        while time.monotonic() < end and not (s._status["step"] == step and s._status["state"] == state):
+            time.sleep(0.005)
+        return s._status["step"] == step and s._status["state"] == state
+
+    def take_status(self, s, take):
+        with open(os.path.join(s.session_dir, "takes", take, "take.json")) as f:
+            return json.load(f)["status"]
+
+    def redos(self, s, take):
+        return [e["id"] for e in self.prompts(s, take) if e["event"] == "redo"]
+
+    def test_redo_in_the_countdown(self):
+        # R in the countdown starts this step again (it used to go back to the one before).
+        seen = {}
+        def steer(s, states):
+            self.assertTrue(self.at(s, "other", "countdown"))
+            s.redo()
+            seen["ready"] = self.at(s, "other", "ready")
+            seen["recording"] = s._recording
+        s, _, _ = self.run_two(steer)
+        self.assertEqual(s.state, "done")
+        self.assertTrue(seen["ready"])
+        self.assertFalse(seen["recording"], "the ready screen isn't recorded")
+        takes = self.files(s)[0]["takes"]
+        self.assertEqual(self.redos(s, takes[-1]), ["other"])
+        self.assertEqual(self.redos(s, takes[-2]), [])
+
+    def test_back_into_the_take_before(self):
+        # Back at a new take's first ready screen: the take before stays open until its Next,
+        # so its last step (with the nudge it follows on from) goes again in the same take.
+        seen = {}
+        def steer(s, states):
+            self.assertTrue(self.at(s, "other", "ready"))
+            seen["can_redo"] = s._status["can_redo"]
+            s.redo()
+            seen["back"] = self.at(s, "nudge", "ready")
+        s, _, _ = self.run_two(steer)
+        self.assertEqual(s.state, "done")
+        self.assertTrue(seen["can_redo"])
+        self.assertTrue(seen["back"], "back to the chain's head, not to hold")
+        takes = self.files(s)[0]["takes"]
+        self.assertEqual(len(takes), 3, takes)        # the opener's dots, one, two: no extra take
+        one = self.prompts(s, takes[1])
+        self.assertEqual(self.redos(s, takes[1]), ["nudge"])
+        self.assertEqual([e["id"] for e in one if e["event"] == "step"], ["nudge", "hold", "nudge", "hold"])
+        self.assertEqual(self.take_status(s, takes[1]), "complete")
+
+    def test_redo_a_step_that_follows_on(self):
+        # R while holding: the nudge goes again with it.
+        def steer(s, states):
+            self.assertTrue(self.at(s, "hold", "running"))
+            s.redo()
+            self.assertTrue(self.at(s, "nudge", "ready"))
+        s, _, _ = self.run_two(steer)
+        self.assertEqual(s.state, "done")
+        takes = self.files(s)[0]["takes"]
+        steps = [e["id"] for e in self.prompts(s, takes[1]) if e["event"] == "step"]
+        self.assertEqual(self.redos(s, takes[1]), ["nudge"])
+        self.assertEqual(steps, ["nudge", "hold", "nudge", "hold"])
+
+    def test_stop_or_skip_at_the_next_ready_screen(self):
+        # The take before is finished, not stopped or skipped, when the next take's ready screen
+        # is where it ends.
+        def stop(s, states):
+            self.assertTrue(self.at(s, "other", "ready"))
+            s.stop(wait=0)
+        s, _, _ = self.run_two(stop)
+        self.assertEqual(s.state, "stopped")
+        takes = self.files(s)[0]["takes"]
+        self.assertEqual(len(takes), 2)
+        self.assertEqual(self.take_status(s, takes[1]), "complete")
+
+        def skip(s, states):
+            self.assertTrue(self.at(s, "other", "ready"))
+            s.skip()
+        s, _, _ = self.run_two(skip)
+        self.assertEqual(s.state, "done")
+        takes = self.files(s)[0]["takes"]
+        self.assertEqual(len(takes), 2, "two was skipped before it started")
+        self.assertEqual(self.take_status(s, takes[1]), "complete")
+
+    def test_nothing_to_go_back_to(self):
+        # After a reseat Back is off: the headset sits differently now.
+        rnd = [{"id": "a", "kind": "prompt", "dot": [0, 0], "seconds": 1, "text": "A"},
+               {"id": "off", "kind": "reseat", "text": "off and on"},
+               {"id": "b", "kind": "prompt", "dot": [0, 0], "seconds": 1, "text": "B"}]
+        write_round(self.tmp, rnd)
+        seen = {}
+        def steer(s, states):
+            self.assertTrue(self.at(s, "b", "ready"))
+            seen["can_redo"] = s._status["can_redo"]
+        s, states, _ = self.run_round("t", steer=steer, speed=10, next_after=0.4, rounds_dir=self.tmp,
+                                      fit_coach=False, profile={"eyes_ipd": 0.063})
+        self.assertEqual(s.state, "done")
+        self.assertFalse(seen["can_redo"], "nothing goes back over a reseat")
+
     def test_pause_stops_the_recording(self):
         def steer(s, states):
             end = time.monotonic() + 30

gaze/rec/tests/test_sounds.py

@@ -24,7 +24,8 @@ class SoundsTest(unittest.TestCase):
         shutil.rmtree(self.tmp, ignore_errors=True)
 
     def test_wavs(self):
-        self.assertEqual(set(sounds.SOUNDS), {"blink", "close", "open"})
+        self.assertEqual(set(sounds.SOUNDS), {"blink", "close", "open", "go", "done"})
+        self.assertEqual(len({sounds.SOUNDS[n] for n in ("blink", "close", "go", "done")}), 4, "told apart by ear")
         self.assertEqual(sounds.SOUNDS["close"], sounds.SOUNDS["open"], "open is the same beep again")
         for name, (hz, secs) in sounds.SOUNDS.items():
             path = sounds.wav_path(name)

gaze/test/blind-recalibration-test.py

@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+"""Offline test of our own eye tracker's calibration going blind (gaze/gazecheck.py, OWN_GONE):
+a full calibration while ours is calibrated but sends no gaze. Live 2026-10-18 a calibration
+with the left eye closed mapped only the right eye; the next one, with the right eye closed,
+took no dot, since every click waited for our tracker's gaze ("Waiting: the eye tracker isn't
+sending a gaze"). Now the dot stands in for the gaze while there's none, as in the first
+calibration (first-calibration-test.py), and ft-eyes checks the pupils it sees.
+
+Same harness as first-calibration-test.py: ft-gazed's Service with its sockets renamed and HOME
+in a temp folder, a fake pointer helper (gaze mode on, headset worn), a fake ft-gaze (SteamVR
+sees both eyes; ours sends a gaze while OWN_FLAG exists), a fake ft-eyes control socket
+(calibrated), and no panel. Nothing reaches the live gaze service, the pointer helper, ft-eyes,
+or SteamVR, so it's safe next to them.
+
+  gaze/test/blind-recalibration-test.py
+"""
+import os
+import tempfile
+
+HOME = tempfile.mkdtemp(prefix="ft-gaze-blind-recal-test-")
+os.environ["HOME"] = HOME  # before gazecal: its STATE, and frametop.conf, follow HOME
+
+import importlib.machinery  # noqa: E402
+import importlib.util  # noqa: E402
+import json  # noqa: E402
+import selectors  # noqa: E402
+import shutil  # noqa: E402
+import socket  # noqa: E402
+import subprocess  # noqa: E402
+import sys  # noqa: E402
+import threading  # noqa: E402
+import time  # noqa: E402
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+GAZE = os.path.join(HERE, "..")
+sys.path.insert(0, GAZE)
+loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(GAZE, "ft-gazed"))
+gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
+loader.exec_module(gazed)
+import gazecheck  # noqa: E402  (the module ft-gazed imported)
+
+tag = f"ft_gaze_blind_recal_test_{os.getpid()}"
+gazed.ME = f"\0{tag}_gazed"
+gazed.POINTER = gazecheck.POINTER = f"\0{tag}_helper"
+gazecheck.SCREENS = f"\0{tag}_screens"
+gazecheck.PANEL = f"\0{tag}_panel"
+gazed.EYES_SOCKET = gazecheck.EYES = f"\0{tag}_eyes"
+gazed.read_settings = lambda: ("own", "auto", "auto", 55.0)
+gazed.TrackedEye = lambda: lambda now=None: None  # both eyes, whatever SteamVR's settings say
+DOTS = 4
+real_dots = gazecheck.check_dots
+gazecheck.check_dots = lambda kind, own: real_dots(kind, own)[:DOTS]  # a short calibration
+gazecheck.Fusion.MIN = DOTS   # ...whose dots still fit a fusion (SteamVR's eyes at each)
+gazecheck.LOOKED_AWAY = 1e9   # the fake SteamVR's eyes stay put whatever the dot (looked-away-test.py has that check)
+logs = []
+gazed.log = gazecheck.log = lambda msg: logs.append(msg)
+
+# The fake ft-gaze: 90 samples a second, SteamVR sees both eyes, ours sends a gaze (the right
+# eye's, at 1, 2) while OWN_FLAG exists, and {"ok":0} as with a stale file otherwise.
+FAKE = os.path.join(HOME, "ft-gaze")
+OWN_FLAG = os.path.join(HOME, "own-gaze")
+with open(FAKE, "w") as f:
+    f.write('''import json, os, select, sys, time
+flag = sys.argv[1]
+while True:
+    if select.select([sys.stdin], [], [], 1 / 90)[0]:
+        if not os.read(0, 4096):
+            break
+    own = {"hy": 1.0, "hp": 2.0, "eyes": [None, [1.0, 2.0]]} if os.path.exists(flag) else {"ok": 0}
+    print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
+          "open": [0.8, 0.8]}, "left": {"hy": 0.0, "hp": 2.0}, "right": {"hy": 2.0, "hp": 2.0},
+          "own": own}}), flush=True)
+''')
+SLEEPER = [sys.executable, "-c", "import sys; sys.stdin.read()"]  # quits when its stdin closes
+
+
+def own_gaze(on):
+    if on:
+        open(OWN_FLAG, "w").close()
+    elif os.path.exists(OWN_FLAG):
+        os.remove(OWN_FLAG)
+
+
+def start_helper(self):
+    """ft-gaze, straight from here instead of the dev container."""
+    self.proc = subprocess.Popen([sys.executable, FAKE, OWN_FLAG], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                                 stderr=subprocess.PIPE)
+    self.proc_sources = self.wanted_sources()
+    os.set_blocking(self.proc.stdout.fileno(), False)
+    os.set_blocking(self.proc.stderr.fileno(), False)
+    self.sel.register(self.proc.stdout, selectors.EVENT_READ, "stdout")
+    self.sel.register(self.proc.stderr, selectors.EVENT_READ, "stderr")
+    self.buf = b""
+
+
+def start_eyes(self):
+    """ft-eyes' process: a stand-in. Its control socket is the fake below."""
+    self.eyes_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
+    os.set_blocking(self.eyes_proc.stderr.fileno(), False)
+    self.sel.register(self.eyes_proc.stderr, selectors.EVENT_READ, "eyes")
+
+
+def start_panel(self):
+    self.panel_proc = subprocess.Popen(SLEEPER, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
+    os.set_blocking(self.panel_proc.stderr.fileno(), False)
+    self.sel.register(self.panel_proc.stderr, selectors.EVENT_READ, "panel")
+
+
+gazed.Service.start_helper = start_helper
+gazed.Service.start_eyes = start_eyes
+gazecheck.Checks.start_panel = start_panel
+
+# The fake ft-eyes: calibrated (the right eye only, as after a calibration with the left closed).
+eyes_state = {"cal": {"made": "test", "dots": 27}, "points": []}
+eyes = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+eyes.bind(gazed.EYES_SOCKET)
+eyes.settimeout(0.2)
+
+
+def eyes_answer():
+    while True:
+        try:
+            data, addr = eyes.recvfrom(512)
+        except socket.timeout:
+            continue
+        except OSError:
+            return
+        w = data.decode().split()
+        if w[0] == "status":
+            reply = json.dumps({"calibration": eyes_state["cal"], "calibrating": False, "dots": 0, "mapped": ["right"],
+                                "eyes": {"right": {"reseat": False}, "left": {"reseat": False}}})
+        elif w[0] == "calib-start":
+            eyes_state["points"] = []
+            reply = "ok"
+        elif w[0] == "calib-point":
+            eyes_state["points"].append(tuple(map(float, w[1:5])))
+            reply = "ok 0 50 nan 1.00 (left eye only: the right eye was seen in only 0 frames)"
+        elif w[0] == "calib-fit":
+            eyes_state["cal"] = {"made": "test", "dots": len(eyes_state["points"])}
+            reply = f"ok {len(eyes_state['points'])} dots, fit median 1.00 deg, eyes right not used (0 dots, needs 12), left 1.00 ({len(eyes_state['points'])} dots)"
+        elif w[0] == "click":
+            continue
+        else:
+            reply = f"fail unknown command {w[0]}"
+        if addr:
+            eyes.sendto(reply.encode(), addr)
+
+
+threading.Thread(target=eyes_answer, daemon=True).start()
+
+helper_state = {"reply": "ok on worn", "heard": []}
+helper = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+helper.bind(gazed.POINTER)
+helper.settimeout(0.2)
+
+
+def helper_answer():
+    while True:
+        try:
+            data, addr = helper.recvfrom(512)
+        except socket.timeout:
+            continue
+        except OSError:
+            return
+        if data.startswith(b"gaze ?") and addr:
+            helper.sendto(helper_state["reply"].encode(), addr)
+        else:
+            helper_state["heard"].append(data.decode())
+
+
+threading.Thread(target=helper_answer, daemon=True).start()
+own_gaze(True)
+svc = gazed.Service(None, False, gazed.POINTER)
+threading.Thread(target=svc.run, daemon=True).start()
+ctl = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+ctl.bind("")
+ctl.settimeout(3)
+
+
+def ask(cmd):
+    ctl.sendto(cmd.encode(), gazed.ME)
+    return ctl.recv(65536).decode()
+
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(("ok    " if ok else "FAIL  ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+def wait(cond, seconds):
+    end = time.monotonic() + seconds
+    while time.monotonic() < end:
+        if cond():
+            return True
+        time.sleep(0.05)
+    return cond()
+
+
+def check_state():
+    return svc.checks.check or {}
+
+
+def click_dot(i, settle=0.0):
+    """Wait for dot i, then click: is it taken (within 2 s)?"""
+    wait(lambda: check_state().get("i") == i and not check_state().get("done_at"), 3)
+    time.sleep(gazecheck.CHECK_SETTLE + gazecheck.CHECK_WINDOW + 0.1 + settle)
+    before = check_state().get("captured", 0)
+    ask("calaccept")
+    return wait(lambda: check_state().get("captured", 0) > before, 2)
+
+
+check("ours runs, calibrated", ask("wake 60") == "ok" and wait(lambda: svc.checks.calibrated() is True, 6), True)
+check("SteamVR sees the eyes (the fake ft-gaze is sending)", wait(svc.checks.eyes_seen, 6), True)
+check("calibrate: a full calibration opens",
+      ask("calibrate").startswith("ok") and wait(lambda: check_state().get("kind") == "full", 6), True)
+check("not blind: ours is calibrated", check_state().get("blind"), False)
+
+check("dot 1, ours sends a gaze: taken", click_dot(0), True)
+check("...on our gaze, not blind", (check_state().get("run_blind"), check_state().get("went_blind")), (False, False))
+
+# Dot 2: the eye ours maps closes, and its gaze stops. The old way (OWN_GONE never) the click waits.
+own_gaze(False)
+gazecheck.OWN_GONE = 1e9
+check("dot 2, no gaze from ours, the old way: the click takes nothing", click_dot(1), False)
+check("...and the panel says it waits for a gaze",
+      wait(lambda: "isn't sending a gaze" in check_state().get("note", ""), 2), True)
+gazecheck.OWN_GONE = 0.5
+check("now: the dot stands in for the gaze, and the click waiting there takes it",
+      wait(lambda: check_state().get("captured") == 2, 3), True)
+check("...said once in the log", sum("our tracker sends no gaze" in m for m in logs), 1)
+t0, t1, yaw, pitch = eyes_state["points"][-1]
+dot = gazecheck.check_dots("full", True)[1]
+check("ft-eyes got dot 2's direction", (round(yaw, 3), round(pitch, 3)), (round(dot[0], 3), round(dot[1], 3)))
+
+own_gaze(True)
+check("dot 3, ours sends a gaze again: taken on it", click_dot(2) and not check_state().get("run_blind"), True)
+own_gaze(False)
+check("dot 4, gone again: blind once OWN_GONE passes, taken", click_dot(3, gazecheck.OWN_GONE), True)
+t0, t1, yaw, pitch = eyes_state["points"][-1]
+check("its window is the look up to the click (about CHECK_WINDOW)", abs((t1 - t0) - gazecheck.CHECK_WINDOW) < 0.15, True)
+check("still said once in the log", sum("our tracker sends no gaze" in m for m in logs), 1)
+check("all dots: ours fits its calibration", wait(lambda: not svc.checks.check and len(eyes_state["points"]) == DOTS, 5), True)
+check("the fit's reply in the log", any("left 1.00 (4 dots)" in m for m in logs), True)
+
+# A check of our gaze (five) has nothing to check without one: it waits for ours to send, and
+# never goes blind.
+check("five check without our gaze: it waits for ours to send", ask("fivecheck"), "ok waking the eye tracker first")
+check("...and doesn't open meanwhile", wait(lambda: svc.checks.check is not None, 3), False)
+own_gaze(True)
+check("ours sends: the five check opens", wait(lambda: check_state().get("kind") == "five", 6), True)
+own_gaze(False)
+check("its click without our gaze takes nothing", click_dot(0, gazecheck.OWN_GONE), False)
+check("...and it never went blind", (check_state().get("run_blind"), check_state().get("went_blind")), (False, False))
+ask("calquit")
+
+print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+svc.running = False
+time.sleep(0.7)
+shutil.rmtree(HOME, ignore_errors=True)
+os._exit(1 if failures else 0)

gaze/test/first-calibration-test.py

@@ -49,6 +49,8 @@ gazed.TrackedEye = lambda: lambda now=None: None  # both eyes, whatever SteamVR'
 DOTS = 4
 real_dots = gazecheck.check_dots
 gazecheck.check_dots = lambda kind, own: real_dots(kind, own)[:DOTS]  # a short calibration
+gazecheck.Fusion.MIN = DOTS   # ...whose dots still fit a fusion (SteamVR's eyes at each)
+gazecheck.LOOKED_AWAY = 1e9   # the fake SteamVR's eyes stay put whatever the dot (looked-away-test.py has that check)
 logs = []
 gazed.log = gazecheck.log = lambda msg: logs.append(msg)
 
@@ -61,7 +63,8 @@ while True:
         if not os.read(0, 4096):
             break
     print(json.dumps({"t": time.monotonic(), "src": {"mmap1": {"hy": 1.0, "hp": 2.0, "unc": [0.001, 0.001],
-          "open": [0.8, 0.8]}, "own": {"ok": 0}}}), flush=True)
+          "open": [0.8, 0.8]}, "left": {"hy": 0.0, "hp": 2.0}, "right": {"hy": 2.0, "hp": 2.0},
+          "own": {"ok": 0}}}), flush=True)
 ''')
 SLEEPER = [sys.executable, "-c", "import sys; sys.stdin.read()"]  # quits when its stdin closes
 
@@ -285,6 +288,16 @@ check("the service sees it calibrated", wait(lambda: svc.checks.calibrated() is
 check("and gaze mode stays on", "gaze off" in helper_state["heard"], False)
 check("and no second calibration opens", wait(lambda: svc.checks.check is not None, 3), False)
 check("one calibration in the log", sum(m.startswith("full check:") for m in logs), 1)
+fusion = gazecheck.Fusion.load(gazecheck.FUSION)
+check("SteamVR's eyes at the dots ours took: a fusion saved", (fusion.info.get("dots"), fusion.info.get("how"))
+      if fusion else None, (DOTS, "calibration"))
+dots = [d[:2] for d in real_dots("full", True)[:DOTS]]
+check("its points: SteamVR's eyes' mean, and the dot", [p[:2] for p in fusion.points] if fusion else None,
+      [[1.0, 2.0]] * DOTS)
+check("...the dots", [tuple(round(v, 3) for v in p[2:]) for p in fusion.points] if fusion else None,
+      [tuple(round(v, 3) for v in d) for d in dots])
+check("the service loaded it", wait(lambda: svc.fusion is not None and svc.fusion.ready(), 3), True)
+check("GAZE_FUSION unset: not in use", (svc.fusing, svc.status()["fusion"]["setting"]), (False, "off"))
 
 print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
 svc.running = False

gaze/test/fusion-test.py

@@ -0,0 +1,178 @@
+#!/usr/bin/env python3
+"""Offline test of the fusion (gazecal.Fusion, ft-gazed's GAZE_FUSION): SteamVR's affine from
+calibration dots, the clicks' offset per wear, which tracker carries the gaze when, saving and
+loading fusion.json, the setting, and ft-gazed's live gaze with it on. ft-gazed's Service is
+built without its sockets, and HOME is a temp folder, so nothing reaches the live gaze service.
+
+  gaze/test/fusion-test.py
+"""
+import os
+import tempfile
+
+HOME = tempfile.mkdtemp(prefix="ft-fusion-test-")
+os.environ["HOME"] = HOME  # before gazecal: its STATE, and frametop.conf, follow HOME
+
+import importlib.machinery  # noqa: E402
+import importlib.util  # noqa: E402
+import math  # noqa: E402
+import shutil  # noqa: E402
+import sys  # noqa: E402
+from collections import deque  # noqa: E402
+from pathlib import Path  # noqa: E402
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(HERE, ".."))
+from gazecal import EyeWeights, Fusion, GazeLock  # noqa: E402
+
+loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(HERE, "..", "ft-gazed"))
+gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
+loader.exec_module(gazed)
+
+failures = []
+
+
+def check(what, got, want):
+    if got != want:
+        failures.append(what)
+        print(f"FAIL {what}: got {got!r}, want {want!r}", flush=True)
+    else:
+        print(f"ok   {what}", flush=True)
+
+
+def near(a, b, tol=1e-6):
+    return a is not None and b is not None and all(abs(x - y) < tol for x, y in zip(a, b))
+
+
+# --- gazecal.Fusion ---
+f = Fusion()
+check("not fitted: not ready, no SteamVR gaze", (f.ready(), f.steam_gaze((0, 0), (0, 0))), (False, None))
+# SteamVR's eyes' mean reads 1 degree high everywhere: the affine's offset takes it out
+f.fit([(y, p + 1, y, p) for y in (-20, -10, 0, 10, 20) for p in (-10, 0, 10)], {"how": "test"})
+check("fitted on 15 dots: ready", (f.ready(), f.info["dots"], f.info["how"]), (True, 15, "test"))
+check("an offset: taken out", near(f.steam_gaze((14.0, 6.0), (16.0, 6.0)), (15.0, 5.0)), True)
+check("...leaving nothing on its dots", max(f.misses()) < 1e-6, True)
+# ...and 10 % wide too: the gain is taken out as well without the ridge (with it, a little short:
+# on 2026-10-17's runs it fitted other sittings better)
+wide = Fusion()
+wide.RIDGE = 0.0
+wide.fit([(1.1 * y, 1.1 * p + 1, y, p) for y in (-20, -10, 0, 10, 20) for p in (-10, 0, 10)])
+check("a gain, no ridge: taken out", near(wide.steam_gaze((1.1 * 15 - 1, 1.1 * 5 + 1), (1.1 * 15 + 1, 1.1 * 5 + 1)),
+                                         (15.0, 5.0), 1e-6), True)
+wide.RIDGE = Fusion.RIDGE
+wide.fit(wide.points)
+check("with the ridge: most of it", 0.0 < max(wide.misses()) < 0.6, True)
+check("one SteamVR eye: no gaze from it", f.steam_gaze((1.0, 2.0), None), None)
+check("the mean of the eyes", Fusion.mean((0.0, 2.0), (2.0, 4.0)), (1.0, 3.0))
+
+# Who carries the gaze
+ours, steam = (2.0, 0.0), (0.0, 2.0)
+check("both our eyes and SteamVR's: half and half", f.combine(ours, 2, steam), (1.0, 1.0))
+check("one of our eyes: SteamVR's", f.combine(ours, 1, steam), steam)
+check("none of ours: SteamVR's", f.combine(None, 0, steam), steam)
+check("no SteamVR gaze: ours", f.combine(ours, 1, None), ours)
+check("a share of 1: ours where both are found", f.combine(ours, 2, steam, share=1.0), ours)
+
+# The clicks' offset: per wear, a median, stale clicks left out
+g = Fusion()
+g.fit([(y, p, y, p - 1) for y in (-10, 0, 10) for p in (-10, 0, 10)])   # 1 degree low... corrected
+g.wear = 100.0
+check("no clicks: no offset", g.offset(), (0.0, 0.0))
+check("a click 0.5 off: its miss", round(g.click((0.0, 0.0), (0.0, 0.0), 0.5, -1.0), 3), 0.5)
+check("...and it's the offset", near(g.offset(), (0.5, 0.0)), True)
+check("a click 20 degrees off (a look elsewhere): not taken", (round(g.click((0, 0), (0, 0), 20.0, -1.0), 1),
+                                                              near(g.offset(), (0.5, 0.0))), (20.0, True))
+g.wear = 200.0
+check("the headset on again: the old wear's clicks don't count", g.offset(), (0.0, 0.0))
+check("steam_gaze without the clicks", near(g.steam_gaze((3.0, 3.0), (3.0, 3.0), clicks=False), (3.0, 2.0)), True)
+
+# Saving and loading
+path = Path(HOME) / "fusion.json"
+f.save(path)
+h = Fusion.load(path)
+check("saved and loaded: the same correction", near(h.steam_gaze((5.0, 5.0), (7.0, 5.0)), f.steam_gaze((5.0, 5.0), (7.0, 5.0))),
+      True)
+check("...and its info", (h.info["how"], h.info["dots"], len(h.points)), ("test", 15, 15))
+check("a missing file: None", Fusion.load(Path(HOME) / "none.json"), None)
+path.write_text("{ broken")
+check("a broken file: None", Fusion.load(path), None)
+
+# --- GAZE_FUSION ---
+conf = Path(HOME) / ".config" / "frametop.conf"
+conf.parent.mkdir(parents=True, exist_ok=True)
+check("no frametop.conf: off", gazed.read_fusion(), False)
+for text, want in (("GAZE_FUSION=on\n", True), ("GAZE_FUSION = ON  # yes\n", True), ("GAZE_FUSION=off\n", False),
+                   ("GAZE_FUSION=yes\n", False), ("# GAZE_FUSION=on\n", False), ("GAZE_TRACKER=own\n", False)):
+    conf.write_text(text)
+    check(f"frametop.conf {text.strip()!r}", gazed.read_fusion(), want)
+
+
+# --- ft-gazed's live gaze: the parts of Service the samples go through, no sockets ---
+def service(fusion_on=True, one=None):
+    svc = gazed.Service.__new__(gazed.Service)
+    svc.override, svc.tracker, svc.source, svc.one_eye = None, "own", "mmap1", one
+    svc.counts = dict.fromkeys(("samples", "sent", "blinks", "one_eye", "one_eye_used", "lost_left", "lost_right",
+                                "looking_down", "dropped", "fused", "steam_only"), 0)
+    svc.opens, svc.lost = (deque(maxlen=90), deque(maxlen=90)), [False, False]
+    svc.fix, svc.last_sample, svc.low_until = GazeLock(radius=1.0), 0.0, [0.0, 0.0]
+    svc.bridge = gazed.EyeBridge()
+    svc.weights = {"own": EyeWeights()}
+    svc.fusion_on = fusion_on
+    svc.fusion = Fusion()
+    svc.fusion.fit([(y, p + 1, y, p) for y in (-10, 0, 10) for p in (-10, 0, 10)])   # SteamVR 1 degree high
+    svc.checks = type("Checks", (), {"active": False, "pending": None})()
+    svc.sent = []
+    svc.send = lambda t, hy, hp, rhy, rhp, eyes: svc.sent.append((round(hy, 3), round(hp, 3), eyes))
+    return svc
+
+
+def feed(svc, n=30, own_eyes=((4.0, 2.0), (4.0, 2.0)), steam=True, opens=(0.8, 0.8), unc=(0.001, 0.001)):
+    for i in range(n):
+        src = {"mmap1": {"unc": list(unc), "open": list(opens)},
+               "own": {"hy": 4.0, "hp": 2.0, "eyes": [list(e) if e else None for e in own_eyes]}}
+        if steam:
+            src["left"], src["right"] = {"hy": 1.0, "hp": 3.0}, {"hy": 3.0, "hp": 3.0}   # mean (2, 3): corrected (2, 2)
+        svc.on_eyes_sample({"t": i / 90, "src": src}, True)
+
+
+svc = service()
+check("fusing with our tracker, on, and fitted", svc.fusing, True)
+check("ft-gaze prints SteamVR's eyes too", svc.wanted_sources(), "own,mmap1,left,right")
+feed(svc)
+check("both our eyes: half ours (4, 2), half SteamVR's corrected (2, 2)", svc.sent[-1][:2], (3.0, 2.0))
+check("...counted as fused", (svc.counts["fused"], svc.counts["steam_only"]), (30, 0))
+check("...and the eyes sent with it are ours (for lessons)", svc.sent[-1][2], [(4.0, 2.0), (4.0, 2.0)])
+svc = service()
+feed(svc, own_eyes=((4.0, 2.0), None))
+check("one of our eyes: SteamVR's", svc.sent[-1][:2], (2.0, 2.0))
+check("...counted", (svc.counts["steam_only"], svc.counts["one_eye"]), (30, 30))
+svc = service()
+feed(svc, own_eyes=(None, None))
+check("neither of ours: SteamVR's carries it", (len(svc.sent), svc.sent[-1][:2] if svc.sent else None), (30, (2.0, 2.0)))
+svc = service()
+feed(svc, steam=False)
+check("SteamVR's eyes missing: ours", svc.sent[-1][:2], (4.0, 2.0))
+svc = service()
+feed(svc, n=60, unc=(0.02, 0.001))
+check("SteamVR lost its left eye: ours", svc.sent[-1][:2], (4.0, 2.0))
+svc = service()
+feed(svc, n=60, opens=(0.0, 0.0))
+check("a blink (both lids down): nothing", (svc.sent, svc.counts["blinks"]), ([], 60))
+svc = service(one=1)
+feed(svc)
+check("SteamVR tracking one eye: no fusion", (svc.sent[-1][:2], svc.counts["fused"]), ((4.0, 2.0), 0))
+svc = service(fusion_on=False)
+check("GAZE_FUSION off: not fusing, SteamVR's eyes not asked for", (svc.fusing, svc.wanted_sources()),
+      (False, "own,mmap1"))
+feed(svc)
+check("...ours alone", svc.sent[-1][:2], (4.0, 2.0))
+svc = service()
+svc.fusion = Fusion()
+check("no fitted fusion: not fusing", svc.fusing, False)
+svc = service()
+svc.tracker = "steam"
+svc.models = {name: gazed.Correction() for name in gazed.SOURCES}
+check("SteamVR's tracker: no fusion", svc.fusing, False)
+
+shutil.rmtree(HOME, ignore_errors=True)
+print("FAILED: " + ", ".join(failures) if failures else "all passed")
+sys.exit(1 if failures else 0)

gaze/test/gaze-lock-test.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+"""Offline test of the fixation lock on the gaze's direction in the room (gazecal.GazeLock) and
+of ft-gazed's blink settle for our tracker (BLINK_SETTLE). Made-up samples only: ft-gazed's
+Service is built without its sockets, so nothing reaches the live gaze service.
+
+  gaze/test/gaze-lock-test.py
+"""
+import importlib.machinery
+import importlib.util
+import math
+import os
+import random
+import sys
+from collections import deque
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(HERE, ".."))
+from gazecal import EyeWeights, Fixation, GazeLock, angles, direction, rotate  # noqa: E402
+
+loader = importlib.machinery.SourceFileLoader("ftgazed", os.path.join(HERE, "..", "ft-gazed"))
+gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader("ftgazed", loader))
+loader.exec_module(gazed)
+
+failures = []
+
+
+def check(what, ok, detail=""):
+    if not ok:
+        failures.append(what)
+        print(f"FAIL {what} {detail}", flush=True)
+
+
+def head(yaw, pitch=0.0, roll=0.0):
+    """A head pose's quaternion [w, x, y, z]: yaw about +Y (left), then pitch about +X (up), then
+    roll about -Z, as a headset's pose matrix would give."""
+    def q(axis, deg):
+        h = math.radians(deg) / 2
+        return [math.cos(h)] + [math.sin(h) * a for a in axis]
+
+    def mul(a, b):
+        w1, x1, y1, z1 = a
+        w2, x2, y2, z2 = b
+        return [w1 * w2 - x1 * x2 - y1 * y2 - z1 * z2, w1 * x2 + x1 * w2 + y1 * z2 - z1 * y2,
+                w1 * y2 - x1 * z2 + y1 * w2 + z1 * x2, w1 * z2 + x1 * y2 - y1 * x2 + z1 * w2]
+    return mul(mul(q((0, 1, 0), yaw), q((1, 0, 0), pitch)), q((0, 0, -1), roll))
+
+
+def room(q, yaw, pitch):
+    """Where a head-relative gaze points in the room, as (yaw, pitch)."""
+    return angles(rotate(q, direction(yaw, pitch)))
+
+
+def head_rel(q, ryaw, rpitch):
+    return angles(rotate(q, direction(ryaw, rpitch), inverse=True))
+
+
+def off(a, b):
+    return math.degrees(math.acos(max(-1.0, min(1.0, sum(x * y for x, y in zip(direction(*a), direction(*b)))))))
+
+
+# --- The conventions: ft-gaze's Direction and Angles, and the quaternion as a pose rotation
+check("straight ahead is -Z", direction(0, 0) == (-0.0, 0.0, -1.0), direction(0, 0))
+check("yaw turns left (-X), pitch looks up", direction(90, 0)[0] < -0.99 and direction(0, 90)[1] > 0.99)
+rng = random.Random(1)
+worst = 0.0
+for _ in range(500):
+    q = head(rng.uniform(-180, 180), rng.uniform(-60, 60), rng.uniform(-30, 30))
+    y, p = rng.uniform(-50, 50), rng.uniform(-40, 40)
+    back = head_rel(q, *room(q, y, p))
+    worst = max(worst, abs(back[0] - y), abs(back[1] - p))
+check("into the room and back", worst < 1e-9, worst)
+check("a head turned 30 deg left, eyes ahead: the room's 30 left", max(map(abs, (room(head(30), 0, 0)[0] - 30,
+                                                                              room(head(30), 0, 0)[1]))) < 1e-9)
+check("a head pitched 10 up, eyes 5 up: the room's 15 up", abs(room(head(0, 10), 0, 5)[1] - 15) < 1e-9)
+
+# --- A look at a spot in the room while the head turns (the eyes turn the other way)
+def turning(speed, lock, poses=True, noise=0.05, seconds=2.0):
+    """The pointer's place in the room (drawn with the newest head, as the helper does) while the
+    eyes hold room yaw 10, pitch 5 and the head turns at `speed` deg/s: the worst miss, and the
+    jumps over 0.5 deg between samples."""
+    rng = random.Random(2)
+    worst, jumps, last = 0.0, 0, None
+    for i in range(int(90 * seconds)):
+        t = i / 90
+        q = head(-20 + speed * t, 3.0, 2.0)
+        hy, hp = head_rel(q, 10.0 + rng.gauss(0, noise), 5.0 + rng.gauss(0, noise))
+        if isinstance(lock, GazeLock):
+            out = lock(hy, hp, t, q if poses else None, q if poses else None)
+        else:
+            out = lock(hy, hp, t, 1.0)
+        at = room(q, *out)
+        if i >= 3:
+            worst = max(worst, off(at, (10.0, 5.0)))
+            jumps += last is not None and off(at, last) > 0.5
+        last = at
+    return worst, jumps
+
+
+w, j = turning(20, GazeLock())
+check("room lock, head turning 20 deg/s: the pointer stays on the spot", w < 0.3 and j == 0, (w, j))
+w, j = turning(60, GazeLock())
+check("room lock, head turning 60 deg/s: still on it", w < 0.3 and j == 0, (w, j))
+w, j = turning(20, Fixation(1.0))
+check("the head-relative lock rides along and snaps back (why the room lock)", w > 0.8 and j >= 10, (w, j))
+w, j = turning(20, GazeLock(), poses=False)
+check("no poses (an older ft-gaze): the head-relative lock", w > 0.8 and j >= 10, (w, j))
+
+# A look elsewhere moves it at once (two samples), and a still head is as before
+lock = GazeLock()
+q = head(15, -5, 1)
+for i in range(30):
+    out = lock(2.0, 1.0, i / 90, q, q)
+check("still: the look", off(out, (2.0, 1.0)) < 1e-6, out)
+lock(20.0, 1.0, 30 / 90, q, q)
+out = lock(20.0, 1.0, 31 / 90, q, q)
+check("a saccade: there after two samples", off(out, (20.0, 1.0)) < 1e-6, out)
+one = lock(2.0, 1.0, 32 / 90, q, q)
+check("one stray sample is held off", off(one, (20.0, 1.0)) < 1e-6, one)
+fix, lock2 = Fixation(1.0), GazeLock()
+same = 0.0
+for i in range(200):
+    y, p = 3 + 0.3 * math.sin(i / 7), -2 + 0.3 * math.cos(i / 5)
+    a, b = fix(y, p, i / 90, 1.0), lock2(y, p, i / 90, q, q)
+    same = max(same, off(a, b))
+check("a still head: as the head-relative lock", same < 0.02, same)
+lock2(3, -2, 3.0, q, q)
+lock2(3, -2, 3.1, None, None)
+check("switching to no poses starts over", lock2.count == 0 and lock2.head.count == 1)
+# Newest pose differs from the sample's: the room direction is kept, re-expressed in the newest head
+lock3 = GazeLock()
+then, now = head(0), head(4)
+out = lock3(10.0, 0.0, 0.0, then, now)
+check("drawn with the newest head", abs(out[0] - 6.0) < 1e-9 and abs(out[1]) < 1e-9, out)
+
+
+# --- ft-gazed's own path: an eye stays out for BLINK_SETTLE after its lids came down
+def service():
+    svc = gazed.Service.__new__(gazed.Service)
+    svc.override, svc.tracker, svc.source, svc.one_eye = None, "own", "mmap1", None
+    svc.counts = dict.fromkeys(("samples", "sent", "blinks", "one_eye", "one_eye_used", "lost_left", "lost_right",
+                                "looking_down", "dropped"), 0)
+    svc.opens, svc.lost = (deque(maxlen=90), deque(maxlen=90)), [False, False]
+    svc.fix, svc.last_sample, svc.low_until = GazeLock(radius=1.0), 0.0, [0.0, 0.0]
+    svc.bridge = gazed.EyeBridge()
+    svc.weights = {"own": EyeWeights()}
+    svc.sent = []
+    svc.send = lambda t, hy, hp, rhy, rhp, eyes: svc.sent.append((t, hy, hp))
+    return svc
+
+
+def blink(settle):
+    """The right eye's lids down for 3 samples, and its pupil found wrong (30, -20) for 3 more
+    after they're up: the gaze sent from 60 on."""
+    saved = gazed.BLINK_SETTLE
+    gazed.BLINK_SETTLE = settle
+    try:
+        svc = service()
+        q = head(0)
+        for i in range(120):
+            closing = 60 <= i < 63
+            right = [30.0, -20.0] if 60 <= i < 66 else [6.0, 2.0]
+            svc.on_eyes_sample({"t": i / 90, "head": {"q": q}, "src": {
+                "mmap1": {"unc": [0.001, 0.001], "open": [0.8, 0.0 if closing else 0.8]},
+                "own": {"hy": 5.0, "hp": 2.0, "hq": q, "eyes": [[4.0, 2.0], right]}}}, True)
+    finally:
+        gazed.BLINK_SETTLE = saved
+    return svc, [s for s in svc.sent if s[0] >= 60 / 90]
+
+
+svc, sent = blink(gazed.BLINK_SETTLE)
+worst = max(off(s[1:], (5.0, 2.0)) for s in sent)
+check("settle: the bad finds after the lids came up don't reach the pointer", worst < 1.1, worst)
+check("settle: then both eyes again", off(sent[-1][1:], (5.0, 2.0)) < 0.2, sent[-1])
+check("settle: counted as one-eye samples", svc.counts["one_eye"] >= 6, svc.counts)
+_, sent = blink(0.0)
+worst = max(off(s[1:], (5.0, 2.0)) for s in sent)
+check("without it, they do (the test's control)", worst > 5, worst)
+
+print("FAILED: " + ", ".join(failures) if failures else "all passed")
+sys.exit(1 if failures else 0)

gaze/test/looked-away-test.py

@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+"""Offline test of the full calibration's looked-away check (gaze/gazecheck.py, Checks.looked_away):
+a dot for our tracker where SteamVR's two eyes were more than LOOKED_AWAY from it, past their
+usual offset in that calibration, isn't taken, and ft-eyes isn't asked. Live 2026-10-18: the
+palm menu opened the Steam menu mid-calibration, and three dots were taken 17-35 degrees from
+where the user looked (both trackers agreed). A SteamVR that is off by itself (its eyes in one
+place whatever the dot) mustn't block the calibration: past AWAY_GIVE_UP dots in a row, the
+check stops.
+
+Drives Checks.capture with synthetic looks and a fake service: nothing reaches the live gaze
+service, the pointer helper, ft-eyes, or SteamVR. HOME is a temp folder (checks.jsonl goes there).
+
+  gaze/test/looked-away-test.py
+"""
+import os
+import tempfile
+
+HOME = tempfile.mkdtemp(prefix="ft-gaze-looked-away-test-")
+os.environ["HOME"] = HOME  # before gazecal: its STATE follows HOME
+
+import selectors  # noqa: E402
+import shutil  # noqa: E402
+import sys  # noqa: E402
+import types  # noqa: E402
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+import gazecheck  # noqa: E402
+
+tag = f"ft_gaze_looked_away_test_{os.getpid()}"
+gazecheck.PANEL = f"\0{tag}_panel"
+gazecheck.POINTER = f"\0{tag}_helper"
+gazecheck.EYES = f"\0{tag}_eyes"
+logs = []
+gazecheck.log = lambda msg: logs.append(msg)
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(f"{'ok  ' if ok else 'FAIL'}  {label}" + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+class Weights:
+    def add(self, miss):
+        pass
+
+
+svc = types.SimpleNamespace(one_eye=None, weights={"own": Weights(), "steam": Weights()})
+checks = gazecheck.Checks(svc, selectors.DefaultSelector())
+asked = []
+checks.ask_eyes = lambda command, timeout, done: asked.append(command)
+DOTS = gazecheck.check_dots("full", True)
+
+
+def new_check():
+    checks.check = {"kind": "full", "own": True, "dots": DOTS, "i": 0, "tries": 0, "skipped": 0, "captured": 0,
+                    "points": {}, "reasons": {}, "fit_reasons": set(), "note": "", "blind": False, "fusion": [],
+                    "away_dots": [], "away_off": False, "steam_offsets": []}
+
+
+def look(at, steam_off=(0.0, 0.0), n=30):
+    """A look at `at` (yaw, pitch) by SteamVR's eyes (each this far from it, plus steam_off), our
+    tracker's at the dot: the capture window as on_sample keeps it."""
+    y, p = at[0] + steam_off[0], at[1] + steam_off[1]
+    src = {"mmap1": {"unc": [0.001, 0.001], "open": [0.8, 0.8]}, "left": {"hy": y - 1.0, "hp": p},
+           "right": {"hy": y + 1.0, "hp": p}, "own": {"ok": 1, "eyes": [list(at), list(at)]}}
+    return [(0.0, {"t": 100.0 + k / 90, "src": src}, y, p) for k in range(n)]
+
+
+def capture(i, at, steam_off=(0.0, 0.0)):
+    """Dot i shown, a click while the eyes are at `at`: was ft-eyes asked (the check let it go on)?"""
+    c = checks.check
+    c["i"], c["done_at"] = i, None
+    before = len(asked)
+    checks.capture(look(at, steam_off))
+    return len(asked) > before
+
+
+def answer_ok():
+    """ft-eyes takes the dot it was asked about (point_done's own path, with "ok")."""
+    c = checks.check
+    yaw, pitch, rnd = c["dots"][c["i"]]
+    rec = {"true": [yaw, pitch], "steam": [(yaw, pitch), (yaw, pitch)], "own": True}
+    checks.point_done(rec, [0.0, 0.0], "ok")
+
+
+# Looking at the dots: each taken, as before.
+new_check()
+for i in range(3):
+    check(f"dot {i + 1}: looking at it, ft-eyes is asked", capture(i, DOTS[i][:2]), True)
+    answer_ok()
+check("no notes for those", checks.check["note"], "")
+
+# Looking 25 degrees off (the Steam menu, mid-calibration): not taken, and the panel says why.
+dot = DOTS[3][:2]
+check("dot 4: looked 25 degrees away: ft-eyes isn't asked", capture(3, dot, (25.0, 0.0)), False)
+check("...the panel says why", "looked away" in checks.check["note"], True)
+check("...and the dot stays for another try", (checks.check["i"], checks.check["tries"]), (3, 1))
+check("...the reason is counted", checks.check["reasons"], {"you looked away from the dot (by SteamVR's eyes)": 1})
+check("dot 4, second try, looking at it: taken", capture(3, dot), True)
+answer_ok()
+
+# 14 degrees off: under LOOKED_AWAY, taken (SteamVR's own error at the corners reached 9 live).
+check("dot 5: 14 degrees off: taken", capture(4, DOTS[4][:2], (0.0, 14.0)), True)
+answer_ok()
+
+# A SteamVR that is off by itself, 20 degrees at every dot: the usual offset counts once it has
+# AWAY_OFFSET_MIN dots, so every dot is taken.
+new_check()
+for i in range(gazecheck.AWAY_OFFSET_MIN):
+    capture(i, DOTS[i][:2], (3.0, 0.0))   # a small offset first: these are taken
+    answer_ok()
+check("its usual offset in this calibration is learned", len(checks.check["steam_offsets"]), gazecheck.AWAY_OFFSET_MIN)
+check("then 18.5 degrees the same way: 15.5 past the usual 3, refused",
+      capture(3, DOTS[3][:2], (18.5, 0.0)), False)
+check("and 14 degrees the same way: taken (11 past the usual 3)", capture(3, DOTS[3][:2], (14.0, 0.0)), True)
+answer_ok()
+
+# A SteamVR off by 20 degrees from the start: the first dots are refused, then the check stops
+# (AWAY_GIVE_UP dots in a row) and the rest are taken.
+new_check()
+logs.clear()
+check("SteamVR 20 degrees off, dot 1: refused", capture(0, DOTS[0][:2], (20.0, 0.0)), False)
+check("dot 1, second try: refused, and skipped", (capture(0, DOTS[0][:2], (20.0, 0.0)), checks.check["skipped"]),
+      (False, 1))
+check("dot 2: refused", capture(1, DOTS[1][:2], (20.0, 0.0)), False)
+check("dot 2 again: refused, and skipped", (capture(1, DOTS[1][:2], (20.0, 0.0)), checks.check["skipped"]), (False, 2))
+check("dot 3: a third dot in a row: SteamVR is off, taken", capture(2, DOTS[2][:2], (20.0, 0.0)), True)
+check("...and the log says the check is off", any("looked-away check is off" in m for m in logs), True)
+answer_ok()
+check("dot 4: taken (the check stays off)", capture(3, DOTS[3][:2], (20.0, 0.0)), True)
+
+# Our tracker alone without SteamVR's eyes (one eye tracked, or SteamVR lost them): no check.
+new_check()
+svc.one_eye = 0
+check("one eye tracked by SteamVR: no check, 30 degrees off is taken", capture(0, DOTS[0][:2], (30.0, 0.0)), True)
+svc.one_eye = None
+
+# The reason's words, for the note and a failed calibration's status.
+check("reject_reason for it", gazecheck.reject_reason("looked away (25 deg by SteamVR)", None),
+      ("looked away", "you looked away from the dot (by SteamVR's eyes)", False))
+
+print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+shutil.rmtree(HOME, ignore_errors=True)
+sys.exit(1 if failures else 0)

gaze/test/mmap-layout-test.cpp

@@ -162,6 +162,40 @@ void NoMapping() {
     CHECK(f.eyes.Counter() == f.n);
 }
 
+// QuatJson (the head poses printed for the gaze service's room lock): for poses turned every
+// way, including past 90 degrees where each of its four branches is taken, the quaternion turns
+// vectors as the pose matrix does.
+void Quaternions() {
+    auto axis = [](Vec3 a, double deg) {  // a rotation matrix about unit axis a
+        vr::HmdMatrix34_t m{};
+        const Vec3 cols[3] = {RotateAbout({1, 0, 0}, a, deg * M_PI / 180), RotateAbout({0, 1, 0}, a, deg * M_PI / 180),
+                              RotateAbout({0, 0, 1}, a, deg * M_PI / 180)};
+        for (int c = 0; c < 3; ++c) m.m[0][c] = cols[c].x, m.m[1][c] = cols[c].y, m.m[2][c] = cols[c].z;
+        return m;
+    };
+    double worst = 0;
+    int n = 0;
+    for (int ax = 0; ax < 7; ++ax) {
+        const Vec3 a = Normalize(Vec3{double(ax & 1), double((ax >> 1) & 1), double((ax >> 2) & 1)} +
+                                 Vec3{0.1 * ax, -0.07 * ax, 0.05});
+        for (double deg = -179; deg <= 180; deg += 23) {
+            const vr::HmdMatrix34_t m = axis(a, deg);
+            double q[4];
+            CHECK(std::sscanf(QuatJson(m).c_str(), "[%lf,%lf,%lf,%lf]", &q[0], &q[1], &q[2], &q[3]) == 4);
+            const Vec3 u{q[1], q[2], q[3]};
+            for (const Vec3 v : {Vec3{0, 0, -1}, Vec3{1, 0, 0}, Vec3{0.3, 0.8, -0.5}}) {
+                const Vec3 c = Cross(u, v);
+                const Vec3 byQ = v + c * (2 * q[0]) + Cross(u, c) * 2;
+                worst = std::max(worst, Length(byQ - Rotate(m, v)));
+            }
+            ++n;
+        }
+    }
+    CHECK(n > 100);
+    CHECK(worst < 1e-5);
+    if (worst >= 1e-5) std::printf("quaternion off by %g\n", worst);
+}
+
 }  // namespace
 
 int main() {
@@ -171,6 +205,7 @@ int main() {
     TornWrite();
     ReadsTheLayoutFound();
     NoMapping();
+    Quaternions();
     if (failures) {
         std::printf("%d failed\n", failures);
         return 1;

gaze/test/one-eye-calibration-test.py

@@ -0,0 +1,254 @@
+#!/usr/bin/env python3
+"""Offline test of our tracker with one eye lost (2026-10-18): a calibration dot is taken with
+the eye that was seen when the other wasn't (ft-eyes calib-point), each eye is fitted on its
+own dots and an eye seen at too few is left out (calib-fit, eyes_model.Calibration.fit), and
+ft-gazed eases the step when an eye drops out (gazecal.EyeBridge). A second person's eyes each
+lost the pupil looking 20 degrees outward, so until now their outer dots failed, and the rest
+fitted both eyes from the middle only.
+
+Pupil positions are written into ft-eyes' eye histories as its frames would leave them: no
+cameras, SteamVR or live gaze service. Needs the tracker's Python (numpy, OpenCV).
+
+  scripts/in-box /opt/frametop/venv/bin/python gaze/test/one-eye-calibration-test.py
+"""
+import importlib.machinery
+import importlib.util
+import json
+import math
+import os
+import shutil
+import sys
+import tempfile
+from pathlib import Path
+
+HOME = tempfile.mkdtemp(prefix="ft-eyes-one-eye-test-")
+os.environ.update(HOME=HOME, FT_EYES_CAMS="/nonexistent", FT_EYES_STATE=os.path.join(HOME, "eyes"),
+                  FT_EYES_MODEL="classic")
+GAZE = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(GAZE))
+sys.path.insert(0, str(GAZE / "tracker"))
+
+import gazecheck  # noqa: E402
+from gazecal import EyeBridge, EyeWeights, GazeLock  # noqa: E402
+
+loader = importlib.machinery.SourceFileLoader("fteyes", str(GAZE / "tracker" / "ft-eyes"))
+fe = importlib.util.module_from_spec(importlib.util.spec_from_loader(loader.name, loader))
+loader.exec_module(fe)
+loader = importlib.machinery.SourceFileLoader("ftgazed", str(GAZE / "ft-gazed"))
+gazed = importlib.util.module_from_spec(importlib.util.spec_from_loader(loader.name, loader))
+loader.exec_module(gazed)
+
+failures = []
+
+
+def check(what, got, want):
+    ok = got == want
+    print(f"{'ok  ' if ok else 'FAIL'}  {what}" + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(what)
+
+
+DOTS = [d[:2] for d in gazecheck.check_dots("full", True)]   # 27 dots, out to +-20
+CENTRE = {0: (300.0, 200.0), 1: (330.0, 205.0)}               # right, left: pupil looking ahead
+
+
+def pupil(eye, yaw, pitch):
+    """Where an eye's pupil sits for a gaze: 4 px a degree, yaw +left moves it to lower x."""
+    x0, y0 = CENTRE[eye]
+    return x0 - 4.0 * yaw + 0.02 * yaw * pitch, y0 - 4.0 * pitch
+
+
+class Clock:
+    t = 100.0
+
+
+def look(tracker, yaw, pitch, seen=(0, 1), jitter=()):
+    """Half a second at a dot: 45 frames per eye that's seen (jitter: those that jump about),
+    then calib-point over that window."""
+    t0 = Clock.t
+    for i in range(45):
+        t = t0 + i / 90
+        for e in tracker.eyes:
+            if e.eye not in seen:
+                continue
+            x, y = pupil(e.eye, yaw, pitch)
+            if e.eye in jitter:
+                x += 0.5 * (i - 22)   # drifting: half its frames over 5.5 px from the middle
+            e.history.append((t, x, y, math.nan, math.nan))
+    Clock.t = t0 + 1.0
+    return tracker.command(f"calib-point {t0:.6f} {t0 + 0.5:.6f} {yaw:.4f} {pitch:.4f}")
+
+
+def outward_lost(yaw):
+    """The second person: the right eye lost looking 14+ degrees right (yaw -), the left 14+ left."""
+    return tuple(e for e in (0, 1) if not (e == 0 and yaw <= -14 or e == 1 and yaw >= 14))
+
+
+# --- calib-point: one eye is enough for a dot ---
+tracker = fe.Tracker()
+check("calib-start", tracker.command("calib-start"), "ok")
+r = look(tracker, 0.0, 0.0)
+check("both eyes seen: taken", r.split()[:3], ["ok", "45", "45"])
+check("...with no note", "only" in r, False)
+r = look(tracker, -20.0, 0.0, seen=(1,))
+check("right eye lost: taken with the left", r.startswith("ok 0 45"), True)
+check("...and says so", r.endswith("(left eye only: the right eye was seen in only 0 frames)"), True)
+check("...the dot holds the left eye alone", sorted(tracker.dots[-1]["eye"]), [1])
+r = look(tracker, 10.0, 0.0, jitter=(1,))
+check("left eye jumping about: taken with the right", "(right eye only: the left eye moved" in r, True)
+check("...the dot holds the right eye alone", sorted(tracker.dots[-1]["eye"]), [0])
+n = len(tracker.dots)
+r = look(tracker, 0.0, 10.0, seen=())
+check("neither eye seen: not taken", r, "fail neither eye was seen enough (right 0 frames, left 0 frames)")
+r = look(tracker, 0.0, 10.0, jitter=(0, 1))
+check("both moved: not taken, as before", r, "fail the right eye moved (5.5 px)")
+r = look(tracker, 0.0, 10.0, seen=(1,), jitter=(1,))
+check("one lost and the other moved: the move is the reason", r, "fail the left eye moved (5.5 px)")
+check("...none of those kept", len(tracker.dots), n)
+
+# The panel's words for them (gazecheck.reject_reason).
+check("reject_reason: neither eye", gazecheck.reject_reason(
+    "fail neither eye was seen enough (right 3 frames, left 0 frames)", None),
+    ("eyes not seen", "our tracker saw neither eye well enough", True))
+check("reject_reason: one eye moved, as before", gazecheck.reject_reason("fail the left eye moved (12.0 px)", None),
+      ("left eye moved", "your left eye moved while you looked", False))
+
+# --- calib-fit: a full calibration where each eye is lost looking outward ---
+tracker = fe.Tracker()
+tracker.command("calib-start")
+for yaw, pitch in DOTS:
+    look(tracker, yaw, pitch, seen=outward_lost(yaw))
+seen = {c: sum(1 for k in tracker.dots if c in k["eye"]) for c in (0, 1)}
+check("all 27 dots taken", len(tracker.dots), 27)
+check("each eye at the dots it was seen at", seen, {0: 22, 1: 22})
+r = tracker.command("calib-fit")
+check("fitted", r.startswith("ok 27 dots, fit median 0.0"), True)
+check("...each eye's dots in the reply", r.endswith("(22 dots)") and "(22 dots), left 0.0" in r, True)
+check("both eyes mapped", tracker.status()["mapped"], ["right", "left"])
+check("...the dots per eye saved with it", tracker.cal.info["eye_dots"], {"right": 22, "left": 22})
+right, left = tracker.eyes
+check("each eye calibrated", (right.cal is not None, left.cal is not None), (True, True))
+g = tracker.cal.gaze(1, *pupil(1, -20.0, 0.0))
+check("the left eye at 20 right (only it saw that dot): on it", (round(g[0], 1), round(g[1], 1)), (-20.0, 0.0))
+rows = [json.loads(x) for x in open(fe.CALIBRATION.with_name("calibration-dots.jsonl"))]
+check("calibration-dots.jsonl keeps the one-eye dots", sorted(len(x["eye"]) for x in rows)[:4], [1, 1, 1, 1])
+check("...and its spreads come back from them", sorted(fe.spread_from_dots(tracker.cal)), [0, 1])
+loaded = fe.eyes_model.Calibration.load(fe.CALIBRATION)
+check("the saved calibration maps both", loaded.eyes(), [0, 1])
+
+# --- calib-fit with one eye seen at too few dots: the other alone ---
+tracker = fe.Tracker()
+tracker.command("calib-start")
+for i, (yaw, pitch) in enumerate(DOTS):
+    look(tracker, yaw, pitch, seen=(1,) if i % 3 else (0, 1))   # the right eye at 9 of 27
+r = tracker.command("calib-fit")
+check("right eye at 9 dots: fitted with the left alone", r.endswith("(27 dots)")
+      and "right not used (9 dots, needs 12)" in r, True)
+check("...the left mapped only", tracker.status()["mapped"], ["left"])
+right, left = tracker.eyes
+check("...the right eye gives no gaze", (right.cal, left.cal is not None), (None, True))
+check("...a click teaches the left only", tracker.command(f"click {Clock.t:.6f} 0 0").split()[1:3], ["nan", "nan"])
+fresh = fe.Tracker()
+check("...and a restart says so", (fresh.cal.eyes(), fresh.eyes[0].cal), ([1], None))
+
+# --- calib-fit with too few dots for either eye ---
+tracker = fe.Tracker()
+tracker.command("calib-start")
+for i, (yaw, pitch) in enumerate(DOTS[:14]):
+    look(tracker, yaw, pitch, seen=(0,) if i % 2 else (1,))
+r = tracker.command("calib-fit")
+check("7 dots each: no calibration", r, "fail neither eye was seen at enough dots (right 7, left 7; need 12)")
+check("...the last one stays", fe.eyes_model.Calibration.load(fe.CALIBRATION).eyes(), [1])
+
+# --- gazecal.EyeBridge: no jump when an eye drops out ---
+b = EyeBridge()
+for i in range(90):   # both eyes, a degree apart: their mean is the gaze
+    out = b(i / 90, [(4.0, 2.0), (6.0, 2.0)], (5.0, 2.0))
+check("both eyes: the combined gaze", out, (5.0, 2.0))
+t = 1.0
+out = b(t, [None, (6.0, 2.0)], (6.0, 2.0))
+check("the left drops out: the gaze stays put", (abs(out[0] - 5.0) < 0.05, out[1]), (True, 2.0))
+out = b(t + 0.3, [None, (6.0, 2.0)], (6.0, 2.0))
+check("...eases toward the right eye", 5.5 < out[0] < 5.8, True)
+out = b(t + 2.0, [None, (6.0, 2.0)], (6.0, 2.0))
+check("...and is the right eye's after 2 s", round(out[0], 2), 6.0)
+out = b(t + 2.1, [(4.0, 2.0), (6.0, 2.0)], (5.0, 2.0))
+check("both back: the combined gaze", out, (5.0, 2.0))
+check("no eyes learned yet: as it comes", EyeBridge()(0.0, [None, (6.0, 2.0)], (6.0, 2.0)), (6.0, 2.0))
+
+
+# --- ft-gazed's own path uses it ---
+def service():
+    svc = gazed.Service.__new__(gazed.Service)
+    svc.override, svc.tracker, svc.source, svc.one_eye = None, "own", "mmap1", None
+    svc.models = {name: gazed.Correction() for name in gazed.SOURCES}
+    svc.counts = dict.fromkeys(("samples", "sent", "blinks", "one_eye", "one_eye_used", "lost_left", "lost_right",
+                                "looking_down", "dropped"), 0)
+    from collections import deque
+    svc.opens, svc.vergence = (deque(maxlen=90), deque(maxlen=90)), deque(maxlen=90)
+    svc.lost, svc.bad_at, svc.fallback = [False, False], [0.0, 0.0], gazed.EyeFallback()
+    svc.fix, svc.last_sample, svc.low_until = GazeLock(radius=1.0), 0.0, [0.0, 0.0]
+    svc.bridge = EyeBridge()
+    svc.fix = lambda gy, gp, t, then, q: (gy, gp)   # what goes into the lock
+    svc.weights, svc.fusion_on, svc.fusion = {"own": EyeWeights()}, False, None
+    svc.sent = []
+    svc.send = lambda t, hy, hp, rhy, rhp, eyes: svc.sent.append((round(hy, 2), round(hp, 2)))
+    return svc
+
+
+svc = service()
+for i in range(90):
+    svc.on_eyes_sample({"t": i / 90, "src": {"mmap1": {"unc": [0.001, 0.001], "open": [0.8, 0.8]},
+                                             "own": {"hy": 5.0, "hp": 2.0, "eyes": [[4.0, 2.0], [6.0, 2.0]]}}}, True)
+svc.on_eyes_sample({"t": 1.0, "src": {"mmap1": {"unc": [0.001, 0.001], "open": [0.8, 0.8]},
+                                      "own": {"hy": 6.0, "hp": 2.0, "eyes": [None, [6.0, 2.0]]}}}, True)
+check("ft-gazed, our tracker: both eyes, then the left lost: no jump",
+      (svc.sent[-2], abs(svc.sent[-1][0] - 5.0) < 0.05), ((5.0, 2.0), True))
+check("...counted as one eye", svc.counts["one_eye"], 1)
+
+
+# --- a new calibration starts the eye weights over (ft-gazed's note_own_calibration) ---
+logs = []
+gazed.log = lambda msg: logs.append(msg)
+svc = service()
+svc.bias, svc.own_made, svc.dirty = "auto", None, False
+svc.own = {"calibration": {"made": "2026-10-18 20:58:26", "dots": 27}}
+svc.note_own_calibration()
+check("the first calibration it hears of: adopted, nothing dropped", (svc.own_made, logs), ("2026-10-18 20:58:26", []))
+for _ in range(6):
+    svc.weights["own"].add([2.0, 0.5])   # the left eye's mapping was the worse one
+before = svc.weights["own"].weights()
+check("misses on that calibration lean toward the right eye", before[1] > 0.9, True)
+svc.note_own_calibration()
+check("the same calibration again: kept", svc.weights["own"].weights(), before)
+svc.own = {"calibration": {"made": "2026-10-18 21:11:33", "dots": 26}}
+svc.note_own_calibration()
+check("a new calibration: the misses start over", svc.weights["own"].misses, [[], []])
+check("...the eyes count alike until each has MIN again", tuple(svc.weights["own"].weights()), (0.5, 0.5))
+check("...said in the log", sum("eye weights start over" in m for m in logs), 1)
+check("...and saved with the stamp", (svc.own_made, svc.dirty), ("2026-10-18 21:11:33", True))
+svc.own = {}
+svc.note_own_calibration()
+check("a status without a calibration changes nothing", svc.own_made, "2026-10-18 21:11:33")
+
+# The stamp lives with the misses, so a restart keeps them and a calibration made meanwhile doesn't.
+svc.lives = {name: gazed.PointerLessons() for name in gazed.SOURCES}
+svc.weights["own"].add([1.0, 1.0])
+gazed.STATE.mkdir(parents=True, exist_ok=True)
+svc.save_lessons()
+again = service()
+again.bias, again.lives = "auto", {name: gazed.PointerLessons() for name in gazed.SOURCES}
+again.source = "mmap1"
+again.load_lessons()
+check("after a restart: the stamp and the misses", (again.own_made, again.weights["own"].misses),
+      ("2026-10-18 21:11:33", [[1.0], [1.0]]))
+again.own = {"calibration": {"made": "2026-10-18 21:11:33"}}
+again.note_own_calibration()
+check("...the same calibration: kept", again.weights["own"].misses, [[1.0], [1.0]])
+again.own = {"calibration": {"made": "2026-10-18 21:30:00"}}
+again.note_own_calibration()
+check("...one made while it was down: started over", again.weights["own"].misses, [[], []])
+
+shutil.rmtree(HOME, ignore_errors=True)
+print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+sys.exit(1 if failures else 0)

gaze/test/one-eye-test.py

@@ -21,7 +21,7 @@ from gazecal import (  # noqa: E402
     Correction,
     EyeFallback,
     EyeWeights,
-    Fixation,
+    GazeLock,
     TrackedEye,
     steady_samples,
     tracked_eye,
@@ -112,7 +112,8 @@ def service(one, source="mmap1"):
                                 "looking_down", "dropped"), 0)
     svc.opens, svc.vergence = (deque(maxlen=90), deque(maxlen=90)), deque(maxlen=90)
     svc.lost, svc.bad_at, svc.fallback = [False, False], [0.0, 0.0], EyeFallback()
-    svc.fix, svc.last_sample = Fixation(radius=1.0), 0.0
+    svc.fix, svc.last_sample, svc.low_until = GazeLock(radius=1.0), 0.0, [0.0, 0.0]
+    svc.bridge = gazed.EyeBridge()
     svc.correction = lambda name, hy, hp: (0.0, 0.0)
     svc.sent = []
     svc.send = lambda t, hy, hp, rhy, rhp, eyes: svc.sent.append((hy, hp))

gaze/test/size-guard-test.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Offline test of ft-eyes' size guard (eyes_pupil.SizeGuard, EYES_SIZE_GUARD): synthetic frames
+fed to its eyes as the live loop does. A find two thirds of the pupil's usual size in one eye,
+while the other's is as usual, counts as not found: no gaze from it, the search stays on the
+last pupil, the finds log marks it "guard". With the guard off, it's taken. Needs the
+tracker's Python (numpy, OpenCV); no cameras or SteamVR.
+
+  scripts/in-box /opt/frametop/venv/bin/python gaze/test/size-guard-test.py
+"""
+import importlib.machinery
+import importlib.util
+import json
+import os
+import sys
+import tempfile
+from pathlib import Path
+
+import cv2
+import numpy as np
+
+HOME = tempfile.mkdtemp(prefix="ft-eyes-size-guard-test-")
+TRACKER = Path(__file__).resolve().parents[1] / "tracker"
+
+
+def load(rule):
+    os.environ.update(HOME=HOME, FT_EYES_CAMS="/nonexistent", FT_EYES_STATE=os.path.join(HOME, "eyes"),
+                      FT_EYES_MODEL="classic", FT_EYES_SIZE_GUARD=rule)
+    loader = importlib.machinery.SourceFileLoader(f"fteyes_{rule}", str(TRACKER / "ft-eyes"))
+    mod = importlib.util.module_from_spec(importlib.util.spec_from_loader(loader.name, loader))
+    loader.exec_module(mod)
+    return mod
+
+
+def frame(a, b):
+    img = np.full((400, 512), 100, np.uint8)
+    cv2.ellipse(img, (256, 200), (a, b), 0, 0, 360, 0, -1)
+    return img
+
+
+failed = 0
+
+
+def check(ok, what):
+    global failed
+    print(("ok   " if ok else "FAIL ") + what)
+    failed += not ok
+
+
+for rule, refused in (("both", True), ("off", False)):
+    fe = load(rule)
+    tracker = fe.Tracker()
+    log = os.path.join(HOME, f"finds-{rule}.jsonl")
+    tracker.command(f"finds start {log}")
+    right, left = tracker.eyes
+    t = 100.0
+    for i in range(40):           # both pupils as usual
+        right.feed(t, frame(30, 24))
+        left.feed(t + 0.001, frame(30, 24))
+        t += 1 / 90
+    last = right.last
+    left.feed(t, frame(30, 24))
+    right.feed(t + 0.001, frame(20, 15))   # 0.67 of usual, the left eye as usual
+    tracker.command("finds stop")
+    rows = [json.loads(x) for x in open(log)]
+    check(tracker.status()["size_guard"] == rule, f"{rule}: status says the rule")
+    check((right.refused == 1) == refused, f"{rule}: the small find refused: {right.refused}")
+    check((right.last is last) == refused, f"{rule}: the search stays on the last pupil")
+    check(bool(rows[-1].get("guard")) == refused, f"{rule}: the finds log marks it: {rows[-1]}")
+    check(abs(rows[-1]["a"] - 20) < 1.5, f"{rule}: ...with its size")
+    check(left.refused == 0 and sum(1 for r in rows if r.get("guard")) == (1 if refused else 0),
+          f"{rule}: nothing else refused")
+    counts = tracker.status()["size_guard_counts"]
+    got = {e: (c["kept"] >= 30, c["refused"], c["restarts"]) for e, c in counts.items()}
+    want = {"right": (True, 1, 0), "left": (True, 0, 0)} if refused else \
+        {e: (False, 0, 0) for e in ("right", "left")}   # the rule off: nothing judged
+    check(got == want, f"{rule}: status counts the guard's finds: {counts}")
+    zero = {e: {"kept": 0, "refused": 0, "restarts": 0} for e in (0, 1)}
+    _, line = fe.guard_line(tracker.guard, zero)
+    check(("right refused 1 of " in line and "left refused 0 of " in line and "restarts" not in line) if refused
+          else line == "", f"{rule}: its minute's log line: {line!r}")
+    if refused:
+        # The right pupil really smaller (one eye's light) for over GUARD_RESET: its usual size
+        # starts over, once, and that's counted.
+        was = {e: tracker.guard.counts(e) for e in (0, 1)}
+        t += 0.01
+        for i in range(int(fe.eyes_pupil.GUARD_RESET * 90) + 30):
+            left.feed(t, frame(30, 24))
+            right.feed(t + 0.001, frame(20, 15))
+            t += 1 / 90
+        check(tracker.guard.counts(1)["restarts"] == 0 and tracker.guard.counts(0)["restarts"] == 1,
+              f"{rule}: a pupil that stayed smaller: one restart: {tracker.status()['size_guard_counts']}")
+        _, line = fe.guard_line(tracker.guard, was)
+        check(", 1 restarts" in line, f"{rule}: ...in the log line: {line!r}")
+
+print(f"{'FAILED' if failed else 'ok'}: size guard")
+sys.exit(1 if failed else 0)

gaze/test/test_eyes_pupil.py

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""Offline pupil detection: shared preprocessing preserves both closing attempts.
+
+Synthetic images exercise normal pupils, glints, the small-kernel retry, lost eyes,
+and windowed searches. No camera, calibration, or SteamVR connection is needed.
+"""
+import sys
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import cv2
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tracker"))
+import eyes_pupil as pupil
+
+
+def frame(retry=False):
+    img = np.full((400, 512), 100, np.uint8)
+    centre = (44, 200) if retry else (256, 200)
+    if retry:
+        img[:, :11] = 0  # large closing joins the pupil to this edge region
+    cv2.ellipse(img, centre, (30, 24), 0, 0, 360, 0, -1)
+    return img
+
+
+def separate_attempts(img, ox, oy):
+    """The old preprocessing order: each attempt gets a newly calculated mask."""
+    for close in (pupil.CLOSE, pupil.CLOSE_TIGHT):
+        dark = (cv2.GaussianBlur(img, (5, 5), 0) < pupil.DARK).astype(np.uint8)
+        p = pupil._find(img, dark, ox, oy, close)
+        if p is not None:
+            return p
+    return None
+
+
+class PupilTest(unittest.TestCase):
+    def test_first_attempt(self):
+        img = frame()
+        with patch.object(pupil.cv2, "GaussianBlur", wraps=cv2.GaussianBlur) as blur, \
+                patch.object(pupil, "_find", wraps=pupil._find) as find:
+            p = pupil.find_pupil(img)
+        self.assertEqual((blur.call_count, find.call_count), (1, 1))
+        self.assertAlmostEqual(p["x"], 256, delta=0.1)
+        self.assertAlmostEqual(p["y"], 200, delta=0.1)
+
+    def test_retry_preserves_mask_and_kernels(self):
+        img = frame(retry=True)
+        dark = (cv2.GaussianBlur(img, (5, 5), 0) < pupil.DARK).astype(np.uint8)
+        self.assertIsNone(pupil._find(img, dark, 0, 0))
+        expected = pupil._find(img, dark, 0, 0, pupil.CLOSE_TIGHT)
+        self.assertIsNotNone(expected)
+        kernels = {k: v.copy() for k, v in pupil._KERNELS.items()}
+        with patch.object(pupil.cv2, "GaussianBlur", wraps=cv2.GaussianBlur) as blur, \
+                patch.object(pupil, "_find", wraps=pupil._find) as find:
+            self.assertEqual(pupil._find_either(img, 0, 0), expected)
+        self.assertEqual((blur.call_count, find.call_count), (1, 2))
+        first, second = (call.args[1] for call in find.call_args_list)
+        self.assertIs(first, second)
+        np.testing.assert_array_equal(first, dark)
+        for k, v in kernels.items():
+            np.testing.assert_array_equal(pupil._KERNELS[k], v)
+
+    def test_exact_results_for_full_and_cropped_images(self):
+        rng = np.random.default_rng(512400)
+        images = [np.zeros((400, 512), np.uint8), np.full((400, 512), 255, np.uint8),
+                  frame(), frame(retry=True)]
+        glints = frame()
+        for y in (194, 207):
+            cv2.circle(glints, (256, y), 2, 255, -1)
+        images.append(glints)
+        for _ in range(12):
+            img = rng.integers(35, 180, (400, 512), np.uint8)
+            cv2.ellipse(img, (int(rng.integers(30, 480)), int(rng.integers(25, 375))),
+                        (int(rng.integers(10, 45)), int(rng.integers(8, 25))),
+                        int(rng.integers(0, 180)), 0, 360, 0, -1)
+            images.append(img)
+        for i, img in enumerate(images):
+            for x, y in ((0, 0), (11, 15), (200, 120)):
+                roi = img if (x, y) == (0, 0) else img[y:y + 180, x:x + 200]
+                before = roi.copy()
+                with self.subTest(image=i, offset=(x, y)):
+                    self.assertEqual(pupil._find_either(roi, x, y), separate_attempts(roi, x, y))
+                    np.testing.assert_array_equal(roi, before)
+
+    def test_window_hint_and_full_frame_fallback(self):
+        img = frame()
+        expected = pupil.find_pupil(img)
+        self.assertEqual(pupil.find_pupil(img, expected), expected)
+        self.assertEqual(pupil.find_pupil(img, {"x": 50, "y": 50, "a": 5}), expected)
+        self.assertEqual(pupil.find_pupil(np.zeros_like(img), expected), None)
+
+    def test_a_much_smaller_spot_isnt_the_pupil(self):
+        """The right camera's dark spot of 2026-10-17: darker and smaller than the pupil, so it
+        wins on score unless the pupil's recent size rules it out."""
+        img = np.full((400, 512), 100, np.uint8)
+        cv2.ellipse(img, (256, 200), (30, 24), 0, 0, 360, 20, -1)
+        cv2.ellipse(img, (215, 250), (10, 7), 0, 0, 360, 0, -1)
+        self.assertAlmostEqual(pupil.find_pupil(img)["x"], 215, delta=0.5)
+        p = pupil.find_pupil(img, smallest=15)
+        self.assertAlmostEqual(p["x"], 256, delta=0.5)
+        self.assertAlmostEqual(pupil.find_pupil(img, p, 15)["x"], 256, delta=0.5)
+        self.assertIsNone(pupil.find_pupil(img, p, 40))
+
+    def test_pupil_size(self):
+        size = pupil.PupilSize()
+        for i in range(pupil.SIZE_FINDS - 1):
+            size.add(i / 90, {"a": 20.0})
+        self.assertEqual(size.smallest(9 / 90), 0.0)  # too few finds yet
+        size.add(9 / 90, {"a": 24.0})
+        size.add(10 / 90, None)  # lost frames don't count
+        self.assertAlmostEqual(size.smallest(10 / 90), pupil.SMALLEST * 20.0)
+        self.assertEqual(size.smallest(10 / 90 + pupil.SIZE_WINDOW + 0.2), 0.0)  # all of them too old
+
+
+def warm(guard, t=0.0, a=(20.0, 20.0)):
+    """Both eyes' usual sizes: GUARD_FINDS finds each, 90 Hz, from t. Returns the next t."""
+    for i in range(pupil.GUARD_FINDS):
+        for e in (0, 1):
+            assert guard.keep(e, t + i / 90, a[e])
+    return t + pupil.GUARD_FINDS / 90
+
+
+class SizeGuardTest(unittest.TestCase):
+    def test_a_misfind_in_one_eye(self):
+        g = pupil.SizeGuard("both")
+        t = warm(g)
+        self.assertTrue(g.keep(1, t, 19.0))
+        self.assertFalse(g.keep(0, t + 0.001, 13.0))   # 0.65 of usual, the other eye at 0.95
+        self.assertTrue(g.keep(0, t + 1 / 90, 19.5))
+        self.assertEqual(g.refused, {0: 1, 1: 0})
+
+    def test_light_shrinks_both(self):
+        """Both pupils down to under half their usual size: kept with "both", refused with "long"."""
+        for rule, kept in (("both", True), ("long", False)):
+            g = pupil.SizeGuard(rule)
+            t = warm(g)
+            ok = []
+            for i in range(20):   # narrowing over 20 frames, both eyes alike
+                a = 20.0 * (1 - 0.03 * i)
+                ok += [g.keep(0, t, a), g.keep(1, t + 0.001, a)]
+                t += 1 / 90
+            with self.subTest(rule=rule):
+                self.assertEqual(all(ok), kept)
+
+    def test_other_eye_not_seen(self):
+        g = pupil.SizeGuard("both")
+        t = warm(g) + 1.0   # the other eye's last find is old
+        self.assertTrue(g.keep(0, t, 13.0))    # 0.65: over GUARD_ALONE
+        self.assertFalse(g.keep(0, t + 0.01, 11.0))
+
+    def test_refusals_going_on_start_over(self):
+        g = pupil.SizeGuard("long")
+        t = warm(g)
+        ok = [g.keep(0, t + i / 90, 10.0) for i in range(int(pupil.GUARD_RESET * 90) + 5)]
+        self.assertFalse(ok[0])
+        self.assertTrue(ok[-1])        # taken again after GUARD_RESET s
+        self.assertTrue(g.keep(0, t + 2.0, 10.0))
+
+    def test_off_and_unknown(self):
+        self.assertTrue(pupil.SizeGuard("off").keep(0, 0.0, 1.0))
+        self.assertEqual(pupil.SizeGuard("nonsense").rule, "both")
+
+
+if __name__ == "__main__":
+    unittest.main()

gaze/tracker/build.sh

@@ -13,6 +13,7 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
 "$root/scripts/sync.sh" >/dev/null
 exec "$root/scripts/frame.sh" -C gaze/tracker 'set -e; mkdir -p build
 gcc -std=gnu11 -O2 -Wall -Wextra -pthread -o build/ft-eyegrab ft-eyegrab.c
+strip --strip-unneeded build/ft-eyegrab
 max=$(objdump -T build/ft-eyegrab | grep -oE "GLIBC_[0-9.]+" | sort -uV | tail -1)
 echo "built build/ft-eyegrab, newest glibc symbol: $max"
 [ "$(printf "%s\n" "$max" GLIBC_2.39 | sort -V | tail -1)" = GLIBC_2.39 ] || { echo "needs newer glibc than the host has" >&2; exit 1; }

gaze/tracker/eyes_model.py

@@ -75,13 +75,26 @@ def pair_mid(pair):
     return ((pair[0][0] + pair[1][0]) / 2, (pair[0][1] + pair[1][1]) / 2)
 
 
+def eye_of(click, eye):
+    """A click's features for one eye, or None: `eye` is a list by eye (the lab tools) or a dict
+    of the eyes seen (ft-eyes' calibration dots, which may hold one)."""
+    e = click["eye"]
+    return e.get(eye) if isinstance(e, dict) else e[eye]
+
+
 class Calibration:
     """The three fits per eye. Build from clicks (ft-eyes-score's features) or load from JSON.
 
     `spread` is each eye's RMS miss (degrees) on the clicks its pupil fit was made from.
     `combine` weights the eyes by its inverse square: on practice2 (one headset position,
     leave-one-out) that gave 0.75 median against 0.84 for the plain average, since one eye
-    is usually much better than the other (left 0.73, right 1.32 there)."""
+    is usually much better than the other (left 0.73, right 1.32 there).
+
+    Each eye is fitted on the clicks it was seen in, so a click may hold one eye only, and an
+    eye seen in under MIN_CLICKS is left out (the calibration then maps the other alone). A
+    second person's eyes (2026-10-17 rounds) each lost the pupil looking 20 degrees outward:
+    with only the clicks that saw both, each eye's fit had none out there and guessed (frame
+    errors 8.1 degrees median, 84 p90, held-out takes); each on its own clicks, 2.3 and 5.0."""
 
     def __init__(self, fits=None, info=None, spread=None):
         self.fits = fits or {}   # (name, eye) -> Quad
@@ -93,21 +106,26 @@ class Calibration:
         truth = lambda cs: np.array([k["truth"] for k in cs])  # noqa: E731
         fits, spread = {}, {}
         for c in (0, 1):
-            cs = [k for k in clicks if k["eye"][c] is not None]
+            cs = [k for k in clicks if eye_of(k, c) is not None]
             if len(cs) < MIN_CLICKS:
                 continue
-            fits["pupil", c] = Quad([k["eye"][c]["pupil"] for k in cs], truth(cs))
-            miss = fits["pupil", c]([k["eye"][c]["pupil"] for k in cs]) - truth(cs)
+            pupils = [eye_of(k, c)["pupil"] for k in cs]
+            fits["pupil", c] = Quad(pupils, truth(cs))
+            miss = fits["pupil", c](pupils) - truth(cs)
             spread[c] = float(np.sqrt(np.mean(np.sum(miss ** 2, axis=1))))
-            fits["where", c] = Quad(truth(cs), [k["eye"][c]["pupil"] for k in cs])
-            gs = [k for k in cs if k["eye"][c]["mid"] is not None]
+            fits["where", c] = Quad(truth(cs), pupils)
+            gs = [k for k in cs if eye_of(k, c)["mid"] is not None]
             if len(gs) >= MIN_CLICKS:
-                fits["glint", c] = Quad([k["eye"][c]["pupil"] - k["eye"][c]["mid"] for k in gs], truth(gs))
+                fits["glint", c] = Quad([eye_of(k, c)["pupil"] - eye_of(k, c)["mid"] for k in gs], truth(gs))
         return cls(fits, info, spread)
 
     def has(self, name, eye):
         return (name, eye) in self.fits
 
+    def eyes(self):
+        """The eyes it maps (0 right, 1 left)."""
+        return [c for c in (0, 1) if self.has("pupil", c)]
+
     def slip(self, eye, rows):
         """Median slip in pixels from pair sightings `rows` (t, px, py, mx, my), or None."""
         if not self.has("glint", eye) or len(rows) < SLIP_MIN:

gaze/tracker/eyes_pupil.py

@@ -5,7 +5,14 @@ background are just as dark, but they touch the image edge, so any dark region t
 the edge is dropped. Closing the glints' holes can join the pupil to that background (the
 left camera's, when you look more than about 20 degrees left), so when nothing is found
 the search runs again with a smaller closing.
+
+A dark spot well inside the frame can pass all that too: on 2026-10-17 the right camera took
+one about 9 px across for the pupil (23 px) about once a second, for 3-7 frames at a time, and
+the gaze jumped 60 degrees. A pupil doesn't shrink to half its size in a frame, so a find that
+much smaller than the last half second's (PupilSize) is skipped.
 """
+from collections import deque
+
 import cv2
 import numpy as np
 
@@ -22,42 +29,44 @@ MAX_ASPECT = 3.0   # long / short axis; the steep camera sees a squashed pupil
 GLINT = 200        # glints are near-saturated spots on or by the pupil
 CLOSE = 7          # px: closes the glints' holes in the pupil (the right eye's need this much)
 CLOSE_TIGHT = 3    # px: the retry, keeps a pupil near the dark background apart from it
+_KERNELS = {size: np.ones((size, size), np.uint8) for size in {3, CLOSE, CLOSE_TIGHT}}
 
 
 NEAR = 70          # the windowed search: this many pixels, or 3 pupil radii, around a hint
 
 
-def find_pupil(frame, near=None):
+def find_pupil(frame, near=None, smallest=0.0):
     """Return dict(x, y, a, b, angle, area, fill, glints) or None.
 
     `near` (a previous result) searches a window around it first (0.4 ms, not 1.4-2.1);
-    if the pupil isn't wholly inside the window it falls back to the whole frame."""
+    if the pupil isn't wholly inside the window it falls back to the whole frame. Blobs whose
+    long semi-axis is under `smallest` px don't count (PupilSize.smallest)."""
     if near is not None:
         r = int(max(NEAR, 3 * near['a']))
         x0, y0 = max(int(near['x']) - r, 0), max(int(near['y']) - r, 0)
-        p = _find_either(frame[y0:int(near['y']) + r, x0:int(near['x']) + r], x0, y0)
+        p = _find_either(frame[y0:int(near['y']) + r, x0:int(near['x']) + r], x0, y0, smallest)
         if p is not None:
             p['glints'] = find_glints(frame, p)
             return p
-    p = _find_either(frame, 0, 0)
+    p = _find_either(frame, 0, 0, smallest)
     if p is not None:
         p['glints'] = find_glints(frame, p)
     return p
 
 
-def _find_either(img, ox, oy):
-    p = _find(img, ox, oy)
-    return p if p is not None else _find(img, ox, oy, CLOSE_TIGHT)
+def _find_either(img, ox, oy, smallest=0.0):
+    # Both closing sizes use the same input mask; morphologyEx returns a new one.
+    dark = (cv2.GaussianBlur(img, (5, 5), 0) < DARK).astype(np.uint8)
+    p = _find(img, dark, ox, oy, smallest=smallest)
+    return p if p is not None else _find(img, dark, ox, oy, CLOSE_TIGHT, smallest)
 
 
-def _find(img, ox, oy, close=CLOSE):
-    """The pupil in `img` (a window at ox, oy of the frame), with frame coordinates. A dark
+def _find(img, dark, ox, oy, close=CLOSE, smallest=0.0):
+    """The pupil in `img` (a window at ox, oy), using its prepared `dark` mask. A dark
     region touching the window's edge doesn't count: it's background, rim, or cut off."""
-    f = cv2.GaussianBlur(img, (5, 5), 0)
-    dark = (f < DARK).astype(np.uint8)
     # Glints punch bright holes in the pupil; close them so the blob stays whole.
-    dark = cv2.morphologyEx(dark, cv2.MORPH_CLOSE, np.ones((close, close), np.uint8))
-    dark = cv2.morphologyEx(dark, cv2.MORPH_OPEN, np.ones((3, 3), np.uint8))
+    dark = cv2.morphologyEx(dark, cv2.MORPH_CLOSE, _KERNELS[close])
+    dark = cv2.morphologyEx(dark, cv2.MORPH_OPEN, _KERNELS[3])
     n, lab, stats, _ = cv2.connectedComponentsWithStats(dark, connectivity=8)
     h, w = img.shape
     best = None
@@ -74,7 +83,7 @@ def _find(img, ox, oy, close=CLOSE):
             continue
         (ex, ey), (d1, d2), ang = cv2.fitEllipse(c)
         a, b = max(d1, d2) / 2, min(d1, d2) / 2
-        if b < 3 or a / b > MAX_ASPECT:
+        if b < 3 or a / b > MAX_ASPECT or a < smallest:
             continue
         fill = area / (np.pi * a * b)
         if fill < MIN_FILL or fill > 1.25:
@@ -90,6 +99,116 @@ def _find(img, ox, oy, close=CLOSE):
     return best[1] if best else None
 
 
+SMALLEST = 0.5     # a find under this share of the pupil's recent size is something else
+SIZE_WINDOW = 0.5  # s of finds the recent size comes from
+SIZE_FINDS = 10    # and at least this many of them
+
+
+class PupilSize:
+    """One eye's recent pupil size, the median long semi-axis of its finds over the last
+    SIZE_WINDOW s. A pupil narrows in bright light over a second or so, and turning the eye
+    changes how big the camera sees it by a few percent per frame; in the practice captures
+    (2026-09-29) under 0.4 % of finds were below half this median, many of them strays as well.
+    Only smaller finds are refused: if a wrong spot gets in (after a loss, with no recent size),
+    the real pupil isn't refused when it's back, and the median follows it."""
+
+    def __init__(self):
+        self.finds = deque()   # (t, a)
+
+    def smallest(self, t):
+        """The smallest long semi-axis a find at time t may have, in px (0: no limit yet)."""
+        while self.finds and self.finds[0][0] < t - SIZE_WINDOW:
+            self.finds.popleft()
+        if len(self.finds) < SIZE_FINDS:
+            return 0.0
+        return SMALLEST * float(np.median([a for _, a in self.finds]))
+
+    def add(self, t, p):
+        if p is not None:
+            self.finds.append((t, p['a']))
+
+
+GUARD_RULES = ("both", "long", "off")
+GUARD_LONG = 10.0   # s of kept finds an eye's usual size comes from
+GUARD_FINDS = 20    # and at least this many of them
+GUARD_BOTH = 0.75   # "both": a find under this share of the other eye's ratio to its usual size
+GUARD_ALONE = 0.6   # without the other eye just now (or "long"): under this share of its usual size
+GUARD_FRESH = 0.05  # s: the other eye's last find counts this long
+GUARD_RESET = 1.0   # s of refusals in a row: the pupil really did change, its usual size starts over
+
+
+class SizeGuard:
+    """Both eyes' usual pupil sizes, for the misfinds PupilSize lets through (2026-10-17, two
+    Gaze Bench runs): a dark spot two thirds of the pupil's size, held for a few frames, so the
+    last half second's median follows it in. Here each eye's usual size is the median long
+    semi-axis of its kept finds over the last GUARD_LONG s, and a find is judged by its ratio to
+    that. The pupils narrow and widen together in light (the consensual reflex), so with
+    "both" a find is refused when its ratio is under GUARD_BOTH of the other eye's latest
+    (capped at 1): light shrinks both, a misfind one. On those runs (8884 finds) good finds sat
+    at 0.84 of the usual size or more 95 % of the time, the right eye's misfinds at a median of
+    0.77 and a quarter of them under 0.52. "long" is the ratio alone, under GUARD_ALONE, which
+    is also the rule while the other eye isn't seen. A refused find counts as not found. If
+    refusals go on for GUARD_RESET s, the eye's usual size starts over, so a pupil that did
+    change (light with one eye lost) is taken again. Each eye's kept and refused finds and those
+    restarts are counted (counts), so a session shows whether it refused real pupils: many
+    restarts, or refusals while both eyes looked fine, say it did."""
+
+    def __init__(self, rule="both"):
+        self.rule = rule if rule in GUARD_RULES else "both"
+        self.finds = {0: deque(), 1: deque()}   # (t, a) of kept finds
+        self.usual = {0: None, 1: None}         # the median, recomputed every few finds
+        self.since = {0: 0, 1: 0}
+        self.last = {0: None, 1: None}          # (t, ratio) of each eye's last kept find
+        self.refusing = {0: None, 1: None}      # when the current run of refusals began
+        self.refused = {0: 0, 1: 0}
+        self.kept = {0: 0, 1: 0}
+        self.restarts = {0: 0, 1: 0}            # runs of refusals that went on GUARD_RESET s
+
+    def ratio(self, e, t, a):
+        h = self.finds[e]
+        while h and h[0][0] < t - GUARD_LONG:
+            h.popleft()
+            self.usual[e] = None
+        if len(h) < GUARD_FINDS:
+            return None
+        if self.usual[e] is None or self.since[e] >= 10:
+            self.usual[e], self.since[e] = float(np.median([x for _, x in h])), 0
+        return a / self.usual[e]
+
+    def keep(self, e, t, a):
+        """Is eye e's find at time t, long semi-axis a px, its pupil? (It learns from the kept ones.)"""
+        if self.rule == "off":
+            return True
+        r = self.ratio(e, t, a)
+        ok = True
+        if r is not None:
+            o = self.last[1 - e]
+            if self.rule == "both" and o is not None and t - o[0] <= GUARD_FRESH:
+                ok = r >= GUARD_BOTH * min(1.0, o[1])
+            else:
+                ok = r >= GUARD_ALONE
+        if not ok:
+            if self.refusing[e] is None:
+                self.refusing[e] = t
+            elif t - self.refusing[e] > GUARD_RESET:
+                self.finds[e].clear()
+                self.usual[e], ok, r = None, True, None
+                self.restarts[e] += 1
+        if not ok:
+            self.refused[e] += 1
+            return False
+        self.refusing[e] = None
+        self.kept[e] += 1
+        self.finds[e].append((t, a))
+        self.since[e] += 1
+        self.last[e] = (t, 1.0 if r is None else r)
+        return True
+
+    def counts(self, e):
+        """Eye e's finds judged so far: {"kept", "refused", "restarts"} (none with the rule off)."""
+        return {"kept": self.kept[e], "refused": self.refused[e], "restarts": self.restarts[e]}
+
+
 GLINT_RING = 140   # a glint sits on dark iris or pupil: its surroundings are below this
 GLINT_PAIR = (5, 45)  # the two LEDs' reflections: this far apart, in pixels, mostly vertical
 

gaze/tracker/ft-eyes

@@ -28,8 +28,13 @@ Control socket: abstract datagram "@ft_eyes"; each command gets one reply line.
   calib-start                  a new calibration: collect dots from now on
   calib-point T0 T1 YAW PITCH  a dot you looked at from T0 to T1 (CLOCK_MONOTONIC_RAW) in
                                that direction (head-relative degrees): "ok N0 N1 SD0 SD1"
-                               (frames and spread in px per eye, right first) or "fail WHY"
-  calib-fit                    fit the dots, save, and start the shifts and clicks over
+                               (frames and spread in px per eye, right first), then
+                               " (EYE eye only: WHY)" if one eye wasn't seen enough or moved
+                               (the dot is kept for the other), or "fail WHY"
+  calib-fit                    fit the dots, save, and start the shifts and clicks over. Each
+                               eye is fitted on the dots it was seen at; one seen at too few
+                               (eyes_model.MIN_CLICKS) is left out, and the other maps the
+                               gaze alone ("..., left not used (9 dots, needs 12)")
   click T YAW PITCH            a click: you were looking there just before T; teaches the
                                shift: "ok DX0 DY0 DX1 DY1" (the shift each eye measured)
   fit                          JSON: the last second of pupil finds per camera, for the fit
@@ -37,8 +42,9 @@ Control socket: abstract datagram "@ft_eyes"; each command gets one reply line.
   finds start PATH | finds stop
                                append every frame looked at to PATH as JSON lines (the eye
                                recorder's own.jsonl): {"t", "cam", "x", "y", "a", "b", "angle",
-                               "glints": [[x, y], ...], "gaze": [yaw, pitch] once calibrated},
-                               or {"t", "cam", "lost": true}; with the net also "net": {"gaze",
+                               "glints": [[x, y], ...], "gaze": [yaw, pitch] once calibrated,
+                               "guard": true if the size guard refused it}, or {"t", "cam",
+                               "lost": true}; with the net also "net": {"gaze",
                                "x", "y", "open", "visible"}. "finds stop" replies "ok ROWS"
 
 The eye net (eyes_net.py) instead of the pupil finder: EYES_MODEL in ~/.config/frametop.conf (or
@@ -53,9 +59,17 @@ service restarts ft-eyes when EYES_MODEL changes. If the model can't load, ft-ey
 so and uses the pupil finder. While the fit coach asks ("fit") or finds are logged, the pupil
 finder runs too, for them.
 
+The pupil finder's size guard (eyes_pupil.SizeGuard): a find much smaller than that eye's usual
+size, judged against how the other eye's size compares with its own (both narrow in light),
+counts as not found. EYES_SIZE_GUARD in frametop.conf (or FT_EYES_SIZE_GUARD): both (the
+default), long (against its own usual size only) or off. Read at start. Each eye's kept and
+refused finds, and its usual size's restarts, are in the status ("size_guard_counts", since the
+start), and once a minute that it refused any, a log line says how many ("size guard, the last
+60 s: right refused 40 of 3100 finds; left refused 0 of 3200 finds").
+
 Environment, for replays (lab/ft-eyes-e2e): FT_EYES_CAMS, FT_EYES_GAZE, FT_EYES_STATE (the
-state folder), FT_EYES_SOCKET (the socket's name), FT_EYES_MODEL. With FT_EYES_CAMS set, the want file isn't
-touched.
+state folder), FT_EYES_SOCKET (the socket's name), FT_EYES_MODEL, FT_EYES_SIZE_GUARD. With
+FT_EYES_CAMS set, the want file isn't touched.
 
 /dev/shm/frametop-eyes-gaze, 128 bytes, little-endian (mirrored in ft-gaze.cpp):
    0 u32 seq      odd while it's being written: read it before and after, retry if it moved
@@ -121,6 +135,27 @@ def load_net():
 
 NET = load_net()
 
+
+def size_guard_rule():
+    """EYES_SIZE_GUARD in ~/.config/frametop.conf (FT_EYES_SIZE_GUARD wins): eyes_pupil.SizeGuard's
+    rule, both (the default), long or off. Read at start."""
+    v = os.environ.get("FT_EYES_SIZE_GUARD")
+    if v is None:
+        v = ""
+        try:
+            for line in (Path.home() / ".config" / "frametop.conf").read_text().splitlines():
+                k, _, val = line.split("#", 1)[0].partition("=")
+                if k.strip() == "EYES_SIZE_GUARD":
+                    v = val.strip().strip("\"'")
+        except OSError:
+            pass
+    v = v.strip().lower()
+    return v if v in eyes_pupil.GUARD_RULES else "both"
+
+
+SIZE_GUARD = size_guard_rule()
+GUARD_REPORT = 60.0  # s between the size guard's log lines (only when it refused something)
+
 CAMS = os.environ.get("FT_EYES_CAMS", "/dev/shm/frametop-eyes-cams")  # overrides for replays
 OUT = os.environ.get("FT_EYES_GAZE", "/dev/shm/frametop-eyes-gaze")
 WANT = None if "FT_EYES_CAMS" in os.environ else Path("/dev/shm/frametop-eyes-want")
@@ -222,6 +257,9 @@ class Eye:
         self.shift = eyes_model.Shift()
         self.history = deque()  # (t, x, y, glint mid x, y)
         self.last = None        # the previous pupil (window hint)
+        self.size = eyes_pupil.PupilSize()   # its recent size: much smaller finds are something else
+        self.guard = None       # both eyes' usual sizes (eyes_pupil.SizeGuard, Tracker's), for the pupil finder
+        self.refused = 0
         self.gaze = None        # (t, yaw, pitch, x, y)
         self.lost = 0
         self.frames = self.found = 0
@@ -250,12 +288,18 @@ class Eye:
             if self.lost % LOST_EVERY:
                 return
         t0 = time.perf_counter()
-        p = eyes_pupil.find_pupil(img, self.last)
-        self.last = p
+        p = eyes_pupil.find_pupil(img, self.last, self.size.smallest(t))
+        self.size.add(t, p)
+        # A find the size guard refuses counts as not found, and the search stays where the last one was.
+        refused = p is not None and self.guard is not None and not self.guard.keep(self.eye, t, p["a"])
+        if refused:
+            self.refused += 1
+        else:
+            self.last = p
         if self.obs is not None:
             self.obs.add(self.eye, t, p)
             self.fed = (t, time.clock_gettime(time.CLOCK_MONOTONIC_RAW))
-        if p is not None:
+        if p is not None and not refused:
             self.found += 1
             self.lost = 0
             pair = eyes_pupil.glint_pair(p)
@@ -270,7 +314,7 @@ class Eye:
                 g = self.cal.gaze(self.eye, p["x"], p["y"], self.shift.value)
                 self.gaze = (t, float(g[0]), float(g[1]), p["x"], p["y"])
         if self.finds is not None:
-            self.finds.write(self, t, p)
+            self.finds.write(self, t, p, refused=refused)
         self.work += time.perf_counter() - t0
 
     def feed_net(self, t, img):
@@ -280,8 +324,9 @@ class Eye:
         classic = self.finds is not None or time.monotonic() < self.coach[0]
         p = None
         if classic:
-            p = eyes_pupil.find_pupil(img, self.last)
+            p = eyes_pupil.find_pupil(img, self.last, self.size.smallest(t))
             self.last = p
+            self.size.add(t, p)
             if self.obs is not None:
                 self.obs.add(self.eye, t, p)
                 self.fed = (t, time.clock_gettime(time.CLOCK_MONOTONIC_RAW))
@@ -317,7 +362,7 @@ class Finds:
         self.f = open(path, "a", buffering=1 << 16)
         self.rows = 0
 
-    def write(self, eye, t, p, net=None):
+    def write(self, eye, t, p, net=None, refused=False):
         if p is None:
             row = {"t": round(t, 6), "cam": eye.eye, "lost": True}
         else:
@@ -326,6 +371,8 @@ class Finds:
                    "glints": [[round(float(x), 1), round(float(y), 1)] for x, y in p.get("glints") or []]}
             if eye.gaze and eye.gaze[0] == t and not NET:
                 row["gaze"] = [round(eye.gaze[1], 3), round(eye.gaze[2], 3)]
+            if refused:
+                row["guard"] = True
         if net is not None:
             row["net"] = {"gaze": [round(v, 3) for v in net["gaze"]], "x": round(net["x"], 1), "y": round(net["y"], 1),
                           "open": round(net["open"], 3), "visible": round(net["visible"], 3)}
@@ -347,8 +394,9 @@ class Tracker:
         self.calibrating = False
         self.fit = fitcoach.Observations()
         coach = [0.0]
+        self.guard = eyes_pupil.SizeGuard(SIZE_GUARD)
         for e in self.eyes:
-            e.obs, e.coach = self.fit, coach
+            e.obs, e.coach, e.guard = self.fit, coach, self.guard
         if CALIBRATION.exists():
             self.cal = eyes_model.Calibration.load(CALIBRATION)
             if self.cal.info.get("feature", "pupil") != FEATURE:
@@ -417,23 +465,44 @@ class Tracker:
                 return "fail no calibration started"
             t0, t1, yaw, pitch = map(float, w[1:])
             got = {e.eye: e.window(t0, t1) for e in self.eyes}
+            # Each eye that was seen and held still counts: the dot is taken with one eye if
+            # the other was lost (looking far toward that eye's ear, say) or jumped about.
+            why = {}
             for c, g in got.items():
                 if g is None or g["n"] < CALIB_MIN:
-                    return f"fail the {EYES[c]} eye was seen in only {0 if g is None else g['n']} frames"
-                if g["spread"] > CALIB_SPREAD:
-                    return f"fail the {EYES[c]} eye moved ({g['spread']:.1f} px)"
-            self.dots.append(dict(truth=(yaw, pitch), eye={c: dict(pupil=g["pupil"], mid=g["mid"]) for c, g in got.items()}))
-            return "ok {} {} {:.2f} {:.2f}".format(got[0]["n"], got[1]["n"], got[0]["spread"], got[1]["spread"])
+                    why[c] = f"the {EYES[c]} eye was seen in only {0 if g is None else g['n']} frames"
+                elif g["spread"] > CALIB_SPREAD:
+                    why[c] = f"the {EYES[c]} eye moved ({g['spread']:.1f} px)"
+            if len(why) == 2:
+                moved = [c for c in (0, 1) if "moved" in why[c]]
+                if moved:
+                    return "fail " + why[moved[0]]
+                return "fail neither eye was seen enough ({})".format(
+                    ", ".join(f"{EYES[c]} {0 if got[c] is None else got[c]['n']} frames" for c in (0, 1)))
+            self.dots.append(dict(truth=(yaw, pitch), eye={c: dict(pupil=g["pupil"], mid=g["mid"])
+                                                           for c, g in got.items() if c not in why}))
+            n = [0 if got[c] is None else got[c]["n"] for c in (0, 1)]
+            spread = [math.nan if got[c] is None else got[c]["spread"] for c in (0, 1)]
+            reply = "ok {} {} {:.2f} {:.2f}".format(*n, *spread)
+            for c, text in why.items():
+                reply += f" ({EYES[1 - c]} eye only: {text})"
+            return reply
         if w[0] == "calib-fit":
             if len(self.dots) < eyes_model.MIN_CLICKS:
                 return f"fail only {len(self.dots)} dots (need {eyes_model.MIN_CLICKS})"
+            seen = {c: sum(1 for k in self.dots if c in k["eye"]) for c in (0, 1)}
             cal = eyes_model.Calibration.fit(self.dots, {"made": time.strftime("%Y-%m-%d %H:%M:%S"),
                                                        "dots": len(self.dots), "from": "probe calibration",
-                                                       "feature": FEATURE})
-            if not all(cal.has(n, c) for n in ("pupil", "where") for c in (0, 1)):
-                return "fail not enough dots with both eyes"
-            errs = [float(np.hypot(*(np.mean([cal.gaze(c, *k["eye"][c]["pupil"]) for c in (0, 1)], axis=0)
-                                     - k["truth"]))) for k in self.dots]
+                                                       "feature": FEATURE,
+                                                       "eye_dots": {EYES[c]: seen[c] for c in (0, 1)}})
+            if not cal.eyes():
+                return "fail neither eye was seen at enough dots ({}; need {})".format(
+                    ", ".join(f"{EYES[c]} {seen[c]}" for c in (0, 1)), eyes_model.MIN_CLICKS)
+            errs = []
+            for k in self.dots:
+                g = cal.combine({c: cal.gaze(c, *v["pupil"]) for c, v in k["eye"].items() if cal.has("pupil", c)})
+                if g is not None:
+                    errs.append(float(np.hypot(*(g - k["truth"]))))
             if CALIBRATION.exists():
                 CALIBRATION.replace(CALIBRATION.with_name(time.strftime("calibration-%Y%m%d-%H%M%S.json")))
             cal.save(CALIBRATION)
@@ -448,7 +517,9 @@ class Tracker:
                                                     "mid": None if v["mid"] is None else v["mid"].tolist()}
                                                 for c, v in k["eye"].items()}}) + "\n")
             return (f"ok {len(self.dots)} dots, fit median {np.median(errs):.2f} deg, eyes "
-                    + ", ".join(f"{EYES[c]} {cal.spread[c]:.2f}" for c in sorted(cal.spread)))
+                    + ", ".join(f"{EYES[c]} {cal.spread[c]:.2f} ({seen[c]} dots)" if c in cal.spread
+                                else f"{EYES[c]} not used ({seen[c]} dots, needs {eyes_model.MIN_CLICKS})"
+                                for c in (0, 1)))
         if w[0] == "click" and len(w) == 4:
             if not self.cal:
                 return "fail not calibrated"
@@ -473,8 +544,10 @@ class Tracker:
 
     def status(self):
         return {"calibration": self.cal.info if self.cal else None, "calibrating": self.calibrating,
+                "mapped": [EYES[c] for c in self.cal.eyes()] if self.cal else None,
                 "feature": FEATURE, "model": NET.folder if NET else None,
-                "dots": len(self.dots),
+                "dots": len(self.dots), "size_guard": None if NET else self.guard.rule,
+                "size_guard_counts": None if NET else {EYES[e]: self.guard.counts(e) for e in (0, 1)},
                 "eyes": {EYES[e.eye]: {"shift": e.shift.value.tolist(), "clicks": len(e.shift.meas),
                                        "jump": bool(np.any(e.shift.jump)), "jump_px": e.shift.jump.tolist(),
                                        "reseat": e.shift.reseated} for e in self.eyes}}
@@ -558,6 +631,18 @@ def below_steamvr():
         print(f"ft-eyes: SCHED_BATCH: {e}", file=sys.stderr, flush=True)
 
 
+def guard_line(guard, was):
+    """The size guard since `was` (its counts then): (its counts now, a log line, or "" if it
+    refused nothing). Its own line, once a minute: ft-gazed keeps it (it drops the fps lines)."""
+    now = {e: guard.counts(e) for e in (0, 1)}
+    d = {e: {k: now[e][k] - was[e][k] for k in now[e]} for e in (0, 1)}
+    if not any(d[e]["refused"] for e in (0, 1)):
+        return now, ""
+    parts = [f"{EYES[e]} refused {d[e]['refused']} of {d[e]['kept'] + d[e]['refused']} finds"
+             + (f", {d[e]['restarts']} restarts" if d[e]["restarts"] else "") for e in (0, 1)]
+    return now, f"size guard, the last {GUARD_REPORT:.0f} s: " + "; ".join(parts)
+
+
 def main():
     below_steamvr()
     verbose = "-v" in sys.argv
@@ -580,12 +665,14 @@ def main():
     cal = tracker.cal
     print("ft-eyes: " + (f"the eye net {NET.name} ({NET.folder}); " if NET else "")
           + (f"calibration from {cal.info.get('made')} ({cal.info.get('from', cal.info.get('capture'))})"
-                           if cal else "not calibrated: run the probe's calibration with the Own tracker")
+             + ("" if len(cal.eyes()) == 2 else f", the {' and '.join(EYES[c] for c in cal.eyes())} eye only")
+             if cal else "not calibrated: run the probe's calibration with the Own tracker")
           + f"; waiting for {CAMS}", file=sys.stderr, flush=True)
     cams = wait_for_cams(sock, tracker, want)
     print("ft-eyes: frames found, tracking", file=sys.stderr, flush=True)
     seen = [cams.count(0), cams.count(1)]
-    report = time.monotonic()
+    report = guard_at = time.monotonic()
+    guard_was = {e: tracker.guard.counts(e) for e in (0, 1)}
     arrived = [0.0, 0.0]  # when each camera's newest frame was seen (monotonic)
     while True:
         serve(sock, tracker)
@@ -621,13 +708,18 @@ def main():
                 parts = []
                 for e in eyes:
                     s = e.shift.value
-                    parts.append(f"{EYES[e.eye]} {e.frames / 5:.0f} fps, found {e.found / max(e.frames, 1):.0%}, "
+                    parts.append(f"{EYES[e.eye]} {e.frames / 5:.0f} fps, found {e.found / max(e.frames, 1):.0%}"
+                                 + (f" ({e.refused} refused by size)" if e.refused else "") + ", "
                                  f"{e.work / max(e.frames, 1) * 1000:.2f} ms/frame, shift ({s[0]:+.1f},{s[1]:+.1f})"
                                  f" from {len(e.shift.meas)} clicks" + (", jump" if np.any(e.shift.jump) else ""))
-                    e.frames = e.found = 0
+                    e.frames = e.found = e.refused = 0
                     e.work = 0.0
                 print("ft-eyes: " + "; ".join(parts), file=sys.stderr, flush=True)
             report = now
+            if now - guard_at >= GUARD_REPORT:
+                guard_at, guard_was, line = now, *guard_line(tracker.guard, guard_was)
+                if line and not NET:
+                    print("ft-eyes: " + line, file=sys.stderr, flush=True)
             if cams.replaced():
                 print("ft-eyes: frames went away; waiting", file=sys.stderr, flush=True)
                 cams = wait_for_cams(sock, tracker, want)

hand-recorder.md

@@ -54,7 +54,7 @@ The [standalone recorder's README](https://github.com/Frametop/frametop-hand-rec
    curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --release --experimental
    ```
 
-   This downloads Frametop's newest experimental release (about 1.4 GB) and installs it without compiling anything. The installer asks what to install: say yes to the Hand Recorder, and to hand tracking if you want to record testing sessions. It asks for your `sudo` password for those. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, the same command updates it, and the Hand Recorder with it.
+   This downloads Frametop's newest experimental release (about 1.2 GB) and installs it without compiling anything. The installer asks what to install: say yes to the Hand Recorder, and to hand tracking if you want to record testing sessions. It asks for your `sudo` password for those. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, the same command updates it, and the Hand Recorder with it.
 4. Choose Launch a program → Desktop again, and open Frametop Hand Recorder from the application menu.
 
 Installing `Frametop.zip` with [FrameDrop](https://framedropvr.com), or by hand from the [releases](https://github.com/Frametop/frametop/releases), works too: leave Hand Recorder ticked in the installer's window.

hands/README.md

@@ -102,7 +102,7 @@ Options:
 - `--sensor S`: only the mono cameras whose sensor name contains S.
 - `--status S`: a status line every S seconds (0: never).
 
-It exits when XRService exits, when a camera's buffers keep going stale, which means XRService has reallocated them, or when a camera stops while the others keep running (3 s), which means XRService has moved it to another device (the colour module was unplugged or plugged in). The service starts it again, and it attaches to the new buffers.
+It exits when XRService exits, when a camera's buffers keep going stale, which means XRService has reallocated them, or when a camera stops while the others keep running (3 s), which means XRService has moved it to another device (the colour module was unplugged or plugged in). The service starts it again, and it attaches to the new buffers: after 1 s the first time, then 1.7, 2.9 and 5 s while SteamVR runs (`RestartSteps`; it can't start itself again, having dropped its capabilities). ft-hands waits for the new ring (up to 15 s) and starts itself again on its cameras, without waiting out its own service's 5 s.
 
 **Which camera is which:** video9 is `slam_right` (sensor `og01a1b 4-0036`), video13 is `slam_left` (`4-0060`), video6 is `upper_left` and video7 is `upper_right`; without the colour module the side pair is on video0 (`slam_right`) and video3. XRService says so in its log: `Found camera 'slam_left': ... v4l_subdev=/dev/v4l-subdev30` names each sensor, and each `TrackingCameraInit` line gives the device and subdev it opened. ft-hands and `camcheck.py` name the devices that way. Until 2026-10-05 they named them by the `TrackingCameraInit` index instead, which is only the order XRService opens them in (index 0 was `slam_right` on every start logged), and ft-camd bound each buffer queue to a device by the order of XRService's file descriptors, which changes when XRService restarts its cameras. The two mistakes made the side names come out swapped on most starts and right on some. ft-camd now asks each device which buffers it holds (`VIDIOC_QUERYBUF` names the descriptor XRService queued at each index) and only falls back to the order if that fails. With the names swapped, every hand is seen by one camera only, at the wrong depth, and the hand holes land beside the hands. ft-hands still checks by itself, in case (`track/sides.h`, `HANDS_SWAP_SIDES=auto`):
 - Whenever a hand's landmarks are found in two cameras at once (one of them a side camera), it intersects the rays through the 21 landmarks twice: once with the calibrations as named, once with the two side cameras exchanged. The same hand seen the right way meets within a few mm, in front of both cameras and as far away as its size says. The wrong way misses by centimetres or meets behind a camera.
@@ -134,6 +134,7 @@ Options:
 - `--nice N`: niceness. Default 5, so the VR stack wins contested CPUs.
 - `--no-publish`: don't write the hands and gestures files.
 - Parked: while `/run/user/UID/frametop-hands/park` exists, ft-hands processes no frames (the models stay loaded), publishes no hands, and ends any gesture down, as lost; it writes `0` to `hands-want` there every second, and ft-camd stops copying the mono cameras' frames while that file is fresh (3 s) and says 0. The input relay parks it while Frametop is paused for a VR game with hand interaction on (`POINTER_HANDS=1`), instead of stopping the services, and unparks it while the dashboard is open, so the hands work there; it tracks again from a palm search. Not while recording.
+- Frame sets: a set is the side pair's newer frames, taken within 3 ms, with the upper pair's from the same moment (it waits up to 8 ms after the sides for theirs). XRService keeps the uppers in step with the sides, but after a colour module hot plug they can stay out of step: on 2026-10-17 XRService logged `Set upper camera driver not in-sync` and never `in-sync` again, the uppers ran at 40 fps with no dark frames, and XRService's own tracking replaced them with dummy frames. Waiting for all four to meet left 6-7 sets a second (27-30 normally). Now, when the uppers met fewer than half of the recent sets, ft-hands says so, drops their views and tracks with the side pair alone, at the full rate; when they meet 95% again it takes them back. Taking the headset off and putting it back on restarts the camera streams, which normally brings the uppers in step within half a second (untested after a hot plug). A recording keeps only full sets.
 - `--no-gestures`: hands for the cutouts only. No pinch or grip detection, so nothing reaches the pointer and a closing hand doesn't raise the rate to 30 Hz. The gestures file is removed at start. `ft-cutouts` runs it this way.
 - `--record DIR`, `--record-for S`, `--record-hz N`: save every frame set for S seconds (default 120) to `DIR/sets.bin`, or at most N sets a second with `--record-hz` (the hand recorder uses 10). Every set is about 80 MB/s. Sending the tracker SIGUSR1 (`pkill -USR1 -x ft-hands`) starts a recording in `~/.local/share/frametop/hands/rec-<time>` without a restart. Recordings are images of your hands and room: they stay on the headset unless you move them.
 - `--record-only`: record without tracking or publishing, so it can run beside the live tracker. Give it `--record DIR`, since SIGUSR1 would reach both trackers. With `ft-camd --with-dark`, recordings also hold each camera's newest dark frame as `<name>_dk`, which doubles the rate. With `--with-color`, each colour camera's newest frame is saved with every set, as `color_video<N>`, which adds about 70 MB/s. Run the recorder at normal I/O priority: idle I/O priority stalled a 165 MB/s recording.
@@ -228,7 +229,7 @@ ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gest
 
 With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/ft-pointer.cpp`) reads the gestures file every frame. It's off by default.
 
-**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. The ray comes up as the thumb nears the index or middle tip (`POINTER_RAY_READY`). With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), a three-finger pinch (thumb, index and middle together) scrolls as the hand moves, and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`).
+**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand in front and pointing, palm away, high or low (not while it types: the typing guard; `POINTER_RAY_RAISED=30` brings back the old limit, the knuckles within 30 cm under the eyes), shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. The ray comes up as the thumb nears the index or middle tip (`POINTER_RAY_READY`). With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), a three-finger pinch (thumb, index and middle together) scrolls as the hand moves, and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`).
 
 - **Pinch to click, in gaze mode.** Your eyes aim and your hands click, Vision Pro style, and the hand ray stays off (`POINTER_GAZE_HAND_RAY`). A pinch locks the pointer where you were looking just before the fingers began to close (75 ms before ft-hands saw the pinch, since the eyes often move on as the hand closes), and a quick tap clicks there. Pinch, hold and move the hand to nudge the pointer: nothing moves until the hand has moved 4 degrees (about 3 cm at arm's length), then the pointer follows at 0.3 times the hand's movement. Letting go clicks where the pointer is, and the correction is a lesson for the gaze tracker, as with the mouse. Held still for `POINTER_GAZE_HOLD` (0.5 s) instead, the pinch is a real press at that point, and the hand drags at `POINTER_GRIP_GAIN`. A middle pinch is a right click the same way. Hold a pinch with one hand and pinch with the other: both buttons are down, left for the first hand and right for the second, each until its own pinch opens (a first pinch not yet pressed becomes a press at once, where the pointer is), as when you hold both of a mouse's buttons. The rules and timings are in `pointer/helper/gazehand.h`.
 - **Without gaze mode,** a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is.

hands/frametop-camd.service

@@ -5,6 +5,9 @@ Documentation=file://@REPO@/hands/README.md
 # It borrows XRService's camera buffers, so it comes and goes with SteamVR.
 After=steamvr.service
 PartOf=steamvr.service
+# Room for the quick restarts below, while XRService is away (8 starts in 30 s at most).
+StartLimitIntervalSec=30
+StartLimitBurst=12
 
 [Service]
 # On the host: the dev container can't reach XRService. Its file capabilities (set by
@@ -15,7 +18,12 @@ PartOf=steamvr.service
 # Add --with-color to try the colour cameras (ft-hands then picks them by the light).
 ExecStart=@REPO@/hands/build/ft-camd --status 60
 Restart=always
-RestartSec=5
+# 1 s at first, so hand tracking is back soon after a colour module hot plug (it can't restart
+# itself: it has dropped its capabilities), then longer: 1.7, 2.9, 5 s (the count only resets
+# when SteamVR starts it).
+RestartSec=1
+RestartSteps=3
+RestartMaxDelaySec=5
 TimeoutStopSec=5
 
 [Install]

hands/include/fh_ray.h

@@ -58,7 +58,11 @@
 //   tracked     a two-camera (STEREO) frame within      a frame within lostFor (250 ms); one-camera
 //               stereoWithin (200 ms)                   depth is 2-6 cm off, fine for a short gap
 //   raised      K at most raisedShow (0.30 m) below     raisedHide (0.40 m): gaze pinches sat
-//               the eyes                                0.35-0.45 m below the eyes, elbow resting
+//               the eyes (0: off; the pointer helper    0.35-0.45 m below the eyes, elbow resting
+//               turns it off: user, 2026-10-19, "hands
+//               still recognized even if they are low,
+//               just not if they are typing", which its
+//               typing guard holds back)
 //   forward     K at least forwardShow (0.25 m) ahead   forwardHide (0.20 m)
 //               of the eyes, along the torso's yaw
 //   palm away   palm normal . D >= palmShow (0.3)       palmHide (0.0): palm toward you is reserved
@@ -324,7 +328,7 @@ inline Tests Evaluate(const Params &p, const Body &body, const HandFrame &f, con
         if (!s) r.fails += r.fails.empty() ? name : std::string(",") + name;
         show = show && s, keep = keep && k;
     };
-    test(r.below <= p.raisedShow, r.below <= p.raisedHide, "raised");
+    if (p.raisedShow > 0) test(r.below <= p.raisedShow, r.below <= std::max(p.raisedShow, p.raisedHide), "raised");
     test(r.ahead >= p.forwardShow, r.ahead >= p.forwardHide, "forward");
     test(r.palm >= p.palmShow, r.palm >= p.palmHide, "palm");
     if (r.forearmUsed) test(r.forearm >= p.forearmShow, r.forearm >= p.forearmHide, "forearm");

hands/models.json

@@ -1,7 +1,7 @@
 {
  "catalog": 1,
  "kind": "hands",
- "latest": "v16b",
+ "latest": "v19b",
  "default": "latest",
  "baseline": "stock",
  "metrics": [
@@ -19,11 +19,34 @@
    "note": "Of the held-out views where the thumb and index tips touch, the share the model reads as touching."}
  ],
  "models": [
+  {
+   "name": "v19b",
+   "title": "Frametop v19b",
+   "kind": "custom",
+   "status": "recommended",
+   "date": "2026-10-19",
+   "summary": "Taught by the bounty rounds' prompts: held pinches and middle-finger touches read closed far more often.",
+   "notes": "v13b-p1's landmark model trained further on v16b's data plus Frametop Hands contributions #15 (pokes), #16 (middle-finger taps) and #17 (card grabs). New: where a bounty round's prompt asked for a fingertip touch, training pulls those tips together, and where it asked for a resting hand, it keeps them apart. The palm detector is v13b-p1's, unchanged. Same networks as stock, so the same CPU cost.\n\nWhat to expect against v16b: on prompted acts held back from training, a held thumb-index pinch reads closed 58% of the time (v16b 21%) and a held thumb-middle touch 100% (v16b 33%). The replays track the same hands, and finger points and hand shape are as close as before. The held-out pinch number below is lower (69% against 80%) because it counts touches by the teacher model's labels, which miss many real touches; v16b followed them more closely.\n\nKnown weak spots: it reads the thumb and index tips a few millimetres further apart than v16b, so a light pinch may need a firmer squeeze, and a resting hand reads as touching a little more often for some people.",
+   "cost": "Same as stock: two networks, fp16, on the cores hand tracking already uses.",
+   "bench": {"replay_bright": 961, "replay_dim": 1225, "replay_lit": 1434, "found": 98.6, "landmark_px": 3.8,
+             "within5": 94.9, "shape_mm": 8.0, "side_ok": 96.7, "pinch_closed": 69.0},
+   "format": 1,
+   "rev": "52d3a657abcf0cef7db48f90c5fefb281ff4b5b7",
+   "files": {
+    "NOTICE": "96fc8914e0220e51222362d141e0495e3425081d9898c21899b63e68df358ca6",
+    "hand.ncnn.bin": "010f96a9316d0d935f2baf4951770130402bddc126b0066b4d8b3ba2754fdc9a",
+    "hand.ncnn.param": "54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd",
+    "palm.ncnn.bin": "9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909",
+    "palm.ncnn.param": "c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e",
+    "LICENSE": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
+    "README.md": "3393752d047bc3dd4e23fca47e3c29e4ae37c13be8dae1dae3412be55f4085f0"
+   }
+  },
   {
    "name": "v16b",
    "title": "Frametop v16b",
    "kind": "custom",
-   "status": "recommended",
+   "status": "older",
    "date": "2026-10-16",
    "summary": "v13b-p1 plus the pinch and tap bounty rounds: pinches and taps read closed more often.",
    "notes": "v13b-p1's landmark model trained further with Frametop Hands contributions #12 (index taps), #13 (pinches) and #14. The palm detector is v13b-p1's, unchanged. Same networks as stock, so the same CPU cost.\n\nWhat to expect against v13b-p1: a thumb touching the index finger is read as touching more often (80% of held-out touches against 71%), so pinches and taps register more reliably. Finger points and hand shape are as close as before, and the replays track the same hands.\n\nKnown weak spots: a thumb touching the middle finger is still often read as a near miss, and empty views are taken for a hand slightly more often than with v13b-p1 (0.37% against 0.22% of empty crops).",

hands/models.pin

@@ -5,12 +5,12 @@
 # needs to run them: raise it when a model needs a newer ft-hands, and older installs keep theirs.
 # Made by frame-hands' model-repo/pin.py.
 format 1
-rev d96e56599b0af18fc05af2a0a4489985914b754a
-model v16b
-file NOTICE 0d31ad8d108fc01e9fc4c5612eb5a9b16d90f9a971e6677cafe4969c2388fc06
-file hand.ncnn.bin c602ad2344bd1676a628fc8ab4fca2f53574b9f12bc97c705f47762d23f5f941
+rev 52d3a657abcf0cef7db48f90c5fefb281ff4b5b7
+model v19b
+file NOTICE 96fc8914e0220e51222362d141e0495e3425081d9898c21899b63e68df358ca6
+file hand.ncnn.bin 010f96a9316d0d935f2baf4951770130402bddc126b0066b4d8b3ba2754fdc9a
 file hand.ncnn.param 54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd
 file palm.ncnn.bin 9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909
 file palm.ncnn.param c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e
 file LICENSE 41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0
-file README.md 322fc0b146d301865592958ba4dd512bf249836816b578f0919a2f671f3f014c
+file README.md 3393752d047bc3dd4e23fca47e3c29e4ae37c13be8dae1dae3412be55f4085f0

hands/rec/CONSENT.md

@@ -43,7 +43,7 @@ The recorder doesn't record sound, your name, your email address or your Steam a
 
 - Recordings stay on your headset, in `~/.local/share/frametop/hands/contrib`. Nothing is uploaded automatically.
 - Before you share anything, you can watch every recording in the Review page. You can delete any stretch of a recording, a whole take or a whole session. Export leaves out what you deleted.
-- You upload the export yourself, from the Upload page. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
+- You upload the export yourself: from the Upload page, or with Export and upload, which sends it once it's exported. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
 
 ## Where it goes
 

hands/rec/DESIGN.md

@@ -402,7 +402,7 @@ Short rounds (about 3 min recorded, 4 with reading) of what the hand tracker nee
 
 - `bounties.json` is the board: `{"schema": 1, "updated", "bounties": [{"id", "title", "blurb", "minutes", "priority", "script"}]}`. A newer copy fetched from the dataset (`hub.py board`, the window's "Get the newest board", into `BASE/board-cache.json`) can only describe, reorder, reprioritise or retire bounties: each one's steps come from `bounties/<script>.json`, shipped with the recorder. A bounty whose script this recorder lacks shows "Needs a newer recorder".
 - A round script is a normal script (above) with `"record_hz": 30`: hand size first (the no-hands check), then one scene section, and at most 30 s of plain steps after it when a gesture needs no window (the palm up). At 30 sets a second a round is about 2.5 min recorded, about 2.1 MB a second exported.
-- The window opens on the Bounties page: the board by priority, then least done first, then in the board's own order (a fetched board ranks bounties of one priority by listing them; before 2026-10-15 the title broke ties), each with Record (Record again once done), "Get the newest board", and below them the full session and the quick round. Picking a round goes to the consent page if this contributor hasn't agreed to the current version, else straight to the checklist; the checklist's Round section shows the pick (or the full/quick choice) with "Pick another…" back to the board, and hides what a bounty doesn't use (objects within reach, controllers, guidance). Its camera check and Start then run the script (`session.py --bounty ID`; `--board` lists them). When the round is done, the Session page's "Export and upload" exports it and opens the Upload page for that export once the export finishes (`exportDone`). session.json gets `"bounty"`, the upload's title and description name it, and Review shows its title by the session. A round that ends `done` (not a dry run) counts in profile.json's `bounties`.
+- The window opens on the Bounties page: the board by priority, then least done first, then in the board's own order (a fetched board ranks bounties of one priority by listing them; before 2026-10-15 the title broke ties), each with Record (Record again once done), "Get the newest board", and below them the full session and the quick round. Picking a round goes to the consent page if this contributor hasn't agreed to the current version, else straight to the checklist; the checklist's Round section shows the pick (or the full/quick choice) with "Pick another…" back to the board, and hides what a bounty doesn't use (objects within reach, controllers, guidance). Its camera check and Start then run the script (`session.py --bounty ID`; `--board` lists them). When the round is done, the Session page's "Export and upload" queues its export to upload by itself once exported (logged in), or else exports it and opens the Upload page for that export once the export finishes (`exportDone`). session.json gets `"bounty"`, the upload's title and description name it, and Review shows its title by the session. A round that ends `done` (not a dry run) counts in profile.json's `bounties`.
 
 The first four (2026-10-14), each on the gestures Frametop uses:
 
@@ -490,7 +490,9 @@ The board can list one script under two bounties: `daylight-taps` is the index-t
     2. Log in to Hugging Face. The page shows whether someone is logged in (`whoami`). Log in runs `hub.py login`, opens the link in the browser, and shows the code large, with Copy code and Cancel. "Use another account" logs in again. A classic read-only token counts as not logged in.
     3. Upload, with a note to accept the dataset's terms the first time.
 
-    Upload has Cancel, the phase with a progress bar (a share while the export is checked, a sweep while it's sent, as `huggingface_hub` reports no progress), the pull request's link once it's open, with "plug in the headset and leave it plugged in until this says Uploaded", and Uploaded at the end. If this export was uploaded before, the page says so, and uploading it again asks first. A stale export can't be uploaded.
+    Upload has Cancel, the phase with a progress bar (a share while the export is checked, a sweep while it's sent, as `huggingface_hub` reports no progress), the pull request's link once it's open, with "plug in the headset and leave it plugged in until this says Uploaded", and Uploaded at the end. If this export was uploaded before, the page says so, and uploading it again asks first. A stale export can't be uploaded. An export and an upload of different sessions run together (2026-10-17: Upload used to wait for any export); a session's own export and upload wait for each other, since an export replaces its old one when it ends. One export and one upload at a time. `tests/test_upload_during_export.py` covers it.
+
+    Exports queue (2026-10-17, "so they don't have to keep coming back to the headset"): Export, or Export and upload, while another export runs puts it in a queue on the Export page, run in order. Export and upload (logged in, and the dataset takes contributions) uploads it by itself once its export is done: uploads wait their turn too, one at a time, and an export uploaded before (the same SHA256SUMS) isn't sent again. Each waiting export has a "then upload" box and Remove; Cancel stops only the running export (and its upload), and the next one starts. A failed export or upload doesn't hold up the rest. The page's Done list says how each went, with the pull request's link, for whoever comes back to the headset. The queue lives in the window: closing it asks, then cancels them all. A session waiting to export or upload can't be deleted, nor its export. The awake unit's workers now bring it to the reasons current when they run, so an export's end and its upload's start can't leave it stopped (`recorder/tests/test_stay_awake.py`). `tests/test_export_queue.py` covers the queue.
   - **Staying awake:** while an export or an upload runs, the window holds a host unit, `frametop-handrec-awake.service`, running `systemd-inhibit --what=sleep:idle --mode=block ... sleep infinity`, and stops it when the last of them ends (or on quit). It's meant to keep Steam from putting the Frame to sleep with the headset off; whether Steam's sleep honours a logind block inhibitor is still to be checked on the device. Exports and uploads are done in the headset: the export page only notes that VR may stutter a little while it compresses.
   - **While the texts are drafts**, Upload stays off unless `FT_HANDREC_ALLOW_UPLOAD=1`, so the maintainer can rehearse against a private test repo. `FT_HANDREC_DATASET` overrides `HF_DATASET`. `ft-handrec --hub-dry-run` makes Upload a dry run: no network, so it isn't held back by the drafts.
   - **Rehearsal: `hands/rec/rehearse.sh [--repo ID]`** runs it all without the headset, in the dev container, in one `frame-job --local` scope when frame-job is installed. `ft-ringplay` plays 30 s of a recording into a ring in `/run/user/UID`. A tracking ft-hands that's already running is used, or one is started on that ring. `ft-handpanel --no-vr` stands in for the panel. `session.py --no-start --next-after 0.3` records a three-section test script (a prompt, a two-cue sweep, no hands) in step mode, three parts of 6 s (countdown and hold), about 360 MB once exported. Then `takes.py` exports, `validate.py` checks, and `hub.py` uploads: a dry run by default, or for real to `--repo ID` with `FT_HANDREC_ALLOW_UPLOAD=1`. It prints a summary, deletes its temporary folders (camera images of a room) and stops everything it started, Ctrl+C included. The `--no-vr` panel logs no poses, so `poses.jsonl` is missing there (a warning).

hands/rec/ft_handrec.py

@@ -266,6 +266,9 @@ class Backend(RecorderBackend):
         except hub.HubError:
             return hub.HF_TEST_DATASET if self._consent_kind == "testing" else hub.HF_DATASET
 
+    def _session_label(self, session):
+        return session_label(session)
+
     def _session_dataset(self, session):
         try:
             return hub.dataset_id(hub.session_kind(self.store, session))
@@ -716,6 +719,8 @@ class Backend(RecorderBackend):
             return "exporting a recording"
         if self.uploading:
             return "uploading a recording"
+        if self._upload_queue:
+            return "waiting to upload a recording"
         return ""
 
     def shutdown(self):
@@ -887,7 +892,7 @@ class Backend(RecorderBackend):
 
     @Slot(str)
     def deleteSession(self, session):
-        if self._active_guard(session):
+        if self._active_guard(session) or self._busy_guard(session):
             return
         try:
             self.store.delete_session(session)
@@ -900,7 +905,7 @@ class Backend(RecorderBackend):
     def _freeable(self):
         busy = {self._session_id if self.sessionActive else "", self._export_session if self.exporting else "",
                 (self._upload or {}).get("session", "") if self.uploading else ""}
-        return [f for f in self.store.freeable() if f["id"] not in busy]
+        return [f for f in self.store.freeable() if f["id"] not in busy and not self._queued(f["id"])]
 
     @Property(int, notify=sessionsRelay)
     def freeableCount(self):

hands/rec/main.qml

@@ -85,6 +85,24 @@ Kirigami.ApplicationWindow {
         return n + " " + word + (n === 1 ? "" : "s")
     }
 
+    // A session's label, from its id (the Export page's queue).
+    function sessionLabel(id) {
+        for (const s of backend.sessions)
+            if (s.id === id)
+                return s.label
+        return id
+    }
+    // The session's export can upload by itself once done: logged in, and its dataset takes
+    // contributions (or a dry run).
+    function canUploadAfter(id) {
+        return backend.canUploadAfter && (backend.hubDryRun || backend.uploadAllowedFor(id))
+    }
+    // Waiting to export, or exporting now.
+    function exportPending(id) {
+        return backend.exportSessionId === id && backend.exporting
+               || backend.exportQueue.some(q => q.session === id)
+    }
+
     function show(page) {
         pageStack.clear()
         pageStack.push(page)
@@ -197,9 +215,11 @@ Kirigami.ApplicationWindow {
             close.accepted = false
             confirm.ask("Stop the session?", "A session is recording. Closing stops it; what's recorded so far is kept.",
                         () => { root.quitting = true; Qt.quit() })
-        } else if (backend.uploading && !quitting) {
+        } else if ((backend.uploading || backend.exporting || backend.uploadQueue.length > 0) && !quitting) {
             close.accepted = false
-            confirm.ask("Cancel the upload?", "An upload is running. Closing cancels it; you can start it again later.",
+            confirm.ask("Cancel the exports and uploads?",
+                        "An export or upload is running or waiting (the Export page lists them). Closing cancels "
+                        + "them all; an export cancelled keeps nothing. You can start them again later.",
                         () => { root.quitting = true; Qt.quit() })
         }
     }
@@ -1356,15 +1376,22 @@ Kirigami.ApplicationWindow {
                         icon.name: "media-playback-stop"
                         onClicked: backend.stopSession()
                     }
+                    // Logged in: the export uploads by itself once done (queued behind any other).
+                    // Not yet: the Upload page follows the export, to log in there.
                     Controls.Button {
-                        visible: !backend.sessionActive && backend.sessionId !== "" && !backend.exporting
+                        visible: !backend.sessionActive && backend.sessionId !== ""
+                        enabled: !root.exportPending(backend.sessionId)
                         focusPolicy: Qt.NoFocus
                         text: "Export and upload"
                         icon.name: "cloud-upload"
                         onClicked: {
                             root.chosenSession = backend.sessionId
-                            root.sendAfterExport = true
-                            backend.exportSession(backend.sessionId, false)
+                            if (root.canUploadAfter(backend.sessionId)) {
+                                backend.queueExport(backend.sessionId, false, true)
+                            } else {
+                                root.sendAfterExport = true
+                                backend.exportSession(backend.sessionId, false)
+                            }
                             root.show(exportPage)
                         }
                     }
@@ -1808,6 +1835,12 @@ Kirigami.ApplicationWindow {
                         return s
                 return null
             }
+            // The chosen session is exporting, or waiting to.
+            readonly property bool pending: chosen !== null && root.exportPending(chosen.id)
+            readonly property bool canExport: chosen !== null && !pending && backend.zstdFound && !chosen.active
+                                              && !(backend.uploading && backend.upload.session === chosen.id)
+                                              && backend.uploadQueue.indexOf(chosen.id) < 0
+            Component.onCompleted: if (backend.login.state === "unknown") backend.checkLogin()
 
             Timer {
                 interval: 5000
@@ -1826,6 +1859,15 @@ Kirigami.ApplicationWindow {
                     type: Kirigami.MessageType.Information
                     text: "Exporting your session: the Upload page follows by itself once it's done."
                 }
+                Kirigami.InlineMessage {
+                    Layout.fillWidth: true
+                    visible: backend.uploadAfter.length + backend.uploadQueue.length > 0
+                    position: Kirigami.InlineMessage.Position.Header
+                    type: Kirigami.MessageType.Positive
+                    text: "You can take the headset off and leave it plugged in: it stays awake until the "
+                          + "exports and uploads below are done, and they show here how they went. Keep this "
+                          + "window open: closing it cancels them."
+                }
                 Kirigami.InlineMessage {
                     Layout.fillWidth: true
                     visible: exportView.worn && backend.exporting
@@ -1850,7 +1892,6 @@ Kirigami.ApplicationWindow {
                     model: backend.sessions
                     textRole: "label"
                     valueRole: "id"
-                    enabled: !backend.exporting
                     Component.onCompleted: {
                         const i = indexOfValue(root.chosenSession)
                         currentIndex = i >= 0 ? i : (count > 0 ? 0 : -1)
@@ -1866,32 +1907,73 @@ Kirigami.ApplicationWindow {
                     Kirigami.FormData.label: "Export:"
                     visible: exportView.chosen !== null
                     text: !exportView.chosen ? ""
+                          : exportView.pending ? (backend.exportSessionId === exportView.chosen.id && backend.exporting
+                                                  ? "exporting now" : "waiting its turn, below")
                           : exportView.chosen.export_stale ? "changed since it was exported: export it again"
                           : exportView.chosen.exported ? "ready, " + exportView.chosen.exportText : "not exported yet"
                 }
+                // Exports run one at a time; one asked for meanwhile waits its turn. "Export and
+                // upload" sends it by itself once it's exported.
                 RowLayout {
                     Controls.Button {
-                        text: backend.exporting ? "Exporting…"
-                              : exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
+                        text: exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
                         icon.name: "document-export"
-                        enabled: exportView.chosen !== null && !backend.exporting && backend.zstdFound
-                                 && !exportView.chosen.active
-                        onClicked: backend.exportSession(exportView.chosen.id, false)
+                        enabled: exportView.canExport
+                        onClicked: backend.queueExport(exportView.chosen.id, false, false)
                     }
+                    Controls.Button {
+                        text: "Export and upload"
+                        icon.name: "cloud-upload"
+                        enabled: exportView.canExport && root.canUploadAfter(exportView.chosen.id)
+                        onClicked: backend.queueExport(exportView.chosen.id, false, true)
+                    }
+                }
+                Controls.Label {
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    visible: exportView.chosen !== null && !backend.hubDryRun
+                             && (!backend.canUploadAfter || !backend.uploadAllowedFor(exportView.chosen.id))
+                    wrapMode: Text.Wrap
+                    opacity: 0.7
+                    text: !backend.canUploadAfter ? "To upload by itself once exported, log in on the Upload page first."
+                          : "Contributions aren't open yet, so it can't upload by itself."
+                }
+                Controls.Label {
+                    Layout.maximumWidth: Kirigami.Units.gridUnit * 26
+                    visible: backend.exporting && exportView.chosen !== null && !exportView.pending
+                    wrapMode: Text.Wrap
+                    opacity: 0.7
+                    text: "Another export is running: this one waits its turn."
+                }
+
+                // The running export
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Exporting now"
+                    visible: backend.exporting
+                }
+                RowLayout {
+                    Kirigami.FormData.label: "Session:"
+                    visible: backend.exporting
+                    Controls.Label { text: root.sessionLabel(backend.exportSessionId) }
                     // Shows the moment Export is pressed: the first progress can take a few seconds.
                     Controls.BusyIndicator {
-                        visible: backend.exporting
-                        running: visible
+                        running: backend.exporting
                         implicitWidth: Kirigami.Units.gridUnit * 1.5
                         implicitHeight: implicitWidth
                     }
                     Controls.Button {
-                        visible: backend.exporting
                         text: "Cancel"
                         icon.name: "dialog-cancel"
                         onClicked: { root.sendAfterExport = false; backend.cancelExport() }
                     }
                 }
+                Controls.CheckBox {
+                    visible: backend.exporting
+                    text: "Upload it once exported"
+                    checked: backend.uploadAfter.indexOf(backend.exportSessionId) >= 0
+                    enabled: checked || root.canUploadAfter(backend.exportSessionId)
+                    onToggled: backend.setUploadAfter(backend.exportSessionId, checked)
+                }
                 // Drawn here: the desktop style's ProgressBar paints nothing without a QApplication.
                 Rectangle {
                     Kirigami.FormData.label: "Progress:"
@@ -1929,7 +2011,7 @@ Kirigami.ApplicationWindow {
                     ]
                 }
                 RowLayout {
-                    visible: exportView.chosen !== null && exportView.chosen.exported && !backend.exporting
+                    visible: exportView.chosen !== null && exportView.chosen.exported && !exportView.pending
                     Controls.Button {
                         text: "Upload"
                         icon.name: "go-next"
@@ -1938,6 +2020,8 @@ Kirigami.ApplicationWindow {
                     Controls.Button {
                         text: "Delete export"
                         icon.name: "edit-delete"
+                        enabled: !(backend.uploading && backend.upload.session === exportView.chosen.id)
+                                 && backend.uploadQueue.indexOf(exportView.chosen.id) < 0
                         onClicked: {
                             const sid = exportView.chosen.id
                             confirm.ask("Delete this export?", "The export is deleted; the session's recordings stay.",
@@ -1945,6 +2029,107 @@ Kirigami.ApplicationWindow {
                         }
                     }
                 }
+
+                // Waiting: exports, then uploads
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Waiting to export"
+                    visible: backend.exportQueue.length > 0
+                }
+                Repeater {
+                    model: backend.exportQueue
+                    delegate: RowLayout {
+                        required property var modelData
+                        required property int index
+                        Kirigami.FormData.label: (index + 1) + "."
+                        Controls.Label { text: root.sessionLabel(modelData.session) }
+                        Controls.CheckBox {
+                            text: "then upload"
+                            checked: modelData.upload
+                            enabled: checked || root.canUploadAfter(modelData.session)
+                            onToggled: backend.setUploadAfter(modelData.session, checked)
+                        }
+                        Controls.ToolButton {
+                            icon.name: "list-remove"
+                            text: "Remove"
+                            display: Controls.AbstractButton.IconOnly
+                            Controls.ToolTip.text: text
+                            Controls.ToolTip.visible: hovered
+                            onClicked: backend.unqueueExport(modelData.session)
+                        }
+                    }
+                }
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Uploading"
+                    visible: backend.uploading || backend.uploadQueue.length > 0
+                }
+                RowLayout {
+                    Kirigami.FormData.label: "Now:"
+                    visible: backend.uploading
+                    Controls.Label {
+                        text: root.sessionLabel(backend.upload.session || "") + (backend.upload.text ? ": " + backend.upload.text : "")
+                    }
+                    Controls.Button {
+                        text: "Show"
+                        icon.name: "go-next"
+                        flat: true
+                        onClicked: { root.chosenSession = backend.upload.session; root.show(uploadPage) }
+                    }
+                }
+                Repeater {
+                    model: backend.uploadQueue
+                    delegate: RowLayout {
+                        required property var modelData
+                        required property int index
+                        Kirigami.FormData.label: "Then:"
+                        Controls.Label { text: root.sessionLabel(modelData) }
+                        Controls.ToolButton {
+                            icon.name: "list-remove"
+                            text: "Don't upload it"
+                            display: Controls.AbstractButton.IconOnly
+                            Controls.ToolTip.text: text
+                            Controls.ToolTip.visible: hovered
+                            onClicked: backend.unqueueUpload(modelData)
+                        }
+                    }
+                }
+
+                // How they went, for whoever comes back to the headset
+                Kirigami.Separator {
+                    Kirigami.FormData.isSection: true
+                    Kirigami.FormData.label: "Done"
+                    visible: backend.doneLog.length > 0
+                }
+                Repeater {
+                    model: backend.doneLog
+                    delegate: RowLayout {
+                        required property var modelData
+                        Kirigami.Icon {
+                            source: modelData.error ? "dialog-error" : "checkmark"
+                            implicitWidth: Kirigami.Units.iconSizes.small
+                            implicitHeight: implicitWidth
+                        }
+                        Controls.Label {
+                            Layout.maximumWidth: Kirigami.Units.gridUnit * 24
+                            wrapMode: Text.Wrap
+                            text: modelData.text
+                        }
+                        Controls.ToolButton {
+                            visible: modelData.link !== ""
+                            icon.name: "internet-services"
+                            text: "Open"
+                            onClicked: Qt.openUrlExternally(modelData.link)
+                        }
+                    }
+                }
+                Controls.Button {
+                    visible: backend.doneLog.length > 0
+                    text: "Clear"
+                    icon.name: "edit-clear-history"
+                    flat: true
+                    onClicked: backend.clearDoneLog()
+                }
             }
 
             footer: Controls.Label {
@@ -1953,8 +2138,8 @@ Kirigami.ApplicationWindow {
                 opacity: 0.7
                 text: "Export leaves out the ranges you deleted, compresses the rest and adds a manifest and "
                       + "checksums, in " + backend.exportsDir + ". It runs at the lowest priority "
-                      + "and takes a few minutes per round; the headset stays awake until it's done. Nothing is "
-                      + "uploaded until you press Upload on the Upload page."
+                      + "and takes a few minutes per round, one at a time; the headset stays awake until they're "
+                      + "done. Nothing is uploaded unless you press Export and upload here, or Upload on the Upload page."
             }
         }
     }
@@ -1980,7 +2165,7 @@ Kirigami.ApplicationWindow {
             readonly property string blocked: {
                 if (session === "") return "Choose an export"
                 if (chosen && chosen.export_stale) return "Export the session again first"
-                if (backend.exporting) return "Wait for the export to finish"
+                if (root.exportPending(session)) return "Wait for its export to finish"
                 if (backend.uploading) return ""
                 if (backend.hubDryRun) return ""
                 if (!backend.uploadAllowedFor(session)) return "Contributions aren't open yet"
@@ -1993,6 +2178,14 @@ Kirigami.ApplicationWindow {
             Connections {
                 target: backend
                 function onSessionsChanged() { uploadView.refreshInfo() }
+                // An upload started by itself (queued on the Export page): show it.
+                function onUploadChanged() {
+                    const i = backend.uploading ? exportBox.indexOfValue(backend.upload.session) : -1
+                    if (i >= 0 && i !== exportBox.currentIndex) {
+                        exportBox.currentIndex = i
+                        exportBox.keep = backend.upload.session
+                    }
+                }
             }
             Component.onCompleted: {
                 refreshInfo()
@@ -2072,8 +2265,19 @@ Kirigami.ApplicationWindow {
                             textRole: "label"
                             valueRole: "id"
                             enabled: !backend.uploading
-                            Component.onCompleted: currentIndex = Math.max(0, indexOfValue(root.chosenSession))
-                            onActivated: root.chosenSession = currentValue
+                            // An export that finishes meanwhile adds a row (newest first): stay on the
+                            // same export, the uploading one while an upload runs.
+                            property string keep: ""
+                            Component.onCompleted: {
+                                currentIndex = Math.max(0, indexOfValue(root.chosenSession))
+                                keep = currentValue || ""
+                            }
+                            onActivated: root.chosenSession = keep = currentValue
+                            onModelChanged: Qt.callLater(() => {
+                                const i = indexOfValue(backend.uploading ? backend.upload.session : keep)
+                                currentIndex = i >= 0 ? i : (count > 0 ? Math.max(0, currentIndex) : -1)
+                                keep = currentValue || ""
+                            })
                         }
                         Controls.Label {
                             opacity: 0.7
@@ -2187,6 +2391,14 @@ Kirigami.ApplicationWindow {
                               + "\">its page</a>. Upload checks the export, opens your pull request, then sends the files."
                         onLinkActivated: link => Qt.openUrlExternally(link)
                     }
+                    Controls.Label {
+                        Layout.fillWidth: true
+                        visible: backend.uploadQueue.length + backend.uploadAfter.length > 0
+                        wrapMode: Text.Wrap
+                        opacity: 0.7
+                        text: "Queued on the Export page to upload after this one, by themselves: "
+                              + root.plural(backend.uploadQueue.length + backend.uploadAfter.length, "export") + "."
+                    }
                     RowLayout {
                         Controls.Button {
                             text: backend.uploading ? "Uploading…" : backend.hubDryRun ? "Upload (dry run)" : "Upload"

hands/rec/tests/test_export_queue.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""Exports queue up and run one at a time; one set to upload uploads by itself once it's
+exported, after any upload before it (recorder/window.py queueExport, 2026-10-17: "users can
+queue exports, and trigger an auto upload when the export is finished"). Needs PySide6 (the dev
+container); skipped without it.
+
+  python3 hands/rec/tests/test_export_queue.py
+"""
+import os
+import shutil
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(HERE))
+import takes  # noqa: E402
+from test_validate import make_session  # noqa: E402
+
+A, B, C = "20261003-184200", "20261003-190000", "20261003-193000"
+
+
+class ExportQueueTest(unittest.TestCase):
+    def setUp(self):
+        try:
+            import ft_handrec
+            from recorder import window
+        except ImportError as e:
+            self.skipTest(f"no PySide6: {e}")
+        self.base = tempfile.mkdtemp(prefix="handrec-export-queue-test-")
+        for sid in (A, B, C):
+            make_session(self.base, sid=sid)
+        self.store = takes.Store(self.base)
+        self.backend = ft_handrec.Backend(self.store, {"dry_run": True}, hub_dry_run=True, ignore_cameras=True)
+        self.backend._stay_awake = lambda why, on: None
+        # Nothing runs: exports start and end by hand (finish), uploads start a fake hub.py.
+        self.started = []    # sessions whose export started, in order
+        self.sent = []       # sessions whose upload started, in order
+        real_start = self.backend._start_export
+
+        def start_export(session, keep_notes):
+            real_start(session, keep_notes)
+            self.started.append(session)
+        self.backend._start_export = start_export
+        popen = mock.patch.object(window.subprocess, "Popen", side_effect=self.popen)
+        popen.start()
+        self.addCleanup(popen.stop)
+        thread = mock.patch.object(self.backend, "_thread")
+        thread.start()
+        self.addCleanup(thread.stop)
+
+    def tearDown(self):
+        shutil.rmtree(self.base, ignore_errors=True)
+
+    def popen(self, argv, **kw):
+        self.sent.append(argv[argv.index("upload") + 1])
+        return mock.Mock()
+
+    def finish_export(self, error=""):
+        self.backend._on_export_finished("" if error else self.base, error)
+
+    def finish_upload(self, ok=True):
+        self.backend._on_upload_finished({"done": {"dry_run": True}} if ok else {"error": {"kind": "hub", "text": "no"}})
+
+    def test_exports_run_in_turn(self):
+        for sid in (A, B, C):
+            self.backend.queueExport(sid, False, False)
+        self.assertEqual(self.started, [A])
+        self.assertEqual([q["session"] for q in self.backend.exportQueue], [B, C])
+        self.finish_export()
+        self.assertEqual(self.started, [A, B])
+        self.finish_export("disk full")   # a failed one doesn't hold up the rest
+        self.assertEqual(self.started, [A, B, C])
+        self.finish_export()
+        self.assertFalse(self.backend.exporting)
+        self.assertEqual(self.sent, [])   # none was set to upload
+
+    def test_queued_once(self):
+        self.backend.queueExport(A, False, False)
+        self.backend.queueExport(B, False, False)
+        self.backend.queueExport(B, False, True)   # again, now to upload: the same place in the queue
+        self.backend.queueExport(A, False, False)
+        self.assertEqual(self.backend.exportQueue, [{"session": B, "upload": True}])
+        self.assertEqual(self.started, [A])
+
+    def test_upload_after_export(self):
+        self.backend.queueExport(A, False, True)
+        self.assertEqual(self.sent, [])
+        self.finish_export()
+        self.assertEqual(self.sent, [A])
+        self.assertEqual(self.backend.upload["session"], A)
+
+    def test_no_upload_after_a_failed_or_cancelled_export(self):
+        self.backend.queueExport(A, False, True)
+        self.finish_export("disk full")
+        self.backend.queueExport(B, False, True)
+        self.backend.cancelExport()
+        self.finish_export("cancelled")
+        self.assertEqual(self.sent, [])
+        self.assertTrue(self.backend.doneLog[0]["error"])
+
+    def test_uploads_wait_their_turn(self):
+        for sid in (A, B, C):
+            self.backend.queueExport(sid, False, sid != B)
+        self.finish_export()              # A exported: it uploads
+        self.finish_export()              # B exported: not set to upload
+        self.finish_export()              # C exported: waits for A's upload
+        self.assertEqual(self.sent, [A])
+        self.assertEqual(self.backend.uploadQueue, [C])
+        self.finish_upload(ok=False)      # a failed upload doesn't hold up the next
+        self.assertEqual(self.sent, [A, C])
+        self.finish_upload()
+        self.assertFalse(self.backend.uploading)
+        self.assertEqual(self.backend.uploadQueue, [])
+
+    def test_upload_after_can_change(self):
+        self.backend.queueExport(A, False, False)
+        self.backend.queueExport(B, False, True)
+        self.backend.setUploadAfter(A, True)
+        self.backend.setUploadAfter(B, False)
+        self.finish_export()
+        self.finish_upload()
+        self.finish_export()
+        self.assertEqual(self.sent, [A])
+
+    def test_unqueue(self):
+        self.backend.queueExport(A, False, False)
+        self.backend.queueExport(B, False, True)
+        self.backend.unqueueExport(B)
+        self.finish_export()
+        self.assertEqual(self.started, [A])
+        self.assertEqual(self.sent, [])
+
+    def test_queued_session_isnt_deleted(self):
+        self.backend.queueExport(A, False, False)
+        self.backend.queueExport(B, False, False)
+        self.backend.deleteSession(B)
+        self.assertTrue(os.path.isdir(os.path.join(self.base, "sessions", B)))
+
+    def test_no_second_upload_of_the_same_export(self):
+        self.backend.queueExport(A, False, True)
+        with mock.patch.object(self.backend, "uploadInfo", return_value={"previous": {"export_sha": "x"}, "uploads": 1}):
+            self.finish_export()
+        self.assertEqual(self.sent, [])
+        self.assertIn("uploaded before", self.backend.doneLog[-1]["text"])
+
+
+if __name__ == "__main__":
+    unittest.main()

hands/rec/tests/test_upload_during_export.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""An export and an upload of different sessions go on together; the same session waits
+(recorder/window.py exportSession and startUpload, 2026-10-17: an upload was refused while
+any other session exported). Needs PySide6 (the dev container); skipped without it.
+
+  python3 hands/rec/tests/test_upload_during_export.py
+"""
+import os
+import shutil
+import sys
+import tempfile
+import threading
+import unittest
+from unittest import mock
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(HERE))
+import takes  # noqa: E402
+from test_validate import make_session  # noqa: E402
+
+A, B = "20261003-184200", "20261003-190000"
+
+
+class UploadDuringExportTest(unittest.TestCase):
+    def setUp(self):
+        try:
+            import ft_handrec
+            from recorder import window
+        except ImportError as e:
+            self.skipTest(f"no PySide6: {e}")
+        self.window = window
+        self.base = tempfile.mkdtemp(prefix="handrec-upload-export-test-")
+        make_session(self.base, sid=A)
+        make_session(self.base, sid=B)
+        self.store = takes.Store(self.base)
+        self.backend = ft_handrec.Backend(self.store, {"dry_run": True}, hub_dry_run=True, ignore_cameras=True)
+        self.backend._stay_awake = lambda why, on: None
+
+    def tearDown(self):
+        shutil.rmtree(self.base, ignore_errors=True)
+
+    def exporting(self, session):
+        self.backend._export_cancel = threading.Event()
+        self.backend._export_session = session
+
+    def uploading(self, session):
+        self.backend._upload_proc = object()
+        self.backend._upload = {"session": session}
+
+    def upload(self, session):
+        """startUpload(session): whether it started hub.py."""
+        with mock.patch.object(self.window.subprocess, "Popen") as popen, \
+                mock.patch.object(self.backend, "_thread"):
+            self.backend.startUpload(session, False)
+        return popen.called
+
+    def export(self, session):
+        """exportSession(session): whether it started the export."""
+        with mock.patch.object(self.backend, "_thread") as thread:
+            self.backend.exportSession(session, False)
+        return thread.called
+
+    def test_upload_while_another_exports(self):
+        self.exporting(A)
+        self.assertTrue(self.upload(B))
+        self.assertTrue(self.backend.uploading)
+
+    def test_upload_waits_for_its_own_export(self):
+        self.exporting(A)
+        self.assertFalse(self.upload(A))
+
+    def test_export_while_another_uploads(self):
+        self.uploading(B)
+        self.assertTrue(self.export(A))
+        self.assertEqual(self.backend.exportSessionId, A)
+
+    def test_no_export_of_the_uploading_session(self):
+        self.uploading(B)
+        self.assertFalse(self.export(B))
+
+    def test_one_export_at_a_time(self):
+        self.exporting(A)
+        self.assertFalse(self.export(B))
+        self.assertEqual(self.backend.exportQueue, [{"session": B, "upload": False}])   # it waits its turn
+
+
+if __name__ == "__main__":
+    unittest.main()

hands/tests/skeleton_test.cpp

@@ -153,6 +153,36 @@ int main() {
         CHECK(std::fabs(left.len[5] - got.len[5]) < 1e-9);
     }
 
+    // a left hand stuck fitted as a right one (the mirror won while its fingers were straight)
+    // goes back to the model's side once its fingers curl a little: the mirror follows them only
+    // by bending its own backward, and at 20 degrees it fits well enough to stay (2026-10-19,
+    // before the guard: right in every frame, knuckles at -28 degrees)
+    {
+        Skeleton sk;
+        const double D = M_PI / 180;
+        const Shape truth = Skeleton::canonical(1.0);
+        Rot R0 = exp_rot({0.3, 0.2, 0.1}), fr[21];
+        const V3 t0{-0.12, -0.2, -0.38};
+        double q0[kParams] = {};
+        for (int c = 0; c < 4; ++c) q0[6 + 5 * c + 1] = 20 * D, q0[6 + 5 * c + 3] = 20 * D, q0[6 + 5 * c + 4] = 10 * D;
+        V3 pts[21];
+        Skeleton::pose(truth, -1, R0, t0, q0, pts, fr);
+        Fit f;
+        f.valid = true, f.side = 1, f.flipped = true, f.ns = 1'000'000'000;
+        int64_t ns = f.ns;
+        int asRight = 0;
+        for (int i = 0; i < 10; ++i) {
+            ns += 33'000'000;
+            sk.update(f, pts, false, 0.1, 2, 0.0015, nullptr, {0, 0, 0}, 1.0, ns);
+            asRight += f.side == 1;
+        }
+        double mcp = 0;
+        for (int c = 0; c < 4; ++c) mcp += f.qf[6 + 5 * c + 1] / 4;
+        std::printf("flipped left hand, curled: %d of 10 frames as right, mean knuckle bend %.0f deg\n", asRight, mcp / D);
+        CHECK(asRight == 0 && f.valid && !f.flipped);
+        CHECK(mcp > 15 * D);
+    }
+
     // the elbow: behind and below a hand held in front, 2.8 palms away
     {
         const V3 wrist{0.15, -0.25, -0.35}, knuckle{0.15, -0.22, -0.43};

hands/track/main.cpp

@@ -599,6 +599,63 @@ int main(int argc, char **argv) {
     tracker.set_hold(reacquire > 0 ? hold : 0.3);
     tracker.set_skeleton(skeleton);
     std::map<std::string, std::vector<uint8_t>> pixels;
+    // A mono set is the side pair's newer frames (all the mono cameras' without both sides) with
+    // the upper pair's from the same moment. XRService keeps the uppers in step with the sides,
+    // but after a colour module hot plug they may stay out of step ("Set upper camera driver not
+    // in-sync", 2026-10-17: 40 fps against the sides' 30, no dark frames, for good). Waiting for
+    // all four to meet then left 6-7 sets a second; out of step, sets hold the side pair alone
+    // (until the uppers keep meeting them again).
+    std::vector<std::string> lead, follow;
+    for (auto &[name, i] : index)
+        (!have_sides || name == "slam_left" || name == "slam_right" ? lead : follow).push_back(name);
+    bool follow_in_step = true;
+    double follow_rate = 1;      // how often the uppers met a set lately (a moving average)
+    uint64_t prev_t0 = 0, prev_t1 = 0;
+    // a camera's frame from within 3 ms of the set's frames (taken t0 to t1), newer than after, or 0
+    auto follow_frame = [&](int i, uint64_t t0, uint64_t t1, uint64_t after) -> uint64_t {
+        uint64_t n = ring.latest(i);
+        for (int k = 0; k < 4 && n > after; ++k, --n) {
+            fh_ring_slot_t m;
+            if (!ring.meta(i, n, &m)) continue;
+            if (m.capture_ns + 3'000'000 < t1) break;   // older than the set: earlier ones are too
+            if (m.capture_ns <= t0 + 3'000'000) return n;
+        }
+        return 0;
+    };
+    // Did camera i take a frame within 3 ms of t0 to t1? 1 yes, 0 no, -1 can't tell: the ring
+    // (4 frames) no longer reaches back that far, as when a step took long.
+    auto met_set = [&](int i, uint64_t t0, uint64_t t1) -> int {
+        const uint64_t latest = ring.latest(i);
+        for (uint64_t n = latest; n > 0 && latest - n < 4; --n) {
+            fh_ring_slot_t m;
+            if (!ring.meta(i, n, &m)) return -1;
+            if (m.capture_ns + 3'000'000 < t1) return 0;
+            if (m.capture_ns <= t0 + 3'000'000) return 1;
+        }
+        return -1;
+    };
+    // A set used or passed by: did the uppers meet the one before it? (theirs have come by now)
+    auto note_step = [&](uint64_t t0, uint64_t t1) {
+        int met = 1;
+        if (!follow.empty() && prev_t1)
+            for (const std::string &name : follow) met = std::min(met, met_set(index[name], prev_t0, prev_t1));
+        if (!follow.empty() && prev_t1 && met >= 0) {
+            follow_rate += 0.1 * (met - follow_rate);
+            if (follow_in_step && follow_rate < 0.5) {
+                follow_in_step = false;
+                for (const std::string &name : follow) tracker.drop_camera(name);
+                std::printf("upper cameras out of step with the side ones (XRService's \"upper camera driver not in-sync\", "
+                            "after a colour module hot plug): tracking with the side pair alone. Taking the headset off "
+                            "and putting it back on may bring them back.\n");
+                std::fflush(stdout);
+            } else if (!follow_in_step && follow_rate > 0.95) {
+                follow_in_step = true;
+                std::printf("upper cameras in step with the side ones again: tracking with all four\n");
+                std::fflush(stdout);
+            }
+        }
+        prev_t0 = t0, prev_t1 = t1;
+    };
     // The flag buffer (see the top), and its request file.
     std::unique_ptr<FlagBuffer> flags;
     const std::string flag_file = run_dir() + "/flag";
@@ -894,37 +951,53 @@ int main(int argc, char **argv) {
         uint64_t tmin = UINT64_MAX, dq = 0;
 
         if (mono_driven) {
-            // a new frame set: every mono camera has a newer frame, taken at the same moment
+            // a new frame set: the side pair has newer frames, taken at the same moment (see lead)
             std::map<std::string, uint64_t> latest;
             bool ready = true;
-            for (auto &[name, i] : index) {
-                latest[name] = ring.latest(i);
+            for (const std::string &name : lead) {
+                latest[name] = ring.latest(index[name]);
                 ready = ready && latest[name] > last[name];
             }
             if (!ready) {
                 std::this_thread::sleep_for(std::chrono::milliseconds(2));
                 continue;
             }
-            // not needed at the current rate, and not recorded: skip it without copying images
-            if (track && !rec && !g_record) {
-                uint64_t t0 = UINT64_MAX, t1 = 0;
-                bool ok = true;
-                for (auto &[name, i] : index) {
-                    fh_ring_slot_t meta;
-                    ok = ok && ring.meta(i, latest[name], &meta);
-                    if (ok) t0 = std::min(t0, meta.capture_ns), t1 = std::max(t1, meta.capture_ns);
-                }
-                if (ok && t1 - t0 <= 3'000'000 && t0 < next_ns) {
-                    for (auto &[name, i] : index) last[name] = latest[name];
-                    continue;
-                }
+            uint64_t t0 = UINT64_MAX, t1 = 0, lead_dq = 0;
+            bool met = true;
+            for (const std::string &name : lead) {
+                fh_ring_slot_t meta;
+                met = met && ring.meta(index[name], latest[name], &meta);
+                if (met) t0 = std::min(t0, meta.capture_ns), t1 = std::max(t1, meta.capture_ns), lead_dq = std::max(lead_dq, meta.dqbuf_ns);
             }
+            if (!met || t1 - t0 > 3'000'000) {   // one is a frame ahead: wait for the other
+                std::this_thread::sleep_for(std::chrono::milliseconds(1));
+                continue;
+            }
+            // the uppers' frames from the same moment: in step, theirs may be a few ms behind
+            std::map<std::string, uint64_t> with;
+            if (follow_in_step)
+                for (const std::string &name : follow)
+                    if (const uint64_t n = follow_frame(index[name], t0, t1, last[name])) with[name] = n;
+            if (follow_in_step && with.size() < follow.size() && mono_ns() < lead_dq + 8'000'000) {
+                std::this_thread::sleep_for(std::chrono::milliseconds(1));
+                continue;
+            }
+            const bool full = with.size() == follow.size();
+            // not needed at the current rate, and not recorded: skip it without copying images
+            if (track && !rec && !g_record && t0 < next_ns) {
+                note_step(t0, t1);
+                for (const std::string &name : lead) last[name] = latest[name];
+                for (auto &[name, n] : with) last[name] = n;
+                continue;
+            }
+            for (auto &[name, n] : with) latest[name] = n;
             std::vector<SetFrame> frames;
             uint64_t tmax = 0;
             bool ok = true;
-            for (auto &[name, i] : index) {
+            for (auto &[name, n] : latest) {
+                const int i = index[name];
                 fh_ring_slot_t meta;
-                ok = ok && ring.read(i, latest[name], pixels[name], &meta);
+                ok = ok && ring.read(i, n, pixels[name], &meta);
                 if (!ok) break;
                 const auto &c = ring.camera(i);
                 images[name] = {pixels[name].data(), int(c.width), int(c.height), int(c.width)};
@@ -937,7 +1010,8 @@ int main(int argc, char **argv) {
                 std::this_thread::sleep_for(std::chrono::milliseconds(1));
                 continue;
             }
-            for (auto &[name, i] : index) last[name] = latest[name];
+            note_step(t0, t1);
+            for (auto &[name, n] : latest) last[name] = n;
             if (g_record && !rec) {
                 g_record = 0;
                 char name[64];
@@ -957,7 +1031,7 @@ int main(int argc, char **argv) {
                     std::printf("recording done: %zu sets, %zu dropped\n", n, d);
                     std::fflush(stdout);
                     if (!track) break;
-                } else if (due) {
+                } else if (due && full) {   // a set without the uppers isn't recorded
                     for (auto &[name, i] : dark) {   // the newest dark and color frames, as they are
                         fh_ring_slot_t meta;
                         const uint64_t n = ring.latest(i);
@@ -1090,6 +1164,9 @@ int main(int argc, char **argv) {
                             s.handoff_miss, s.dups, s.splits, s.created, s.merged, s.forgotten,
                             !rec ? "" : ("  recorded " + std::to_string(rec->written()) + " dropped " +
                                          std::to_string(rec->dropped())).c_str());
+            if (!follow.empty() && !follow_in_step)
+                std::printf("        upper cameras out of step with the side ones (met %.0f%% of sets lately): side pair alone\n",
+                            100 * follow_rate);
             if (flags) {
                 const FlagBuffer::Stats fs = flags->stats();
                 std::printf("        flag buffer: %zu sets, %zu crops (%zu ghosts) in %.1f MB, %.0f us a crop, %zu dropped, %zu flags written%s%s\n",
@@ -1136,5 +1213,28 @@ int main(int argc, char **argv) {
         grip.release(int64_t(mono_ns()));
         gestures.write(pinch, grip, mono_ns(), &middle);
     }
+    // ft-camd stopped: a new one comes when XRService moves the cameras (a colour module hot plug)
+    // or restarts. Wait for it here and start again on its cameras, rather than exit and wait out
+    // the unit's RestartSec as well. Its ring is a new file (camd.c unlinks the old one).
+    if (exit_code == 2 && publish && track && !g_stop) {
+        rec.reset();   // writes out what's queued
+        Ring next;
+        ino_t next_ino = ring_st.st_ino;
+        for (const uint64_t t = mono_ns(); !g_stop && mono_ns() - t < 15'000'000'000ull;
+             std::this_thread::sleep_for(std::chrono::milliseconds(100))) {
+            struct stat st;
+            std::string e;
+            if (next_ino == ring_st.st_ino && stat(ring_path.c_str(), &st) == 0 && st.st_ino != ring_st.st_ino &&
+                next.open(ring_path.c_str(), e))
+                next_ino = st.st_ino;
+            if (next_ino != ring_st.st_ino && next.alive()) {
+                std::printf("ft-camd is back: starting again on its cameras\n");
+                std::fflush(stdout);
+                execv("/proc/self/exe", argv);
+                std::perror("execv /proc/self/exe");
+                break;
+            }
+        }
+    }
     return exit_code;
 }

hands/track/ringplay.cpp

@@ -3,11 +3,15 @@
 // run after run. For A/B tests of how the tracker runs.
 //
 //   ft-ringplay DIR --ring PATH [--from S] [--to S] [--loop] [--cpus 0,1]
+//               [--upper-skew MS] [--upper-drift MS_PER_S]
 //
 // Frames are stamped as they're published, so the tracker's latency figures stay
 // meaningful. Cameras carry their calibration name and no device node (ft-hands maps
 // them by name), so a recording made with the right names needs no --swap-sides.
 // Dark frames (<name>_dk) are skipped. Needs no root: the ring is an ordinary file.
+// --upper-skew and --upper-drift stamp the upper cameras' frames MS (plus MS_PER_S a second,
+// modulo a 30 fps frame) later than the sides', as after a colour module hot plug that left
+// them out of step (hands/README.md, "Frame sets").
 #include "record.h"
 
 extern "C" {
@@ -20,6 +24,7 @@ extern "C" {
 #include <unistd.h>
 
 #include <algorithm>
+#include <cmath>
 #include <cerrno>
 #include <csignal>
 #include <cstdio>
@@ -87,12 +92,13 @@ bool is_dark(const char *name) {
 
 int main(int argc, char **argv) {
     if (argc < 2 || argv[1][0] == '-') {
-        std::fprintf(stderr, "usage: %s DIR --ring PATH [--from S] [--to S] [--loop] [--cpus 0,1]\n", argv[0]);
+        std::fprintf(stderr, "usage: %s DIR --ring PATH [--from S] [--to S] [--loop] [--cpus 0,1] [--upper-skew MS] "
+                             "[--upper-drift MS_PER_S]\n", argv[0]);
         return 1;
     }
     const std::string dir = argv[1];
     std::string ring_path;
-    double from = 0, to = 1e9;
+    double from = 0, to = 1e9, upper_skew_ms = 0, upper_drift_ms = 0;
     bool loop = false;
     std::vector<int> cpus = {0, 1};
     for (int i = 2; i < argc; ++i) {
@@ -102,6 +108,8 @@ int main(int argc, char **argv) {
         else if (a == "--from" && more) from = std::atof(argv[++i]);
         else if (a == "--to" && more) to = std::atof(argv[++i]);
         else if (a == "--loop") loop = true;
+        else if (a == "--upper-skew" && more) upper_skew_ms = std::atof(argv[++i]);
+        else if (a == "--upper-drift" && more) upper_drift_ms = std::atof(argv[++i]);
         else if (a == "--cpus" && more) {
             cpus.clear();
             for (char *p = argv[++i]; *p;) {
@@ -185,6 +193,7 @@ int main(int argc, char **argv) {
     std::fflush(stdout);
 
     uint64_t published = 0, rounds = 0;
+    const uint64_t t_play = clock_ns(CLOCK_MONOTONIC);
     for (;;) {
         // one pass over [from, to]: each set goes out at its recorded offset from the first
         while (have && set_time(cams) - rec0 < from && !g_stop) {
@@ -202,6 +211,7 @@ int main(int argc, char **argv) {
                 nanosleep(&ts, nullptr);
             }
             const uint64_t raw = clock_ns(CLOCK_MONOTONIC_RAW), mono = clock_ns(CLOCK_MONOTONIC);
+            const double skew_ms = std::fmod(upper_skew_ms + upper_drift_ms * (mono - t_play) * 1e-9, 1000.0 / 30);
             for (size_t r = 0; r < pub.size(); ++r) {
                 fh_ring_cam_t &rc = hdr->cams[r];
                 if (px[pub[r]].size() != size_t(rc.width) * rc.height) continue;
@@ -212,6 +222,7 @@ int main(int argc, char **argv) {
                 std::memcpy(reinterpret_cast<uint8_t *>(s + 1), px[pub[r]].data(), px[pub[r]].size());
                 s->frame = n;
                 s->capture_ns = raw;   // taken now, as a live camera's frame would be
+                if (!std::strncmp(rc.name, "upper", 5)) s->capture_ns += uint64_t(std::max(0.0, skew_ms) * 1e6);
                 s->dqbuf_ns = mono;
                 s->publish_ns = mono;
                 __atomic_store_n(&s->seq, 2 * n + 2, __ATOMIC_RELEASE);

hands/track/skeleton.cpp

@@ -620,6 +620,29 @@ void Skeleton::update(Fit &f, const V3 pts[21], bool right, double right_score,
     double crms[5] = {};
     double cost = solve(s, side, pts, sl, sd, eye, prev, p_.iterations, R, t, q, &rms, p_.limits, crms, p_.warm > 0);
     if (cost < 0) return;
+    // A mirror image follows fingers that bend toward the palm only by bending its own backward
+    // (the knuckles allow -35 degrees): a fit as the other side than the model's call that does,
+    // on average over the four fingers, is the wrong side however well it fits. Such a fit goes
+    // back to the model's side at once, and a mirror that needs it never wins.
+    auto backward = [](const double *qq) {
+        double m = 0;
+        for (int c = 0; c < 4; ++c) m += qq[6 + 5 * c + 1];
+        return m / 4 < -10 * D;
+    };
+    if (side != model_side && backward(q)) {
+        Rot R2;
+        V3 t2;
+        double q2[kParams] = {}, rms2 = 0, crms2[5] = {};
+        const Shape s2 = shape(model_side, scale);
+        const double cost2 = solve(s2, model_side, pts, sl, sd, eye, nullptr, p_.iterations, R2, t2, q2, &rms2, p_.limits,
+                                   crms2, p_.warm > 0);
+        if (cost2 >= 0) {
+            side = model_side, s = s2, R = R2, t = t2, rms = rms2, cost = cost2;
+            std::copy(q2, q2 + kParams, q);
+            std::copy(crms2, crms2 + 5, crms);
+            f.votes = 0;
+        }
+    }
     bool better = false;
     if (rms > 0.015) {
         Rot R2;
@@ -629,7 +652,7 @@ void Skeleton::update(Fit &f, const V3 pts[21], bool right, double right_score,
         double crms2[5] = {};
         const double cost2 = solve(s2, -side, pts, sl, sd, eye, continuing && f.side == -side ? f.q : nullptr,
                                    p_.iterations, R2, t2, q2, &rms2, p_.limits, crms2, p_.warm > 0);
-        better = cost2 >= 0 && cost2 < 0.5 * cost;
+        better = cost2 >= 0 && cost2 < 0.5 * cost && !(-side != model_side && backward(q2));
         if (better && ++f.votes >= kFlipFrames) {
             side = -side, s = s2, R = R2, t = t2, rms = rms2, cost = cost2;
             std::copy(q2, q2 + kParams, q);

host/linux/README.md

@@ -49,7 +49,7 @@ sudo ufw allow Vibepollo
 
 ## Good to know
 
-- The resolution switch and the HDR guard of Frametop's build are Windows-only for now. On Linux, a monitor keeps its own resolution and the stream scales it.
+- The resolution switch, the HDR guard and the Frame's microphone are Windows-only for now. On Linux, a monitor keeps its own resolution and the stream scales it, and Send my microphone does nothing.
 - Vibepollo runs as system services. `sudo systemctl status vibepollo-session-controller.service vibepollo.service` shows them, and `sudo journalctl -u vibepollo-session-controller.service -u vibepollo.service --since '-10 minutes'` shows their log. Don't start `vibepollo.service` yourself: the session controller starts it when you log in.
 - Settings, the Web UI login and pairings are in `/var/lib/vibepollo`.
 - Vibepollo's [Linux install guide](https://github.com/Frametop/frametop-vibepollo/blob/frametop-2.0.0-3/docs/linux/install.md) has the details and troubleshooting.

host/windows/README.md

@@ -37,6 +37,7 @@ Frametop's build is Vibepollo 2.0.0 with a few changes, from [Frametop/frametop-
 - An HDR monitor's HDR is off while it streams, and back on after. An HDR desktop in the stream looked washed out.
 - A virtual display that goes away leaves your monitor layout as it was.
 - The Web UI doesn't stall after a client disconnects.
+- It can take the Frame's microphone, for calls and voice chat on the PC. Turn on Send my microphone on the PC's card in Remote Displays. While its displays stream, apps on the PC can record it as "Microphone (Steam Streaming Microphone)". That's Steam's virtual microphone: the build installs it from Steam's drivers the first time if it's missing, and leaves your default microphone as it was. It only goes over the encrypted stream.
 
 ## Stream over a Steam Link dongle
 

input-settings/ft_input_settings.py

@@ -4,7 +4,8 @@
 A Kirigami (QML) app with a Python backend. It runs in the dev container and talks
 to the input relay over its control socket (@frametop_relay):
   - Devices: every USB/Bluetooth mouse and keyboard, a live activity light to
-    identify them, and a role for each (3D pointer, pass through, ignore).
+    identify them, and a role for each (3D pointer, pass through, keyboard + 3D pointer for a
+    keyboard with a touchpad, ignore).
   - Buttons: press a button or key on a pointer device, then pick an action.
   - Controllers: the same for the Frame controllers' buttons, minus the gaze actions (gaze
     mode is a mouse feature). They're read by the pointer helper through SteamVR input
@@ -54,7 +55,7 @@ import sys
 import threading
 import time
 
-from PySide6.QtCore import Property, QObject, QSocketNotifier, QTimer, QUrl, Signal, Slot
+from PySide6.QtCore import Property, QObject, QProcess, QSocketNotifier, QTimer, QUrl, Signal, Slot
 from PySide6.QtGui import QGuiApplication, QIcon
 from PySide6.QtQml import QQmlApplicationEngine
 from PySide6.QtQuickControls2 import QQuickStyle
@@ -166,6 +167,9 @@ HAND_INTERACTIONS = [
     ("POINTER_RAY_READY", True, "6", "Laser", "Laser only when about to pinch",
      "A hand's laser shows once its thumb and index are within 6 cm, and goes when they open past 10 cm. "
      "Off: whenever the hand is raised and pointing."),
+    ("POINTER_RAY_RAISED", False, "30", "Laser", "No laser for a low hand",
+     "A hand's laser shows only with the knuckles within 30 cm under your eyes, and goes past 40 cm. "
+     "Off: a low hand points too. Hands typing on a keyboard never do."),
     ("POINTER_RAY_FIST_CURL", True, "1.1", "Laser", "No laser or clicks in a fist",
      "Closing the hand drops its laser, and a pinch made while the hand closes into a fist doesn't click."),
     ("POINTER_PINCH_NO_RAY", False, "1", "Laser", "Pinch with no laser clicks at the pointer",
@@ -219,7 +223,10 @@ VR_KEYBOARD_MODES = {
     "button": "Only with a mapped mouse or controller button",
     "never": "Never",
 }
-ROLE_LABELS = {"pointer": "3D pointer", "passthrough": "Pass through", "ignore": "Ignore"}
+# "combo" splits a device by node (the relay's node_role): a keyboard's touchpad is the pointer, its
+# keys pass through. The Devices page offers it only for devices with a mouse and a keyboard node.
+ROLE_LABELS = {"pointer": "3D pointer", "passthrough": "Pass through", "combo": "Keyboard + 3D pointer",
+               "ignore": "Ignore"}
 # Frame controller buttons the pointer helper can read (pointer/helper/vrbuttons.h). The
 # system button stays SteamVR's.
 CONTROLLER_BUTTONS = {
@@ -410,7 +417,7 @@ def write_conf_value(key, value):
         f.write("\n".join(lines) + "\n")
 
 
-def host(*cmd):
+def host(*cmd, timeout=20):
     """Run a command on the SteamOS host (we live in the dev container), on the host's user
     bus. The app itself has the bus of the session it was opened from (ft-input-settings),
     in the Frametop desktop a private one from dbus-run-session; distrobox-host-exec run on
@@ -419,7 +426,7 @@ def host(*cmd):
     runner = ["distrobox-host-exec"] if shutil.which("distrobox-host-exec") else []
     env = dict(os.environ, DBUS_SESSION_BUS_ADDRESS=f"unix:path=/run/user/{os.getuid()}/bus")
     try:
-        return subprocess.run(runner + list(cmd), capture_output=True, text=True, timeout=20, env=env)
+        return subprocess.run(runner + list(cmd), capture_output=True, text=True, timeout=timeout, env=env)
     except (OSError, subprocess.TimeoutExpired) as e:
         return subprocess.CompletedProcess(cmd, 1, "", str(e))
 
@@ -481,6 +488,7 @@ class Backend(QObject):
         self._combo_tap_code = None  # that modifier's key as it is (Right Ctrl: 97)
         self._voice = {}  # ft-voiced's status, {} when it isn't running
         self._voice_enabled = False  # its service starts at login (systemctl, refreshVoice)
+        self._model_fetching = ""  # the whisper model voice/models.sh is downloading
         self._vr = {}  # the helper's vrstatus, {} when it doesn't answer
         self._vr_at = 0.0
         self._gaze = {}  # ft-gazed's status, {} when it isn't running
@@ -631,8 +639,10 @@ class Backend(QObject):
                     self._combo_tap = None  # a mouse button: no tap
             elif t == "event":
                 self.activity.emit(msg["id"])
+                # Not a Keyboard + 3D pointer device's keys: its keyboard node passes through, so
+                # its button map never sees them.
                 if (self._capture_id and msg["id"] == self._capture_id and msg["type"] == "key"
-                        and msg["value"] == 1):
+                        and msg["value"] == 1 and msg.get("role") != "passthrough"):
                     code = int(msg["code"])
                     self._capture_id = ""
                     self.captured.emit(code, self.codeName(code))
@@ -672,6 +682,8 @@ class Backend(QObject):
                 d["name"] = n["name"].replace(" Mouse", "")  # the Z3's nodes are "Z3 Mouse"/"Z3 Keyboard"
         for d in grouped.values():
             d["connected"] = True
+            if rules.get(d["id"], {}).get("role") == "combo":
+                d["role"] = "combo"  # its nodes say pointer and passthrough
         for device_id in set(rules) | {i for i, b in bindings.items() if b}:
             if device_id in grouped:
                 continue
@@ -1204,18 +1216,21 @@ class Backend(QObject):
         self.message.emit("Gaze dot: " + ("always shown" if on else "shown only while the mouse moves it"), False)
 
     # --- key combinations ("key_bindings") ---
-    def comboName(self, combo):
+    def comboName(self, combo, action=None):
+        """A combination's name. A lone modifier is a tap, or held with dictate_hold (the relay
+        starts dictation on its press and stops it on the release)."""
         parts = [int(c) for c in combo.split("+") if c.isdigit()]
+        tap = "" if action == "dictate_hold" else " tap"
         if len(parts) == 1 and parts[0] in MODIFIER_NAMES:
-            return MODIFIER_NAMES[parts[0]] + " tap"
+            return MODIFIER_NAMES[parts[0]] + tap
         if len(parts) == 1 and parts[0] in RIGHT_MODIFIER_NAMES:
-            return RIGHT_MODIFIER_NAMES[parts[0]] + " tap"
+            return RIGHT_MODIFIER_NAMES[parts[0]] + tap
         return "+".join(MODIFIER_NAMES.get(c) or self.codeName(c).removeprefix("KEY_").title() for c in parts)
 
     @Property("QVariantList", notify=mappingsChanged)
     def keyShortcuts(self):
         bound = key_bindings(read_json(RULES_PATH))
-        return [{"combo": c, "label": self.comboName(c), "action": a, "actionLabel": action_label(a)}
+        return [{"combo": c, "label": self.comboName(c, a), "action": a, "actionLabel": action_label(a)}
                 for c, a in sorted(bound.items())]
 
     @Property("QVariantList", constant=True)
@@ -1361,9 +1376,102 @@ class Backend(QObject):
 
     @Property("QVariantList", notify=mappingsChanged)
     def dictationKeys(self):
-        return [{"combo": c, "label": self.comboName(c), "actionLabel": action_label(a)}
+        return [{"combo": c, "label": self.comboName(c, a), "actionLabel": action_label(a)}
                 for c, a in sorted(key_bindings(read_json(RULES_PATH)).items()) if a in VOICE_ACTIONS]
 
+    @Property("QVariantList", constant=True)
+    def dictationKeyModes(self):
+        return [{"value": "dictate", "text": "Tap to start, tap again to stop"},
+                {"value": "dictate_hold", "text": "Hold to talk"}]
+
+    @Property(str, notify=mappingsChanged)
+    def dictationKeyMode(self):
+        """dictate or dictate_hold when all of dictation's keys do the same, else ""."""
+        modes = {a for a in key_bindings(read_json(RULES_PATH)).values() if a in VOICE_ACTIONS}
+        return modes.pop() if len(modes) == 1 else ""
+
+    @Slot(str)
+    def setDictationKeyMode(self, action):
+        """Every dictation key toggles (dictate) or is held to talk (dictate_hold)."""
+        if action not in VOICE_ACTIONS:
+            return
+        rules = read_json(RULES_PATH)
+        bound = key_bindings(rules)
+        if not any(a in VOICE_ACTIONS for a in bound.values()):
+            self.message.emit("Dictation has no keys: add one on the Keyboard page", True)
+            return
+        rules["key_bindings"] = {c: action if a in VOICE_ACTIONS else a for c, a in bound.items()}
+        self._save_rules(rules)
+        self.message.emit("Dictation keys: " + ("hold to talk" if action == "dictate_hold"
+                                                 else "tap to start, tap again to stop"), False)
+
+    @Property("QVariantList", constant=True)
+    def voiceLanguages(self):
+        return [{"value": k, "text": v} for k, v in voice_settings.LANGUAGES.items()]
+
+    @Slot(str)
+    def setVoiceLanguage(self, language):
+        if language in voice_settings.LANGUAGES:
+            s = voice_settings.load()
+            s["language"] = language
+            want = voice_settings.model_for(s)
+            self._set_voice("language", language, "Dictation language: " + voice_settings.LANGUAGES[language]
+                            + ("" if want == s["model"] else f" (with whisper {want})"))
+            if not voice_settings.model_installed(want):
+                self._fetch_model(want)
+
+    @Property("QVariantList", notify=voiceChanged)
+    def voiceModels(self):
+        """Whisper models for the Voice page: English-only ones only while the language is English."""
+        english = voice_settings.load()["language"] == "en"
+        return [{"value": name, "installed": voice_settings.model_installed(name),
+                 "text": f"{name} ({mb} MB{', English only' if langs == 'en' else ''}"
+                         f"{'' if voice_settings.model_installed(name) else ', downloads when chosen'})"}
+                for name, (_, mb, langs) in voice_settings.WHISPER_MODELS.items() if english or langs != "en"]
+
+    @Property(str, notify=voiceChanged)
+    def voiceModel(self):
+        """The model ft-voiced uses (voice_settings.model_for)."""
+        return voice_settings.model_for(voice_settings.load())
+
+    @Property(str, notify=voiceChanged)
+    def voiceModelFetching(self):
+        return self._model_fetching
+
+    @Slot(str)
+    def setVoiceModel(self, name):
+        if name not in voice_settings.WHISPER_MODELS or self._model_fetching:
+            return
+        if voice_settings.model_installed(name):
+            self._set_voice("model", name, f"Whisper model: {name}")
+        else:
+            self._fetch_model(name, then_choose=True)
+
+    def _fetch_model(self, name, then_choose=False):
+        """voice/models.sh model NAME, on the host in the background (up to 574 MB); then the
+        model is chosen. ft-voiced restarts the whisper service when its model changes."""
+        if self._model_fetching:
+            return
+        self._model_fetching = name
+        self.voiceChanged.emit()
+        self.message.emit(f"Downloading whisper {name} ({voice_settings.WHISPER_MODELS[name][1]} MB)…", False)
+
+        def job():
+            result = host(os.path.join(REPO, "voice", "models.sh"), "model", name, timeout=3600)
+            if result.returncode != 0 or not voice_settings.model_installed(name):
+                err = (result.stderr.strip() or result.stdout.strip() or "no answer").splitlines()
+                self.message.emit(f"Couldn't download whisper {name}: " + (err[-1] if err else "")[:160], True)
+            elif then_choose:
+                s = voice_settings.load()
+                s["model"] = name
+                voice_settings.save(s)
+                self.message.emit(f"Whisper model: {name}, downloaded", False)
+            else:
+                self.message.emit(f"Whisper {name} downloaded", False)
+            self._model_fetching = ""  # last: the page waits for this
+            self.voiceChanged.emit()
+        threading.Thread(target=job, daemon=True).start()
+
     @Property("QVariantList", notify=voiceChanged)
     def voiceCommands(self):
         commands, _ = voice_commands.parse(voice_settings.load()["commands"])
@@ -1458,6 +1566,13 @@ class Backend(QObject):
         """The full calibration in the panel fixed to the headset (gaze/gazecheck.py)."""
         self._gaze_check("calibrate", "Calibration: look at each dot in the headset; right click or Meta+K stops")
 
+    @Slot()
+    def openTour(self):
+        """Frametop Tour (tour/ft_tour.py), in this container as it runs."""
+        ok, _pid = QProcess.startDetached(sys.executable, [os.path.join(REPO, "tour", "ft_tour.py")])
+        if not ok:
+            self.message.emit("Couldn't start Frametop Tour", True)
+
     def _gaze_check(self, command, done):
         if self._send(command, GAZED):
             self._check_asked = time.monotonic()

input-settings/main.qml

@@ -26,7 +26,9 @@ Kirigami.ApplicationWindow {
             Kirigami.Action { text: "Ignored panels"; icon.name: "view-hidden"; onTriggered: root.show(ignorePage) },
             Kirigami.Action { text: "Gaze"; icon.name: "view-visible"; onTriggered: root.show(gazePage) },
             Kirigami.Action { text: "Hands"; icon.name: "transform-browse"; onTriggered: root.show(handsPage) },
-            Kirigami.Action { text: "Bluetooth"; icon.name: "preferences-system-bluetooth"; onTriggered: root.show(bluetoothPage) }
+            Kirigami.Action { text: "Bluetooth"; icon.name: "preferences-system-bluetooth"; onTriggered: root.show(bluetoothPage) },
+            Kirigami.Action { separator: true },
+            Kirigami.Action { text: "Frametop Tour"; icon.name: "help-about"; onTriggered: backend.openTour() }
         ]
     }
 
@@ -371,7 +373,12 @@ Kirigami.ApplicationWindow {
                         }
 
                         Controls.ComboBox {
-                            model: backend.roles
+                            // Keyboard + 3D pointer only where it splits something: a mouse node and a keyboard
+                            // node. A device that isn't connected has no nodes to tell, so it's offered there too.
+                            model: backend.roles.filter(r => r.value !== "combo" || row.modelData.role === "combo"
+                                                        || !row.modelData.connected
+                                                        || (row.modelData.kinds.indexOf("mouse") >= 0
+                                                            && row.modelData.kinds.indexOf("keyboard") >= 0))
                             textRole: "text"
                             valueRole: "value"
                             Component.onCompleted: currentIndex = indexOfValue(row.modelData.role)
@@ -423,7 +430,7 @@ Kirigami.ApplicationWindow {
                 }
             ]
             property string deviceId: pointerDevices.length > 0 ? pointerDevices[Math.max(0, deviceBox.currentIndex)].id : ""
-            property var pointerDevices: backend.devices.filter(d => d.role === "pointer")
+            property var pointerDevices: backend.devices.filter(d => d.role === "pointer" || d.role === "combo")
             property int capturedCode: -1
             property string capturedName: ""
             property bool capturing: false
@@ -488,49 +495,49 @@ Kirigami.ApplicationWindow {
                             onClicked: { backend.setMapping(bpage.deviceId, bpage.capturedCode, newAction.currentValue); bpage.capturedCode = -1 }
                         }
                     }
-                }
 
-                Kirigami.Heading {
-                    visible: bpage.rows.length > 0
-                    level: 3
-                    text: "Current mappings"
-                }
+                    Kirigami.Separator {
+                        visible: bpage.rows.length > 0
+                        Kirigami.FormData.isSection: true
+                        Kirigami.FormData.label: "Current mappings"
+                    }
 
-                Repeater {
-                    model: bpage.rows
-                    delegate: RowLayout {
-                        required property var modelData
-                        Layout.fillWidth: true
-                        spacing: Kirigami.Units.largeSpacing
-                        Controls.Label {
-                            text: modelData.name
-                            font.family: "monospace"
-                            Layout.preferredWidth: Kirigami.Units.gridUnit * 10
+                    Repeater {
+                        model: bpage.rows
+                        delegate: RowLayout {
+                            required property var modelData
+                            Layout.fillWidth: true
+                            spacing: Kirigami.Units.largeSpacing
+                            Controls.Label {
+                                text: modelData.name
+                                font.family: "monospace"
+                                Layout.preferredWidth: Kirigami.Units.gridUnit * 10
+                            }
+                            Controls.ComboBox {
+                                model: backend.actions
+                                textRole: "text"
+                                valueRole: "value"
+                                Layout.preferredWidth: Kirigami.Units.gridUnit * 13
+                                Component.onCompleted: currentIndex = indexOfValue(modelData.action)
+                                onActivated: backend.setMapping(bpage.deviceId, modelData.code, currentValue)
+                            }
+                            Controls.Label {
+                                text: modelData.custom ? (modelData.isDefault ? "changed" : "custom") : "default"
+                                opacity: 0.6
+                                Layout.preferredWidth: Kirigami.Units.gridUnit * 4
+                            }
+                            // Remove: your own binding goes away (the button passes through again).
+                            // Reset: a changed built-in binding goes back to its default.
+                            // Unbind: an untouched built-in binding is set to "Do nothing".
+                            Controls.Button {
+                                text: modelData.custom ? (modelData.isDefault ? "Reset" : "Remove") : "Unbind"
+                                icon.name: modelData.custom ? (modelData.isDefault ? "edit-undo" : "edit-delete-remove")
+                                                            : "list-remove"
+                                onClicked: modelData.custom ? backend.removeMapping(bpage.deviceId, modelData.code)
+                                                            : backend.unbind(bpage.deviceId, modelData.code)
+                            }
+                            Item { Layout.fillWidth: true }
                         }
-                        Controls.ComboBox {
-                            model: backend.actions
-                            textRole: "text"
-                            valueRole: "value"
-                            Layout.preferredWidth: Kirigami.Units.gridUnit * 13
-                            Component.onCompleted: currentIndex = indexOfValue(modelData.action)
-                            onActivated: backend.setMapping(bpage.deviceId, modelData.code, currentValue)
-                        }
-                        Controls.Label {
-                            text: modelData.custom ? (modelData.isDefault ? "changed" : "custom") : "default"
-                            opacity: 0.6
-                            Layout.preferredWidth: Kirigami.Units.gridUnit * 4
-                        }
-                        // Remove: your own binding goes away (the button passes through again).
-                        // Reset: a changed built-in binding goes back to its default.
-                        // Unbind: an untouched built-in binding is set to "Do nothing".
-                        Controls.Button {
-                            text: modelData.custom ? (modelData.isDefault ? "Reset" : "Remove") : "Unbind"
-                            icon.name: modelData.custom ? (modelData.isDefault ? "edit-undo" : "edit-delete-remove")
-                                                        : "list-remove"
-                            onClicked: modelData.custom ? backend.removeMapping(bpage.deviceId, modelData.code)
-                                                        : backend.unbind(bpage.deviceId, modelData.code)
-                        }
-                        Item { Layout.fillWidth: true }
                     }
                 }
 
@@ -724,7 +731,7 @@ Kirigami.ApplicationWindow {
             title: "Keyboard"
             // Pass-through keyboards connected now: with "no_keyboard", the keyboard waits for none.
             // A program's uinput keyboard (a remote desktop's, say) doesn't count.
-            property var keyboards: backend.devices.filter(d => d.connected && d.role === "passthrough"
+            property var keyboards: backend.devices.filter(d => d.connected && (d.role === "passthrough" || d.role === "combo")
                                                                 && d.kinds.indexOf("keyboard") >= 0 && !d.uinput)
 
             Kirigami.FormLayout {
@@ -896,6 +903,18 @@ Kirigami.ApplicationWindow {
                         icon.name: "input-keyboard"
                         onClicked: root.show(keyboardPage)
                     }
+                    Controls.ComboBox {
+                        Kirigami.FormData.label: "Keys do:"
+                        model: backend.dictationKeyModes
+                        textRole: "text"
+                        valueRole: "value"
+                        enabled: backend.dictationKeys.length > 0
+                        displayText: backend.dictationKeyMode ? currentText : "Some tap, some are held"
+                        readonly property string want: backend.dictationKeyMode
+                        onWantChanged: currentIndex = indexOfValue(want)
+                        Component.onCompleted: currentIndex = indexOfValue(want)
+                        onActivated: backend.setDictationKeyMode(currentValue)
+                    }
                     Repeater {
                         model: backend.voiceButtons
                         delegate: RowLayout {
@@ -947,6 +966,41 @@ Kirigami.ApplicationWindow {
                         onActivated: backend.setVoiceControllerMode(currentValue)
                     }
 
+                    Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Language" }
+                    Controls.ComboBox {
+                        Kirigami.FormData.label: "You speak:"
+                        model: backend.voiceLanguages
+                        textRole: "text"
+                        valueRole: "value"
+                        Layout.preferredWidth: Kirigami.Units.gridUnit * 12
+                        readonly property string want: vpage.vs.language
+                        onWantChanged: currentIndex = indexOfValue(want)
+                        Component.onCompleted: currentIndex = indexOfValue(want)
+                        onActivated: backend.setVoiceLanguage(currentValue)
+                    }
+                    Controls.ComboBox {
+                        Kirigami.FormData.label: "Whisper model:"
+                        model: backend.voiceModels
+                        textRole: "text"
+                        valueRole: "value"
+                        enabled: !backend.voiceModelFetching
+                        displayText: backend.voiceModelFetching ? "Downloading " + backend.voiceModelFetching + "…" : currentText
+                        Layout.preferredWidth: Kirigami.Units.gridUnit * 18
+                        readonly property string want: backend.voiceModel
+                        onWantChanged: currentIndex = indexOfValue(want)
+                        onModelChanged: currentIndex = indexOfValue(want)
+                        Component.onCompleted: currentIndex = indexOfValue(want)
+                        onActivated: backend.setVoiceModel(currentValue)
+                    }
+                    Controls.Label {
+                        text: vpage.vs.language === "en"
+                              ? "Moonshine types English as you talk. Whisper types it when \"As you talk\" is off, and hears the wake word's commands."
+                              : "Whisper types other languages, slower than Moonshine types English. Bigger models hear better and take longer; large-v3-turbo can make VR stutter."
+                        wrapMode: Text.Wrap
+                        opacity: 0.7
+                        Layout.maximumWidth: Kirigami.Units.gridUnit * 24
+                    }
+
                     Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Typing" }
                     Controls.Switch {
                         Kirigami.FormData.label: "As you talk:"
@@ -994,8 +1048,8 @@ Kirigami.ApplicationWindow {
                     opacity: 0.7
                     text: "Speech is recognised on the headset; nothing leaves it. Text is typed where you type: the "
                           + "desktop window you clicked last, or Steam and games. It waits while you hold Ctrl, Alt, "
-                          + "Shift or Meta, so letters don't turn into shortcuts. Recordings under 2 seconds are "
-                          + "dropped. A key works only on its own: with another modifier held it's a shortcut, not "
+                          + "Shift or Meta, so letters don't turn into shortcuts. A key held to talk "
+                          + "dictates while it's down; pressing another key with it cancels. A key works only on its own: with another modifier held it's a shortcut, not "
                           + "dictation. Controller buttons still do what they do in games. After the wake word, a "
                           + "command goes to voice/hooks/on-command, or runs a program you set in "
                           + "~/.config/frametop-voice.json (\"commands\"; see voice/README.md)."
@@ -1306,7 +1360,8 @@ Kirigami.ApplicationWindow {
                         text: (backend.pauseGesture.first === "" ? "No gesture: a mapped button, a key combination, or ft-pause still work."
                                : "Pauses and resumes Frametop, in games too. The game still gets the presses, so pick buttons it "
                                  + "doesn't use this way. Two buttons count as together when both go down within a third of a "
-                                 + "second, and twice means within 0.7 s.")
+                                 + "second, and twice means within 0.7 s. A gesture with a button on each controller works with "
+                                 + "one controller too: while only one is connected, its button alone, clicked twice, does it.")
                               + (gmpage.st.relay && backend.pauseGesture.first !== "" && !gmpage.st.gestureReader
                                  ? " Not reading the controllers right now (SteamVR isn't running?)." : "")
                     }
@@ -1633,6 +1688,8 @@ Kirigami.ApplicationWindow {
                     text: gpage.status.calibration_samples > 0
                           ? gpage.status.calibration_samples + " points ("
                             + (gpage.status.tracker === "own" ? "Own tracker, " + gpage.status.calibration_made
+                               + (gpage.status.own_mapped && gpage.status.own_mapped.length === 1
+                                  ? ", " + gpage.status.own_mapped[0] + " eye only" : "")
                                : gpage.status.kind === "eyes" ? gpage.status.model + ", each eye" : gpage.status.model) + ")"
                           : "none yet: use Calibrate"
                 }

input/game_pause.py

@@ -26,7 +26,10 @@ Toggled by:
     games (the pointer helper only reads those outside games).
   - A controller gesture, by default both thumbsticks clicked together twice ("pause_gesture").
     It's read from vrserver's web socket (vrws.py), which works in games and takes nothing from
-    them, so the game sees the clicks too.
+    them, so the game sees the clicks too. Once it goes off, it's ignored for GESTURE_COOLDOWN:
+    on 2026-10-16 a resume was followed by a pause 2 s later, which left Frametop paused.
+    A gesture on both controllers, with only one of them connected, is that controller's button
+    alone, clicked twice (one_controller).
   - input/ft-pause on|off|toggle|status, which uses the relay's control socket
     ("pause on|off|toggle|?").
   - VR games, with "pause_auto" (on by default): the pointer helper says "vrgame 1|0" as a SteamVR
@@ -39,7 +42,7 @@ Settings in ~/.config/frametop-input.json (Frametop Input Settings, Game optimiz
 (bool), "pause_gesture" ({"buttons": one or two controller buttons, "presses": 1 or 2}, or null
 for none; one button always takes two presses), "pause_desktop" ("hide" or "close"), and
 "pause_sound" (a sound on pause and resume; bool). The state outlives a relay restart in
-STATE_PATH.
+STATE_PATH, and a desktop that starts while paused is told (screens_seen), so it hides too.
 """
 import json
 import os
@@ -77,6 +80,7 @@ RESUME_DELAY = 5.0  # after a VR game ends: loading the next one can end one sce
 GAME_STALE = 12.0  # the helper repeats "vrgame" every 5 s; silence this long means no game (or no SteamVR)
 CHORD_WINDOW = 0.3  # a two-button gesture's buttons go down within this of each other
 DOUBLE_WINDOW = 0.7  # a double press's second press comes within this of the first
+GESTURE_COOLDOWN = 3.0  # once the gesture goes off, it's ignored this long (a triple press, a repeat)
 LOOKUP_EVERY = 30.0  # the controllers are looked up again this often (and sooner when they may have changed)
 LOOKUP_GAP = 3.0  # but not more often than this for a message from a device it doesn't know
 
@@ -103,6 +107,17 @@ def gesture_of(rules, buttons):
     return chosen, 1 if len(chosen) == 2 and spec.get("presses") == 1 else 2
 
 
+def one_controller(buttons, sides):
+    """A two-button gesture with a button on each controller, while only one of the two is connected
+    (`sides`: the sides there are): that one's button alone, pressed twice, so a single controller
+    can still pause. None otherwise."""
+    by_side = {b.split("/", 1)[0]: b for b in buttons}
+    if len(buttons) != 2 or len(by_side) != 2:
+        return None
+    there = [b for side, b in by_side.items() if side in sides]
+    return (there[0],) if len(there) == 1 else None
+
+
 def settings_of(rules, buttons):
     return {"auto": rules.get("pause_auto", True) is not False,
             "gesture": gesture_of(rules, buttons),
@@ -114,13 +129,14 @@ class Gesture:
     """The pause gesture, from button presses and releases. A press of a two-button gesture is
     both buttons going down within CHORD_WINDOW of each other, so a button held for a while (to
     sprint, say) and the other clicked doesn't count. Two presses within DOUBLE_WINDOW make a
-    double press."""
+    double press. Once it goes off, it's ignored for GESTURE_COOLDOWN."""
 
     def __init__(self, buttons, presses):
         self.buttons, self.presses = tuple(buttons), presses
         self.down = {}  # button -> when it went down
         self.first = None  # when the first press of a double press was
         self.fired = None
+        self.ignored = None  # a gesture the cooldown dropped: how long after the last one (for the log)
 
     def reset(self):
         self.down.clear()
@@ -142,8 +158,9 @@ class Gesture:
             self.first = now
             return False
         self.first = None
-        if self.fired is not None and now - self.fired < 1.0:
-            return False  # a triple press is still one
+        if self.fired is not None and now - self.fired < GESTURE_COOLDOWN:
+            self.ignored = now - self.fired
+            return False
         self.fired = now
         return True
 
@@ -303,6 +320,7 @@ class ControllerWatch(threading.Thread):
             last_poll = 0.0
             self.tracker = TouchTracker()
             touch_was = False
+            alone = None  # the one-controller gesture in use (one_controller), or None
             while True:
                 g = self.current()
                 touch = self.touch_feed
@@ -341,8 +359,17 @@ class ControllerWatch(threading.Thread):
                         self.log(f"{'pause gesture' if g is not None else 'touch feed'}: following "
                                  f"{' and '.join(sorted(sides.values())) or 'no'} "
                                  f"controller{'s' if len(sides) != 1 else ''} on SteamVR's web socket")
-                wanted = {} if g is None else \
-                    {b: (b.split("/", 1)[0], f"/input/{b.split('/', 1)[1]}/click") for b in g.buttons}
+                single = None if g is None else one_controller(g.buttons, set(sides.values()))
+                if single is None:
+                    if alone is not None and g is not None:
+                        self.log(f"pause gesture: both controllers are there again, back to {' and '.join(g.buttons)}")
+                    alone = None
+                elif alone is None or alone.buttons != single:
+                    alone = Gesture(single, 2)
+                    self.log(f"pause gesture: only one controller, so {single[0]} clicked twice")
+                use = alone or g
+                wanted = {} if use is None else \
+                    {b: (b.split("/", 1)[0], f"/input/{b.split('/', 1)[1]}/click") for b in use.buttons}
                 for text in ws.messages(timeout=1.0):
                     device = device_of(text)
                     if device and device not in sides:
@@ -363,12 +390,15 @@ class ControllerWatch(threading.Thread):
                         self.to_helper(self.tracker.feed(msg.get("device"), components))
                     for button, (s, component) in wanted.items():
                         if s == side and component in components:
-                            if g.feed(button, bool(components[component]), time.monotonic()):
+                            if use.feed(button, bool(components[component]), time.monotonic()):
                                 self.log("pause gesture")
                                 try:
                                     self.sock.sendto(b"pause toggle gesture", RELAY)
                                 except OSError:
                                     pass
+                            elif g.ignored is not None:
+                                self.log(f"pause gesture ignored: {g.ignored:.1f} s after the last one")
+                                g.ignored = None
         finally:
             ws.close()
 
@@ -508,6 +538,7 @@ class GamePause:
         self.hands_always = False  # HANDS_MODE=always: hand tracking runs through games (set_hands)
         self.dashboard = False  # the SteamVR dashboard is open (the helper's "vrdash")
         self.parked = None  # hand tracking parked, as last applied (None: not yet)
+        self.screens_pid = None  # the ft-screens that last said where typing goes (screens_seen)
         self.screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)
         self.worker = Worker(log, self.save)
         self.watch = ControllerWatch(log)
@@ -568,6 +599,19 @@ class GamePause:
         except OSError:
             pass  # the desktop isn't running
 
+    def screens_seen(self, pid):
+        """From the relay: ft-screens `pid` said where typing goes, as it does every second. One
+        that started while paused never heard "pause on" and shows its screens, while the relay
+        keeps the 3D mouse off: on 2026-10-16 an update restarted the desktop and the relay, the
+        relay came back paused from STATE_PATH, and the desktop looked on with a dead pointer.
+        So a new ft-screens hears the pause."""
+        if not pid or pid == self.screens_pid:
+            return
+        self.screens_pid = pid
+        if self.paused:
+            self.log(f"telling ft-screens ({pid}) that Frametop is paused")
+            self.tell_screens(True)
+
     def tell_helper(self):
         """The pointer helper's hand interaction runs in a VR game only while Frametop isn't paused
         (toggled back on during the game) or the dashboard is open: "paused 1|0", on a change and

input/input-relay.py

@@ -10,11 +10,14 @@ the physical devices as they come and go.
 Every USB or Bluetooth mouse and keyboard is a candidate. A physical device is
 identified by its Bluetooth address (EVIOCGUNIQ) or USB bus:vendor:product:name,
 so all its event nodes share one role (the Swiftpoint Z3 has a mouse node and a
-keyboard node for its extra buttons). Roles, from ~/.config/frametop-input.json
-(written by the Frametop Input Settings app):
+keyboard node for its extra buttons), except with "combo". Roles, from
+~/.config/frametop-input.json (written by the Frametop Input Settings app):
   pointer      grabbed; drives the universal 3D mouse (default for devices with a mouse node)
   passthrough  keys go to the desktop; grabbed only while typing goes there, otherwise only observed,
                for the key combinations below (default for keyboards)
+  combo        a keyboard with a touchpad (its touchpad is a mouse node): mouse nodes are pointer,
+               keyboard nodes passthrough, as if they were two devices. Never a default: the
+               Z3's keyboard node has to stay with its mouse, and nothing on the nodes tells the two apart
   ignore       not grabbed, only observed for identification in the settings app
 Buttons and keys of pointer devices go through a per-device map to actions
 (left, right, middle, back, scroll_up, scroll_down, dashboard, recenter,
@@ -50,7 +53,10 @@ either side, here Ctrl+Alt+G): the combination does the action, and its last key
 A modifier on its own ("125": Meta) is a tap: pressed with no other modifier held, and released
 with no other key, mouse button, or scroll in between; the desktop gets an F24 press before its
 release, so Plasma's launcher doesn't open on a Meta tap that's bound. A tap's left-hand code is
-either side's; a right-hand code ("97": Right Ctrl) is that key's alone, and comes first. A rules
+either side's; a right-hand code ("97": Right Ctrl) is that key's alone, and comes first. A lone
+modifier bound to dictate_hold is hold to talk instead: dictation starts on its press (alone) and
+stops on its release, and a key, mouse button or scroll in between makes it a shortcut, which
+cancels that dictation quietly ("cancel keyboard quiet"). A rules
 file without "key_bindings" gets DEFAULT_KEY_BINDINGS (Meta tap: steam_menu, Meta+J: gaze_left,
 Meta+K: gaze_right, Meta+Shift+F: float_toggle), plus a Right Ctrl tap for dictate once Frametop
 Voice is installed (VOICE_KEY_BINDINGS); one with its own, even an empty one, doesn't. The float,
@@ -67,6 +73,9 @@ reaches games; see pointer/helper/vrbuttons.h).
 
 In gaze mode outside games, the helper keeps the pointer ("gazeawake 1", repeated every 5
 seconds; "gazeawake 0" or silence ends it): the pointer isn't released when the mouse is idle.
+When something else let the pointer go, the gaze takes it back and the helper says "gazewoke"
+(the pointer is on again here too). The hides here say why: "hide toggle" (pointer_toggle: the
+gaze leaves it off until something wakes it), "hide idle", "hide pause".
 Typing on a keyboard sends the helper "typing" (at most 4 times a second): it takes no hand
 pinches right after a key, since typing touches thumb to index like a pinch.
 
@@ -79,16 +88,21 @@ Back) goes there too, and ft-screens gives it the screen the pointer is on, not
 typing goes to (it releases the button itself when the pointer leaves the screens, they
 hide, or Frametop pauses). Other buttons and keys from KEY_OK up don't go there.
 Typing goes to the panel clicked last, and ft-screens says which ("keyboard
-desktop|steam" on the control socket, every second). While it's the desktop,
+desktop|steam SHOWN" on the control socket, every second; SHOWN 1|0, whether the screens
+show). While it's the desktop,
 pass-through keyboards are grabbed, so gamescope, which reads every keyboard itself,
 doesn't type them into its focused app too. Without word from ft-screens for 3 seconds
 they're released.
 
 Dictation (Frametop Voice) types here: its text comes as "type TEXT" on the control socket, only
 from this user, and goes out key by key, TYPE_STEP apart, where typing goes: to ft-screens for the
-desktop, else on the virtual keyboard, which SteamVR passes on to Steam and games (US layout,
-ASCII; the voice service turns the rest into ASCII or drops it). A character waits while a
-modifier is held on a keyboard, which would make it a shortcut (Ctrl+letter).
+desktop, else on the virtual keyboard, which SteamVR passes on to Steam and games. On the
+desktop each character is the keys that type it on the desktop's keyboard layout (Plasma's
+kxkbrc in Frametop's desktop config, worked out with libxkbcommon), Shift and AltGr included, so a
+German QWERTZ desktop gets "z" from the Z key; Steam and games get a US layout. A character the
+layout has no key for (a dead key's accent, another script) is dropped, and the log says which.
+A character waits while a modifier is held on a keyboard, which would make it a shortcut
+(Ctrl+letter).
 
 Frametop's keyboard (ft-screens' key panel): the desktop's input method (input/ft-textinput)
 says "textfield 1|0" when a text field on the desktop gains or loses keyboard focus, and
@@ -100,6 +114,18 @@ doesn't count), "button" (only the keyboard_toggle action opens it), or "never"
 open when the text field loses focus, until its Close key, keyboard_toggle, or a layout reset
 (ft-layout reset) closes it.
 
+Full Keyboard (keyboard/, TaiKeid's, run with --frametop; keyboard/FRAMETOP.md) takes over from
+it while it runs. It says "kbhello OPEN" on the control socket every second, only taken from
+this user, and gets "kb DESKTOP SURFACE" back: whether typing goes to the desktop, and whether
+the screens show with Frametop not paused (it may show over them with the dashboard closed).
+The relay tells ft-screens "fullkb OPEN", and while that's fresh ft-screens passes its
+"vrkeyboard" commands on as "kbpanel show|hide", which the relay sends Full Keyboard as "kb
+show|hide". Its typing for the desktop comes as "kbkey CODE 0|1" and "kbtext TEXT", only from
+the address that said hello: they go in order with dictation's typing, keys as they are (a held
+modifier doesn't hold them back), text through the desktop's layout like dictation's, and
+unlogged. Its keys come up where they went down, and when it stops (no hello for 3 seconds),
+the ones it holds come up.
+
 Volume keys, from every device that has them (the headset's own buttons included),
 are handled here: wpctl steps the default output. Nothing else may see a volume key,
 because gamescope aborts on one when no window has keyboard focus, which ends the
@@ -115,6 +141,12 @@ pointer helper decides per side whether a hand holds a controller or is bare
 state as "paused 1|0" (hands work in a VR game only with the dashboard open or Frametop toggled
 back on).
 
+The gaze calibration panel (gaze/panel/ft-gazepanel; the pointer helper says "calpanel 1" while
+it's up): Enter on a pass-through keyboard (either Enter, with no modifier held) takes the
+panel's dot instead of typing: "calkey 1" to the helper, which answers the panel as it does a
+left click. Its repeats and its release go nowhere either. That works while the relay has the
+keyboard (typing goes to the desktop); while typing goes to Steam, Steam sees the Enter too.
+
 Frametop can pause for VR games (game_pause.py: by hand, with a controller gesture, or by itself
 while a game runs). Paused, it stops the services that cost the game CPU and GPU, and the relay
 plays a plain one: the pointer devices feed the virtual mouse and keyboard as without pointer
@@ -132,10 +164,12 @@ Control socket (abstract datagram @frametop_relay, JSON replies to the sender):
   reload            re-read both config files, re-apply roles, tell the helper
   vrcapture <s>     take every controller button for s seconds (0: stop), so the settings
                     app can capture one; watchers see them as events with id frame_controller
-  vrbtn, vrhello, gazeawake   from the pointer helper (above)
+  vrbtn, vrhello, gazeawake, gazewoke   from the pointer helper (above)
   vrgame 1|0        from the pointer helper: a VR game runs (on a change and every 5 s), for pausing
   vrdash 1|0        from the pointer helper (hand interaction): the dashboard is open, so hand
                     tracking parked for a game wakes (game_pause.py, PARK_PATH)
+  calpanel 1|0      from the pointer helper: a gaze calibration panel is up (renewed every second,
+                    lapses 3 s after the last), so Enter answers it (above)
   textfield 1|0     from the desktop's input method (above)
   pause on|off|toggle [reason]   pause Frametop or resume it (input/ft-pause; the gesture reader)
   pause ?           the pause state, as {"t": "pause", ...}
@@ -155,6 +189,9 @@ kernel's evdev and uinput interfaces.
 import array
 import atexit
 import collections
+import configparser
+import ctypes
+import ctypes.util
 import errno
 import fcntl
 import json
@@ -179,6 +216,7 @@ SCROLLS = {0x06, REL_WHEEL, 0x0B, 0x0C}  # REL_HWHEEL, REL_WHEEL and their _HI_R
 BTN_LEFT, BTN_RIGHT, BTN_MIDDLE, BTN_SIDE, BTN_EXTRA = 0x110, 0x111, 0x112, 0x113, 0x114
 BTN_MOUSE, BTN_TASK = 0x110, 0x117  # mouse buttons, the first and the last
 KEY_LEFTMETA, KEY_RIGHTMETA = 125, 126
+KEY_ENTER, KEY_KPENTER = 28, 96
 KEY_MUTE, KEY_VOLUMEDOWN, KEY_VOLUMEUP = 113, 114, 115
 # Volume keys are remapped to KEY_MACRO28, KEY_MACRO29 and KEY_MACRO30: above 255, so X11
 # can't carry them, and bound to nothing in the default keymap.
@@ -259,12 +297,22 @@ KEY_F24 = 194  # sent to the desktop with a Meta combination (see the top)
 # Key combinations ("key_bindings"): modifiers, each side's code folded into the left one's.
 MODIFIERS = {29: 29, 97: 29, 42: 42, 54: 42, 56: 56, 100: 56, 125: 125, 126: 125}
 KEY_LEFTSHIFT = 42
-# Dictation's text, typed on a US layout (see the top): character -> (code, with Shift).
-US_KEYS = {" ": (57, False)}
+KEY_RIGHTALT = 100  # AltGr: the third level of most layouts
+# Dictation's text on a US layout (see the top), for Steam and games, and the desktop without a
+# layout of its own: character -> (code, modifiers held while it's pressed).
+US_KEYS = {" ": (57, ())}
 for _row, _shifted, _first in (("1234567890-=", "!@#$%^&*()_+", 2), ("qwertyuiop[]", "QWERTYUIOP{}", 16),
                                ("asdfghjkl;'`", 'ASDFGHJKL:"~', 30), ("\\zxcvbnm,./", "|ZXCVBNM<>?", 43)):
     for _i, (_plain, _shift) in enumerate(zip(_row, _shifted)):
-        US_KEYS[_plain], US_KEYS[_shift] = (_first + _i, False), (_first + _i, True)
+        US_KEYS[_plain], US_KEYS[_shift] = (_first + _i, ()), (_first + _i, (KEY_LEFTSHIFT,))
+# The desktop's keyboard layout: Plasma's kxkbrc in the Frametop desktop's config folder (its
+# XDG_CONFIG_HOME, session/frametop-session.sh), where its System Settings saves it.
+KXKBRC = os.path.expanduser("~/.config/frametop/kxkbrc")
+# The keys tried for each character on it, with these held (the first that types it wins): the
+# main block and the key left of Z on ISO keyboards (KEY_102ND: "<" on a German one), not the
+# keypad, which some apps read as keys of their own.
+LAYOUT_CODES = (*range(2, 54), 57, 86)
+LAYOUT_MODS = ((), (KEY_LEFTSHIFT,), (KEY_RIGHTALT,), (KEY_LEFTSHIFT, KEY_RIGHTALT))
 TYPE_STEP = 0.004  # seconds from a typed key's press to its release, and on to the next key
 TYPE_MAX = 20000  # characters waiting to be typed, at most
 VR_KEYBOARD_MODES = ("always", "no_keyboard", "button", "never")  # when Frametop's keyboard opens
@@ -407,6 +455,107 @@ def read_config(path=os.path.expanduser("~/.config/frametop.conf")):
     return conf
 
 
+def kxkbrc_layout(path=KXKBRC):
+    """The desktop's keyboard layout as (layout, variant, options, model): the first of kxkbrc's
+    LayoutList and VariantList; None when there's no file, no list, or layouts are switched off
+    there (Use=false: KWin's default, a US layout, then)."""
+    cp = configparser.ConfigParser(interpolation=None, strict=False)
+    cp.optionxform = str  # its keys are case-sensitive
+    try:
+        if not cp.read(path):
+            return None
+    except configparser.Error:
+        return None
+    sec = cp["Layout"] if cp.has_section("Layout") else {}
+
+    def first(key):
+        return (sec.get(key) or "").split(",")[0].strip()
+    if (sec.get("Use") or "true").strip().lower() == "false" or not first("LayoutList"):
+        return None
+    options = sec.get("Options", "") if (sec.get("ResetOldOptions") or "").strip().lower() == "true" else ""
+    return first("LayoutList"), first("VariantList"), options.strip(), (sec.get("Model") or "pc105").strip()
+
+
+class XkbNames(ctypes.Structure):
+    _fields_ = [("rules", ctypes.c_char_p), ("model", ctypes.c_char_p), ("layout", ctypes.c_char_p),
+                ("variant", ctypes.c_char_p), ("options", ctypes.c_char_p)]
+
+
+def layout_keys(layout, variant="", options="", model="pc105"):
+    """{character: (code, modifiers)} for a keyboard layout, through libxkbcommon: every key in
+    LAYOUT_CODES with each of LAYOUT_MODS held, and what it types. Dead keys type nothing, so
+    their accents aren't in it. Raises OSError without libxkbcommon or a keymap for the layout."""
+    lib = ctypes.CDLL(ctypes.util.find_library("xkbcommon") or "libxkbcommon.so.0")
+    lib.xkb_context_new.restype = ctypes.c_void_p
+    lib.xkb_context_new.argtypes = [ctypes.c_int]
+    lib.xkb_context_unref.argtypes = [ctypes.c_void_p]
+    lib.xkb_keymap_new_from_names.restype = ctypes.c_void_p
+    lib.xkb_keymap_new_from_names.argtypes = [ctypes.c_void_p, ctypes.POINTER(XkbNames), ctypes.c_int]
+    lib.xkb_keymap_unref.argtypes = [ctypes.c_void_p]
+    lib.xkb_state_new.restype = ctypes.c_void_p
+    lib.xkb_state_new.argtypes = [ctypes.c_void_p]
+    lib.xkb_state_unref.argtypes = [ctypes.c_void_p]
+    lib.xkb_state_update_key.argtypes = [ctypes.c_void_p, ctypes.c_uint32, ctypes.c_int]
+    lib.xkb_state_key_get_utf8.argtypes = [ctypes.c_void_p, ctypes.c_uint32, ctypes.c_char_p, ctypes.c_size_t]
+    ctx = lib.xkb_context_new(0)
+    if not ctx:
+        raise OSError("libxkbcommon: no context")
+    names = XkbNames(b"evdev", model.encode() or None, layout.encode(), variant.encode() or None,
+                     options.encode() or None)
+    keymap = lib.xkb_keymap_new_from_names(ctx, ctypes.byref(names), 0)
+    if not keymap:
+        lib.xkb_context_unref(ctx)
+        raise OSError(f"no keymap for layout {layout}{f'({variant})' if variant else ''}")
+    keys = {}
+    buf = ctypes.create_string_buffer(16)
+    try:
+        for mods in LAYOUT_MODS:
+            state = lib.xkb_state_new(keymap)
+            for m in mods:
+                lib.xkb_state_update_key(state, m + 8, 1)  # evdev code + 8: the XKB keycode; 1: down
+            for code in LAYOUT_CODES:
+                n = lib.xkb_state_key_get_utf8(state, code + 8, buf, len(buf))
+                ch = buf.raw[:n].decode(errors="ignore") if 0 < n < len(buf) else ""
+                if len(ch) == 1 and ch.isprintable() and ch not in keys:
+                    keys[ch] = (code, mods)
+            lib.xkb_state_unref(state)
+    finally:
+        lib.xkb_keymap_unref(keymap)
+        lib.xkb_context_unref(ctx)
+    return keys
+
+
+class DesktopLayout:
+    """The keys that type dictation's characters on the desktop (layout_keys), made again when
+    kxkbrc changes; US_KEYS without a layout or libxkbcommon."""
+
+    def __init__(self, path=None):
+        self.path, self.stamp, self.keys, self.name = path, False, US_KEYS, "us"  # path None: KXKBRC
+
+    def refresh(self):
+        path = self.path or KXKBRC
+        try:
+            st = os.stat(path)
+            stamp = (st.st_mtime_ns, st.st_size)
+        except OSError:
+            stamp = None
+        if (path, stamp) == self.stamp:
+            return
+        self.stamp = (path, stamp)
+        found = kxkbrc_layout(path) if stamp else None
+        name = f"{found[0]}({found[1]})" if found and found[1] else found[0] if found else "us"
+        keys = US_KEYS
+        if found and name != "us":
+            try:
+                keys = layout_keys(*found)
+            except OSError as e:
+                log(f"typing on the desktop: {e}; a US layout instead")
+                name = "us"
+        if name != self.name or keys is not self.keys:
+            log(f"typing on the desktop: layout {name}")
+        self.name, self.keys = name, keys
+
+
 def default_key_bindings():
     """The key combinations a rules file without its own gets."""
     return {**DEFAULT_KEY_BINDINGS, **(VOICE_KEY_BINDINGS if os.path.exists(VOICE_UNIT) else {})}
@@ -681,7 +830,7 @@ class Pointer:
             self.send("recenter")
         elif name == "pointer_toggle":
             if self.active:
-                self.send("hide")
+                self.send("hide toggle")  # gaze mode leaves it off until something wakes it
                 self.active = False
                 log("pointer off (toggle)")
             else:
@@ -755,7 +904,7 @@ class Pointer:
             self.send("scroll 0 0")
             self.scroll_until = None
         if self.active and now - self.last_used > self.idle and now >= self.gaze_awake_until:
-            self.send("hide")
+            self.send("hide idle")
             self.active = False
             log("pointer off (idle)")
 
@@ -798,7 +947,7 @@ class Pointer:
         self.dx = self.dy = self.pending = 0
         self.gaze_awake_until = 0.0
         if self.active or releases:
-            self.send("hide")
+            self.send("hide pause")
         if self.active:
             self.active = False
             log("pointer off (paused)")
@@ -831,6 +980,18 @@ class Node:
                 "kinds": kinds, "role": self.role, "grabbed": self.grabbed, "uinput": self.uinput}
 
 
+def node_role(rule, node, nodes):
+    """A candidate node's role, from its device's rule in frametop-input.json ({} for none) and
+    the nodes there are: "combo" splits the device by node (see the top), the other roles
+    take every node of it, and without one a device with a mouse node is a pointer."""
+    if rule.get("role") == "combo":
+        return "pointer" if node.is_mouse else "passthrough"
+    if rule.get("role") in ("pointer", "passthrough", "ignore"):
+        return rule["role"]
+    has_mouse = any(n.is_mouse for n in nodes if n.id == node.id) or node.is_mouse
+    return "pointer" if has_mouse else "passthrough"
+
+
 def probe(path):
     """Open a node if it is a USB or Bluetooth mouse or keyboard, or has volume keys,
     else return None."""
@@ -892,9 +1053,13 @@ def main():
     # vr_capture_until: every controller button is taken until then (the settings app capturing one).
     # pointer: the 3D mouse while it's in use, pointer_conf: the one the config asks for (they
     # differ while Frametop is paused).
+    # shown_until: ft-screens shows the desktop's screens until then (it says so with "keyboard").
+    # kb_addr, kb_until: Full Keyboard's address, and until when it counts as running (its
+    # hello renews it); kb_said: the "kb DESKTOP SURFACE" it was told last.
     state = {"pointer": None, "pointer_conf": None, "rules": {},
              "desktop_until": 0.0, "typing_applied": None, "vr_capture_until": 0.0, "vr_bind_retry": False,
-             "pointer_holding": False}
+             "pointer_holding": False, "calpanel_until": 0.0, "shown_until": 0.0,
+             "kb_addr": None, "kb_until": 0.0, "kb_said": None}
 
     def pause_changed(paused):
         """Frametop paused or resumed (game_pause.py): the relay's own part."""
@@ -1067,29 +1232,70 @@ def main():
 
     held_modifiers = set()  # on any keyboard, folded (MODIFIERS)
     tap_code = None  # the tap's key as it is (a right-hand code isn't folded), see modifier_tap
+    talk_code = None  # a modifier bound to dictate_hold, held alone: dictation runs until its release
+    talk_hidden = False  # its press kept from the desktop, so the text typed meanwhile isn't a shortcut
     held_meta = set()  # the Meta keys held, as they are (KEY_LEFTMETA, KEY_RIGHTMETA)
     meta_hidden = set()  # held Meta keys the desktop was told came up (a combination; key_binding)
     combos_down = {}  # key code -> the action its combination started (released with it)
+    cal_keys = set()  # Enter keys whose press answered the gaze calibration panel (cal_enter)
+
+    def cal_enter(code, value, now):
+        """Enter while the gaze calibration panel is up (see the top): its press goes to the helper
+        as "calkey 1", and the press, its repeats and its release aren't typed. True if taken."""
+        if value == 1 and code in (KEY_ENTER, KEY_KPENTER) and now < state["calpanel_until"] \
+                and not held_modifiers and code not in combos_down:
+            cal_keys.add(code)
+            try:
+                screens_sock.sendto(b"calkey 1", HELPER)
+            except OSError:
+                pass  # helper not running (SteamVR not running)
+            log("Enter: the gaze calibration panel's")
+            return True
+        if code in cal_keys:
+            if value == 0:
+                cal_keys.discard(code)
+            return True
+        return False
 
     def key_binding(code, value, now):
         """A key from a keyboard: does it complete a key combination ("key_bindings")? True if
         it was taken for one (then it isn't typed)."""
-        nonlocal tap, tap_code
+        nonlocal tap, tap_code, talk_code, talk_hidden
         if code in MODIFIERS:
             mod = MODIFIERS[code]
+            hide = value == 2 and code == talk_code and talk_hidden  # its repeats, as its press
             if value == 1:
+                if talk_code is not None and code != talk_code:
+                    cancel_talk()  # another modifier with it: a shortcut (Ctrl+Shift+T)
                 tap = mod if not held_modifiers and not combos_down else None
                 tap_code = code
+                if tap is not None and talk_code is None and tap_action(mod, code)[1] == "dictate_hold":
+                    # Hold to talk: dictation starts on the press, not the release. The desktop
+                    # doesn't get the press (cancel_talk sends it after all for a shortcut), so
+                    # live text is typed there while it's held. Steam and games read the keyboard
+                    # themselves: there the text waits for the release (typing_tick).
+                    talk_code, talk_hidden, hide = code, True, True
+                    do_action("dictate_hold", 1, now, "keyboard")
+                    log(f"key combination {tap_action(mod, code)[0]}: dictate_hold (held)")
             (held_modifiers.add if value else held_modifiers.discard)(mod)
             if code in (KEY_LEFTMETA, KEY_RIGHTMETA):
                 (held_meta.add if value else held_meta.discard)(code)
                 if value == 0 and code in meta_hidden:
                     meta_hidden.discard(code)
                     return True  # the desktop already had it come up (below)
+            if value == 0 and code == talk_code:
+                talk_code, tap = None, None
+                hide, talk_hidden = talk_hidden, False
+                if not hide:
+                    # As with a tap (modifier_tap): a key in between, so a Meta release isn't one.
+                    to_screens(KEY_F24, 1)
+                    to_screens(KEY_F24, 0)
+                do_action("dictate_hold", 0, now, "keyboard")
+                return hide
             if value == 0 and tap == mod:
                 tap = None
                 modifier_tap(mod, tap_code, now)
-            return False
+            return hide
         if value == 1 and code not in combos_down and meta_hidden:
             # Another key while Meta is still held after a combination: the desktop gets Meta
             # back first, so Meta+that key still works there.
@@ -1123,16 +1329,35 @@ def main():
         log(f"key combination {combo}: {action}")
         return True
 
-    def modifier_tap(mod, code, now):
-        """A modifier pressed and released alone: its binding, if it has one ("125": Meta tap).
-        A right-hand key's own binding ("97": Right Ctrl) comes before the folded one."""
+    def tap_action(mod, code):
+        """A lone modifier's combination and action, (combo, None) when it does nothing now. A
+        right-hand key's own binding ("97": Right Ctrl) comes before the folded one."""
         bindings = state["rules"]["key_bindings"]
         combo = str(code) if code != mod and str(code) in bindings else str(mod)
         action = bindings.get(combo)
         if not known_action(action) or action in ("key", "none") or (pause.paused and not works_paused(action)):
-            return
-        if action == "dictate_hold":
-            action = "dictate"  # a tap is over before it's held: it starts, and the next one stops
+            return combo, None
+        return combo, action
+
+    def cancel_talk():
+        """A key, mouse button or scroll while a hold-to-talk modifier is held: it was a shortcut
+        (Ctrl+C), so the dictation it started is dropped, without its cancel tone."""
+        nonlocal talk_code, talk_hidden
+        if talk_code is not None:
+            if talk_hidden:
+                to_screens(talk_code, 1)  # the desktop gets the modifier before the key it's for
+            talk_code, talk_hidden = None, False
+            try:
+                screens_sock.sendto(b"cancel keyboard quiet", VOICE)
+            except OSError:
+                pass
+            log("dictate_hold: cancelled (another key while held: a shortcut)")
+
+    def modifier_tap(mod, code, now):
+        """A modifier pressed and released alone: its binding, if it has one ("125": Meta tap)."""
+        combo, action = tap_action(mod, code)
+        if action is None or action == "dictate_hold":
+            return  # nothing, or hold to talk (key_binding), which started on the press
         # The desktop gets a key in between before the release goes there, so the tap isn't one
         # there too: a Meta tap would open Plasma's launcher.
         to_screens(KEY_F24, 1)
@@ -1166,17 +1391,33 @@ def main():
     next_scan = 0.0
 
     volume = Volume()
-    typing = collections.deque()  # (code, shift): dictation's characters still to type
-    typing_down = []  # the character pressed last, (code, shift, to the desktop): its release is next
+    # What's still to type, in order: a character (str) from dictation, ("quiet", character)
+    # from Full Keyboard (its text isn't logged), or ("key", code, value), a key Full Keyboard
+    # pressed or released.
+    typing = collections.deque()
+    typing_down = []  # the character pressed last, (code, modifiers, to the desktop): its release is next
     typing_at = 0.0  # the next press or release goes then
+    untyped = []  # characters with no key where typing went, logged when the queue is empty (None: quiet)
+    kb_held = {}  # keys Full Keyboard holds down: code -> to the desktop
+    desktop_layout = DesktopLayout()
 
-    def type_text(text):
-        """Queue dictation's text (see the top)."""
-        keys = [US_KEYS[c] for c in text if c in US_KEYS]
+    def type_text(text, quiet=False):
+        """Queue dictation's text (see the top), or with quiet, Full Keyboard's. Which keys type
+        it is worked out as it goes, where typing goes then."""
+        desktop_layout.refresh()
+        chars = [c for c in text if c.isprintable()]
         room = max(0, TYPE_MAX - len(typing))
-        typing.extend(keys[:room])
-        log(f"typing {min(len(keys), room)} characters for Frametop Voice"
-            + (f" ({len(keys) - room} dropped: too many waiting)" if len(keys) > room else ""))
+        typing.extend((("quiet", c) for c in chars[:room]) if quiet else chars[:room])
+        if not quiet or len(chars) > room:
+            log(f"typing {min(len(chars), room)} characters for {'Full Keyboard' if quiet else 'Frametop Voice'}"
+                + (f" ({len(chars) - room} dropped: too many waiting)" if len(chars) > room else ""))
+
+    def report_untyped(desktop):
+        if untyped and not typing:
+            where = f"the desktop's layout ({desktop_layout.name})" if desktop else "a US layout (Steam)"
+            said = "".join(dict.fromkeys(c for c in untyped if c is not None))
+            log(f"typing: {len(untyped)} characters {where} has no key for" + (f": {said}" if said else ""))
+            untyped.clear()
 
     def type_key(code, value, desktop):
         if desktop:
@@ -1185,6 +1426,14 @@ def main():
             keyboard.emit(EV_KEY, code, value)
             keyboard.sync()
 
+    def typing_blocked(now):
+        """A modifier is held (the text would be shortcuts), except a hold-to-talk key the
+        desktop never got, when typing goes there."""
+        held = held_modifiers
+        if talk_hidden and now < state["desktop_until"]:
+            held = held - {MODIFIERS[talk_code]}
+        return bool(held)
+
     def typing_tick(now):
         """Release the character pressed last, or press the next, when it's time. A new one
         waits while a modifier is held."""
@@ -1192,23 +1441,44 @@ def main():
         if now < typing_at:
             return
         if typing_down:
-            code, shift, desktop = typing_down.pop()
+            code, mods, desktop = typing_down.pop()
             type_key(code, 0, desktop)
-            if shift:
-                type_key(KEY_LEFTSHIFT, 0, desktop)
-        elif typing and not held_modifiers:
-            code, shift = typing.popleft()
+            for m in reversed(mods):
+                type_key(m, 0, desktop)
+        elif typing and typing[0][0] == "key":
+            # Full Keyboard's key, with its modifiers its own: a held one doesn't stop it. It
+            # comes up where it went down.
+            _, code, value = typing.popleft()
+            if value:
+                if code in kb_held:
+                    return  # down already
+                desktop = kb_held[code] = now < state["desktop_until"]
+            elif code in kb_held:
+                desktop = kb_held.pop(code)
+            else:
+                return  # not down
+            type_key(code, value, desktop)
+        elif typing and not typing_blocked(now):
             desktop = now < state["desktop_until"]
-            if shift:
-                type_key(KEY_LEFTSHIFT, 1, desktop)
+            keys = desktop_layout.keys if desktop else US_KEYS
+            item = typing.popleft()
+            ch = item if isinstance(item, str) else item[1]
+            if ch not in keys:
+                untyped.append(ch if isinstance(item, str) else None)
+                report_untyped(desktop)
+                return  # the next one right away
+            code, mods = keys[ch]
+            for m in mods:
+                type_key(m, 1, desktop)
             type_key(code, 1, desktop)
-            typing_down.append((code, shift, desktop))
+            typing_down.append((code, mods, desktop))
+            report_untyped(desktop)
         else:
             return
         typing_at = now + TYPE_STEP
 
     def typing_timeout(now, default):
-        if typing_down or (typing and not held_modifiers):
+        if typing_down or (typing and (typing[0][0] == "key" or not typing_blocked(now))):
             return max(0.0, min(default, typing_at - now))
         return default
 
@@ -1255,11 +1525,7 @@ def main():
         return node.candidate
 
     def role_of(node):
-        rule = state["rules"]["devices"].get(node.id, {})
-        if rule.get("role") in ("pointer", "passthrough", "ignore"):
-            return rule["role"]
-        has_mouse = any(n.is_mouse for n in nodes.values() if n.id == node.id) or node.is_mouse
-        return "pointer" if has_mouse else "passthrough"
+        return node_role(state["rules"]["devices"].get(node.id, {}), node, nodes.values())
 
     def release_held(node):
         for code in node.held:
@@ -1289,7 +1555,9 @@ def main():
         if not screens_down and not held_meta and not held_modifiers:
             return
         down = physically_down()
-        for code in sorted(screens_down - down):
+        typed = {k for code, mods, desktop in typing_down if desktop for k in (code, *mods)}  # dictation's, coming up next
+        typed |= {code for code, desktop in kb_held.items() if desktop}  # Full Keyboard's
+        for code in sorted(screens_down - down - typed):
             to_screens(code, 0)
             log(f"key {code} released on the desktop: no keyboard holds it")
         for code in [c for c in held_meta if c not in down]:
@@ -1352,14 +1620,46 @@ def main():
         except OSError:
             watchers.pop(addr, None)
 
+    def kb_tell(now, message=None):
+        """Tell Full Keyboard message, or with none, "kb DESKTOP SURFACE" if that changed: typing
+        goes to the desktop 0|1, and the screens show with Frametop not paused 0|1 (it may show
+        over them with the dashboard closed). From the control socket: it takes word from
+        @frametop_relay only."""
+        addr = state["kb_addr"]
+        if addr is None:
+            return
+        if message is None:
+            message = (f"kb {int(now < state['desktop_until'])}"
+                       f" {int(now < state['shown_until'] and not pause.paused)}")
+            if message == state["kb_said"]:
+                return
+            state["kb_said"] = message
+        try:
+            control.sendto(message.encode(), addr)
+        except OSError:
+            pass  # gone: its hellos stop, and kb_gone follows
+
+    def kb_gone(why):
+        """Full Keyboard stopped or restarted: its keys still to come are dropped, and the keys
+        it holds come up where they went down."""
+        state["kb_addr"], state["kb_said"] = None, None
+        kept = [item for item in typing if isinstance(item, str) or item[0] != "key"]
+        typing.clear()
+        typing.extend(kept)
+        for code, desktop in sorted(kb_held.items()):
+            type_key(code, 0, desktop)
+        kb_held.clear()
+        log(f"Full Keyboard {why}")
+
     def handle_control(now):
         while True:
             try:
                 data, anc, _, addr = control.recvmsg(65536, socket.CMSG_SPACE(creds.size))
             except BlockingIOError:
+                kb_tell(now)  # what changed reaches Full Keyboard at once
                 return
-            uid = next((creds.unpack(d[:creds.size])[1] for level, kind, d in anc
-                        if level == socket.SOL_SOCKET and kind == socket.SCM_CREDENTIALS), None)
+            pid, uid = next((creds.unpack(d[:creds.size])[:2] for level, kind, d in anc
+                             if level == socket.SOL_SOCKET and kind == socket.SCM_CREDENTIALS), (None, None))
             words = data.decode(errors="replace").split()
             cmd = words[0] if words else ""
             # One malformed datagram must not end the relay: it would drop every grab,
@@ -1368,8 +1668,12 @@ def main():
             try:
                 if cmd == "keyboard":
                     # From ft-screens (unbound, no reply): where typing goes, repeated every second.
+                    # "keyboard desktop|steam 1|0": the third word, whether the screens show.
                     desktop = len(words) > 1 and words[1] == "desktop" and not pause.paused
                     state["desktop_until"] = now + 3.0 if desktop else 0.0
+                    state["shown_until"] = now + 3.0 if len(words) > 2 and words[2] == "1" else 0.0
+                    if uid == os.getuid():
+                        pause.screens_seen(pid)  # a desktop started while paused hides too
                     continue
                 if cmd == "vrbtn" and len(words) == 3 and words[1] in VR_BUTTONS and words[2] in ("0", "1"):
                     vr_button(words[1], int(words[2]), now)
@@ -1404,10 +1708,60 @@ def main():
                     else:
                         log(f"type from another user ({uid}): ignored")
                     continue
+                if cmd == "kbhello":
+                    # Full Keyboard (keyboard/, --frametop), every second and when it opens or
+                    # closes: "kbhello OPEN". Only from this user and a bound address: what it
+                    # types goes to the desktop. It gets the word on every hello, and ft-screens
+                    # hears it's there ("fullkb OPEN"), so it opens Full Keyboard, not its own.
+                    if uid != os.getuid() or not addr:
+                        continue
+                    if addr != state["kb_addr"]:
+                        if state["kb_addr"] is not None:
+                            kb_gone("restarted")
+                        log("Full Keyboard connected")
+                    state["kb_addr"], state["kb_until"], state["kb_said"] = addr, now + 3.0, None
+                    kb_tell(now)
+                    opened = "1" if len(words) > 1 and words[1] == "1" else "0"
+                    try:
+                        screens_sock.sendto(f"fullkb {opened}".encode(), SCREENS)
+                    except OSError:
+                        pass  # ft-screens not running
+                    continue
+                if cmd in ("kbkey", "kbtext"):
+                    # Full Keyboard's typing, while it was told it goes to the desktop: "kbkey CODE
+                    # 0|1" (a key's press or release) and "kbtext TEXT" (what its input methods
+                    # composed). Only from the Full Keyboard that said hello. In order with what's
+                    # still to type, where typing goes then.
+                    if uid != os.getuid() or addr is None or addr != state["kb_addr"]:
+                        continue
+                    if cmd == "kbtext":
+                        type_text(data[len(b"kbtext "):].decode(errors="replace"), quiet=True)
+                    elif (len(words) == 3 and words[1].isdigit() and 0 < int(words[1]) < BTN_MISC
+                          and words[2] in ("0", "1") and len(typing) < TYPE_MAX):
+                        typing.append(("key", int(words[1]), int(words[2])))
+                    continue
+                if cmd == "kbpanel" and len(words) == 2 and words[1] in ("show", "hide"):
+                    # From ft-screens (unbound): open or close Full Keyboard, which it does for the
+                    # "vrkeyboard" commands while Full Keyboard is there.
+                    if uid == os.getuid():
+                        kb_tell(now, f"kb {words[1]}")
+                    continue
+                if cmd == "calpanel" and len(words) == 2:
+                    # From the pointer helper (see the top). Only from this user: it takes Enter.
+                    if uid == os.getuid():
+                        state["calpanel_until"] = now + 3.0 if words[1] == "1" else 0.0
+                    continue
                 if cmd == "gazeawake" and len(words) == 2:
                     if state["pointer"]:
                         state["pointer"].gaze_awake_until = now + 12.0 if words[1] == "1" else 0.0
                     continue
+                if cmd == "gazewoke":
+                    # The gaze took the pointer back (gaze mode owns it): it's on, as a wake leaves it.
+                    p = state["pointer"]
+                    if p and not p.active:
+                        p.active, p.last_used = True, now
+                        log("pointer on (gaze)")
+                    continue
                 if not addr:
                     continue  # unbound sender, nowhere to reply
                 if cmd == "devices":
@@ -1442,7 +1796,7 @@ def main():
         if etype == EV_REL and now - node.last_watch < 0.05:
             return  # motion: enough for an activity light
         node.last_watch = now
-        msg = {"t": "event", "id": node.id, "path": node.path, "name": node.name,
+        msg = {"t": "event", "id": node.id, "path": node.path, "name": node.name, "role": node.role,
                "type": EV_NAMES.get(etype, str(etype)), "code": code, "value": value}
         for addr, until in list(watchers.items()):
             if now > until:
@@ -1515,6 +1869,9 @@ def main():
             vr_bind(now)
         volume.tick(now)
         typing_tick(now)
+        if state["kb_addr"] is not None and now >= state["kb_until"]:
+            kb_gone("stopped (no word for 3 seconds)")
+        kb_tell(now)
         pause.tick(now)
         if state["vr_capture_until"] and now >= state["vr_capture_until"]:
             vr_bind(now)  # capture over: back to the mapped buttons
@@ -1544,6 +1901,7 @@ def main():
                               value, now)
                 if (etype == EV_KEY and value == 1 and code not in MODIFIERS) or (etype == EV_REL and code in SCROLLS):
                     tap = None  # a key, button or scroll in between: a modifier's release isn't a tap
+                    cancel_talk()
                 if etype == EV_KEY and ((node.remapped and code in VOLUME_ORIGINAL)
                                         or (node.grabbed and code in VOLUME_STANDIN)):
                     volume.key(fd, code, value, now)
@@ -1559,6 +1917,8 @@ def main():
                 if node.role != "pointer":
                     # Observed only, unless typing goes to the desktop. Key combinations work on
                     # any pass-through keyboard.
+                    if node.role == "passthrough" and etype == EV_KEY and cal_enter(code, value, now):
+                        continue
                     if node.role == "passthrough" and etype == EV_KEY and code < BTN_MISC and key_binding(code, value, now):
                         continue
                     if node.role == "passthrough" and etype == EV_KEY:

input/test/keys-test.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """Offline test of the input relay's key combinations and modifier taps, of which keys and
-mouse buttons reach the desktop (ft-screens), and of dictation (Frametop Voice): the keys that
-start and stop it, and the text it has the relay type.
+mouse buttons reach the desktop (ft-screens), of dictation (Frametop Voice): the keys that
+start and stop it, and the text it has the relay type, and of Full Keyboard's typing.
 
 Runs the relay's main() against a fake pass-through keyboard, a fake mouse, and two fake nodes
 with volume keys, a keyboard's Consumer Control node and the headset's gpio-keys (pipes), with
@@ -28,6 +28,9 @@ relay_path = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "inp
 tag = f"ft_keys_test_{os.getpid()}"
 with open(relay_path) as f:
     src = f.read().replace('"\\0frametop_relay"', f'"\\0{tag}_relay"')
+screens_socket = "screens_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)"
+assert src.count(screens_socket) == 1, "relay's screens socket constructor changed"
+src = src.replace(screens_socket, "screens_sock = _test_screens_sender")
 relay = importlib.util.module_from_spec(importlib.util.spec_from_loader("relay", loader=None))
 relay.__file__ = os.path.abspath(relay_path)
 sys.path.insert(0, os.path.dirname(os.path.abspath(relay_path)))  # its game_pause and vrws
@@ -59,7 +62,7 @@ class StubPause:
     def configure(self, *args):
         pass
 
-    game_state = tick = helper_started = controllers_changed = set_hands = dashboard_state = configure
+    game_state = tick = helper_started = controllers_changed = set_hands = dashboard_state = screens_seen = configure
 
 
 stub = {}
@@ -174,7 +177,10 @@ def probe(path):
 
 relay.probe = probe
 
-screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+# A connected pair holds all 16 startup releases even when the kernel limits unconnected
+# datagram queues to 10 messages. The relay still uses real sendto(), including other peers.
+relay._test_screens_sender, screens = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)
+relay._test_screens_sender.setblocking(False)
 screens.bind(relay.SCREENS)
 screens.settimeout(0.05)
 voice = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)  # Frametop Voice's socket
@@ -233,12 +239,13 @@ def to_relay(text):
     c.close()
 
 
-def keys_for(text):
-    """The presses and releases that type text, as (code, value), Shift included."""
+def keys_for(text, keys=None):
+    """The presses and releases that type text, as (code, value), modifiers included: on a US
+    layout, or keys ({character: (code, modifiers)})."""
     out = []
     for ch in text:
-        code, shift = relay.US_KEYS[ch]
-        out += ([(42, 1)] if shift else []) + [(code, 1), (code, 0)] + ([(42, 0)] if shift else [])
+        code, mods = (keys or relay.US_KEYS)[ch]
+        out += [(m, 1) for m in mods] + [(code, 1), (code, 0)] + [(m, 0) for m in reversed(mods)]
     return out
 
 
@@ -394,7 +401,7 @@ def tests():
                 m = helper.recv(256).decode()
             except socket.timeout:
                 return got
-            if m == "hide" or (m.startswith("btn ") and not m.startswith("btn a ")):
+            if m.startswith("hide") or (m.startswith("btn ") and not m.startswith("btn a ")):
                 got.append(m)
 
     conf["POINTER"] = "1"
@@ -404,7 +411,7 @@ def tests():
     check("pointer mode: a held left button reaches the helper", clicks(), ["btn trigger 1"])
     pause("on")
     check("pause with the left button held: it comes up, then the pointer hides", clicks(),
-          ["btn trigger 0", "hide"])
+          ["btn trigger 0", "hide pause"])
     key(relay.BTN_LEFT, 0, ms_w)
     check("paused: its release reaches no one", clicks(), [])
     pause("off")
@@ -488,8 +495,19 @@ def tests():
     key(CTRL, 1); key(CTRL, 0); key(RCTRL, 1); key(RCTRL, 0)
     check("a Ctrl tap bound by its left code: either side", dictation(), ["toggle keyboard", "toggle keyboard"])
     use({"97": "dictate_hold"})
-    key(RCTRL, 1); key(RCTRL, 0)
-    check("a tap bound to dictate_hold toggles (a tap is over before it's held)", dictation(), ["toggle keyboard"])
+    key(RCTRL, 1)
+    check("Right Ctrl bound to dictate_hold: starts on the press", dictation(), ["start keyboard"])
+    key(RCTRL, 0)
+    check("...and stops on the release", dictation(), ["stop keyboard"])
+    key(CTRL, 1); key(CTRL, 0)
+    check("...Left Ctrl doesn't", dictation(), [])
+    key(RCTRL, 1); key(F, 1)
+    check("Right Ctrl+F while held: a shortcut, so that dictation is cancelled quietly", dictation(),
+          ["start keyboard", "cancel keyboard quiet"])
+    key(F, 0); key(RCTRL, 0)
+    check("...and its release does nothing more", dictation(), [])
+    key(SHIFT, 1); key(RCTRL, 1); key(RCTRL, 0); key(SHIFT, 0)
+    check("Right Ctrl with Shift held: not on its own, no dictation", dictation(), [])
     use({"29+42+33": "dictate_hold"})
     key(CTRL, 1); key(SHIFT, 1); key(F, 1)
     check("Ctrl+Shift+F held (dictate_hold): starts", dictation(), ["start keyboard"])
@@ -533,6 +551,73 @@ def tests():
     time.sleep(0.3)
     check("type: to the desktop while typing goes there", typed(), [f"key {c} {v}" for c, v in keys_for("Ok.")])
     check("...and not to the virtual keyboard", emitted(), [])
+    use({"97": "dictate_hold"})
+    dictation()
+    key(RCTRL, 1)
+    to_relay("type ok")
+    time.sleep(0.3)
+    check("held to talk: live text reaches the desktop while Right Ctrl is held, without it",
+          typed(), [f"key {c} {v}" for c, v in keys_for("ok")])
+    key(RCTRL, 2); key(RCTRL, 0)
+    check("...nor its repeats or release", typed(), [])
+    check("...and dictation ran from press to release", dictation(), ["start keyboard", "stop keyboard"])
+    key(RCTRL, 1); key(C, 1); key(C, 0); key(RCTRL, 0)
+    check("held to talk, then C: the desktop gets Right Ctrl before C (a shortcut after all)", typed(),
+          ["key 97 1", f"key {C} 1", f"key {C} 0", "key 97 0"])
+    check("...and that dictation is cancelled quietly", dictation(), ["start keyboard", "cancel keyboard quiet"])
+    key(RCTRL, 1); key(SHIFT, 1)
+    check("held to talk, then Shift: cancelled, and the desktop gets both", (dictation(), typed()),
+          (["start keyboard", "cancel keyboard quiet"], ["key 97 1", f"key {SHIFT} 1"]))
+    key(SHIFT, 0); key(RCTRL, 0)
+    typed()
+    to_relay("keyboard steam")
+    key(RCTRL, 1)
+    to_relay("type ab")
+    time.sleep(0.3)
+    check("held to talk while typing goes to Steam (it sees the key itself): the text waits", emitted(), [])
+    key(RCTRL, 0)
+    time.sleep(0.3)
+    check("...until the release", emitted(), [("keyboard", c, v) for c, v in keys_for("ab")])
+    dictation()
+    to_relay("keyboard desktop")
+    use({})
+
+    # The desktop's own layout (kxkbrc): German QWERTZ, through libxkbcommon.
+    rc = os.path.join(OUT, "kxkbrc")
+    with open(rc, "w") as f:
+        f.write("[Layout]\nLayoutList=de,us\nUse=true\n")
+    relay.KXKBRC = rc
+    try:
+        de = relay.layout_keys("de")
+    except OSError as e:
+        de = None
+        print(f"skip  German layout typing: {e}")
+    if de:
+        check("German layout: z is the US Y key, y the US Z key, @ is AltGr+Q, < the ISO key",
+              (de["z"], de["y"], de["@"], de["<"], de["\u00e4"]),
+              ((21, ()), (44, ()), (16, (100,)), (86, ()), (40, ())))
+        to_relay("type way @ f\u00fcr")
+        time.sleep(0.3)
+        check("type on a German desktop: its keys, AltGr and umlauts included", typed(),
+              [f"key {c} {v}" for c, v in keys_for("way @ f\u00fcr", de)])
+        to_relay("type a\u00e9b")
+        time.sleep(0.3)
+        check("...a character it has no key for (a dead key's accent) is dropped", typed(),
+              [f"key {c} {v}" for c, v in keys_for("ab", de)])
+        check("...and the log says which", [m for m in LOG if m.startswith("typing: 1 characters")][-1:],
+              ["typing: 1 characters the desktop's layout (de) has no key for: \u00e9"])
+        to_relay("keyboard steam")
+        to_relay("type way")
+        time.sleep(0.3)
+        check("...Steam and games still get a US layout", emitted(), [("keyboard", c, v) for c, v in keys_for("way")])
+        to_relay("keyboard desktop")
+    with open(rc, "w") as f:
+        f.write("[Layout]\nLayoutList=de\nUse=false\n")
+    to_relay("type way")
+    time.sleep(0.3)
+    check("layouts switched off in kxkbrc (Use=false): a US layout", typed(),
+          [f"key {c} {v}" for c, v in keys_for("way")])
+    relay.KXKBRC = os.path.join(OUT, "no-kxkbrc")
     to_relay("keyboard steam")
 
     check("an empty command isn't an action", relay.known_action("command:  "), False)
@@ -569,6 +654,133 @@ def tests():
           (text_field(), said()), ([], ["text field focused: not opening Frametop's keyboard (Keyboard setting: never)"]))
     vr_keyboard["mode"] = None
     use(None)
+
+    # The gaze calibration panel (the helper's "calpanel 1"): Enter takes its dot ("calkey 1" to
+    # the helper) and isn't typed, its press, repeats and release; with a modifier held, or once
+    # the panel is gone or lapsed, Enter is typed again.
+    ENTER, KPENTER = relay.KEY_ENTER, relay.KEY_KPENTER
+
+    def cal_keys():
+        got = []
+        while True:
+            try:
+                m = helper.recv(256).decode()
+            except socket.timeout:
+                return got
+            if m.startswith("calkey"):
+                got.append(m)
+
+    def panel(on):
+        to_relay(f"calpanel {on}")
+        time.sleep(0.1)
+
+    use({})
+    typed(), cal_keys()
+    key(ENTER, 1); key(ENTER, 0)
+    check("Enter without the panel: typed", (cal_keys(), typed()), ([], ["key 28 1", "key 28 0"]))
+    panel(1)
+    key(ENTER, 1); key(ENTER, 2); key(ENTER, 2); key(ENTER, 0)
+    check("panel up: Enter takes the dot, once", cal_keys(), ["calkey 1"])
+    check("...and its press, repeats and release aren't typed", typed(), [])
+    key(KPENTER, 1); key(KPENTER, 0)
+    check("panel up: keypad Enter too", (cal_keys(), typed()), (["calkey 1"], []))
+    key(SHIFT, 1); key(ENTER, 1); key(ENTER, 0); key(SHIFT, 0)
+    check("panel up, Shift held: Enter is typed", (cal_keys(), typed()),
+          ([], ["key 42 1", "key 28 1", "key 28 0", "key 42 0"]))
+    key(A, 1); key(A, 0)
+    check("panel up: other keys are typed", (cal_keys(), typed()), ([], ["key 30 1", "key 30 0"]))
+    key(ENTER, 1); panel(0); key(ENTER, 0)
+    check("panel gone while Enter is held: it took the dot, and its release isn't typed",
+          (cal_keys(), typed()), (["calkey 1"], []))
+    key(ENTER, 1); key(ENTER, 0)
+    check("panel gone: Enter is typed", (cal_keys(), typed()), ([], ["key 28 1", "key 28 0"]))
+    panel(1)
+    time.sleep(3.1)  # no renewal: the claim lapses after 3 s
+    key(ENTER, 1); key(ENTER, 0)
+    check("panel claim lapsed: Enter is typed", (cal_keys(), typed()), ([], ["key 28 1", "key 28 0"]))
+
+    # Full Keyboard (keyboard/, --frametop): it says hello, hears where typing goes and whether
+    # the screens show, and types on the desktop through the relay.
+    use({})
+    kb = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    kb.bind("")  # autobound, as Full Keyboard's
+    kb.settimeout(0.05)
+
+    def heard():
+        """What Full Keyboard was sent since the last call."""
+        got = []
+        while True:
+            try:
+                got.append(kb.recv(256).decode())
+            except socket.timeout:
+                return got
+
+    def from_kb(text, sock=kb):
+        sock.sendto(text.encode(), f"\0{tag}_relay")
+        time.sleep(0.1)
+
+    typed(), emitted()
+    from_kb("kbhello 0")
+    check("Full Keyboard's hello: the word back (Steam, nothing shown)", heard(), ["kb 0 0"])
+    check("...and ft-screens hears it's there", [m for m in typed() if m.startswith("fullkb")], ["fullkb 0"])
+    check("...logged", [m for m in LOG if m.startswith("Full Keyboard")][-1:], ["Full Keyboard connected"])
+    to_relay("keyboard desktop 1")
+    time.sleep(0.1)
+    check("typing goes to the desktop, the screens show: it hears at once", heard(), ["kb 1 1"])
+    from_kb("kbhello 1")
+    check("a hello always gets the word", heard(), ["kb 1 1"])
+    check("...and ft-screens hears it's up", [m for m in typed() if m.startswith("fullkb")], ["fullkb 1"])
+    from_kb("kbkey 30 1")
+    from_kb("kbkey 30 0")
+    check("its keys reach the desktop", (typed(), emitted()), (["key 30 1", "key 30 0"], []))
+    LOG.clear()
+    from_kb("kbtext a\u00e9b")
+    from_kb(f"kbkey {ENTER} 1")
+    from_kb(f"kbkey {ENTER} 0")
+    check("its text through the desktop's layout, then its key, in order", typed(),
+          [f"key {c} {v}" for c, v in keys_for("ab")] + ["key 28 1", "key 28 0"])
+    check("...what the layout can't type isn't logged by name",
+          [m for m in LOG if m.startswith("typing")], ["typing: 1 characters the desktop's layout (us) has no key for"])
+    from_kb(f"kbkey {SHIFT} 1")
+    to_relay("keyboard steam 1")
+    time.sleep(0.1)
+    check("typing goes to Steam: it hears", heard(), ["kb 0 1"])
+    from_kb(f"kbkey {SHIFT} 0")
+    from_kb("kbkey 30 1")
+    from_kb("kbkey 30 0")
+    check("a key held from before comes up on the desktop", typed(), [f"key {SHIFT} 1", f"key {SHIFT} 0"])
+    check("...and new ones go to the virtual keyboard", emitted(), [("keyboard", 30, 1), ("keyboard", 30, 0)])
+    other = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
+    other.bind("")
+    to_relay("keyboard desktop 1")
+    from_kb("kbkey 30 1", other)
+    from_kb("kbtext x", other)
+    check("keys and text from another address are ignored", (typed(), emitted()), ([], []))
+    other.close()
+    heard()
+    to_relay("kbpanel show")
+    check("ft-screens' kbpanel show: Full Keyboard opens", heard(), ["kb show"])
+    to_relay("kbpanel hide")
+    check("...and hide", heard(), ["kb hide"])
+    pause("on")
+    time.sleep(0.1)
+    check("paused: typing to Steam, and not over the screens", heard(), ["kb 0 0"])
+    pause("off")
+    to_relay("keyboard desktop 1")
+    from_kb("kbhello 1")
+    heard()
+    from_kb("kbkey 30 1")
+    time.sleep(1.2)  # the relay's once-a-second check (reconcile_desktop_keys)
+    check("its held key stays down on the desktop past a second", typed(), ["fullkb 1", "key 30 1"])
+    time.sleep(2.2)  # no hello for 3 s
+    check("it stops saying hello: its held key comes up", typed(), ["key 30 0"])
+    check("...logged", [m for m in LOG if m.startswith("Full Keyboard")][-1:],
+          ["Full Keyboard stopped (no word for 3 seconds)"])
+    from_kb("kbkey 30 1")
+    check("gone: its keys reach nothing until it says hello again", (typed(), emitted()), ([], []))
+    kb.close()
+    to_relay("keyboard steam")
+    use(None)
     print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
     shutil.rmtree(OUT, ignore_errors=True)
     os._exit(1 if failures else 0)
@@ -576,4 +788,5 @@ def tests():
 
 threading.Thread(target=tests, daemon=True).start()
 sys.argv = [relay_path, "--no-grab"]
+relay.KXKBRC = os.path.join(OUT, "no-kxkbrc")  # a US desktop until the layout tests
 relay.main()

input/test/pause-buttons-test.py

@@ -60,7 +60,7 @@ p.action("left", 1, 10.0)  # a mouse click (or drag) held: btn trigger 1
 check("press reaches the helper", p.sock.sent, ["show", "recenter", "btn trigger 1"])
 mark = len(p.sock.sent)
 p.stand_down()  # Frametop pauses while it's held
-check("stand_down releases the held button, then hides", since(p, mark), ["btn trigger 0", "hide"])
+check("stand_down releases the held button, then hides", since(p, mark), ["btn trigger 0", "hide pause"])
 
 # The release that pausing drops later is already covered: the button is no longer down,
 # so a stray second stand_down sends nothing.
@@ -74,7 +74,7 @@ p.action("left", 1, 10.0)
 p.action("right", 1, 10.5)
 mark = len(p.sock.sent)
 p.stand_down()
-check("both held buttons are released, then it hides", since(p, mark), ["btn b 0", "btn trigger 0", "hide"])
+check("both held buttons are released, then it hides", since(p, mark), ["btn b 0", "btn trigger 0", "hide pause"])
 
 # A mapped controller button and a key combination take the same path as the mouse.
 p = pointer()
@@ -83,7 +83,7 @@ p.action("back", 1, 10.1, "keyboard")  # a key combination mapped to back
 mark = len(p.sock.sent)
 p.stand_down()
 check("controller and key combination clicks are released too", since(p, mark),
-      ["btn joystick 0", "btn x 0", "hide"])
+      ["btn joystick 0", "btn x 0", "hide pause"])
 
 # ---------------------------------------------------------------- the pointer already off
 # A release with no "hide" after it would wake a helper that wakes on any btn, connecting the
@@ -93,18 +93,19 @@ p.idle = 30.0
 p.action("left", 1, 10.0)
 for t in (10.3, 10.4, 41.0):  # the claim pulse, then 30 s with no mouse input
     p.tick(t)
-check("held 30 s with no mouse input: the pointer goes off", (p.active, p.sock.sent[-1]), (False, "hide"))
+check("held 30 s with no mouse input: the pointer goes off", (p.active, p.sock.sent[-1]), (False, "hide idle"))
 mark = len(p.sock.sent)
 p.stand_down()
-check("idle with a button held: the release, then hide", since(p, mark), ["btn trigger 0", "hide"])
+check("idle with a button held: the release, then hide", since(p, mark), ["btn trigger 0", "hide pause"])
 
 p = pointer()
 p.action("left", 1, 10.0)
 p.action("pointer_toggle", 1, 10.5)
+check("pointer_toggle says why it hides (gaze mode leaves it off)", p.sock.sent[-1], "hide toggle")
 mark = len(p.sock.sent)
 p.stand_down()
 check("pointer toggled off with a button held: the release, then hide", since(p, mark),
-      ["btn trigger 0", "hide"])
+      ["btn trigger 0", "hide pause"])
 
 # ---------------------------------------------------------------- the ordinary path
 p = pointer()
@@ -112,7 +113,7 @@ p.action("left", 1, 10.0)
 p.action("left", 0, 11.0)  # released before the pause
 mark = len(p.sock.sent)
 p.stand_down()
-check("a button released before the pause: stand_down only hides", since(p, mark), ["hide"])
+check("a button released before the pause: stand_down only hides", since(p, mark), ["hide pause"])
 
 p = pointer()
 p.idle = 30.0
@@ -130,7 +131,7 @@ p.action("gaze_left", 1, 10.0, "keyboard")  # Meta+J held
 p.action("gaze_drag", 1, 10.1, "mouse")
 mark = len(p.sock.sent)
 p.stand_down()
-check("gaze holds: stand_down sends only hide (the helper ends them)", since(p, mark), ["hide"])
+check("gaze holds: stand_down sends only hide (the helper ends them)", since(p, mark), ["hide pause"])
 
 print()
 if failures:

input/test/pause-test.py

@@ -52,7 +52,12 @@ check("sticks 0.5 s apart, twice: nothing", presses(g, both(0, gap=0.5, hold=0.6
 g = gp.Gesture((L, R), 2)
 check("three double presses in a row: once", presses(g, both(0) + both(0.3) + both(0.6)), 1)
 g = gp.Gesture((L, R), 2)
-check("again after a second: twice in all", presses(g, both(0) + both(0.3) + both(2.0) + both(2.3)), 2)
+check("again 2 s later (2026-10-16: resumed, then paused again): ignored",
+      presses(g, both(0) + both(0.3) + both(2.0) + both(2.3)), 1)
+check("...and says how long after the last one", round(g.ignored, 1), 2.0)
+g = gp.Gesture((L, R), 2)
+check("again after the cooldown: twice in all",
+      presses(g, both(0) + both(0.3) + both(gp.GESTURE_COOLDOWN + 0.5) + both(gp.GESTURE_COOLDOWN + 0.8)), 2)
 g = gp.Gesture((L, R), 2)
 check("repeated down states count once",
       presses(g, [(0, L, True), (0.01, L, True), (0.02, R, True), (0.03, R, True), (0.1, L, False), (0.1, R, False)]
@@ -65,6 +70,18 @@ check("one button double-clicked: goes off",
 g = gp.Gesture(("right/b",), 2)
 check("other buttons don't count", presses(g, [(0, "right/a", True), (0.2, "right/a", True)]), 0)
 
+check("one controller: its stick alone", gp.one_controller((L, R), {"right"}), (R,))
+check("one controller (left): its stick alone", gp.one_controller((L, R), {"left"}), (L,))
+check("both controllers: the gesture as set", gp.one_controller((L, R), {"left", "right"}), None)
+check("no controllers: nothing", gp.one_controller((L, R), set()), None)
+check("two buttons on one controller: as set", gp.one_controller(("right/a", "right/b"), {"right"}), None)
+check("one button: as set", gp.one_controller(("right/b",), {"right"}), None)
+g = gp.Gesture(gp.one_controller((L, R), {"right"}), 2)
+check("one controller: its stick clicked twice goes off",
+      presses(g, [(0, R, True), (0.1, R, False), (0.3, R, True), (0.4, R, False)]), 1)
+g = gp.Gesture(gp.one_controller((L, R), {"right"}), 2)
+check("one controller: its stick clicked once, nothing", presses(g, [(0, R, True), (0.1, R, False)]), 0)
+
 VR = ("left/thumbstick", "right/thumbstick", "right/b", "right/a")
 check("no pause_gesture: both sticks twice", gp.gesture_of({}, VR), ((L, R), 2))
 check("pause_gesture null: none", gp.gesture_of({"pause_gesture": None}, VR), None)
@@ -208,6 +225,22 @@ p.toggle("gesture", 0)
 q = gp.GamePause(lambda *a: None, changes.append, VR)
 check("a relay restarted while paused stays paused", (q.paused, q.reason), (True, "gesture"))
 check("and stops the services again", q.worker.jobs, ["enforce"])
+told = []
+q.tell_screens = told.append
+q.screens_seen(4242)
+check("the desktop that reports in first hears the pause (2026-10-16: it showed, the pointer was off)",
+      told, [True])
+q.screens_seen(4242)
+check("...once, not every second", told, [True])
+q.screens_seen(None)
+check("a sender without credentials changes nothing", told, [True])
+q.screens_seen(4343)
+check("a restarted desktop (new pid) hears it again", told, [True, True])
+p.tell_screens = told.append
+told.clear()
+p.toggle("gesture", 1)
+p.screens_seen(5151)
+check("running: a new desktop hears nothing", told, [False])
 
 # ---------------------------------------------------------------- hand tracking parked for games
 parked = lambda: os.path.exists(gp.PARK_PATH)  # noqa: E731

input/test/roles-test.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+"""Offline test of the input relay's device roles (node_role): the defaults, a saved role
+taking every node of a device, and "combo" (Keyboard + 3D pointer) splitting a keyboard with a
+touchpad by node. It loads input-relay.py without running it: no devices, sockets, or grabs.
+
+  input/test/roles-test.py [RELAY]   (default: input/input-relay.py next to this folder)
+"""
+import importlib.util
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+relay_path = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "input-relay.py")
+relay = importlib.util.module_from_spec(importlib.util.spec_from_loader("relay", loader=None))
+relay.__file__ = os.path.abspath(relay_path)
+sys.path.insert(0, os.path.dirname(os.path.abspath(relay_path)))  # its game_pause and vrws
+with open(relay_path) as f:
+    exec(compile(f.read(), relay_path, "exec"), relay.__dict__)
+
+failures = []
+
+
+def check(label, got, want):
+    ok = got == want
+    print(("ok    " if ok else "FAIL  ") + label + ("" if ok else f": got {got!r}, want {want!r}"), flush=True)
+    if not ok:
+        failures.append(label)
+
+
+def node(name, uniq, is_mouse, is_keyboard, bus=relay.BUS_BLUETOOTH):
+    return relay.Node("/dev/input/event999", -1, name, bus, 0x045E, 0x005F, uniq, is_mouse, is_keyboard)
+
+
+def roles(rule, nodes):
+    return [relay.node_role(rule, n, nodes) for n in nodes]
+
+
+# The keyboard of 2026-10-16: "Bluetooth 5.1 Keyboard" and "... Keyboard Mouse" (its touchpad), one address.
+combo = [node("Bluetooth 5.1 Keyboard", "d1:01:0b:5d:87:05", False, True),
+         node("Bluetooth 5.1 Keyboard Mouse", "d1:01:0b:5d:87:05", True, False)]
+check("one device: both nodes have the address as id", {n.id for n in combo}, {"d1:01:0b:5d:87:05"})
+check("no rule: the whole device is a pointer (it has a mouse node)", roles({}, combo), ["pointer", "pointer"])
+check("combo: keys pass through, the touchpad is the pointer", roles({"role": "combo"}, combo),
+      ["passthrough", "pointer"])
+check("pointer saved: every node", roles({"role": "pointer"}, combo), ["pointer", "pointer"])
+check("passthrough saved: every node", roles({"role": "passthrough"}, combo), ["passthrough", "passthrough"])
+check("ignore saved: every node", roles({"role": "ignore"}, combo), ["ignore", "ignore"])
+check("an unknown role: the default", roles({"role": "bogus"}, combo), ["pointer", "pointer"])
+
+# The Swiftpoint Z3: its keyboard node is its extra buttons, so it stays with its mouse by default.
+z3 = [node("Z3 Mouse", "aa:bb:cc:dd:ee:ff", True, False), node("Z3 Keyboard", "aa:bb:cc:dd:ee:ff", False, True)]
+check("Z3, no rule: both nodes pointer", roles({}, z3), ["pointer", "pointer"])
+
+kb = [node("TK-KB006 Keyboard", "11:22:33:44:55:66", False, True)]
+check("a plain keyboard: pass through", roles({}, kb), ["passthrough"])
+check("combo on a plain keyboard: still pass through", roles({"role": "combo"}, kb), ["passthrough"])
+mouse = [node("Some Mouse", "77:88:99:aa:bb:cc", True, False)]
+check("combo on a plain mouse: still the pointer", roles({"role": "combo"}, mouse), ["pointer"])
+both = [node("One Node Combo", "dd:ee:ff:00:11:22", True, True)]
+check("combo on one node with keys and motion: the pointer (keys go through its button map)",
+      roles({"role": "combo"}, both), ["pointer"])
+
+print("FAILED: " + ", ".join(failures) if failures else "all passed", flush=True)
+sys.exit(1 if failures else 0)

install.sh

@@ -117,14 +117,17 @@ step "6/12 power service (turns the displays off while the headset isn't used, e
 build "$root/power/build.sh"
 "$root/power/run.sh" install
 
-step "7/12 multi-screen desktop (ft-screens), Frametop Input Settings, Display Settings, and Remote Displays"
+step "7/12 multi-screen desktop (ft-screens), its keyboard (Full Keyboard), Frametop Input Settings, Display Settings, Remote Displays, and the Tour"
 build "$root/screens/build.sh"
 build "$root/stream/build.sh"  # ft-stream: remote displays (Frametop Remote Displays)
+build "$root/keyboard/build.sh"  # Frametop's keyboard: Full Keyboard, by TaiKeid (keyboard/FRAMETOP.md)
 "$root/desktops.sh" install >/dev/null
 "$root/input-settings/install.sh"
 "$root/display-settings/install.sh"
+"$root/tour/install.sh"
 "$root/remote-displays/install.sh"
 "$root/remote/install.sh"
+"$root/keyboard/run.sh" install
 on_frame "sed -i 's/^POINTER=0/POINTER=1/' ~/.config/frametop.conf; grep -q '^POINTER=' ~/.config/frametop.conf || echo 'POINTER=1' >> ~/.config/frametop.conf"
 echo "the launcher's Desktop entry now opens the multi-screen desktop; 3D mouse on (POINTER=1 in ~/.config/frametop.conf)"
 "$root/scripts/conf-migrate.sh"

justfile

@@ -14,13 +14,13 @@ test: test-python test-c test-bash
 # python3 has to import the dnf Qt stack and the locked packages together;
 # without that check the Qt tests would only skip. session/test is not here:
 # test_accessibility.py needs the host's AT-SPI and PyGObject, test_notifyfwd.py
-# the host's dbus-daemon and PyGObject.
+# and test_gtksettings.py the host's dbus-daemon and PyGObject.
 test-python:
     #!/usr/bin/env bash
     set -uo pipefail
     python3 -c 'import PySide6, numpy, cv2, sherpa_onnx' || { echo "python3 can't import PySide6, numpy, cv2, and sherpa_onnx"; exit 1; }
     status=0
-    for t in models/test/*.py input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py recorder/tests/*.py gaze/test/*.py gaze/rec/tests/*.py pack/test/*.py voice/test/*.py spotwatch/test/*.py; do
+    for t in models/test/*.py input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py recorder/tests/*.py gaze/test/*.py gaze/rec/tests/*.py pack/test/*.py voice/test/*.py spotwatch/test/*.py tour/test/*.py; do
         [ -f "$t" ] || continue
         if python3 "$t" >/dev/null; then echo "$t: ok"; else echo "$t: FAILED"; status=1; fi
     done
@@ -53,6 +53,9 @@ test-c:
         g++ -std=c++17 -Wall -Wextra -Werror -I. "$t" -o "build/tests/$n"
         "build/tests/$n" >/dev/null && echo "$t: ok"
     done
+    g++ -std=c++17 -O2 -Wall -Wextra -Werror screens/tests/rounded-test.cpp \
+      -o build/tests/rounded-test
+    build/tests/rounded-test && echo "screens/tests/rounded-test.cpp: ok"
 
 # shell scripts: syntax-gate everything that ships. No git: the image has
 # none, and a mounted checkout can trip "dubious ownership" — find lists

keyboard/.clang-format

@@ -0,0 +1,11 @@
+BasedOnStyle: LLVM
+IndentWidth: 4
+ColumnLimit: 105
+NamespaceIndentation: None
+AllowShortIfStatementsOnASingleLine: Never
+AllowShortLoopsOnASingleLine: false
+AllowShortFunctionsOnASingleLine: InlineOnly
+BreakBeforeBraces: Attach
+PointerAlignment: Left
+SortIncludes: CaseSensitive
+InsertBraces: true

keyboard/.editorconfig

@@ -0,0 +1,12 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+indent_style = space
+indent_size = 4
+trim_trailing_whitespace = true
+
+[*.{json,md,yml}]
+indent_size = 2

keyboard/.gitattributes

@@ -0,0 +1,2 @@
+# Preserve vendored SDK formatting; enforce whitespace checks on our own sources.
+third_party/** -whitespace

keyboard/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,30 @@
+---
+name: Bug report
+about: Report a problem with Full Keyboard on Steam Frame
+title: ''
+labels: ''
+assignees: ''
+---
+
+## What happened?
+
+Describe the expected and actual behavior.
+
+## Steps to reproduce
+
+1.
+2.
+3.
+
+## Setup
+
+- Full Keyboard version:
+- SteamOS / runtime version:
+- App and version:
+- Desktop window or floating app:
+- Keyboard layout, language, and theme:
+- Controller model:
+
+## Screenshot or log, if useful
+
+Remove passwords, private text, clipboard contents, and account details before posting.

keyboard/.github/pull_request_template.md

@@ -0,0 +1,7 @@
+## Change
+
+What problem does this solve, and what happens after the change?
+
+## Validation
+
+List relevant tests and say whether this was checked in the headset or only with automated tests. Note untested behavior.

keyboard/.gitignore

@@ -0,0 +1,12 @@
+/build/
+/out/
+/dist/
+/.cache/
+__pycache__/
+*.pyc
+CMakeUserPresets.json
+compile_commands.json
+.env
+.env.*
+/docs/**/*.drawio
+/_bugs.md

keyboard/AGENTS.md

@@ -0,0 +1,20 @@
+# Full Keyboard for Steam Frame development
+
+Follow any enclosing workspace instructions. This file covers project engineering; keep machine-specific notes, deployment records, and credentials outside this repository. The public project name is Full Keyboard for Steam Frame, the app label is Full Keyboard, and internal identifiers remain framekeyboard.
+
+- Read `README.md`, `docs/plan.md`, and the relevant architecture section before implementation. Mark milestone checkboxes only after their acceptance conditions pass.
+- The approved appearance is `design/index.html`. Native key geometry is `layouts/en-us.json`; visual constants are `themes/graphite.json`. Keep fixed-size press movement, the flat case, shallow key sides, and left Copy/Paste keys.
+- Cross-build deployment artifacts on the development host. Keep host and ARM64 output separate. Do not maintain a source checkout on Frame or run an ARM64 artifact on the host.
+- Treat file-based layouts, languages and themes with in-VR selection as core requirements. Read `docs/configuration.md`; keep key geometry, input mapping and styles independent. Do not equate relabeled keys with working language support.
+- Keep rendering, key state, input backend, and VR lifecycle separate. No actual input delivery from UI rendering code.
+- Default development behavior must not send keys to the user's focused application. Test input with a dedicated receiver before application validation.
+- Do not log typed text, credentials, clipboard contents, or browser field values. Do not add clipboard reads for debugging.
+- Distinguish logical key IDs, Linux evdev codes, and Steam key codes. Never reuse raw numeric codes across backends.
+- Every held key needs a release path on cancellation, target change, disconnect, hide, and shutdown. The stock keyboard must remain usable if the native process dies.
+- Keep manual launching by default. Explicit Start with SteamOS requires replacement and starts hidden through a user-local session unit; previews must never configure autostart. In replacement mode, X dismisses the panel while the app remains resident; only Settings exposes Quit Full Keyboard. With replacement off, X exits as before. Optional native keyboard replacement is default-off, uses temporary feature-checked hooks, preserves Steam typing targets, and must restore stock independently when the app stops progressing. Close native presentation completely, including its scene geometry; do not use browser-window hiding to suppress it. Leave requests native when their original owner cannot be restored safely. Never patch Steam files or persist a global summon remap.
+- Feature-check private runtime interfaces and fail without changing Steam files or global input settings.
+- Vendor external code only with a pinned revision and its license/attribution. The sibling overlay is a reference, not an implicit build dependency.
+- Git stays local until publication is requested. Do not copy build sysroots, credentials, machine logs, or recovery backups into Git.
+
+- Use `.clang-format` for our C++ files. Keep functions focused and comment non-obvious state ownership, coordinate/code conversions and cleanup ordering. Do not reformat vendored code without reason.
+- Run `ctest --test-dir build/host --output-on-failure` for core changes. Run the separate uinput smoke test only against its exclusively grabbed device; never turn it into a test of the user's focused app.

keyboard/CHANGELOG.md

@@ -0,0 +1,73 @@
+# Changelog
+
+## 0.8.1: Update handoff fix
+
+- Fixed in-app update shutdown reopening Steam's native keyboard. The old typing session and queued input now end before the updated keyboard starts, while Quit and failure recovery still restore native.
+- Kept background replacement updates hidden until a fresh keyboard request, even if Start with SteamOS was disabled during the session. Manual updates and updates with replacement disabled still show the panel normally.
+
+When updating from 0.7.0 or 0.8.0, Steam's native keyboard may still open once because the old version handles its own shutdown. Dismiss it; the next keyboard request can open Full Keyboard.
+
+## 0.8.0: Typing reliability and Steam dashboard fixes
+
+- Added an ordered output queue so rapid two-controller typing, Backspace and arrow keys keep their press order while delivery is pending. Hiding, dismissal, target changes and native handoff discard old queued input and ignore late results.
+- Made local-app text delivery asynchronous so a slow compositor does not block the keyboard. Input connections recover in the background without replaying old keys or text. Bundled the fixed libei 1.6.0 client to prevent crashes during interrupted input negotiation, without changing system libraries.
+- Fixed Steam search typing reaching a local app behind the dashboard. Text stays bound to the selected Steam field, follows switches between supported Steam and local-app targets, and stops when the field loses focus or disappears.
+- Improved native keyboard handoff and cleanup when owners change, the helper fails, or Steam's UI stalls. Abandoned installation attempts are cleaned up, native restoration is checked, and repeated failures use bounded retries.
+- Fixed the first keyboard request after background startup and SteamVR session restart, and restored replacement readiness after headset sleep or tracking loss without requiring an app relaunch.
+- Fixed Escape dismissal in Steam fields and dashboard sessions. Releasing a queued Backspace repeat now completes its delivery without another deletion, preventing a missing acknowledgement from forcing native fallback.
+- Removed the routine pending-text notice that flashed during typing. Reduced bridge polling and inspection work while idle, retaining independent recovery checks and bounded diagnostics that never include typed text.
+- Preserved saved profile selections when startup temporarily falls back to another supported profile. Missing or incompatible profiles now produce a useful fallback reason, and custom key IDs no longer collide with toolbar controls.
+- Fixed Japanese Romaji input depending on cosmetic key labels instead of resolved key symbols, and stopped idle Space from incorrectly starting composition.
+- Made Reload wait for pending settings work while the VR loop stays responsive. Unrelated preference changes can still save when startup configuration fails, and storage errors remain visible.
+- Manual reinstalls and downgrades now use fresh release directories. Replacement removes only the old marked production release after successful activation, or startup confirmation for in-app updates, while preserving settings and unmarked development releases.
+- Refactored VR lifecycle coordination and expanded requirements-based tests for routing, queued cancellation, dismissal, native fallback, recovery and controller ownership.
+
+## 0.7.0: Movement, native replacement and updates
+
+- Added a movement line below the keyboard with native-style colors and hover feedback. Hold it with the controller laser or AUX head cursor to move the keyboard; the same controller's thumbstick adjusts distance.
+- Renamed Appearance to General and added **Use Drag to Move** to select grip movement instead of the default laser line. Pinning blocks both movement methods and hides the line.
+- Added **Auto leveling** in General, enabled by default, to level small sideways tilts after releasing movement. Aligned the General checkboxes with their labels on the right.
+- Added experimental **Replace native keyboard** for supported dashboard, Steam-field and local-app requests. It is off by default and requires Steam Developer Mode with an available local debugger connection. Independent recovery restores the native keyboard when Full Keyboard becomes unavailable or stops progressing.
+- In replacement mode, × dismisses the panel while the app stays ready for the next request. **Quit Full Keyboard** appears in Settings, and **Use Steam keyboard** returns the current request to Steam.
+- Added optional **Start with SteamOS**, available only with replacement enabled. It starts the keyboard and recovery helpers in the background without showing the panel, ready for native keyboard requests. Startup after a device reboot has been manually verified.
+- Added **Check for updates** at the top of Settings. Checks run only when requested, report the latest-version result briefly, and keep a newer release available to download and install until the app exits.
+- Updates verify release checksums, archive contents, ARM64 architecture, version and profiles before installation. They restart the keyboard, preserve settings and startup preferences, and roll back if the new app fails to complete its startup checks. Failures show a short reason and retain normal logging.
+- Improved service recovery across SteamVR restarts and update handoffs. Saved startup preferences and owned service files stay consistent on Apply, Reload and launch, including after upgrades. Deliberate Quit pauses service restart, while ordinary manual launches retain successful exit codes.
+- Improved movement release detection, cancellation of native key repeats, and recovery from stalled or signal-stopped keyboard processes, with bounded allowances for slow work and SteamVR waits.
+
+## 0.6.0: Settings and keyboard preferences
+
+- Rebuilt Settings as separate cards for languages/layouts and appearance, with horizontal card scrolling and independent vertical scrolling within each card.
+- Added Layout, Language, and Theme dropdowns with five visible choices and scrolling. Menus can extend beyond the keyboard case and dismiss when clicked outside.
+- Moved Back, Apply and save, and Reload config into the Settings toolbar.
+- Added a Favorite checkbox for layout/language pairs. When favorites exist, a main-toolbar language button cycles the saved default and unique favorites while retaining the active theme and saved default.
+- Added a persistent pin button after the zoom controls to prevent accidental grip movement while retaining dashboard following, resizing, and recentering.
+- Added global Num Lock persistence across restarts and profile changes, plus a Hide numpad preference that narrows the case while keeping the remaining key sizes.
+- Fixed native keypad output disagreeing with restored Num Lock in physical uinput and external JIS modes. Digits and decimal wait for confirmed target lock state; keypad Enter and operators remain usable without lock feedback.
+- Fixed toolbar buttons overlapping Close when hiding the numpad in custom layouts.
+- Made favorite cycling faster and moved preference writes off the input thread, with ordered saves, shutdown flushing, and save-error reporting.
+- Fixed fractional dropdown scrolling and preserved precise wheel input in the desktop preview.
+- Added a British English layout and language profile, contributed by [Cr0ss0vr](https://github.com/cr0ss0vr).
+
+## 0.5.0: First release
+
+Full Keyboard for Steam Frame is a separately launched alternative keyboard for the Frame dashboard and local apps. It does not replace the stock keyboard or send input to streamed VR applications.
+
+- Composition commits before ordinary spaces, digits and punctuation, with failed-delivery protection.
+- Chinese/Korean engines load on demand; missing libraries leave the rest of the keyboard usable.
+- Added French, Spanish, Italian, Brazilian ABNT2, Russian and Ukrainian profiles, Simplified/Traditional Pinyin candidates, and Korean two-set Hangul composition. Language settings and system-layout limitations are documented.
+- Sticky left Super and momentary native right Super, with monochrome Frame and SteamOS icons.
+- Full-size keyboard with F1–F12, arrow and navigation keys, numpad, Ctrl/Alt/Shift, and dedicated Copy/Paste buttons.
+- Layout-independent Unicode character delivery, local Shift/Caps/AltGr and accents, plus native navigation, shortcuts and key repeat.
+- Controller laser typing, per-controller press/release haptics, and subtle hover feedback.
+- Grip movement and rotation, same-hand thumbstick depth adjustment, size controls, and horizon alignment after release.
+- Dashboard position and rotation following, saved placement, and relaunch-to-recenter recovery.
+- Dashboard-only visibility and input, with cancellation on hiding, tracking loss, or disconnect.
+- Editable JSON layouts, languages, and themes, with in-VR selection, favorites, and profile reload.
+- English and German profiles, plus Japanese romaji/kana composition, kanji candidates, and an external JIS IME mode.
+- Graphite and Midnight themes, a native desktop preview, and image export.
+- Incremental key redraws and native pixel uploads where supported, with an RGBA compatibility fallback and event-driven idle rendering.
+- User-local ARM64 installation, preserved settings, versioned rollback, release checksums, and manual installation instructions.
+- MIT-licensed project code, build documentation, and automated regression tests.
+
+See the [README](README.md) for installation and controls, and [runtime documentation](docs/runtime.md) for language requirements and known limitations.

keyboard/CMakeLists.txt

@@ -0,0 +1,270 @@
+cmake_minimum_required(VERSION 3.24)
+project(framekeyboard VERSION 0.8.1 LANGUAGES C CXX)
+
+set(CMAKE_CXX_STANDARD 20)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_EXTENSIONS OFF)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+# Keep build-machine directories out of __FILE__, assertions and debug paths.
+# RelWithDebInfo remains useful locally; packaging strips only its installed copy.
+if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
+    add_compile_options("-ffile-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=."
+                        "-ffile-prefix-map=${CMAKE_CURRENT_BINARY_DIR}=./build")
+    if(CMAKE_SYSROOT)
+        add_compile_options("-ffile-prefix-map=${CMAKE_SYSROOT}=/")
+    endif()
+endif()
+find_package(Python3 REQUIRED COMPONENTS Interpreter)
+find_package(PkgConfig REQUIRED)
+find_package(Threads REQUIRED)
+pkg_check_modules(NATIVE REQUIRED IMPORTED_TARGET cairo pangocairo xkbcommon json-c)
+pkg_check_modules(WAYLAND_CLIENT REQUIRED IMPORTED_TARGET wayland-client)
+# SteamOS libei 1.4.1 crashes on an aborted negotiation. Keep the fixed client
+# inside this app; never replace the runtime library used by Steam itself.
+add_subdirectory(third_party/libei)
+find_program(WAYLAND_SCANNER wayland-scanner REQUIRED)
+set(IME_XML "${CMAKE_CURRENT_SOURCE_DIR}/third_party/gamescope/gamescope-input-method.xml")
+set(IME_HEADER "${CMAKE_CURRENT_BINARY_DIR}/generated/gamescope-input-method-client.h")
+set(IME_CODE "${CMAKE_CURRENT_BINARY_DIR}/generated/gamescope-input-method-protocol.c")
+add_custom_command(OUTPUT "${IME_HEADER}" "${IME_CODE}"
+    COMMAND "${CMAKE_COMMAND}" -E make_directory "${CMAKE_CURRENT_BINARY_DIR}/generated"
+    COMMAND "${WAYLAND_SCANNER}" client-header "${IME_XML}" "${IME_HEADER}"
+    COMMAND "${WAYLAND_SCANNER}" private-code "${IME_XML}" "${IME_CODE}"
+    DEPENDS "${IME_XML}" VERBATIM)
+pkg_check_modules(SDL REQUIRED IMPORTED_TARGET sdl2)
+pkg_check_modules(VULKAN REQUIRED IMPORTED_TARGET vulkan)
+
+file(GLOB LAYOUT_FILES CONFIGURE_DEPENDS layouts/*.json)
+file(GLOB PROFILE_FILES CONFIGURE_DEPENDS layouts/*.json languages/*.json themes/*.json)
+set(PROFILE_HEADER "${CMAKE_CURRENT_BINARY_DIR}/generated/bundled_profiles.hpp")
+add_custom_command(OUTPUT "${PROFILE_HEADER}"
+    COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tools/validate_layout.py"
+            ${LAYOUT_FILES}
+    COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/tools/embed_defaults.py"
+            "${CMAKE_CURRENT_SOURCE_DIR}" "${PROFILE_HEADER}"
+    DEPENDS ${PROFILE_FILES} tools/embed_defaults.py tools/validate_layout.py
+    VERBATIM)
+add_library(keyboard_core src/vr_lifecycle.cpp src/vr_runtime.cpp src/config.cpp src/input.cpp src/output.cpp src/panel.cpp src/settings_ui.cpp src/settings_writer.cpp src/startup.cpp src/app.cpp src/instance.cpp src/placement.cpp src/placement_store.cpp src/grip.cpp src/ei_input.cpp src/japanese.cpp src/cjk.cpp src/text_input.cpp src/native_replacement.cpp src/updates.cpp src/frametop.cpp "${IME_HEADER}" "${IME_CODE}" "${PROFILE_HEADER}")
+target_include_directories(keyboard_core PUBLIC include PRIVATE "${CMAKE_CURRENT_BINARY_DIR}/generated")
+target_link_libraries(keyboard_core PUBLIC PkgConfig::NATIVE framekeyboard_ei PkgConfig::WAYLAND_CLIENT ${CMAKE_DL_LIBS} Threads::Threads)
+target_compile_options(keyboard_core PRIVATE -Wall -Wextra -Wpedantic -Wconversion)
+# Frame already supplies these engines. Minimal host builds can still run the
+# ordinary keyboard; unsupported composition profiles fail explicitly on Apply.
+pkg_check_modules(GLIB REQUIRED IMPORTED_TARGET glib-2.0)
+pkg_check_modules(PYZY QUIET IMPORTED_TARGET pyzy-1.0)
+pkg_check_modules(HANGUL QUIET IMPORTED_TARGET libhangul)
+foreach(engine PYZY HANGUL)
+    if(${engine}_FOUND)
+        if(engine STREQUAL "PYZY")
+            set(module pinyin)
+        else()
+            set(module hangul)
+        endif()
+        add_library(cjk_${module} MODULE src/cjk_engine.cpp)
+        target_link_libraries(cjk_${module} PRIVATE PkgConfig::GLIB PkgConfig::${engine})
+        target_compile_definitions(cjk_${module} PRIVATE FRAMEKEYBOARD_HAVE_${engine}=1)
+        target_compile_options(cjk_${module} PRIVATE -Wall -Wextra -Wpedantic -Wconversion)
+        if(CMAKE_CROSSCOMPILING)
+            target_link_options(cjk_${module} PRIVATE -fuse-ld=lld)
+        endif()
+        set_target_properties(cjk_${module} PROPERTIES
+            PREFIX "" OUTPUT_NAME "framekeyboard-${module}"
+            LIBRARY_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/engines"
+            CXX_VISIBILITY_PRESET hidden VISIBILITY_INLINES_HIDDEN YES)
+        add_dependencies(keyboard_core cjk_${module})
+        install(TARGETS cjk_${module} LIBRARY DESTINATION lib/framekeyboard)
+    elseif(CMAKE_CROSSCOMPILING)
+        message(FATAL_ERROR "Frame releases require ${engine} headers and libraries in the sysroot")
+    else()
+        message(STATUS "${engine} unavailable: this host build cannot compose that language")
+    endif()
+endforeach()
+
+# Link the matching architecture from the sysroot; embed only the device runtime path.
+if(CMAKE_CROSSCOMPILING)
+    set(OPENVR_LIBRARY "${CMAKE_SYSROOT}/opt/steamvr/bin/linuxarm64/libopenvr_api.so" CACHE FILEPATH "OpenVR library")
+    set(OPENVR_RUNTIME_PATH "/opt/steamvr/bin/linuxarm64")
+else()
+    find_library(OPENVR_LIBRARY openvr_api HINTS
+        "$ENV{HOME}/.local/share/Steam/steamapps/common/SteamVR/bin/linux64"
+        "$ENV{HOME}/.local/share/Steam/ubuntu12_64" REQUIRED)
+    # Frametop's build names SteamVR's folder first (keyboard/build.sh).
+    if(NOT OPENVR_RUNTIME_PATH)
+        get_filename_component(OPENVR_RUNTIME_PATH "${OPENVR_LIBRARY}" DIRECTORY)
+    endif()
+endif()
+add_library(overlay_texture third_party/overlay_texture/vk_texture.cpp)
+target_include_directories(overlay_texture PUBLIC third_party/overlay_texture third_party/openvr)
+target_link_libraries(overlay_texture PUBLIC PkgConfig::VULKAN "${OPENVR_LIBRARY}")
+add_executable(framekeyboard src/main.cpp src/preview.cpp src/vr.cpp)
+add_executable(framekeyboard-native-guard src/native_restore_guard.cpp)
+target_include_directories(framekeyboard-native-guard PRIVATE third_party/openvr)
+target_link_libraries(framekeyboard-native-guard PRIVATE PkgConfig::NATIVE "${OPENVR_LIBRARY}")
+set_target_properties(framekeyboard-native-guard PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE INSTALL_RPATH "${OPENVR_RUNTIME_PATH}")
+add_dependencies(framekeyboard framekeyboard-native-guard)
+target_link_libraries(framekeyboard PRIVATE keyboard_core overlay_texture PkgConfig::SDL)
+target_compile_options(framekeyboard PRIVATE -Wall -Wextra -Wpedantic -Wconversion)
+target_compile_definitions(framekeyboard PRIVATE FRAMEKEYBOARD_VERSION="${PROJECT_VERSION}")
+# Development binaries find these beside themselves; packages use share/framekeyboard.
+# Do not embed an absolute source-tree path in the executable for this fallback.
+file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/vr")
+foreach(manifest actions.json frame-controller.json)
+    configure_file("vr/${manifest}" "vr/${manifest}" COPYONLY)
+endforeach()
+set_target_properties(framekeyboard PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE INSTALL_RPATH "${OPENVR_RUNTIME_PATH}")
+
+file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/bridge")
+foreach(bridge_file native_bridge.py steam_bridge.js)
+    configure_file("bridge/${bridge_file}" "bridge/${bridge_file}" COPYONLY)
+endforeach()
+file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/update")
+configure_file(update/update_helper.py update/update_helper.py COPYONLY)
+include(CTest)
+if(BUILD_TESTING)
+    add_executable(vr-lifecycle-tests tests/vr_lifecycle_tests.cpp)
+    target_link_libraries(vr-lifecycle-tests PRIVATE keyboard_core)
+    target_compile_options(vr-lifecycle-tests PRIVATE -Wall -Wextra -Wpedantic)
+    add_test(NAME vr-lifecycle COMMAND vr-lifecycle-tests)
+    add_executable(ei-handshake-tests tests/ei_handshake_tests.cpp)
+    target_link_libraries(ei-handshake-tests PRIVATE framekeyboard_ei)
+    add_test(NAME ei-handshake COMMAND ei-handshake-tests)
+    add_executable(output-tests tests/output_tests.cpp)
+    target_link_libraries(output-tests PRIVATE keyboard_core)
+    add_test(NAME ordered-output COMMAND output-tests)
+    add_executable(startup-tests tests/startup_tests.cpp)
+    target_link_libraries(startup-tests PRIVATE keyboard_core)
+    add_test(NAME startup COMMAND startup-tests)
+    find_program(SYSTEMD_ANALYZE systemd-analyze)
+    if(SYSTEMD_ANALYZE AND NOT CMAKE_CROSSCOMPILING)
+        add_test(NAME startup-unit COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/startup_unit_tests.py"
+            "$<TARGET_FILE:startup-tests>" "${SYSTEMD_ANALYZE}")
+    endif()
+    add_executable(update-tests tests/update_tests.cpp)
+    target_link_libraries(update-tests PRIVATE keyboard_core)
+    add_test(NAME updates COMMAND update-tests)
+    if(NOT CMAKE_CROSSCOMPILING)
+        add_test(NAME update-helper COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/update_helper_tests.py"
+            "${CMAKE_CURRENT_SOURCE_DIR}/update/update_helper.py")
+    endif()
+    add_executable(frametop-tests tests/frametop_tests.cpp)
+    target_link_libraries(frametop-tests PRIVATE keyboard_core)
+    add_test(NAME frametop COMMAND frametop-tests)
+    add_executable(native-replacement-tests tests/native_replacement_tests.cpp)
+    target_link_libraries(native-replacement-tests PRIVATE keyboard_core)
+    add_test(NAME native-replacement COMMAND native-replacement-tests)
+    if(NOT CMAKE_CROSSCOMPILING)
+        add_test(NAME timing-ordering COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/timing_order_tests.py"
+            "${CMAKE_CURRENT_SOURCE_DIR}")
+        add_test(NAME native-helper-spawn COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/native_helper_spawn_tests.py"
+            "$<TARGET_FILE:native-replacement-tests>"
+            "${CMAKE_CURRENT_SOURCE_DIR}/bridge/native_bridge.py")
+        add_test(NAME native-bridge-transport COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/native_bridge_transport_tests.py"
+            "${CMAKE_CURRENT_SOURCE_DIR}/bridge/native_bridge.py")
+        add_test(NAME replacement-measurement COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/replacement_measurement_tests.py"
+            "${CMAKE_CURRENT_SOURCE_DIR}/scripts/measure-replacement.py")
+        find_program(NODE_EXECUTABLE node)
+        if(NODE_EXECUTABLE)
+            add_test(NAME native-bridge COMMAND "${NODE_EXECUTABLE}"
+                "${CMAKE_CURRENT_SOURCE_DIR}/tests/native_bridge_tests.js"
+                "${CMAKE_CURRENT_SOURCE_DIR}/bridge/steam_bridge.js")
+        endif()
+    endif()
+    # Opt-in timing probe: never sends input; --vr uses its own hidden overlay.
+    add_executable(render-benchmark tests/render_benchmark.cpp)
+    target_link_libraries(render-benchmark PRIVATE keyboard_core overlay_texture Threads::Threads)
+    target_compile_options(render-benchmark PRIVATE -Wall -Wextra -Wpedantic)
+    set_target_properties(render-benchmark PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE INSTALL_RPATH "${OPENVR_RUNTIME_PATH}")
+    add_executable(language-tests tests/language_tests.cpp)
+    target_link_libraries(language-tests PRIVATE keyboard_core)
+    target_compile_options(language-tests PRIVATE -Wall -Wextra -Wpedantic)
+    add_test(NAME languages COMMAND language-tests --core)
+    add_test(NAME korean-input COMMAND language-tests --korean)
+    add_test(NAME chinese-input COMMAND language-tests --chinese)
+    set_tests_properties(korean-input chinese-input PROPERTIES SKIP_RETURN_CODE 77)
+    if(NOT CMAKE_CROSSCOMPILING)
+        add_test(NAME optional-engines COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/engine_loading_tests.py" "$<TARGET_FILE:language-tests>")
+    endif()
+    add_executable(japanese-tests tests/japanese_tests.cpp)
+    target_link_libraries(japanese-tests PRIVATE keyboard_core)
+    add_test(NAME japanese-input COMMAND japanese-tests)
+    add_executable(settings-tests tests/settings_tests.cpp)
+    target_link_libraries(settings-tests PRIVATE keyboard_core)
+    target_compile_options(settings-tests PRIVATE -Wall -Wextra -Wpedantic)
+    add_test(NAME settings COMMAND settings-tests)
+    add_executable(keyboard-tests tests/keyboard_tests.cpp)
+    add_executable(render-tests tests/render_tests.cpp)
+    target_link_libraries(render-tests PRIVATE keyboard_core)
+    target_compile_options(render-tests PRIVATE -Wall -Wextra -Wpedantic)
+    add_test(NAME rendering COMMAND render-tests)
+    target_link_libraries(keyboard-tests PRIVATE keyboard_core)
+    target_compile_options(keyboard-tests PRIVATE -Wall -Wextra -Wpedantic)
+    add_test(NAME keyboard-core COMMAND keyboard-tests)
+    add_executable(placement-instance-tests tests/placement_instance_tests.cpp)
+    target_link_libraries(placement-instance-tests PRIVATE keyboard_core Threads::Threads)
+    target_compile_options(placement-instance-tests PRIVATE -Wall -Wextra -Wpedantic)
+    add_test(NAME placement-instance COMMAND placement-instance-tests)
+    if(NOT CMAKE_CROSSCOMPILING)
+        add_test(NAME installer COMMAND "${Python3_EXECUTABLE}"
+            "${CMAKE_CURRENT_SOURCE_DIR}/tests/installer_tests.py"
+            "${CMAKE_CURRENT_SOURCE_DIR}" "$<TARGET_FILE:framekeyboard>")
+        pkg_check_modules(EIS QUIET IMPORTED_TARGET libeis-1.0>=1.6)
+        if(EIS_FOUND)
+            add_executable(ei-recovery-tests tests/ei_recovery_tests.cpp)
+            target_link_libraries(ei-recovery-tests PRIVATE keyboard_core PkgConfig::EIS Threads::Threads)
+            pkg_check_modules(WAYLAND_SERVER QUIET IMPORTED_TARGET wayland-server)
+            if(WAYLAND_SERVER_FOUND)
+                set(IME_SERVER_HEADER "${CMAKE_CURRENT_BINARY_DIR}/generated/gamescope-input-method-server.h")
+                add_custom_command(OUTPUT "${IME_SERVER_HEADER}"
+                    COMMAND "${WAYLAND_SCANNER}" server-header "${IME_XML}" "${IME_SERVER_HEADER}"
+                    DEPENDS "${IME_XML}" VERBATIM)
+                target_sources(ei-recovery-tests PRIVATE "${IME_SERVER_HEADER}")
+                target_include_directories(ei-recovery-tests PRIVATE "${CMAKE_CURRENT_BINARY_DIR}/generated")
+                target_link_libraries(ei-recovery-tests PRIVATE PkgConfig::WAYLAND_SERVER)
+                target_compile_definitions(ei-recovery-tests PRIVATE FRAMEKEYBOARD_TEST_WAYLAND_SERVER=1)
+            endif()
+            add_test(NAME ei-recovery COMMAND ei-recovery-tests)
+        endif()
+    endif()
+    # Opt-in only: maps a disposable window and verifies focus before each key.
+    pkg_check_modules(X11 REQUIRED IMPORTED_TARGET x11)
+    add_executable(unicode-target-probe tests/unicode_target_probe.cpp)
+    target_link_libraries(unicode-target-probe PRIVATE keyboard_core PkgConfig::X11)
+    add_executable(japanese-target-probe tests/japanese_target_probe.cpp)
+    target_link_libraries(japanese-target-probe PRIVATE keyboard_core PkgConfig::X11)
+    add_executable(ei-target-probe tests/ei_target_probe.cpp)
+    target_link_libraries(ei-target-probe PRIVATE keyboard_core PkgConfig::X11)
+    # Opt-in read-only validation against the installed Frame controller models.
+    add_executable(stick-component-probe tests/stick_component_probe.cpp)
+    target_link_libraries(stick-component-probe PRIVATE keyboard_core overlay_texture)
+    set_target_properties(stick-component-probe PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE INSTALL_RPATH "${OPENVR_RUNTIME_PATH}")
+    add_executable(vr-panel-probe tests/vr_panel_probe.cpp)
+    target_link_libraries(vr-panel-probe PRIVATE keyboard_core overlay_texture)
+    set_target_properties(vr-panel-probe PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE INSTALL_RPATH "${OPENVR_RUNTIME_PATH}")
+    # Opt-in: sends physical haptic output, so it must not run in CTest.
+    add_executable(haptics-probe tests/haptics_probe.cpp)
+    target_link_libraries(haptics-probe PRIVATE keyboard_core overlay_texture)
+    set_target_properties(haptics-probe PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE INSTALL_RPATH "${OPENVR_RUNTIME_PATH}")
+    # Intentionally excluded from CTest: requires an accessible kernel uinput device.
+    add_executable(uinput-smoke-test tests/uinput_smoke.cpp)
+    target_link_libraries(uinput-smoke-test PRIVATE keyboard_core)
+endif()
+install(TARGETS framekeyboard framekeyboard-native-guard RUNTIME DESTINATION bin)
+install(DIRECTORY docs DESTINATION .)
+install(DIRECTORY layouts languages themes config vr DESTINATION share/framekeyboard)
+install(FILES bridge/native_bridge.py bridge/steam_bridge.js DESTINATION share/framekeyboard/bridge)
+install(FILES update/update_helper.py DESTINATION share/framekeyboard/update)
+install(FILES third_party/README.md DESTINATION third_party)
+install(FILES third_party/openvr/LICENSE DESTINATION third_party/openvr)
+install(FILES third_party/overlay_texture/LICENSE DESTINATION third_party/overlay_texture)
+install(FILES third_party/libei/COPYING third_party/libei/README.md third_party/libei/provenance.json DESTINATION third_party/libei)
+
+install(FILES third_party/gamescope/gamescope-input-method.xml DESTINATION third_party/gamescope)
+
+install(FILES LICENSE README.md CHANGELOG.md CONTRIBUTING.md DESTINATION .)

keyboard/CMakePresets.json

@@ -0,0 +1,25 @@
+{
+  "version": 5,
+  "cmakeMinimumRequired": {"major": 3, "minor": 24, "patch": 0},
+  "configurePresets": [
+    {
+      "name": "host",
+      "displayName": "Host development build",
+      "generator": "Ninja",
+      "binaryDir": "${sourceDir}/build/host",
+      "cacheVariables": {"CMAKE_BUILD_TYPE": "Debug"}
+    },
+    {
+      "name": "frame-arm64",
+      "displayName": "Steam Frame ARM64 cross-build",
+      "generator": "Ninja",
+      "binaryDir": "${sourceDir}/build/frame-arm64",
+      "toolchainFile": "${sourceDir}/cmake/steam-frame-aarch64.cmake",
+      "cacheVariables": {"CMAKE_BUILD_TYPE": "RelWithDebInfo"}
+    }
+  ],
+  "buildPresets": [
+    {"name": "host", "configurePreset": "host"},
+    {"name": "frame-arm64", "configurePreset": "frame-arm64"}
+  ]
+}

keyboard/CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing
+
+Full Keyboard for Steam Frame welcomes bug reports, review, and focused fixes. The app serves the Frame dashboard and local applications. Optional native keyboard replacement is experimental and must preserve native fallback when the app is unavailable. Typing into streamed VR apps is outside the current scope.
+
+## Report a bug
+
+Include the app version, SteamOS/runtime version, controller model, affected app, desktop versus floating-window mode, selected language/layout, reproduction steps, and expected/actual result. Include a screenshot or a short log only if it helps. Remove passwords, private text, account details, and clipboard contents.
+
+For Japanese input, include a harmless example reading, the expected conversion, and the selected Romaji/Kana/JIS mode. Feedback from fluent Japanese users is especially useful.
+
+## Dependencies and third-party code
+
+Do not bundle whole third-party projects inside this repository, including through submodules or source snapshots. Keep external projects separate and use installed libraries, tools, or services through their documented interfaces. Normal builds and installation must not implicitly download external projects or large runtime assets.
+
+Occasional reference files, headers, protocol definitions, and small helpers are fine when they serve a clear purpose. Record their upstream source, exact revision, local modifications, and applicable licenses and attribution. Check embedded components and transitive dependencies; a project's root license may not cover every file. Preserve the required notices in source and release packages.
+
+Document external dependencies and any separate setup they require. Keep optional dependencies optional during build, installation, and startup. Do not enable autostart, start external services, or change system configuration implicitly.
+
+## Optional features and capabilities
+
+Optional features must leave ordinary keyboard operation usable when a dependency is absent, loading, incompatible, or disconnected. Verify readiness and capabilities through the dependency's interface rather than assuming that an installed file or running process is sufficient. Keep integration-specific code behind a defined interface so the UI and profile data do not depend on a particular implementation.
+
+External interfaces must have a documented version and capability contract. Report capabilities for the active configuration, including supported languages, formats, and operations where relevant. Define compatibility and locale matching explicitly. Validate requests against current capabilities, handle capability changes, and reject unsupported operations without silently changing the requested behavior.
+
+Express capability-dependent controls declaratively in profile data. Account for the selected layout, language, and input backend. Use a small validated set of conditions; do not execute scripts or arbitrary expressions from profile files. Hidden controls must be absent from both rendering and hit testing. Keep cancellation controls accessible during active operations, and apply visibility changes without stranding held input or moving another action under a pressed pointer.
+
+These are contribution requirements, not a claim that every interface or profile condition already exists. Include parsing, validation, documentation, and tests when introducing a new contract or schema field.
+
+## Version ownership
+
+Only the project owner, TaiKeid, may bump the project version. Contributors must not change the version in `CMakeLists.txt` or update release-version references as part of their contributions. The owner handles these changes during release preparation.
+
+## Change the code
+
+Read [building](docs/building.md), [architecture](docs/architecture.md), and [configuration](docs/configuration.md). Keep renderer, key state, backend, and profile data separate. Use `.clang-format` for project C++; preserve vendored code's style and attribution.
+
+Run the core test suites for behavior changes. Test input with fake sinks or a dedicated receiver, never an arbitrary focused app. Cover lost releases, tracking loss, disconnects, hiding, and shutdown when changing interaction ownership. Add comments where state lifetime, coordinate transforms, or event ordering are not obvious.
+
+Submit focused PRs and describe the behavior, validation, and remaining limits. Separate unrelated fixes, features, UI changes, and refactoring when they can be reviewed independently. Do not include sysroots, builds, device logs, personal paths, credentials, or user settings. Do not mix publication changes into unrelated PRs.
+
+Every owned resource and subprocess needs a cleanup path for cancellation, hide, disconnect, and shutdown. Invalidate late asynchronous results and guard delayed input against destination changes. Do not terminate shared external services. Bound buffers and waits, and keep blocking work off the VR loop. Deliver text through the existing Unicode transport and retain unsent text when delivery fails. Do not log or persist user input.
+
+Verify installation and packaging from a clean checkout without relying on untracked files or private local setup. For affected paths, cover missing dependencies, capability changes, unsupported configurations, resource cleanup, stale results, visibility and hit testing, non-US text delivery, and failed commits. State whether validation ran on the host, an ARM64 cross-build, the Frame, or in the headset. An ARM64 build alone does not verify runtime behavior on the device.
+
+## Review priorities
+
+Input cancellation, focus routing, two-controller ownership, placement persistence, resource cleanup, parsing of custom profiles, and runtime compatibility deserve close review. Tests and successful builds do not replace in-headset validation; say which one was performed.

keyboard/FRAMETOP.md

@@ -0,0 +1,48 @@
+# Full Keyboard in Frametop
+
+This folder is [Full Keyboard for Steam Frame](https://github.com/TaiKeid/steam-frame-full-keyboard) by TaiKeid and contributors, under the MIT License ([LICENSE](LICENSE), Copyright (c) 2026 TaiKeid and contributors). Its own third-party parts and their licences are in [third_party/README.md](third_party/README.md). Frametop runs it as its keyboard: on the desktop's screens, in the SteamVR dashboard, and in place of Steam's own keyboard.
+
+It was imported at 0.8.1, commit `ef915ef75169ac886b7de86d18df2e59150cde2f`, with `git subtree --squash`; the squash commit carries TaiKeid as its author. Everything here is theirs except the changes listed below.
+
+## Frametop's changes
+
+- `--frametop` (`src/main.cpp`): the VR keyboard, typing through gamescope (libei) as before, started enabled and hidden. Its settings live in `~/.config/frametop-keyboard`. It has no updater and no startup unit: Frametop's service starts it with SteamVR, and it updates with Frametop. The first run turns the Steam keyboard replacement on. Quitting it from its panel exits with status 3, so the service leaves it closed until SteamVR starts again.
+- `src/frametop.cpp`, `include/framekeyboard/frametop.hpp`: the link to Frametop's input relay (below), and a key sink that sends typing to the desktop or to Steam.
+- `src/vr.cpp`: the panel can show over Frametop's screens with the dashboard closed, while Frametop asked for it; lasers can click it there; it tells Frametop whether it's up.
+- `src/placement.cpp`, `include/framekeyboard/placement.hpp`: in Frametop's mode it opens where Frametop's own keyboard did, 0.70 m ahead and 0.35 m under the eyes, facing them, instead of 0.85 m ahead and 0.65 m under, so a hand's laser reaches its bottom row without lowering the hand past where Frametop puts the laser away. Under the dashboard it sits as before.
+- `src/app.cpp`: no Updates button and no Start with SteamOS setting in Frametop's mode.
+- `src/japanese.cpp`, `tests/japanese_tests.cpp`: Anthy also loads as `libanthy-unicode.so.0`, Fedora's name for it.
+- `CMakeLists.txt`: the above, `frametop-tests`, and `OPENVR_RUNTIME_PATH` can be set, so the build loads SteamVR's own `libopenvr_api` first like Frametop's other programs.
+- Frametop's own files: `build.sh`, `run.sh`, `frametop-keyboard.service`, and this file.
+
+## How it works with Frametop
+
+Frametop's desktop runs in a nested KWin that gamescope never sees, so keys sent to gamescope can't reach it. The input relay (`input/input-relay.py`) owns typing for the desktop already: it knows where typing goes (the panel clicked last, from ft-screens), and it types Frametop Voice's text there. Full Keyboard talks to it over the relay's abstract datagram socket, `@frametop_relay`:
+
+- Full Keyboard says `kbhello OPEN` every second, and at once when it opens or closes. The relay answers each hello, and any change, with `kb DESKTOP SURFACE`: whether typing goes to the desktop, and whether the screens show with Frametop not paused (then it may show over them with the dashboard closed).
+- While typing goes to the desktop, its keys go to the relay as `kbkey CODE 0|1` and composed text as `kbtext TEXT`. The relay types them on the desktop's seat through ft-screens, in order with dictation's typing; the text goes through the desktop's keyboard layout and isn't logged. Otherwise they go to gamescope as before. A key comes up where it went down, whichever way typing goes by then, and when Full Keyboard stops, the relay lets go of the keys it held.
+- Opening it: the relay tells ft-screens it runs (`fullkb OPEN`), and ft-screens then passes on its `vrkeyboard` commands (a desktop text field gets focus, a mapped button, spotwatch, a layout reset) as `kbpanel show|hide`, which the relay sends Full Keyboard as `kb show|hide`. Frametop's older panel (`screens/keyboard.cpp`) is used only while Full Keyboard isn't running.
+- A click on its panel (`org.framekeyboard.*` overlays) doesn't take typing away from the desktop.
+- Its Steam keyboard replacement (`bridge/`) is theirs, unchanged.
+
+Only same-user messages count on either side. Full Keyboard takes word only from the relay's own name, and sends desktop typing only while that word is under 3 seconds old, so a stopped relay never leaves typing addressed to a name another process could take. The relay takes keys and text only from the address that said hello.
+
+Known limits: on the desktop, keys go by their place on the keyboard, so pick the same layout in Full Keyboard as the desktop's. Text the desktop's layout has no key for (Chinese, Japanese, Korean, emoji) can't be typed on the desktop yet; it reaches Steam's fields as before.
+
+## Building and running
+
+```
+keyboard/build.sh                          # build/frametop/framekeyboard, in the dev container
+FULL_KEYBOARD_TESTS=1 keyboard/build.sh    # with its tests (ctest)
+keyboard/run.sh install | uninstall        # frametop-keyboard.service, starts with SteamVR
+keyboard/run.sh status | restart | log [lines]
+input/test/keys-test.py                    # the relay's side, offline
+```
+
+## Updating from upstream
+
+```
+git subtree pull --prefix keyboard https://github.com/TaiKeid/steam-frame-full-keyboard <tag> --squash
+```
+
+Then build with its tests, run `input/test/keys-test.py`, and update the version and commit above.

keyboard/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 TaiKeid and contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

keyboard/README.md

@@ -0,0 +1,137 @@
+# Full Keyboard for Steam Frame
+
+A full-size alternative to Steam Frame's built-in virtual keyboard, made for everyday use on the headset. It adds proper arrow keys, a numpad, Ctrl and Alt, and dedicated Copy and Paste buttons for browsing, signing in, editing text, and using local apps.
+
+**Full Keyboard is launched manually by default.** By default, SteamOS's keyboard button opens the original keyboard. An experimental **Replace native keyboard** option in Experimental redirects supported requests while Full Keyboard is running. It requires Steam Developer Mode and falls back to the original keyboard when unavailable. Enable **Start with SteamOS** in the same card to start replacement in the background at login. It stays hidden until summoned. See [native replacement](docs/native-replacement.md).
+
+![Full Keyboard with the Graphite theme](docs/images/keyboard-graphite.png)
+
+*The default English layout and Graphite theme, captured from the app's native renderer.*
+
+## What it does
+
+- Full-size layout with F1–F12, navigation keys, arrow keys, numpad, Ctrl, Alt, and Shift.
+- Dedicated Copy and Paste icons that send Ctrl+C and Ctrl+V.
+- Native Enter and numpad Enter keys. The focused app decides whether Enter submits or inserts a newline.
+- Controller laser typing, press/release haptic clicks, and subtle hover feedback.
+- Move with the laser handle below the keyboard, or enable grab movement in Settings. Size controls are available in both modes.
+- Dashboard following, saved placement, and relaunch-to-recenter recovery.
+- Editable layouts, languages, and themes, with a settings menu inside VR.
+- English, German, French, Spanish, Italian, Brazilian Portuguese, Russian, Ukrainian, Japanese, Chinese and Korean options. Chinese has Simplified and Traditional Pinyin; Korean uses two-set Hangul.
+
+## Where it works
+
+Full Keyboard is a quality-of-life tool for the **on-device experience**. It is designed for the Frame dashboard and local apps, including desktop and floating app windows. Focus a text field in the app, then type using the controller laser.
+
+It is **not a keyboard for streamed VR applications such as VRChat**. Closing the dashboard hides the keyboard and releases its input. It does not capture controls or send keys to a streamed VR app. Reopening the dashboard brings it back.
+
+This is an early release. App focus, non-US keymaps, and Japanese input can behave differently across applications; see [known limitations and troubleshooting](docs/runtime.md). Japanese composition has automated and dedicated-receiver coverage, but still needs feedback from fluent Japanese users.
+
+## Install manually
+
+No AI tools, compilation, `sudo`, or SteamOS write access are needed for the release package.
+
+1. On the **Steam Frame**, open the [GitHub Releases page](https://github.com/TaiKeid/steam-frame-full-keyboard/releases) in a browser. Download `framekeyboard-0.8.1-aarch64.tar.gz` and `SHA256SUMS` from the release assets into **Downloads**. Do not download GitHub's automatic source-code archive for this installation.
+2. Open a terminal in the Frame's desktop environment and run:
+
+   ```sh
+   cd ~/Downloads
+   sha256sum -c SHA256SUMS
+   tar -xzf framekeyboard-0.8.1-aarch64.tar.gz
+   cd framekeyboard-0.8.1-aarch64
+   bash install.sh
+   ```
+
+   Continue only if the checksum reports `OK`. For a newer version, use its archive and extracted directory names.
+3. Return to the Frame's dashboard and open **Full Keyboard** from the app launcher. You can also launch it from a terminal:
+
+   ```sh
+   ~/.local/bin/framekeyboard
+   ```
+
+The installer adds a desktop application entry named **Full Keyboard**. If you also want a Steam Library shortcut, use Steam's **Add a Non-Steam Game** option and select Full Keyboard. A manually created shortcut has its own name; rename an old FrameKeyboard shortcut in its Properties if needed.
+
+Everything installs under your home directory. The installer preserves settings and custom profiles, replaces the current production release, and does not enable autostart or alter Steam files. Development deployments and older unmarked release directories are preserved. Internal filenames and commands remain `framekeyboard`.
+
+To update from the installed keyboard, open Settings and press **Check for updates** at the top center. If a newer stable release is available, press **Download and install update**. The keyboard verifies the archive, installs it and restarts, preserving your settings. The old production release stays until the new one confirms a healthy start, then is removed. A failed start rolls back automatically. See [updates](docs/updates.md).
+
+For a manual update, quit Full Keyboard first. With native replacement enabled, use Settings → Quit Full Keyboard. Download the new release assets and repeat the installation steps. The installer supports reinstalling the same version or installing an older one, and refuses while Full Keyboard is running. Manual rollback means reinstalling an older release package; see [rollback and removal](docs/runtime.md#update-rollback-and-removal).
+
+## Use the keyboard
+
+| Action | Control |
+| --- | --- |
+| Type | Point at a key and press the controller trigger |
+| Use a shortcut | Tap Ctrl, Alt, or Shift, then another key; or hold the modifier with one controller and press a key with the other |
+| Super shortcuts | Tap the left Frame icon, then a key, or hold it with the other controller |
+| Desktop application menu | Tap the right SteamOS icon for a native right Super press/release; the desktop controls its action |
+| Clear a tapped modifier | Tap it again |
+| Copy / paste | Use the two icons on the left; select text first when copying |
+| Move / rotate | Unpin, point at the line below the keyboard, hold the trigger, move your hand, then release |
+| Use the headset cursor | Press AUX to show the head cursor. Aim at keys with your head and click AUX; hold AUX on the movement line to move the keyboard |
+| Use grab movement | Enable **Use Drag to Move** in the **General** settings card and apply. Point anywhere on the keyboard and hold grab/grip; the same hand's thumbstick adjusts depth |
+| Prevent accidental movement | Toggle the pin icon; the keyboard still follows the dashboard |
+| Cycle favorite layouts and languages | Press the language abbreviation beside the pin icon |
+| Move farther / closer | While holding the movement line or grabbing, push that same controller's thumbstick up / down |
+| Change size | Use the smaller / larger icons in the toolbar |
+| Hide the numpad | In Settings, check General → Hide numpad and Apply and save |
+| Change layout, language, or theme | Open the gear icon, choose profiles, then **Apply and save** |
+| Recenter | Use the recenter icon, or launch Full Keyboard again |
+| Close | Use × to hide the panel in replacement mode; otherwise it exits the app |
+| Quit replacement mode | Open Settings → Quit Full Keyboard |
+
+**If the keyboard has drifted out of reach, launch the app again.** When it is already running, another launch recenters the existing keyboard beneath the current dashboard instead of opening a second one. Its size is preserved. If the dashboard is closed, recentering waits until it opens.
+
+Closing and reopening normally restores the last position, rotation, size and Num Lock state. Numpad visibility is saved globally across all layouts. Moving or tilting the dashboard carries the keyboard with it, preserving your chosen offset. With General → Auto leveling enabled, a sideways lean within 5° eases level over 500 ms after releasing movement. Auto leveling is on by default; disable it to keep the angle you set. This preserves the desk-like forward tilt and never fights your hand during a grab.
+
+## Languages and customization
+
+Select a language in Settings and **Apply and save**. The normal Frame launcher
+sends characters from that layout independently of the system keyboard layout,
+including Shift, Caps Lock, AltGr and dead-key accents. No matching SteamOS layout
+or relaunch is needed. A fresh install defaults to English; later launches use
+your saved selection. System locale and system layout switching do not change it.
+See [language setup and switching](docs/languages.md).
+
+For Japanese, pick **Romaji**, **Kana**, or **JIS** from the Language dropdown in Settings. Romaji and Kana compose text in the keyboard and use the system Anthy dictionary for kanji conversion. Space converts; Enter commits the composition. Another Enter sends a normal Enter key. JIS mode requires an existing system IME. See [Japanese input](docs/japanese.md).
+
+Chinese and Korean compose locally and can be selected alongside English and integrated Japanese without a system layout change. [The language guide](docs/languages.md#chinese-and-korean-composition) explains candidates, Hangul, and input limitations.
+
+Copy a bundled JSON profile from `~/.local/share/framekeyboard/current/share/framekeyboard/` into the matching folder below, edit it, then use the **Reload config** icon in Settings:
+
+```text
+~/.config/framekeyboard/layouts/
+~/.config/framekeyboard/languages/
+~/.config/framekeyboard/themes/
+```
+
+Use a new profile `id` to add a choice, or the same `id` to override a bundled one. [Configuration reference](docs/configuration.md) covers fields, favorites, validation, and saved placement.
+
+## How it works
+
+The keyboard is a native C++20 app. Cairo/Pango draw the keys, Vulkan/OpenVR display the panel, and libei sends native controls to the Frame's local Gamescope compositor. A feature-checked Gamescope text-input protocol sends characters resolved from the selected layout, including Japanese, Chinese and Korean composition. It does not run a browser or need an internet connection.
+
+Copy and Paste send shortcuts; the app does not read clipboard contents. It does not log typed text or save composition history. Placement and profile preferences are stored locally.
+
+The visible panel polls input and follows the dashboard at a target 60 Hz. While hidden, polling drops to 4 Hz. It redraws the texture only when its display changes or a key animation runs. With replacement enabled, closing the panel keeps the app resident for the next native request; Settings → Quit Full Keyboard exits it. With replacement off, closing the keyboard exits the process. Battery impact has not been benchmarked.
+
+## Development and feedback
+
+- [Build from source](docs/building.md)
+- [Architecture and code map](docs/architecture.md)
+- [Runtime and troubleshooting](docs/runtime.md)
+- [Manual acceptance checks](docs/acceptance.md)
+- [Roadmap](docs/plan.md) and [changelog](CHANGELOG.md)
+- [Contributing and bug reports](CONTRIBUTING.md)
+- [Release checklist](docs/releasing.md)
+- [Third-party code and licenses](third_party/README.md)
+
+Please report the app version, SteamOS version, affected app, whether it is a desktop or floating window, and steps to reproduce. Remove private text from screenshots and logs.
+
+## License
+
+Project code is available under the [MIT License](LICENSE). Vendored components keep their [upstream licenses](third_party/README.md).
+
+## AI-generated project disclaimer
+
+This project is fully AI-generated. The author is unfamiliar with C++, so please have patience and understanding. Bugs and rough edges are expected; careful reviews, clear bug reports, and contributions are welcome. Third-party components retain their original authorship and licenses.

keyboard/bridge/native_bridge.py

@@ -0,0 +1,634 @@
+#!/usr/bin/env python3
+"""App-scoped CDP transport. Only localhost Steam is contacted; no listeners or files are created."""
+import base64
+from collections import deque
+import hashlib
+import ipaddress
+import json
+import math
+import os
+from pathlib import Path
+import select
+import signal
+import socket
+import struct
+import subprocess
+import sys
+import time
+from urllib.parse import urlsplit
+from urllib.request import HTTPRedirectHandler, ProxyHandler, build_opener
+
+MAX_PACKET = 32768
+GLOBAL = "__fullKeyboardNativeV1"
+INSTALL_TIMEOUT = 2.0
+ACTIVE_POLL = 0.035
+IDLE_POLL = 0.075
+
+
+def poll_interval(state):
+    # Missing/new/inconsistent metadata must retain the ordinary fast cadence.
+    busy = ('open', 'eligible', 'active', 'external', 'externalCandidate', 'externalRestore')
+    return IDLE_POLL if state.get('idle') is True and not any(state.get(k) for k in busy) else ACTIVE_POLL
+
+
+def install_expression(source, token, external_supported, expires):
+    reference = f"globalThis.{GLOBAL}"
+    # Python and CEF run on the same device. A command queued beyond its local
+    # deadline must not install hooks later, even if CDP cleanup cannot get through.
+    return (f"(() => {{ if (Date.now() >= {expires}) return null; "
+            f"if ({reference}) {{ if (!{reference}.bridge.expired()) return false; "
+            f"{reference}.bridge.dispose(); delete {reference}; }} "
+            f"const bridge = ({source})(globalThis, undefined, {str(external_supported).lower()}); "
+            f"if (Date.now() >= {expires}) {{ bridge.dispose(); return null; }} "
+            f"{reference} = {{token: {json.dumps(token)}, bridge}}; return true; }})()")
+
+
+def cleanup_expression(token, restore=True):
+    reference = f"globalThis.{GLOBAL}"
+    argument = "" if restore else "false"
+    return (f"(() => {{ const b = {reference}; if (b?.token === {json.dumps(token)}) {{"
+            f"b.bridge.dispose({argument}); delete {reference}; }} }})()")
+
+
+class BridgeUnavailable(ValueError):
+    """Only fixed, non-sensitive codes may cross the private status channel."""
+    def __init__(self, reason):
+        self.reason = reason if reason in {"debugger-blocked", "steam-incompatible",
+                                          "replacement-busy", "reply-timeout"} else "helper-unavailable"
+        super().__init__(self.reason)
+
+
+def require_loopback_debugger():
+    """Require the debugger endpoint we use to be bound to loopback."""
+    for table in ("tcp", "tcp6"):
+        try:
+            lines = Path("/proc/net", table).read_text().splitlines()[1:]
+        except FileNotFoundError:
+            if table == "tcp6":
+                continue  # IPv6 can be disabled by the device.
+            raise BridgeUnavailable("debugger-blocked") from None
+        except OSError:
+            raise BridgeUnavailable("debugger-blocked") from None
+        for line in lines:
+            fields = line.split()
+            if fields[3] != "0A":
+                continue
+            host, port = fields[1].split(":")
+            if int(port, 16) != 8080:
+                continue
+            raw = bytes.fromhex(host)
+            if sys.byteorder == "little":
+                raw = b"".join(raw[i:i + 4][::-1] for i in range(0, len(raw), 4))
+            address = ipaddress.ip_address(raw)
+            mapped = getattr(address, "ipv4_mapped", None)
+            if not (mapped or address).is_loopback:
+                raise BridgeUnavailable("debugger-blocked")
+
+
+class NoDebuggerRedirects(HTTPRedirectHandler):
+    def redirect_request(self, *_args, **_kwargs):
+        raise BridgeUnavailable("debugger-blocked")
+
+
+class NativeGuard:
+    """Separate native lease. EOF restores an armed exact-overlay request without CDP."""
+    def __init__(self):
+        self.socket = self.process = None
+        self.armed = False
+        executable = Path(__file__).resolve().parents[3] / "bin/framekeyboard-native-guard"
+        if not executable.is_file():
+            executable = Path(__file__).resolve().parent.parent / "framekeyboard-native-guard"
+            if not executable.is_file():
+                raise FileNotFoundError()
+        parent, child = socket.socketpair(socket.AF_UNIX, socket.SOCK_SEQPACKET)
+        self.socket = parent
+        parent.settimeout(.5)
+        try:
+            self.process = subprocess.Popen([str(executable), "--fd", str(child.fileno())],
+                                            pass_fds=(child.fileno(),), stdin=subprocess.DEVNULL,
+                                            stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+            if json.loads(parent.recv(2048)).get("ok") is not True:
+                raise ValueError("native-guard-unavailable")
+        except BaseException:
+            self.close()
+            raise
+        finally:
+            child.close()
+
+    def command(self, op, **fields):
+        data = json.dumps(dict(op=op, **fields), separators=(",", ":")).encode()
+        if self.socket.send(data) != len(data):
+            raise ValueError("native-guard-backpressure")
+        return json.loads(self.socket.recv(2048)).get("ok") is True
+
+    def prepare(self, candidate, frame):
+        ok = self.command("prepare", key=candidate["key"], flags=candidate["flags"],
+                          requireActive=candidate.get("requireActive") is True,
+                          progress=frame["progress"], sent=frame["sent"], ready=frame["ready"])
+        if ok:
+            self.armed = True
+        return ok
+
+    def renew(self, frame):
+        self.command("renew", progress=frame["progress"], sent=frame["sent"], ready=frame["ready"])
+
+    def cancel(self):
+        if not self.command("cancel"):
+            raise ValueError("native-guard-cancel-failed")
+        self.armed = False
+
+    def restore(self):
+        if not self.command("restore"):
+            raise ValueError("native-guard-restore-failed")
+        self.armed = False
+
+    def close(self):
+        if self.socket:
+            self.socket.close()
+            self.socket = None
+        # An active native fallback belongs to this client. Leave the guard alive
+        # until Steam reports its closure, or VR shutdown would close it immediately.
+        if self.process:
+            try:
+                self.process.wait(timeout=.1)
+            except subprocess.TimeoutExpired:
+                pass
+
+
+class CDP:
+    def __init__(self):
+        self.socket = None
+        self.pending = b""
+        self.sequence = 0
+        try:
+            require_loopback_debugger()
+            self.network_checked = time.monotonic()
+            opener = build_opener(ProxyHandler({}), NoDebuggerRedirects())
+            with opener.open("http://127.0.0.1:8080/json/list", timeout=0.4) as response:
+                data = response.read(1024 * 1024 + 1)
+            if len(data) > 1024 * 1024:
+                raise ValueError("target-list-too-large")
+            targets = json.loads(data)
+            target = next((t for t in targets if t.get("title") == "SharedJSContext"), None)
+            if target is None:
+                raise BridgeUnavailable("steam-incompatible")
+            url = urlsplit(target["webSocketDebuggerUrl"])
+            if url.scheme != "ws" or url.hostname not in ("localhost", "127.0.0.1") or url.port != 8080:
+                raise BridgeUnavailable("debugger-blocked")
+            self.socket = socket.create_connection(("127.0.0.1", 8080), timeout=0.3)
+            key = base64.b64encode(os.urandom(16)).decode()
+            path = url.path + ("?" + url.query if url.query else "")
+            self.socket.sendall((f"GET {path} HTTP/1.1\r\nHost: 127.0.0.1:8080\r\n"
+                                 "Upgrade: websocket\r\nConnection: Upgrade\r\n"
+                                 f"Sec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\n\r\n").encode())
+            while b"\r\n\r\n" not in self.pending:
+                chunk = self.socket.recv(4096)
+                if not chunk:
+                    raise EOFError()
+                self.pending += chunk
+                if len(self.pending) > 16384:
+                    raise ValueError("invalid-upgrade")
+            header, self.pending = self.pending.split(b"\r\n\r\n", 1)
+            accept = base64.b64encode(hashlib.sha1((key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode()).digest())
+            if not header.startswith(b"HTTP/1.1 101 ") or accept.lower() not in header.lower():
+                raise ValueError("upgrade-refused")
+        except BaseException:
+            self.close()
+            raise
+
+    def close(self):
+        if self.socket:
+            self.socket.close()
+            self.socket = None
+
+    def peek(self, count, deadline):
+        while len(self.pending) < count:
+            remaining = deadline - time.monotonic()
+            if remaining <= 0:
+                raise TimeoutError()
+            self.socket.settimeout(remaining)
+            data = self.socket.recv(min(65536, count - len(self.pending)))
+            if not data:
+                raise EOFError()
+            self.pending += data
+        return self.pending[:count]
+
+    def receive_frame(self, deadline):
+        # Do not consume a partial frame on timeout. The next call must be able
+        # to drain a late reply, including one interrupted inside its payload.
+        first, second = self.peek(2, deadline)
+        length, header = second & 127, 2
+        if length == 126:
+            length, header = struct.unpack("!H", self.peek(4, deadline)[2:4])[0], 4
+        elif length == 127:
+            length, header = struct.unpack("!Q", self.peek(10, deadline)[2:10])[0], 10
+        if length > 1024 * 1024 or second & 128 or not first & 0x80 or first & 0x70:
+            raise ValueError("invalid-debug-frame")
+        data = self.peek(header + length, deadline)[header:]
+        self.pending = self.pending[header + length:]
+        return first & 15, data
+
+    def frame(self, opcode, data, deadline):
+        size = len(data)
+        header = bytes([0x80 | opcode])
+        if size < 126:
+            header += bytes([0x80 | size])
+        elif size < 65536:
+            header += bytes([0xfe]) + struct.pack("!H", size)
+        else:
+            header += bytes([0xff]) + struct.pack("!Q", size)
+        mask = os.urandom(4)
+        self.socket.settimeout(max(0.001, deadline - time.monotonic()))
+        try:
+            self.socket.sendall(header + mask + bytes(b ^ mask[i % 4] for i, b in enumerate(data)))
+        except OSError as error:
+            # A partial write cannot safely be retried on the same stream.
+            self.close()
+            raise ConnectionError("debugger-write-failed") from error
+
+    def evaluate(self, expression, timeout=0.4):
+        if time.monotonic() - self.network_checked >= 1:
+            require_loopback_debugger()
+            self.network_checked = time.monotonic()
+        self.sequence += 1
+        command = {"id": self.sequence, "method": "Runtime.evaluate", "params": {
+            "expression": expression, "returnByValue": True, "silent": True}}
+        deadline = time.monotonic() + timeout
+        self.frame(1, json.dumps(command, ensure_ascii=True).encode(), deadline)
+        while time.monotonic() < deadline:
+            opcode, payload = self.receive_frame(deadline)
+            if opcode == 9:
+                self.frame(10, payload, deadline)
+                continue
+            if opcode == 8:
+                raise EOFError()
+            if opcode != 1:
+                continue
+            message = json.loads(payload)
+            if message.get("id") != self.sequence:
+                continue
+            if "exceptionDetails" in message.get("result", {}) or "error" in message:
+                raise BridgeUnavailable("steam-incompatible")
+            return message.get("result", {}).get("result", {}).get("value")
+        raise TimeoutError()
+
+
+def send(parent, value):
+    data = json.dumps(value, ensure_ascii=True, separators=(",", ":")).encode()
+    if len(data) > MAX_PACKET or parent.send(data) != len(data):
+        raise ValueError("parent-backpressure")
+
+
+def report_unavailable(parent, reason):
+    # Closing a seqpacket peer with unread frames discards its final reply with
+    # ECONNRESET. Refuse new writes and drain the bounded queue before reporting.
+    parent.shutdown(socket.SHUT_RD)
+    for _ in range(1024):
+        if not parent.recv(MAX_PACKET + 1, socket.MSG_DONTWAIT):
+            break
+    send(parent, {"op": "unavailable", "reason": reason})
+
+
+class WatchdogProgress:
+    """Private pipe from the overlay watchdog, independent of the VR/main loop."""
+    def __init__(self, fd):
+        self.fd = fd
+        self.pending = b""
+        self.sent = None
+        self.slow_until = 0
+        self.runtime_until = 0
+
+    def poll(self):
+        if self.fd is None:
+            return
+        while True:
+            try:
+                data = os.read(self.fd, 4096)
+            except BlockingIOError:
+                break
+            if not data:
+                break
+            self.pending += data
+            while len(self.pending) >= 24:
+                sent, slow_until, runtime_until = struct.unpack("=ddd", self.pending[:24])
+                self.pending = self.pending[24:]
+                if math.isfinite(sent) and 0 < sent <= time.monotonic() and \
+                        (self.sent is None or sent > self.sent):
+                    self.sent = sent
+                    self.slow_until = slow_until if math.isfinite(slow_until) and \
+                        0 < slow_until <= sent + 30 else 0
+                    self.runtime_until = runtime_until if math.isfinite(runtime_until) and \
+                        0 < runtime_until <= sent + 30 else 0
+
+    def close(self):
+        if self.fd is not None:
+            os.close(self.fd)
+            self.fd = None
+
+
+class ParentRecovery:
+    def __init__(self):
+        self.terminated_at = None
+        self.boot_gap = None
+        self.resume_until = 0
+        self.resume_at = 0
+        self.resume_frame = None
+
+    @staticmethod
+    def parent_stopped(owner):
+        try:
+            # comm can contain spaces and parentheses. The state follows its
+            # final closing parenthesis; only a signal stop uses uppercase T.
+            fields = Path(f"/proc/{owner}/stat").read_text().rsplit(") ", 1)[1].split()
+            return fields[0] == "T"
+        except (OSError, IndexError):
+            return False  # An unreadable state leaves the normal deadlines in place.
+
+    def observe_clock(self, now, boot_now, frame_sent):
+        # BOOTTIME includes system suspend; MONOTONIC and the frame clock do not.
+        gap = boot_now - now
+        if self.boot_gap is not None and gap - self.boot_gap > .5 and self.resume_frame != frame_sent:
+            self.resume_at, self.resume_until, self.resume_frame = now, now + 30, frame_sent
+        self.boot_gap = gap
+        if frame_sent > self.resume_at:
+            self.resume_until = 0
+
+    def check(self, owner, watch, frame_sent, watchdog_sent, slow_until=0, runtime_until=0):
+        now = time.monotonic()
+        if not owner or not watch or os.getppid() != owner:
+            self.terminated_at = None
+            return False
+        stopped = self.parent_stopped(owner)
+        if os.getppid() != owner:
+            self.terminated_at = None
+            return False
+        if stopped:
+            # Neither the main loop nor its watchdog can run after SIGSTOP, and
+            # SIGTERM would remain pending. Remove the frozen panel's owner now.
+            os.kill(owner, signal.SIGKILL)
+            return True
+        if now - frame_sent < 3:
+            self.terminated_at = None
+            return False
+        if self.terminated_at is None and (now < self.resume_until or
+                                          now < min(runtime_until, frame_sent + 30)):
+            return False
+        # A live overlay watchdog does not prove main-loop progress. Only an
+        # explicit slow operation may delay recovery, never beyond 30 seconds
+        # from the last completed frame. It cannot cancel a termination begun.
+        if self.terminated_at is None and watchdog_sent is not None and \
+                0 <= now - watchdog_sent < 3 and now < min(slow_until, frame_sent + 30):
+            return False
+        if self.terminated_at is None:
+            os.kill(owner, signal.SIGTERM)
+            self.terminated_at = now
+        elif now - self.terminated_at >= 1:
+            os.kill(owner, signal.SIGKILL)
+            return True
+        return False
+
+
+def run(parent, client_factory=CDP, parent_pid=None, guard_factory=NativeGuard, health_fd=None):
+    client = None
+    guard = None
+    install_attempted = False
+    watchdog = WatchdogProgress(health_fd)
+    recovery_watchdog = ParentRecovery()
+    token = os.urandom(16).hex()
+    reference = f"globalThis.{GLOBAL}"
+    failure_reason = "debugger-unavailable"
+    restore_native = True
+    try:
+        client = client_factory()
+        try:
+            guard = guard_factory()
+        except (OSError, ValueError, EOFError):
+            pass  # Steam targets remain supported when the native adapter is unavailable.
+        failure_reason = "helper-unavailable"
+        source = Path(__file__).with_name("steam_bridge.js").read_text()
+        failure_reason = "steam-incompatible"
+        # Never displace another live instance. A dead owner's lease must expire first.
+        install_attempted = True
+        result = client.evaluate(install_expression(source, token, guard is not None,
+                                                   int((time.time() + INSTALL_TIMEOUT) * 1000)),
+                                 timeout=INSTALL_TIMEOUT)
+        if result is None:
+            raise TimeoutError()  # The expression itself rejected a late installation.
+        if result is not True:
+            raise BridgeUnavailable("replacement-busy")
+        failure_reason = "connection-lost"
+        frame = {"progress": 0, "ready": False, "ack": 0, "fallback": False,
+                 "sent": time.monotonic()}
+        frame_sent = time.monotonic()
+        last_parent = time.monotonic()
+        watch_main_loop = False
+        fallback_request = None
+        external_ack = 0
+        next_poll = 0.0
+        pending = deque()
+        held_keys = {}
+        pending_repeats = set()
+        stopping = False
+        consecutive_timeouts = 0
+
+        def evaluate(expression):
+            nonlocal consecutive_timeouts
+            try:
+                result = client.evaluate(expression)
+            except TimeoutError:
+                consecutive_timeouts += 1
+                if consecutive_timeouts >= 3:
+                    raise BridgeUnavailable("reply-timeout") from None
+                raise
+            consecutive_timeouts = 0
+            return result
+
+        while True:
+            if not Path(__file__).is_file() or not Path(__file__).with_name("steam_bridge.js").is_file():
+                break  # Removing an installed app also withdraws a still-running bridge.
+            readable, _, _ = select.select([parent], [], [],
+                                           0 if pending else max(0, next_poll - time.monotonic()))
+            # Drain releases before dispatching more work. CDP can be slower than
+            # repeat generation; initial clicks stay ordered, repeats do not pile up.
+            received = 0
+            while readable:
+                received += 1
+                if received > 256:
+                    raise ValueError("parent-backlog")
+                packet = parent.recv(MAX_PACKET + 1)
+                if not packet:
+                    stopping = True
+                    break
+                if len(packet) > MAX_PACKET:
+                    raise ValueError("oversize-packet")
+                message = json.loads(packet)
+                operation = message.get("op")
+                if operation == "stop":
+                    # Only an explicit prepared-update shutdown may suppress
+                    # recovery. EOF, signals and malformed packets still restore.
+                    restore_native = message.get("restore") is not False
+                    stopping = True
+                    break
+                if operation == "frame":
+                    if type(message.get("progress")) is not int or type(message.get("ready")) is not bool:
+                        raise ValueError("invalid-frame")
+                    frame = {k: message[k] for k in ("progress", "ready", "ack", "sent")}
+                    # Restoration acknowledgements must survive subsequent frame packets.
+                    frame["externalRestored"] = external_ack
+                    if guard:
+                        guard.renew(frame)
+                    if message.get("fallback") is True:
+                        fallback_request = message.get("fallbackRequest", 0)
+                        next_poll = 0
+                    last_parent = time.monotonic()
+                    frame_sent = float(message["sent"])
+                    watch_main_loop = message.get("watch") is True
+                elif operation == "key-up":
+                    if type(message.get("id")) is not int or type(message.get("request")) is not int:
+                        raise ValueError("invalid-release")
+                    key_id = message["id"]
+                    held = held_keys.get(key_id)
+                    if held and held[0] == message["request"]:
+                        del held_keys[key_id]
+                        cancelled_repeat = key_id in pending_repeats
+                        pending_repeats.discard(key_id)
+                        pending = deque(m for m in pending if not (
+                            m["op"] == "key" and m["value"] == 2 and m["id"] == key_id))
+                        if cancelled_repeat:
+                            # No deletion after release, but the parent still
+                            # needs completion for the repeat it submitted.
+                            send(parent, {"op": "key-result", "id": key_id, "ok": True})
+                elif operation == "dismiss":
+                    if type(message.get("request")) is not int:
+                        raise ValueError("invalid-dismissal")
+                    pending.append(message)
+                elif operation in ("text", "key"):
+                    if type(message.get("request")) is not int or not isinstance(message.get("text"), str):
+                        raise ValueError("invalid-input")
+                    if len(message["text"].encode()) > 8192:
+                        raise ValueError("oversize-input")
+                    if operation == "key":
+                        if type(message.get("id")) is not int or message["id"] <= 0 or \
+                                type(message.get("value")) is not int or message["value"] not in (1, 2):
+                            raise ValueError("invalid-key")
+                        key_id = message["id"]
+                        identity = (message["request"], message["text"])
+                        if message["value"] == 1:
+                            if key_id in held_keys or len(held_keys) >= 128:
+                                raise ValueError("invalid-key-hold")
+                            held_keys[key_id] = identity
+                            pending.append(message)
+                        elif held_keys.get(key_id) == identity and key_id not in pending_repeats:
+                            pending_repeats.add(key_id)
+                            pending.append(message)
+                    else:
+                        pending.append(message)
+                else:
+                    raise ValueError("unknown-operation")
+                if len(pending) > 128:
+                    raise ValueError("input-backlog")
+                readable, _, _ = select.select([parent], [], [], 0)
+            if stopping:
+                break
+            if pending:
+                message = pending.popleft()
+                operation = message["op"]
+                try:
+                    if operation == "dismiss":
+                        ok = evaluate(f"{reference}?.bridge.dismiss({message['request']}) === true")
+                        next_poll = 0
+                    else:
+                        if operation == "key" and message["value"] == 2:
+                            pending_repeats.discard(message["id"])
+                        # The JS side checks the target and lease again at delivery.
+                        ok = evaluate(f"{reference}?.bridge.send({message['request']},"
+                                      f"{json.dumps(message['text'], ensure_ascii=True)},"
+                                      f"{str(operation == 'key').lower()}) === true")
+                except TimeoutError:
+                    ok = False  # Unknown delivery fails this request closed.
+                if operation != "dismiss":
+                    send(parent, {"op": f"{operation}-result", "id": message["id"], "ok": ok is True})
+                if ok is not True:
+                    fallback_request = message["request"]
+            watchdog.poll()
+            recovery_watchdog.observe_clock(time.monotonic(), time.clock_gettime(time.CLOCK_BOOTTIME), frame_sent)
+            if not select.select([parent], [], [], 0)[0] and recovery_watchdog.check(
+                    parent_pid, watch_main_loop, frame_sent, watchdog.sent, watchdog.slow_until,
+                    watchdog.runtime_until):
+                break
+            if time.monotonic() - last_parent > 0.7 or not 0 <= time.monotonic() - frame_sent < 0.7:
+                frame["ready"] = False
+            if time.monotonic() >= next_poll:
+                poll_frame = {**frame, "fallback": fallback_request is not None,
+                              "fallbackRequest": fallback_request}
+                try:
+                    result = evaluate(f"{reference}?.bridge.step({json.dumps(poll_frame)})")
+                    if not isinstance(result, dict) or result.get("version") != 1 or result.get("unavailable"):
+                        raise BridgeUnavailable("steam-incompatible")
+                    if guard:
+                        recovery = result.get("externalRestore")
+                        if recovery and recovery["request"] != external_ack:
+                            guard.restore()
+                            external_ack = frame["externalRestored"] = recovery["request"]
+                        candidate = result.get("externalCandidate")
+                        if candidate:
+                            # Dismissal and resummon can both occur between polls. A
+                            # candidate has no logical owner, so any old lease must
+                            # be cancelled before the next exact target is armed.
+                            if guard.armed:
+                                guard.cancel()
+                            if guard.prepare(candidate, frame):
+                                result = evaluate(f"{reference}?.bridge.prepareExternal("
+                                                         f"{candidate['request']},{json.dumps(candidate['key'])})")
+                                if not isinstance(result, dict):
+                                    raise BridgeUnavailable("steam-incompatible")
+                        recovery = result.get("externalRestore")
+                        if recovery and recovery["request"] != external_ack:
+                            guard.restore()
+                            external_ack = frame["externalRestored"] = recovery["request"]
+                        elif not result.get("external") and guard.armed:
+                            guard.cancel()
+                except TimeoutError:
+                    # step is idempotent. Keep the last state/fallback request;
+                    # the JS lease and C++ freshness timeout remain independent.
+                    next_poll = time.monotonic() + ACTIVE_POLL
+                    continue
+                fallback_request = None
+                send(parent, {"op": "state", **result})
+                next_poll = time.monotonic() + poll_interval(result)
+    except (Exception, SystemExit) as error:
+        # Errors are fixed status codes. Never echo CDP expressions, input packets or field values.
+        try:
+            reason = error.reason if isinstance(error, BridgeUnavailable) else failure_reason
+            if isinstance(error, TimeoutError) and failure_reason == "steam-incompatible":
+                reason = "install-timeout"
+            report_unavailable(parent, reason)
+        except Exception:
+            pass
+    finally:
+        if client:
+            if install_attempted:
+                try:
+                    # A lost reply does not prove the install failed. The token
+                    # prevents this attempt's cleanup from removing a successor.
+                    client.evaluate(cleanup_expression(token, restore_native))
+                    if guard and guard.armed:
+                        if restore_native:
+                            guard.restore()
+                        else:
+                            guard.cancel()
+                except Exception:
+                    pass  # Steam-side watchdog restores presentation even if CDP disappeared.
+            client.close()
+        if guard:
+            guard.close()
+        watchdog.close()
+        parent.close()
+
+
+if __name__ == "__main__":
+    # SIGTERM follows the same cleanup path as EOF from the owning application.
+    signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
+    if len(sys.argv) != 5 or sys.argv[1] != "--fd" or sys.argv[3] != "--health-fd":
+        sys.exit(2)
+    run(socket.socket(fileno=int(sys.argv[2])), parent_pid=os.getppid(), health_fd=int(sys.argv[4]))

keyboard/bridge/steam_bridge.js

@@ -0,0 +1,672 @@
+// Evaluated only in Steam's SharedJSContext. Never read field or clipboard text.
+// Native presentation is closed; input stays bound to the owning Steam window.
+(function installNativeBridge(root, testStore, externalSupported) {
+    "use strict";
+    const VERSION = 1, LEASE_MS = 900, REQUEST_MS = 400, IDLE_INSPECT_MS = 100;
+    const now = () => root.performance.now();
+    let store = testStore;
+    if (!store) {
+        let require;
+        const chunks = root.webpackChunksteamui;
+        if (!chunks || typeof chunks.push !== "function") throw Error("steam-ui-unavailable");
+        const id = Symbol("framekeyboard");
+        chunks.push([[id], {}, r => { require = r; }]);
+        const index = chunks.findIndex(c => c[0]?.[0] === id);
+        if (index >= 0) chunks.splice(index, 1);
+        for (const key of Object.keys(require?.m || {})) {
+            if (!String(require.m[key]).includes("get VRKeyboardStatus()")) continue;
+            for (const value of Object.values(require(key))) {
+                if (value && typeof value === "object" && "VRKeyboardStatus" in value &&
+                    value.m_VRKeyboardState) { store = value; break; }
+            }
+            if (store) break;
+        }
+    }
+    const popup = root.g_PopupManager?.GetPopupForVROverlayKey?.("valve.steam.gamepadui.keyboard");
+    const win = popup?.m_popup || popup?.window;
+    const windowApi = win?.SteamClient?.Window;
+    const windows = root.SteamUIStore?.WindowStore?.SteamUIWindows;
+    const managers = Array.isArray(windows) ? windows.map(w => w.VirtualKeyboardManager).filter(Boolean) : [];
+    if (!store?.m_VRKeyboardState || !win?.document || !windowApi || !managers.length ||
+        typeof windowApi.ShowWindow !== "function" || typeof windowApi.BringToFront !== "function")
+        throw Error("steam-hooks-unavailable");
+    function nativeStatus() {
+        // VRKeyboardStatus also contains initial text. Read only these raw metadata fields.
+        if (testStore) {
+            const s = store.VRKeyboardStatus;
+            return {open: s.bIsOpen, key: s.sOverlayKey, dispatch: s.bDispatchEventsToSteamVR,
+                flags: s.bMinimal ? 13 : 0};
+        }
+        const s = store.m_VRKeyboardState;
+        if (typeof s?.m_bIsOpen !== "boolean") throw Error("steam-status-changed");
+        return {open: s.m_bIsOpen, key: s.m_sOpenForOverlayKey,
+            dispatch: s.m_bDispatchEventsToSteamVR, flags: s.m_eVRKeyboardFlags};
+    }
+    const hooks = [], owners = [];
+    const keyboardApi = root.SteamClient?.OpenVR?.Keyboard;
+    const externalHide = keyboardApi?.Hide;
+    const localTarget = s => externalSupported && s.dispatch === false && s.flags === 13 &&
+        /^gamescope\.gamescope-[0-9]+(?:\.window\.[0-9]+)?$/.test(s.key) && typeof externalHide === "function";
+    let preparedExternal = null, externalRestore = null, handoff = null;
+    const localFrames = new Map();
+    function selectedFrame() {
+        const f = root.vrFrameStore?.activeFrame;
+        if (!f || !Number.isSafeInteger(f.frameID)) return null;
+        return {id: f.frameID, visible: f.isVisible === true,
+            keys: Array.from(f.steamUIVROverlayKeysInPages || []).filter(key => ownSteam(key))};
+    }
+    function hook(object, key, wrap) {
+        const descriptor = Object.getOwnPropertyDescriptor(object, key), original = object[key];
+        if (typeof original !== "function" || descriptor && !descriptor.configurable && !descriptor.writable)
+            throw Error("steam-hook-readonly");
+        const replacement = wrap(original);
+        Object.defineProperty(object, key, {configurable: true, writable: true, value: replacement});
+        hooks.push(() => {
+            if (object[key] !== replacement) return;
+            if (descriptor) Object.defineProperty(object, key, descriptor);
+            else delete object[key];
+        });
+        return original;
+    }
+    const ownSteam = key => typeof key === "string" && (key === "valve.steam.gamepadui.main" ||
+        key === "valve.steam.gamepadui.barpopup" || key.startsWith("valve.steam.gamepadui.appoverlay."));
+    let native = nativeStatus(), previousOpen = native.open, previousKey = native.key;
+    let ignoreOpen = native.open, rawDeadline = 0, restoringUntil = 0;
+    let restoringOwner = null, restoreTimer;
+    let logical = null, closingNative = false, deadline = 0, request = 1, acknowledged = 0;
+    let expectedClose = null;
+    let eventSequence = 0;
+    const events = [];
+    function record(code) {
+        events.push({id: ++eventSequence, request, code});
+        if (events.length > 32) events.shift();
+    }
+    let lease = 0, progress = -1, disposed = false, inspecting = false, interceptedShow = false;
+    let lastClock = now(), lastStep = now(), lastInspection = now(), timer, subscription;
+    const oldWindowShow = windowApi.ShowWindow;
+    const style = win.document.createElement("style");
+    style.setAttribute("data-framekeyboard-native", "1");
+    style.textContent = ":root, body { opacity: 0 !important; pointer-events: none !important; }";
+    let styled = false;
+    function setHidden(hidden) {
+        if (hidden && !styled) { win.document.documentElement.appendChild(style); styled = true; }
+        if (!hidden && styled) { style.remove(); styled = false; }
+    }
+    const fresh = () => !disposed && now() < lease;
+    function editingDocument(manager) {
+        const d = manager.m_ownerWindow?.document;
+        return d && typeof d.execCommand === "function" &&
+            typeof d.getSelection?.()?.modify === "function" &&
+            typeof d.defaultView?.KeyboardEvent === "function" &&
+            d.queryCommandSupported?.("insertText") && d.queryCommandSupported?.("delete") ? d : null;
+    }
+    const editableIn = (e, d) => e?.isConnected && e.ownerDocument === d && !e.disabled && !e.readOnly &&
+        (e.tagName === "TEXTAREA" || e.tagName === "INPUT" &&
+            ["text", "search", "password", "tel", "url"].includes(e.type));
+    function routeFor(owner) {
+        const m = owner.manager;
+        if (owner.document?.visibilityState === "hidden") return null;
+        if (typeof m.m_ActiveElementProps?.onTextEntered === "function") return {kind: "callback"};
+        const d = owner.document;
+        if (!d || m.m_ownerWindow?.document !== d) return null;
+        const e = d.activeElement;
+        const ref = m.m_currentVirtualKeyboardRef || m.m_lastActiveVirtualKeyboardRef;
+        if (editableIn(e, d) && typeof ref?.BIsElementValidForInput === "function" &&
+            (owner.focusRefs.get(ref) === e || ref.BIsElementValidForInput() === true) && (m.m_currentVirtualKeyboardRef ||
+                typeof ref.SetAsCurrentVirtualKeyboardTarget === "function"))
+            return {kind: "dom", element: e, ref};
+        // A button can summon the panel without a field. Such a session can
+        // navigate this window but never sends text to global compositor focus.
+        return !m.m_currentVirtualKeyboardRef && !m.m_ActiveElementProps ? {kind: "dashboard"} : null;
+    }
+    const routeReady = owner => !!routeFor(owner);
+    function sendDom(session, text, key) {
+        const d = session.owner.document, element = session.route.element;
+        if (!key) {
+            // With no field, ordinary typing has no recipient. Never queue it
+            // for a later field or forward it to a background application.
+            return !element || d.execCommand("insertText", false, text) === true;
+        }
+        if (text === "VKClose") { releaseLogical(false); return true; }
+        const codes = {Backspace: 8, Enter: 13, Tab: 9, ArrowLeft: 37, ArrowUp: 38,
+            ArrowRight: 39, ArrowDown: 40};
+        if (!(text in codes)) return false;
+        if (element && text === "Enter" && session.props?.onEnterKeyPress != null) {
+            const enter = session.props.onEnterKeyPress;
+            const action = typeof enter === "function" ? enter.call(session.props) : enter;
+            if (!action) return true;
+            if (action === "VKClose") { releaseLogical(false); return true; }
+            if (action !== "Enter") return false;
+        }
+        const target = element || d.activeElement || d.body;
+        const event = type => new d.defaultView.KeyboardEvent(type,
+            {key: text, code: text, keyCode: codes[text], which: codes[text],
+                bubbles: true, cancelable: true});
+        const accepted = target.dispatchEvent(event("keydown"));
+        try {
+            if (!accepted || element && d.activeElement !== element) return true;
+            if (text === "Tab") {
+                const items = [...d.querySelectorAll("input,textarea,button,a[href],[tabindex]")]
+                    .filter(e => !e.disabled && e.tabIndex >= 0 && e.getClientRects().length)
+                    .sort((a, b) => (a.tabIndex || Infinity) - (b.tabIndex || Infinity));
+                const next = items[(items.indexOf(d.activeElement) + 1) % items.length];
+                next?.focus();
+            } else if (element) {
+                if (text === "Backspace") d.execCommand("delete", false, null);
+                else if (text.startsWith("Arrow")) {
+                    const backward = text === "ArrowLeft" || text === "ArrowUp";
+                    d.getSelection().modify("move", backward ? "backward" : "forward",
+                        text === "ArrowLeft" || text === "ArrowRight" ? "character" : "line");
+                } else if (text === "Enter") {
+                    if (element.tagName === "TEXTAREA") d.execCommand("insertText", false, "\n");
+                    else element.form?.requestSubmit?.();
+                }
+            }
+            return true;
+        } finally { target.dispatchEvent(event("keyup")); }
+    }
+    function potential(s) {
+        return localTarget(s) || ownSteam(s.key) && (s.dispatch === false || (s.flags & 1) !== 0) &&
+            owners.some(o => o.key === s.key && routeReady(o));
+    }
+    function candidate(s) {
+        if (localTarget(s)) return preparedExternal?.key === s.key ? preparedExternal : null;
+        const choices = owners.filter(o => {
+            const route = routeFor(o);
+            return o.key === s.key && route && (route.kind !== "callback" ||
+                typeof o.manager.m_fnVROnTextEnteredOverride === "function");
+        });
+        return potential(s) && choices.length === 1 ? choices[0] : null;
+    }
+    const targetOf = manager => manager.m_currentVirtualKeyboardRef || manager.m_ActiveElementProps || null;
+    function fieldRequest(s) {
+        const owner = candidate(s), intent = owner?.fieldIntent;
+        return owner && !owner.external && intent && now() < intent.until &&
+            intent.target === targetOf(owner.manager) ? owner : null;
+    }
+    function buttonRequest(s) {
+        if (localTarget(s)) return true;
+        const owner = candidate(s);
+        // Dashboard buttons bypass the field manager. A retained field reference
+        // can be stale after returning from an app, so it is not proof of focus.
+        return owner?.fieldHook && !fieldRequest(s);
+    }
+    function blocking() {
+        return fresh() && !ignoreOpen && !restoringUntil &&
+            (logical || !native.open || potential(native) && now() < rawDeadline);
+    }
+    function restoreWindow() {
+        setHidden(false);
+        if (interceptedShow && native.open) oldWindowShow.call(windowApi);
+        interceptedShow = false;
+    }
+    function unsubscribe() {
+        if (typeof subscription === "function") subscription();
+        else subscription?.unregister?.();
+        subscription = null;
+    }
+    function finishRestore(success) {
+        const owner = restoringOwner;
+        restoringOwner = null;
+        restoringUntil = 0;
+        root.clearTimeout(restoreTimer);
+        if (disposed) unsubscribe();
+        if (owner) record(success ? "restore-complete" : "restore-failed");
+        if (owner && !success) owner.setOpen.call(owner.manager, false);
+    }
+    function checkRestore(final = false) {
+        if (!restoringOwner) return;
+        const status = nativeStatus();
+        if (status.open && status.key === restoringOwner.key) finishRestore(true);
+        else if (status.open || final || now() >= restoringUntil) finishRestore(false);
+    }
+    function releaseLogical(restore, closeNative = true,
+        reason = restore ? "release-fallback" : "release-dismissed") {
+        const session = logical;
+        // Expiry runs before ordinary competing-owner inspection, and disposal
+        // may precede a status callback. Always check the current provider here.
+        try { native = nativeStatus(); } catch (_) {
+            // Unknown presentation cannot authorize Show/Hide. Still release
+            // our retained manager even if Steam's status interface changed.
+            native = {open: false};
+            restore = closeNative = false;
+        }
+        if (restore && session && !session.owner.external &&
+            session.owner.document?.visibilityState === "hidden") {
+            restore = closeNative = false;
+            reason = "owner-hidden";
+        } else if (restore && session?.route?.kind === "dom") {
+            const current = routeFor(session.owner);
+            if (current?.kind !== "dom" || current.element !== session.route.element ||
+                targetOf(session.owner.manager) !== session.target) {
+                // Blur can precede Steam clearing its ref, even on expiry or
+                // helper shutdown. Do not revive that field or hide a newer one.
+                restore = closeNative = false;
+                reason = "target-cleared";
+            }
+        }
+        logical = null;
+        handoff = null;
+        expectedClose = null;
+        closingNative = false;
+        acknowledged = 0;
+        ++request;
+        ignoreOpen = true;
+        if (!restore) interceptedShow = false;
+        restoreWindow();
+        if (!session) return;
+        if (restore && native.open && native.key !== session.owner.key) {
+            record("competing-owner");
+            if (!session.owner.external) session.owner.setOpen.call(session.owner.manager, false);
+            preparedExternal = null;
+            return;
+        }
+        record(reason);
+        if (session.owner.external) {
+            if (restore) {
+                externalRestore = {key: session.owner.key, request};
+                restoringUntil = now() + REQUEST_MS;
+            } else if (closeNative) session.owner.hide.call(keyboardApi);
+            preparedExternal = null;
+        } else if (restore &&
+            (!session.target || targetOf(session.owner.manager))) {
+            // Preserve the field while the exact owning window's original provider opens again.
+            restoringUntil = now() + REQUEST_MS;
+            restoringOwner = session.owner;
+            // Disposal removes the hooks and regular timer. This bounded check
+            // still clears a failed restore without touching a newer provider.
+            restoreTimer = root.setTimeout(() => {
+                try { checkRestore(true); } catch (_) { finishRestore(false); }
+            }, REQUEST_MS);
+            session.owner.show.call(session.owner.api);
+            oldWindowShow.call(windowApi);
+        } else {
+            // A missing field cannot justify closing another provider's newer
+            // native request while fallback withdraws the old session.
+            if (closeNative && (!restore || !native.open || native.key === session.owner.key))
+                session.owner.hide.call(session.owner.api);
+            // Hide can fail or complete asynchronously. An already-open native
+            // keyboard for this owner must retain its ordinary Steam state.
+            let status;
+            try { status = nativeStatus(); } catch (_) { /* Ownership must still be released. */ }
+            if (!status?.open || status.key !== session.owner.key)
+                session.owner.setOpen.call(session.owner.manager, false);
+        }
+    }
+    function begin(owner) {
+        const route = owner.external ? null : routeFor(owner);
+        // Steam clears its current ref on X. Restore only its own still-valid
+        // last field, in response to this new explicit native summon.
+        if (route?.kind === "dom" && !owner.manager.m_currentVirtualKeyboardRef)
+            route.ref.SetAsCurrentVirtualKeyboardTarget();
+        const frame = selectedFrame();
+        logical = {owner, target: owner.external ? owner.key : targetOf(owner.manager),
+            props: owner.manager?.m_ActiveElementProps, route, frame: frame?.id};
+        handoff = null;
+        if (owner.external && frame?.visible && frame.keys.length === 0) {
+            localFrames.delete(frame.id);
+            localFrames.set(frame.id, owner.key);
+            if (localFrames.size > 16) localFrames.delete(localFrames.keys().next().value);
+        }
+        ++request;
+        record("begin");
+        acknowledged = 0;
+        deadline = now() + REQUEST_MS;
+        closingNative = !!native.open;
+        expectedClose = !native.open || owner.external ? null : {owner, request};
+        owner.fieldIntent = null;
+        if (!owner.external) owner.setOpen.call(owner.manager, true);
+        // Window.HideWindow leaves dashboard scene/hit geometry alive and can
+        // trigger focus-driven reopening. Close presentation through its provider.
+        if (native.open) owner.hide.call(owner.external ? keyboardApi : owner.api);
+    }
+    function inspect() {
+        if (inspecting || disposed) return;
+        inspecting = true;
+        try {
+            const time = now();
+            lastInspection = time;
+            if (time - lastClock > LEASE_MS || time < lastClock) lease = 0;
+            lastClock = time;
+            if (win.closed || root.g_PopupManager.GetPopupForVROverlayKey("valve.steam.gamepadui.keyboard") !== popup)
+                throw Error("steam-popup-changed");
+            native = nativeStatus();
+            checkRestore();
+            if (!restoringOwner && restoringUntil && (native.open || time >= restoringUntil)) restoringUntil = 0;
+            if (!native.open && !restoringUntil) ignoreOpen = false;
+            const newOpen = native.open && (!previousOpen || native.key !== previousKey);
+            if (newOpen) rawDeadline = time + REQUEST_MS;
+            previousOpen = native.open;
+            previousKey = native.key;
+            const frame = selectedFrame();
+            if (logical && frame && logical.frame != null && logical.frame !== frame.id && !native.open) {
+                // Frame selection can leave a hidden Steam input active in its
+                // document. It is not the typing owner after an app switch.
+                const key = frame.keys.length === 1 ? frame.keys[0] : localFrames.get(frame.id);
+                releaseLogical(false, false, "owner-changed");
+                if (fresh() && frame.visible && key) {
+                    handoff = {key, frame: frame.id, until: time + REQUEST_MS};
+                    ignoreOpen = false;
+                }
+            }
+            if (handoff) {
+                if (!fresh() || time >= handoff.until || native.open)
+                    handoff = null;
+                else if (handoff.previous != null) {
+                    // Visibility and selected-tab metadata arrive separately.
+                    // Stop old input immediately, then wait within the same
+                    // bounded handoff for the new visible selected frame.
+                    if (frame?.visible && frame.id !== handoff.previous) {
+                        const key = frame.keys.length === 1 ? frame.keys[0] : localFrames.get(frame.id);
+                        handoff = key ? {...handoff, key, frame: frame.id, previous: null} : null;
+                    }
+                } else if (!frame?.visible || frame.id !== handoff.frame) handoff = null;
+            }
+            if (handoff && ownSteam(handoff.key)) {
+                const choices = owners.filter(o => o.key === handoff.key && routeReady(o));
+                if (choices.length === 1) begin(choices[0]);
+            }
+            if (logical && !logical.owner.external && logical.owner.document?.visibilityState === "hidden") {
+                const previous = logical.frame;
+                releaseLogical(false, false, "owner-hidden");
+                if (fresh() && previous != null)
+                    handoff = {previous, until: time + REQUEST_MS};
+            }
+            if (logical) {
+                if (!fresh() || time >= deadline && acknowledged !== request) {
+                    releaseLogical(true, true, fresh() ? "request-timeout" : "lease-expired");
+                } else if (newOpen && !closingNative && acknowledged === request && buttonRequest(native)) {
+                    // The resident panel is already open, even if its previous app
+                    // was closed. Consume this button toggle across supported owners.
+                    releaseLogical(false);
+                } else if (newOpen && !closingNative && fieldRequest(native)) {
+                    const owner = fieldRequest(native), previous = logical.owner;
+                    logical = null;
+                    expectedClose = null;
+                    preparedExternal = null;
+                    if (!previous.external && previous !== owner)
+                        previous.setOpen.call(previous.manager, false);
+                    begin(owner);
+                    record("target-changed");
+                } else if (native.open && (native.key !== logical.owner.key || !potential(native))) {
+                    // A different provider can open before our close acknowledgement.
+                    // Its native session must survive; never reopen the old owner over it.
+                    const previous = logical.owner;
+                    logical = null;
+                    expectedClose = null;
+                    if (!previous.external && native.key !== previous.key)
+                        previous.setOpen.call(previous.manager, false);
+                    closingNative = false;
+                    acknowledged = 0;
+                    ++request;
+                    record("competing-owner");
+                    ignoreOpen = true;
+                    preparedExternal = null;
+                } else if (closingNative) {
+                    if (!native.open && !expectedClose) closingNative = false;
+                } else if (newOpen) {
+                    const owner = candidate(native);
+                    const changedTarget = owner !== logical.owner ||
+                        owner && !owner.external && targetOf(owner.manager) !== logical.target;
+                    if (changedTarget) {
+                        const previous = logical.owner;
+                        logical = null;
+                        expectedClose = null;
+                        if (!previous.external && previous !== owner)
+                            previous.setOpen.call(previous.manager, false);
+                        if (owner) begin(owner);
+                        else { acknowledged = 0; ++request; ignoreOpen = true; }
+                    } else {
+                        // Native presentation is closed, so the next same-target
+                        // native button Show toggles our resident panel off.
+                        releaseLogical(false);
+                    }
+                } else if (logical.owner.external) {
+                    // Gamescope input stays on the existing local libei/Unicode backend.
+                } else {
+                    const m = logical.owner.manager;
+                    // React can replace props after every character while the field
+                    // reference stays stable. Refresh callbacks without retargeting.
+                    logical.props = m.m_ActiveElementProps;
+                    const target = targetOf(m), route = routeFor(logical.owner);
+                    if (!target && target !== logical.target || logical.route?.kind === "dom" &&
+                        (!route || route.kind === "dashboard")) {
+                        // Field loss proves only that Steam no longer owns
+                        // this target. It cannot authorize libei input into
+                        // whichever local app currently has compositor focus.
+                        releaseLogical(false, true, "target-cleared");
+                    } else if (!route || route.kind === "callback" &&
+                        typeof m.m_fnVROnTextEnteredOverride !== "function") {
+                        releaseLogical(true);
+                    } else if (target !== logical.target || route.kind !== logical.route?.kind ||
+                        route.element !== logical.route?.element) {
+                        logical.target = target;
+                        logical.route = route;
+                        ++request;
+                        acknowledged = 0;
+                        deadline = time + REQUEST_MS;
+                        record("target-changed");
+                    }
+                }
+            } else if (native.open && !ignoreOpen && !restoringUntil) {
+                const owner = candidate(native);
+                if (fresh() && owner && time < rawDeadline) begin(owner);
+                else if (!fresh() || !potential(native) || time >= rawDeadline) ignoreOpen = true;
+            }
+            // A show can arrive before its native status. Replay it when that
+            // request falls back, even if we never acquired a logical session.
+            if (blocking()) setHidden(true);
+            else restoreWindow();
+        } finally { inspecting = false; }
+    }
+    function idle() {
+        return !logical && !native.open && !handoff && !preparedExternal && !externalRestore &&
+            !restoringOwner && !restoringUntil && !expectedClose && !closingNative && !interceptedShow;
+    }
+    function snapshot() {
+        const eligible = !!logical && !closingNative && !ignoreOpen;
+        return {version: VERSION, request, open: !!logical || !!native.open || !!handoff, eligible, idle: idle(),
+            active: eligible && acknowledged === request && fresh(),
+            local: !!logical?.owner.external,
+            external: logical?.owner.external ? {key: logical.owner.key, request} : null,
+            externalCandidate: handoff && !native.open && externalSupported &&
+                /^gamescope\.gamescope-[0-9]+(?:\.window\.[0-9]+)?$/.test(handoff.key) ?
+                {key: handoff.key, flags: 13, request, requireActive: true} :
+                !logical && !ignoreOpen && native.open && localTarget(native) &&
+                fresh() && now() < rawDeadline ? {key: native.key, flags: native.flags, request} : null,
+            externalRestore, events: events.slice()};
+    }
+    function dispose(restore = true) {
+        if (disposed) return;
+        lease = 0;
+        try {
+            let closeNative = true;
+            if (restore === false) {
+                // Update cleanup can race a new owner's native request. Release
+                // our logical flag without hiding that unrelated presentation.
+                try {
+                    const current = nativeStatus();
+                    closeNative = !current.open || current.key === logical?.owner.key;
+                } catch (_) { closeNative = false; }
+            }
+            releaseLogical(restore !== false, closeNative);
+            restoreWindow();
+        } finally {
+            disposed = true;
+            root.clearInterval(timer);
+            // Observe a pending restore until its bounded check completes. A
+            // later summon for this owner must cancel cleanup before another
+            // bridge can take it over and close native again.
+            if (!restoringOwner) unsubscribe();
+            else if (!subscription) checkRestore(true);
+            for (const undo of hooks.reverse()) undo();
+        }
+    }
+    try {
+        const hookedApis = new Map();
+        for (const manager of managers) {
+            const api = manager.m_ownerWindow?.SteamClient?.OpenVR?.Keyboard;
+            const key = manager.m_Instance?.GetVROverlayKey?.();
+            if (!ownSteam(key) || typeof api?.Show !== "function" || typeof api?.Hide !== "function" ||
+                typeof manager.SetVRKeyboardOpenForThisInstance !== "function") continue;
+            const originals = hookedApis.get(api) || {show: api.Show, hide: api.Hide};
+            const owner = {manager, api, key, document: editingDocument(manager), focusRefs: new WeakMap(), ...originals};
+            const d = owner.document;
+            if (d && typeof d.addEventListener === "function" && typeof d.removeEventListener === "function" &&
+                typeof manager.SetVirtualKeyboardActiveRef === "function") {
+                // Steam's shared-context predicate can consult the wrong document.
+                // Associate a ref only when Steam selects it synchronously inside
+                // this owner's actual DOM focus/click event. Never infer it from polling.
+                const focused = event => { owner.focusEvent = event; };
+                d.addEventListener("focus", focused, true);
+                d.addEventListener("click", focused, true);
+                hooks.push(() => {
+                    d.removeEventListener("focus", focused, true);
+                    d.removeEventListener("click", focused, true);
+                    owner.focusEvent = null;
+                });
+                hook(manager, "SetVirtualKeyboardActiveRef", original => function(ref, ...args) {
+                    const event = owner.focusEvent, e = event?.target;
+                    if (event?.eventPhase && e === d.activeElement && editableIn(e, d) &&
+                        ref && typeof ref === "object") owner.focusRefs.set(ref, e);
+                    return original.call(this, ref, ...args);
+                });
+            }
+            if (typeof manager.SetVirtualKeyboardShownInternal === "function") {
+                owner.fieldHook = true;
+                hook(manager, "SetVirtualKeyboardShownInternal", original => function(open, ...args) {
+                    if (open && targetOf(this))
+                        owner.fieldIntent = {target: targetOf(this), until: now() + REQUEST_MS};
+                    else if (!open) owner.fieldIntent = null;
+                    return original.call(this, open, ...args);
+                });
+            }
+            owner.setOpen = hook(manager, "SetVRKeyboardOpenForThisInstance", original => function(open) {
+                if (!open && logical?.owner.manager === this) {
+                    // Consume exactly the close reaction caused by this request's
+                    // Hide. Keep acquisition pending if status arrives first.
+                    if (closingNative && expectedClose?.owner === owner &&
+                        expectedClose.request === request && !nativeStatus().open) {
+                        expectedClose = null;
+                        inspect();
+                        return;
+                    }
+                    // Steam has independently closed its field. Never restore it.
+                    releaseLogical(false, false, "steam-closed");
+                }
+                return original.call(this, open);
+            });
+            owner.keyDown = hook(manager, "HandleVirtualKeyDown", original => function(...args) {
+                if (logical?.owner.manager === this && fresh()) return;
+                return original.apply(this, args);
+            });
+            owners.push(owner);
+            if (!hookedApis.has(api)) {
+                hookedApis.set(api, originals);
+                hook(api, "Hide", original => function(...args) {
+                    if (logical?.owner.api === this) releaseLogical(false);
+                    return original.apply(this, args);
+                });
+            }
+        }
+        if (!owners.length) throw Error("steam-target-restoration-unavailable");
+        for (const name of ["ShowWindow", "BringToFront"]) hook(windowApi, name, original => function(...args) {
+            try { inspect(); if (blocking()) { interceptedShow = true; return; } }
+            catch (_) { dispose(); }
+            return original.apply(this, args);
+        });
+        // Ignore every callback argument, including initial field text.
+        const keyboard = root.SteamClient?.OpenVR?.Keyboard;
+        if (typeof keyboard?.RegisterForStatus === "function") subscription = keyboard.RegisterForStatus(() => {
+            try { if (disposed) checkRestore(); else inspect(); } catch (_) { dispose(); }
+        });
+        inspect();
+        timer = root.setInterval(() => {
+            try {
+                const time = now();
+                // Keep the independent 50 ms watchdog. Only an idle full scan
+                // can be coalesced with a recent helper/event inspection. Lease
+                // expiry must still remove suppression at the ordinary deadline.
+                if (!idle() || time < lastClock || (styled && !fresh()) ||
+                    time - lastInspection >= IDLE_INSPECT_MS) inspect();
+                if (!fresh() && now() - lastStep > 1500) dispose();
+            } catch (_) { dispose(); }
+        }, 50);
+    } catch (error) { dispose(); throw error; }
+    return {
+        version: VERSION,
+        expired() { return !fresh(); },
+        diagnostics() { return {...snapshot(), route: logical?.route?.kind || null,
+            nativeOpen: !!native.open, closingNative,
+            expectedClose: !!expectedClose, session: !!logical, restoring: !!restoringOwner}; },
+        step(frame) {
+            if (disposed) return {version: VERSION, unavailable: true};
+            lastStep = now();
+            try {
+                if (Number.isSafeInteger(frame.progress) && frame.progress > progress) {
+                    progress = frame.progress;
+                    lease = frame.ready ? now() + LEASE_MS : 0;
+                }
+                inspect();
+                if (externalRestore && frame.externalRestored === externalRestore.request) externalRestore = null;
+                if (frame.fallback && frame.fallbackRequest === request) releaseLogical(true);
+                if (snapshot().eligible && fresh() && frame.ack === request &&
+                    (acknowledged === request || now() < deadline)) {
+                    if (acknowledged !== request) record("acknowledged");
+                    acknowledged = request;
+                }
+                inspect();
+                return snapshot();
+            } catch (_) { dispose(); return {version: VERSION, unavailable: true}; }
+        },
+        send(generation, text, key = false) {
+            inspect();
+            if (!snapshot().active || generation !== request || typeof text !== "string" ||
+                text.length > 8192 || typeof key !== "boolean" || key &&
+                !["Backspace", "Enter", "Tab", "ArrowLeft", "ArrowRight", "ArrowUp", "ArrowDown", "VKClose"].includes(text))
+                return false;
+            const session = logical, manager = session.owner.manager;
+            if (session.owner.external) return false;
+            const previous = manager.m_fnVROnTextEnteredOverride;
+            if (session.route?.kind !== "callback") {
+                const ok = sendDom(session, text, key);
+                inspect();
+                return ok;
+            }
+            const targetText = session.props?.onTextEntered;
+            if (typeof targetText !== "function") return false;
+            // The renderer's override can call VRKeyboard.SendText, which requires
+            // an open native session. Keep Enter/Done handling in the manager, but
+            // dispatch this one call only to its retained field callback.
+            const override = value => targetText(value);
+            manager.m_fnVROnTextEnteredOverride = override;
+            try {
+                if (key) session.owner.keyDown.call(manager, text, false);
+                else targetText(text);
+            }
+            finally {
+                if (manager.m_fnVROnTextEnteredOverride === override)
+                    manager.m_fnVROnTextEnteredOverride = previous;
+            }
+            inspect();
+            return true;
+        },
+        dismiss(generation) {
+            inspect();
+            if (!snapshot().active || generation !== request) return false;
+            releaseLogical(false);
+            inspect();
+            return true;
+        },
+        prepareExternal(generation, key) {
+            inspect();
+            const pending = snapshot().externalCandidate;
+            if (!pending || pending.request !== generation || pending.key !== key) return snapshot();
+            preparedExternal = {external: true, key, api: keyboardApi, hide: externalHide};
+            if (handoff && !native.open) begin(preparedExternal);
+            inspect();
+            return snapshot();
+        },
+        dispose,
+    };
+})

keyboard/build.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Build Full Keyboard (TaiKeid's, vendored here: FRAMETOP.md) in the dev container on the
+# Frame: build/frametop/framekeyboard, which frametop-keyboard.service runs with --frametop.
+# It finds its layouts, themes and Steam bridge beside itself, in build/frametop. Like
+# Frametop's other OpenVR programs it loads SteamVR's own libopenvr_api first. Its tests:
+# FULL_KEYBOARD_TESTS=1 keyboard/build.sh (ctest in build/frametop).
+set -euo pipefail
+root=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+"$root/scripts/sync.sh" >/dev/null
+exec "$root/scripts/frame.sh" -C keyboard "set -e
+. ../scripts/openvr.sh
+tests=${FULL_KEYBOARD_TESTS:-0}
+cmake -S . -B build/frametop -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo \
+  -DBUILD_TESTING=\$([ \$tests = 1 ] && echo ON || echo OFF) \
+  -DOPENVR_LIBRARY=\$openvr_lib/libopenvr_api.so -DOPENVR_RUNTIME_PATH=\$openvr_rpath >/dev/null
+cmake --build build/frametop
+[ \$tests = 1 ] && (cd build/frametop && XDG_RUNTIME_DIR=\${XDG_RUNTIME_DIR:-/run/user/\$(id -u)} ctest --output-on-failure)
+echo 'built build/frametop/framekeyboard'"

keyboard/cmake/steam-frame-aarch64.cmake

@@ -0,0 +1,24 @@
+# Development headers and libraries copied from a matching Steam Frame system.
+# This directory is supplied by the builder and is never bundled in the repository.
+set(CMAKE_SYSTEM_NAME Linux)
+set(CMAKE_SYSTEM_PROCESSOR aarch64)
+set(CMAKE_SYSROOT "$ENV{FRAMEKEYBOARD_SYSROOT}")
+if(NOT IS_ABSOLUTE "${CMAKE_SYSROOT}" OR NOT EXISTS "${CMAKE_SYSROOT}/usr/include")
+    message(FATAL_ERROR "Set FRAMEKEYBOARD_SYSROOT to an absolute Frame sysroot path; see docs/building.md")
+endif()
+set(CMAKE_C_COMPILER clang)
+set(CMAKE_C_COMPILER_TARGET aarch64-unknown-linux-gnu)
+set(CMAKE_C_FLAGS_INIT "--gcc-toolchain=${CMAKE_SYSROOT}/usr")
+set(CMAKE_CXX_COMPILER clang++)
+set(CMAKE_CXX_COMPILER_TARGET aarch64-unknown-linux-gnu)
+set(CMAKE_CXX_FLAGS_INIT "--gcc-toolchain=${CMAKE_SYSROOT}/usr")
+set(CMAKE_EXE_LINKER_FLAGS_INIT "-fuse-ld=lld")
+set(CMAKE_MODULE_LINKER_FLAGS_INIT "-fuse-ld=lld")
+set(CMAKE_FIND_ROOT_PATH "${CMAKE_SYSROOT}")
+set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
+set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
+set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
+set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
+set(ENV{PKG_CONFIG_SYSROOT_DIR} "${CMAKE_SYSROOT}")
+set(ENV{PKG_CONFIG_LIBDIR} "${CMAKE_SYSROOT}/usr/lib/pkgconfig:${CMAKE_SYSROOT}/usr/share/pkgconfig")
+set(ENV{PKG_CONFIG_PATH} "")