Recorder tests: the eye dataset has three recipients since key 3 (dody)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.dockerignore

@@ -19,3 +19,6 @@ frametop-report-*.txt
 # the fine-tuned hand models: the image fetches its own pinned copy (hands/models.sh)
 hands/models/student/
 hands/models/.student-*
+# model versions downloaded from Input Settings (models/catalog.py): each install fetches its own
+hands/models/versions/
+gaze/tracker/models/

.gitignore

@@ -11,3 +11,5 @@ frametop-report-*.txt
 *.pgm
 .venv/
 .frame-job.d/
+# eye net versions picked in Input Settings (models/catalog.py downloads them; CC BY-NC 4.0)
+gaze/tracker/models/

README.md

@@ -35,9 +35,9 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
 
    It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/Frametop/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
 
-   The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.1 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
+   The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.4 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
 
-   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you six things along the way: whether to install voice dictation (yes by default; it downloads about 320 MB of speech models), hand tracking (experimental, yes by default), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. Hand tracking, the eye tracker, and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
+   The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks along the way whether to install each optional part: voice dictation (yes by default; it downloads about 320 MB of speech models), hand tracking (experimental, yes by default), in a release the [Hand Recorder](https://frametop.github.io/frametop/hand-recorder) (yes by default; it records your hands for Frametop's open hand dataset), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), in a release the Eye Recorder (yes by default, with our eye tracker; it records your eyes for Frametop's eye tracking dataset), and the Bluetooth fixes, then whether to restart SteamVR. The stable version asks only about gaze mode, the eye tracker, and the Bluetooth fixes. Hand tracking, the Hand Recorder, the eye tracker, and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
 
 After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings. SteamOS's own single-screen desktop is still there, as Native Desktop in the same list.
 

docs/design.md

@@ -24,6 +24,8 @@ ft-screens replaces it. It's a small wlroots compositor that hosts the nested KW
 
 Frames go to SteamVR the same way gamescope sends them: OpenVR's `IVRIPCResourceManagerClient` imports each DMA-BUF (`ImportDmabuf`), and the overlay shows it as a `TextureType_SharedTextureHandle` texture. There's no copy and no size limit. The Frame's SteamVR supports this interface, but the OpenVR header bundled with SteamVR's samples predates it, so the build fetches a pinned header from Valve's openvr repository.
 
+The small textures ft-screens draws itself (the controls, the hand glows, "grab that"'s halo, the carousel) go the same way, through `screens/overlaytex.cpp`: one linear DMA-BUF per distinct image, imported once and shared. Not `SetOverlayRaw`: the Frame's SteamVR never gives back the memory block each call takes, so a process's 201st call fails ("Refusing to create memory block because 201 blocks are already outstanding" in the journal) and the overlay keeps its old texture at its new size. That was the glow bug of 2026-10-15, after about 40 minutes of hand use. Other programs call it only a few times, or as a fallback.
+
 Before settling on this, we tried KWin's screencast virtual outputs. KWin 6.2.5 crashed (`std::out_of_range` in `textureForOutput`) streaming one while nested in gamescope, and its headless backend doesn't implement virtual outputs at all.
 
 A few wlroots details worth knowing: `wlr_shm_create` wants DRM format codes, not `wl_shm` ones, and KWin asks for server-side decorations before its first commit, when setting the mode would assert, so ft-screens answers on the first commit.
@@ -174,7 +176,7 @@ Volume keys must never reach gamescope. With the openvr backend, gamescope sends
 
 Frametop's keyboard opens by itself for a text field on the desktop. The apps run inside the nested KWin, so only KWin knows when a text field has focus, and the way it tells anyone is its input method protocol (`zwp_input_method_v1`): KWin starts one input method program and activates it whenever the focused app turns on text input. `input/ft-textinput` is that program, speaking the Wayland wire protocol directly so it needs nothing but Python on the host. It only reports focus. The gamescope session puts `QT_IM_MODULE=xim` and `GTK_IM_MODULE=xim` in the systemd user environment; with those, Qt and GTK apps use X input methods and never turn on Wayland text input, so the session script drops them.
 
-The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, as the old `@frametop_keys` could), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU and uploaded with `SetOverlayRaw` when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
+The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, as the old `@frametop_keys` could), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU into one of three shared buffers when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
 
 The Frame controllers can be mapped like mouse buttons, but they aren't input devices on the host: they reach SteamVR over the headset's own radio, and no evdev or hidraw node exists for them. So only a SteamVR client can read them. Overlay apps normally get controller input only while they have input focus, which a background helper never has. SteamVR's experimental global action set priority (`steamvr/globalActionSetPriority`, "Enable global input from overlays") lets an overlay's action set receive input anyway, and takes the inputs it binds from the scene app. Binding every button would take them all from games, so the helper's action manifest puts each button in an action set of its own, and it activates only the sets of mapped buttons. The mapping itself stays in the relay, which does the action, so mice and controllers share one list of actions.
 

docs/reference.md

@@ -316,7 +316,7 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke
 
 It costs a lot of the headset's CPU. `install.sh` offers it (yes by default, `--no-hands` skips it, and it needs `sudo` for ft-camd's file capabilities), and a release brings `ft-camd` and `ft-hands` built in its image. Installed, it runs while the Frametop desktop does (`HANDS_MODE=frametop`).
 
-Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen (a round hole per hand, `HANDS_MATTE=circle`; `mesh` and `capsules` are the older shapes). ft-handview (`frametop-handview.service`) draws your hands as a 3D hand model posed by the tracking: over games and the void with Room View off, and over the hole with it on (`HANDS_VIEW_ROOMVIEW=1`, the default). `HANDS_VIEW_MODEL` picks the model: `asset` (the default), `babylon`, `mrtk3`, `godot`, `mrtk2`, `robot` (a robot hand built in code from the tracked skeleton: light shells, dark joints, cyan accents), or `capsules` for the older tubes; `HANDS_VIEW_STYLE=ghost|solid` the look. The models, in `screens/assets/hands` with their licences: the WebXR Input Profiles' generic hand (`asset`; MIT, Copyright (c) 2019 Amazon), Babylon.js's hand meshes by Patrick Ryan (`babylon`; CC BY 4.0, https://github.com/BabylonJS/Assets), MRTK3's OpenXR hand (`mrtk3`; BSD-3-Clause, Copyright (c) 2025, Mixed Reality Toolkit Contributors), Godot XR Tools' hands by DigitalN8m4r3 (`godot`; CC0 1.0) and MRTK2's rigged hand (`mrtk2`; MIT, Copyright (c) Microsoft Corporation). The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode your eyes aim and your hands click, Vision Pro style: a pinch clicks where you were looking as it began, and the hand ray stays off (`POINTER_GAZE_HAND_RAY=1` brings it back beside the gaze). Pinch, hold and move the hand a few centimetres to nudge the pointer slowly onto the target, then let go to click there; the correction teaches the gaze tracker, as the mouse's do. Pinch and hold still for half a second to press and drag with the hand. A middle pinch right-clicks the same way. Hold a pinch with one hand and pinch with the other to hold both buttons, left and right, each until its pinch opens. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1` and gaze mode off, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (a pinch clicks, a middle pinch right-clicks or scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". When tracking goes wrong, `ft-handsctl flag [note]` (or a voice command through frame-voice, "flag that") saves the last 12 seconds of hand crops and what the tracker made of them for the offline tools: ft-hands keeps them in RAM (`HANDS_FLAG_BUFFER=1`, about 50 MB), the drawn hands pulse cyan when a flag is taken (`HANDS_FLAG_FLASH`, `HANDS_FLAG_TONE`), and flags go to `HANDS_FLAG_DIR`. See [hands/README.md](../hands/README.md), "Flags".
+Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen (a round hole per hand, `HANDS_MATTE=circle`; `mesh` and `capsules` are the older shapes). ft-handview (`frametop-handview.service`) draws your hands as a 3D hand model posed by the tracking: over games and the void with Room View off, and over the hole with it on (`HANDS_VIEW_ROOMVIEW=1`, the default). `HANDS_VIEW_MODEL` picks the model: `asset` (the default), `babylon`, `mrtk3`, `godot`, `mrtk2`, `robot` (a robot hand built in code from the tracked skeleton: light shells, dark joints, cyan accents), or `capsules` for the older tubes; `HANDS_VIEW_STYLE=ghost|solid` the look. The models, in `screens/assets/hands` with their licences: the WebXR Input Profiles' generic hand (`asset`; MIT, Copyright (c) 2019 Amazon), Babylon.js's hand meshes by Patrick Ryan (`babylon`; CC BY 4.0, https://github.com/BabylonJS/Assets), MRTK3's OpenXR hand (`mrtk3`; BSD-3-Clause, Copyright (c) 2025, Mixed Reality Toolkit Contributors), Godot XR Tools' hands by DigitalN8m4r3 (`godot`; CC0 1.0) and MRTK2's rigged hand (`mrtk2`; MIT, Copyright (c) Microsoft Corporation). The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode your eyes aim and your hands click, Vision Pro style: a pinch clicks where you were looking as it began, and the hand ray stays off (`POINTER_GAZE_HAND_RAY=1` brings it back beside the gaze). Pinch, hold and move the hand a few centimetres to nudge the pointer slowly onto the target, then let go to click there; the correction teaches the gaze tracker, as the mouse's do. Pinch and hold still for half a second to press and drag with the hand. A middle pinch right-clicks the same way. Hold a pinch with one hand and pinch with the other to hold both buttons, left and right, each until its pinch opens. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1` and gaze mode off, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (it comes up as the thumb nears the index or middle tip; a pinch clicks, a middle pinch right-clicks or scrolls, a three-finger pinch scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". When tracking goes wrong, `ft-handsctl flag [note]` (or a voice command through frame-voice, "flag that") saves the last 12 seconds of hand crops and what the tracker made of them for the offline tools: ft-hands keeps them in RAM (`HANDS_FLAG_BUFFER=1`, about 50 MB), the drawn hands pulse cyan when a flag is taken (`HANDS_FLAG_FLASH`, `HANDS_FLAG_TONE`), and flags go to `HANDS_FLAG_DIR`. See [hands/README.md](../hands/README.md), "Flags".
 
 - `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. The Hand Recorder's installer (`hands/rec/install.sh`) sets them the same way. `hands/run.sh uncaps` takes them back while neither the services nor the Hand Recorder is installed. `hands/run.sh uninstall` and `hands/rec/install.sh uninstall` run it after removing their own part, and `uninstall.sh` always takes them back.
 - `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.

gaze/README.md

@@ -60,7 +60,7 @@ The tracker stops when the headset is off your head. SteamVR also calibrates gaz
 `gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
 
 - `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye, or 15 without `gaze/eye-rate.py on`) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). It also adds it to SteamOS's update keep list (`/etc/atomic-update.conf.d/frametop-eyegrab.conf`): an update deletes `/etc` files that list doesn't name, and without the grabber gaze mode falls back to SteamVR's tracker and its calibration. `scripts/doctor.sh` reports a grabber an update deleted.
-- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
+- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`. Instead of the pupil finder it can run an eye net (`eyes_net.py`): `EYES_MODEL` is `classic` (the default), `latest`, or a version from `gaze/tracker/models.json`, picked with its notes and benchmarks on Input Settings' Gaze page, which downloads it into `gaze/tracker/models/NAME` (`models/README.md`). Each net keeps its own calibration in `eyes/net-NAME/`.
 - `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
 
 Ground rules, for anyone changing it:

gaze/ft-gazed

@@ -65,7 +65,10 @@ relearn the offset.
 Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while
 GAZE_TRACKER=own and the gaze is in use (below), or the gaze probe asks for it ("eyes SECONDS",
 a lease the probe renews). It reads the eye-camera frames the root service frametop-eyegrab
-copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs.
+copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs. Which eye net it runs
+(EYES_MODEL: models/catalog.py) is read at its start, so ft-eyes restarts when that changes; starting
+it also runs `models/ft-models auto eyes` in the background (the catalog from GitHub, and the net
+EYES_MODEL needs if it isn't downloaded), and ft-eyes moves to a net that brings.
 
 Idle: ft-gaze and our own tracker run only while the gaze is in use: gaze mode on with someone
 wearing the headset (the pointer helper says both: "gaze ? headset" -> "ok on|off worn|away"), a
@@ -135,6 +138,9 @@ from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correcti
                      EyeWeights, Fixation, LiveCorrection, SteamEyeLog, TrackedEye)
 from gazecheck import Checks  # noqa: E402
 
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "models"))
+import catalog  # noqa: E402  (which eye net EYES_MODEL means: models/catalog.py)
+
 REPO = Path(__file__).resolve().parents[1]
 HELPER = REPO / "gaze" / "build" / "ft-gaze"
 IN_BOX = REPO / "scripts" / "in-box"  # runs a program in the container it was built in
@@ -160,6 +166,7 @@ LOOK = 0.3        # seconds of samples before that moment make the look (the pro
 RETRY = 3.0       # seconds before starting ft-gaze again
 EYES_RETRY = 10.0  # seconds before starting ft-eyes again after it stopped on its own
 EYES_LEASE_MAX = 120.0
+MODELS_CLI = REPO / "models" / "ft-models"  # "auto eyes": the eye catalog, and the net EYES_MODEL needs
 SETTLE = 0.3      # seconds after an eye is found again before the fallback learns from it
 IDLE_AFTER = 30.0  # seconds after the gaze was last in use before ft-gaze and our tracker stop
 WAKE_SETTLE = 15.0  # seconds after waking that the tracker may take to start sending
@@ -257,6 +264,8 @@ class Service:
         self.eyes_proc = None  # ft-eyes, while it runs
         self.eyes_until = 0.0  # the probe's lease (monotonic time)
         self.eyes_restart_at = 0.0
+        self.eyes_model = self.eyes_model_now()  # what EYES_MODEL means now: ft-eyes restarts when it changes
+        self.models_proc = None  # ft-models auto eyes, while it runs
         self.awake = False      # ft-gaze (and our tracker) run: the gaze is in use (see the top)
         self.idle_at = 0.0      # idle from then, unless it's in use again first
         self.woke_at = 0.0
@@ -293,8 +302,39 @@ class Service:
 
     # --- Settings, calibration and lessons ---
 
+    @staticmethod
+    def eyes_model_now():
+        """(EYES_MODEL, the folder it resolves to): ft-eyes reads it at start."""
+        try:
+            r = catalog.resolve("eyes")
+            return r["setting"], r["folder"]
+        except Exception as e:  # noqa: BLE001 (a broken catalog: ft-eyes falls back to the pupil finder)
+            return f"error {e}", None
+
+    def check_eyes_model(self):
+        now = self.eyes_model_now()
+        if now != self.eyes_model:
+            log(f"eye model {now[0]}" + (f" ({now[1]})" if now[1] else "") + (": restarting ft-eyes" if self.eyes_proc else ""))
+            self.eyes_model = now
+            if self.eyes_proc:
+                self.stop_eyes()
+                self.eyes_restart_at = 0.0
+
+    def models_auto(self):
+        """The eye catalog from GitHub (at most every 15 minutes) and the net EYES_MODEL needs, if it's
+        missing, in the background: ft-eyes starts with what's here, and moves to the new one after."""
+        if self.models_proc and self.models_proc.poll() is None:
+            return
+        try:
+            self.models_proc = subprocess.Popen([sys.executable, str(MODELS_CLI), "auto", "eyes"], stdin=subprocess.DEVNULL,
+                                                stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
+                                                start_new_session=True)
+        except OSError as e:
+            log(f"couldn't check for eye models: {e}")
+
     def load_settings(self):
         self.conf_mtime = mtime(CONF)
+        self.check_eyes_model()
         tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
         if (tracker, bias) != (self.tracker, self.bias):
             auto = ""
@@ -558,6 +598,8 @@ class Service:
             log(f"ft-eyes isn't built: run {REPO}/gaze/tracker/build.sh")
             self.eyes_restart_at = time.monotonic() + 30
             return
+        self.models_auto()
+        self.eyes_model = self.eyes_model_now()
         env = dict(os.environ)
         env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
         self.eyes_proc = subprocess.Popen([str(IN_BOX), str(EYES_PYTHON), str(EYES_PROG), "-v", "--watch-stdin"],
@@ -920,6 +962,9 @@ class Service:
         if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
                                               and own_installed() != (self.tracker == "own")):
             self.load_settings()
+        if self.models_proc and self.models_proc.poll() is not None:
+            self.models_proc = None
+            self.check_eyes_model()   # a net it fetched: ft-eyes moves to it
         self.update_awake()
         want = self.eyes_wanted()
         if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:

gaze/rec/CONSENT.md

@@ -1,6 +1,6 @@
-# Recording your eyes for the Frametop eye dataset (DRAFT, not reviewed yet)
+# Recording your eyes for the Frametop eye dataset
 
-Version: 2026-10-14-draft
+Version: 2026-10-16
 
 Frametop's eye tracking needs a better model than it has: one that keeps working when you look far to the side, blink, squint, wear glasses or makeup, or when the headset moves on your face. To train it, we're collecting recordings of many people's eyes into a dataset that only the maintainers can read. This page explains what the eye recorder records, where it goes, who can see it, and what you agree to if you take part. Please read all of it.
 
@@ -55,7 +55,7 @@ The recorder doesn't record sound, the headset's outward cameras, your name, you
 ## Where it goes, and who can read it
 
 - The dataset is hosted on Hugging Face (<https://huggingface.co/datasets/DeeJanuz/frametop-eyes>), whose servers may be outside your country, for example in the USA. It's gated: people must accept its terms to download it, and what they get has the eye images and health answers encrypted.
-- Only the maintainers hold the keys that decrypt them. Today that's DeeJanuz, with one key on their own computer and a backup copy stored offline. The keys are never uploaded. If a key ever leaked, uploads made to it could be read: the maintainers would say so on the dataset's page and stop using it.
+- Only the maintainers hold the keys that decrypt them. Today that's DeeJanuz, with a key on each of two of their own computers (one of them trains the models) and a backup copy stored offline. The keys are never uploaded. If a key ever leaked, uploads made to it could be read: the maintainers would say so on the dataset's page and stop using it.
 - Decrypted images are kept only on the maintainers' own computers, for training and checking models, and aren't shared with anyone else or uploaded to other services.
 - Recordings stay in the dataset until they're withdrawn or the dataset is taken down.
 

gaze/rec/DESIGN.md

@@ -1,6 +1,6 @@
 # Eye recorder: design (2026-10-14)
 
-The eye recorder guides a person through recording their eyes with the headset's two eye cameras, at known gaze targets and through the cases where eye tracking breaks. It also fits the headset: it shows whether each eye sits where the cameras see it best, helps set the IPD, and catches the headset shifting on the face, with instructions to put it back. Recordings stay on the headset until the person uploads them. Only the maintainers can read what's uploaded (below, "Privacy").
+The eye recorder guides a person through recording their eyes with the headset's two eye cameras, at known gaze targets and through the cases where eye tracking breaks. It also fits the headset: it shows whether each eye sits where the cameras see it best, shows the IPD setting, and catches the headset shifting on the face, with instructions to put it back. Recordings stay on the headset until the person uploads them. Only the maintainers can read what's uploaded (below, "Privacy").
 
 It's the hand recorder's sibling (`hands/rec/DESIGN.md`): the same session runner pattern, headset button, review, export and Hugging Face upload, with eye cameras instead of hand cameras. Like the hand recorder it ships in Frametop and, later, as a standalone app for people without Frametop (Frametop/frametop-eye-recorder, Frametop as a submodule).
 
@@ -69,7 +69,7 @@ What it can tell, and the instruction it gives:
 | A glint missing, or extra bright spots | Glasses reflections, or the lens fogged or smudged | "Tilt your glasses" / "Wipe the lenses" |
 | An eye found in under half the frames, looking straight ahead | Lashes on the lens, a droopy lid, or the eye outside the camera's view | The per-eye card says which |
 
-The IPD row depends on the first headset test. If the cameras move with the lenses, a pupil's sideways offset in its image directly says how far that eye is from its lens axis, and the coach turns the IPD wheel's direction into "wider" or "narrower" until both offsets are about zero, at any IPD. If the cameras are fixed, the offset depends on the IPD setting too, and the coach needs the reference at each IPD step: the fitment step's IPD sweep (below, "Bounty board") builds that.
+The IPD row depends on the first headset test. If the cameras move with the lenses, a pupil's sideways offset in its image directly says how far that eye is from its lens axis, and the coach turns the IPD wheel's direction into "wider" or "narrower" until both offsets are about zero, at any IPD. If the cameras are fixed, the offset depends on the IPD setting too, and the coach needs the reference at each IPD step, built from rounds recorded at different settings (there's no IPD sweep: the setting can't be changed with the headset on).
 
 **Put-on and shift check.** After a good fit, the coach keeps that fit as the person's baseline (`~/.local/state/frametop/gaze/fit-baseline.json`: per eye, the pupil looking straight ahead and the glint pair, with the IPD setting). Then:
 - When the headset goes back on (frames resume after ft-eyes' 3 s gap), it shows the fit card for a few seconds while the person looks ahead. If the pupils sit more than a threshold (start with 1.5 mm equivalent, tuned from data) from the baseline, it says which way to move the headset, live, until it's back. If the IPD setting changed, it says so first.
@@ -79,10 +79,12 @@ In the recorder the fit coach runs as each round's fitment step, and the shift w
 ## Bounty board
 
 People pick what to record from a board of bounties. Each bounty is one round of about 5 minutes, in step mode: the headset's right button starts each step (press = next or pause, two presses = redo, hold = stop), as in the hand recorder. Every round starts the same way:
-1. **Fitment** (not recorded as a take, but its readings go into fit.json): the fit coach until the headset sits well (below, "Fit coach"). Skippable after a warning, and the round records that it was skipped. On a contributor's first round, and whenever the IPD setting has changed since their last, it ends with an IPD sweep: turn the wheel slowly from narrow to wide and back while looking at a centre dot, with the IPD logged as it changes (the panel looks 30 times a second, and on SteamVR's IPD event). Its own take, `NN-ipd`, before the nine dots.
+1. **Fitment** (not recorded as a take, but its readings go into fit.json): the fit coach until the headset sits well (below, "Fit coach"). Skippable after a warning, and the round records that it was skipped. (The coach is off for now, 2026-10-16: then it only asks for a comfortable fit.) It shows the IPD setting, read from SteamVR when the round starts. The setting can't be changed with the headset on (2026-10-16), so there's no IPD step: a wrong setting means stopping, setting it, and starting again. Each take's ipd.jsonl still logs it.
 2. **Nine dots** (recorded): a 3x3 grid to 20 degrees on the mid background. It's the round's own reference, so each round can be labelled alone, even after a re-seat.
 3. **The bounty's steps.**
 
+How to build a new one, and the rules a round script must follow (beeps for anything done with the eyes shut, cue words as labels, timing, reach, comfort): `rounds/README.md`.
+
 Every round uploads on its own. The board marks the bounties you've done, by your contributor id in `profile.json`, with how many times. You can do one again.
 
 The launch bounties:
@@ -90,7 +92,7 @@ The launch bounties:
 | Bounty | What | Edge case it covers |
 |---|---|---|
 | View edges | Dots at the panel's edges and corners: 30 degrees to the sides, 23 up and down, about 35 in the corners (the rec panel is 64 degrees across; every round's dots are checked against it) | Eyes turned far: the lid, the glints lost, the pupil foreshortened |
-| Blinks and lids | Blink on cue, close both for 2 s, wink each eye, squint, eyes wide, half-closed, looking down with lids lowered | Closed and partly closed eyes; one eye missing |
+| Blinks and lids | Blink on a beep, close both eyes on a beep and open them on the next (sounds.py: the eyes are shut, so words can't cue them), wink each eye, squint, eyes wide, half-closed, looking down with lids lowered | Closed and partly closed eyes; one eye missing |
 | Headset slips | Nudge the headset up, down, left, right, tilt it; then take it off and put it back on; dots after each | Headset shifting on the face: the put-on check's data |
 | Glasses | A 5x5 grid and the edges with glasses or inserts on (offered when the checklist says so) | Lens reflections, the glasses' frame in view |
 | Jumps and pursuit | A dot jumping between random spots (0.5 to 1.5 s each), then moving on circles, lines and a figure eight at two speeds | Saccades, overshoot, smooth pursuit, the tracker's lag |
@@ -137,7 +139,7 @@ ft-gazepanel gains `show rec`: a large head-locked panel (FULL_DEG, 64 degrees,
     session.json                the bounty, checklist, versions, takes, eye camera rate, IPD at start
     fit.json                    the fit coach's readings: the fitment step, and any slip warnings
     health.json                 the opt-in health answers (encrypted in the export)
-    takes/<NN>-<step group>/    numbered in order: NN-ipd (when there's a sweep), NN-dots (the nine dots), then the bounty's
+    takes/<NN>-<step group>/    numbered in order: NN-dots (the nine dots), then the bounty's
       cam0/000000.zst ...       right eye: one zstd chunk per second, frames back to back (512x400 u8)
       cam1/000000.zst ...       left eye
       index.tsv                 per frame: camera, chunk, offset in it, slot, CLOCK_MONOTONIC_RAW time
@@ -161,7 +163,7 @@ Uploads: `gaze/rec/hub.py` sends nothing to the real dataset until `CONSENT.md`
 
 ## Build order
 
-1. **Headset test 1:** do the eye cameras move with the IPD wheel? (Look at one dot, turn the wheel end to end, watch ft-eyes' pupil x.) Decides the IPD guidance.
+1. **Headset test 1:** do the eye cameras move with the lenses when the IPD changes? The setting can't be changed with the headset on, so compare nine-dot rounds recorded at two settings (ft-eyes' pupil x at the centre dot). Decides the IPD guidance.
 2. Pull the shared recorder code out of `hands/rec/` (hand recorder tests pass).
 3. `eyerec.py` (frame writer) and its replay test against a capture.
 4. `fitcoach.py` with readings from a capture, and the fit card on the existing fit panel.

gaze/rec/UPLOAD.md

@@ -1,4 +1,3 @@
-DRAFT: this text waits for its review, with CONSENT.md.
 ### About uploading
 
 - **What's sent:** the export in `@EXPORT_PATH@` (@EXPORT_SIZE@), into `contributions/@CONTRIBUTOR@/@ROUND@` in [@DATASET@](https://huggingface.co/datasets/@DATASET@). Upload first checks that every file is complete and matches its checksum, and that the eye images and health answers are encrypted.

gaze/rec/eyerec.py

@@ -165,6 +165,8 @@ class Writer:
             got = False
             for c in CAMERAS:
                 n = share.count(c)
+                if n <= seen[c]:   # nothing new (and never step back: that would copy frames twice)
+                    continue
                 if n - seen[c] > share.slots:   # fell behind: those are gone
                     with self._lock:
                         if self._on:

gaze/rec/ft-eyerec

@@ -1,6 +1,9 @@
 #!/bin/bash
-# Launch the Frametop Eye Recorder from a Plasma session on the Frame host.
-# The app runs in the dev container (PySide6, Kirigami and zstd come from Fedora there).
+# Launch the Frametop Eye Recorder from a Plasma session on the Frame host (its app menu entry,
+# gaze/rec/install.sh).
+# The app runs in Frametop's container (PySide6, Kirigami and zstd come from Fedora there; a
+# release's image has the recorders' cryptography and huggingface_hub in /opt/frametop/venv, which
+# distrobox leaves off the front of PATH, so it goes first here).
 # podman needs the real XDG_RUNTIME_DIR and the real user bus (to reach systemd for
 # the container's cgroup; the Frametop session runs on a private bus from
 # dbus-run-session). The session's Wayland socket and bus go to the app itself.
@@ -14,8 +17,8 @@ export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
 # From the home folder: the app's host commands (distrobox-host-exec) run in its folder on the
 # host, and a folder only the container has, such as /run/host/tmp, makes them all fail.
 cd "$HOME" || exit 1
-"$here/../../scripts/container-up.sh"
-exec "$HOME/.local/bin/distrobox" enter dev -- env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
+exec "$here/../../scripts/in-box" env PATH="/opt/frametop/venv/bin:/opt/frametop/bin:$PATH:/usr/sbin:/sbin" \
+  WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
   XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
   QT_QPA_PLATFORM="wayland;xcb" \
   python3 "$here/ft_eyerec.py" "$@"

gaze/rec/ft_eyerec.py

@@ -416,11 +416,11 @@ class Backend(RecorderBackend):
         if not mod or not b:
             return ""
         try:
-            plan = mod.build_plan(mod.load_round(mod.round_path(b["script"])), 0, ipd_sweep=False)
+            plan = mod.build_plan(mod.load_round(mod.round_path(b["script"])), 0)
         except (OSError, ValueError) as e:
             return f"Couldn't read the round: {e}"
-        return (f"{len(plan)} steps, about {mod.plan_seconds(plan) / 60:.0f} min recorded, after the headset's fit "
-                "check (and an IPD sweep on your first round, or when the IPD setting changed)")
+        return (f"{len(plan)} steps, about {mod.plan_seconds(plan) / 60:.0f} min recorded, after putting the "
+                "headset on")
 
     # --- the checklist and health answers
     @Property("QVariantMap", notify=profileRelay)

gaze/rec/hub.py

@@ -8,7 +8,8 @@ nobody types a token anywhere. An upload:
   2. stops if this export (same SHA256SUMS) was uploaded before, unless --again;
   3. stops unless CONSENT.md and UPLOAD.md are here and neither is a draft. FT_EYEREC_ALLOW_UPLOAD=1
      opens it for a rehearsal, and only to another dataset than the real one (FT_EYEREC_DATASET):
-     the real one takes nothing before the texts are reviewed;
+     the real one takes nothing before the texts are reviewed; and it stops for an export made under
+     a draft consent text (agree to the current one, then export again);
   4. checks the login and access to the dataset, opens the pull request first, uploads the folder
      to contributions/<contributor>/<round> in it, and records it under "uploads" in session.json.
 --dry-run does all of it except the network calls and the records, and says what it would upload.
@@ -155,6 +156,9 @@ def upload(store, rnd, dry_run=False, again=False, progress=None, log=None):
     if not dry_run and not upload_allowed(repo):
         raise HubError("closed", "Eye contributions aren't open yet: the consent and upload texts are waiting for a "
                                  f"legal review. (A rehearsal: {ALLOW_ENV}=1 with {DATASET_ENV} set to a test dataset.)")
+    if repo == HF_DATASET and str(summary.get("consent_version") or "").endswith("draft"):
+        raise HubError("draft-consent", "This export was made when you'd agreed to a draft of the consent text. "
+                                        "Agree to the current one on the Welcome page, then export the round again.")
     path_in_repo = f"contributions/{summary.get('contributor', '')}/{rnd}"
     message, description = describe(summary, report, sha, report.bytes)
     files = []

gaze/rec/install.sh

@@ -5,6 +5,10 @@
 # runs) and add "Frametop Eye Recorder" to the app menu (for Frametop's desktop, as the hand
 # recorder's). It doesn't change Frametop's gaze setup; the window offers SteamVR's 90 fps eye
 # cameras (gaze/eye-rate.py) when they're slower.
+# A release (FRAME_RELEASE) brings the gaze tools built in its image (pack/Containerfile), so there it
+# only adds the menu entry; its install.sh offers this.
+# The recorder reads the eye cameras through our eye tracker's frame grabber
+# (frametop-eyegrab.service, gaze/tracker/install.sh): without it, the window says so.
 # Usage: gaze/rec/install.sh            install or update
 #        gaze/rec/install.sh uninstall  remove the menu entry. Recordings stay where they are.
 set -euo pipefail
@@ -14,17 +18,23 @@ entry='~/.local/share/applications/frametop-eyerec.desktop'
 
 case ${1:-install} in
   install)
-    echo "== 1/3 dev container packages"
-    "$root/setup/dev-container.sh"
-    echo "== 2/3 the gaze tools: ft-gaze, ft-gazepanel and ft-eyes"
-    "$root/gaze/build.sh"
-    "$root/gaze/tracker/build.sh"
-    echo "== 3/3 the menu entry"
+    if [ "$FRAME_RELEASE" = 1 ]; then
+      echo "== the menu entry"
+    else
+      echo "== 1/3 dev container packages"
+      "$root/setup/dev-container.sh"
+      echo "== 2/3 the gaze tools: ft-gaze, ft-gazepanel and ft-eyes"
+      "$root/gaze/build.sh"
+      "$root/gaze/tracker/build.sh"
+      echo "== 3/3 the menu entry"
+    fi
     fill_template "$root/gaze/rec/ft-eyerec.desktop" |
       on_frame "mkdir -p ~/.local/share/applications && cat > $entry"
     echo
     echo "Installed. On Frametop's desktop, open \"Frametop Eye Recorder\" from the app menu."
     echo "Recordings go to ~/.local/share/frametop/eyes/contrib."
+    on_frame 'systemctl is-enabled -q frametop-eyegrab.service 2>/dev/null' ||
+      echo "It needs our eye tracker's frame grabber, which isn't installed: gaze/tracker/install.sh"
     ;;
   uninstall)
     on_frame "rm -f $entry"

gaze/rec/main.qml

@@ -298,7 +298,7 @@ Kirigami.ApplicationWindow {
                     padding: Kirigami.Units.largeSpacing
                     wrapMode: Text.Wrap
                     text: "Each bounty is a short round of about 5 minutes for one case where eye tracking struggles. "
-                          + "Pick one; the most wanted come first. Every round starts with fitting the headset and "
+                          + "Pick one; the most wanted come first. Every round starts with putting the headset on and "
                           + "nine dots. You can do a bounty again." + (backend.boardUpdated ? " (Board of "
                           + backend.boardUpdated + ".)" : "")
                 }
@@ -603,10 +603,10 @@ Kirigami.ApplicationWindow {
                 Controls.Label {
                     Layout.maximumWidth: Kirigami.Units.gridUnit * 26
                     wrapMode: Text.Wrap
-                    text: "Press Start, then put the headset on. A panel in the headset first helps you fit it: "
-                          + "look at the dot and move the headset until the panel says good (this window shows it "
-                          + "too). On your first round, and whenever the IPD setting changed, you then turn the IPD "
-                          + "wheel slowly from end to end. Then nine dots, then the bounty's steps. Look at each dot "
+                    text: "Press Start, then put the headset on so it's comfortable and the picture is sharp, "
+                          + "and look at the dot on the panel in the headset. It shows the headset's IPD setting: "
+                          + "it can't be changed with the headset on, so set it before you start. Then nine dots, "
+                          + "then the bounty's steps. Look at each dot "
                           + "until its ring fills, and keep your head still unless a step says otherwise.\n\n"
                           + "Each step waits until you're ready: press the button on the right side of the headset, "
                           + "or Space or Next in this window. A 3-2-1 countdown follows, then the step. Nothing is "
@@ -735,13 +735,6 @@ Kirigami.ApplicationWindow {
                         font: Kirigami.Theme.smallFont
                         text: "A first guess: the coach has been measured on one face so far."
                     }
-                    Controls.Label {
-                        visible: sessionView.st.ipd !== undefined && sessionView.st.ipd !== null
-                        Layout.fillWidth: true
-                        wrapMode: Text.Wrap
-                        opacity: 0.7
-                        text: "The headset's IPD setting: " + ((sessionView.st.ipd || 0) * 1000).toFixed(1) + " mm"
-                    }
                 }
             }
 
@@ -805,7 +798,8 @@ Kirigami.ApplicationWindow {
                     FitCard {
                         Layout.alignment: Qt.AlignTop
                         Layout.preferredWidth: Kirigami.Units.gridUnit * 18
-                        visible: sessionView.state === "fitment" || (sessionView.state === "paused" && sessionView.card !== null)
+                        visible: (sessionView.state === "fitment" && sessionView.st.fit_coach !== false)
+                                 || (sessionView.state === "paused" && sessionView.card !== null)
                         card: sessionView.card
                     }
 
@@ -821,6 +815,16 @@ Kirigami.ApplicationWindow {
                             text: sessionView.state === "fitment" ? "Headset position: " + (sessionView.st.fit_words || "")
                                                                   : (sessionView.st.prompt || "")
                         }
+                        // The IPD setting (SteamOS's, through SteamVR), shown while the headset goes on
+                        Controls.Label {
+                            Layout.fillWidth: true
+                            visible: sessionView.state === "fitment"
+                            wrapMode: Text.Wrap
+                            text: sessionView.st.ipd === undefined || sessionView.st.ipd === null
+                                  ? "Couldn't read the IPD setting from SteamVR."
+                                  : "Your IPD setting: " + (sessionView.st.ipd * 1000).toFixed(1) + " mm. If that's not "
+                                    + "right for you, stop (Esc), take the headset off, set it, and start again."
+                        }
                         // The countdown's 3, 2, 1, then a cue's word, big
                         Controls.Label {
                             visible: !!sessionView.st.big && sessionView.state !== "paused"
@@ -849,7 +853,9 @@ Kirigami.ApplicationWindow {
                             visible: sessionView.waiting && sessionView.state !== "paused"
                             opacity: 0.7
                             text: sessionView.state === "fitment"
-                                  ? "Press Next once the fit says good. Pressing it before warns first; a second press starts anyway."
+                                  ? (sessionView.st.fit_coach === false
+                                     ? "Put the headset on comfortably, with a sharp picture, look at the dot, then press Next."
+                                     : "Press Next once the fit says good. Pressing it before warns first; a second press starts anyway.")
                                   : (sessionView.st.ready_text || "Ready? Press Space or click Next")
                                     + ". A 3-2-1 countdown starts the recording."
                             wrapMode: Text.Wrap

gaze/rec/recipients.txt

@@ -1,4 +1,6 @@
 # The eye dataset's recipients (gaze/rec/DESIGN.md "Files"): exports encrypt each chunk and health.json
-# to all of them (recorder/age.py). Key 1 is kept on the maintainer's training machine, key 2 offline.
+# to all of them (recorder/age.py). Key 1 is kept on the 7i, key 2 offline,
+# key 3 on dody (the GPU training machine, which decrypts pull requests itself).
 age1a0tc98pj2s7gdfgrv4gam6ah0qv69ymf3t2cc25hcwepmats69hst35z2e
 age1r0zlq7kcyg4dz4y3vkl9j36jw3zc6ly6reddh9ag5kfy6aw0fy6s5hepat
+age1e5vwgc03s9mk3dj5j39lcgxg6qyvkml77qh7g8svygar49r2kd7svldt70

gaze/rec/rounds/README.md

@@ -0,0 +1,106 @@
+# Building an eye bounty
+
+A bounty is two things:
+
+- **A board entry** in `gaze/rec/bounties.json`: id, title, blurb, minutes, priority, the script it runs,
+  and what it needs from the checklist (`recorder/bounties.py` has the format).
+- **A round script**, `gaze/rec/rounds/<script>.json`: the steps (`session.py`'s docstring has the
+  format of every step kind).
+
+Scripts ship with the recorder. The dataset's copy of the board (`DeeJanuz/frametop-eyes`,
+`bounties.json`) can only describe, reorder, reprioritise or retire bounties, so a new bounty reaches
+people only with a Frametop release that carries its script.
+
+`load_round` enforces the rules marked **(checked)**: a script that breaks one doesn't load, and the
+tests fail.
+
+## What the round is for
+
+Every recorded second becomes training data, labelled by where the dot was. So:
+
+- **Know what each step labels.** In most steps the eyes are on a dot, and the dot's position is the
+  label. Where they aren't (eyes shut, reading, looking around freely), the cue words or the step's
+  kind say so, and training leaves those frames out or treats them as their own case.
+- **One edge case per bounty.** Say in the board's blurb and the DESIGN.md table which case it covers
+  and why the trackers fail there.
+- **Rounds open the same way.** The recorder adds putting the headset on and the nine dots before every
+  round. Don't repeat them. The group `dots` is theirs **(checked)**.
+
+## Cues the person can't see get a beep
+
+The panel can't cue someone whose eyes are shut. A cue that asks for that plays a beep
+(`gaze/rec/sounds.py`) as its third field: `["Close your eyes", 2.5, "close"]`.
+
+- **Blink:** a cue whose word asks for a blink plays `blink` **(checked)**.
+- **Both eyes shut:** a cue asking to close or shut both eyes plays `close` **(checked)**, and the next
+  cue plays `open` **(checked)**, so the person hears when to open them. A step never ends with the eyes
+  shut.
+- **Say it in the step's text**, which is read before Next: "Close your eyes when you hear a beep, and
+  then open your eyes when you hear the beep again." The words still show on the panel too.
+- **One eye or half-closed** (winks, squints, half-closed lids) keep words alone: the open eye reads
+  them. They may beep too.
+- **No voice.** Beeps only; generated speech sounds odd in the headset. A new kind of cue sound goes
+  into `sounds.SOUNDS` and the tests.
+
+## Cue words are labels
+
+Training reads the cue words in `prompts.jsonl` (`frame-eyes/train/data/ingest.py`, `shut_cams`):
+
+- A word with **blink, close, shut, wink or yawn** marks the eyes as shut while it shows. **Left** or
+  **right** limits that to one eye, and **half** means the lids are partly open, so the frames keep
+  their labels.
+- Use those words only for that. "Open", "Look", "Relax" and "Normal" are the words for going back.
+- Keep words short (40 characters at most): they show big on the panel.
+
+## Timing
+
+- **Dots:** a hold of 1.2 s or more after a settle of about 0.6 s (the defaults). The eyes take that
+  long to land on a dot and stay.
+- **Shut eyes:** 2 s or more closed, then 2 s or more open before anything else, so the eyes settle back
+  on the dot.
+- **Blinks on a beep:** uneven gaps, 1.2 to 2 s, so the beep can't be predicted.
+- **Length:** about 4 to 6 minutes recorded. Check with `python3 gaze/rec/session.py <script> --plan`, and
+  put that, rounded, in the board's `minutes`.
+
+## Where things can go
+
+- **Dots and paths stay on the panel** **(checked)**: about 30 degrees to the sides, 23 up and down, less
+  in the corners (`off_panel`).
+- **Distance:** 0.25 to 10 m **(checked)**. 1.5 m unless distance is what the bounty is about.
+- **Head still,** unless the bounty is about moving it: say "Keep your head still" in the text of dot steps.
+
+## Takes
+
+- Steps of the same `group` share one take, and a group's steps must follow each other **(checked)**.
+  Group steps that a reviewer would delete together.
+- A `reseat` step isn't recorded, and a round with one gets no shift check.
+- Every step waits for Next unless it has `"ready": false`. Only chain steps that follow on naturally.
+
+## Comfort, safety and privacy
+
+- Comfortable eye turns only: "as far as is comfortable", never "as far as you can".
+- No flashing: brightness changes are steps of a second or more (`light` levels), never a flicker.
+- Nothing that asks for sound, the outward cameras, or anything about the person beyond the checklist.
+  Talking is fine (it moves the face), but the microphone is never recorded.
+- Text to read is neutral and made up: no names, places or questions about the person.
+
+## Writing the text
+
+- Plain words, one instruction per step, the action first: "Look at each dot until its ring fills."
+- `|` starts a new line on the panel, `||` leaves a blank line.
+- Say what to do, not why. The why goes in the round's `intro` or the board's blurb.
+
+## Before it ships
+
+1. `python3 gaze/rec/session.py <script> --plan`: the steps and their length.
+2. The tests, in the dev container: `cd gaze/rec && for t in tests/test_*.py; do python3 $t; done`.
+   `test_shipped_rounds_load` loads every script; update its count.
+3. A dry run: `python3 gaze/rec/session.py <script> --dry-run --speed 20 --next-after 0.1`.
+4. A replay end to end: `frame-eyes/tools/eyerec-dryrun <bounty>` (record, export, validate), and the
+   export through `eyes_hub.py check` and `ingest.py` on dody. Delete those rounds afterwards: their
+   labels are wrong.
+5. A real round in the headset, with a note in `HEADSET-TESTS.md`: is every cue clear, and is the
+   length right?
+6. Add the bounty to `gaze/rec/bounties.json` and a row to DESIGN.md's bounty table. After a release
+   carries the script, put it on the dataset's board too (`frame-eyes/dataset-repo/bounties.json`,
+   uploaded to `DeeJanuz/frametop-eyes`).

gaze/rec/rounds/blinks.json

@@ -2,12 +2,12 @@
  "schema": 1,
  "id": "blinks",
  "title": "Blinks and lids",
- "intro": "Blinks, closed and half-closed eyes. Keep looking at the dot whenever your eyes are open.",
+ "intro": "Blinks, closed and half-closed eyes, some of them on a beep. Keep looking at the dot whenever your eyes are open.",
  "steps": [
   {
    "id": "blink-cue",
    "kind": "prompt",
-   "text": "Look at the dot and blink each time the word says so.",
+   "text": "Look at the dot and blink once on each beep.",
    "dot": [
     0,
     0
@@ -15,42 +15,50 @@
    "cues": [
     [
      "Blink",
-     1.5
+     1.6,
+     "blink"
     ],
     [
      "Blink",
-     1.5
+     1.3,
+     "blink"
     ],
     [
      "Blink",
-     1.5
+     2.0,
+     "blink"
     ],
     [
      "Blink",
-     1.5
+     1.5,
+     "blink"
     ],
     [
      "Blink",
-     1.5
+     1.8,
+     "blink"
     ],
     [
      "Blink",
-     1.5
+     1.2,
+     "blink"
     ],
     [
      "Blink",
-     1.5
+     1.7,
+     "blink"
     ],
     [
      "Blink",
-     1.5
+     1.5,
+     "blink"
     ]
    ]
   },
   {
    "id": "closed",
    "kind": "prompt",
-   "text": "Look at the dot, then close both eyes when it says so.",
+   "text": "Close your eyes when you hear a beep, and then open your eyes when you hear the beep again.",
    "dot": [
     0,
     0
@@ -61,20 +69,24 @@
      2
     ],
     [
-     "Close both eyes",
-     2.5
+     "Close your eyes",
+     2.5,
+     "close"
     ],
     [
-     "Open",
-     2
+     "Open your eyes",
+     2,
+     "open"
     ],
     [
-     "Close both eyes",
-     2.5
+     "Close your eyes",
+     2.5,
+     "close"
     ],
     [
-     "Open",
-     2
+     "Open your eyes",
+     2,
+     "open"
     ]
    ]
   },

gaze/rec/session.py

@@ -8,11 +8,13 @@ Then, in step mode:
 
   1. Fitment, not a take: the fit coach (gaze/fitcoach.py) on a centre dot until the headset
      sits well and the person presses Next; Next before that warns, a second press goes on and
-     the round records it was skipped. Its readings go into fit.json.
-  2. On the contributor's first round, or when SteamVR's IPD setting changed since their last,
-     an IPD sweep: one take, the wheel turned end to end while looking at a centre dot.
-  3. Nine dots, a take: a 3x3 grid to 20 degrees on the mid background, the round's reference.
-  4. The bounty's steps, from its round script (rounds/<script>.json, below).
+     the round records it was skipped. Its readings go into fit.json. The coach is off for now
+     (FIT_COACH): then this only asks for a comfortable fit, Next goes on, and its readings are
+     kept without being shown. Either way it shows the IPD setting, read from SteamVR when the
+     round starts: it can't be changed with the headset on, so there's no IPD step, and a wrong
+     setting means stopping and setting it first.
+  2. Nine dots, a take: a 3x3 grid to 20 degrees on the mid background, the round's reference.
+  3. The bounty's steps, from its round script (rounds/<script>.json, below).
 
 Each step that's "ready" shows first and waits for Next; then a 3-2-1 countdown, recorded, then
 the step. Steps of the same group share a take (takes/NN-group): the eye frames (eyerec.py),
@@ -34,7 +36,9 @@ Round scripts, rounds/<id>.json: {"schema": 1, "id", "title", "intro", "shift_ch
   path     "path": "circle|line|eight <numbers>" (the panel's path command), "seconds"
   jumps    "count", "range": [yaw, pitch] (a box), "each": [shortest, longest] s
   prompt   "dot": [yaw, pitch] or null (none), and "seconds" or "cues": [[word, s]...] (shown
-           big in turn), "cue_distance": {word: metres}
+           big in turn; [word, s, sound] also plays a beep as it starts, for steps done with the
+           eyes shut: sound blink, close or open, sounds.py; rounds/README.md has the rules,
+           and load_round enforces them), "cue_distance": {word: metres}
   light    "dot", "levels": [[background, s]...]
   world    "point": [x, y, z] (metres from the head when it starts, -z ahead), "seconds"
   reseat   not recorded: the text, Next, then a fit reading
@@ -76,6 +80,7 @@ sys.path.insert(0, GAZE)
 sys.path.insert(0, HERE)
 import eyerec  # noqa: E402  (gaze/rec/eyerec.py: the frame writer)
 import fitcoach  # noqa: E402  (gaze/fitcoach.py)
+import sounds  # noqa: E402  (gaze/rec/sounds.py: the beeps)
 from recorder import bounties  # noqa: E402
 from recorder.button import BUTTON_DEVICE, ButtonReader, find_button, mouse_connected, read_input_devices  # noqa: E402
 from recorder.common import (clean_text, clock_sample, in_container, mono_ns, os_version,  # noqa: E402
@@ -110,7 +115,6 @@ FIT_ASK_S = 0.5           # the fit coach's reading, this often during the fitme
 FIT_GOOD_S = 2.0          # OK this long: the fit is good
 READING_WAIT_S = 6.0      # a reseat's fit reading waits this long for both eyes
 CENTRE_DEG = 1.0          # a dot this near the middle gets the shift check after its hold
-SPLIT_IPD_M = 0.0005      # the IPD setting changed: an IPD sweep
 VERSION = "ft-eyerec 0.1"
 READY_TEXT = "Ready? Press Space or click Next"
 READY_BUTTON = "Ready? Press the button on the right side of the headset"
@@ -124,11 +128,20 @@ FIT_TITLE = "Headset fit"
 FIT_TEXT = "Look at the dot. Adjust the headset until this says good."
 FIT_ADJUST = "Adjust the headset until it says good"
 FIT_SKIP_NOTE = "The fit isn't good yet: recordings may be less useful. Press again to start anyway."
-IPD_STEP = {"id": "ipd", "kind": "prompt", "group": "ipd", "dot": [0, 0], "seconds": 25,
-            "text": "Keep looking at the dot and turn the IPD wheel slowly: all the way narrow, then all the "
-                    "way wide, then back to where the picture is sharpest."}
+# The fit coach is off until it's trained on more people's fits (2026-10-16: its hints jumped
+# around and "good" was out of reach). Off, the fitment only asks for a comfortable fit and a look
+# at the dot, Next goes straight on, and the coach's readings still go into fit.json, silently:
+# they're what it will be trained on. No baseline then, so no slip notes either.
+# FT_EYEREC_FIT_COACH=1 (or session.py --fit-coach) turns it back on.
+FIT_COACH = os.environ.get("FT_EYEREC_FIT_COACH", "") == "1"
+FIT_OFF_TITLE = "Put the headset on"
+FIT_OFF_TEXT = "Put the headset on so it's comfortable and the picture is sharp. Then look at the dot."
+# The IPD setting is the headset's (SteamOS, read through SteamVR), shown at the fitment. It can't be
+# changed with the headset on, so a wrong one means stopping the round.
+IPD_TEXT = "Your IPD setting: {mm:.1f} mm. If that's not right for you, stop, take the headset off and set it."
+IPD_UNKNOWN = "Couldn't read the IPD setting from SteamVR."
 NINE_DOTS_TEXT = "Look at each dot until its ring fills. Keep your head still."
-DONE_SCREEN = {"title": "All done", "text": "Thank you! You can take off the headset.|Review the round in the Eye "
+DONE_SCREEN = {"title": "All done", "text": "Thank you! This recording session will close. Continue in the application.|Review the round in the Eye "
                                             "recorder window.", "seconds": 5}
 STOPPED_SCREEN = {"title": "Stopped", "text": "What was recorded so far is saved. Review it in the Eye recorder window.",
                   "seconds": 4}
@@ -203,6 +216,32 @@ def _dot(v, what):
     return d
 
 
+def cue_needs(word):
+    """The sound a cue must have because the person can't see the panel then (rounds/README.md):
+    "blink" for a blink, "close" for shutting both eyes, else None. Half-closed eyes and one eye
+    (a wink: the other one reads the panel) don't need one."""
+    w = word.lower()
+    if "half" in w or "left" in w or "right" in w:
+        return None
+    if "blink" in w:
+        return "blink"
+    if "close" in w or "shut" in w:
+        return "close"
+    return None
+
+
+def _check_sounds(cues, sid):
+    """Eyes-shut cues beep, and the cue after a "close" plays "open": with the eyes shut, the
+    words can't tell anyone when to open them."""
+    for k, (word, _secs, *sound) in enumerate(cues):
+        need = cue_needs(word)
+        if need and sound != [need]:
+            raise ValueError('%s: cue "%s" needs the sound "%s" (the eyes are shut then; rounds/README.md)'
+                             % (sid, word, need))
+        if sound == ["close"] and (k + 1 == len(cues) or cues[k + 1][2:] != ["open"]):
+            raise ValueError('%s: after "%s" comes a cue with the sound "open" (rounds/README.md)' % (sid, word))
+
+
 def _step(raw, i):
     """One step, checked and filled in."""
     if not isinstance(raw, dict):
@@ -263,12 +302,16 @@ def _step(raw, i):
         st["dot"] = None if "dot" in raw and raw["dot"] is None else _dot(raw.get("dot", [0, 0]), sid + " dot")
         cues = raw.get("cues")
         if cues is not None:
+            want = "%s: cues want [[word, seconds]...] or [word, seconds, sound] (sound: %s)" % (
+                sid, ", ".join(sounds.SOUNDS))
             if not isinstance(cues, list) or not cues:
-                raise ValueError("%s: cues want [[word, seconds]...]" % sid)
-            st["cues"] = [[clean_text(c[0])[:40], _num(c[1], 0.2, 60, sid + " cue")] for c in cues
-                          if isinstance(c, list) and len(c) == 2 and isinstance(c[0], str)]
+                raise ValueError(want)
+            st["cues"] = [[clean_text(c[0])[:40], _num(c[1], 0.2, 60, sid + " cue")] + c[2:] for c in cues
+                          if isinstance(c, list) and len(c) in (2, 3) and isinstance(c[0], str)
+                          and (len(c) == 2 or c[2] in sounds.SOUNDS)]
             if len(st["cues"]) != len(cues):
-                raise ValueError("%s: cues want [[word, seconds]...]" % sid)
+                raise ValueError(want)
+            _check_sounds(st["cues"], sid)
             cd = raw.get("cue_distance") or {}
             st["cue_distance"] = {str(k): _num(v, 0.25, 10, sid + " cue distance") for k, v in cd.items()}
         else:
@@ -311,8 +354,8 @@ def load_round(path):
             raise ValueError("%s: group %s comes back after another" % (path, s["group"]))
         groups.add(s["group"])
         last = s["group"]
-    if groups & {"dots", "ipd"}:
-        raise ValueError("%s: the groups dots and ipd are the opener's" % path)
+    if "dots" in groups:
+        raise ValueError("%s: the group dots is the opener's" % path)
     rid = raw.get("id")
     return {"id": rid if isinstance(rid, str) else os.path.basename(path)[:-5], "title": str(raw.get("title", rid)),
             "intro": str(raw.get("intro", "")), "shift_check": raw.get("shift_check", True) is not False and
@@ -346,13 +389,10 @@ def nine_dots(rng):
             "shuffle": False}
 
 
-def build_plan(rnd, seed, ipd_sweep):
+def build_plan(rnd, seed):
     """The round's steps in order, the opener's first, shuffles and jumps made from seed."""
     rng = random.Random(seed)
-    steps = []
-    if ipd_sweep:
-        steps.append(_step(IPD_STEP, 0))
-    steps.append(nine_dots(rng))
+    steps = [nine_dots(rng)]
     for s in rnd["steps"]:
         s = json.loads(json.dumps(s))
         if s["kind"] == "dots" and s["shuffle"]:
@@ -561,7 +601,7 @@ class Session:
     def __init__(self, base_dir, script, *, bounty=None, profile=None, checklist=None, health=None, dry_run=False,
                  speed=1.0, on_status=None, next_after=None, cams=None, panel_bin=None, panel_no_vr=False,
                  steamvr_gaze=True, button=True, button_device=None, seed=None, rounds_dir=None,
-                 profile_path=None, eyes_python=None):
+                 profile_path=None, eyes_python=None, fit_coach=None):
         self.base_dir = os.path.abspath(os.path.expanduser(base_dir or BASE_DIR))
         self.script = script
         self.round = load_round(round_path(script, rounds_dir))
@@ -575,6 +615,7 @@ class Session:
         self.steamvr_gaze = steamvr_gaze and not dry_run
         self.button, self.button_device = button, button_device
         self.eyes_python = eyes_python or EYES_PYTHON
+        self.fit_coach = FIT_COACH if fit_coach is None else bool(fit_coach)
         self.seed = seed
         self.print = print
         self.quiet_units = () if dry_run or cams else QUIET_UNITS
@@ -589,10 +630,11 @@ class Session:
         self._status = {"state": "starting", "bounty": self.bounty["id"], "title": "", "step": "", "step_index": 0,
                         "step_count": 0, "prompt": "", "note": "", "take": None, "error": "", "waiting": False,
                         "countdown": 0, "big": "", "can_redo": False, "fit": None, "fit_words": "", "fit_card": None,
-                        "ipd": None, "ready_text": READY_TEXT, "button": False, "seconds_left": 0.0}
+                        "ipd": None, "ready_text": READY_TEXT, "button": False, "seconds_left": 0.0,
+                        "fit_coach": self.fit_coach}
         self._last_emit = 0.0
         self._log_file = None
-        self._panel = self._eyes = None
+        self._panel = self._eyes = self._sounds = None
         self._procs = {}            # name: Popen
         self._gaze_thread = None
         self._gaze_lock = threading.Lock()
@@ -739,6 +781,7 @@ class Session:
     def _setup(self):
         self._panel = Socket(PANEL_SOCKET, self.dry_run, fake_panel, self._log, self.print)
         self._eyes = Socket(EYES_SOCKET, self.dry_run, fake_eyes(self.ref), self._log)
+        self._sounds = sounds.Player(on=not self.dry_run, log=self._log)
         if not self.dry_run:
             for path, build in ((self.panel_bin, "gaze/build.sh"), (FT_EYES, "gaze/tracker/build.sh")):
                 if not os.path.exists(path):
@@ -755,9 +798,8 @@ class Session:
         r = self._panel.cmd("ipd", reply=True)
         self._ipd = float(r.split()[1]) if r and r.startswith("ok ") else None
         self._status["ipd"] = self._ipd
-        last = self.profile.get("eyes_ipd")
-        sweep = self._ipd is not None and (not isinstance(last, (int, float)) or abs(self._ipd - last) >= SPLIT_IPD_M)
-        self.plan = build_plan(self.round, self.seed, sweep)
+        self._log("IPD setting: %s" % ("%.1f mm" % (self._ipd * 1000) if self._ipd is not None else "unknown"))
+        self.plan = build_plan(self.round, self.seed)
         self._fit["reference"] = {"source": self.ref.source, "faces": self.ref.faces, "made": self.ref.made,
                                   "provisional": self.ref.provisional} if self.ref else None
         self._session_json = {
@@ -765,7 +807,7 @@ class Session:
             "title": self.bounty.get("title") or self.round["title"], "board_updated": self.bounty.get("updated"),
             "checklist": self.checklist, "recorder": VERSION, "os": os_version(), "steamvr": steamvr_version(),
             "started": time.strftime("%Y-%m-%dT%H:%M:%S%z"), "ended": None, "status": "running", "seed": self.seed,
-            "dry_run": self.dry_run, "replay": self.cams or None, "ipd_start": self._ipd, "ipd_sweep": sweep,
+            "dry_run": self.dry_run, "replay": self.cams or None, "ipd_start": self._ipd,
             "fit": None, "eye_fps": None, "takes": [],
             "plan": [{"id": s["id"], "kind": s["kind"], "group": s["group"]} for s in self.plan]}
         if self.health:
@@ -1029,6 +1071,9 @@ class Session:
         self._panel.set("title", "title " + clean_text(title))
         self._panel.set("text", "text " + clean_text(text))
 
+    def _ipd_text(self):
+        return IPD_UNKNOWN if self._ipd is None else IPD_TEXT.format(mm=self._ipd * 1000)
+
     # --- the fit coach
     def _reading(self, previous=None, baseline=None):
         """The fit coach's reading of ft-eyes' last second, and the finds it's from (or None, None)."""
@@ -1039,10 +1084,13 @@ class Session:
 
     def _fitment(self):
         """The fit coach on a centre dot until the fit is good and Next is pressed; Next before
-        that warns, and a second press goes on (fit.json records it was skipped)."""
+        that warns, and a second press goes on (fit.json records it was skipped). With the coach
+        off (FIT_COACH), _fitment_quiet instead."""
+        if not self.fit_coach:
+            return self._fitment_quiet()
         title = "%s: %s" % (FIT_TITLE, self._session_json["title"])
         intro = self.round["intro"]
-        self._show(title, FIT_TEXT + ("||" + intro if intro else ""))
+        self._show(title, FIT_TEXT + "||" + self._ipd_text() + ("||" + intro if intro else ""))
         self._panel.cmd("dot 0 0 look")
         f = {"reading": None, "summary": None, "since": 0.0, "good": False, "ask": 0.0, "warned": False}
         started = mono_ns()
@@ -1069,7 +1117,7 @@ class Session:
                     self._panel.set("note", "note ")
             self._panel.cmd("dot 0 0 %s" % ("done" if good else "look"))
             self._panel.set("action", "action " + (self._status["ready_text"] if good or f["warned"] else FIT_ADJUST))
-            self._show(title, "Headset position: " + words + ("||" + intro if intro else ""))
+            self._show(title, "Headset position: " + words + "||" + self._ipd_text() + ("||" + intro if intro else ""))
             self._emit(force=False, fit=r["state"] if r else None, fit_words=words, fit_card=fit_card(r))
 
         self._redo_ok = False
@@ -1098,6 +1146,37 @@ class Session:
         self._panel.cmd("dot 0 0 off")
         self._emit(note="", fit=None, fit_words="", fit_card=None)
 
+    def _fitment_quiet(self):
+        """The fitment with the coach off: a comfortable fit and the centre dot, Next goes on.
+        The coach's readings are kept in fit.json, but nothing of them is shown."""
+        intro = self.round["intro"]
+        self._show("%s: %s" % (FIT_OFF_TITLE, self._session_json["title"]),
+                   FIT_OFF_TEXT + "||" + self._ipd_text() + ("||" + intro if intro else ""))
+        self._panel.cmd("dot 0 0 look")
+        self._panel.set("action", "action " + self._status["ready_text"])
+        f = {"reading": None, "ask": 0.0}
+        started = mono_ns()
+
+        def tick(dt):
+            f["ask"] -= dt
+            if f["ask"] > 0:
+                return
+            f["ask"] = FIT_ASK_S
+            r, _ = self._reading(f["reading"])
+            if r is not None:
+                f["reading"] = r
+                self._fit["readings"].append({"t": mono_ns(), "phase": "fitment", **compact(r)})
+
+        self._redo_ok = False
+        self._await_next(tick)
+        self._fit["fitment"] = {"from": started, "to": mono_ns(), "result": "off", "ipd": self._ipd,
+                                "last": compact(f["reading"]) if f["reading"] else None}
+        self._session_json["fit"] = "off"
+        self._save_session()
+        self._save_fit()
+        self._log("fitment: the coach is off%s" % ((", it read " + coach_words(f["reading"])) if f["reading"] else ""))
+        self._panel.cmd("dot 0 0 off")
+
     def _shift_check(self, step):
         """After a centre dot's hold: has the headset moved since the fitment?"""
         if not self.round["shift_check"] or not self._fit["baseline"]:
@@ -1134,7 +1213,8 @@ class Session:
         if r:
             self._fit["readings"].append({"t": mono_ns(), "phase": "reseat", "step": step["id"], **compact(r)})
             words = coach_words(r)
-            self._panel.set("note", "note Headset position: " + words if r["state"] != "ok" else "note ")
+            if self.fit_coach:
+                self._panel.set("note", "note Headset position: " + words if r["state"] != "ok" else "note ")
             self._log("    reseat: %s" % words)
         self._save_fit()
 
@@ -1295,7 +1375,7 @@ class Session:
                    countdown=0, big="", seconds_left=round(step_seconds(st), 1))
 
     def _title(self, st):
-        return {"ipd": "IPD sweep", "dots": "Nine dots"}.get(st["group"], self.round["title"])
+        return {"dots": "Nine dots"}.get(st["group"], self.round["title"])
 
     def _first_dot(self, st):
         """Where the eyes should be when the step starts: shown during its countdown."""
@@ -1390,12 +1470,17 @@ class Session:
             self._wait(st["seconds"])
             return
         total, gone = sum(c[1] for c in st["cues"]), 0.0
-        for word, secs in st["cues"]:
+        for word, secs, *sound in st["cues"]:
             self._panel.set("big", "big " + word)
             dist = st["cue_distance"].get(word)
             if dist:
                 self._panel.cmd("distance %.3f" % dist)
-            self._event("cue", id=st["id"], word=word, **({"distance": dist} if dist else {}))
+            self._event("cue", id=st["id"], word=word, **({"distance": dist} if dist else {}),
+                        **({"sound": sound[0]} if sound else {}))
+            if sound:
+                self._sounds.play(sound[0])
+                if self.dry_run:
+                    self.print("sound: " + sound[0])
             self._emit(big=word)
             self._wait(secs, bar=False)
             gone += secs
@@ -1468,6 +1553,7 @@ def main():
     ap.add_argument("--no-headset-button", action="store_true", help="don't read the headset's button")
     ap.add_argument("--button-device", metavar="PATH", help="test: read the button from this device or FIFO")
     ap.add_argument("--seed", type=int, help="the shuffles and jumps (default: from the round's id)")
+    ap.add_argument("--fit-coach", action="store_true", help="the fit coach on (off by default for now: FIT_COACH)")
     ap.add_argument("--glasses", choices=("none", "glasses", "inserts", "contacts"), help="the checklist's answer")
     a = ap.parse_args()
 
@@ -1504,8 +1590,8 @@ def main():
         print("round %s: %s" % (script, e))
         return 2
     if a.plan:
-        plan = build_plan(rnd, a.seed or 0, ipd_sweep=False)
-        print("%s: %d steps, about %.1f min recorded (plus the fitment, and the IPD sweep on a first round)"
+        plan = build_plan(rnd, a.seed or 0)
+        print("%s: %d steps, about %.1f min recorded (plus the fitment)"
               % (rnd["title"], len(plan), plan_seconds(plan) / 60))
         for s in plan:
             extra = len(s["dots"]) if "dots" in s else ""
@@ -1536,7 +1622,8 @@ def main():
     s = Session(base, script, bounty=entry, profile=profile, checklist=checklist, dry_run=a.dry_run, speed=a.speed,
                 on_status=on_status, next_after=a.next_after, cams=a.cams, panel_bin=a.panel, panel_no_vr=a.panel_no_vr,
                 steamvr_gaze=not a.no_steamvr_gaze, button=not a.no_headset_button and (not a.dry_run or a.button_device),
-                button_device=a.button_device, seed=a.seed, profile_path=profile_path, eyes_python=a.eyes_python)
+                button_device=a.button_device, seed=a.seed, profile_path=profile_path, eyes_python=a.eyes_python,
+                fit_coach=True if a.fit_coach else None)
 
     def on_signal(*_):
         print("stopping", flush=True)

gaze/rec/sounds.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""The eye recorder's sound cues: plain beeps for steps done with the eyes shut, where the panel's
+words can't be seen. A round script names one as a cue's third field (session.py: "cues":
+[[word, seconds, sound]...]); it plays as the cue starts, and prompts.jsonl's "cue" line says which.
+
+  blink   a short high beep: blink once
+  close   a beep: close your eyes
+  open    the same beep again: open them
+
+No voice, no sound files: the beeps are made here (a sine with a short fade, so it doesn't click)
+into the run folder, and played with PipeWire's pw-play on the default output (the headset's
+speakers). A beep starts some tens of ms after its cue line (pw-play's start and its 20 ms
+latency); a person's reaction to it takes several times that.
+
+  sounds.py              play each one, a second apart
+  sounds.py NAME...      play those
+"""
+import math
+import os
+import shutil
+import struct
+import subprocess
+import sys
+import time
+import wave
+
+RATE = 48000
+VOLUME = 0.6        # pw-play's stream volume; the beep itself is at half scale
+# name: (pitch Hz, seconds)
+SOUNDS = {
+    "blink": (1000.0, 0.09),
+    "close": (660.0, 0.18),
+    "open": (660.0, 0.18),
+}
+FADE_S = 0.005
+
+
+def _dir():
+    d = os.path.join(os.environ.get("XDG_RUNTIME_DIR") or "/run/user/%d" % os.getuid(), "frametop", "eyerec-sounds")
+    os.makedirs(d, exist_ok=True)
+    return d
+
+
+def wav_path(name):
+    """The sound's WAV file, made the first time (and again when its pitch or length changed)."""
+    hz, secs = SOUNDS[name]
+    path = os.path.join(_dir(), "%s-%d-%d.wav" % (name, hz, secs * 1000))
+    if not os.path.exists(path):
+        n, fade = int(RATE * secs), int(RATE * FADE_S)
+        frames = bytearray()
+        for i in range(n):
+            env = min(1.0, i / fade, (n - 1 - i) / fade)
+            frames += struct.pack("<h", int(0.5 * 32767 * env * math.sin(2 * math.pi * hz * i / RATE)))
+        with wave.open(path + ".tmp", "wb") as w:
+            w.setnchannels(1)
+            w.setsampwidth(2)
+            w.setframerate(RATE)
+            w.writeframes(bytes(frames))
+        os.replace(path + ".tmp", path)
+    return path
+
+
+class Player:
+    """Plays sounds without waiting for them. Off (dry runs, tests): it only says what it would play.
+    Without pw-play, or when a sound fails, it logs that once and the round goes on silent."""
+
+    def __init__(self, on=True, log=None):
+        self.log = log or (lambda text: None)
+        self.pw_play = shutil.which("pw-play") if on else None
+        self.procs = []
+        self.warned = False
+        if on and not self.pw_play:
+            self._warn("no pw-play: the sound cues are off")
+        if self.pw_play:
+            for name in SOUNDS:     # made now, not while a step runs
+                wav_path(name)
+
+    def _warn(self, text):
+        if not self.warned:
+            self.warned = True
+            self.log(text)
+
+    def play(self, name):
+        if name not in SOUNDS:
+            raise ValueError("no sound %r" % name)
+        if not self.pw_play:
+            return False
+        self.procs = [p for p in self.procs if p.poll() is None]
+        env = dict(os.environ, XDG_RUNTIME_DIR=os.environ.get("XDG_RUNTIME_DIR") or "/run/user/%d" % os.getuid())
+        try:
+            self.procs.append(subprocess.Popen(
+                [self.pw_play, "--latency", "20ms", "--volume", str(VOLUME), "--media-role", "Notification",
+                 wav_path(name)], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
+                env=env))
+            return True
+        except OSError as e:
+            self._warn("can't play the sound cues: %s" % e)
+            return False
+
+    def close(self):
+        for p in self.procs:
+            try:
+                p.wait(timeout=1)
+            except subprocess.TimeoutExpired:
+                p.kill()
+        self.procs = []
+
+
+def main(argv):
+    names = argv or list(SOUNDS)
+    bad = [n for n in names if n not in SOUNDS]
+    if bad:
+        sys.exit("no sound %s; there are: %s" % (", ".join(bad), ", ".join(SOUNDS)))
+    p = Player(log=lambda text: print(text, file=sys.stderr))
+    for i, name in enumerate(names):
+        if i:
+            time.sleep(1.0)
+        print(name)
+        p.play(name)
+    p.close()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

gaze/rec/tests/test_hub.py

@@ -63,16 +63,35 @@ class HubTest(unittest.TestCase):
         self.assertEqual(hub.rhub.uploads(hub.meta_path(self.store, RID)), [])
 
     def test_gate(self):
-        self.assertFalse(hub.texts_ready(), "the texts are drafts")
-        with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: ""}):
-            self.assertFalse(hub.upload_allowed(hub.dataset_id()), "never the real dataset before the review")
-            with self.assertRaises(hub.HubError) as cm:
-                hub.upload(self.store, RID)
-            self.assertEqual(cm.exception.kind, "closed")
-        with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: "a/test"}):
+        self.assertTrue(hub.texts_ready(), "the texts are reviewed (2026-10-16)")
+        with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "", hub.DATASET_ENV: ""}):
             self.assertTrue(hub.upload_allowed(hub.dataset_id()))
-        with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "", hub.DATASET_ENV: "a/test"}):
-            self.assertFalse(hub.upload_allowed(hub.dataset_id()))
+        # While a text is a draft: never the real dataset, a rehearsal only with ALLOW_ENV.
+        with mock.patch.object(hub, "texts_ready", return_value=False):
+            with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: ""}):
+                self.assertFalse(hub.upload_allowed(hub.dataset_id()), "never the real dataset before the review")
+                with self.assertRaises(hub.HubError) as cm:
+                    hub.upload(self.store, RID)
+                self.assertEqual(cm.exception.kind, "closed")
+            with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: "a/test"}):
+                self.assertTrue(hub.upload_allowed(hub.dataset_id()))
+            with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "", hub.DATASET_ENV: "a/test"}):
+                self.assertFalse(hub.upload_allowed(hub.dataset_id()))
+
+    def test_draft_consent_blocks(self):
+        # An export made under a draft consent text never goes to the real dataset.
+        prof = os.path.join(self.store.base, "profile.json")
+        with open(prof) as f:
+            p = json.load(f)
+        p["consent"]["version"] = "2026-10-14-draft"
+        with open(prof, "w") as f:
+            json.dump(p, f)
+        shutil.rmtree(self.store.export_dir(RID))
+        self.store.export(RID, low_priority=False)
+        with mock.patch.dict(os.environ, {hub.DATASET_ENV: ""}):
+            with self.assertRaises(hub.HubError) as cm:
+                hub.upload(self.store, RID, dry_run=True)
+        self.assertEqual(cm.exception.kind, "draft-consent")
 
     def test_invalid_blocks(self):
         with open(os.path.join(self.store.export_dir(RID), "health.json"), "w") as f:

gaze/rec/tests/test_session.py

@@ -69,6 +69,21 @@ class RoundTest(unittest.TestCase):
         self.bad([{"id": "a", "kind": "dots", "group": "g", "dots": [[0, 0]]}, {"id": "b", "kind": "dots", "dots": [[0, 0]]},
                   {"id": "c", "kind": "dots", "group": "g", "dots": [[0, 0]]}], "comes back")
         self.bad([{"id": "a", "kind": "dots", "group": "dots", "dots": [[0, 0]]}], "opener")
+        self.bad([{"id": "a", "kind": "prompt", "cues": [["Blink", 1, "honk"]]}], "sound: blink")
+        self.bad([{"id": "a", "kind": "prompt", "cues": [["Blink", 1, "blink", 2]]}], "cues want")
+        # The eyes-shut rules (rounds/README.md)
+        self.bad([{"id": "a", "kind": "prompt", "cues": [["Blink", 1]]}], 'needs the sound "blink"')
+        self.bad([{"id": "a", "kind": "prompt", "cues": [["Close both eyes", 2], ["Open", 2]]}], 'needs the sound "close"')
+        self.bad([{"id": "a", "kind": "prompt", "cues": [["Shut your eyes", 2, "close"]]}], 'comes a cue with the sound "open"')
+        self.bad([{"id": "a", "kind": "prompt", "cues": [["Close your eyes", 2, "close"], ["Open", 2]]}], '"open"')
+        for word in ("Close your left eye", "Half closed", "Squint", "Open"):
+            session.load_round(write_round(self.tmp, [{"id": "a", "kind": "prompt", "cues": [[word, 1]]}]))
+
+    def test_sound_cues(self):
+        r = session.load_round(write_round(self.tmp, [{"id": "a", "kind": "prompt",
+                                                       "cues": [["Look", 1], ["Blink", 1.5, "blink"]]}]))
+        self.assertEqual(r["steps"][0]["cues"], [["Look", 1.0], ["Blink", 1.5, "blink"]])
+        self.assertEqual(session.step_seconds(r["steps"][0]), 2.5)
 
     def test_grid_and_defaults(self):
         r = session.load_round(write_round(self.tmp, [{"id": "a", "kind": "dots", "grid": [3, 2, 10, 5], "ready": False},
@@ -81,18 +96,17 @@ class RoundTest(unittest.TestCase):
 
     def test_plan(self):
         rnd = session.load_round(os.path.join(session.ROUNDS_DIR, "jumps.json"))
-        a = session.build_plan(rnd, 7, ipd_sweep=True)
-        self.assertEqual([s["group"] for s in a[:3]], ["ipd", "dots", "jumps"])
-        nine = a[1]["dots"]
+        a = session.build_plan(rnd, 7)
+        self.assertEqual([s["group"] for s in a[:2]], ["dots", "jumps"])
+        nine = a[0]["dots"]
         self.assertEqual((nine[0], nine[-1], len(nine)), ([0.0, 0.0], [0.0, 0.0], 10))
         self.assertEqual(sorted(map(tuple, nine[1:-1])), sorted(map(tuple, (d for d in session.grid(3, 3, 20, 20) if d != [0, 0]))))
-        jumps = a[2]
+        jumps = a[1]
         self.assertEqual(len(jumps["dots"]), 40)
         self.assertTrue(all(0.5 <= t <= 1.5 for t in jumps["times"]))
         self.assertTrue(all(not session.off_panel(*d) for d in jumps["dots"]))
-        self.assertEqual(a, session.build_plan(rnd, 7, ipd_sweep=True), "the same seed, the same round")
-        self.assertNotEqual(a[1]["dots"], session.build_plan(rnd, 8, ipd_sweep=True)[1]["dots"])
-        self.assertEqual(session.build_plan(rnd, 7, ipd_sweep=False)[0]["group"], "dots")
+        self.assertEqual(a, session.build_plan(rnd, 7), "the same seed, the same round")
+        self.assertNotEqual(a[0]["dots"], session.build_plan(rnd, 8)[0]["dots"])
         self.assertGreater(session.plan_seconds(a), 60)
 
 
@@ -171,6 +185,7 @@ class DryRunTest(unittest.TestCase):
         with open(path, "w") as f:
             json.dump(profile or {}, f)
         states = []
+        kw.setdefault("fit_coach", True)   # the coach's own tests; test_fitment_coach_off has it off
         s = session.Session(self.tmp, script, profile=profile, dry_run=True, speed=kw.pop("speed", 40),
                             next_after=kw.pop("next_after", 0.25), profile_path=path, button=False,
                             on_status=lambda st: states.append(dict(st)), **kw)
@@ -198,8 +213,11 @@ class DryRunTest(unittest.TestCase):
         s, states, profile = self.run_round("depth")
         self.assertEqual(s.state, "done")
         sj, fit = self.files(s)
-        self.assertEqual(sj["takes"], ["01-ipd", "02-dots", "03-near", "04-mid", "05-far", "06-near-far"])
-        self.assertEqual((sj["status"], sj["fit"], sj["ipd_sweep"], sj["ipd_start"]), ("done", "good", True, 0.063))
+        # No IPD step: the setting is read and shown at the fitment.
+        self.assertEqual(sj["takes"], ["01-dots", "02-near", "03-mid", "04-far", "05-near-far"])
+        self.assertEqual((sj["status"], sj["fit"], sj["ipd_start"]), ("done", "good", 0.063))
+        self.assertNotIn("ipd_sweep", sj)
+        self.assertIn(session.IPD_TEXT.format(mm=63.0), "\n".join(self.out))
         self.assertEqual(fit["fitment"]["result"], "good")
         self.assertIsNotNone(fit["baseline"])
         self.assertEqual(profile["bounties"]["depth"]["rounds"], 1)
@@ -207,16 +225,29 @@ class DryRunTest(unittest.TestCase):
         for take in sj["takes"]:
             with open(os.path.join(s.session_dir, "takes", take, "take.json")) as f:
                 self.assertEqual(json.load(f)["status"], "complete", take)
-        events = [e["event"] for e in self.prompts(s, "03-near")]
+        events = [e["event"] for e in self.prompts(s, "02-near")]
         self.assertEqual(events[:4], ["take", "ready", "rec", "step"])
         self.assertEqual(events[-3:], ["wait", "rec", "end"])
         self.assertIn("ft_eyerecpanel: distance 0.500", self.out)
         self.assertIn("ft_eyerecpanel: distance 5.000", self.out)
-        # The IPD is the same next time: no sweep.
         s2, _, profile = self.run_round("depth", profile=profile)
         self.assertEqual(self.files(s2)[0]["takes"][0], "01-dots")
         self.assertEqual(profile["bounties"]["depth"]["rounds"], 2)
 
+    def test_blink_beeps(self):
+        # The blinks round's beeps: each in prompts.jsonl's cue line, played as the cue starts.
+        s, _, _ = self.run_round("blinks", profile={"eyes_ipd": 0.063})
+        self.assertEqual(s.state, "done")
+        played = [line[len("sound: "):] for line in self.out if line.startswith("sound: ")]
+        self.assertEqual(played, ["blink"] * 8 + ["close", "open", "close", "open"])
+        sj, _ = self.files(s)
+        [take] = [t for t in sj["takes"] if t.endswith("blink-cue")]
+        cues = [e for e in self.prompts(s, take) if e["event"] == "cue"]
+        self.assertEqual([(e["word"], e.get("sound")) for e in cues], [("Blink", "blink")] * 8)
+        [take] = [t for t in sj["takes"] if t.endswith("closed")]
+        cues = [e.get("sound") for e in self.prompts(s, take) if e["event"] == "cue"]
+        self.assertEqual(cues, [None, "close", "open", "close", "open"])
+
     def test_fitment_warns_then_skips(self):
         # Next before the fit has been good for FIT_GOOD_S: a warning, then the round goes on.
         s, states, _ = self.run_round("reading", next_after=0.01, speed=20)
@@ -225,6 +256,21 @@ class DryRunTest(unittest.TestCase):
         self.assertTrue(any(st["note"] == session.FIT_SKIP_NOTE for st in states))
         self.assertIsNone(fit["baseline"])
 
+    def test_fitment_coach_off(self):
+        # The default for now: no coaching or warning, the first Next goes on, and the coach's
+        # readings are still kept; no baseline, so no slip checks.
+        self.assertFalse(session.FIT_COACH)
+        s, states, _ = self.run_round("reading", next_after=0.6, speed=20, fit_coach=False)
+        sj, fit = self.files(s)
+        self.assertEqual(s.state, "done")
+        self.assertEqual((sj["fit"], fit["fitment"]["result"]), ("off", "off"))
+        self.assertTrue(fit["readings"])
+        self.assertIsNone(fit["baseline"])
+        self.assertFalse(fit["slips"])
+        self.assertFalse(any(st["note"] for st in states))
+        self.assertFalse(any(st["fit_words"] or st["fit_card"] for st in states))
+        self.assertTrue(all(st["fit_coach"] is False for st in states))
+
     def test_skip_redo_stop(self):
         def steer(s, states):
             def wait_for(cond):

gaze/rec/tests/test_sounds.py

@@ -0,0 +1,52 @@
+#!/usr/bin/env python3
+"""sounds.py: the beeps' WAV files, and a player that never stops a round."""
+import os
+import shutil
+import sys
+import tempfile
+import unittest
+import wave
+from unittest import mock
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.join(HERE, ".."))
+import sounds  # noqa: E402
+
+
+class SoundsTest(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.mkdtemp(prefix="eyerec-sounds-")
+        self.env = mock.patch.dict(os.environ, {"XDG_RUNTIME_DIR": self.tmp})
+        self.env.start()
+
+    def tearDown(self):
+        self.env.stop()
+        shutil.rmtree(self.tmp, ignore_errors=True)
+
+    def test_wavs(self):
+        self.assertEqual(set(sounds.SOUNDS), {"blink", "close", "open"})
+        self.assertEqual(sounds.SOUNDS["close"], sounds.SOUNDS["open"], "open is the same beep again")
+        for name, (hz, secs) in sounds.SOUNDS.items():
+            path = sounds.wav_path(name)
+            self.assertTrue(path.startswith(self.tmp))
+            with wave.open(path) as w:
+                self.assertEqual((w.getnchannels(), w.getsampwidth(), w.getframerate()), (1, 2, sounds.RATE))
+                self.assertEqual(w.getnframes(), int(sounds.RATE * secs))
+                frames = w.readframes(w.getnframes())
+            first, last = int.from_bytes(frames[:2], "little", signed=True), int.from_bytes(frames[-2:], "little", signed=True)
+            self.assertEqual((first, last), (0, 0), "faded in and out: no click")
+
+    def test_player_off_and_missing(self):
+        logged = []
+        self.assertFalse(sounds.Player(on=False, log=logged.append).play("blink"))
+        self.assertEqual(logged, [])
+        with mock.patch.object(sounds.shutil, "which", return_value=None):
+            p = sounds.Player(log=logged.append)
+            self.assertFalse(p.play("close"))
+        self.assertEqual(logged, ["no pw-play: the sound cues are off"])
+        with self.assertRaises(ValueError):
+            p.play("honk")
+
+
+if __name__ == "__main__":
+    unittest.main()

gaze/tracker/build.sh

@@ -21,6 +21,12 @@ if [ "${FRAME_IN_BOX:-0}" = 1 ] && [ -x /opt/frametop/venv/bin/python ]; then
   python3 -m venv --without-pip build/venv
   /opt/frametop/venv/bin/python -c "import sysconfig; print(sysconfig.get_path(\"purelib\"))" \
     >"$(build/venv/bin/python -c "import sysconfig; print(sysconfig.get_path(\"purelib\"))")/frametop-image.pth"
+  # The eye net (eyes_net.py) needs ncnn, which the image does not have yet: add it here. It needs
+  # the network; without it ft-eyes uses the pupil finder.
+  if ! build/venv/bin/python -c "import ncnn" 2>/dev/null; then
+    { build/venv/bin/python -m ensurepip >/dev/null && build/venv/bin/python -m pip install -q --disable-pip-version-check \
+        --no-deps "$(grep -E "^ncnn==" requirements.txt)"; } || echo "build/venv: no ncnn (the eye net needs it)" >&2
+  fi
 elif ! cmp -s requirements.txt build/venv/requirements.done; then
   rm -rf build/venv
   python3 -m venv build/venv

gaze/tracker/eyes_net.py

@@ -0,0 +1,56 @@
+"""The eye net: one eye camera's frame in, that eye's raw gaze out (frame-eyes' train/gaze).
+
+A model folder holds model.json and the ncnn files it names:
+  {"name": "eyes-v0", "variant": "s100", "input": [100, 128], "gaze_scale": 30, "out": 6,
+   "param": "model.ncnn.param", "bin": "model.ncnn.bin", "in": "in0", "outname": "out0", ...}
+The net's output holds both cameras' heads (OUT numbers each, camera 0's first): gaze yaw and
+pitch / gaze_scale, pupil x / 512 and y / 400, eye open (logit), pupil visible (logit).
+
+The gaze is raw: each person's own offset is left to the calibration (eyes_model.Calibration,
+with this gaze where the pupil centre would be). So a calibration belongs to one model: ft-eyes
+keeps each model's in its own folder.
+
+ncnn fp16 on one thread: 1.4 ms a frame for s100 on an A720 core (frame-eyes notes/e0-runtime.md).
+"""
+import json
+import math
+import os
+
+import numpy as np
+
+W, H = 512, 400
+
+
+class Net:
+    def __init__(self, folder):
+        import cv2
+        import ncnn
+        self.cv2 = cv2
+        self.folder = folder
+        with open(os.path.join(folder, "model.json")) as f:
+            self.info = json.load(f)
+        self.name = str(self.info["name"])
+        self.in_h, self.in_w = self.info["input"]
+        self.scale = float(self.info.get("gaze_scale", 30.0))
+        self.out = int(self.info.get("out", 6))
+        self.in_name, self.out_name = self.info.get("in", "in0"), self.info.get("outname", "out0")
+        net = ncnn.Net()
+        net.opt.num_threads = 1
+        net.opt.use_vulkan_compute = False
+        net.opt.use_fp16_storage = net.opt.use_fp16_arithmetic = net.opt.use_fp16_packed = True
+        for key, load in (("param", net.load_param), ("bin", net.load_model)):
+            if load(os.path.join(folder, self.info[key])) != 0:
+                raise RuntimeError(f"{folder}: can't load {self.info[key]}")
+        self.ncnn, self.net = ncnn, net
+
+    def __call__(self, img, cam):
+        """img: one camera's 400x512 uint8 frame -> dict(gaze=(yaw, pitch), x, y, open, visible),
+        open and visible as probabilities."""
+        x = self.cv2.resize(img, (self.in_w, self.in_h), interpolation=self.cv2.INTER_AREA)
+        x = x.astype(np.float32)[None] * (1.0 / 255.0)
+        ex = self.net.create_extractor()
+        ex.input(self.in_name, self.ncnn.Mat(x))
+        o = np.array(ex.extract(self.out_name)[1]).reshape(-1)[cam * self.out:(cam + 1) * self.out]
+        sig = lambda v: 1.0 / (1.0 + math.exp(-float(v)))  # noqa: E731
+        return {"gaze": (float(o[0]) * self.scale, float(o[1]) * self.scale), "x": float(o[2]) * W,
+                "y": float(o[3]) * H, "open": sig(o[4]), "visible": sig(o[5])}

gaze/tracker/ft-eyes

@@ -38,10 +38,23 @@ Control socket: abstract datagram "@ft_eyes"; each command gets one reply line.
                                append every frame looked at to PATH as JSON lines (the eye
                                recorder's own.jsonl): {"t", "cam", "x", "y", "a", "b", "angle",
                                "glints": [[x, y], ...], "gaze": [yaw, pitch] once calibrated},
-                               or {"t", "cam", "lost": true}. "finds stop" replies "ok ROWS"
+                               or {"t", "cam", "lost": true}; with the net also "net": {"gaze",
+                               "x", "y", "open", "visible"}. "finds stop" replies "ok ROWS"
+
+The eye net (eyes_net.py) instead of the pupil finder: EYES_MODEL in ~/.config/frametop.conf (or
+FT_EYES_MODEL), picked on Input Settings' Gaze page: latest, a version's name (gaze/tracker/models.json,
+downloaded into gaze/tracker/models/NAME by models/catalog.py), or a model folder from frame-eyes'
+training (anything with a "/"); classic or unset is the pupil finder. Each eye's raw gaze from the net
+then stands where the pupil centre would: the calibration, the shifts clicks teach and the dots'
+spread are in its degrees, and there are no glints (no slip estimate or jump notice). Each model
+has its own state folder, net-NAME/ (calibration.json, state.json, clicks.jsonl), so switching
+asks for a calibration once per model and the classic one is kept. Read at start: the gaze
+service restarts ft-eyes when EYES_MODEL changes. If the model can't load, ft-eyes says
+so and uses the pupil finder. While the fit coach asks ("fit") or finds are logged, the pupil
+finder runs too, for them.
 
 Environment, for replays (lab/ft-eyes-e2e): FT_EYES_CAMS, FT_EYES_GAZE, FT_EYES_STATE (the
-state folder), FT_EYES_SOCKET (the socket's name). With FT_EYES_CAMS set, the want file isn't
+state folder), FT_EYES_SOCKET (the socket's name), FT_EYES_MODEL. With FT_EYES_CAMS set, the want file isn't
 touched.
 
 /dev/shm/frametop-eyes-gaze, 128 bytes, little-endian (mirrored in ft-gaze.cpp):
@@ -81,11 +94,40 @@ import eyes_model  # noqa: E402
 import eyes_pupil  # noqa: E402
 import fitcoach  # noqa: E402
 
+def load_net():
+    """The eye net EYES_MODEL picks (FT_EYES_MODEL wins), or None (the pupil finder). models/catalog.py
+    resolves it: classic (or unset: the catalog's default), latest (the newest version, once downloaded),
+    a downloaded version's name, or a folder of one's own (anything with a "/")."""
+    try:
+        sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "models"))
+        import catalog
+        r = catalog.resolve("eyes")
+    except Exception as e:  # noqa: BLE001 (a broken catalog mustn't stop the tracker)
+        print(f"ft-eyes: the model catalog didn't load ({type(e).__name__}: {e}); using the pupil finder",
+              file=sys.stderr, flush=True)
+        return None
+    if not r["folder"]:
+        if r["setting"] not in ("classic", "steamvr"):
+            print(f"ft-eyes: EYES_MODEL={r['setting']}: {r['why']}; using the pupil finder", file=sys.stderr, flush=True)
+        return None
+    try:
+        import eyes_net
+        return eyes_net.Net(r["folder"])
+    except Exception as e:  # noqa: BLE001 (any failure: say so and track with the pupil finder)
+        print(f"ft-eyes: the eye net in {r['folder']} didn't load ({type(e).__name__}: {e}); using the pupil finder",
+              file=sys.stderr, flush=True)
+        return None
+
+
+NET = load_net()
+
 CAMS = os.environ.get("FT_EYES_CAMS", "/dev/shm/frametop-eyes-cams")  # overrides for replays
 OUT = os.environ.get("FT_EYES_GAZE", "/dev/shm/frametop-eyes-gaze")
 WANT = None if "FT_EYES_CAMS" in os.environ else Path("/dev/shm/frametop-eyes-want")
 OUT_SIZE = 128
-CALIBRATION = Path(os.environ.get("FT_EYES_STATE", Path.home() / ".local/state/frametop/gaze/eyes")) / "calibration.json"
+STATE_BASE = Path(os.environ.get("FT_EYES_STATE", Path.home() / ".local/state/frametop/gaze/eyes"))
+FEATURE = "net:" + NET.name if NET else "pupil"   # what the calibration maps: each model has its own
+CALIBRATION = (STATE_BASE / ("net-" + NET.name) if NET else STATE_BASE) / "calibration.json"
 W, H = 512, 400
 FRESH = 0.03          # an eye's reading counts toward the output for this long (s)
 LOST_EVERY = 3        # while an eye is lost, search the whole frame only every 3rd frame
@@ -98,7 +140,8 @@ HISTORY = 12.0        # seconds of pupil positions kept per eye, for dots and cl
 GAP = 3.0             # s without frames: the headset was off, and may sit differently now
 CLICK_BEFORE = 0.3    # a click's frames: the 300 ms before it (like the probe's fixation)
 CALIB_MIN = 15        # frames an eye needs in a calibration dot's window
-CALIB_SPREAD = 4.0    # px: more than this and the eye moved during the dot
+CALIB_SPREAD = 2.5 if NET else 4.0   # px (the net: degrees): more than this and the eye moved during the dot
+CLASSIC_FOR = 3.0     # s: with the net, the pupil finder runs this long after the fit coach asks
 # Waiting for frames. They come only as a counter in shared memory, so there's nothing to block
 # on: sleep until a camera's next frame is due, then look every POLL. ft-eyegrab passes each one
 # on within a ms or two of when its camera starts the one after, so they arrive 11.1 ms apart,
@@ -187,6 +230,7 @@ class Eye:
         self.obs = None         # the fit coach's window (fitcoach.Observations), shared by both eyes
         self.fed = None         # (frame time, CLOCK_MONOTONIC_RAW then) of the newest frame fed
         self.finds = None       # the finds log (Tracker.finds), shared by both eyes
+        self.coach = [0.0]      # with the net: until when the fit coach wants pupils (monotonic), shared
 
     def use(self, cal, shift=None):
         self.cal = cal if cal is not None and cal.has("pupil", self.eye) else None
@@ -199,6 +243,8 @@ class Eye:
         if self.last_t is not None and t - self.last_t > GAP:
             self.shift.reseat()
         self.last_t = t
+        if NET:
+            return self.feed_net(t, img)
         if self.last is None:
             self.lost += 1
             if self.lost % LOST_EVERY:
@@ -227,6 +273,31 @@ class Eye:
             self.finds.write(self, t, p)
         self.work += time.perf_counter() - t0
 
+    def feed_net(self, t, img):
+        """The net's raw gaze is the feature: history, calibration and shifts as for a pupil."""
+        t0 = time.perf_counter()
+        r = NET(img, self.eye)
+        classic = self.finds is not None or time.monotonic() < self.coach[0]
+        p = None
+        if classic:
+            p = eyes_pupil.find_pupil(img, self.last)
+            self.last = p
+            if self.obs is not None:
+                self.obs.add(self.eye, t, p)
+                self.fed = (t, time.clock_gettime(time.CLOCK_MONOTONIC_RAW))
+        if r["visible"] >= 0.5:
+            self.found += 1
+            fx, fy = r["gaze"]
+            self.history.append((t, fx, fy, math.nan, math.nan))
+            while self.history and self.history[0][0] < t - HISTORY:
+                self.history.popleft()
+            if self.cal:
+                g = self.cal.gaze(self.eye, fx, fy, self.shift.value)
+                self.gaze = (t, float(g[0]), float(g[1]), r["x"], r["y"])
+        if self.finds is not None:
+            self.finds.write(self, t, p, r)
+        self.work += time.perf_counter() - t0
+
     def window(self, t0, t1):
         """Median pupil and glint midpoint over [t0, t1], the frame count, and the spread."""
         rows = np.array([r for r in self.history if t0 <= r[0] <= t1]).reshape(-1, 5)
@@ -246,13 +317,18 @@ class Finds:
         self.f = open(path, "a", buffering=1 << 16)
         self.rows = 0
 
-    def write(self, eye, t, p):
+    def write(self, eye, t, p, net=None):
         if p is None:
             row = {"t": round(t, 6), "cam": eye.eye, "lost": True}
         else:
             row = {"t": round(t, 6), "cam": eye.eye, "x": round(float(p["x"]), 2), "y": round(float(p["y"]), 2),
                    "a": round(float(p["a"]), 2), "b": round(float(p["b"]), 2), "angle": round(float(p["angle"]), 1),
                    "glints": [[round(float(x), 1), round(float(y), 1)] for x, y in p.get("glints") or []]}
+            if eye.gaze and eye.gaze[0] == t and not NET:
+                row["gaze"] = [round(eye.gaze[1], 3), round(eye.gaze[2], 3)]
+        if net is not None:
+            row["net"] = {"gaze": [round(v, 3) for v in net["gaze"]], "x": round(net["x"], 1), "y": round(net["y"], 1),
+                          "open": round(net["open"], 3), "visible": round(net["visible"], 3)}
             if eye.gaze and eye.gaze[0] == t:
                 row["gaze"] = [round(eye.gaze[1], 3), round(eye.gaze[2], 3)]
         self.f.write(json.dumps(row) + "\n")
@@ -270,10 +346,16 @@ class Tracker:
         self.dots = []  # calibration dots so far, in ft-eyes-score's click form
         self.calibrating = False
         self.fit = fitcoach.Observations()
+        coach = [0.0]
         for e in self.eyes:
-            e.obs = self.fit
+            e.obs, e.coach = self.fit, coach
         if CALIBRATION.exists():
             self.cal = eyes_model.Calibration.load(CALIBRATION)
+            if self.cal.info.get("feature", "pupil") != FEATURE:
+                print(f"ft-eyes: {CALIBRATION} is for {self.cal.info.get('feature', 'pupil')}, not {FEATURE}:"
+                      " calibrate again", file=sys.stderr, flush=True)
+                self.cal = None
+        if self.cal:
             if not self.cal.spread:
                 self.cal.spread = spread_from_dots(self.cal)
         shifts = {}
@@ -319,6 +401,7 @@ class Tracker:
                 return "ok"
             return "fail finds start PATH | finds stop"
         if w[0] == "fit":
+            self.eyes[0].coach[0] = time.monotonic() + CLASSIC_FOR
             # In frame time, which a replay's frames keep from the recording: the newest frame's,
             # plus how long ago it came (frames stopping, the headset off, empty the window).
             fed = [e.fed for e in self.eyes if e.fed]
@@ -345,7 +428,8 @@ class Tracker:
             if len(self.dots) < eyes_model.MIN_CLICKS:
                 return f"fail only {len(self.dots)} dots (need {eyes_model.MIN_CLICKS})"
             cal = eyes_model.Calibration.fit(self.dots, {"made": time.strftime("%Y-%m-%d %H:%M:%S"),
-                                                       "dots": len(self.dots), "from": "probe calibration"})
+                                                       "dots": len(self.dots), "from": "probe calibration",
+                                                       "feature": FEATURE})
             if not all(cal.has(n, c) for n in ("pupil", "where") for c in (0, 1)):
                 return "fail not enough dots with both eyes"
             errs = [float(np.hypot(*(np.mean([cal.gaze(c, *k["eye"][c]["pupil"]) for c in (0, 1)], axis=0)
@@ -389,6 +473,7 @@ class Tracker:
 
     def status(self):
         return {"calibration": self.cal.info if self.cal else None, "calibrating": self.calibrating,
+                "feature": FEATURE, "model": NET.folder if NET else None,
                 "dots": len(self.dots),
                 "eyes": {EYES[e.eye]: {"shift": e.shift.value.tolist(), "clicks": len(e.shift.meas),
                                        "jump": bool(np.any(e.shift.jump)), "jump_px": e.shift.jump.tolist(),
@@ -493,7 +578,8 @@ def main():
     sock.bind(SOCKET)
     sock.setblocking(False)
     cal = tracker.cal
-    print("ft-eyes: " + (f"calibration from {cal.info.get('made')} ({cal.info.get('from', cal.info.get('capture'))})"
+    print("ft-eyes: " + (f"the eye net {NET.name} ({NET.folder}); " if NET else "")
+          + (f"calibration from {cal.info.get('made')} ({cal.info.get('from', cal.info.get('capture'))})"
                            if cal else "not calibrated: run the probe's calibration with the Own tracker")
           + f"; waiting for {CAMS}", file=sys.stderr, flush=True)
     cams = wait_for_cams(sock, tracker, want)

gaze/tracker/lab/ft-eyes-replay

@@ -2,10 +2,12 @@
 """ft-eyes-replay: play a recording into the shared-frame layout, as ft-eyegrab --share
 would, at the recorded pace, to test ft-eyes without the headset.
 
-Usage: lab/py lab/ft-eyes-replay NAME [PATH] [--from S] [--for S]
+Usage: lab/py lab/ft-eyes-replay NAME [PATH] [--from S] [--for S] [--now]
 NAME is a recording (a bare name is in eyes_lab.CAPTURES). PATH defaults to
 /dev/shm/frametop-eyes-cams-replay; run ft-eyes with FT_EYES_CAMS=PATH (and FT_EYES_GAZE=...
-so it doesn't overwrite the live output).
+so it doesn't overwrite the live output). Frames keep the recording's times, unless --now: then
+they're stamped with today's CLOCK_MONOTONIC_RAW as they're written, as ft-eyegrab's are (for
+tools that line them up with something live, such as the eye recorder's dots).
 """
 import mmap
 import os
@@ -42,6 +44,7 @@ def main(argv):
     count = [0, 0]
     t0 = idx[0, 3] + start
     wall0 = time.monotonic()
+    shift = time.clock_gettime(time.CLOCK_MONOTONIC_RAW) - t0 if "--now" in argv else 0.0
     try:
         for i in range(len(frames)):
             t = idx[i, 3]
@@ -58,7 +61,7 @@ def main(argv):
             seq = struct.unpack_from("<Q", mm, off)[0]
             struct.pack_into("<Q", mm, off, seq + 1)
             mm[off + 64:off + 64 + W * H] = frames[i].tobytes()
-            struct.pack_into("<dQII", mm, off + 8, t, n, cam, int(idx[i, 1]))
+            struct.pack_into("<dQII", mm, off + 8, t + shift, n, cam, int(idx[i, 1]))
             struct.pack_into("<Q", mm, off, seq + 2)
             count[cam] = n + 1
             struct.pack_into("<Q", mm, 24 + 8 * cam, n + 1)

gaze/tracker/models.json

@@ -0,0 +1,59 @@
+{
+ "catalog": 1,
+ "kind": "eyes",
+ "latest": "eyes-v0",
+ "default": "classic",
+ "baseline": "steamvr",
+ "metrics": [
+  {"key": "fresh", "label": "Right after calibrating, median", "unit": "°", "better": "lower",
+   "note": "One person, a later recording after taking the headset off and on again, 124 click targets."},
+  {"key": "fresh_p90", "label": "Right after calibrating, 90% of looks within", "unit": "°", "better": "lower"},
+  {"key": "reseat_clicks", "label": "After a re-seat, old calibration + click corrections, median", "unit": "°", "better": "lower",
+   "note": "The headset was taken off and put back on; the calibration is the one from before, and the pointer's clicks correct it."},
+  {"key": "reseat_clicks_p90", "label": "After a re-seat, with click corrections, 90% within", "unit": "°", "better": "lower"},
+  {"key": "reseat", "label": "After a re-seat, old calibration alone, median", "unit": "°", "better": "lower"}
+ ],
+ "models": [
+  {
+   "name": "classic",
+   "title": "Frametop classic (pupil and glints)",
+   "kind": "classic",
+   "status": "builtin",
+   "summary": "Our hand-written tracker: finds each pupil and its glints. Best once your clicks have corrected it.",
+   "notes": "No learned model: it finds the pupil and the two glints in each eye camera's frame and maps them with your calibration. The glints show when the headset slips, so the pointer's clicks correct it quickly after a re-seat. Right after a calibration it's less precise than SteamVR's.",
+   "cost": "CPU, in Frametop's container, while the gaze is in use. Not yet measured side by side with the nets.",
+   "bench": {"fresh": 2.83, "fresh_p90": 4.71, "reseat_clicks": 1.25, "reseat_clicks_p90": 3.35, "reseat": 14.03}
+  },
+  {
+   "name": "steamvr",
+   "title": "SteamVR (Valve's tracker)",
+   "kind": "valve",
+   "status": "builtin",
+   "summary": "Valve's own eye tracking, as SteamVR reports it, with Frametop's calibration on top.",
+   "notes": "The most precise right after calibrating, and it needs nothing installed. It drifts when the headset moves, and our click corrections help it less than the classic tracker.",
+   "cost": "None for Frametop: Valve's tracker runs on the headset's own processors.",
+   "bench": {"fresh": 1.18, "fresh_p90": 2.32, "reseat_clicks": 3.11, "reseat_clicks_p90": 5.43, "reseat": 3.73}
+  },
+  {
+   "name": "eyes-v0",
+   "title": "Frametop eye net v0",
+   "kind": "custom",
+   "status": "testing",
+   "date": "2026-10-16",
+   "summary": "Experimental: our first deep-learning eye net. Better than classic right after calibrating, worse after the headset moves.",
+   "notes": "A small network that reads each eye camera's frame and gives a raw gaze, which your calibration fits (each net keeps its own calibration, so calibrate once after switching).\n\nTrained on one person in one headset seating, so it doesn't carry over well when the headset sits differently: after a re-seat, the classic tracker with click corrections is more than twice as precise. Try it to help test the pipeline; later versions trained on more people should do better.",
+   "cost": "CPU, while the gaze is in use: 1.4 ms per pair of eye frames in a benchmark (12% of one big core at 90 frames a second), about 1.2 ms per frame inside ft-eyes.",
+   "bench": {"fresh": 1.61, "fresh_p90": 3.23, "reseat_clicks": 2.87, "reseat_clicks_p90": 6.41, "reseat": 4.0},
+   "format": 1,
+   "rev": "5f4341cc53d698f622368c22bf4bae1addbe20a0",
+   "files": {
+    "NOTICE": "2894eeb7d9269890ea54a5546b3fd7ee1c7e4cba0d0c1c64f8207dab68de7acf",
+    "model.json": "9d7e252137348148597b1a951066053868d4504229520459bb31e95795cd2101",
+    "model.ncnn.bin": "230cac3489333257ad99ad1ce5ce6fa76fe60a80f4fa92fb8bc78c7f50a9a44a",
+    "model.ncnn.param": "ec82393354ff522bbd17be57d96a68ebdd1337915f48dfa86ac989f4756f984b",
+    "LICENSE": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
+    "README.md": "2625af9abfe4bd350c268f39c0f7ee315cd40f83ada2f93253a0abf68c2f78c2"
+   }
+  }
+ ]
+}

gaze/tracker/requirements.txt

@@ -3,3 +3,6 @@
 # VTK, GDAL, and over a gigabyte of map data; these wheels are about 165 MB.
 numpy==2.5.3
 opencv-python-headless==5.0.0.93
+# The eye net (eyes_net.py, EYES_MODEL): ncnn's own wheel. Not in the image yet: build.sh adds
+# it to build/venv there from this line.
+ncnn==1.0.20260526

hand-recorder.md

@@ -4,64 +4,80 @@ title: Install the Frametop Hand Recorder
 
 # Install the Frametop Hand Recorder
 
-The Hand Recorder records your hands with the Steam Frame's tracking cameras for Frametop's open hand dataset ([DeeJanuz/frametop-hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) on Hugging Face). These are the Konsole commands to install it.
+The Hand Recorder records your hands with the Steam Frame's tracking cameras for Frametop's open hand dataset ([DeeJanuz/frametop-hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) on Hugging Face), which trains Frametop's hand tracking.
 
-> **Fixed in Frametop 0.2.1 (October 5, 2026):** the recorder now works on headsets without the Arcturus color passthrough module too. If you installed it earlier and the camera check stopped with "Not all of the headset's tracking cameras are running (ft-camd publishes only 2 of 4 mono cameras ...)", run the commands under [Update](#update).
+You pick a **bounty**: a short round, about 4 minutes, of what the hand tracker needs most right now. Windows appear in front of you in the headset, and you poke, tap, pinch and grab them as you would real ones while the cameras record your hands. Afterwards you watch what was recorded, delete anything you don't want to share, and upload it from your own Hugging Face account. Nothing leaves the headset until you press Upload.
+
+With Frametop's hand tracking installed there's also a **testing session**: up to 3 minutes of using Frametop as you always do. When the tracking gets something wrong, press the headset button to mark the moment. New versions of the hand tracking are tested on these recordings before they ship. Testing sessions are never used for training and aren't published: they're encrypted on your headset so that only the maintainer can open them.
 
 Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions and help with recording.
 
-These steps install Frametop first, and the recorder runs inside Frametop's desktop. To record **bounty rounds** (short rounds of taps, pinches and grabs, about 4 minutes each) without installing Frametop, use the standalone [Frametop Hand Recorder](https://github.com/Frametop/frametop-hand-recorder) instead: it runs from the SteamVR dashboard and installs with one command. Bounty rounds go to the same public dataset. In Frametop itself they need the experimental version (`--experimental` instead of `--stable` below).
+There are two ways to install it. Both record the same rounds for the same dataset.
 
-Installing a Frametop release (Frametop.zip, FrameDrop, or `get.sh --release`) instead? Its installer offers the Hand Recorder, built, so there's nothing to compile: leave Hand Recorder ticked (or answer yes) and skip step 2. To add it to a release later, run `~/.local/share/frametop/releases/current/hands/rec/install.sh`.
+- [**Without Frametop**](#without-frametop): the standalone Hand Recorder runs from SteamVR's dashboard. One command installs it.
+- [**With Frametop**](#with-frametop): Frametop's experimental release brings the recorder already built. It runs on Frametop's desktop.
 
 ## Before you start
 
-- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/Frametop/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
+- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/Frametop/frametop/blob/experimental/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first round.
 - You need a Steam Frame on the stable SteamOS release (not the beta), an internet connection, and a keyboard (Bluetooth, or the on-screen one).
-- You need a `sudo` password. If you've never set one, run `passwd` in Konsole first.
-- Recordings are several gigabytes per round, and uploading one needs about the same again free while it runs. `df -h ~` shows your free space.
+- You need a `sudo` password: the camera broker needs it once to read the cameras. If you've never set one, run `passwd` in Konsole first.
+- A round and its export take up to about 30 GB. You can delete both once the round is uploaded. `df -h ~` shows your free space.
 
-## Open Konsole
+## Without Frametop
+
+1. In the headset, open the dashboard, then **Launch a program** → **Konsole**.
+2. Run:
+
+   ```
+   curl -fsSL https://raw.githubusercontent.com/Frametop/frametop-hand-recorder/main/get-hand-recorder.sh | bash
+   ```
+
+   It asks for your `sudo` password once.
+3. Open the dashboard, then **Launch a program** → **Frametop Hand Recorder**.
+
+To update, run the command again: it installs the newest release. To uninstall:
+
+```
+~/.local/share/frametop-hand-recorder/current/get-hand-recorder.sh --uninstall
+```
+
+The [standalone recorder's README](https://github.com/Frametop/frametop-hand-recorder#readme) says what the installer changes.
+
+## With Frametop
 
 1. In the launcher, choose Launch a program → Desktop.
 2. In the application menu, open System → Konsole.
+3. Run:
 
-## 1. Install Frametop
+   ```
+   curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --release --experimental
+   ```
+
+   This downloads Frametop's newest experimental release (about 1.4 GB) and installs it without compiling anything. The installer asks what to install: say yes to the Hand Recorder, and to hand tracking if you want to record testing sessions. It asks for your `sudo` password for those. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, the same command updates it, and the Hand Recorder with it.
+4. Choose Launch a program → Desktop again, and open Frametop Hand Recorder from the application menu.
+
+Installing `Frametop.zip` with [FrameDrop](https://framedropvr.com), or by hand from the [releases](https://github.com/Frametop/frametop/releases), works too: leave Hand Recorder ticked in the installer's window.
+
+To add the recorder to a release you've already installed:
 
 ```
-curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
+~/.local/share/frametop/releases/current/hands/rec/install.sh
 ```
 
-This clones Frametop into `~/frametop` and runs its installer. The first run downloads 1–2 GB. The installer asks a few questions (gaze mode, the eye tracker, the Bluetooth fixes); the defaults are fine. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, this updates it.
+From a clone of the experimental branch (`get.sh --experimental`, without `--release`), `~/frametop/hands/rec/install.sh` builds it, which takes a few minutes. Frametop's stable version (0.2.2) has an older recorder without bounty rounds.
 
-## 2. Install the Hand Recorder
+## Record
 
-After SteamVR restarts, choose Launch a program → Desktop again, open Konsole, and run:
+Open the recorder. It starts on **Bounties**, with the rounds needed most at the top. Press **Record** on one. The first time, it shows the consent text. Then a short checklist checks the cameras, and the round starts in the headset.
 
-```
-~/frametop/hands/rec/install.sh
-```
+While you record, use the button on the right side of the headset: press it to start each step, press it twice to record a step again, and hold it to stop. In a testing session, a press marks a moment and a hold finishes.
 
-This builds the camera broker, the hand tracker and the headset panel (the first build takes a few minutes), asks for your `sudo` password once to let the camera broker read the cameras, and adds Frametop Hand Recorder to the application menu.
-
-## 3. Record
-
-Open Frametop Hand Recorder from the desktop's application menu. It walks you through the consent text, a short checklist, the recording, a review of what you recorded, and the upload to Hugging Face. Nothing leaves the headset until you press Upload.
-
-## Update
-
-```
-curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
-~/frametop/hands/rec/install.sh
-```
-
-Run the second command after SteamVR has restarted, as in the install.
+After the round, **Export and upload** prepares it and opens the upload page, where you log in to Hugging Face. Your upload arrives as a pull request on the dataset, and the maintainer checks it before it's merged.
 
 ## Uninstall
 
-```
-~/frametop/hands/rec/install.sh uninstall
-```
+For the standalone recorder, see [Without Frametop](#without-frametop). For Frametop's, run its installer with `uninstall`: `~/.local/share/frametop/releases/current/hands/rec/install.sh uninstall` for a release, or `~/frametop/hands/rec/install.sh uninstall` for a clone.
 
 This removes the menu entry. With your `sudo` password, it also takes back the camera broker's permission to read the cameras. If you also installed Frametop's live hand tracking, the camera broker keeps that permission, because live hand tracking still uses it. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/Frametop/frametop#uninstall) in the README.
 

hands/.gitignore

@@ -13,3 +13,5 @@ taps-out/
 tools/taps-*.sh
 # recorded demo clips (hands/rec/demo/camgifs.py) show the room around the recorder: never shipped
 rec/demos-recorded/
+# hand model versions picked in Input Settings (models/catalog.py downloads them; CC BY-NC 4.0)
+models/versions/

hands/README.md

@@ -50,7 +50,8 @@ Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides
 - `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
 - `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
 - `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
-- `HANDS_MODELS` (unset: `hands/models/student`, the fine-tuned models, when `hands/models.sh` fetched them, else `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to try other models without passing options to every launcher. The fine-tuned folders lack the `*-int8` files, so `--int8` without `--models` uses the stock ones.
+- `HANDS_MODEL=latest`: which hand models to run, picked on Input Settings' Hands page with notes and benchmarks: `latest` (`hands/models/student`, kept current by `models.sh`), `stock` (`hands/models/ncnn`, Google's MediaPipe models), or a version from `hands/models.json`, downloaded into `hands/models/versions/NAME` (see `models/README.md`). A version that isn't downloaded falls back to latest.
+- `HANDS_MODELS` (unset: what `HANDS_MODEL` picks; by default `hands/models/student`, the fine-tuned models, when `hands/models.sh` fetched them, else `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to try other models without passing options to every launcher. The fine-tuned folders lack the `*-int8` files, so `--int8` without `--models` uses the stock ones.
 - `HANDS_MODELS_UPDATE=1`: when hand tracking starts, fetch the fine-tuned models GitHub's branch pins if they changed, without a new release (`models.sh` below). `0` keeps the ones it came with.
 - `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
 - `HANDS_SKELETON=1`: fit a skeleton with your bone lengths and publish it (see "Skeleton" below); `0` publishes the filtered landmarks as before. As `--skeleton`.
@@ -227,7 +228,7 @@ ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gest
 
 With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/ft-pointer.cpp`) reads the gestures file every frame. It's off by default.
 
-**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`).
+**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. The ray comes up as the thumb nears the index or middle tip (`POINTER_RAY_READY`). With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), a three-finger pinch (thumb, index and middle together) scrolls as the hand moves, and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`).
 
 - **Pinch to click, in gaze mode.** Your eyes aim and your hands click, Vision Pro style, and the hand ray stays off (`POINTER_GAZE_HAND_RAY`). A pinch locks the pointer where you were looking just before the fingers began to close (75 ms before ft-hands saw the pinch, since the eyes often move on as the hand closes), and a quick tap clicks there. Pinch, hold and move the hand to nudge the pointer: nothing moves until the hand has moved 4 degrees (about 3 cm at arm's length), then the pointer follows at 0.3 times the hand's movement. Letting go clicks where the pointer is, and the correction is a lesson for the gaze tracker, as with the mouse. Held still for `POINTER_GAZE_HOLD` (0.5 s) instead, the pinch is a real press at that point, and the hand drags at `POINTER_GRIP_GAIN`. A middle pinch is a right click the same way. Hold a pinch with one hand and pinch with the other: both buttons are down, left for the first hand and right for the second, each until its own pinch opens (a first pinch not yet pressed becomes a press at once, where the pointer is), as when you hold both of a mouse's buttons. The rules and timings are in `pointer/helper/gazehand.h`.
 - **Without gaze mode,** a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is.

hands/frametop-hands.service

@@ -14,6 +14,8 @@ PartOf=steamvr.service
 # every 15 minutes, never more than a minute, offline or failing it starts with the ones it has).
 ExecStartPre=-/usr/bin/pkill -x ft-hands
 ExecStartPre=-/usr/bin/timeout 60 @REPO@/hands/models.sh auto
+# A version picked by name (HANDS_MODEL, Input Settings' Hands page) that isn't downloaded yet.
+ExecStartPre=-/usr/bin/timeout 60 /usr/bin/python3 @REPO@/models/ft-models auto hands
 ExecStart=@REPO@/scripts/in-box @REPO@/hands/build/ft-hands --status 60
 ExecStopPost=-/usr/bin/pkill -x ft-hands
 Restart=always

hands/include/fh_ray.h

@@ -70,10 +70,12 @@
 //   not near    not near a panel (ft-screens' near      near: at once (near wins)
 //               claim, or the fingertip at an overlay)
 //   not a fist  no grip down                            a grip down
-//   ready       the thumb and index tips within         readyHide (0.10 m) apart: the laser is
-//               readyShow (0.06 m) (user, 2026-10-11:   for pinching, not a hand held up (0: off;
-//               "the aiming laser only activates when   frames without the gap pass)
-//               your fingers get within a distance")
+//   ready       the thumb tip within readyShow (0.06    readyHide (0.10 m) apart: the laser is
+//               m) of the index or the middle tip       for pinching, not a hand held up (0: off;
+//               (user, 2026-10-11: "the aiming laser    frames without either gap pass). The
+//               only activates when your fingers get    middle counts for its right click and the
+//               within a distance"; 2026-10-15: also    three-finger scroll
+//               for the middle and three-finger taps)
 //   open        not a fist: HandFrame::curl (the more   fistCurl - 0.05: a hand closing into a
 //               open of the index and the other three   fist drops its laser, so the fist's
 //               fingers) at least fistCurl (1.1)        thumb-to-index touch isn't a click
@@ -147,7 +149,7 @@ struct Params {
     double forearmShow = -35, forearmHide = -50, elbowConfMin = 0.5;
     double pitchMin = -45, pitchMax = 60, pitchMinHide = -60, pitchMaxHide = 75;
     double yawShow = 70, yawHide = 80;
-    double readyShow = 0.06, readyHide = 0.10;   // the thumb and index tips apart (0: no "ready" test)
+    double readyShow = 0.06, readyHide = 0.10;   // the thumb to the nearer of the index and middle tips (0: no "ready" test)
     double fistCurl = 1.1;                       // HandFrame::curl under this: a fist (0: no "open" test)
     // filter (see the top)
     double minCutoff = 0.8, beta = 0.06, dCutoff = 1.0, originMinCutoff = 1.5, originBeta = 0.5;
@@ -276,7 +278,8 @@ struct HandFrame {
     double elbowConf = 0;    // 0: the arm model's guess
     bool grip = false;       // a grip (closed hand) is down
     bool near = false;       // near a panel (see "not near" at the top)
-    double gap = -1;         // the thumb and index tips apart (m); < 0 unknown (the "ready" test passes)
+    double gap = -1;         // the thumb and index tips apart (m); < 0 unknown
+    double gap2 = -1;        // the thumb and middle tips apart (m); < 0 unknown (both: the "ready" test passes)
     double curl = -1;        // how open the hand is (FistCurl): about 1.9 open, under 1.1 a fist; < 0 unknown
 };
 
@@ -328,7 +331,8 @@ inline Tests Evaluate(const Params &p, const Body &body, const HandFrame &f, con
     test(r.pitch >= p.pitchMin && r.pitch <= p.pitchMax, r.pitch >= p.pitchMinHide && r.pitch <= p.pitchMaxHide, "aim");
     test(r.yawOff <= p.yawShow, r.yawOff <= p.yawHide, "yaw");
     test(!f.grip, !f.grip, "grip");
-    if (p.readyShow > 0 && f.gap >= 0) test(f.gap <= p.readyShow, f.gap <= std::max(p.readyShow, p.readyHide), "ready");
+    const double gap = f.gap < 0 ? f.gap2 : f.gap2 < 0 ? f.gap : std::min(f.gap, f.gap2);
+    if (p.readyShow > 0 && gap >= 0) test(gap <= p.readyShow, gap <= std::max(p.readyShow, p.readyHide), "ready");
     if (p.fistCurl > 0 && f.curl >= 0) test(f.curl >= p.fistCurl, f.curl >= p.fistCurl - 0.05, "fist");
     r.show = show, r.keep = keep;
     return r;

hands/models.json

@@ -0,0 +1,80 @@
+{
+ "catalog": 1,
+ "kind": "hands",
+ "latest": "v16b",
+ "default": "latest",
+ "baseline": "stock",
+ "metrics": [
+  {"key": "replay_bright", "label": "Hands tracked, bright room (90 s replay)", "unit": "", "better": "higher",
+   "note": "The maintainer's recordings replayed through ft-hands with its default settings."},
+  {"key": "replay_dim", "label": "Hands tracked, dim room (120 s replay)", "unit": "", "better": "higher"},
+  {"key": "replay_lit", "label": "Hands tracked, lit room (120 s replay)", "unit": "", "better": "higher"},
+  {"key": "found", "label": "Hand found in ft-hands' crops (held-out people)", "unit": "%", "better": "higher",
+   "note": "Contributors the model never trained on (Frametop Hands #4, #7, #8)."},
+  {"key": "landmark_px", "label": "Finger point error, median (held-out people)", "unit": "px", "better": "lower"},
+  {"key": "within5", "label": "Points within 5% of hand size (held-out people)", "unit": "%", "better": "higher"},
+  {"key": "shape_mm", "label": "3D hand shape error, median (held-out people)", "unit": "mm", "better": "lower"},
+  {"key": "side_ok", "label": "Left or right hand told correctly (held-out people)", "unit": "%", "better": "higher"},
+  {"key": "pinch_closed", "label": "Thumb-index pinch read closed (held-out people)", "unit": "%", "better": "higher",
+   "note": "Of the held-out views where the thumb and index tips touch, the share the model reads as touching."}
+ ],
+ "models": [
+  {
+   "name": "v16b",
+   "title": "Frametop v16b",
+   "kind": "custom",
+   "status": "recommended",
+   "date": "2026-10-16",
+   "summary": "v13b-p1 plus the pinch and tap bounty rounds: pinches and taps read closed more often.",
+   "notes": "v13b-p1's landmark model trained further with Frametop Hands contributions #12 (index taps), #13 (pinches) and #14. The palm detector is v13b-p1's, unchanged. Same networks as stock, so the same CPU cost.\n\nWhat to expect against v13b-p1: a thumb touching the index finger is read as touching more often (80% of held-out touches against 71%), so pinches and taps register more reliably. Finger points and hand shape are as close as before, and the replays track the same hands.\n\nKnown weak spots: a thumb touching the middle finger is still often read as a near miss, and empty views are taken for a hand slightly more often than with v13b-p1 (0.37% against 0.22% of empty crops).",
+   "cost": "Same as stock: two networks, fp16, on the cores hand tracking already uses.",
+   "bench": {"replay_bright": 960, "replay_dim": 1225, "replay_lit": 1434, "found": 98.7, "landmark_px": 3.8,
+             "within5": 94.7, "shape_mm": 8.1, "side_ok": 96.6, "pinch_closed": 80.0},
+   "format": 1,
+   "rev": "d96e56599b0af18fc05af2a0a4489985914b754a",
+   "files": {
+    "NOTICE": "0d31ad8d108fc01e9fc4c5612eb5a9b16d90f9a971e6677cafe4969c2388fc06",
+    "hand.ncnn.bin": "c602ad2344bd1676a628fc8ab4fca2f53574b9f12bc97c705f47762d23f5f941",
+    "hand.ncnn.param": "54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd",
+    "palm.ncnn.bin": "9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909",
+    "palm.ncnn.param": "c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e",
+    "LICENSE": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
+    "README.md": "322fc0b146d301865592958ba4dd512bf249836816b578f0919a2f671f3f014c"
+   }
+  },
+  {
+   "name": "v13b-p1",
+   "title": "Frametop v13b-p1",
+   "kind": "custom",
+   "status": "older",
+   "date": "2026-10-14",
+   "summary": "Retrained on the Steam Frame's own cameras: about twice as many hands found in bright rooms as stock, and steadier fingers.",
+   "notes": "MediaPipe's palm detector and hand landmark model, retrained on the Frametop Hands dataset (contributions #3 to #7), the maintainer's recordings and flagged tracking failures. Same networks as stock, so the same CPU cost.\n\nWhat to expect against stock: hands are found far more often, most of all in bright rooms (twice as many tracked in the bright replay, 1.3 to 1.8 times in dim and lit ones), the finger points sit about a quarter closer, and the 3D hand shape is about 40% closer.\n\nKnown weak spot: a thumb touching the middle finger is often read as a near miss, so middle-finger pinches (right click) can need a firmer touch.",
+   "cost": "Same as stock: two networks, fp16, on the cores hand tracking already uses.",
+   "bench": {"replay_bright": 960, "replay_dim": 1224, "replay_lit": 1433, "found": 98.7, "landmark_px": 3.8,
+             "within5": 94.6, "shape_mm": 8.3, "side_ok": 96.5, "pinch_closed": 71.3},
+   "format": 1,
+   "rev": "41afd78fbd4527456fc4cbd840ffc9f39fdc939a",
+   "files": {
+    "NOTICE": "0e6b4fa8ea37665cb699abc36fcf1ee0ffaa63a78dfee788be6613dea6607f84",
+    "hand.ncnn.bin": "d6a6f00d845fb9ee490276d84b1e978da884361c372614546ed2b1b4d2f5c449",
+    "hand.ncnn.param": "54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd",
+    "palm.ncnn.bin": "9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909",
+    "palm.ncnn.param": "c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e",
+    "LICENSE": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
+    "README.md": "29c6c5460e47412a9a37b22d6a7c8a9ed266a872bbcce93d3867f9e56cfe0ca1"
+   }
+  },
+  {
+   "name": "stock",
+   "title": "MediaPipe (stock, off the shelf)",
+   "kind": "stock",
+   "status": "builtin",
+   "summary": "Google's original palm and hand models, unchanged. Comes with Frametop.",
+   "notes": "Trained on colour photos from ordinary lenses. The Frame's tracking cameras are infrared, monochrome and fisheye, so these lose hands often, most of all in bright rooms. Useful to compare, or to go back to if a retrained version tracks worse for you.",
+   "cost": "Two networks, fp16, on the cores hand tracking already uses.",
+   "bench": {"replay_bright": 442, "replay_dim": 912, "replay_lit": 782, "found": 94.8, "landmark_px": 5.2,
+             "within5": 88.5, "shape_mm": 14.3, "side_ok": 93.8, "pinch_closed": 7.8}
+  }
+ ]
+}

hands/models.pin

@@ -5,12 +5,12 @@
 # needs to run them: raise it when a model needs a newer ft-hands, and older installs keep theirs.
 # Made by frame-hands' model-repo/pin.py.
 format 1
-rev 41afd78fbd4527456fc4cbd840ffc9f39fdc939a
-model v13b-p1
-file NOTICE 0e6b4fa8ea37665cb699abc36fcf1ee0ffaa63a78dfee788be6613dea6607f84
-file hand.ncnn.bin d6a6f00d845fb9ee490276d84b1e978da884361c372614546ed2b1b4d2f5c449
+rev d96e56599b0af18fc05af2a0a4489985914b754a
+model v16b
+file NOTICE 0d31ad8d108fc01e9fc4c5612eb5a9b16d90f9a971e6677cafe4969c2388fc06
+file hand.ncnn.bin c602ad2344bd1676a628fc8ab4fca2f53574b9f12bc97c705f47762d23f5f941
 file hand.ncnn.param 54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd
 file palm.ncnn.bin 9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909
 file palm.ncnn.param c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e
 file LICENSE 41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0
-file README.md 29c6c5460e47412a9a37b22d6a7c8a9ed266a872bbcce93d3867f9e56cfe0ca1
+file README.md 322fc0b146d301865592958ba4dd512bf249836816b578f0919a2f671f3f014c

hands/rec/bounties/card-grabs.json

@@ -12,7 +12,7 @@
  "done": {
   "title": "All done",
   "seconds": 6,
-  "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
+  "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
  },
  "stopped": {
   "title": "Stopped",

hands/rec/bounties/hand-gestures.json

@@ -12,7 +12,7 @@
  "done": {
   "title": "All done",
   "seconds": 6,
-  "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
+  "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
  },
  "stopped": {
   "title": "Stopped",

hands/rec/bounties/index-taps.json

@@ -12,7 +12,7 @@
  "done": {
   "title": "All done",
   "seconds": 6,
-  "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
+  "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
  },
  "stopped": {
   "title": "Stopped",

hands/rec/bounties/middle-taps.json

@@ -12,7 +12,7 @@
  "done": {
   "title": "All done",
   "seconds": 6,
-  "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
+  "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
  },
  "stopped": {
   "title": "Stopped",

hands/rec/bounties/pinches.json

@@ -12,7 +12,7 @@
  "done": {
   "title": "All done",
   "seconds": 6,
-  "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
+  "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
  },
  "stopped": {
   "title": "Stopped",

hands/rec/bounties/pokes.json

@@ -12,7 +12,7 @@
  "done": {
   "title": "All done",
   "seconds": 6,
-  "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
+  "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
  },
  "stopped": {
   "title": "Stopped",

hands/rec/ft-handrec

@@ -1,6 +1,8 @@
 #!/bin/bash
 # Launch the Frametop Hand Recorder from a Plasma session on the Frame host.
-# The app runs in Frametop's container (PySide6, Kirigami and zstd come from Fedora there).
+# The app runs in Frametop's container (PySide6, Kirigami and zstd come from Fedora there; a
+# release's image has the recorders' cryptography and huggingface_hub in /opt/frametop/venv, which
+# distrobox leaves off the front of PATH, so it goes first here).
 # podman needs the real XDG_RUNTIME_DIR and the real user bus (to reach systemd for
 # the container's cgroup; the Frametop session runs on a private bus from
 # dbus-run-session). The session's Wayland socket and bus go to the app itself.
@@ -14,7 +16,8 @@ export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
 # From the home folder: the app's host commands (distrobox-host-exec) run in its folder on the
 # host, and a folder only the container has, such as /run/host/tmp, makes them all fail.
 cd "$HOME" || exit 1
-exec "$here/../../scripts/in-box" env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
+exec "$here/../../scripts/in-box" env PATH="/opt/frametop/venv/bin:/opt/frametop/bin:$PATH:/usr/sbin:/sbin" \
+  WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
   XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
   QT_QPA_PLATFORM="wayland;xcb" \
   python3 "$here/ft_handrec.py" "$@"

hands/rec/script-v2.json

@@ -4,7 +4,7 @@
  "intro_s": 4,
  "between_s": 3,
  "welcome": {"title": "Hand recorder", "seconds": 8, "text": "Follow the steps on this panel. Each one plays a short clip of what to do: copy it. In the clips the palm is light and the back of the hand is dark.|The button on the right side of the headset goes on when a step waits for you, and pauses while recording."},
- "done": {"title": "All done", "seconds": 6, "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."},
+ "done": {"title": "All done", "seconds": 6, "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."},
  "stopped": {"title": "Stopped", "seconds": 4, "text": "What was recorded so far is saved. Review it in the Hand recorder window."},
  "objects": {"pencil": "pencil or pen", "phone": "phone", "cup": "cup", "keyboard": "keyboard", "mouse": "mouse", "gamepad": "gamepad", "small": "small thing"},
  "sections": [

hands/rec/script.json

@@ -5,7 +5,7 @@
  "welcome": {"title": "Hand recorder", "seconds": 7,
   "text": "Follow the steps on this panel. Each one shows what to do with your hands, and where. A bar shows the time left.|The button on the right side of the headset goes on when a step waits for you, and pauses while recording."},
  "done": {"title": "All done", "seconds": 6,
-  "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."},
+  "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."},
  "stopped": {"title": "Stopped", "seconds": 4,
   "text": "What was recorded so far is saved. Review it in the Hand recorder window."},
  "cue_names": {"flat": "Flat", "flat-back": "Backs", "spread": "Spread", "open": "Open", "fist": "Fist",

hands/track/main.cpp

@@ -45,8 +45,9 @@
 // HANDS_CAMERAS, HANDS_BRIGHT, HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT (which
 // colour camera is passthrough_left: color_video0 or color_video3), HANDS_COLOR_CROP
 // (subtract or none: tools/check_color.py tells both), HANDS_MISREAD_GUARD (0 or 1: the
-// tracker's guards for fine-tuned landmark models, Tracker::set_misread_guard), HANDS_MODELS (as
-// --models: a folder with palm.ncnn.* and hand.ncnn.*, e.g. fine-tuned ones), HANDS_REACQUIRE (seconds
+// tracker's guards for fine-tuned landmark models, Tracker::set_misread_guard), HANDS_MODEL (latest,
+// stock, or a version's name: models/catalog.py), HANDS_MODELS (as --models: a folder with palm.ncnn.*
+// and hand.ncnn.*, used instead of HANDS_MODEL), HANDS_REACQUIRE (seconds
 // the palm search aims at a lost hand, 0 off: Tracker::set_reacquire), HANDS_PAIR_SIDES (0 or 1: two
 // hands are one left and one right, Tracker::set_pair_sides), HANDS_PINCH_DIP (0 or 1: pinches also
 // begin on a sharp dip of the thumb-index distance, PinchParams::dip), HANDS_PINCH_FIST_CURL (1.2; 0 off:
@@ -69,6 +70,7 @@
 #include <unistd.h>
 
 #include <algorithm>
+#include <cctype>
 #include <cmath>
 #include <ctime>
 #include <memory>
@@ -456,10 +458,29 @@ int main(int argc, char **argv) {
         std::fflush(stdout);
         return true;
     };
-    if (models.empty())
-        models = model_root + (!int8 && access((model_root + "student/hand.ncnn.param").c_str(), R_OK) == 0 &&
-                                       access((model_root + "student/palm.ncnn.param").c_str(), R_OK) == 0
-                                   ? "student" : "ncnn");
+    // HANDS_MODEL (Input Settings' Hands page, models/catalog.py): latest (models/student, which
+    // hands/models.sh keeps current), stock (models/ncnn), or a downloaded version (models/versions/NAME).
+    // None of the fine-tuned ones has 8-bit files, so --int8 always takes the stock ones.
+    auto has_models = [](const std::string &dir) {
+        return access((dir + "/hand.ncnn.param").c_str(), R_OK) == 0 && access((dir + "/palm.ncnn.param").c_str(), R_OK) == 0;
+    };
+    if (models.empty()) {
+        const std::string pick = setting("HANDS_MODEL");
+        const bool name_ok = !pick.empty() && pick.size() <= 64 && std::isalnum((unsigned char)pick[0]) &&
+                             pick.find_first_not_of("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._-") ==
+                                 std::string::npos;
+        if (pick == "stock" || pick == "off" || pick == "none") {
+            models = model_root + "ncnn";
+        } else if (name_ok && pick != "latest") {
+            if (!int8 && has_models(model_root + "versions/" + pick)) models = model_root + "versions/" + pick;
+            else if (!int8)
+                std::fprintf(stderr, "HANDS_MODEL=%s isn't downloaded (Frametop Input Settings > Hands, or "
+                             "models/ft-models fetch hands %s): using the latest\n", pick.c_str(), pick.c_str());
+        } else if (!pick.empty() && pick != "latest") {
+            std::fprintf(stderr, "HANDS_MODEL=%s isn't a model name: using the latest\n", pick.c_str());
+        }
+        if (models.empty()) models = model_root + (!int8 && has_models(model_root + "student") ? "student" : "ncnn");
+    }
     if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
         (publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
         (!record.empty() && !start_recording(record, err))) {

input-settings/ft_input_settings.py

@@ -23,14 +23,17 @@ to the input relay over its control socket (@frametop_relay):
     go to ~/.config/frametop-voice.json (voice/voice_settings.py); ft-voiced reads them again
     within 2 s, and answers "status" on its control socket.
   - Gaze: the pointer's gaze mode (@ft_pointer_helper "gaze") and the gaze service
-    (gaze/ft-gazed, @ft_gazed: status, forget, reload), with its eye tracker (SteamVR's or
-    our own) and eye bias (GAZE_TRACKER, GAZE_EYE in frametop.conf).
+    (gaze/ft-gazed, @ft_gazed: status, forget, reload), with its eye tracker and eye bias
+    (GAZE_TRACKER, GAZE_EYE in frametop.conf). The tracker is picked from a catalog (models/catalog.py,
+    gaze/tracker/models.json): SteamVR's, our classic one, or one of our eye nets (EYES_MODEL: latest,
+    which follows new versions, or a version), each with notes and benchmarks, downloaded on use.
   - Hands: when hand tracking runs (HANDS_MODE: with the Frametop desktop, always with SteamVR,
     or off; hands/handsmode.py on the host starts and stops its services), the hand cutouts'
     margin (HANDS_MATTE_MARGIN, sent live to ft-screens: "cutouts margin DEG"), and three hand
     inputs, each on its own (handsmode.METHODS): clicks at the gaze (POINTER_HANDS,
     POINTER_GAZE_PINCH), the hand ray with gaze off (POINTER_HAND_RAY), and direct touch on
-    Frametop's panels (HANDS_NEAR, sent live to ft-screens: "handinput on|off").
+    Frametop's panels (HANDS_NEAR, sent live to ft-screens: "handinput on|off"), and the hand
+    models (HANDS_MODEL, from hands/models.json: latest, a version, or stock MediaPipe), the same way.
   - Bluetooth: paired devices, and re-applying the Bluetooth LE fixes after pairing.
   - A warning on every page when SteamVR won't load the ft_pointer driver (blocked after a
     crash, disabled, or SteamVR in safe mode) or hasn't loaded it (@ft_pointer doesn't answer
@@ -71,6 +74,12 @@ GAZE_PROBE = os.path.join(REPO, "gaze", "probe", "ft-gazeprobe")
 sys.path.insert(0, os.path.join(REPO, "hands"))
 import handsmode  # noqa: E402
 HANDSMODE = os.path.join(REPO, "hands", "handsmode.py")
+# The hand and eye model catalogs: versions, notes, benchmarks, downloads (standard library only).
+sys.path.insert(0, os.path.join(REPO, "models"))
+import catalog as model_catalog  # noqa: E402
+# How the pickers tag each entry: by status, and builtins by what they are.
+MODEL_TAGS = {"recommended": "Recommended", "testing": "Testing", "older": "Older"}
+MODEL_KIND_TAGS = {"stock": "Off the shelf", "valve": "Valve", "classic": "Built in", "custom": "Frametop"}
 SCREENS = "\0ft_screens"
 HANDS_MODES = {"frametop": "With Frametop: while the Frametop desktop runs",
                "always": "Always: whenever SteamVR runs, in games and the SteamVR dashboard too",
@@ -454,6 +463,7 @@ class Backend(QObject):
     captured = Signal(int, str)  # code, name
     capturedController = Signal(str, str)  # button, label
     shortcutCaptureChanged = Signal()
+    modelsChanged = Signal()
     message = Signal(str, bool)  # text, is error
 
     def __init__(self):
@@ -485,6 +495,8 @@ class Backend(QObject):
         self._hands = {}  # handsmode.py's status, {} until it answers
         self._hands_lock = threading.Lock()  # one handsmode.py at a time
         self._hands_refreshing = False
+        self._model_busy = {"hands": "", "eyes": ""}  # what a model download is doing, per kind
+        self._model_views = {}
         self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
         self.sock.bind("")  # autobind an abstract address the relay can reply to
         self.sock.setblocking(False)
@@ -1594,6 +1606,166 @@ class Backend(QObject):
         self._send(f"cutouts margin {value}", SCREENS)
         self.handsChanged.emit()
 
+    # --- models (hands and eyes: models/catalog.py) ---
+    def _model_view(self, kind):
+        """The picker's view of a kind: the catalog's entries as options, a "latest" one first (it
+        follows new versions), which one the settings pick ("choice") and which version runs ("running")."""
+        try:
+            v = model_catalog.view(kind)
+        except (OSError, ValueError, model_catalog.CatalogError) as e:
+            return {"error": f"The {kind} model list didn't load: {e}", "options": [], "metrics": []}
+        conf = read_conf()
+        by = {m["name"]: m for m in v["models"]}
+        choice = v["setting"]
+        note = ""
+        if kind == "eyes" and self.gazeTracker == "steam":
+            choice = "steamvr"
+        elif "/" in choice:
+            note = f"EYES_MODEL is a folder of your own ({choice}); pick one here to replace it."
+            choice = ""
+        if kind == "hands" and (os.environ.get("FT_HANDS_MODELS") or conf.get("HANDS_MODELS")):
+            note = ("HANDS_MODELS in frametop.conf names a folder of its own, and ft-hands uses that instead "
+                    "of what's picked here. Remove it to use this choice.")
+        running = v["latest"] if choice == "latest" else choice
+        if kind == "hands" and choice == "latest":
+            # hands/models.sh keeps latest in hands/models/student, on its own pin.
+            here = ""
+            try:
+                with open(os.path.join(REPO, "hands", "models", "student", "PIN")) as f:
+                    here = next((ln.split()[1] for ln in f if ln.startswith("model ")), "")
+            except (OSError, IndexError):
+                pass
+            running = here or "stock"
+            if not here:
+                note = note or (f"This install has no fine-tuned models yet, so stock runs: {v['latest']} comes "
+                                "when hand tracking next starts online (hands/models.sh).")
+            elif here != v["latest"]:
+                note = note or f"This install still has {here}: {v['latest']} comes when hand tracking next starts."
+        elif kind == "eyes" and choice == "latest":
+            running = v["resolved"]["name"] or "classic"
+            if not v["resolved"]["folder"]:
+                note = note or f"{v['latest']} isn't downloaded yet: the classic tracker runs until it is."
+        opts = []
+        latest = by.get(v["latest"])
+        if latest:
+            opts.append(dict(latest, name="latest", target=latest["name"], follows=True,
+                             title=f"Latest (follows updates): {latest['title']}",
+                             tag=MODEL_TAGS.get(latest["status"], ""),
+                             installed=True if kind == "hands" else latest["installed"]))
+        for m in v["models"]:
+            tag = MODEL_TAGS.get(m["status"]) or MODEL_KIND_TAGS.get(m["kind"], "")
+            opts.append(dict(m, target=m["name"], follows=False, tag=tag))
+        for o in opts:
+            o["bench"] = {k: (None if val is None else float(val)) for k, val in o["bench"].items()}
+        checked = time.strftime("%H:%M", time.localtime(v["checked"])) if v["checked"] else ""
+        return {"options": opts, "metrics": v["metrics"], "byName": by, "latest": v["latest"],
+                "baseline": v["baseline"], "choice": choice, "running": running, "note": note,
+                "source": v["source"], "checked": checked, "error": ""}
+
+    def _models_update(self):
+        self._model_views = {k: self._model_view(k) for k in ("hands", "eyes")}
+        self.modelsChanged.emit()
+
+    @Property("QVariantMap", notify=modelsChanged)
+    def handModels(self):
+        if "hands" not in self._model_views:   # (no signal from inside a getter: a binding loop)
+            self._model_views["hands"] = self._model_view("hands")
+        return self._model_views["hands"]
+
+    @Property("QVariantMap", notify=modelsChanged)
+    def eyeModels(self):
+        if "eyes" not in self._model_views:   # (no signal from inside a getter: a binding loop)
+            self._model_views["eyes"] = self._model_view("eyes")
+        return self._model_views["eyes"]
+
+    @Property("QVariantMap", notify=modelsChanged)
+    def modelBusy(self):
+        return self._model_busy
+
+    def _model_job(self, kind, work):
+        """Run a download (or a catalog check) off the UI thread, one per kind at a time."""
+        if self._model_busy[kind]:
+            return
+
+        def job():
+            try:
+                work()
+            except (OSError, ValueError, model_catalog.CatalogError) as e:
+                self.message.emit(str(e), True)
+            finally:
+                self._model_busy[kind] = ""
+                self._models_update()
+        self._model_busy[kind] = "working…"
+        self.modelsChanged.emit()
+        threading.Thread(target=job, daemon=True).start()
+
+    def _busy(self, kind, text):
+        self._model_busy[kind] = text
+        self.modelsChanged.emit()
+
+    @Slot()
+    def reloadModels(self):
+        """Read the catalogs and the downloads again (the pages call it when they open)."""
+        self._models_update()
+
+    @Slot(str)
+    def refreshModels(self, kind):
+        """The kind's catalog from GitHub (this install's branch): new versions and notes."""
+        if kind not in model_catalog.SPECS:
+            return
+
+        def work():
+            self._busy(kind, "checking for new versions…")
+            ok, msg = model_catalog.refresh(kind)
+            self.message.emit(("Hand" if kind == "hands" else "Eye") + " models: " + msg, not ok)
+        self._model_job(kind, work)
+
+    @Slot(str, str)
+    def useModel(self, kind, name):
+        """Pick a model: download it first if it needs that, then write the setting and apply it.
+        hands: HANDS_MODEL (latest, stock or a version), and hand tracking restarts to load it.
+        eyes: steamvr is GAZE_TRACKER=steam; anything else is our tracker (GAZE_TRACKER=own) with
+        EYES_MODEL (classic, latest or a version), which the gaze service applies (it restarts ft-eyes)."""
+        if kind not in model_catalog.SPECS or not (name == "latest" or model_catalog.NAME.fullmatch(name)):
+            return
+        view = self._model_views.get(kind) or self._model_view(kind)
+        target = view["latest"] if name == "latest" else name
+        e = view["byName"].get(target)
+        if e is None:
+            return
+
+        def work():
+            if not e["builtin"] and not (kind == "hands" and name == "latest"):
+                if not e["usable"]:
+                    raise model_catalog.CatalogError(f"{target} needs a newer Frametop")
+                if not e["installed"]:
+                    model_catalog.fetch(kind, target, progress=lambda t: self._busy(kind, t + "…"))
+            if kind == "hands":
+                write_conf_value("HANDS_MODEL", name)
+                r = host("systemctl", "--user", "try-restart", "frametop-hands.service")
+                running = r.returncode == 0 and host("systemctl", "--user", "is-active", "--quiet",
+                                                     "frametop-hands.service").returncode == 0
+                self.message.emit(f"Hand model: {e['title']}" + (", hand tracking restarted" if running
+                                                                  else " (used when hand tracking starts)"), False)
+                self.handsChanged.emit()
+            elif name == "steamvr":
+                self._set_gaze("GAZE_TRACKER", "steam", "Eye tracker: SteamVR's")
+            else:
+                write_conf_value("EYES_MODEL", name)
+                self._set_gaze("GAZE_TRACKER", "own", f"Eye tracker: {e['title']}"
+                               + ("" if e["builtin"] else " (calibrate it once: each net keeps its own calibration)"))
+        self._model_job(kind, work)
+
+    @Slot(str, str)
+    def removeModel(self, kind, name):
+        """Delete a downloaded version (not the one in use)."""
+        view = self._model_views.get(kind) or self._model_view(kind)
+        if kind not in model_catalog.SPECS or name in (view["running"], view["choice"]):
+            return
+        if model_catalog.remove(kind, name):
+            self.message.emit(f"Removed the download of {name}", False)
+        self._models_update()
+
     # --- bluetooth ---
     @Property("QVariantList", notify=bluetoothChanged)
     def bluetooth(self):

input-settings/main.qml

@@ -54,6 +54,212 @@ Kirigami.ApplicationWindow {
         })[backend.driverBlock] || ""
     }
 
+    // A model picker (Hands and Gaze pages): the catalog's versions (models/catalog.py) with their notes
+    // and benchmarks, set against the one running (viewing that one: against the baseline, stock or
+    // SteamVR). Choosing a version only shows it; "Use" downloads it if needed and switches to it.
+    // The catalog's texts come from GitHub, so they're shown as plain text.
+    component ModelPicker: ColumnLayout {
+        id: picker
+        property string kind
+        property var view: ({})
+        readonly property string busy: (backend.modelBusy || {})[kind] || ""
+        property string viewed: ""
+        readonly property var options: view.options || []
+        readonly property var opt: options.find(o => o.name === (viewed || view.choice)) || options[0] || ({})
+        readonly property string running: view.running || ""
+        readonly property string otherName: opt.target !== running ? running
+                                            : opt.target !== view.baseline ? (view.baseline || "") : ""
+        readonly property var other: otherName ? ((view.byName || {})[otherName] || null) : null
+        readonly property bool inUse: opt.name === view.choice
+        spacing: Kirigami.Units.smallSpacing
+        Layout.maximumWidth: Kirigami.Units.gridUnit * 30
+
+        function fmt(v, unit) {
+            if (v === undefined || v === null) return "–"
+            const n = Math.abs(v) >= 100 ? Math.round(v) : Math.round(v * 100) / 100
+            return n + (unit === "" || unit === "%" || unit === "°" ? unit : " " + unit)
+        }
+        function better(m, v, o) {
+            if (v === undefined || v === null || o === undefined || o === null || v === o) return 0
+            return (m.better === "lower" ? v < o : v > o) ? 1 : -1
+        }
+
+        Kirigami.InlineMessage {
+            Layout.fillWidth: true
+            visible: !!picker.view.error
+            type: Kirigami.MessageType.Error
+            text: picker.view.error || ""
+        }
+        Repeater {
+            model: picker.options
+            delegate: Controls.RadioButton {
+                required property var modelData
+                text: modelData.title + (modelData.tag ? "  ·  " + modelData.tag : "")
+                      + (modelData.name === picker.view.choice ? "  ·  in use" : "")
+                      + (!modelData.usable ? "  ·  needs a newer Frametop"
+                         : !modelData.installed ? "  ·  not downloaded" : "")
+                font.bold: modelData.name === picker.view.choice
+                checked: picker.opt.name === modelData.name
+                onToggled: if (checked) picker.viewed = modelData.name
+            }
+        }
+        Controls.Label {
+            visible: !!picker.view.note
+            text: picker.view.note || ""
+            textFormat: Text.PlainText
+            Layout.fillWidth: true
+            wrapMode: Text.WordWrap
+            color: Kirigami.Theme.neutralTextColor
+            font: Kirigami.Theme.smallFont
+        }
+
+        // The one being looked at: what it is, what to expect, what it costs, and its numbers.
+        Kirigami.Separator { Layout.fillWidth: true; visible: !!picker.opt.name }
+        Controls.Label {
+            text: (picker.opt.follows ? "Latest: " + picker.opt.target : picker.opt.title || "")
+                  + (picker.opt.date ? " (" + picker.opt.date + ")" : "")
+            textFormat: Text.PlainText
+            font.bold: true
+        }
+        Controls.Label {
+            visible: picker.opt.follows === true
+            text: "Moves to each new version we recommend, without asking. Pick a version below it to stay on that one."
+            Layout.fillWidth: true
+            wrapMode: Text.WordWrap
+            opacity: 0.7
+            font: Kirigami.Theme.smallFont
+        }
+        Controls.Label {
+            text: picker.opt.summary || ""
+            visible: text !== ""
+            textFormat: Text.PlainText
+            Layout.fillWidth: true
+            wrapMode: Text.WordWrap
+        }
+        Controls.Label {
+            text: picker.opt.notes || ""
+            visible: text !== ""
+            textFormat: Text.PlainText
+            Layout.fillWidth: true
+            wrapMode: Text.WordWrap
+            opacity: 0.85
+        }
+        Controls.Label {
+            text: picker.opt.cost ? "Cost: " + picker.opt.cost : ""
+            visible: text !== ""
+            textFormat: Text.PlainText
+            Layout.fillWidth: true
+            wrapMode: Text.WordWrap
+            opacity: 0.7
+            font: Kirigami.Theme.smallFont
+        }
+        GridLayout {
+            visible: (picker.view.metrics || []).some(m => picker.opt.bench && picker.opt.bench[m.key] !== undefined)
+            columns: 3
+            columnSpacing: Kirigami.Units.largeSpacing
+            rowSpacing: 2
+            Controls.Label { text: "Benchmark"; font.bold: true; Layout.fillWidth: true }
+            Controls.Label { text: picker.opt.target || ""; textFormat: Text.PlainText; font.bold: true }
+            Controls.Label {
+                text: picker.other ? picker.otherName + (picker.otherName === picker.running ? " (in use)" : "") : ""
+                textFormat: Text.PlainText
+                font.bold: true
+                opacity: 0.7
+            }
+            Repeater {
+                model: picker.view.metrics || []
+                delegate: Controls.Label {
+                    required property var modelData
+                    required property int index
+                    text: modelData.label
+                    textFormat: Text.PlainText
+                    Layout.row: index + 1
+                    Layout.column: 0
+                    Layout.fillWidth: true
+                    wrapMode: Text.WordWrap
+                    font: Kirigami.Theme.smallFont
+                    Controls.ToolTip.visible: !!modelData.note && mhover.hovered
+                    Controls.ToolTip.text: modelData.note || ""
+                    HoverHandler { id: mhover }
+                }
+            }
+            Repeater {
+                model: picker.view.metrics || []
+                delegate: Controls.Label {
+                    required property var modelData
+                    required property int index
+                    readonly property var v: (picker.opt.bench || {})[modelData.key]
+                    readonly property int b: picker.other ? picker.better(modelData, v, (picker.other.bench || {})[modelData.key]) : 0
+                    text: picker.fmt(v, modelData.unit)
+                    Layout.row: index + 1
+                    Layout.column: 1
+                    color: b > 0 ? Kirigami.Theme.positiveTextColor : b < 0 ? Kirigami.Theme.negativeTextColor
+                                                                       : Kirigami.Theme.textColor
+                }
+            }
+            Repeater {
+                model: picker.view.metrics || []
+                delegate: Controls.Label {
+                    required property var modelData
+                    required property int index
+                    text: picker.other ? picker.fmt((picker.other.bench || {})[modelData.key], modelData.unit) : ""
+                    Layout.row: index + 1
+                    Layout.column: 2
+                    opacity: 0.7
+                }
+            }
+        }
+        Controls.Label {
+            visible: (picker.view.metrics || []).some(m => picker.opt.bench && picker.opt.bench[m.key] !== undefined)
+            text: "Green: better than " + (picker.otherName || "the other") + ", red: worse. Hover a row for how it was measured."
+            textFormat: Text.PlainText
+            opacity: 0.7
+            font: Kirigami.Theme.smallFont
+        }
+        RowLayout {
+            Controls.Button {
+                text: picker.inUse ? "In use"
+                      : !picker.opt.installed ? "Download and use"
+                      : "Use this"
+                icon.name: picker.inUse ? "checkmark" : !picker.opt.installed ? "download" : "dialog-ok-apply"
+                enabled: !picker.inUse && picker.opt.usable !== false && picker.busy === "" && !!picker.opt.name
+                onClicked: backend.useModel(picker.kind, picker.opt.name)
+            }
+            Controls.Button {
+                text: "Remove download"
+                icon.name: "edit-delete"
+                visible: !picker.opt.builtin && !picker.opt.follows && picker.opt.installed === true
+                         && picker.opt.target !== picker.running && picker.opt.name !== picker.view.choice
+                enabled: picker.busy === ""
+                onClicked: backend.removeModel(picker.kind, picker.opt.name)
+            }
+            Controls.Button {
+                text: "Check for new versions"
+                icon.name: "view-refresh"
+                enabled: picker.busy === ""
+                onClicked: backend.refreshModels(picker.kind)
+            }
+            Controls.BusyIndicator {
+                visible: picker.busy !== ""
+                running: visible
+                Layout.preferredHeight: Kirigami.Units.iconSizes.medium
+                Layout.preferredWidth: Layout.preferredHeight
+            }
+            Controls.Label {
+                visible: picker.busy !== ""
+                text: picker.busy
+                opacity: 0.7
+                font: Kirigami.Theme.smallFont
+            }
+        }
+        Controls.Label {
+            text: "List from " + (picker.view.source || "this install")
+                  + (picker.view.checked ? ", checked " + picker.view.checked : "")
+            opacity: 0.6
+            font: Kirigami.Theme.smallFont
+        }
+    }
+
     function show(page) {
         pageStack.clear()
         pageStack.push(page)
@@ -1285,26 +1491,22 @@ Kirigami.ApplicationWindow {
                     opacity: 0.7
                     font: Kirigami.Theme.smallFont
                 }
-                RowLayout {
-                    Kirigami.FormData.label: "Eye tracker:"
-                    Controls.RadioButton {
-                        text: "Own tracker (recommended)"
-                        checked: backend.gazeTracker === "own"
-                        onToggled: if (checked) backend.setGazeTracker("own")
-                    }
-                    Controls.RadioButton {
-                        text: "SteamVR"
-                        checked: backend.gazeTracker === "steam"
-                        onToggled: if (checked) backend.setGazeTracker("steam")
-                    }
+                // Whose eye tracking: SteamVR's (GAZE_TRACKER=steam), or ours (own) with the classic
+                // pupil finder or an eye net (EYES_MODEL: latest or a version).
+                Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Eye tracker" }
+                ModelPicker {
+                    Kirigami.FormData.label: "Tracker:"
+                    kind: "eyes"
+                    view: backend.eyeModels
+                    Component.onCompleted: backend.reloadModels()
                 }
                 Controls.Label {
                     // Ours is the default once it's installed (GAZE_TRACKER=auto, gaze/ft-gazed).
                     visible: backend.gazeTracker === "steam" && !backend.gazeOwnInstalled
-                    text: "Our own tracker is more accurate. Install it with gaze/tracker/install.sh in the "
-                          + "Frametop folder, in a terminal (asks for sudo)"
+                    text: "Our own tracker (the classic one and the eye nets) needs its frame grabber. Install it "
+                          + "with gaze/tracker/install.sh in the Frametop folder, in a terminal (asks for sudo)"
                           + (backend.gazeTrackerAuto ? "; gaze then uses it, and you calibrate it once."
-                                                     : ", then pick Own tracker here and calibrate it once.")
+                                                     : ", then pick it here and calibrate it once.")
                     Layout.maximumWidth: Kirigami.Units.gridUnit * 30
                     wrapMode: Text.WordWrap
                     opacity: 0.7
@@ -1354,6 +1556,7 @@ Kirigami.ApplicationWindow {
                     opacity: 0.7
                     font: Kirigami.Theme.smallFont
                 }
+                Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Gaze pointer tuning" }
                 Repeater {
                     model: backend.gazeSettings
                     delegate: RowLayout {
@@ -1454,7 +1657,7 @@ Kirigami.ApplicationWindow {
                       + "a pinch clicks where you look; pinch, hold and move the hand to nudge the pointer first. Pinch nudge "
                       + "starts after: how far the hand moves before the pointer does. Pinch nudge speed: how far the pointer "
                       + "moves for each degree of hand movement after that. Eye tracker: SteamVR's, or our own "
-                      + "(gaze/tracker), which keeps its own calibration. Eye bias: the gaze "
+                      + "(gaze/tracker), the classic pupil finder or an eye net; each keeps its own calibration. Eye bias: the gaze "
                       + "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as "
                       + "much, and Auto weights each by how far off it was at your recent nudges."
             }
@@ -1532,6 +1735,15 @@ Kirigami.ApplicationWindow {
                     }
                 }
 
+                // Which hand models ft-hands runs (HANDS_MODEL): latest, a version, or stock MediaPipe.
+                Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Hand model" }
+                ModelPicker {
+                    Kirigami.FormData.label: "Model:"
+                    kind: "hands"
+                    view: backend.handModels
+                    Component.onCompleted: backend.reloadModels()
+                }
+
                 // Three ways the hands work the desktop, each on its own (handsmode.METHODS).
                 Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Hand input" }
                 Controls.Switch {

install.sh

@@ -8,15 +8,17 @@
 # distrobox the release brings, makes the release's container from its image, and builds nothing.
 #
 # Usage: ./install.sh [--yes] [--no-voice] [--no-hands] [--no-hand-recorder] [--no-eye-tracker]
-#                    [--no-bluetooth | --bluetooth]
-#   --yes           don't ask; installs voice dictation and gaze mode, and hand tracking, the Hand
-#                   Recorder and our eye tracker if sudo can run without a password prompt; skips
-#                   the Bluetooth fixes and the SteamVR restart
+#                    [--no-eye-recorder] [--no-bluetooth | --bluetooth]
+#   --yes           don't ask; installs voice dictation, gaze mode and the Eye Recorder, and hand
+#                   tracking, the Hand Recorder and our eye tracker if sudo can run without a
+#                   password prompt; skips the Bluetooth fixes and the SteamVR restart
 #   --no-voice      don't offer voice dictation (Frametop Voice: it downloads about 320 MB)
 #   --no-hands      don't offer hand tracking or the Hand Recorder
 #   --no-hand-recorder  don't offer the Hand Recorder (a release's; a clone builds it with
 #                   hands/rec/install.sh)
-#   --no-eye-tracker  don't offer our own eye tracker
+#   --no-eye-tracker  don't offer our own eye tracker (or the Eye Recorder, which needs it)
+#   --no-eye-recorder  don't offer the Eye Recorder (a release's; a clone builds it with
+#                   gaze/rec/install.sh)
 #   --no-bluetooth  don't offer the Bluetooth fixes
 #   --bluetooth     install the Bluetooth fixes without asking (with --yes: if sudo can run
 #                   without a password prompt)
@@ -25,17 +27,18 @@ set -euo pipefail
 root=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
 . "$root/scripts/_env.sh"
 
-assume_yes=0 bluetooth=1 eye_tracker=1 hands=1 hand_recorder=1 voice=1
+assume_yes=0 bluetooth=1 eye_tracker=1 eye_recorder=1 hands=1 hand_recorder=1 voice=1
 for arg in "$@"; do
   case $arg in
     --yes) assume_yes=1 ;;
     --no-bluetooth) bluetooth=0 ;;
     --bluetooth) bluetooth=2 ;;
-    --no-eye-tracker) eye_tracker=0 ;;
+    --no-eye-tracker) eye_tracker=0 eye_recorder=0 ;;
+    --no-eye-recorder) eye_recorder=0 ;;
     --no-voice) voice=0 ;;
     --no-hands) hands=0 hand_recorder=0 ;;
     --no-hand-recorder) hand_recorder=0 ;;
-    -h|--help) sed -n '2,22p' "$0"; exit 0 ;;
+    -h|--help) sed -n '2,24p' "$0"; exit 0 ;;
     *) echo "unknown option: $arg" >&2; exit 2 ;;
   esac
 done
@@ -224,6 +227,27 @@ else
   echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
 fi
 
+# The Eye Recorder (gaze/rec), as the Hand Recorder: a release brings it built, so it's only the menu
+# entry, which needs our eye tracker's frame grabber (step 11). An entry from the release before
+# points into that release's folder: it's renewed without asking. A clone builds it with
+# gaze/rec/install.sh (a few minutes), so it isn't offered here.
+if [ "$FRAME_RELEASE" = 1 ]; then
+  recorder="the Eye Recorder (records your eyes for Frametop's eye tracking dataset)"
+  if [ "$eye_recorder" = 0 ]; then
+    echo "$recorder: skipped. Add it later with: gaze/rec/install.sh"
+  elif on_frame '[ -e ~/.local/share/applications/frametop-eyerec.desktop ]'; then
+    "$root/gaze/rec/install.sh" install >/dev/null && echo "$recorder: updated" ||
+      echo "$recorder: couldn't update it (above). Try again with: gaze/rec/install.sh"
+  elif ! on_frame 'systemctl is-enabled -q frametop-eyegrab.service 2>/dev/null'; then
+    echo "$recorder: skipped: it needs our eye tracker. Install both later with: gaze/tracker/install.sh, then gaze/rec/install.sh"
+  elif ask "Add the Eye Recorder to the app menu? It records your eyes for Frametop's eye tracking dataset, which trains our eye tracker; nothing leaves the headset until you press Upload, and uploads are encrypted." y; then
+    "$root/gaze/rec/install.sh" install ||
+      echo "$recorder: not installed (above). Try again later with: gaze/rec/install.sh"
+  else
+    echo "$recorder: skipped. Add it later with: gaze/rec/install.sh"
+  fi
+fi
+
 step "12/12 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
 if [ "$bluetooth" = 2 ] && [ "$assume_yes" = 1 ] && ! sudo_quiet; then
   echo "skipped: they need your password (sudo), and --yes doesn't ask. Install later with: setup/bluetooth/install.sh install"

justfile

@@ -20,7 +20,7 @@ test-python:
     set -uo pipefail
     python3 -c 'import PySide6, numpy, cv2, sherpa_onnx' || { echo "python3 can't import PySide6, numpy, cv2, and sherpa_onnx"; exit 1; }
     status=0
-    for t in input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py recorder/tests/*.py gaze/test/*.py gaze/rec/tests/*.py pack/test/*.py voice/test/*.py spotwatch/test/*.py; do
+    for t in models/test/*.py input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py recorder/tests/*.py gaze/test/*.py gaze/rec/tests/*.py pack/test/*.py voice/test/*.py spotwatch/test/*.py; do
         [ -f "$t" ] || continue
         if python3 "$t" >/dev/null; then echo "$t: ok"; else echo "$t: FAILED"; status=1; fi
     done

models/README.md

@@ -0,0 +1,50 @@
+# Model versions
+
+Users pick which hand and eye models Frametop runs on Frametop Input Settings' Hands and Gaze pages.
+Each page lists the versions in a catalog, with notes on what to expect and benchmark numbers set
+against the one in use. `catalog.py` reads the catalogs, downloads versions and says what a setting
+runs; `ft-models` is the same from a terminal.
+
+| Kind | Catalog | Setting (frametop.conf) | Downloads go to | Files from |
+|---|---|---|---|---|
+| hands | `hands/models.json` | `HANDS_MODEL`: `latest`, `stock`, a version | `hands/models/versions/NAME` | `DeeJanuz/frametop-hands-models` |
+| eyes | `gaze/tracker/models.json` | `EYES_MODEL`: `classic`, `latest`, a version; `GAZE_TRACKER=steam` for SteamVR's | `gaze/tracker/models/NAME` | `DeeJanuz/frametop-eyes-models` |
+
+- **Latest** follows the catalog's `latest`, so it moves when a new version ships. A version picked
+  by name stays put. For hands, latest is `hands/models/student`, which `hands/models.sh` keeps on
+  `hands/models.pin` (older installs only read that pin). The catalog's latest must be the same
+  files: `models/test/test_catalog.py` checks it.
+- **Where the list comes from.** Installs read their branch's catalog from GitHub (`main` for a
+  stable release, `experimental` for an experimental one or a clone on that branch), so a push
+  reaches them without a release: Input Settings' "Check for new versions", and the services when
+  they start (`ft-models auto`, at most every 15 minutes; `HANDS_MODELS_UPDATE=0` /
+  `EYES_MODELS_UPDATE=0` turn that off). Offline, or when GitHub's copy isn't valid, the copy in
+  the install is used.
+- **Trust.** A catalog only names files. They come from the kind's Hugging Face repo, which is
+  fixed in `catalog.py` and never taken from a catalog. Each download is pinned to a commit and must
+  match its sha256, or it isn't kept. Catalog checks are strict: one bad entry and the whole
+  catalog is ignored. The texts are shown as plain text.
+- **Who applies it.** ft-hands reads `HANDS_MODEL` at start, and Input Settings restarts hand
+  tracking after a change. ft-eyes reads `EYES_MODEL` at start, and the gaze service restarts it
+  when the setting changes. Each eye net keeps its own calibration (`net-NAME/`), so a switch asks
+  for one calibration.
+
+## Shipping a version
+
+1. Upload the version's folder to the kind's Hugging Face repo, next to the earlier ones (hands:
+   frame-hands' `model-repo/`, eyes: frame-eyes' `model-repo/`), with a NOTICE: how it was trained,
+   what it was tested on, the numbers.
+2. Print its files' hashes from that staging folder:
+   `models/ft-models hashes eyes ~/Desktop/Projects/frame-eyes/model-repo eyes-v1`
+3. Add an entry to the catalog: name, title, kind (`custom`), status (`testing` until it's the one
+   to recommend), date, summary, notes (what to expect against the current latest and the
+   baseline), cost, `bench` (the catalog's metrics, measured the same way as the other entries),
+   `format`, `rev` (the Hugging Face commit) and `files`. Set the old latest's status to `older`
+   when a new one is recommended.
+4. To make it latest, set `latest`. For hands, also re-pin `hands/models.pin` with frame-hands'
+   `model-repo/pin.py VERSION REV CHECKOUT`.
+5. Run `python3 models/test/test_catalog.py` and `models/ft-models fetch KIND NAME` (a real
+   download, checked), then push to experimental. Stable installs see it once it's on main.
+
+Raise `format` (and `SPECS[kind]["format"]` in `catalog.py`) when a version needs a newer runtime.
+Older installs then list the version as needing a newer Frametop instead of running it.

models/catalog.py

@@ -0,0 +1,465 @@
+"""The model catalogs: which hand and eye models there are, their notes and benchmarks, and the
+versions someone picked, downloaded and checked.
+
+Two catalogs, one per kind, in the repo and on GitHub:
+  hands  hands/models.json         ft-hands' palm detector + hand landmark model pairs
+  eyes   gaze/tracker/models.json  the eye tracker: SteamVR's, our classic pupil finder, our eye nets
+
+An install reads its branch's catalog from GitHub (main for a stable release, experimental for an
+experimental one or a clone on that branch: models.sh's rule), so a new version and its notes reach
+installs on a push, without a release. Offline, or when GitHub's copy isn't valid, the copy that came
+with the install is used. A catalog only names versions: the files come from the kind's own Hugging
+Face repo (HF_REPO here, never from the catalog), at the commit the entry pins, and every file must
+match the sha256 it lists.
+
+A catalog (JSON):
+  {"catalog": 1, "kind": "hands", "latest": "v13b-p1", "default": "latest", "baseline": "stock",
+   "metrics": [{"key", "label", "unit", "better": "higher"|"lower", "note"}],
+   "models": [{"name", "title", "kind", "status", "date", "summary", "notes", "cost", "bench": {key: n},
+               "format": 1, "rev": "<40 hex>", "files": {"name": "<sha256>"}}]}
+  kind    custom (ours, trained on Frametop data), stock (off the shelf, unchanged), valve (SteamVR's),
+          classic (our hand-written tracker, no model)
+  status  recommended, testing, older, builtin
+  latest  the custom version "Latest" follows: it changes as new versions ship
+  default what an unset setting uses ("latest", or a builtin's name)
+  format  what that version needs of the runtime (FORMATS here); newer ones show, but can't be used
+Builtin entries (no rev or files) ship with Frametop or come from SteamVR.
+
+Downloaded versions live in the kind's store (hands/models/versions/NAME, gaze/tracker/models/NAME),
+with PIN.json, the entry they were fetched by. hands' "latest" is hands/models/student, which
+hands/models.sh keeps on its own pin (models.pin); the catalog's latest must match it (the tests check).
+
+Standard library only: the input settings, the gaze service (on the host) and ft-eyes (in the box)
+import it. CLI: models/ft-models.
+"""
+import fcntl
+import hashlib
+import json
+import os
+import re
+import shutil
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+REPO = Path(__file__).resolve().parents[1]
+GITHUB = "Frametop/frametop"
+CACHE = Path(os.environ.get("XDG_CACHE_HOME") or Path.home() / ".cache") / "frametop" / "models"
+CATALOG_VERSION = 1           # the catalog layout this code reads
+MAX_CATALOG = 128 * 1024      # bytes
+MAX_FILE = 200 * 1024 * 1024  # bytes, per model file
+NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}")
+SHA = re.compile(r"[0-9a-f]{64}")
+REV = re.compile(r"[0-9a-f]{40}")
+KINDS_OF = ("custom", "stock", "valve", "classic")
+STATUSES = ("recommended", "testing", "older", "builtin")
+TEXT_MAX = {"title": 80, "date": 20, "summary": 400, "notes": 4000, "cost": 300}
+METRIC_TEXT_MAX = {"key": 40, "label": 120, "unit": 20, "note": 300}
+
+# Per kind: the catalog, the Hugging Face repo files come from, where versions go, the files a
+# version may have (and must), files at the repo's top, and the newest version format we can run.
+SPECS = {
+    "hands": {
+        "catalog": "hands/models.json",
+        "hf": "DeeJanuz/frametop-hands-models",
+        "store": "hands/models/versions",
+        "files": {"NOTICE", "hand.ncnn.bin", "hand.ncnn.param", "palm.ncnn.bin", "palm.ncnn.param"},
+        "required": {"hand.ncnn.bin", "hand.ncnn.param", "palm.ncnn.bin", "palm.ncnn.param"},
+        "top": {"LICENSE", "README.md"},
+        "format": 1,
+        "setting": "HANDS_MODEL",
+        "update": "HANDS_MODELS_UPDATE",
+    },
+    "eyes": {
+        "catalog": "gaze/tracker/models.json",
+        "hf": "DeeJanuz/frametop-eyes-models",
+        "store": "gaze/tracker/models",
+        "files": {"NOTICE", "model.json", "model.ncnn.bin", "model.ncnn.param"},
+        "required": {"model.json", "model.ncnn.bin", "model.ncnn.param"},
+        "top": {"LICENSE", "README.md"},
+        "format": 1,
+        "setting": "EYES_MODEL",
+        "update": "EYES_MODELS_UPDATE",
+    },
+}
+
+
+class CatalogError(ValueError):
+    pass
+
+
+def _check_text(d, limits, where):
+    for key, n in limits.items():
+        v = d.get(key, "")
+        if not isinstance(v, str) or len(v) > n:
+            raise CatalogError(f"{where}: {key} must be text of at most {n} characters")
+
+
+def validate(cat, kind):
+    """Check a catalog strictly; return it, or raise CatalogError. Anything unexpected fails the
+    whole catalog: GitHub's copy is then ignored for the one that came with the install."""
+    spec = SPECS[kind]
+    if not isinstance(cat, dict):
+        raise CatalogError("not an object")
+    if cat.get("catalog") != CATALOG_VERSION:
+        raise CatalogError(f"catalog version {cat.get('catalog')!r}, this Frametop reads {CATALOG_VERSION}")
+    if cat.get("kind") != kind:
+        raise CatalogError(f"a catalog of {cat.get('kind')!r}, not {kind}")
+    metrics = cat.get("metrics", [])
+    if not isinstance(metrics, list) or len(metrics) > 40:
+        raise CatalogError("metrics must be a list (at most 40)")
+    keys = set()
+    for m in metrics:
+        if not isinstance(m, dict):
+            raise CatalogError("a metric isn't an object")
+        _check_text(m, METRIC_TEXT_MAX, "metric")
+        if not m.get("key") or not m.get("label") or m.get("better", "higher") not in ("higher", "lower"):
+            raise CatalogError(f"metric {m.get('key')!r}: needs key, label, better higher|lower")
+        keys.add(m["key"])
+    models = cat.get("models")
+    if not isinstance(models, list) or not models or len(models) > 200:
+        raise CatalogError("models must be a non-empty list (at most 200)")
+    names = set()
+    for e in models:
+        if not isinstance(e, dict):
+            raise CatalogError("a model isn't an object")
+        name = e.get("name")
+        if not isinstance(name, str) or not NAME.fullmatch(name) or name in ("latest",):
+            raise CatalogError(f"bad model name {name!r}")
+        if name in names:
+            raise CatalogError(f"{name} listed twice")
+        names.add(name)
+        _check_text(e, TEXT_MAX, name)
+        if e.get("kind") not in KINDS_OF or e.get("status") not in STATUSES:
+            raise CatalogError(f"{name}: kind must be one of {KINDS_OF}, status one of {STATUSES}")
+        bench = e.get("bench", {})
+        if not isinstance(bench, dict) or any(k not in keys or not (v is None or isinstance(v, (int, float)))
+                                              or isinstance(v, bool) for k, v in bench.items()):
+            raise CatalogError(f"{name}: bench must map listed metrics to numbers")
+        if e["status"] == "builtin":
+            if "rev" in e or "files" in e:
+                raise CatalogError(f"{name}: a builtin has no rev or files")
+            continue
+        if not isinstance(e.get("format"), int) or e["format"] < 1:
+            raise CatalogError(f"{name}: format must be a positive number")
+        if not isinstance(e.get("rev"), str) or not REV.fullmatch(e["rev"]):
+            raise CatalogError(f"{name}: rev must be a full commit hash")
+        files = e.get("files")
+        if not isinstance(files, dict) or any(not isinstance(v, str) or not SHA.fullmatch(v) for v in files.values()):
+            raise CatalogError(f"{name}: files must map names to sha256s")
+        allowed = spec["files"] | spec["top"]
+        if set(files) - allowed or spec["required"] - set(files):
+            raise CatalogError(f"{name}: files must be among {sorted(allowed)} and include {sorted(spec['required'])}")
+    for key in ("latest", "baseline"):
+        v = cat.get(key)
+        if v is not None and v not in names:
+            raise CatalogError(f"{key} {v!r} isn't listed")
+    latest = cat.get("latest")
+    if latest is not None and next(e for e in models if e["name"] == latest)["status"] == "builtin":
+        raise CatalogError("latest must be a downloadable version")
+    default = cat.get("default", "latest")
+    if default != "latest" and default not in names:
+        raise CatalogError(f"default {default!r} isn't listed")
+    if default == "latest" and latest is None:
+        raise CatalogError("default is latest, but there's no latest")
+    return cat
+
+
+# --- where and which ---
+
+def store(kind, tree=REPO):
+    return Path(tree) / SPECS[kind]["store"]
+
+
+def channel(tree=REPO):
+    """The branch whose catalog an install follows: main or experimental, or None (any other branch:
+    only the copy in the checkout)."""
+    tree = Path(tree)
+    rel = tree / ".frametop-release"
+    if rel.exists():
+        ch = ""
+        for line in rel.read_text().splitlines():
+            if line.startswith("CHANNEL="):
+                ch = line.split("=", 1)[1].strip()
+        return "experimental" if ch == "experimental" else "main"
+    git = tree / ".git"
+    try:
+        if git.is_file():   # a worktree: "gitdir: PATH"
+            git = Path(git.read_text().split(":", 1)[1].strip())
+        head = (git / "HEAD").read_text().strip()
+    except (OSError, IndexError):
+        return None
+    branch = head.removeprefix("ref: refs/heads/") if head.startswith("ref: refs/heads/") else None
+    return branch if branch in ("main", "experimental") else None
+
+
+def read_conf(path=None):
+    path = Path(path) if path else Path.home() / ".config" / "frametop.conf"
+    conf = {}
+    try:
+        for line in path.read_text().splitlines():
+            line = line.split("#", 1)[0].strip()
+            if "=" in line:
+                k, v = line.split("=", 1)
+                conf[k.strip()] = v.strip().strip("\"'")
+    except OSError:
+        pass
+    return conf
+
+
+def bundled(kind, tree=REPO):
+    path = Path(tree) / SPECS[kind]["catalog"]
+    return validate(json.loads(path.read_text()), kind)
+
+
+def cached(kind):
+    """GitHub's copy from the last refresh, if it's still valid: (catalog, branch, fetched unix)."""
+    try:
+        d = json.loads((CACHE / f"{kind}.json").read_text())
+        return validate(d["catalog"], kind), d["branch"], float(d["fetched"])
+    except (OSError, ValueError, KeyError, TypeError):
+        return None
+
+
+def load(kind, tree=REPO):
+    """The catalog to show: GitHub's copy for this install's branch (from the last refresh), else the
+    one in the checkout. Returns (catalog, source), source "github BRANCH" or "this install"."""
+    ch = channel(tree)
+    c = cached(kind)
+    if c and ch and c[1] == ch:
+        return c[0], f"github {ch}"
+    return bundled(kind, tree), "this install"
+
+
+def refresh(kind, tree=REPO, timeout=15):
+    """Fetch the branch's catalog from GitHub into the cache. Returns (ok, message)."""
+    ch = channel(tree)
+    if not ch:
+        return False, "this checkout isn't on main or experimental: only its own catalog"
+    url = f"https://raw.githubusercontent.com/{GITHUB}/{ch}/{SPECS[kind]['catalog']}"
+    try:
+        with urllib.request.urlopen(url, timeout=timeout) as r:
+            data = r.read(MAX_CATALOG + 1)
+    except (urllib.error.URLError, OSError) as e:
+        return False, f"couldn't reach GitHub ({getattr(e, 'reason', e)})"
+    if len(data) > MAX_CATALOG:
+        return False, "GitHub's catalog is too big; keeping this one"
+    try:
+        cat = validate(json.loads(data), kind)
+    except (ValueError, CatalogError) as e:
+        return False, f"GitHub's catalog isn't valid for this Frametop ({e}); keeping this one"
+    CACHE.mkdir(parents=True, exist_ok=True)
+    tmp = CACHE / f".{kind}.json.tmp"
+    tmp.write_text(json.dumps({"branch": ch, "fetched": time.time(), "catalog": cat}))
+    os.replace(tmp, CACHE / f"{kind}.json")
+    return True, f"up to date with {ch}"
+
+
+def entry(cat, name):
+    return next((e for e in cat["models"] if e["name"] == name), None)
+
+
+def installed(kind, name, tree=REPO):
+    """A downloaded version's folder, if it has every file it needs and its pin, else None."""
+    if not NAME.fullmatch(name or ""):
+        return None
+    d = store(kind, tree) / name
+    if (d / "PIN.json").is_file() and all((d / f).is_file() for f in SPECS[kind]["required"]):
+        return d
+    return None
+
+
+def installed_names(kind, tree=REPO):
+    s = store(kind, tree)
+    try:
+        return sorted(p.name for p in s.iterdir() if not p.name.startswith(".") and installed(kind, p.name, tree))
+    except OSError:
+        return []
+
+
+def usable(kind, e):
+    """Can this Frametop run the entry: builtins, and versions whose format it knows."""
+    return e["status"] == "builtin" or e["format"] <= SPECS[kind]["format"]
+
+
+def setting(kind, conf=None):
+    """The kind's setting from frametop.conf (FT_ in the environment wins), or the catalog's default."""
+    key = SPECS[kind]["setting"]
+    v = os.environ.get("FT_" + key)
+    if v is None:
+        v = (conf if conf is not None else read_conf()).get(key, "")
+    return v.strip()
+
+
+# --- downloading ---
+
+def _sha(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for block in iter(lambda: f.read(1 << 20), b""):
+            h.update(block)
+    return h.hexdigest()
+
+
+def verify(kind, name, tree=REPO):
+    """Check a downloaded version's files against its pin: [] or the names that don't match."""
+    d = installed(kind, name, tree)
+    if not d:
+        return ["(not downloaded)"]
+    pin = json.loads((d / "PIN.json").read_text())
+    return [f for f, sha in pin["files"].items() if not (d / f).is_file() or _sha(d / f) != sha]
+
+
+def fetch(kind, name, tree=REPO, cat=None, progress=None, timeout=30):
+    """Download a version into the store, every file checked, then swapped in whole. Returns the
+    folder. Raises CatalogError (or OSError) when it can't."""
+    spec = SPECS[kind]
+    cat = cat or load(kind, tree)[0]
+    e = entry(cat, name)
+    if e is None:
+        raise CatalogError(f"{name} isn't in the catalog")
+    if e["status"] == "builtin":
+        raise CatalogError(f"{name} comes with Frametop: nothing to download")
+    if not usable(kind, e):
+        raise CatalogError(f"{name} needs a newer Frametop")
+    s = store(kind, tree)
+    s.mkdir(parents=True, exist_ok=True)
+    with open(s / ".lock", "w") as lock:
+        fcntl.flock(lock, fcntl.LOCK_EX)
+        d = s / name
+        if installed(kind, name, tree) and json.loads((d / "PIN.json").read_text()) == e and not verify(kind, name, tree):
+            return d
+        new = s / f".{name}.new"
+        new.mkdir(exist_ok=True)
+        for f, sha in sorted(e["files"].items()):
+            path = new / f
+            if path.is_file() and _sha(path) == sha:
+                continue
+            sub = f if f in spec["top"] else f"{name}/{f}"
+            url = f"https://huggingface.co/{spec['hf']}/resolve/{e['rev']}/{sub}"
+            if progress:
+                progress(f"downloading {f}")
+            part = new / (f + ".part")
+            try:
+                with urllib.request.urlopen(url, timeout=timeout) as r, open(part, "wb") as out:
+                    total = 0
+                    while block := r.read(1 << 20):
+                        total += len(block)
+                        if total > MAX_FILE:
+                            raise CatalogError(f"{f} is bigger than {MAX_FILE >> 20} MB")
+                        out.write(block)
+            except (urllib.error.URLError, OSError) as err:
+                part.unlink(missing_ok=True)
+                raise CatalogError(f"couldn't download {f} ({getattr(err, 'reason', err)})") from err
+            except CatalogError:
+                part.unlink(missing_ok=True)
+                raise
+            if _sha(part) != sha:
+                part.unlink()
+                raise CatalogError(f"{f}'s sha256 doesn't match the catalog; not kept")
+            os.replace(part, path)
+        (new / "PIN.json").write_text(json.dumps(e, indent=1))
+        old = s / f".{name}.old"
+        shutil.rmtree(old, ignore_errors=True)
+        if d.exists():
+            os.replace(d, old)
+        os.replace(new, d)
+        shutil.rmtree(old, ignore_errors=True)
+    return d
+
+
+def remove(kind, name, tree=REPO):
+    d = installed(kind, name, tree)
+    if d:
+        shutil.rmtree(d)
+    return bool(d)
+
+
+# --- what a setting means ---
+
+def resolve(kind, value=None, tree=REPO, cat=None):
+    """What the kind's setting runs now: {"setting", "name", "folder", "why"}. name is the version
+    (a catalog name, "latest"'s target), folder its downloaded folder or None (a builtin, or not
+    downloaded: then the runtime falls back and "why" says so). A value with a "/" is a folder of
+    one's own (for development), used as is."""
+    cat = cat or load(kind, tree)[0]
+    value = value if value is not None else setting(kind)
+    if not value:
+        value = cat.get("default", "latest")
+    if "/" in value:
+        return {"setting": value, "name": None, "folder": os.path.expanduser(value), "why": "a folder of your own"}
+    if value.lower() in ("off", "none"):
+        value = "classic" if kind == "eyes" else "stock"
+    if value == "latest":
+        name = cat.get("latest")
+        if kind == "hands":   # hands/models.sh keeps latest in hands/models/student
+            return {"setting": "latest", "name": name, "folder": None, "why": "latest (hands/models/student)"}
+        folder = installed(kind, name, tree) if name else None
+        if folder:
+            return {"setting": "latest", "name": name, "folder": str(folder), "why": "latest"}
+        # Not fetched yet: the newest custom one downloaded, while the auto update fetches it.
+        have = [e["name"] for e in cat["models"] if e["kind"] == "custom" and installed(kind, e["name"], tree)]
+        if have:
+            return {"setting": "latest", "name": have[0], "folder": str(installed(kind, have[0], tree)),
+                    "why": f"latest is {name}, not downloaded yet: using {have[0]}"}
+        return {"setting": "latest", "name": None, "folder": None, "why": f"latest is {name}, not downloaded yet"}
+    e = entry(cat, value)
+    if e is not None and e["status"] == "builtin":
+        return {"setting": value, "name": value, "folder": None, "why": e.get("title") or value}
+    folder = installed(kind, value, tree)
+    if folder:
+        return {"setting": value, "name": value, "folder": str(folder), "why": value}
+    return {"setting": value, "name": None, "folder": None, "why": f"{value} isn't downloaded"}
+
+
+def auto(kind, tree=REPO, every=15 * 60):
+    """What the services run when they start: refresh the catalog (at most every 15 minutes), then
+    fetch what the setting needs if it's missing. Not with KIND_MODELS_UPDATE=0. Returns a message."""
+    conf = read_conf()
+    upd = os.environ.get("FT_" + SPECS[kind]["update"], conf.get(SPECS[kind]["update"], "1"))
+    if upd.strip() == "0":
+        return f"{kind} models: not updating ({SPECS[kind]['update']}=0)"
+    c = cached(kind)
+    msg = ""
+    if not c or time.time() - c[2] > every:
+        msg = refresh(kind, tree)[1]
+    cat = load(kind, tree)[0]
+    value = setting(kind, conf) or cat.get("default", "latest")
+    want = cat.get("latest") if value == "latest" else value
+    if kind == "hands" and value == "latest":
+        return f"hands models: latest is hands/models.sh's ({msg})" if msg else "hands models: latest is hands/models.sh's"
+    e = entry(cat, want) if want and NAME.fullmatch(want) else None
+    if e is None or e["status"] == "builtin" or not usable(kind, e):
+        return f"{kind} models: nothing to fetch for {value!r}" + (f" ({msg})" if msg else "")
+    have = installed(kind, want, tree)
+    if have and json.loads((have / "PIN.json").read_text()) == e:
+        return f"{kind} models: {want} is here" + (f" ({msg})" if msg else "")
+    fetch(kind, want, tree, cat)
+    return f"{kind} models: fetched {want}"
+
+
+def view(kind, tree=REPO):
+    """Everything the input settings show for a kind: the catalog with each entry's state."""
+    cat, source = load(kind, tree)
+    c = cached(kind)
+    value = setting(kind) or cat.get("default", "latest")
+    have = set(installed_names(kind, tree))
+    models = []
+    for e in cat["models"]:
+        m = {k: e.get(k, "") for k in ("name", "title", "kind", "status", "date", "summary", "notes", "cost")}
+        m["bench"] = e.get("bench", {})
+        m["builtin"] = e["status"] == "builtin"
+        m["installed"] = m["builtin"] or e["name"] in have
+        m["usable"] = usable(kind, e)
+        m["size_mb"] = None
+        models.append(m)
+    listed = {e["name"] for e in cat["models"]}
+    for name in sorted(have - listed):   # downloaded once, no longer listed: still usable
+        models.append({"name": name, "title": name, "kind": "custom", "status": "older", "date": "",
+                       "summary": "No longer in the catalog; downloaded here.", "notes": "", "cost": "",
+                       "bench": {}, "builtin": False, "installed": True, "usable": True, "size_mb": None})
+    return {"kind": kind, "latest": cat.get("latest"), "default": cat.get("default", "latest"),
+            "baseline": cat.get("baseline"), "metrics": cat.get("metrics", []), "models": models,
+            "setting": value, "resolved": resolve(kind, value, tree, cat), "source": source,
+            "checked": c[2] if c else None}

models/ft-models

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""ft-models: the hand and eye model catalogs (models/catalog.py) from a terminal.
+
+  ft-models list hands|eyes [--json]    the versions, their notes, and what's in use
+  ft-models refresh hands|eyes          fetch this install's branch's catalog from GitHub
+  ft-models fetch hands|eyes NAME       download a version (every file checked against the catalog)
+  ft-models verify hands|eyes NAME      check a downloaded version's files again
+  ft-models remove hands|eyes NAME      delete a downloaded version
+  ft-models resolve hands|eyes [VALUE]  what the setting (or VALUE) runs now, as JSON
+  ft-models auto hands|eyes             what the services run at start: refresh at most every
+                                        15 minutes, fetch what the setting needs if it's missing
+  ft-models hashes hands|eyes STAGING VERSION
+                                        for a new catalog entry: the "files" of STAGING/VERSION and
+                                        STAGING's LICENSE and README.md, as uploaded (models/README.md)
+
+Pick a model in Frametop Input Settings (Hands and Gaze pages), or set HANDS_MODEL / EYES_MODEL in
+~/.config/frametop.conf: latest (follows new versions), a version's name, or a builtin (stock for
+hands; classic for eyes, and GAZE_TRACKER=steam for SteamVR's).
+"""
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import catalog  # noqa: E402
+
+
+def main(argv):
+    if len(argv) < 2 or argv[0] not in ("list", "refresh", "fetch", "verify", "remove", "resolve", "auto", "hashes") \
+            or argv[1] not in catalog.SPECS:
+        print(__doc__.strip(), file=sys.stderr)
+        return 2
+    cmd, kind, rest = argv[0], argv[1], argv[2:]
+    try:
+        if cmd == "list":
+            v = catalog.view(kind)
+            if "--json" in rest:
+                print(json.dumps(v, indent=1))
+                return 0
+            print(f"{kind} models ({v['source']}); setting {v['setting']!r} -> {v['resolved']['why']}")
+            for m in v["models"]:
+                mark = "*" if m["name"] == v["resolved"]["name"] else " "
+                latest = " (latest)" if m["name"] == v["latest"] else ""
+                state = "builtin" if m["builtin"] else "downloaded" if m["installed"] else \
+                    "needs a newer Frametop" if not m["usable"] else "not downloaded"
+                print(f"{mark} {m['name']:<16} {m['status']:<12} {state:<15} {m['title']}{latest}")
+                if m["summary"]:
+                    print(f"  {'':<16} {m['summary']}")
+            return 0
+        if cmd == "refresh":
+            ok, msg = catalog.refresh(kind)
+            print(f"{kind} catalog: {msg}")
+            return 0 if ok else 1
+        if cmd == "resolve":
+            print(json.dumps(catalog.resolve(kind, rest[0] if rest else None)))
+            return 0
+        if cmd == "auto":
+            print(catalog.auto(kind))
+            return 0
+        if cmd == "hashes" and len(rest) == 2:
+            staging, version = Path(rest[0]), rest[1]
+            spec = catalog.SPECS[kind]
+            files = {p.name: catalog._sha(p) for p in sorted((staging / version).iterdir()) if p.is_file()}
+            files.update({n: catalog._sha(staging / n) for n in sorted(spec["top"]) if (staging / n).is_file()})
+            if set(files) - spec["files"] - spec["top"] or spec["required"] - set(files):
+                print(f"ft-models: a {kind} version has {sorted(spec['files'])} (needs {sorted(spec['required'])}),"
+                      f" plus {sorted(spec['top'])} at the top; found {sorted(files)}", file=sys.stderr)
+                return 1
+            print(json.dumps({"files": files}, indent=1))
+            return 0
+        if not rest:
+            print(f"ft-models {cmd} {kind} NAME", file=sys.stderr)
+            return 2
+        name = rest[0]
+        if cmd == "fetch":
+            d = catalog.fetch(kind, name, progress=lambda s: print(s, flush=True))
+            print(f"{kind} model {name} in {d}")
+        elif cmd == "verify":
+            bad = catalog.verify(kind, name)
+            print(f"{kind} model {name}: " + ("all files match" if not bad else "doesn't match: " + ", ".join(bad)))
+            return 1 if bad else 0
+        elif cmd == "remove":
+            print(f"{kind} model {name}: " + ("removed" if catalog.remove(kind, name) else "wasn't downloaded"))
+    except (catalog.CatalogError, OSError, ValueError) as e:
+        print(f"ft-models: {e}", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

models/test/test_catalog.py

@@ -0,0 +1,236 @@
+#!/usr/bin/env python3
+"""models/catalog.py: the catalogs in the repo are valid and agree with hands/models.pin, a bad catalog
+is refused whole, downloads keep only files that match, and settings resolve as the runtimes expect.
+No network: downloads go to a fake urlopen. Run: python3 models/test/test_catalog.py"""
+import copy
+import hashlib
+import io
+import json
+import os
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE.parent))
+import catalog  # noqa: E402
+
+REPO = HERE.parents[1]
+
+
+def sha(b):
+    return hashlib.sha256(b).hexdigest()
+
+
+FILES = {"model.json": b'{"name": "eyes-t1"}', "model.ncnn.param": b"param", "model.ncnn.bin": b"weights",
+         "NOTICE": b"notice", "LICENSE": b"licence", "README.md": b"card"}
+REV = "a" * 40
+
+
+def eye_catalog(**over):
+    cat = {"catalog": 1, "kind": "eyes", "latest": "eyes-t1", "default": "classic", "baseline": "steamvr",
+           "metrics": [{"key": "fresh", "label": "Fresh", "unit": "°", "better": "lower"}],
+           "models": [
+               {"name": "classic", "title": "Classic", "kind": "classic", "status": "builtin", "bench": {"fresh": 2.8}},
+               {"name": "steamvr", "title": "SteamVR", "kind": "valve", "status": "builtin"},
+               {"name": "eyes-t1", "title": "T1", "kind": "custom", "status": "testing", "format": 1, "rev": REV,
+                "files": {k: sha(v) for k, v in FILES.items()}, "bench": {"fresh": 1.6}},
+               {"name": "eyes-t0", "title": "T0", "kind": "custom", "status": "older", "format": 1, "rev": REV,
+                "files": {k: sha(v) for k, v in FILES.items()}},
+           ]}
+    cat.update(over)
+    return cat
+
+
+class FakeHF:
+    """urlopen for huggingface.co/DeeJanuz/frametop-eyes-models/resolve/REV/...: serves FILES."""
+
+    def __init__(self, files=FILES):
+        self.files, self.urls = dict(files), []
+
+    def __call__(self, url, timeout=None):
+        self.urls.append(url)
+        prefix = f"https://huggingface.co/{catalog.SPECS['eyes']['hf']}/resolve/{REV}/"
+        assert url.startswith(prefix), url
+        name = url[len(prefix):].split("/")[-1]
+        return io.BytesIO(self.files[name])
+
+
+class RepoCatalogs(unittest.TestCase):
+    def test_bundled_valid(self):
+        for kind in catalog.SPECS:
+            cat = catalog.bundled(kind)
+            self.assertEqual(cat["kind"], kind)
+
+    def test_hands_latest_matches_models_pin(self):
+        """hands/models.sh fetches latest by models.pin; the catalog's latest must be the same files."""
+        pin = {"files": {}}
+        for line in (REPO / "hands" / "models.pin").read_text().splitlines():
+            parts = line.split()
+            if not parts or parts[0].startswith("#"):
+                continue
+            if parts[0] == "file":
+                pin["files"][parts[1]] = parts[2]
+            else:
+                pin[parts[0]] = parts[1]
+        cat = catalog.bundled("hands")
+        e = catalog.entry(cat, cat["latest"])
+        self.assertEqual((e["name"], e["rev"], e["files"]), (pin["model"], pin["rev"], pin["files"]))
+
+    def test_texts_have_no_markup(self):
+        """Shown as plain text, but keep the catalogs free of markup anyway."""
+        for kind in catalog.SPECS:
+            for e in catalog.bundled(kind)["models"]:
+                for k in ("title", "summary", "notes", "cost"):
+                    self.assertNotIn("<", e.get(k, ""), (kind, e["name"], k))
+
+
+class Validate(unittest.TestCase):
+    def bad(self, cat):
+        with self.assertRaises(catalog.CatalogError):
+            catalog.validate(cat, "eyes")
+
+    def test_good(self):
+        catalog.validate(eye_catalog(), "eyes")
+
+    def test_refusals(self):
+        c = eye_catalog()
+        self.bad(dict(c, kind="hands"))
+        self.bad(dict(c, catalog=2))
+        self.bad(dict(c, latest="classic"))          # latest must be downloadable
+        self.bad(dict(c, latest="nope"))
+        self.bad(dict(c, default="nope"))
+        for change in (
+            lambda m: m.update(name="../x"),
+            lambda m: m.update(name="a/b"),
+            lambda m: m.update(rev="abc"),
+            lambda m: m["files"].update({"run.sh": "0" * 64}),           # not a model file
+            lambda m: m["files"].update({"model.json": "xyz"}),          # not a sha256
+            lambda m: m["files"].pop("model.ncnn.bin"),                  # a required file missing
+            lambda m: m.update(bench={"unknown": 1}),
+            lambda m: m.update(bench={"fresh": "1"}),
+            lambda m: m.update(status="best"),
+            lambda m: m.update(notes="x" * 5000),
+            lambda m: m.update(format=0),
+        ):
+            c2 = copy.deepcopy(c)
+            change(c2["models"][2])
+            self.bad(c2)
+        c2 = copy.deepcopy(c)
+        c2["models"][0]["rev"] = REV                 # a builtin has nothing to fetch
+        self.bad(c2)
+        c2 = copy.deepcopy(c)
+        c2["models"].append(copy.deepcopy(c2["models"][2]))
+        self.bad(c2)                                 # listed twice
+
+
+class Store(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.tree = Path(self.tmp.name)
+        self.cat = eye_catalog()
+        env = {"XDG_CACHE_HOME": str(self.tree / "cache"), "FT_EYES_MODEL": "", "FT_EYES_MODELS_UPDATE": ""}
+        self.env = mock.patch.dict(os.environ, env)
+        self.env.start()
+        self.cache = mock.patch.object(catalog, "CACHE", self.tree / "cache" / "frametop" / "models")
+        self.cache.start()
+
+    def tearDown(self):
+        self.cache.stop()
+        self.env.stop()
+        self.tmp.cleanup()
+
+    def fetch(self, name="eyes-t1", hf=None):
+        hf = hf or FakeHF()
+        with mock.patch.object(catalog.urllib.request, "urlopen", hf):
+            return catalog.fetch("eyes", name, tree=self.tree, cat=self.cat), hf
+
+    def test_fetch_and_verify(self):
+        d, hf = self.fetch()
+        self.assertEqual(d, self.tree / "gaze/tracker/models/eyes-t1")
+        self.assertEqual({p.name for p in d.iterdir()}, set(FILES) | {"PIN.json"})
+        self.assertEqual(catalog.verify("eyes", "eyes-t1", self.tree), [])
+        self.assertTrue(any(u.endswith(f"/{REV}/LICENSE") for u in hf.urls))     # top files from the top
+        self.assertTrue(any(u.endswith(f"/{REV}/eyes-t1/model.ncnn.bin") for u in hf.urls))
+        _, again = self.fetch()                     # already there and matching: nothing downloaded
+        self.assertEqual(again.urls, [])
+        (d / "model.ncnn.bin").write_bytes(b"changed")
+        self.assertEqual(catalog.verify("eyes", "eyes-t1", self.tree), ["model.ncnn.bin"])
+
+    def test_bad_file_not_kept(self):
+        with self.assertRaises(catalog.CatalogError):
+            self.fetch(hf=FakeHF(dict(FILES, **{"model.ncnn.bin": b"tampered"})))
+        self.assertIsNone(catalog.installed("eyes", "eyes-t1", self.tree))
+        new = self.tree / "gaze/tracker/models/.eyes-t1.new"
+        self.assertFalse(any(p.name.endswith(".part") or p.name == "model.ncnn.bin" for p in new.iterdir()))
+
+    def test_builtin_and_unknown_not_fetched(self):
+        for name in ("classic", "nope"):
+            with self.assertRaises(catalog.CatalogError):
+                self.fetch(name)
+
+    def test_newer_format_not_fetched(self):
+        self.cat["models"][2]["format"] = catalog.SPECS["eyes"]["format"] + 1
+        with self.assertRaises(catalog.CatalogError):
+            self.fetch()
+
+    def test_resolve(self):
+        r = lambda v: catalog.resolve("eyes", v, self.tree, self.cat)  # noqa: E731
+        self.assertEqual(r("")["name"], "classic")              # unset: the catalog's default
+        self.assertIsNone(r("classic")["folder"])
+        self.assertIsNone(r("latest")["folder"])                 # nothing downloaded: the pupil finder
+        self.assertIsNone(r("eyes-t1")["folder"])
+        self.assertEqual(r("/some/folder")["folder"], "/some/folder")
+        self.fetch("eyes-t0")
+        self.assertEqual(r("latest")["name"], "eyes-t0")         # latest missing: a downloaded custom one
+        self.fetch("eyes-t1")
+        self.assertEqual(r("latest")["folder"], str(self.tree / "gaze/tracker/models/eyes-t1"))
+        self.assertEqual(r("eyes-t0")["name"], "eyes-t0")
+
+    def test_hands_latest_is_student(self):
+        cat = catalog.bundled("hands")
+        self.assertIsNone(catalog.resolve("hands", "latest", self.tree, cat)["folder"])
+        self.assertIsNone(catalog.resolve("hands", "stock", self.tree, cat)["folder"])
+
+    def test_channel(self):
+        self.assertIsNone(catalog.channel(self.tree))
+        (self.tree / ".git").mkdir()
+        (self.tree / ".git" / "HEAD").write_text("ref: refs/heads/experimental\n")
+        self.assertEqual(catalog.channel(self.tree), "experimental")
+        (self.tree / ".git" / "HEAD").write_text("ref: refs/heads/my-branch\n")
+        self.assertIsNone(catalog.channel(self.tree))
+        (self.tree / ".frametop-release").write_text("VERSION=x\nCHANNEL=stable\n")
+        self.assertEqual(catalog.channel(self.tree), "main")
+
+    def test_auto_off(self):
+        with mock.patch.dict(os.environ, {"FT_EYES_MODELS_UPDATE": "0"}):
+            self.assertIn("not updating", catalog.auto("eyes", self.tree))
+
+    def test_auto_fetches_setting(self):
+        (self.tree / "gaze/tracker").mkdir(parents=True)
+        (self.tree / "gaze/tracker/models.json").write_text(json.dumps(self.cat))
+        with mock.patch.dict(os.environ, {"FT_EYES_MODEL": "latest"}), \
+                mock.patch.object(catalog, "refresh", lambda kind, tree: (False, "offline")), \
+                mock.patch.object(catalog.urllib.request, "urlopen", FakeHF()):
+            self.assertIn("fetched eyes-t1", catalog.auto("eyes", self.tree))
+            self.assertIn("is here", catalog.auto("eyes", self.tree))
+
+    def test_cached_catalog_only_for_its_branch(self):
+        (self.tree / "gaze/tracker").mkdir(parents=True)
+        (self.tree / "gaze/tracker/models.json").write_text(json.dumps(self.cat))
+        newer = eye_catalog(latest="eyes-t0")
+        catalog.CACHE.mkdir(parents=True)
+        (catalog.CACHE / "eyes.json").write_text(json.dumps({"branch": "experimental", "fetched": 0, "catalog": newer}))
+        self.assertEqual(catalog.load("eyes", self.tree)[0]["latest"], "eyes-t1")   # not on a branch: its own
+        (self.tree / ".frametop-release").write_text("CHANNEL=experimental\n")
+        cat, source = catalog.load("eyes", self.tree)
+        self.assertEqual((cat["latest"], source), ("eyes-t0", "github experimental"))
+        (catalog.CACHE / "eyes.json").write_text(json.dumps({"branch": "experimental", "fetched": 0,
+                                                              "catalog": dict(newer, latest="classic")}))
+        self.assertEqual(catalog.load("eyes", self.tree)[0]["latest"], "eyes-t1")   # invalid cache: ignored
+
+
+if __name__ == "__main__":
+    unittest.main()

pack/install-release.sh

@@ -20,6 +20,7 @@
 #
 # Usage: install-release.sh [--yes] [--any-steamos] [--unpack-only] [--dir DIR]
 #                           [--no-voice] [--no-hands] [--no-hand-recorder] [--no-eye-tracker]
+#                           [--no-eye-recorder]
 #                           [--no-bluetooth | --bluetooth]
 #   --unpack-only  stop after step 2 and the copy: don't run install.sh
 #   --dir DIR      where the releases go (default ~/.local/share/frametop/releases)
@@ -113,7 +114,7 @@ main() {
       --any-steamos) any=1 ;;
       --unpack-only) unpack_only=1 ;;
       --dir) base=${2:?--dir needs a folder}; shift ;;
-      --no-voice|--no-hands|--no-hand-recorder|--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
+      --no-voice|--no-hands|--no-hand-recorder|--no-eye-tracker|--no-eye-recorder|--no-bluetooth|--bluetooth) pass+=("$1") ;;
       -h|--help) sed -n '2,27p' "$0"; return 0 ;;
       *) echo "unknown option: $1" >&2; return 2 ;;
     esac

pack/release-notes.md

@@ -1,6 +1,6 @@
 ## Install
 
-On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1.1 GB), then either:
+On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1.4 GB), then either:
 
 - **With [FrameDrop](https://framedropvr.com) on a PC:** drop Frametop.zip on your paired headset. In the headset, open Frametop in your library and press Play.
 - **On the headset:** unpack Frametop.zip (in the desktop's Dolphin or Konsole) and run `Frametop/frametop-install.sh`.

pointer/helper/ft-pointer.cpp

@@ -319,9 +319,13 @@
 // right: clicked on frametop.screen.0 (right ray, aim 100 ms before, held 0.15 s)") or why not. A middle
 // pinch (thumb to middle finger, fh_gestures v3) is a right click on a tap; held and moved, it
 // scrolls: the virtual stick deflects with the hand's displacement from the begin (0.5 cm dead zone,
-// full at 4 cm; up and down, and across the ray), times POINTER_RAY_SCROLL (1; -1 turns it round).
-// A grip with the ray up does nothing. With the ray down, pinches and grips are gaze gestures, as
-// below ("Hands"). A ray stays up while its pinch is held, and 250 ms after.
+// full at 4 cm; up and down, and across the ray), times POINTER_RAY_SCROLL (1; -1 turns it round);
+// in its first POINTER_RAY_TAP (0.3 s) only past 1.5 cm, so a right click that wobbles still clicks.
+// A three-finger pinch (the thumb on the index and middle tips, both within POINTER_GAZE_SCROLL_GAP,
+// as in gaze mode; whichever finger ft-hands calls it, at the begin or joining before anything went
+// out) scrolls the same way and never clicks (user, 2026-10-15). A middle pinch's aim lock follows
+// the thumb to middle distance. A grip with the ray up does nothing. With the ray down, pinches and
+// grips are gaze gestures, as below ("Hands"). A ray stays up while its pinch is held, and 250 ms after.
 //   Smoothing (fh_ray.h, fh_smooth.h): the ray is read at this loop's time (At), not at the tracker's
 // frames: its One Euro filters run in continuous time, so the pose sent to the driver moves every
 // loop (about 100 Hz) instead of jumping at 15-30 Hz, which the dashboard's dragger turned into jerks.
@@ -1297,8 +1301,11 @@ int main() {
         // ray that follows the pinching hand; side is the gesture's). released: the pinch ended before
         // its press went out, which then clicks (released at releaseAt). For the log: took (it took
         // the ray from the other hand), lockAge (the aim's age at the begin, s; < 0 none), the hit.
+        // rayThree: a three-finger pinch (the thumb on the index and middle tips, either kind): it
+        // scrolls as a middle pinch does, and never clicks.
         int rayKind = 0, raySlot = -1;
         bool pendingPress = false, scrolling = false, pushOn = false, aimSent = false, released = false, took = false;
+        bool rayThree = false;
         Clock::time_point pressAt{}, releaseAt{}, beganAt{};
         double lockAge = -1;
         std::string pressHit;
@@ -2543,6 +2550,7 @@ int main() {
                     f.elbow = fv(h->elbow), f.elbowConf = h->elbowConf;
                     f.grip = gripDown[s];
                     f.gap = h->pinchD[0] >= 0 ? h->pinchD[0] : Length(h->pts[4] - h->pts[8]);
+                    f.gap2 = h->pinchD[1] >= 0 ? h->pinchD[1] : Length(h->pts[4] - h->pts[12]);
                     f.curl = fhray::FistCurl(h->pts, [](Vec3 v) { return Length(v); });
                     // near: ft-screens says so (a fresh report), or the fingertip is at an overlay: a cast
                     // from 20 cm behind the index tip along the finger meets one within 25 cm (the tip
@@ -2564,7 +2572,7 @@ int main() {
                         }
                     }
                     rays[s].Frame(body, f, mayPoint(s), secsOf(tnow));
-                    rayAims[s].Add(tc, rays[s].Current(), rays[s].Shown(), h->pinchD[0]);
+                    rayAims[s].Add(tc, rays[s].Current(), rays[s].Shown(), h->pinchD[0], h->pinchD[1]);
                 }
             }
             for (int s = 0; s < 2; ++s) {
@@ -2891,6 +2899,12 @@ int main() {
             out = hin::ToRoom(head, p);
             return true;
         };
+        // The thumb on both the index and middle tips (POINTER_GAZE_SCROLL_GAP): a three-finger pinch,
+        // whichever finger ft-hands calls it (it needs the thumb nearer the middle by 1 cm for a middle pinch).
+        auto threeFingers = [&](int side) {
+            return sideGap[side][0] >= 0 && sideGap[side][0] < gazeScrollGap && sideGap[side][1] >= 0 &&
+                   sideGap[side][1] < gazeScrollGap;
+        };
         // A ray pinch (see "Hand ray" at the top). side: the gesture's; rs: the ray's (the one that follows
         // the pinching hand). The aim is locked where the ray pointed just before the fingers began to
         // close (rayowner.h), and the dead zone counts from the ray now.
@@ -2914,7 +2928,8 @@ int main() {
             };
             angles(rays[rs].Current().dir, h.refYaw, h.refPitch);
             fhray::Ray lock;
-            if (rayAims[rs].Lock(double(g.begin_ns) / 1e9, handOk && hg.end_m > 0 ? hg.end_m : 0.035, lock, h.lockAge))
+            if (rayAims[rs].Lock(double(g.begin_ns) / 1e9, handOk && hg.end_m > 0 ? hg.end_m : 0.035, lock, h.lockAge,
+                                 kind == 2 ? 1 : 0))
                 angles(lock.dir, yaw, pitch);
             else
                 yaw = h.refYaw, pitch = h.refPitch, h.lockAge = -1;
@@ -2924,11 +2939,13 @@ int main() {
             rayInactive = false;
             const bool ours_ = laserOurs();
             if (!ours_ && !claimPending && !claimHeld) claimPending = true, claimAt = tnow;
-            h.pendingPress = kind == 0;
+            h.rayThree = rayScroll != 0 && threeFingers(side);   // a scroll, not a click (user, 2026-10-15)
+            h.pendingPress = kind == 0 && !h.rayThree;
             h.pressAt = tnow + std::chrono::milliseconds(ours_ ? 0 : 30 + (claimHeld ? 0 : 8));
             hold = h;
             rays[rs].Hold(true, secsOf(tnow));
-            if (debug) std::printf("ray %s %s began\n", rs ? "right" : "left", kind ? "middle pinch" : "pinch");
+            if (debug)
+                std::printf("ray %s %s began\n", rs ? "right" : "left", h.rayThree ? "three-finger pinch" : kind ? "middle pinch" : "pinch");
             if (debug) std::fflush(stdout);
         };
         // What a ray pinch did, for its log line.
@@ -2950,16 +2967,23 @@ int main() {
         // A ray pinch ended (lost: by losing the hand, a grip, or the ray; why says which).
         auto endRayHold = [&](bool lost, const char *why) {
             const int side = hold.side, kind = hold.rayKind;
-            if (!lost && !hold.pressed && (kind == 0 || !hold.scrolling)) {
+            if (!lost && !hold.pressed && !hold.rayThree && (kind == 0 || !hold.scrolling)) {
                 // a tap before its press went out (or a middle tap): it clicks once the press can go
                 // (the press block below), along the locked aim
                 hold.released = hold.pendingPress = true;
                 return;
             }
             if (kind == 0 && hold.pressed && leftHeld) releaseLeft();
-            if (kind == 2 && hold.scrolling) SendTo(out, "ft_pointer", "scroll 0 0");
+            const bool scrolls = kind == 2 || hold.rayThree;
+            if (scrolls && hold.scrolling) SendTo(out, "ft_pointer", "scroll 0 0");
             std::string what;
-            if (kind == 2 && hold.scrolling) what = "scrolled";
+            if (hold.rayThree) {
+                char b[96];
+                std::snprintf(b, sizeof b, "three-finger pinch: %s (stick up to %.2f)%s%s",
+                              hold.scrolling ? "scrolled" : "no scroll, it didn't move", hold.scrollMax, lost ? ": " : "",
+                              lost ? why : "");
+                what = b;
+            } else if (scrolls && hold.scrolling) what = "scrolled";
             else if (!hold.pressed) what = std::string("no click: ") + why;
             else if (lost) what = "pressed on " + hitName(hold.pressHit) + ", let go: " + why;
             else what = (hold.engaged ? "dragged from " : "clicked on ") + hitName(hold.pressHit);
@@ -3095,7 +3119,7 @@ int main() {
                 // no laser up: no press (POINTER_PINCH_NO_RAY); the first one wakes the pointer
                 handUsed = tnow;
                 if (!active) wake(tnow);
-                if (kind == 0) pinchLog(s, kind, "ignored (no ray up: bring thumb and index together to raise the laser)");
+                if (kind == 0) pinchLog(s, kind, "ignored (no ray up: bring the thumb to the index or middle tip to raise the laser)");
             } else {
                 beginHold(s, kind == 1, g);
             }
@@ -3151,6 +3175,8 @@ int main() {
                         const int other = k == 0 ? 2 : k == 2 ? 0 : -1;
                         if (hold.gazeScroll && other >= 0 && (slots[other][s].flags & FH_PINCH_DOWN)) {
                             hold.right = other == 2;   // a three-finger pinch: ft-hands calls the other finger now; it goes on
+                        } else if (hold.src == Src::Ray && hold.rayThree && other >= 0 && (slots[other][s].flags & FH_PINCH_DOWN)) {
+                            hold.rayKind = other;      // (the same on the ray: both kinds' point is the knuckles)
                         } else if (hold.pressed && hold.second >= 0 && hold.second != s) {
                             // the two-hand press (see gazeTwoHand): the other hand's pinch still holds it, and drags now
                             hold.side = hold.second, hold.right = hold.secondKind == 2, hold.second = -1;
@@ -3164,6 +3190,7 @@ int main() {
                     for (PendingBegin &p : pendingBegins)
                         if (p.side == s && p.kind == k && ended) p.ended = true, p.lost = lost;
                     if (!began) continue;
+                    if (k != 1 && hold.src == Src::Ray && hold.rayThree && hold.side == s) continue;   // its other finger
                     if (!allowed) {
                         if (k != 1) pinchLog(s, k, "ignored (hands not allowed: a VR game, or the headset off)");
                         continue;
@@ -3271,16 +3298,30 @@ int main() {
                     }
                 }
             }
-            // A middle pinch held and moved: scrolling, the stick deflected by the hand's displacement
-            // from the begin (0.5 cm dead zone, full at 4 cm): up and down, and across the ray.
-            if (hold.src == Src::Ray && hold.rayKind == 2) {
-                const fh_pinch_t &g = hg.pinch2[hold.side];
+            // The other finger joining a ray pinch just begun, before anything went out (no press, no
+            // scroll, not a tap already over): a three-finger scroll from here.
+            if (hold.src == Src::Ray && !hold.rayThree && !hold.pressed && !hold.scrolling && !hold.released &&
+                (hold.rayKind == 2 || hold.pendingPress) && rayScroll != 0 &&
+                tnow - hold.beganAt < std::chrono::milliseconds(400) && threeFingers(hold.side)) {
+                const fh_pinch_t &g = hold.rayKind == 2 ? hg.pinch2[hold.side] : hg.pinch[hold.side];
+                if ((g.flags & FH_PINCH_TRACKED) && rayRoom(g.point, hg.capture_ns, hold.beginPoint)) {
+                    hold.rayThree = true, hold.pendingPress = false;
+                    if (debug) std::printf("ray %s: the other finger joined, a three-finger pinch\n", hold.raySlot ? "right" : "left");
+                }
+            }
+            // A middle or three-finger pinch held and moved: scrolling, the stick deflected by the hand's
+            // displacement from the begin (0.5 cm dead zone, full at 4 cm): up and down, and across the ray.
+            // A middle pinch's first rayTap (0.3 s) is a tap's: it scrolls only past 1.5 cm then, so a
+            // right click that wobbles still clicks (2026-10-15: two of the user's turned into scrolls).
+            if (hold.src == Src::Ray && (hold.rayKind == 2 || hold.rayThree)) {
+                const fh_pinch_t &g = hold.rayKind == 2 ? hg.pinch2[hold.side] : hg.pinch[hold.side];
                 Vec3 now;
                 if ((g.flags & FH_PINCH_TRACKED) && rayRoom(g.point, hg.capture_ns, now)) {
                     const Vec3 d = now - hold.beginPoint;
                     const Vec3 right = Normalize(Cross(rayDir, {0, 1, 0}));
                     const double dx = Dot(d, right), dy = d.y;
-                    if (!hold.scrolling && std::hypot(dx, dy) > 0.005) hold.scrolling = true;
+                    const bool tap = !hold.rayThree && tnow - hold.beganAt < std::chrono::duration<double>(rayTap);
+                    if (!hold.scrolling && std::hypot(dx, dy) > (tap ? 0.015 : 0.005)) hold.scrolling = true;
                     if (hold.scrolling) {
                         auto axis = [&](double v) {
                             const double a = std::clamp((std::fabs(v) - 0.005) / 0.035, 0.0, 1.0) * rayScroll;
@@ -3289,6 +3330,7 @@ int main() {
                         const double sx = std::clamp(axis(dx), -1.0, 1.0), sy = std::clamp(axis(dy), -1.0, 1.0);
                         if (sx != hold.scrollX || sy != hold.scrollY) {
                             hold.scrollX = sx, hold.scrollY = sy;
+                            hold.scrollMax = std::max(hold.scrollMax, std::max(std::fabs(sx), std::fabs(sy)));
                             char msg[64];
                             std::snprintf(msg, sizeof msg, "scroll %.2f %.2f", sx, sy);
                             SendTo(out, "ft_pointer", msg);
@@ -3376,7 +3418,7 @@ int main() {
 
         // The driving ray (see "Hand ray" at the top): the pointer's origin and direction. During a ray
         // pinch the direction holds still until the ray has turned rayDeadzone, then follows (steer);
-        // a middle pinch holds it. Push and pull move the origin along the ray while a carry is held.
+        // a middle or three-finger pinch holds it. Push and pull move the origin along the ray while a carry is held.
         // rayMode: a hand's ray has the pointer, driving it or (its owner's grace, rayowner.h) holding
         // its last pose.
         const bool rayDrives = active && raySide >= 0 && rays[raySide].Shown();
@@ -3390,7 +3432,7 @@ int main() {
             if (hold.src == Src::Ray && hold.raySlot == raySide) {
                 // a tap's dead zone first: a quick pinch whose hand turned a little still clicks
                 const bool tap = tnow - hold.beganAt < std::chrono::duration<double>(rayTap);
-                if (hold.rayKind == 0) steer(ry, rp, tap ? std::max(rayDeadzone, rayTapDeadzone) : rayDeadzone, 1.0);
+                if (hold.rayKind == 0 && !hold.rayThree) steer(ry, rp, tap ? std::max(rayDeadzone, rayTapDeadzone) : rayDeadzone, 1.0);
             } else {
                 yaw = ry, pitch = std::clamp(rp, -85.0, 85.0);
             }

pointer/helper/rayowner.h

@@ -28,8 +28,8 @@
 // ft-hands has called it the other side for flipHold (0.3 s), or it's gone; then it takes the hand
 // called its side that the other ray doesn't follow.
 //
-// Aim lock: where a ray pinch clicks. ft-hands sees a pinch once the thumb and index tips are
-// within begin_m, and closing them turns the hand (1.3-1.6 deg median in the first 150 ms after the
+// Aim lock: where a ray pinch clicks. ft-hands sees a pinch once the thumb and index tips (the
+// middle's for a middle pinch) are within begin_m, and closing them turns the hand (1.3-1.6 deg median in the first 150 ms after the
 // begin, 2.4-3.5 at the 90th percentile, on recordings; more before it). So the press goes out
 // along the ray as it was just before the fingers began to close: the newest frame, within lockBack
 // (0.35 s) before the begin, with the tips still lockOpen (1 cm) further apart than the pinch's
@@ -217,22 +217,24 @@ private:
 class AimHistory {
 public:
     explicit AimHistory(Params p = {}) : p_(p) {}
-    // A hands frame at capture time t: the ray (filtered), whether it's up, and the thumb to index
-    // distance (m; < 0 unknown).
-    void Add(double t, const fhray::Ray &ray, bool shown, double pinchD) {
-        h_.push_back({t, ray, shown, pinchD});
+    // A hands frame at capture time t: the ray (filtered), whether it's up, and the thumb to index and
+    // thumb to middle distances (m; < 0 unknown).
+    void Add(double t, const fhray::Ray &ray, bool shown, double pinchD, double pinchD2 = -1) {
+        h_.push_back({t, ray, shown, {pinchD, pinchD2}});
         while (!h_.empty() && h_.front().t < t - 1.0) h_.pop_front();
     }
     // The ray to press along for a pinch that began at tBegin (see the top); endD: the pinch's release
-    // distance (end_m). False: no frame with the ray up within lockBack. age: how long before the begin.
-    bool Lock(double tBegin, double endD, fhray::Ray &out, double &age) const {
+    // distance (end_m); finger: 0 the index pinched, 1 the middle. False: no frame with the ray up
+    // within lockBack. age: how long before the begin.
+    bool Lock(double tBegin, double endD, fhray::Ray &out, double &age, int finger = 0) const {
         const double openD = endD + p_.lockOpen;
+        const int fi = finger == 1 ? 1 : 0;
         const S *open = nullptr, *lead = nullptr;
         for (auto it = h_.rbegin(); it != h_.rend(); ++it) {
             if (it->t > tBegin + 1e-6) continue;
             if (it->t < tBegin - p_.lockBack - 1e-6) break;
             if (!it->shown) continue;
-            if (!open && it->d >= openD) open = &*it;
+            if (!open && it->d[fi] >= openD) open = &*it;
             if (!lead && it->t <= tBegin - p_.lockLead + 1e-6) lead = &*it;
         }
         const S *pick = open ? open : lead;
@@ -254,7 +256,7 @@ private:
         double t;
         fhray::Ray ray;
         bool shown;
-        double d;
+        double d[2];
     };
     Params p_;
     std::deque<S> h_;

pointer/tests/handray-test.cpp

@@ -450,6 +450,26 @@ void TestReadyAndFist() {
         double t = 0;
         CHECK(RunUntil(r, b, t, 1.0, true, Pointing) >= 0, "no gap or curl known: the old gate");
     }
+    {   // the middle counts too (user, 2026-10-15): its right click and the three-finger scroll
+        auto middle = [](double gap, double gap2) {
+            return [gap, gap2](double tt) {
+                HandFrame f = Pointing(tt);
+                f.gap = gap, f.gap2 = gap2, f.curl = 1.8;
+                return f;
+            };
+        };
+        Body b;
+        HandRay r(1);
+        double t = 0;
+        CHECK(RunUntil(r, b, t, 1.0, true, middle(0.12, 0.13)) < 0, "both 12-13 cm away: no laser");
+        CHECK(RunUntil(r, b, t, 1.0, true, middle(0.12, 0.04)) >= 0, "the thumb 4 cm from the middle tip: the laser");
+        CHECK(RunUntil(r, b, t, 0.6, false, middle(0.12, 0.08)) < 0, "8 cm from it (under readyHide): it stays");
+        CHECK(RunUntil(r, b, t, 1.0, false, middle(0.14, 0.14)) >= 0, "both past readyHide: it goes");
+        Body b2;
+        HandRay o(1);
+        t = 0;
+        CHECK(RunUntil(o, b2, t, 1.0, true, middle(-1, 0.04)) >= 0, "only the middle's gap known: it counts");
+    }
 }
 
 int main() {

pointer/tests/rayowner-test.cpp

@@ -290,6 +290,16 @@ void TestAimLock() {
     AimHistory f;
     for (double t = 1.0; t <= begin + 1e-9; t += kFrame) f.Add(t, RayAt(t), t > begin - 0.07, 0.01);
     CHECK(f.Lock(begin, 0.035, r, age) && age <= 0.07 + 1e-9 && age > 0.03, "a ray just up: its oldest frame (%.3f)", age);
+    // a middle pinch: the thumb closing on the middle tip with the index held out (its distance
+    // never closes): the lock follows the middle's distance, as the index's for a pinch
+    AimHistory m;
+    for (double t = 1.0; t <= begin + 1e-9; t += kFrame) {
+        const double closing = std::clamp((t - (begin - 0.15)) / 0.15, 0.0, 1.0);
+        m.Add(t, RayAt(-3 * closing), true, 0.07, 0.06 - 0.04 * closing);
+    }
+    CHECK(m.Lock(begin, 0.035, r, age, 1) && std::fabs(YawDeg(r)) < 1.3 && age > 0.05 && age < 0.15,
+          "a middle pinch: from before the middle closed (%.2f deg, %.3f s)", YawDeg(r), age);
+    CHECK(m.Lock(begin, 0.035, r, age, 0) && age < kFrame, "by the index's distance: the begin's own frame (%.3f s)", age);
 }
 
 // fh_ray's ShowForPinch: a pinch brings back a ray that just dropped, not one sent away.

recorder/tests/test_age.py

@@ -76,7 +76,7 @@ class AgeTest(unittest.TestCase):
 
     def test_files_and_recipients(self):
         rec = age.read_recipients(os.path.join(HERE, "..", "..", "gaze", "rec", "recipients.txt"))
-        self.assertEqual(len(rec), 2)
+        self.assertEqual(len(rec), 3)              # keys 1 (7i), 2 (offline), 3 (dody)
         with tempfile.TemporaryDirectory() as d:
             src, dst = os.path.join(d, "a"), os.path.join(d, "a.age")
             with open(src, "wb") as f:

screens/build.sh

@@ -25,6 +25,7 @@ $cc -c -o build/remote.o remote.c
 cxx="g++ -std=c++17 -O2 -Wall -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags egl glesv2 gbm libdrm)"
 $cxx -c -o build/vr.o vr.cpp
 $cxx -c -o build/keyboard.o keyboard.cpp
+$cxx -c -o build/overlaytex.o overlaytex.cpp
 $cxx -c -o build/handcut.o handcut.cpp
 $cxx -c -o build/handrender.o handrender.cpp
 $cxx -c -o build/handinput.o handinput.cpp
@@ -37,7 +38,7 @@ $cxx -c -o build/handskin.o handskin.cpp
 $cxx -c -o build/handrobot.o handrobot.cpp
 $cxx -c -o build/handview.o handview.cpp
 vrlibs="$(pkg-config --libs egl glesv2 gbm) $OPENVR_LIBS"
-g++ -o build/ft-screens build/compositor.o build/remote.o build/vr.o build/keyboard.o build/handcut.o build/handrender.o build/handinput.o build/palmmenu.o build/grabthat.o build/steampanels.o \
+g++ -o build/ft-screens build/compositor.o build/remote.o build/vr.o build/keyboard.o build/overlaytex.o build/handcut.o build/handrender.o build/handinput.o build/palmmenu.o build/grabthat.o build/steampanels.o \
   $(pkg-config --libs wlroots-0.20 wayland-server xkbcommon pixman-1) $vrlibs
 g++ -o build/ft-handtest build/handtest.o build/handcut.o build/handrender.o $vrlibs
 g++ -o build/ft-handview build/handview.o build/handmesh.o build/handskin.o build/handrobot.o build/handcut.o build/handrender.o $vrlibs

screens/grabthat.cpp

@@ -136,7 +136,7 @@ void Detector::Frame(int64_t t, const std::vector<Hand> &hands, const Mat &head,
         if (h->id != sd.handId) sd.recent.clear();
         sd.lastSeen = t, sd.handId = h->id, sd.head = head;
         const Features f = Measure(*h, head);
-        sd.thumb = Dist3(h->p[4], h->p[8]);
+        sd.thumb = std::min(Dist3(h->p[4], h->p[8]), Dist3(h->p[4], h->p[12]));   // (the nearer: either pinch)
         sd.recent.push_back({t, f.fwd, f.palmFwd});
         while (!sd.recent.empty() && t - sd.recent.front().t > int64_t(p_.pullMs * 1e6)) sd.recent.erase(sd.recent.begin());
         double palmNow[3];

screens/grabthat.h

@@ -37,13 +37,15 @@
 // pulling back never comes back far enough, a pinch keeps the other fingers open, and typing or a
 // hand in the lap is never held out with the palm forward.
 //
-// The claim: in Closing, on frames where the thumb tip is claimThumb or more from the index tip,
-// and for claimMs after a fire, the hand is ours. vr.cpp reports it near ("near ... <L> <R>" to the
+// The claim: in Closing, on frames where the thumb tip is claimThumb or more from the index and
+// the middle tips, and for claimMs after a fire, the hand is ours. vr.cpp reports it near ("near ... <L> <R>" to the
 // pointer helper, as the palm menu does), so the grip ft-hands sees as the fist closes isn't a
 // gaze grip (a press and drag at the pointer). ft-hands' grip begins with every finger under 1.2,
 // after the three fingers' mean is under claimCurl (1.5), and with the thumb 3 cm or more from the
 // index tip; a pinch begins under 2 cm, so a pinch made from a hand held out (a ray click with the
-// other fingers curled) stays the helper's. On recordings without the gesture (probe-moves, the
+// other fingers curled) stays the helper's, and so does a middle pinch (its right click: the ring and
+// little curl with the middle; 2026-10-15, the user's right clicks were left to ft-screens). The
+// user's flag keeps the thumb 3.5 cm or more from the middle tip as the fist closes. On recordings without the gesture (probe-moves, the
 // 2026-09-29 and lit recordings, the PR #3 takes: 15 minutes), 1 of 143 pinches and 3 of 12 grips
 // began on a claimed hand (11 pinches without the thumb rule), and nothing fired (one did before
 // the palm rule above: an open palm turned over, then closed and pulled in); nor in the user's
@@ -76,7 +78,7 @@ struct Params {
     double armMs = 200;
     double closeMs = 450;
     double claimCurl = 1.5;              // the middle, ring and little fingers' mean extension
-    double claimThumb = 0.025;           // ...with the thumb tip this far (m) from the index tip: not a pinch
+    double claimThumb = 0.025;           // ...with the thumb tip this far (m) from the index and middle tips: not a pinch
     double closePalm = 0.05, closePalmMs = 100;
     double fist = 1.0;
     double pull = 0.08, pullMs = 500;    // m, within ms
@@ -183,7 +185,7 @@ private:
         int64_t since = 0, lastSeen = 0, lastOpen = 0, quietUntil = 0, claimUntil = 0;
         double eye[3] = {0, 0, 0}, palm[3] = {0, 0, 0}, normal[3] = {0, 0, 0};
         Mat head{};                   // the head on the last frame
-        double thumb = 0;             // thumb tip to index tip now (m)
+        double thumb = 0;             // thumb tip to the nearer of the index and middle tips now (m)
         double holdFrom[3] = {0, 0, 0};   // the arming hold: where the palm was as it began,
         double facingSum = 0;             // ...its facing off the aim summed over its frames
         int facingN = 0;

screens/overlaytex.cpp

@@ -0,0 +1,141 @@
+// See overlaytex.h.
+#include "overlaytex.h"
+
+#include <drm_fourcc.h>
+#include <fcntl.h>
+#include <gbm.h>
+#include <unistd.h>
+
+#include <cstdio>
+#include <cstring>
+#include <map>
+#include <tuple>
+
+namespace overlaytex {
+namespace {
+
+struct Image {
+    gbm_bo *bo = nullptr;
+    int fd = -1;
+    vr::SharedTextureHandle_t handle = 0;
+    int users = 0;         // overlays showing it
+    uint64_t used = 0;     // when last put on one (a counter)
+};
+using Key = std::tuple<int, int, uint64_t>;   // w, h, the pixels' hash
+
+int g_drm = -1;
+gbm_device *g_gbm = nullptr;
+int g_state = 0;   // 0: not tried yet, 1: buffers, -1: SetOverlayRaw
+std::map<Key, Image> g_images;
+std::map<vr::VROverlayHandle_t, Key> g_on;   // what each overlay shows
+uint64_t g_clock = 0;
+
+uint64_t Hash(const uint8_t *px, size_t n) {   // FNV-1a, 8 bytes at a time
+    uint64_t h = 1469598103934665603ull;
+    size_t i = 0;
+    for (; i + 8 <= n; i += 8) {
+        uint64_t w;
+        std::memcpy(&w, px + i, 8);
+        h = (h ^ w) * 1099511628211ull;
+    }
+    for (; i < n; ++i) h = (h ^ px[i]) * 1099511628211ull;
+    return h;
+}
+
+void Drop(Image &im) {
+    if (im.handle) vr::VRIPCResourceManager()->UnrefResource(im.handle);
+    if (im.fd >= 0) close(im.fd);
+    if (im.bo) gbm_bo_destroy(im.bo);
+    im = Image{};
+}
+
+bool Ready() {
+    if (g_state) return g_state > 0;
+    g_drm = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC);
+    if (g_drm >= 0) g_gbm = gbm_create_device(g_drm);
+    g_state = g_gbm ? 1 : -1;
+    if (g_state < 0) std::printf("overlay textures: no render node; using SetOverlayRaw (at most 200 changes)\n");
+    return g_state > 0;
+}
+
+bool Make(Image &im, const uint8_t *px, int w, int h) {
+    im.bo = gbm_bo_create(g_gbm, uint32_t(w), uint32_t(h), GBM_FORMAT_ABGR8888, GBM_BO_USE_RENDERING | GBM_BO_USE_LINEAR);
+    if (!im.bo || (im.fd = gbm_bo_get_fd(im.bo)) < 0) return false;
+    uint32_t stride = 0;
+    void *mapping = nullptr;
+    auto *dst = static_cast<uint8_t *>(gbm_bo_map(im.bo, 0, 0, uint32_t(w), uint32_t(h), GBM_BO_TRANSFER_WRITE, &stride, &mapping));
+    if (!dst) return false;
+    for (int y = 0; y < h; ++y) std::memcpy(dst + size_t(y) * stride, px + size_t(y) * w * 4, size_t(w) * 4);   // (ABGR8888: R, G, B, A in memory)
+    gbm_bo_unmap(im.bo, mapping);
+    vr::DmabufAttributes_t a{};
+    a.unWidth = uint32_t(w), a.unHeight = uint32_t(h);
+    a.unDepth = a.unMipLevels = a.unArrayLayers = a.unSampleCount = 1;
+    a.unFormat = DRM_FORMAT_ABGR8888;
+    a.ulModifier = DRM_FORMAT_MOD_LINEAR;
+    a.unPlaneCount = 1;
+    a.plane[0].unOffset = gbm_bo_get_offset(im.bo, 0);
+    a.plane[0].unStride = gbm_bo_get_stride(im.bo);
+    a.plane[0].nFd = im.fd;
+    return vr::VRIPCResourceManager()->ImportDmabuf(vr::VRApplication_Overlay, &a, &im.handle) && im.handle;
+}
+
+void Release(vr::VROverlayHandle_t o) {
+    auto on = g_on.find(o);
+    if (on == g_on.end()) return;
+    auto it = g_images.find(on->second);
+    if (it != g_images.end() && it->second.users > 0) --it->second.users;
+    g_on.erase(on);
+}
+
+// Past kKeep images: drop the least recently used no overlay shows.
+void Trim() {
+    while (int(g_images.size()) > kKeep) {
+        auto old = g_images.end();
+        for (auto it = g_images.begin(); it != g_images.end(); ++it)
+            if (!it->second.users && (old == g_images.end() || it->second.used < old->second.used)) old = it;
+        if (old == g_images.end()) return;
+        Drop(old->second);
+        g_images.erase(old);
+    }
+}
+
+}  // namespace
+
+vr::EVROverlayError Set(vr::VROverlayHandle_t o, const uint8_t *px, int w, int h) {
+    if (o == vr::k_ulOverlayHandleInvalid) return vr::VROverlayError_UnknownOverlay;
+    if (!Ready()) return vr::VROverlay()->SetOverlayRaw(o, const_cast<uint8_t *>(px), uint32_t(w), uint32_t(h), 4);
+    const Key key{w, h, Hash(px, size_t(w) * h * 4)};
+    auto it = g_images.find(key);
+    if (it == g_images.end()) {
+        Image im;
+        if (!Make(im, px, w, h)) {
+            std::fprintf(stderr, "overlay textures: a %dx%d buffer failed; SetOverlayRaw this once\n", w, h);
+            Drop(im);
+            Release(o);
+            return vr::VROverlay()->SetOverlayRaw(o, const_cast<uint8_t *>(px), uint32_t(w), uint32_t(h), 4);
+        }
+        it = g_images.emplace(key, im).first;
+    }
+    const auto on = g_on.find(o);
+    if (on == g_on.end() || on->second != key) {
+        Release(o);
+        g_on[o] = key, ++it->second.users;
+    }
+    it->second.used = ++g_clock;
+    vr::Texture_t tex = {&it->second.handle, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
+    const vr::EVROverlayError err = vr::VROverlay()->SetOverlayTexture(o, &tex);
+    Trim();
+    return err;
+}
+
+void Forget(vr::VROverlayHandle_t o) { Release(o); }
+
+void Shutdown() {
+    for (auto &[k, im] : g_images) Drop(im);
+    g_images.clear(), g_on.clear();
+    if (g_gbm) gbm_device_destroy(g_gbm);
+    if (g_drm >= 0) close(g_drm);
+    g_gbm = nullptr, g_drm = -1, g_state = 0;
+}
+
+}  // namespace overlaytex

screens/overlaytex.h

@@ -0,0 +1,31 @@
+// Small CPU-drawn textures on ft-screens' overlays (the controls, the glows, the carousel) without
+// SetOverlayRaw. SteamVR never gives back the memory block each SetOverlayRaw call takes: the 201st
+// call in a process fails ("Refusing to create memory block because 201 blocks are already
+// outstanding"), and the overlay keeps its old texture at its new size and place. That was the
+// glow bug of 2026-10-15: after about 40 minutes of hand use, a corner's glow showed as the
+// fingertip dot's disc, as big as the corner's glow, and "grab that"'s halo kept the last panel's
+// shape. Measured with a probe: hidden, shown or transparent, destroyed or not, 20 s apart or
+// not, every call keeps its block.
+//   Here each distinct image goes into a linear DMA-BUF SteamVR imports once (as the keyboard's
+// buffers, keyboard.cpp), kept by its content, and the overlay switches to it with
+// SetOverlayTexture, which has no such limit (600 switches, and 600 imports dropped again, fine).
+// Images no overlay shows are dropped, the least recently used first, past kKeep of them.
+#pragma once
+
+#include <openvr.h>
+
+#include <cstdint>
+
+namespace overlaytex {
+
+constexpr int kKeep = 64;
+
+// w x h pixels, RGBA, straight alpha (as SetOverlayRaw took them), onto overlay o. Falls back to
+// SetOverlayRaw if the buffers can't be made.
+vr::EVROverlayError Set(vr::VROverlayHandle_t o, const uint8_t *px, int w, int h);
+// The overlay is going (before DestroyOverlay): its image may be dropped.
+void Forget(vr::VROverlayHandle_t o);
+// Every image dropped (at shutdown, after the overlays are gone).
+void Shutdown();
+
+}  // namespace overlaytex

screens/tests/handinput_test.cpp

@@ -154,6 +154,7 @@ struct SynthHand {
     double tiltDeg = 20;               // fingertips up: the palm turns toward the head behind it
     int curl = 0;                      // middle, ring and little: 0 open, 1 half curled, 2 a fist
     int curlIndex = 0;                 // ...and the index, likewise (grab that's fist)
+    bool thumbOnMiddle = false;        // ...the thumb tip on the middle's (a middle pinch)
     double curlT = -1, curlIndexT = -1;   // ...or in between (0..2: open, half, fist), a hand closing
     // a pointing hand (PointingHand) whose middle finger the tracker has nearly straight: its tip
     // this much deeper (toward where the index points) than the index tip (NaN: curled)
@@ -192,6 +193,7 @@ void PalmHand(const SynthHand &h, float out[21][3]) {
     if (h.curlT >= 0)
         for (int f = 0; f < 3; ++f) mix(&base[10 + 4 * f], half[f], fist[f], std::clamp(h.curlT, 0.0, 2.0), &out[10 + 4 * f]);
     if (h.curlIndexT >= 0) mix(&base[6], halfIndex, fistIndex, std::clamp(h.curlIndexT, 0.0, 2.0), &out[6]);
+    if (h.thumbOnMiddle) out[4][0] = out[12][0] + 0.008f, out[4][1] = out[12][1], out[4][2] = out[12][2];
     const double r = h.rollDeg * M_PI / 180, a = h.tiltDeg * M_PI / 180;
     double c[3] = {0, 0, 0};
     for (int j = 0; j < 21; ++j) {
@@ -3160,6 +3162,23 @@ void TestGrabThatGesture() {
         for (int k = 1; k <= 6; ++k) q.Frame({StopHand(0.1, -0.15, -0.42 + 0.025 * k, k / 6.0, 2)});
         CHECK(q.fires.empty());
     }
+    {   // a middle pinch from a hand held out (a ray's right click, the other two curled with it): the
+        // pointer helper's, not claimed (2026-10-15: the user's right clicks were "left to ft-screens")
+        auto closing = [](bool middlePinch) {
+            GrabRig r;
+            for (int k = 0; k < 10; ++k) r.Frame({StopHand(0.1, -0.15, -0.42)});
+            for (int k = 0; k < 6; ++k) {
+                SynthHand h = StopHand(0.1, -0.15, -0.42, 0, 1);
+                h.thumbOnMiddle = middlePinch;
+                r.Frame({h});
+            }
+            return r.claimed[1];
+        };
+        const int fist = closing(false), pinch = closing(true);
+        std::printf("grab that: the three curled from a hand held out claim %d frames, with the thumb on the middle %d\n",
+                    fist, pinch);
+        CHECK(fist >= 2 && pinch == 0);
+    }
     {   // a wave: held out open, palm forward, swinging 15 cm either way at 1.5 Hz for 3 s
         GrabRig r;
         for (int k = 0; k < 90; ++k) {

screens/vr.cpp

@@ -107,6 +107,7 @@
 #include "grabthat.h"
 #include "handinput.h"
 #include "softglow.h"
+#include "overlaytex.h"
 #include "palmmenu.h"
 #include "keyboard.h"
 #include "roomview.h"
@@ -584,7 +585,7 @@ vr::VROverlayHandle_t MakeChrome(const char *key, const char *name, const std::v
         std::fprintf(stderr, "openvr: can't create overlay %s: %s\n", key, vr::VROverlay()->GetOverlayErrorNameFromEnum(err));
         return o;
     }
-    vr::VROverlay()->SetOverlayRaw(o, const_cast<uint8_t *>(px.data()), uint32_t(w), uint32_t(h), 4);
+    overlaytex::Set(o, px.data(), w, h);
     vr::VROverlay()->SetOverlayInputMethod(o, vr::VROverlayInputMethod_Mouse);
     // SteamVR's laser and ComputeOverlayIntersection size the hit area from the mouse scale,
     // not the texture: at the default 1x1 it's a square as tall as the control is wide, so the
@@ -599,7 +600,7 @@ void LightBar(Screen &s, bool lit) {
     if (s.barLit == lit) return;
     s.barLit = lit;
     const auto &px = BarTexture(lit);
-    vr::VROverlay()->SetOverlayRaw(s.bar, const_cast<uint8_t *>(px.data()), 256, 24, 4);
+    overlaytex::Set(s.bar, px.data(), 256, 24);
 }
 
 // ---------------------------------------------------------------- wrist guides
@@ -646,7 +647,7 @@ struct GuidePart {
     void Light(bool on, const std::vector<uint8_t> &px, int n) {
         if (int(on) == lit) return;
         lit = on;
-        vr::VROverlay()->SetOverlayRaw(overlay, const_cast<uint8_t *>(px.data()), uint32_t(n), uint32_t(n), 4);
+        overlaytex::Set(overlay, px.data(), n, n);
     }
 };
 struct Guide { GuidePart ring, dot; };
@@ -1172,7 +1173,7 @@ void UpdateCorner(Screen &s, const Mat &p, const std::vector<Mat> &lasers) {
     if (lit != s.cornerLit && s.handle != vr::k_ulOverlayHandleInvalid) {
         s.cornerLit = lit;
         const auto &px = HandleTexture(lit);
-        vr::VROverlay()->SetOverlayRaw(s.handle, const_cast<uint8_t *>(px.data()), 64, 64, 4);
+        overlaytex::Set(s.handle, px.data(), 64, 64);
     }
     if (std::fabs(glow - s.cornerGlow) > 0.02f || (glow == 0) != (s.cornerGlow == 0)) {
         s.cornerGlow = glow;
@@ -1653,7 +1654,7 @@ bool EnsureFloatPanel(Screen &s) {
 
 void DropFloatPanel(Screen &s) {
     for (auto o : s.All())
-        if (o != vr::k_ulOverlayHandleInvalid) vr::VROverlay()->DestroyOverlay(o);
+        if (o != vr::k_ulOverlayHandleInvalid) overlaytex::Forget(o), vr::VROverlay()->DestroyOverlay(o);
     s.overlay = s.bar = s.handle = s.curveButton = s.rollButton = s.dockButton = s.closeButton = s.resetButton =
         vr::k_ulOverlayHandleInvalid;
     s.visible = s.controlsUp = s.barLit = s.cornerLit = false;
@@ -3440,7 +3441,7 @@ void ShowCarousel() {
         vr::VROverlay()->CreateOverlay("frametop.carousel", "Frametop: screens", &g_carousel.overlay) != vr::VROverlayError_None)
         return;
     const vr::VROverlayHandle_t o = g_carousel.overlay;
-    vr::VROverlay()->SetOverlayRaw(o, px.data(), uint32_t(W), uint32_t(H), 4);
+    overlaytex::Set(o, px.data(), W, H);
     // under the screen in the slot, past its bar: 0.45 mm a pixel at 1 m (30 px text: about 0.8 deg)
     const Screen &s = g_screens[g_front.panel];
     const double d = std::max(0.3, Apart(head, g_front.to));
@@ -3520,7 +3521,7 @@ void ShowResetAsk(int panel) {
     if (g_resetAsk.overlay == vr::k_ulOverlayHandleInvalid &&
         vr::VROverlay()->CreateOverlay("frametop.resetask", "Frametop: reset the layout?", &g_resetAsk.overlay) != vr::VROverlayError_None)
         return;
-    vr::VROverlay()->SetOverlayRaw(g_resetAsk.overlay, px.data(), uint32_t(W), uint32_t(H), 4);
+    overlaytex::Set(g_resetAsk.overlay, px.data(), W, H);
     vr::VROverlay()->SetOverlaySortOrder(g_resetAsk.overlay, 10);
     g_resetAsk.panel = panel, g_resetAsk.shownMs = NowMs(), g_resetAsk.px = W;
     g_resetAsk.tipZ[0] = g_resetAsk.tipZ[1] = 1;
@@ -3727,7 +3728,7 @@ void UpdateTargetGlow() {
     const int h = std::clamp(int(std::lround((kTargetW - 2 * m) * H / std::max(1e-6, W))) + 2 * m, 2 * m + 4, 2048);
     if (kTargetW != g_target.texW || h != g_target.texH || m != g_target.texM) {
         const std::vector<uint8_t> px = softglow::TargetTexture(kTargetW, h, m);
-        vr::VROverlay()->SetOverlayRaw(g_target.overlay, const_cast<uint8_t *>(px.data()), kTargetW, uint32_t(h), 4);
+        overlaytex::Set(g_target.overlay, px.data(), kTargetW, h);
         g_target.texW = kTargetW, g_target.texH = h, g_target.texM = m;
         static int logged = -1;
         if (logged != g_target.panel)
@@ -3844,7 +3845,7 @@ void UpdateHandGuides() {
                 if (textures.size() > 48) textures.clear();
                 t = textures.emplace(key, softglow::GuideTexture(hg.g, tw, th, wm, halo, arm, r, edge)).first;
             }
-            vr::VROverlay()->SetOverlayRaw(hg.overlay, const_cast<uint8_t *>(t->second.data()), uint32_t(tw), uint32_t(th), 4);
+            overlaytex::Set(hg.overlay, t->second.data(), tw, th);
             hg.key = key;
         }
         vr::VROverlay()->SetOverlayWidthInMeters(hg.overlay, float(wm));
@@ -4351,7 +4352,7 @@ bool ft_vr_init(void) {
     if (vr::VROverlay()->CreateOverlay("frametop.catcher", "Frametop: release catcher", &g_catcher) ==
         vr::VROverlayError_None) {
         static std::vector<uint8_t> clear(4 * 4 * 4, 0);
-        vr::VROverlay()->SetOverlayRaw(g_catcher, clear.data(), 4, 4, 4);
+        overlaytex::Set(g_catcher, clear.data(), 4, 4);
         vr::VROverlay()->SetOverlayInputMethod(g_catcher, vr::VROverlayInputMethod_Mouse);
         vr::VROverlay()->SetOverlayAlpha(g_catcher, 0);
         vr::VROverlay()->SetOverlayFlag(g_catcher, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, true);
@@ -4385,6 +4386,7 @@ void ft_vr_shutdown(void) {
     keyboard::Destroy();
     std::this_thread::sleep_for(std::chrono::milliseconds(200));
     for (auto &[k, h] : g_imports) vr::VRIPCResourceManager()->UnrefResource(h);
+    overlaytex::Shutdown();
     g_guides.clear();
     g_screens.clear();
     g_imports.clear();
@@ -4580,7 +4582,7 @@ void ft_vr_screen_destroy(int index) {
     const Screen &s = it->second;
     if (s.remote && ScreenPose(s, &p)) g_parked[index] = {p, s.metres, s.curve, s.pinned, s.pinRel, s.alone};
     for (auto o : it->second.All())
-        if (o != vr::k_ulOverlayHandleInvalid) vr::VROverlay()->DestroyOverlay(o);
+        if (o != vr::k_ulOverlayHandleInvalid) overlaytex::Forget(o), vr::VROverlay()->DestroyOverlay(o);
     for (auto &[k, sub] : it->second.subs) vr::VROverlay()->DestroyOverlay(sub.overlay);
     g_screens.erase(it);
 }

scripts/report.sh

@@ -206,6 +206,7 @@ fi
 # looks wrong first. It wakes an idle gaze service for about 20 s, to see the tracker send.
 if has gaze; then
   python3 "$repo/scripts/gaze-report.py" </dev/null 2>&1 || echo "scripts/gaze-report.py failed"
+  python3 "$repo/models/ft-models" list eyes 2>&1 | head -n 12
 fi
 
 # Hand tracking: whether it runs, which models ft-hands loaded and whether they're the pinned ones,
@@ -220,6 +221,7 @@ if has hands; then
   [ -d "$student" ] && "$repo/hands/models.sh" check 2>&1 | tail -n 3
   echo "last checked for newer ones: $(date -r "$repo/hands/models/.student-checked" '+%F %T' 2>/dev/null || echo never)"
   grep -E '^\s*(FT_)?HANDS_MODELS(_UPDATE)?=' ~/.config/frametop.conf 2>/dev/null | sed 's/\s*#.*//; s/^/override (ft-hands loads these instead): /'
+  python3 "$repo/models/ft-models" list hands 2>&1 | head -n 12
   python3 "$repo/hands/camcheck.py" 2>&1 | head -n 30
   echo "sides: $(cat "$XDG_RUNTIME_DIR/frametop-hands/sides.json" 2>/dev/null || echo 'no sides.json')"
   for c in 5 6 7; do   # ft-hands' CPUs (HANDS_CPUS)

session/frametop.conf.example

@@ -42,6 +42,8 @@ POINTER_HEAD_DEADZONE=0.5  # keyboard clicks at the gaze (Meta+J, Meta+K): degre
 POINTER_KEY_TAP=0.25       # keyboard clicks: let go within this (s) and it clicks where the dot was at the press, and tells the gaze tracker it was right
 GAZE_TRACKER=auto          # gaze service: auto = our own eye tracker when it's installed (gaze/tracker/install.sh, which install.sh offers), else SteamVR's | own = ours | steam = SteamVR's. Each keeps its own calibration (Calibrate on Frametop Input Settings' Gaze page)
 GAZE_EYE=auto              # gaze service: eye bias. auto = each eye weighted by how far off it was at your recent nudges | left | right = that eye counts twice
+#EYES_MODEL=classic        # our eye tracker (GAZE_TRACKER own): classic = the pupil finder | latest = the newest eye net, following new versions | a version's name (gaze/tracker/models.json) | a model folder of your own. Pick it on Frametop Input Settings' Gaze page, which downloads it
+EYES_MODELS_UPDATE=1       # our eye tracker: 1 = when it starts, check GitHub for the eye net list and fetch the net EYES_MODEL needs | 0 = only what's downloaded
 GAZE_FIT_WATCH=0           # gaze service, with our eye tracker installed: 1 = the put-on check (the quick check, when the headset goes on, first says which way to move it back to your saved fit; the fit check saves a good one) and the slip note (our tracker in use: a small note when the headset slipped) | 0 = off
 HANDS_MODE=off             # hand tracking (hands/run.sh install; install.sh sets frametop when it installs it), when it runs: frametop = while the Frametop desktop runs | always = whenever SteamVR runs, in games and the dashboard too | off = only by hand (ft-handsctl on). Frametop Input Settings' Hands page applies it at once
 HANDS_SWAP_SIDES=auto      # hand tracking: which side camera is which. auto = ft-hands tells from the hands and fixes ft-camd's names, which some SteamVR restarts swap | 0 = keep ft-camd's | 1 = exchange them
@@ -52,6 +54,7 @@ HANDS_BRIGHT_ON=40         # hand tracking, auto: the colour frames' mean bright
 HANDS_BRIGHT_OFF=25        # ... and back to the mono cameras under this
 HANDS_COLOR_LEFT=color_video0  # hand tracking: which colour camera is passthrough_left (hands/tools/check_color.py tells)
 HANDS_COLOR_CROP=subtract  # hand tracking: how the colour calibration's crop applies, subtract | none (check_color.py tells)
+#HANDS_MODEL=latest        # hand tracking: latest = the fine-tuned models we recommend, following new versions (hands/models/student) | stock = Google's MediaPipe models | a version's name (hands/models.json). Pick it on Frametop Input Settings' Hands page, which downloads it
 #HANDS_MODELS=            # hand tracking: a folder of palm.ncnn.* and hand.ncnn.* to use instead of the fine-tuned models (hands/models/student; hands/models/ncnn has the stock ones)
 HANDS_MODELS_UPDATE=1      # hand tracking: 1 = when it starts, fetch newer fine-tuned models if GitHub's main (stable) or experimental branch pins them (hands/models.pin), without a new release | 0 = keep the ones it came with
 HANDS_MATTE=circle         # hand cutouts: circle = a round hole per hand | mesh = hand-shaped, from the skeleton | capsules = ft-hands' capsules (the old look)
@@ -135,11 +138,11 @@ POINTER_TYPING_RECENT=600  # hands: ... only within this many seconds of a key (
 POINTER_HAND_RAY=1         # hands: a bare hand raised and pointing shows a ray that drives the pointer like a controller's laser (pinch = click, middle pinch = right click or scroll); 0 = gaze pinches only
 POINTER_RAY_DEADZONE=2.5   # hand ray: degrees the ray turns during a pinch before the press drags (a click's own wobble stays inside)
 POINTER_RAY_PUSH=1         # hand ray: while carrying a panel, reaching out pushes it away and pulling in brings it near
-POINTER_RAY_SCROLL=1       # hand ray: scroll speed for a held middle pinch moved up or down (-1 turns the direction round)
+POINTER_RAY_SCROLL=1       # hand ray: scroll speed for a held middle or three-finger pinch moved up or down (-1 turns the direction round; 0 = no ray scrolling, a three-finger pinch is then a pinch)
 POINTER_RAY_RELEASE=2      # hand ray: seconds after the ray goes before the pointer it woke lets go of the laser
 POINTER_RAY_GRACE=0.5      # hand ray: seconds a hand keeps the laser after its ray drops (the other hand's ray takes it after this, or at once by pinching or raising)
 POINTER_RAY_BEAM=0         # hand ray: 0 = only its dot shows (the beam starts just short of it; SteamVR's beam width is one setting for every laser, read only when SteamVR starts) | 1 = the whole beam from the hand
-POINTER_RAY_READY=6        # hand ray: shows only with the thumb and index tips within this many cm (gone past POINTER_RAY_READY_HIDE, 10): a laser for pinching | 0 = whenever the hand points
+POINTER_RAY_READY=6        # hand ray: shows only with the thumb tip within this many cm of the index or middle tip (gone past POINTER_RAY_READY_HIDE, 10): a laser for pinching | 0 = whenever the hand points
 POINTER_RAY_FIST_CURL=1.1  # hand ray: a fist (the index and the other fingers curled under this, tip-to-wrist over knuckle-to-wrist) drops the laser, and a pinch made closing into one doesn't click | 0 = off
 POINTER_PINCH_NO_RAY=0     # hands, gaze off: 1 = a pinch or fist with no laser up presses where the pointer is and drags it (air mouse; most accidental clicks of the 2026-10-11 test) | 0 = it only wakes the pointer
 POINTER_RAY_SMOOTH_MINCUTOFF=0.8  # hand ray: smoothing with the hand still (Hz; lower = steadier, laggier)
@@ -158,7 +161,7 @@ POINTER_GAZE_TWO_HAND=1    # hands in gaze mode, as Meta+J then Meta+K: while on
 POINTER_GAZE_RIGHT_DRAG=0  # hands in gaze mode: an index pinch within this many seconds of the other hand's index tap holds the right button where you look (the tap itself clicks) | 0 = off
 POINTER_TYPING_HIDE=1.5    # gaze mode: the dot and its laser hide this many seconds after each key (0 = never)
 POINTER_PINCH_MAX_SPEED=1.0  # a pinch begun with its palm moving faster than this (m/s, smoothed) within 0.25 s doesn't click (lazy susan strokes, waves) | 0 = off
-POINTER_GAZE_SCROLL=1      # hands in gaze mode: the thumb on both the index and middle tips (within POINTER_GAZE_SCROLL_GAP, 0.035 m) scrolls where you looked as the hand moves (0.5 cm dead zone, full at 4 cm), times this (-1 turns it round) | 0 = off
+POINTER_GAZE_SCROLL=1      # hands in gaze mode: the thumb on both the index and middle tips (within POINTER_GAZE_SCROLL_GAP, 0.035 m, which the hand ray's three-finger scroll uses too) scrolls where you looked as the hand moves (0.5 cm dead zone, full at 4 cm), times this (-1 turns it round) | 0 = off
 POINTER_GAZE_CHORD=0       # hands in gaze mode: 1 = while one hand's pinch is held, the other hand's pinch holds the right button (the chord; the right-drag replaced it)
 POINTER_GAZE_HAND_RAY=0    # hands in gaze mode: 0 = no hand ray, your eyes aim and a pinch clicks where you look | 1 = a raised, pointing hand's ray works beside the gaze
 POINTER_GAZE_PINCH_DEADZONE=4 # hands in gaze mode: degrees (seen from the eye; about 3 cm at arm's length) a held pinch moves before it nudges the pointer; held still for POINTER_GAZE_HOLD it's a press, to drag

uninstall.sh

@@ -120,6 +120,7 @@ main() {
     [ -e "$apps/$f.desktop" ] && entries+=("$apps/$f.desktop")
   done
   [ -e "$apps/frametop-handrec.desktop" ] && entries+=("$apps/frametop-handrec.desktop")
+  [ -e "$apps/frametop-eyerec.desktop" ] && entries+=("$apps/frametop-eyerec.desktop")
   entries+=("$apps"/frametop-profile-*.desktop)  # one per layout profile (layout/ft_layout.py)
   [ -L "$handsctl" ] || handsctl=
   [ -f "$eye_rate" ] && [ -f "$repo/gaze/eye-rate.py" ] || eye_rate=

voice/ft-voiced

@@ -331,6 +331,7 @@ class Voice:
         self.loop = None  # the event loop, for the controller reader's thread
 
         self.kws = self.kws_stream = None  # built from the wake phrases in the settings
+        self.wake_phrases = []
         self.cmd_vad = self._make_vad(self.wcfg["end_silence"])
         self.clip_vad = self._make_vad(0.5)
         self.piece_vad = self._make_vad(0.1)  # per-chunk speech/silence labels for live windows
@@ -348,6 +349,7 @@ class Voice:
     def _make_kws(self, phrases):
         """Open-vocabulary spotter for the phrases: they are spelled with the model's BPE
         tokens at load time, so any English words work without training."""
+        self.wake_phrases = list(phrases or ())
         if not phrases:
             log.info("wake phrases: none")
             return None
@@ -904,7 +906,11 @@ class Voice:
         self.chime("wake")
         self.broadcast("wake", keyword=keyword)
         self.run_hook("wake_hook", keyword, env={"FRAME_VOICE_EVENT": "wake"})
-        self.command = {"chunks": [], "t0": time.monotonic(), "heard": False, "wake_unix": time.time()}
+        self.listen_for_command(time.time())
+
+    def listen_for_command(self, wake_unix, again=False):
+        self.command = {"chunks": [], "t0": time.monotonic(), "heard": False, "wake_unix": wake_unix,
+                        "again": again}
         self.cmd_vad.reset()
         self.update_mic()
 
@@ -919,7 +925,7 @@ class Voice:
         elapsed = time.monotonic() - cmd["t0"]
         if cmd["heard"] and not speaking or elapsed > self.wcfg["max_command_seconds"]:
             self.command = None
-            asyncio.create_task(self.finish_command(np.concatenate(cmd["chunks"]), cmd["wake_unix"]))
+            asyncio.create_task(self.finish_command(np.concatenate(cmd["chunks"]), cmd["wake_unix"], cmd["again"]))
         elif not cmd["heard"] and elapsed > self.wcfg["command_timeout"]:
             self.command = None
             log.info("wake: no command heard")
@@ -929,7 +935,7 @@ class Voice:
             self.reset_kws()
             self.update_mic()
 
-    async def finish_command(self, samples, wake_unix):
+    async def finish_command(self, samples, wake_unix, again=False):
         self.chime("stop")
         try:
             text = await self.transcribe(samples)
@@ -937,6 +943,15 @@ class Voice:
             log.error("transcription failed: %s", err)
             self.broadcast("error", message=str(err))
             return
+        # The capture can hold only the wake phrase's end ("Hey, Computer."): the spotter fires
+        # late, and a pause after it (waiting for the chime) ends the capture before the command.
+        # Then listen once more.
+        text, had_wake = voice_commands.strip_wake(text, self.wake_phrases)
+        if had_wake and not text.strip(" .!?,") and not again and self.command is None and self.recording is None:
+            log.info("wake: only the wake phrase heard, listening again")
+            self.chime("wake")
+            self.listen_for_command(wake_unix, again=True)
+            return
         found = voice_commands.match(text, self.voice_commands) if text else None
         if found is None:
             self.broadcast("command", text=text)

voice/test/test_commands.py

@@ -174,3 +174,21 @@ class Settings(unittest.TestCase):
 
 if __name__ == "__main__":
     unittest.main()
+
+
+class StripWake(unittest.TestCase):
+    PHRASES = ["HEY COMPUTER"]
+
+    def test_wake_then_command(self):
+        self.assertEqual(vc.strip_wake("Hey, Computer. Flag that.", self.PHRASES), ("Flag that.", True))
+
+    def test_misheard_wake_alone(self):
+        text, had = vc.strip_wake("A Computer.", self.PHRASES)
+        self.assertTrue(had)
+        self.assertEqual(text.strip(" .!?,"), "")
+
+    def test_command_only(self):
+        self.assertEqual(vc.strip_wake("Flag that.", self.PHRASES), ("Flag that.", False))
+
+    def test_wake_word_later_stays(self):
+        self.assertEqual(vc.strip_wake("Flag that, Computer missed it", self.PHRASES)[1], False)

voice/voice_commands.py

@@ -36,6 +36,23 @@ def words(text):
     return tuple(w.lower() for w in WORD.findall(text))
 
 
+def strip_wake(text, phrases):
+    """`text` without a wake phrase it starts with, and whether one was there. The recognizer
+    often catches the end of the wake phrase as the command starts ("Hey, Computer. Flag that."),
+    and mishears its first words ("A Computer"), so the phrase's last word
+    within the phrase's length marks it: everything up to it goes."""
+    found = list(WORD.finditer(text or ""))
+    heard = [m.group().lower() for m in found]
+    best = -1
+    for phrase in phrases:
+        pw = words(phrase)
+        if pw and pw[-1] in heard[:len(pw)]:
+            best = max(best, heard[:len(pw)].index(pw[-1]))
+    if best < 0:
+        return text, False
+    return re.sub(r"^\W+", "", text[found[best].end():]), True
+
+
 def normalise(text):
     return " ".join(words(text))