Compare commits

..
Author SHA1 Message Date
DeeJanuzandClaude Opus 5.5 cf51af3452 pack: the PySide export also checks the Eye Recorder (PR #63)
Experimental ships it; its imports are the same five bindings, and the
export run in the dev box keeps QtCore, QtGui, QtNetwork, QtOpenGL, QtQml,
QtQuick and QtQuickControls2 (27 MB of python3-pyside6's 60 MB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:15:49 -06:00
DeeJanuzandClaude Opus 5.5 feab679cf6 Merge PR #63 from cinnacappu/optimize-runtime: smaller release runtime, stripped binaries, less repeated work
Conflicts against experimental:
- pack/Containerfile: experimental's venv groups (voice) and import check,
  without PySide6 there (the runtime stage checks it after the export);
  ft-spotpanel and whisper-server go into /runtime like the rest; hand
  tracking and the Hand Recorder panel stay in /src, copied whole.
- screens/build.sh: experimental's four programs, all stripped.
- justfile: both test lists.
- screens/vr.cpp: experimental's side. The PR's per-frame vector-to-array
  changes there don't fit experimental's code (UpdateCorner takes the
  lasers, HandleSize margins); left out, not ported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:15:29 -06:00
DeeJanuzandClaude Opus 5.5 3b88b7adbc input-settings: hide the Current mappings heading with no mappings (PR #62)
The old Heading was hidden when the device had none; the Separator that
replaced it showed over nothing. Also the ternaries' continuation indent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:14:34 -06:00
DeeJanuzandClaude Opus 5.5 309200b365 Merge PR #62 from SuperTuxii/center-button-mappings: input-settings current mappings in the Buttons page's FormLayout
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:14:18 -06:00
DeeJanuzandClaude Opus 5.5 d6f2e02cd5 Merge PR #61 from jlitewski/fix/43-held-button-crossing: ft-screens keeps a held button's release across screens and on leaving them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:14:18 -06:00
DeeJanuzandClaude Opus 5.5 1c55d10509 Merge main into experimental before the PR round
main's only commit not here is adebd7d (install pages); experimental's
772397e has the same pages, newer (zip size, Eye Recorder), so both
conflicts keep experimental's side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:14:18 -06:00
DeeJanuzandClaude Opus 5.5 fb65872042 Merge eyerec-redo-cues: tipped and off-the-face nudges
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:09:02 -06:00
DeeJanuzandClaude Opus 5.5 d4f11375cd Eye recorder slips: two more nudges, front tipped down and off the face
The kinds of slip a heavy or loose headset makes: the front droops (its
bottom edge down, the top against the forehead), and it comes a few mm off
the face. Same grip, beeps and chain as the other five. "tip" counts as a
hand-move cue word. About 3.0 min recorded now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:09:02 -06:00
DeeJanuzandClaude Opus 5.5 fc54b0e096 Merge eyerec-redo-cues: detailed nudge directions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:02:15 -06:00
DeeJanuzandClaude Opus 5.5 13b6d4d716 Eye recorder slips: say exactly how to move the headset each time
Each nudge's ready screen now says the grip (one hand on the front, thumb
under the bottom edge), the direction, how far, what stays still (level,
the tilt's middle) and that the head and eyes don't move. The intro lists
the five moves and says a little blur is fine. rounds/README.md gets the
rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 17:02:15 -06:00
DeeJanuzandClaude Opus 5.5 454d295e4a Merge eyerec-redo-cues: eye recorder redo/Back fixes, cue rules for moving the headset
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:55:49 -06:00
DeeJanuzandClaude Opus 5.5 2cb0686df0 Eye recorder: redo and Back that work, cue rules for moving the headset
Redo in the countdown started the step before and left the recording on;
Back was off at the first step of every take; and redoing the dots after a
nudge didn't redo the nudge. Now redo during the countdown or a step starts
it again from the start of its chain, and Back at a ready screen reaches the
take before (it stays open until the next step's Next). Nothing goes back
over a reseat. The keys line and the window's button say what redo does.

rounds/README.md gets rules by state (ready screen, countdown, dots, eyes
shut, moving the headset, head or reading). Moving the headset by hand gets
"go" and "done" beeps and load_round checks: a look cue first, a "Let go"
cue in the same chain, 2 s or more to settle, "Keep holding" text while
holding. The slips round follows them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:55:41 -06:00
DeeJanuzandClaude Opus 5.5 d63b5aff75 Merge branch recorder-export-queue into experimental (consent wording)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:36:23 -06:00
DeeJanuzandClaude Opus 5.5 dadb064bab Recorders: the consent texts name Export and upload
"You upload the export yourself" now says it can be from the Upload page or
with Export and upload, which sends it once it's exported. Same promise:
nothing goes up unless the person asks for that round, so the version
stays (user, 2026-10-17).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:36:23 -06:00
DeeJanuzandClaude Opus 5.5 9bc5e00f5a Merge branch recorder-export-queue into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:33:26 -06:00
DeeJanuzandClaude Opus 5.5 ed12112cca Recorders: queue exports, and upload by itself once exported
Export (or Export and upload) while another export runs queues it on the
Export page; exports run in order. Export and upload, when logged in and
the dataset takes contributions, uploads the session by itself once its
export is done, uploads waiting their turn too. Each waiting export can
be removed or set to upload; Cancel stops only the running one. A Done
list says how each went, with the pull request's link, for whoever comes
back to the headset. Sessions waiting to export or upload can't be
deleted. Both the hand and eye recorders get it (recorder/window.py).

The awake unit's workers now reconcile to the current reasons, so an
export ending as its upload starts can't leave the Frame free to sleep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:33:15 -06:00
DeeJanuzandClaude Opus 5.5 c8ef23847c Eye recorder: pace the pose log at 250 Hz
clock_nanosleep has no absolute sleeps on CLOCK_MONOTONIC_RAW on Linux
(ENOTSUP at once), so PoseLog::Run spun: ~160k poses/s, a busy core on
the Frame during eye rounds and ~4.6 GB of poses.jsonl per upload.
Sleep the gap relative on CLOCK_MONOTONIC; the schedule stays on RAW.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 16:33:06 -06:00
DeeJanuz 0a45e0a7bd Merge branch recorder-upload-during-export into experimental 2026-10-17 10:17:05 -06:00
DeeJanuz 8bafc8f55f Merge branch spotwatch-reserve into experimental 2026-10-17 10:17:05 -06:00
DeeJanuzandClaude Opus 5.5 8cebed0448 Recorders: upload one session while another exports
Upload waited for any export, so an exported round couldn't go up while the next one
exported. Exports write beside the old export (.partial) and swap in at the end, so only
the same session conflicts: its upload waits for its own export, and it can't be exported
again while it uploads (that swap would pull the files out from under hub.py). The Upload
page's export picker now stays on its export when another one finishes and joins the list.
Hand and eye recorders both (recorder/window.py is shared).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 10:16:38 -06:00
DeeJanuzandClaude Opus 5.5 fbc4cbde81 spotwatch: a reserved spot nearer than the lazy susan's front slot
The search opened 1 m ahead, where the front slot puts a spun panel, so the two shared a
plane. It now opens 0.75 m ahead at the same angular size (0.42 m wide), and ft-screens
keeps the front slot at least 0.15 m behind it (SCREEN_SPIN_DISTANCE is raised to 0.9 if
set nearer). Its sort order (14) stays over the panels', so a panel nearer than it still
doesn't cover it. The numbers live in spotwatch/reserve.h, shared by both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 10:02:05 -06:00
DeeJanuzandClaude Opus 5.5 381c99479b Merge branch handrec-free-space into experimental
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:38:29 -06:00
DeeJanuzandClaude Opus 5.5 e4d135d373 hand recorder: room to export, and free up space but keep a log of uploads
Exports of long rounds (middle taps) filled the disk and failed with a bare
"Broken pipe". The pre-round check asked for 16 GB, enough to record but not
to export.

- Before you start asks for room to record and export (about 31 GB).
- An export starts only with 0.8x the recordings + 2 GB free, and stops
  cleanly (removing its half) when free space drops under 1.5 GB.
- Messages say how much is needed and point at deleting uploaded rounds.
- New recorder/contributions.py: BASE/contributions.json keeps one entry per
  finished upload (PR link, bounty, size), written by hub.py on upload and
  backfilled from session.json for older uploads.
- Review gets "Free up space" per uploaded round and "Delete uploaded rounds";
  the log is written and checked before anything is deleted. Rounds changed
  since their upload aren't offered. "Your uploads" lists the log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:38:12 -06:00
cappu dfc3c5244f Keep deployment templates out of the private Qt runtime 2026-10-17 01:11:32 -04:00
cappu ca3eb5b341 Preserve Qt image formats and verify both Python entry paths 2026-10-17 00:59:26 -04:00
cappu 191c0709f7 Separate the required Fedora PySide runtime from its full build package 2026-10-17 00:53:56 -04:00
DeeJanuzandClaude Opus 5.5 155fbc50c1 Model picker: one row a version; the latest version's row follows updates
The list had "Latest (follows updates): v16b" and v16b again by name, and
the named one read "not downloaded" (hand models' latest lives in
hands/models/student). Now each version shows once, the latest tagged
Latest, and picking it follows updates. A setting that names the latest
version shows on that row as not following updates, with a note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:18:53 -06:00
cappu 0d71257c86 Keep input fixture startup releases within a connected datagram pair 2026-10-16 23:57:17 -04:00
cappu 7413139f13 Strip native build outputs and discard unused driver and panel sections 2026-10-16 23:57:16 -04:00
cappu 0684a58306 Reduce release build leftovers and repeated runtime work 2026-10-16 23:43:01 -04:00
DeeJanuzandClaude Opus 5.5 29a4bf9b68 Recorder tests: the eye dataset has three recipients since key 3 (dody)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:13:24 -06:00
DeeJanuzandClaude Opus 5.5 88fe0a525b Model picker: v16b in the hand catalog (latest), pinch closure metric
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:18:35 -06:00
DeeJanuzandClaude Opus 5.5 d0a52f3a7c Model picker: choose hand and eye model versions in Input Settings
Catalogs list each kind's versions with notes, cost and benchmark numbers:
hands/models.json (Latest = v13b-p1, stock) and gaze/tracker/models.json
(classic, SteamVR, eyes-v0 as Testing). Installs read their branch's
catalog from GitHub, so a new version reaches them with a push; offline,
the copy in the install is used.

models/catalog.py (+ ft-models CLI) validates catalogs strictly, downloads
versions from the kind's fixed Hugging Face repo pinned to a commit, and
keeps only files matching their sha256. HANDS_MODEL and EYES_MODEL pick
latest (follows new versions), a version by name, or a builtin. ft-hands
and ft-eyes read them at start; Input Settings restarts hand tracking and
ft-gazed restarts ft-eyes when the setting changes. The services fetch
what the setting needs at start (HANDS_MODELS_UPDATE / EYES_MODELS_UPDATE
= 0 turn that off).

Input Settings: "Hand model" on the Hands page, "Eye tracker" on the Gaze
page (replaces the Own/SteamVR switch), tagged Recommended / Testing /
Older / Built in, with a benchmark table against the model in use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:18:07 -06:00
DeeJanuzandClaude Opus 5.5 e5de0d5647 Hand models: v16b (more pinch and tap training from the bounty rounds)
v16b's landmark model adds Frametop Hands contributions #12 (index taps), #13
(pinches) and #14. Held out (#4, #7, #8): thumb-index pinch read closed 71% ->
80%, landmark error and 3D shape level with v13b-p1 (3.8 px, 8.1 mm), slightly
more empty crops taken for a hand (0.22% -> 0.37%). Replays track the same
hands as v13b-p1. Palm detector unchanged (palm-1). HF commit d96e565.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:17:59 -06:00
DeeJanuzandClaude Opus 5.5 eb5bcbb1cf Hand Recorder: the end screen says the recording session will close
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:19:11 -06:00
DeeJanuzandClaude Opus 5.5 2bf643da01 Eye Recorder: fit coach off, IPD shown, beeps for shut eyes, bounty rules
- The fit coach is off until it's trained on more fits
  (FT_EYEREC_FIT_COACH=1 brings it back): the round opens on putting the
  headset on, one Next goes on, its readings are still kept in fit.json.
- No IPD sweep (the IPD can't be changed with the headset on): that
  screen shows the IPD setting read from SteamVR instead.
- sounds.py: generated beeps (blink, close, open) played with pw-play; a
  cue's third field names one. Blinks and lids blinks on uneven beeps
  and closes the eyes on a beep, opening them on the next.
- rounds/README.md: the rules for building a bounty. load_round enforces
  the eyes-shut ones.
- The end screen says the recording session will close.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:19:11 -06:00
DeeJanuzandClaude Opus 5.5 5b83f04f27 Eye Recorder: contributions open
CONSENT.md (version 2026-10-16) and UPLOAD.md are no longer drafts, in the
hand recorder's format; the keys line names the three keys. Uploads go to
DeeJanuz/frametop-eyes (public, gated). An export made under a draft
consent text isn't uploaded there: agree again and export again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:19:11 -06:00
DeeJanuzandClaude Opus 5.5 a8c898f8cf Eye Recorder in the app menu, and offered by the release installer
ft-eyerec runs through scripts/in-box with the image's venv first, as the
Hand Recorder. gaze/rec/install.sh only writes the menu entry in a
release. install.sh offers it after our eye tracker (--no-eye-recorder),
uninstall.sh removes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:19:11 -06:00
DeeJanuzandClaude Opus 5.5 7c6af9c659 Hand Recorder: run with the release image's Python packages
distrobox puts the host's PATH first, so in a release box python3 was
Fedora's, without cryptography or huggingface_hub: Upload and encrypted
export failed. The launcher puts /opt/frametop/venv/bin first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:19:11 -06:00
DeeJanuzandClaude Opus 5.5 f94f3574b2 Eye recorder: never read frames twice; replays on today's clock
eyerec.py's reader skips a camera whose count hasn't moved on, so it
never steps back and copies frames again. ft-eyes-replay --now shifts a
recording's times to the live clock, so a replayed round's targets line
up with its frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:19:11 -06:00
DeeJanuzandClaude Opus 5.5 22b0b7aa21 gaze: our eye tracker can run an eye net (EYES_MODEL)
ft-eyes gains a net mode: eyes_net.py runs an end-to-end gaze model with
ncnn, with its own nine-dot calibration; without a model it stays the
pupil finder. build.sh adds the ncnn wheel to a release box's venv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 17:19:11 -06:00
SuperTuxii 36537fb1ec input-settings: Move current mappings in Buttons menu into FormLayout
Move the current mappings in the Buttons menu into the
Kirigami.FormLayout. This will make them get centered when resizing the
window instead of aligning to the left side. Also use a
Kirigami.Seperator for the "Current mappings" heading to achieve a
similar look as the Keyboard menu.

Signed-off-by: SuperTuxii <123881249+SuperTuxii@users.noreply.github.com>
2026-10-16 22:30:34 +02:00
DeeJanuzandClaude Opus 5.5 5d929288bd Eye recorder: add key 3 (dody, the GPU training machine) as a recipient
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:16:28 -06:00
JLitewski 802cde44f7 ft-screens: release held buttons when the pointer leaves the screens
The same lost release as #43, on the way out: wlr_seat_pointer_notify_clear_focus()
forgets the held buttons, so a press still down when the pointer left every screen (the
3D mouse's or a laser's, not only the relay's) never got its release, and KWin kept a
move or drag going until the next click. Release whatever the seat still holds, to the
surface it was pressed on, before clearing the focus, as release_relay_buttons() already
does for the relay's buttons.
2026-10-16 07:57:42 -04:00
JLitewskiandVaIerus 9df06a5244 ft-screens: keep a held button's release when the pointer crosses screens
Dragging a window by its title bar from one screen onto another and letting go left
it stuck to the pointer until the next click. Entering the new screen's surface went
through wlr_seat_pointer_notify_enter(), which in wlroots 0.20 forgets the held buttons
when the focused surface changes (reset_buttons), so the release found no button and
KWin never got it. With a button held and no grab, enter with wlr_seat_pointer_enter(),
which keeps them: KWin is one client with a surface per screen, so the release belongs
on the new one. A drag within one screen is unchanged.

Fixes #43

Co-authored-by: VaIerus <20468282+VaIerus@users.noreply.github.com>
2026-10-16 07:57:42 -04:00
DeeJanuzandClaude Opus 5.5 ced0ece879 docs: Frametop.zip is about 1.4 GB (exp.12's was 1.39 GB)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 22:21:08 -06:00
DeeJanuzandClaude Opus 5.5 e3156c1690 screens: overlay textures through shared buffers, not SetOverlayRaw
SteamVR never gives back the memory block each SetOverlayRaw call takes,
so ft-screens' 201st call failed and overlays kept stale textures at new
sizes: corner glows showed as big fingertip-dot discs and "grab that"'s
halo kept another panel's shape (2026-10-15 recording, about 40 minutes
in). Each distinct image now goes into a linear DMA-BUF imported once,
shared by content, dropped least recently used past 64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 22:16:07 -06:00
DeeJanuzandClaude Opus 5.5 c1ced884e8 Hand ray: middle and three-finger pinches raise the laser; three-finger scroll
The user, 2026-10-15: right clicks were hard and the dot kept going away.
The laser's "ready" gate now opens with the thumb near the index or the
middle tip, so a middle pinch (right click) or a three-finger pinch has a
laser to aim with. A middle pinch's aim lock follows the thumb to middle
distance.

A three-finger pinch (thumb on the index and middle tips, the gaze
scroll's POINTER_GAZE_SCROLL_GAP) on the ray scrolls as the hand moves and
never clicks, whichever finger ft-hands calls it, at the begin or joining
before the press went out. A middle tap's first POINTER_RAY_TAP scrolls
only past 1.5 cm, so a right click that wobbles still clicks (two of the
day's turned into scrolls).

Grab that's claim skipped a thumb on the middle tip, so a middle pinch
from a hand held out (the ray's pose) looked like the fist closing and
was left to ft-screens (13 of the day's middle pinches). It now needs the
thumb off both tips; the user's flag keeps it 3.5 cm or more from the
middle tip as the fist closes, and still fires twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 21:36:34 -06:00
DeeJanuzandClaude Opus 5.5 d3cd523a8d Voice: listen again when a wake command holds only the wake phrase
frame-voice's d5e62b4, which never reached Frametop Voice: the spotter fires
late, so the command capture can start inside the wake phrase, and a pause
after it (waiting for the chime) ended the capture before the command was
said. A leading wake phrase is now stripped from the transcript, and one that
leaves nothing opens a second listening window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 20:32:29 -06:00
DeeJanuzandClaude Opus 5.5 adebd7d6cf Install pages: the Hand Recorder from the experimental release or standalone
The Pages site serves main: its hand-recorder.md and README install text
now match experimental's (772397e). The recorder page gives the standalone
recorder's one-liner, or get.sh --release --experimental, whose installer
offers the recorder built; the README names the questions each version asks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:08:43 -06:00
DeeJanuzandClaude Opus 5.5 772397e79e Install pages: the Hand Recorder from the experimental release or standalone
hand-recorder.md still installed stable Frametop and built the recorder,
and called the standalone recorder planned. It now gives the two ways:
the standalone recorder's one-liner (dashboard, no Frametop), or
get.sh --release --experimental, whose installer offers the recorder
built. It describes bounty rounds and testing sessions, and the uninstall
for each. The README's install text names the questions each version asks,
the Hand Recorder's included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 17:08:29 -06:00
108 changed files with 5812 additions and 521 deletions

No files matched your search

+3
View File
@@ -19,3 +19,6 @@ frametop-report-*.txt
# the fine-tuned hand models: the image fetches its own pinned copy (hands/models.sh)
hands/models/student/
hands/models/.student-*
# model versions downloaded from Input Settings (models/catalog.py): each install fetches its own
hands/models/versions/
gaze/tracker/models/
+2
View File
@@ -11,3 +11,5 @@ frametop-report-*.txt
*.pgm
.venv/
.frame-job.d/
# eye net versions picked in Input Settings (models/catalog.py downloads them; CC BY-NC 4.0)
gaze/tracker/models/
+2 -2
View File
@@ -35,9 +35,9 @@ You need a Steam Frame with an internet connection, a keyboard (Bluetooth, or th
It asks which version you want: stable (the `main` branch, tested releases) or experimental (the `experimental` branch, the newest features, less tested). Then it clones the repo into `~/frametop` and runs `install.sh`. To choose without the question, add `-s -- --stable` or `-s -- --experimental` after `bash`. By hand, the same is `git clone https://github.com/Frametop/frametop.git ~/frametop`, then `cd ~/frametop` and `./install.sh` (add `--branch experimental` to the clone for experimental).
The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.1 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
The third and fourth choices, stable release and experimental release, download Frametop already built (`Frametop.zip`, about 1.4 GB, from the [releases](https://github.com/Frametop/frametop/releases)) and install it without compiling anything. `-s -- --release` picks the stable release without the question, and `-s -- --release --experimental` the experimental one.
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks you six things along the way: whether to install voice dictation (yes by default; it downloads about 320 MB of speech models), hand tracking (experimental, yes by default), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), and the Bluetooth fixes, then whether to restart SteamVR. Hand tracking, the eye tracker, and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
The installer sets up distrobox in your home folder (the system files aren't touched), a Fedora build container, and everything else. The first run downloads 1–2 GB. It asks along the way whether to install each optional part: voice dictation (yes by default; it downloads about 320 MB of speech models), hand tracking (experimental, yes by default), in a release the [Hand Recorder](https://frametop.github.io/frametop/hand-recorder) (yes by default; it records your hands for Frametop's open hand dataset), gaze mode (experimental, yes by default), our own eye tracker for it (yes by default), in a release the Eye Recorder (yes by default, with our eye tracker; it records your eyes for Frametop's eye tracking dataset), and the Bluetooth fixes, then whether to restart SteamVR. The stable version asks only about gaze mode, the eye tracker, and the Bluetooth fixes. Hand tracking, the Hand Recorder, the eye tracker, and the Bluetooth fixes need your `sudo` password; if you've never set one, run `passwd` first, or skip them for now. SteamVR has to restart once at the end, which closes everything open in VR, including the terminal. Rebooting the headset works too.
After the restart, Launch a program → Desktop opens the multi-screen desktop, with its screens arranged around where you're facing. Frametop Display Settings and Frametop Input Settings are in the desktop's application menu, under Settings. SteamOS's own single-screen desktop is still there, as Native Desktop in the same list.
+3 -1
View File
@@ -24,6 +24,8 @@ ft-screens replaces it. It's a small wlroots compositor that hosts the nested KW
Frames go to SteamVR the same way gamescope sends them: OpenVR's `IVRIPCResourceManagerClient` imports each DMA-BUF (`ImportDmabuf`), and the overlay shows it as a `TextureType_SharedTextureHandle` texture. There's no copy and no size limit. The Frame's SteamVR supports this interface, but the OpenVR header bundled with SteamVR's samples predates it, so the build fetches a pinned header from Valve's openvr repository.
The small textures ft-screens draws itself (the controls, the hand glows, "grab that"'s halo, the carousel) go the same way, through `screens/overlaytex.cpp`: one linear DMA-BUF per distinct image, imported once and shared. Not `SetOverlayRaw`: the Frame's SteamVR never gives back the memory block each call takes, so a process's 201st call fails ("Refusing to create memory block because 201 blocks are already outstanding" in the journal) and the overlay keeps its old texture at its new size. That was the glow bug of 2026-10-15, after about 40 minutes of hand use. Other programs call it only a few times, or as a fallback.
Before settling on this, we tried KWin's screencast virtual outputs. KWin 6.2.5 crashed (`std::out_of_range` in `textureForOutput`) streaming one while nested in gamescope, and its headless backend doesn't implement virtual outputs at all.
A few wlroots details worth knowing: `wlr_shm_create` wants DRM format codes, not `wl_shm` ones, and KWin asks for server-side decorations before its first commit, when setting the mode would assert, so ft-screens answers on the first commit.
@@ -174,7 +176,7 @@ Volume keys must never reach gamescope. With the openvr backend, gamescope sends
Frametop's keyboard opens by itself for a text field on the desktop. The apps run inside the nested KWin, so only KWin knows when a text field has focus, and the way it tells anyone is its input method protocol (`zwp_input_method_v1`): KWin starts one input method program and activates it whenever the focused app turns on text input. `input/ft-textinput` is that program, speaking the Wayland wire protocol directly so it needs nothing but Python on the host. It only reports focus. The gamescope session puts `QT_IM_MODULE=xim` and `GTK_IM_MODULE=xim` in the systemd user environment; with those, Qt and GTK apps use X input methods and never turn on Wayland text input, so the session script drops them.
The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, as the old `@frametop_keys` could), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU and uploaded with `SetOverlayRaw` when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
The keyboard itself is ft-screens' own panel (`screens/keyboard.cpp`). We tried SteamVR's first (`ShowKeyboardForOverlay`), and on the Frame it doesn't fit a desktop. It's Steam's own panel (`valve.steam.gamepadui.keyboard`), which SteamVR mounts in the dashboard's scene, so with the dashboard closed it opened but wasn't drawn. Placing it in the room ourselves (`SetKeyboardTransformAbsolute`) made it show, but SteamVR moves it to whichever overlay the laser goes to and mounts it again, and while it's open the controllers switch to SteamVR's own laser. Our panel is an overlay like the screens' controls: any laser or the 3D mouse clicks it, nothing moves it, and its keys go out as key presses on ft-screens' seat rather than as text handed back to the input method. So nothing typed leaves ft-screens (a socket to the input method could be claimed by any local process, as the old `@frametop_keys` could), apps without text input (X11, Electron) take the keys too, and they mean what the desktop's keyboard layout says. It's drawn on the CPU into one of three shared buffers when a key's look changes; the labels come from stb_truetype, so the container needs no text rendering stack.
The Frame controllers can be mapped like mouse buttons, but they aren't input devices on the host: they reach SteamVR over the headset's own radio, and no evdev or hidraw node exists for them. So only a SteamVR client can read them. Overlay apps normally get controller input only while they have input focus, which a background helper never has. SteamVR's experimental global action set priority (`steamvr/globalActionSetPriority`, "Enable global input from overlays") lets an overlay's action set receive input anyway, and takes the inputs it binds from the scene app. Binding every button would take them all from games, so the helper's action manifest puts each button in an action set of its own, and it activates only the sets of mapped buttons. The mapping itself stays in the relay, which does the action, so mice and controllers share one list of actions.
+1 -1
View File
@@ -316,7 +316,7 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke
It costs a lot of the headset's CPU. `install.sh` offers it (yes by default, `--no-hands` skips it, and it needs `sudo` for ft-camd's file capabilities), and a release brings `ft-camd` and `ft-hands` built in its image. Installed, it runs while the Frametop desktop does (`HANDS_MODE=frametop`).
Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen (a round hole per hand, `HANDS_MATTE=circle`; `mesh` and `capsules` are the older shapes). ft-handview (`frametop-handview.service`) draws your hands as a 3D hand model posed by the tracking: over games and the void with Room View off, and over the hole with it on (`HANDS_VIEW_ROOMVIEW=1`, the default). `HANDS_VIEW_MODEL` picks the model: `asset` (the default), `babylon`, `mrtk3`, `godot`, `mrtk2`, `robot` (a robot hand built in code from the tracked skeleton: light shells, dark joints, cyan accents), or `capsules` for the older tubes; `HANDS_VIEW_STYLE=ghost|solid` the look. The models, in `screens/assets/hands` with their licences: the WebXR Input Profiles' generic hand (`asset`; MIT, Copyright (c) 2019 Amazon), Babylon.js's hand meshes by Patrick Ryan (`babylon`; CC BY 4.0, https://github.com/BabylonJS/Assets), MRTK3's OpenXR hand (`mrtk3`; BSD-3-Clause, Copyright (c) 2025, Mixed Reality Toolkit Contributors), Godot XR Tools' hands by DigitalN8m4r3 (`godot`; CC0 1.0) and MRTK2's rigged hand (`mrtk2`; MIT, Copyright (c) Microsoft Corporation). The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode your eyes aim and your hands click, Vision Pro style: a pinch clicks where you were looking as it began, and the hand ray stays off (`POINTER_GAZE_HAND_RAY=1` brings it back beside the gaze). Pinch, hold and move the hand a few centimetres to nudge the pointer slowly onto the target, then let go to click there; the correction teaches the gaze tracker, as the mouse's do. Pinch and hold still for half a second to press and drag with the hand. A middle pinch right-clicks the same way. Hold a pinch with one hand and pinch with the other to hold both buttons, left and right, each until its pinch opens. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1` and gaze mode off, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (a pinch clicks, a middle pinch right-clicks or scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". When tracking goes wrong, `ft-handsctl flag [note]` (or a voice command through frame-voice, "flag that") saves the last 12 seconds of hand crops and what the tracker made of them for the offline tools: ft-hands keeps them in RAM (`HANDS_FLAG_BUFFER=1`, about 50 MB), the drawn hands pulse cyan when a flag is taken (`HANDS_FLAG_FLASH`, `HANDS_FLAG_TONE`), and flags go to `HANDS_FLAG_DIR`. See [hands/README.md](../hands/README.md), "Flags".
Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen (a round hole per hand, `HANDS_MATTE=circle`; `mesh` and `capsules` are the older shapes). ft-handview (`frametop-handview.service`) draws your hands as a 3D hand model posed by the tracking: over games and the void with Room View off, and over the hole with it on (`HANDS_VIEW_ROOMVIEW=1`, the default). `HANDS_VIEW_MODEL` picks the model: `asset` (the default), `babylon`, `mrtk3`, `godot`, `mrtk2`, `robot` (a robot hand built in code from the tracked skeleton: light shells, dark joints, cyan accents), or `capsules` for the older tubes; `HANDS_VIEW_STYLE=ghost|solid` the look. The models, in `screens/assets/hands` with their licences: the WebXR Input Profiles' generic hand (`asset`; MIT, Copyright (c) 2019 Amazon), Babylon.js's hand meshes by Patrick Ryan (`babylon`; CC BY 4.0, https://github.com/BabylonJS/Assets), MRTK3's OpenXR hand (`mrtk3`; BSD-3-Clause, Copyright (c) 2025, Mixed Reality Toolkit Contributors), Godot XR Tools' hands by DigitalN8m4r3 (`godot`; CC0 1.0) and MRTK2's rigged hand (`mrtk2`; MIT, Copyright (c) Microsoft Corporation). The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode your eyes aim and your hands click, Vision Pro style: a pinch clicks where you were looking as it began, and the hand ray stays off (`POINTER_GAZE_HAND_RAY=1` brings it back beside the gaze). Pinch, hold and move the hand a few centimetres to nudge the pointer slowly onto the target, then let go to click there; the correction teaches the gaze tracker, as the mouse's do. Pinch and hold still for half a second to press and drag with the hand. A middle pinch right-clicks the same way. Hold a pinch with one hand and pinch with the other to hold both buttons, left and right, each until its pinch opens. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1` and gaze mode off, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (it comes up as the thumb nears the index or middle tip; a pinch clicks, a middle pinch right-clicks or scrolls, a three-finger pinch scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". When tracking goes wrong, `ft-handsctl flag [note]` (or a voice command through frame-voice, "flag that") saves the last 12 seconds of hand crops and what the tracker made of them for the offline tools: ft-hands keeps them in RAM (`HANDS_FLAG_BUFFER=1`, about 50 MB), the drawn hands pulse cyan when a flag is taken (`HANDS_FLAG_FLASH`, `HANDS_FLAG_TONE`), and flags go to `HANDS_FLAG_DIR`. See [hands/README.md](../hands/README.md), "Flags".
- `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. The Hand Recorder's installer (`hands/rec/install.sh`) sets them the same way. `hands/run.sh uncaps` takes them back while neither the services nor the Hand Recorder is installed. `hands/run.sh uninstall` and `hands/rec/install.sh uninstall` run it after removing their own part, and `uninstall.sh` always takes them back.
- `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring.
+2 -2
View File
@@ -19,7 +19,7 @@ Spotwatch works while the Frametop desktop is running and Frametop isn't paused
- Searching open windows by title, Frametop's own actions ("Reset layout"), and profiles (decision 4).
- Moving the caret within the query. Typing and Backspace work at the end only.
- A setting for its distance and size (`--distance` and `--width` on ft-spotpanel; 1 m and 0.56 m).
- A setting for its distance and size (`--distance` and `--width` on ft-spotpanel; 0.75 m and 0.42 m, `spotwatch/reserve.h`).
## Decisions
@@ -28,7 +28,7 @@ These were settled with the user on 2026-10-10. The numbers may be cited in the
| # | Question | Decision |
|---|---|---|
| 1 | The look | A rounded glass pill with a results card under it. The glass is faux, because SteamVR's overlays have alpha but no backdrop blur: a translucent dark tint, a soft sheen from the top, a bright rim fading down the edges, and a shadow outside. Real blur would mean reading the panels back every frame on the Frame's GPU |
| 2 | Where it opens | Fixed in the room, centred where the head points (not the eye gaze), at 1 m, upright, tilted no more than 60° up or down. The card hangs below the pill, so the pill stays put as results change |
| 2 | Where it opens | Fixed in the room, centred where the head points (not the eye gaze), at 0.75 m, upright, tilted no more than 60° up or down. The card hangs below the pill, so the pill stays put as results change. That spot is reserved (2026-10-17): the lazy susan's front slot stays at least 0.15 m behind it, and the search draws over any panel nearer than it |
| 3 | An app with several windows | One row per app. Enter goes to its most recently active window, and Shift+Enter or a right click lists the others with the menu items |
| 4 | What it searches | Apps and remote displays only. Hidden remote displays are left out, and disconnected ones show with Connect |
| 5 | Disconnected remote displays | Enter connects the display (`ft-layout remote connect`), waits for its panel, then spins to it |
+1 -1
View File
@@ -60,7 +60,7 @@ The tracker stops when the headset is off your head. SteamVR also calibrates gaz
`gaze/tracker/` is an eye tracker of our own, because SteamVR's is about 1.5 degrees off after the best correction the gaze service can learn, and what's left is mostly look-to-look noise that no correction on top of its output can remove. Ours processes the eye cameras itself: 0.59 degrees (median) in its best live session against 0.83 for SteamVR's with the probe's correction, and after the headset was taken off and put back without recalibrating, 0.58 once your first clicks had taught it where the headset sat (`tracker/findings.md` has the measurements).
- `ft-eyegrab` (C, root, the system service `frametop-eyegrab.service`) copies the eye-camera frames (512x400, 90 fps per eye, or 15 without `gaze/eye-rate.py on`) out of the DMA-BUFs SteamVR's `eyetracking` process holds into `/dev/shm/frametop-eyes-cams`, owned by you. It maps them read-only, and it only copies while someone touches `/dev/shm/frametop-eyes-want` (ft-eyes and the recorder do, every second). Otherwise it holds none of the tracker's buffers. Its unit keeps only the capabilities that needs (`CAP_SYS_PTRACE`, `CAP_DAC_READ_SEARCH`, `CAP_CHOWN`). `gaze/tracker/install.sh` builds it and installs it to `/etc/frametop` with sudo, which it asks for (`uninstall`, `status`, and `log` too). It also adds it to SteamOS's update keep list (`/etc/atomic-update.conf.d/frametop-eyegrab.conf`): an update deletes `/etc` files that list doesn't name, and without the grabber gaze mode falls back to SteamVR's tracker and its calibration. `scripts/doctor.sh` reports a grabber an update deleted.
- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`.
- `ft-eyes` (Python with numpy and OpenCV, in the dev container: `gaze/tracker/build.sh` puts the pinned `requirements.txt` in `gaze/tracker/build/venv`) finds each eye's pupil (dark threshold, closing, ellipse fit) and glint pair (`eyes_pupil.py`), and maps them to a gaze with a quadratic fit per eye (`eyes_model.py`). It follows the headset moving on your face with a per-eye shift, which your clicks teach, and uses the glints only to notice a sudden jump. It publishes the gaze in `/dev/shm/frametop-eyes-gaze` (ft-gaze's source `own`) and takes calibration dots and clicks on `@ft_eyes`. The gaze service runs it while Eye tracker is Own tracker, or while the probe uses it. State (the calibration, each eye's shift, the clicks) is in `~/.local/state/frametop/gaze/eyes/`. Instead of the pupil finder it can run an eye net (`eyes_net.py`): `EYES_MODEL` is `classic` (the default), `latest`, or a version from `gaze/tracker/models.json`, picked with its notes and benchmarks on Input Settings' Gaze page, which downloads it into `gaze/tracker/models/NAME` (`models/README.md`). Each net keeps its own calibration in `eyes/net-NAME/`.
- `lab/` has the tools for improving it on recordings. `ft-eyes-record NAME` (or `ft-eyes-session`, with SteamVR's gaze alongside) records the cameras. `ft-eyes-score` fits and scores on recordings against the probe's practice clicks. `ft-eyes-e2e` runs the whole live path on two recordings (calibrate on one, click through the other). `ft-eyes-replay` plays a recording into a scratch share. Heavy ones are meant for a PC: if you have `frame-job` (a personal tool, not in this repo), `gaze/tracker/.frame-job` sends them there. `lab/py` runs them with that Python (in the dev container on the Frame; on a PC, the same venv from `requirements.txt`, which frame-job's setup makes).
Ground rules, for anyone changing it:
+2 -1
View File
@@ -13,7 +13,8 @@ g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -
-o build/ft-gaze ft-gaze.cpp $OPENVR_LIBS -lpthread
stb=2c980bb59875b0d32144a71867fbdebb2f77cd20
[ -f build/include/stb-$stb ] || { curl -fsSL "https://raw.githubusercontent.com/nothings/stb/$stb/stb_truetype.h" -o build/include/stb_truetype.h && touch build/include/stb-$stb; }
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
g++ -std=c++17 -O2 -ffunction-sections -fdata-sections -Wl,--gc-sections -Wall -Wno-unused-parameter -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags gbm libdrm) \
-o build/ft-gazepanel panel/ft-gazepanel.cpp $OPENVR_LIBS \
$(pkg-config --libs gbm libdrm) -lpthread
strip --strip-unneeded build/ft-gaze build/ft-gazepanel
echo "built build/ft-gaze build/ft-gazepanel"'
+46 -1
View File
@@ -65,7 +65,10 @@ relearn the offset.
Our own tracker (gaze/tracker/ft-eyes) runs here too, in the dev container, while
GAZE_TRACKER=own and the gaze is in use (below), or the gaze probe asks for it ("eyes SECONDS",
a lease the probe renews). It reads the eye-camera frames the root service frametop-eyegrab
copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs.
copies (gaze/tracker/install.sh), which copies them only while ft-eyes runs. Which eye net it runs
(EYES_MODEL: models/catalog.py) is read at its start, so ft-eyes restarts when that changes; starting
it also runs `models/ft-models auto eyes` in the background (the catalog from GitHub, and the net
EYES_MODEL needs if it isn't downloaded), and ft-eyes moves to a net that brings.
Idle: ft-gaze and our own tracker run only while the gaze is in use: gaze mode on with someone
wearing the headset (the pointer helper says both: "gaze ? headset" -> "ok on|off worn|away"), a
@@ -135,6 +138,9 @@ from gazecal import (DEFAULT_MODEL, EYE_FOUND, EYE_LOST, MODELS, STATE, Correcti
EyeWeights, Fixation, LiveCorrection, SteamEyeLog, TrackedEye)
from gazecheck import Checks # noqa: E402
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "models"))
import catalog # noqa: E402 (which eye net EYES_MODEL means: models/catalog.py)
REPO = Path(__file__).resolve().parents[1]
HELPER = REPO / "gaze" / "build" / "ft-gaze"
IN_BOX = REPO / "scripts" / "in-box" # runs a program in the container it was built in
@@ -160,6 +166,7 @@ LOOK = 0.3 # seconds of samples before that moment make the look (the pro
RETRY = 3.0 # seconds before starting ft-gaze again
EYES_RETRY = 10.0 # seconds before starting ft-eyes again after it stopped on its own
EYES_LEASE_MAX = 120.0
MODELS_CLI = REPO / "models" / "ft-models" # "auto eyes": the eye catalog, and the net EYES_MODEL needs
SETTLE = 0.3 # seconds after an eye is found again before the fallback learns from it
IDLE_AFTER = 30.0 # seconds after the gaze was last in use before ft-gaze and our tracker stop
WAKE_SETTLE = 15.0 # seconds after waking that the tracker may take to start sending
@@ -257,6 +264,8 @@ class Service:
self.eyes_proc = None # ft-eyes, while it runs
self.eyes_until = 0.0 # the probe's lease (monotonic time)
self.eyes_restart_at = 0.0
self.eyes_model = self.eyes_model_now() # what EYES_MODEL means now: ft-eyes restarts when it changes
self.models_proc = None # ft-models auto eyes, while it runs
self.awake = False # ft-gaze (and our tracker) run: the gaze is in use (see the top)
self.idle_at = 0.0 # idle from then, unless it's in use again first
self.woke_at = 0.0
@@ -293,8 +302,39 @@ class Service:
# --- Settings, calibration and lessons ---
@staticmethod
def eyes_model_now():
"""(EYES_MODEL, the folder it resolves to): ft-eyes reads it at start."""
try:
r = catalog.resolve("eyes")
return r["setting"], r["folder"]
except Exception as e: # noqa: BLE001 (a broken catalog: ft-eyes falls back to the pupil finder)
return f"error {e}", None
def check_eyes_model(self):
now = self.eyes_model_now()
if now != self.eyes_model:
log(f"eye model {now[0]}" + (f" ({now[1]})" if now[1] else "") + (": restarting ft-eyes" if self.eyes_proc else ""))
self.eyes_model = now
if self.eyes_proc:
self.stop_eyes()
self.eyes_restart_at = 0.0
def models_auto(self):
"""The eye catalog from GitHub (at most every 15 minutes) and the net EYES_MODEL needs, if it's
missing, in the background: ft-eyes starts with what's here, and moves to the new one after."""
if self.models_proc and self.models_proc.poll() is None:
return
try:
self.models_proc = subprocess.Popen([sys.executable, str(MODELS_CLI), "auto", "eyes"], stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
start_new_session=True)
except OSError as e:
log(f"couldn't check for eye models: {e}")
def load_settings(self):
self.conf_mtime = mtime(CONF)
self.check_eyes_model()
tracker, self.tracker_setting, bias, self.nudge_max = read_settings()
if (tracker, bias) != (self.tracker, self.bias):
auto = ""
@@ -558,6 +598,8 @@ class Service:
log(f"ft-eyes isn't built: run {REPO}/gaze/tracker/build.sh")
self.eyes_restart_at = time.monotonic() + 30
return
self.models_auto()
self.eyes_model = self.eyes_model_now()
env = dict(os.environ)
env["XDG_RUNTIME_DIR"] = f"/run/user/{os.getuid()}"
self.eyes_proc = subprocess.Popen([str(IN_BOX), str(EYES_PYTHON), str(EYES_PROG), "-v", "--watch-stdin"],
@@ -920,6 +962,9 @@ class Service:
if mtime(CONF) != self.conf_mtime or (self.tracker_setting == "auto"
and own_installed() != (self.tracker == "own")):
self.load_settings()
if self.models_proc and self.models_proc.poll() is not None:
self.models_proc = None
self.check_eyes_model() # a net it fetched: ft-eyes moves to it
self.update_awake()
want = self.eyes_wanted()
if want and not self.eyes_proc and time.monotonic() >= self.eyes_restart_at:
+6 -2
View File
@@ -644,8 +644,12 @@ struct PoseLog {
next += kPosePeriodNs;
const int64_t now = RawNs();
if (next < now - kPosePeriodNs) next = now;
const timespec ts{time_t(next / 1'000'000'000), long(next % 1'000'000'000)};
clock_nanosleep(CLOCK_MONOTONIC_RAW, TIMER_ABSTIME, &ts, nullptr);
// Linux has no absolute sleeps on CLOCK_MONOTONIC_RAW (EINVAL at once: the loop spun at
// ~160k poses/s), so sleep the gap relative on CLOCK_MONOTONIC.
if (const int64_t wait = next - now; wait > 0) {
const timespec ts{time_t(wait / 1'000'000'000), long(wait % 1'000'000'000)};
clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, nullptr);
}
}
}
};
+4 -4
View File
@@ -1,6 +1,6 @@
# Recording your eyes for the Frametop eye dataset (DRAFT, not reviewed yet)
# Recording your eyes for the Frametop eye dataset
Version: 2026-10-14-draft
Version: 2026-10-16
Frametop's eye tracking needs a better model than it has: one that keeps working when you look far to the side, blink, squint, wear glasses or makeup, or when the headset moves on your face. To train it, we're collecting recordings of many people's eyes into a dataset that only the maintainers can read. This page explains what the eye recorder records, where it goes, who can see it, and what you agree to if you take part. Please read all of it.
@@ -50,12 +50,12 @@ The recorder doesn't record sound, the headset's outward cameras, your name, you
- Recordings stay on your headset, in `~/.local/share/frametop/eyes/contrib`. Nothing is uploaded automatically.
- Before you share anything, you can look through every recording in the Review page. You can delete any stretch of a take, a whole take or a whole round. Export leaves out what you deleted.
- You upload the export yourself, from the Upload page. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
- You upload the export yourself: from the Upload page, or with Export and upload, which sends it once it's exported. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
## Where it goes, and who can read it
- The dataset is hosted on Hugging Face (<https://huggingface.co/datasets/DeeJanuz/frametop-eyes>), whose servers may be outside your country, for example in the USA. It's gated: people must accept its terms to download it, and what they get has the eye images and health answers encrypted.
- Only the maintainers hold the keys that decrypt them. Today that's DeeJanuz, with one key on their own computer and a backup copy stored offline. The keys are never uploaded. If a key ever leaked, uploads made to it could be read: the maintainers would say so on the dataset's page and stop using it.
- Only the maintainers hold the keys that decrypt them. Today that's DeeJanuz, with a key on each of two of their own computers (one of them trains the models) and a backup copy stored offline. The keys are never uploaded. If a key ever leaked, uploads made to it could be read: the maintainers would say so on the dataset's page and stop using it.
- Decrypted images are kept only on the maintainers' own computers, for training and checking models, and aren't shared with anyone else or uploaded to other services.
- Recordings stay in the dataset until they're withdrawn or the dataset is taken down.
+10 -8
View File
@@ -1,6 +1,6 @@
# Eye recorder: design (2026-10-14)
The eye recorder guides a person through recording their eyes with the headset's two eye cameras, at known gaze targets and through the cases where eye tracking breaks. It also fits the headset: it shows whether each eye sits where the cameras see it best, helps set the IPD, and catches the headset shifting on the face, with instructions to put it back. Recordings stay on the headset until the person uploads them. Only the maintainers can read what's uploaded (below, "Privacy").
The eye recorder guides a person through recording their eyes with the headset's two eye cameras, at known gaze targets and through the cases where eye tracking breaks. It also fits the headset: it shows whether each eye sits where the cameras see it best, shows the IPD setting, and catches the headset shifting on the face, with instructions to put it back. Recordings stay on the headset until the person uploads them. Only the maintainers can read what's uploaded (below, "Privacy").
It's the hand recorder's sibling (`hands/rec/DESIGN.md`): the same session runner pattern, headset button, review, export and Hugging Face upload, with eye cameras instead of hand cameras. Like the hand recorder it ships in Frametop and, later, as a standalone app for people without Frametop (Frametop/frametop-eye-recorder, Frametop as a submodule).
@@ -69,7 +69,7 @@ What it can tell, and the instruction it gives:
| A glint missing, or extra bright spots | Glasses reflections, or the lens fogged or smudged | "Tilt your glasses" / "Wipe the lenses" |
| An eye found in under half the frames, looking straight ahead | Lashes on the lens, a droopy lid, or the eye outside the camera's view | The per-eye card says which |
The IPD row depends on the first headset test. If the cameras move with the lenses, a pupil's sideways offset in its image directly says how far that eye is from its lens axis, and the coach turns the IPD wheel's direction into "wider" or "narrower" until both offsets are about zero, at any IPD. If the cameras are fixed, the offset depends on the IPD setting too, and the coach needs the reference at each IPD step: the fitment step's IPD sweep (below, "Bounty board") builds that.
The IPD row depends on the first headset test. If the cameras move with the lenses, a pupil's sideways offset in its image directly says how far that eye is from its lens axis, and the coach turns the IPD wheel's direction into "wider" or "narrower" until both offsets are about zero, at any IPD. If the cameras are fixed, the offset depends on the IPD setting too, and the coach needs the reference at each IPD step, built from rounds recorded at different settings (there's no IPD sweep: the setting can't be changed with the headset on).
**Put-on and shift check.** After a good fit, the coach keeps that fit as the person's baseline (`~/.local/state/frametop/gaze/fit-baseline.json`: per eye, the pupil looking straight ahead and the glint pair, with the IPD setting). Then:
- When the headset goes back on (frames resume after ft-eyes' 3 s gap), it shows the fit card for a few seconds while the person looks ahead. If the pupils sit more than a threshold (start with 1.5 mm equivalent, tuned from data) from the baseline, it says which way to move the headset, live, until it's back. If the IPD setting changed, it says so first.
@@ -79,10 +79,12 @@ In the recorder the fit coach runs as each round's fitment step, and the shift w
## Bounty board
People pick what to record from a board of bounties. Each bounty is one round of about 5 minutes, in step mode: the headset's right button starts each step (press = next or pause, two presses = redo, hold = stop), as in the hand recorder. Every round starts the same way:
1. **Fitment** (not recorded as a take, but its readings go into fit.json): the fit coach until the headset sits well (below, "Fit coach"). Skippable after a warning, and the round records that it was skipped. On a contributor's first round, and whenever the IPD setting has changed since their last, it ends with an IPD sweep: turn the wheel slowly from narrow to wide and back while looking at a centre dot, with the IPD logged as it changes (the panel looks 30 times a second, and on SteamVR's IPD event). Its own take, `NN-ipd`, before the nine dots.
1. **Fitment** (not recorded as a take, but its readings go into fit.json): the fit coach until the headset sits well (below, "Fit coach"). Skippable after a warning, and the round records that it was skipped. (The coach is off for now, 2026-10-16: then it only asks for a comfortable fit.) It shows the IPD setting, read from SteamVR when the round starts. The setting can't be changed with the headset on (2026-10-16), so there's no IPD step: a wrong setting means stopping, setting it, and starting again. Each take's ipd.jsonl still logs it.
2. **Nine dots** (recorded): a 3x3 grid to 20 degrees on the mid background. It's the round's own reference, so each round can be labelled alone, even after a re-seat.
3. **The bounty's steps.**
How to build a new one, and the rules a round script must follow (beeps for anything done with the eyes shut, cue words as labels, timing, reach, comfort): `rounds/README.md`.
Every round uploads on its own. The board marks the bounties you've done, by your contributor id in `profile.json`, with how many times. You can do one again.
The launch bounties:
@@ -90,8 +92,8 @@ The launch bounties:
| Bounty | What | Edge case it covers |
|---|---|---|
| View edges | Dots at the panel's edges and corners: 30 degrees to the sides, 23 up and down, about 35 in the corners (the rec panel is 64 degrees across; every round's dots are checked against it) | Eyes turned far: the lid, the glints lost, the pupil foreshortened |
| Blinks and lids | Blink on cue, close both for 2 s, wink each eye, squint, eyes wide, half-closed, looking down with lids lowered | Closed and partly closed eyes; one eye missing |
| Headset slips | Nudge the headset up, down, left, right, tilt it; then take it off and put it back on; dots after each | Headset shifting on the face: the put-on check's data |
| Blinks and lids | Blink on a beep, close both eyes on a beep and open them on the next (sounds.py: the eyes are shut, so words can't cue them), wink each eye, squint, eyes wide, half-closed, looking down with lids lowered | Closed and partly closed eyes; one eye missing |
| Headset slips | Nudge the headset up, down, left, right, tilt it, tip its front down, pull it off the face, on a beep, holding it through dots, letting go on a low beep; then take it off and put it back on; dots after each | Headset shifting on the face: the put-on check's data |
| Glasses | A 5x5 grid and the edges with glasses or inserts on (offered when the checklist says so) | Lens reflections, the glasses' frame in view |
| Jumps and pursuit | A dot jumping between random spots (0.5 to 1.5 s each), then moving on circles, lines and a figure eight at two speeds | Saccades, overshoot, smooth pursuit, the tracker's lag |
| Depth | Dots at 0.5 m, 1.5 m and 5 m in turn (the panel's distance changes) | Vergence; the eyes' angle with distance |
@@ -137,7 +139,7 @@ ft-gazepanel gains `show rec`: a large head-locked panel (FULL_DEG, 64 degrees,
session.json the bounty, checklist, versions, takes, eye camera rate, IPD at start
fit.json the fit coach's readings: the fitment step, and any slip warnings
health.json the opt-in health answers (encrypted in the export)
takes/<NN>-<step group>/ numbered in order: NN-ipd (when there's a sweep), NN-dots (the nine dots), then the bounty's
takes/<NN>-<step group>/ numbered in order: NN-dots (the nine dots), then the bounty's
cam0/000000.zst ... right eye: one zstd chunk per second, frames back to back (512x400 u8)
cam1/000000.zst ... left eye
index.tsv per frame: camera, chunk, offset in it, slot, CLOCK_MONOTONIC_RAW time
@@ -155,13 +157,13 @@ The export recompresses nothing (the chunks are zstd already), drops deleted ran
## Shared with the hand recorder
Much of `hands/rec/` isn't about hands: the headset button reader, profile and consent, take bookkeeping (deleted ranges, redo, parts, clock samples), the Hugging Face login and upload, validate's manifest and checksum checks, and most of the window. These live in `recorder/` at the top of the tree, with the bounty board, and both recorders use them: `common.py`, `units.py`, `button.py`, `bounties.py`, `files.py` (ranges, checksums, sizes), `checks.py` (validate's report, SHA256SUMS, typed manifest lookups), `hub.py` (login, errors, upload records, the pull request flow), `age.py` (the export's encryption) and `window.py` (the window backend's shared half: consent, disk, session controls, export, login and upload; each recorder's `Backend` subclasses it). The hand recorder's modules keep their names on top of them, and its tests pass.
Much of `hands/rec/` isn't about hands: the headset button reader, profile and consent, take bookkeeping (deleted ranges, redo, parts, clock samples), the Hugging Face login and upload, validate's manifest and checksum checks, and most of the window. These live in `recorder/` at the top of the tree, with the bounty board, and both recorders use them: `common.py`, `units.py`, `button.py`, `bounties.py`, `files.py` (ranges, checksums, sizes), `checks.py` (validate's report, SHA256SUMS, typed manifest lookups), `hub.py` (login, errors, upload records, the pull request flow), `age.py` (the export's encryption) and `window.py` (the window backend's shared half: consent, disk, session controls, export, login and upload; each recorder's `Backend` subclasses it). The hand recorder's modules keep their names on top of them, and its tests pass. The export queue, with Export and upload sending a round by itself once it's exported (2026-10-17), is `window.py`'s too, so both windows have it: `hands/rec/DESIGN.md` (Upload) has how it works.
Uploads: `gaze/rec/hub.py` sends nothing to the real dataset until `CONSENT.md` and `UPLOAD.md` exist and aren't drafts. `FT_EYEREC_ALLOW_UPLOAD=1` opens rehearsals, and only to another dataset (`FT_EYEREC_DATASET`). Rehearsed 2026-10-14 against the private `DeeJanuz/frametop-eyes-test` with a round of random bytes: uploaded, downloaded, checked, and decrypted with both keys by age 1.1.1 on the 7i.
## Build order
1. **Headset test 1:** do the eye cameras move with the IPD wheel? (Look at one dot, turn the wheel end to end, watch ft-eyes' pupil x.) Decides the IPD guidance.
1. **Headset test 1:** do the eye cameras move with the lenses when the IPD changes? The setting can't be changed with the headset on, so compare nine-dot rounds recorded at two settings (ft-eyes' pupil x at the centre dot). Decides the IPD guidance.
2. Pull the shared recorder code out of `hands/rec/` (hand recorder tests pass).
3. `eyerec.py` (frame writer) and its replay test against a capture.
4. `fitcoach.py` with readings from a capture, and the fit card on the existing fit panel.
-1
View File
@@ -1,4 +1,3 @@
DRAFT: this text waits for its review, with CONSENT.md.
### About uploading
- **What's sent:** the export in `@EXPORT_PATH@` (@EXPORT_SIZE@), into `contributions/@CONTRIBUTOR@/@ROUND@` in [@DATASET@](https://huggingface.co/datasets/@DATASET@). Upload first checks that every file is complete and matches its checksum, and that the eye images and health answers are encrypted.
+2
View File
@@ -165,6 +165,8 @@ class Writer:
got = False
for c in CAMERAS:
n = share.count(c)
if n <= seen[c]: # nothing new (and never step back: that would copy frames twice)
continue
if n - seen[c] > share.slots: # fell behind: those are gone
with self._lock:
if self._on:
+7 -4
View File
@@ -1,6 +1,9 @@
#!/bin/bash
# Launch the Frametop Eye Recorder from a Plasma session on the Frame host.
# The app runs in the dev container (PySide6, Kirigami and zstd come from Fedora there).
# Launch the Frametop Eye Recorder from a Plasma session on the Frame host (its app menu entry,
# gaze/rec/install.sh).
# The app runs in Frametop's container (PySide6, Kirigami and zstd come from Fedora there; a
# release's image has the recorders' cryptography and huggingface_hub in /opt/frametop/venv, which
# distrobox leaves off the front of PATH, so it goes first here).
# podman needs the real XDG_RUNTIME_DIR and the real user bus (to reach systemd for
# the container's cgroup; the Frametop session runs on a private bus from
# dbus-run-session). The session's Wayland socket and bus go to the app itself.
@@ -14,8 +17,8 @@ export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
# From the home folder: the app's host commands (distrobox-host-exec) run in its folder on the
# host, and a folder only the container has, such as /run/host/tmp, makes them all fail.
cd "$HOME" || exit 1
"$here/../../scripts/container-up.sh"
exec "$HOME/.local/bin/distrobox" enter dev -- env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
exec "$here/../../scripts/in-box" env PATH="/opt/frametop/venv/bin:/opt/frametop/bin:$PATH:/usr/sbin:/sbin" \
WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
QT_QPA_PLATFORM="wayland;xcb" \
python3 "$here/ft_eyerec.py" "$@"
+7 -4
View File
@@ -337,6 +337,9 @@ class Backend(RecorderBackend):
def _upload_values(self, session, path):
return {"ROUND": session}
def _session_label(self, session):
return round_label(session)
@Property(str, constant=True)
def toolVersion(self):
return takes.tool_version()
@@ -416,11 +419,11 @@ class Backend(RecorderBackend):
if not mod or not b:
return ""
try:
plan = mod.build_plan(mod.load_round(mod.round_path(b["script"])), 0, ipd_sweep=False)
plan = mod.build_plan(mod.load_round(mod.round_path(b["script"])), 0)
except (OSError, ValueError) as e:
return f"Couldn't read the round: {e}"
return (f"{len(plan)} steps, about {mod.plan_seconds(plan) / 60:.0f} min recorded, after the headset's fit "
"check (and an IPD sweep on your first round, or when the IPD setting changed)")
return (f"{len(plan)} steps, about {mod.plan_seconds(plan) / 60:.0f} min recorded, after putting the "
"headset on")
# --- the checklist and health answers
@Property("QVariantMap", notify=profileRelay)
@@ -739,7 +742,7 @@ class Backend(RecorderBackend):
@Slot(str)
def deleteSession(self, rnd):
if self._active_guard(rnd):
if self._active_guard(rnd) or self._busy_guard(rnd):
return
try:
self.store.delete_round(rnd)
+5 -1
View File
@@ -8,7 +8,8 @@ nobody types a token anywhere. An upload:
2. stops if this export (same SHA256SUMS) was uploaded before, unless --again;
3. stops unless CONSENT.md and UPLOAD.md are here and neither is a draft. FT_EYEREC_ALLOW_UPLOAD=1
opens it for a rehearsal, and only to another dataset than the real one (FT_EYEREC_DATASET):
the real one takes nothing before the texts are reviewed;
the real one takes nothing before the texts are reviewed; and it stops for an export made under
a draft consent text (agree to the current one, then export again);
4. checks the login and access to the dataset, opens the pull request first, uploads the folder
to contributions/<contributor>/<round> in it, and records it under "uploads" in session.json.
--dry-run does all of it except the network calls and the records, and says what it would upload.
@@ -155,6 +156,9 @@ def upload(store, rnd, dry_run=False, again=False, progress=None, log=None):
if not dry_run and not upload_allowed(repo):
raise HubError("closed", "Eye contributions aren't open yet: the consent and upload texts are waiting for a "
f"legal review. (A rehearsal: {ALLOW_ENV}=1 with {DATASET_ENV} set to a test dataset.)")
if repo == HF_DATASET and str(summary.get("consent_version") or "").endswith("draft"):
raise HubError("draft-consent", "This export was made when you'd agreed to a draft of the consent text. "
"Agree to the current one on the Welcome page, then export the round again.")
path_in_repo = f"contributions/{summary.get('contributor', '')}/{rnd}"
message, description = describe(summary, report, sha, report.bytes)
files = []
+16 -6
View File
@@ -5,6 +5,10 @@
# runs) and add "Frametop Eye Recorder" to the app menu (for Frametop's desktop, as the hand
# recorder's). It doesn't change Frametop's gaze setup; the window offers SteamVR's 90 fps eye
# cameras (gaze/eye-rate.py) when they're slower.
# A release (FRAME_RELEASE) brings the gaze tools built in its image (pack/Containerfile), so there it
# only adds the menu entry; its install.sh offers this.
# The recorder reads the eye cameras through our eye tracker's frame grabber
# (frametop-eyegrab.service, gaze/tracker/install.sh): without it, the window says so.
# Usage: gaze/rec/install.sh install or update
# gaze/rec/install.sh uninstall remove the menu entry. Recordings stay where they are.
set -euo pipefail
@@ -14,17 +18,23 @@ entry='~/.local/share/applications/frametop-eyerec.desktop'
case ${1:-install} in
install)
echo "== 1/3 dev container packages"
"$root/setup/dev-container.sh"
echo "== 2/3 the gaze tools: ft-gaze, ft-gazepanel and ft-eyes"
"$root/gaze/build.sh"
"$root/gaze/tracker/build.sh"
echo "== 3/3 the menu entry"
if [ "$FRAME_RELEASE" = 1 ]; then
echo "== the menu entry"
else
echo "== 1/3 dev container packages"
"$root/setup/dev-container.sh"
echo "== 2/3 the gaze tools: ft-gaze, ft-gazepanel and ft-eyes"
"$root/gaze/build.sh"
"$root/gaze/tracker/build.sh"
echo "== 3/3 the menu entry"
fi
fill_template "$root/gaze/rec/ft-eyerec.desktop" |
on_frame "mkdir -p ~/.local/share/applications && cat > $entry"
echo
echo "Installed. On Frametop's desktop, open \"Frametop Eye Recorder\" from the app menu."
echo "Recordings go to ~/.local/share/frametop/eyes/contrib."
on_frame 'systemctl is-enabled -q frametop-eyegrab.service 2>/dev/null' ||
echo "It needs our eye tracker's frame grabber, which isn't installed: gaze/tracker/install.sh"
;;
uninstall)
on_frame "rm -f $entry"
+249 -35
View File
@@ -96,6 +96,24 @@ Kirigami.ApplicationWindow {
return n + " " + word + (n === 1 ? "" : "s")
}
// A round's label, from its id (the Export page's queue).
function sessionLabel(id) {
for (const s of backend.sessions)
if (s.id === id)
return s.label
return id
}
// The round's export can upload by itself once done: logged in, and its dataset takes
// contributions (or a dry run).
function canUploadAfter(id) {
return backend.canUploadAfter && (backend.hubDryRun || backend.uploadAllowedFor(id))
}
// Waiting to export, or exporting now.
function exportPending(id) {
return backend.exportSessionId === id && backend.exporting
|| backend.exportQueue.some(q => q.session === id)
}
function show(page) {
pageStack.clear()
pageStack.push(page)
@@ -171,9 +189,11 @@ Kirigami.ApplicationWindow {
close.accepted = false
confirm.ask("Stop the round?", "A round is recording. Closing stops it; what's recorded so far is kept.",
() => { root.quitting = true; Qt.quit() })
} else if (backend.uploading && !quitting) {
} else if ((backend.uploading || backend.exporting || backend.uploadQueue.length > 0) && !quitting) {
close.accepted = false
confirm.ask("Cancel the upload?", "An upload is running. Closing cancels it; you can start it again later.",
confirm.ask("Cancel the exports and uploads?",
"An export or upload is running or waiting (the Export page lists them). Closing cancels "
+ "them all; an export cancelled keeps nothing. You can start them again later.",
() => { root.quitting = true; Qt.quit() })
}
}
@@ -298,7 +318,7 @@ Kirigami.ApplicationWindow {
padding: Kirigami.Units.largeSpacing
wrapMode: Text.Wrap
text: "Each bounty is a short round of about 5 minutes for one case where eye tracking struggles. "
+ "Pick one; the most wanted come first. Every round starts with fitting the headset and "
+ "Pick one; the most wanted come first. Every round starts with putting the headset on and "
+ "nine dots. You can do a bounty again." + (backend.boardUpdated ? " (Board of "
+ backend.boardUpdated + ".)" : "")
}
@@ -603,10 +623,10 @@ Kirigami.ApplicationWindow {
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
wrapMode: Text.Wrap
text: "Press Start, then put the headset on. A panel in the headset first helps you fit it: "
+ "look at the dot and move the headset until the panel says good (this window shows it "
+ "too). On your first round, and whenever the IPD setting changed, you then turn the IPD "
+ "wheel slowly from end to end. Then nine dots, then the bounty's steps. Look at each dot "
text: "Press Start, then put the headset on so it's comfortable and the picture is sharp, "
+ "and look at the dot on the panel in the headset. It shows the headset's IPD setting: "
+ "it can't be changed with the headset on, so set it before you start. Then nine dots, "
+ "then the bounty's steps. Look at each dot "
+ "until its ring fills, and keep your head still unless a step says otherwise.\n\n"
+ "Each step waits until you're ready: press the button on the right side of the headset, "
+ "or Space or Next in this window. A 3-2-1 countdown follows, then the step. Nothing is "
@@ -735,13 +755,6 @@ Kirigami.ApplicationWindow {
font: Kirigami.Theme.smallFont
text: "A first guess: the coach has been measured on one face so far."
}
Controls.Label {
visible: sessionView.st.ipd !== undefined && sessionView.st.ipd !== null
Layout.fillWidth: true
wrapMode: Text.Wrap
opacity: 0.7
text: "The headset's IPD setting: " + ((sessionView.st.ipd || 0) * 1000).toFixed(1) + " mm"
}
}
}
@@ -805,7 +818,8 @@ Kirigami.ApplicationWindow {
FitCard {
Layout.alignment: Qt.AlignTop
Layout.preferredWidth: Kirigami.Units.gridUnit * 18
visible: sessionView.state === "fitment" || (sessionView.state === "paused" && sessionView.card !== null)
visible: (sessionView.state === "fitment" && sessionView.st.fit_coach !== false)
|| (sessionView.state === "paused" && sessionView.card !== null)
card: sessionView.card
}
@@ -821,6 +835,16 @@ Kirigami.ApplicationWindow {
text: sessionView.state === "fitment" ? "Headset position: " + (sessionView.st.fit_words || "")
: (sessionView.st.prompt || "")
}
// The IPD setting (SteamOS's, through SteamVR), shown while the headset goes on
Controls.Label {
Layout.fillWidth: true
visible: sessionView.state === "fitment"
wrapMode: Text.Wrap
text: sessionView.st.ipd === undefined || sessionView.st.ipd === null
? "Couldn't read the IPD setting from SteamVR."
: "Your IPD setting: " + (sessionView.st.ipd * 1000).toFixed(1) + " mm. If that's not "
+ "right for you, stop (Esc), take the headset off, set it, and start again."
}
// The countdown's 3, 2, 1, then a cue's word, big
Controls.Label {
visible: !!sessionView.st.big && sessionView.state !== "paused"
@@ -849,7 +873,9 @@ Kirigami.ApplicationWindow {
visible: sessionView.waiting && sessionView.state !== "paused"
opacity: 0.7
text: sessionView.state === "fitment"
? "Press Next once the fit says good. Pressing it before warns first; a second press starts anyway."
? (sessionView.st.fit_coach === false
? "Put the headset on comfortably, with a sharp picture, look at the dot, then press Next."
: "Press Next once the fit says good. Pressing it before warns first; a second press starts anyway.")
: (sessionView.st.ready_text || "Ready? Press Space or click Next")
+ ". A 3-2-1 countdown starts the recording."
wrapMode: Text.Wrap
@@ -881,9 +907,10 @@ Kirigami.ApplicationWindow {
opacity: 0.7
text: "The dots and instructions appear in the headset. "
+ (sessionView.st.button ? "The button on the right side of the headset: next, pause or "
+ "resume; twice: record the last step again; hold: stop. " : "")
+ "While this window has focus: Space: next · P: pause or resume · R: record the last step "
+ "again · S: skip the rest of the take · Esc: stop. Stopping keeps what's recorded so far."
+ "resume; twice: redo; hold: stop. " : "")
+ "While this window has focus: Space: next · P: pause or resume · R: redo · S: skip the "
+ "rest of the take · Esc: stop. Redo during a step starts it again; before a step "
+ "starts, it goes back to the one before. Stopping keeps what's recorded so far."
}
RowLayout {
// No keyboard focus on the buttons, so Space always reaches the shortcut.
@@ -905,7 +932,8 @@ Kirigami.ApplicationWindow {
visible: backend.sessionActive
enabled: !!sessionView.st.can_redo
focusPolicy: Qt.NoFocus
text: "Redo step"
// At a ready screen it goes back to the step before; else this one again.
text: sessionView.state === "ready" ? "Back to the last step" : "Redo this step"
icon.name: "edit-undo"
onClicked: backend.redo()
}
@@ -1304,6 +1332,12 @@ Kirigami.ApplicationWindow {
return s
return null
}
// The chosen round is exporting, or waiting to.
readonly property bool pending: chosen !== null && root.exportPending(chosen.id)
readonly property bool canExport: chosen !== null && !pending && backend.exportProblem === "" && !chosen.active
&& !(backend.uploading && backend.upload.session === chosen.id)
&& backend.uploadQueue.indexOf(chosen.id) < 0
Component.onCompleted: if (backend.login.state === "unknown") backend.checkLogin()
Timer {
interval: 5000
@@ -1323,6 +1357,15 @@ Kirigami.ApplicationWindow {
text: "Exporting keeps the processor busy for a minute or so, so VR may stutter a little until "
+ "it's done. You can keep using the headset meanwhile."
}
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: backend.uploadAfter.length + backend.uploadQueue.length > 0
position: Kirigami.InlineMessage.Position.Header
type: Kirigami.MessageType.Positive
text: "You can take the headset off and leave it plugged in: it stays awake until the "
+ "exports and uploads below are done, and they show here how they went. Keep this "
+ "window open: closing it cancels them."
}
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: backend.exportProblem !== ""
@@ -1339,7 +1382,6 @@ Kirigami.ApplicationWindow {
model: backend.sessions
textRole: "label"
valueRole: "id"
enabled: !backend.exporting
Component.onCompleted: {
const i = indexOfValue(root.chosenSession)
currentIndex = i >= 0 ? i : (count > 0 ? 0 : -1)
@@ -1355,32 +1397,73 @@ Kirigami.ApplicationWindow {
Kirigami.FormData.label: "Export:"
visible: exportView.chosen !== null
text: !exportView.chosen ? ""
: exportView.pending ? (backend.exportSessionId === exportView.chosen.id && backend.exporting
? "exporting now" : "waiting its turn, below")
: exportView.chosen.export_stale ? "changed since it was exported: export it again"
: exportView.chosen.exported ? "ready, " + exportView.chosen.exportText : "not exported yet"
}
// Exports run one at a time; one asked for meanwhile waits its turn. "Export and
// upload" sends it by itself once it's exported.
RowLayout {
Controls.Button {
text: backend.exporting ? "Exporting…"
: exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
text: exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
icon.name: "document-export"
enabled: exportView.chosen !== null && !backend.exporting && backend.exportProblem === ""
&& !exportView.chosen.active
onClicked: backend.exportSession(exportView.chosen.id, false)
enabled: exportView.canExport
onClicked: backend.queueExport(exportView.chosen.id, false, false)
}
Controls.Button {
text: "Export and upload"
icon.name: "cloud-upload"
enabled: exportView.canExport && root.canUploadAfter(exportView.chosen.id)
onClicked: backend.queueExport(exportView.chosen.id, false, true)
}
}
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
visible: exportView.chosen !== null && !backend.hubDryRun
&& (!backend.canUploadAfter || !backend.uploadAllowedFor(exportView.chosen.id))
wrapMode: Text.Wrap
opacity: 0.7
text: !backend.canUploadAfter ? "To upload by itself once exported, log in on the Upload page first."
: "Contributions aren't open yet, so it can't upload by itself."
}
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
visible: backend.exporting && exportView.chosen !== null && !exportView.pending
wrapMode: Text.Wrap
opacity: 0.7
text: "Another export is running: this one waits its turn."
}
// The running export
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Exporting now"
visible: backend.exporting
}
RowLayout {
Kirigami.FormData.label: "Round:"
visible: backend.exporting
Controls.Label { text: root.sessionLabel(backend.exportSessionId) }
// Shows the moment Export is pressed: the first progress can take a few seconds.
Controls.BusyIndicator {
visible: backend.exporting
running: visible
running: backend.exporting
implicitWidth: Kirigami.Units.gridUnit * 1.5
implicitHeight: implicitWidth
}
Controls.Button {
visible: backend.exporting
text: "Cancel"
icon.name: "dialog-cancel"
onClicked: backend.cancelExport()
}
}
Controls.CheckBox {
visible: backend.exporting
text: "Upload it once exported"
checked: backend.uploadAfter.indexOf(backend.exportSessionId) >= 0
enabled: checked || root.canUploadAfter(backend.exportSessionId)
onToggled: backend.setUploadAfter(backend.exportSessionId, checked)
}
// Drawn here: the desktop style's ProgressBar paints nothing without a QApplication.
Rectangle {
Kirigami.FormData.label: "Progress:"
@@ -1403,7 +1486,7 @@ Kirigami.ApplicationWindow {
text: backend.exportText
}
RowLayout {
visible: exportView.chosen !== null && exportView.chosen.exported && !backend.exporting
visible: exportView.chosen !== null && exportView.chosen.exported && !exportView.pending
Controls.Button {
text: "Upload"
icon.name: "go-next"
@@ -1412,6 +1495,8 @@ Kirigami.ApplicationWindow {
Controls.Button {
text: "Delete export"
icon.name: "edit-delete"
enabled: !(backend.uploading && backend.upload.session === exportView.chosen.id)
&& backend.uploadQueue.indexOf(exportView.chosen.id) < 0
onClicked: {
const rid = exportView.chosen.id
confirm.ask("Delete this export?", "The export is deleted; the round's recordings stay.",
@@ -1419,6 +1504,107 @@ Kirigami.ApplicationWindow {
}
}
}
// Waiting: exports, then uploads
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Waiting to export"
visible: backend.exportQueue.length > 0
}
Repeater {
model: backend.exportQueue
delegate: RowLayout {
required property var modelData
required property int index
Kirigami.FormData.label: (index + 1) + "."
Controls.Label { text: root.sessionLabel(modelData.session) }
Controls.CheckBox {
text: "then upload"
checked: modelData.upload
enabled: checked || root.canUploadAfter(modelData.session)
onToggled: backend.setUploadAfter(modelData.session, checked)
}
Controls.ToolButton {
icon.name: "list-remove"
text: "Remove"
display: Controls.AbstractButton.IconOnly
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: backend.unqueueExport(modelData.session)
}
}
}
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Uploading"
visible: backend.uploading || backend.uploadQueue.length > 0
}
RowLayout {
Kirigami.FormData.label: "Now:"
visible: backend.uploading
Controls.Label {
text: root.sessionLabel(backend.upload.session || "") + (backend.upload.text ? ": " + backend.upload.text : "")
}
Controls.Button {
text: "Show"
icon.name: "go-next"
flat: true
onClicked: { root.chosenSession = backend.upload.session; root.show(uploadPage) }
}
}
Repeater {
model: backend.uploadQueue
delegate: RowLayout {
required property var modelData
required property int index
Kirigami.FormData.label: "Then:"
Controls.Label { text: root.sessionLabel(modelData) }
Controls.ToolButton {
icon.name: "list-remove"
text: "Don't upload it"
display: Controls.AbstractButton.IconOnly
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: backend.unqueueUpload(modelData)
}
}
}
// How they went, for whoever comes back to the headset
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Done"
visible: backend.doneLog.length > 0
}
Repeater {
model: backend.doneLog
delegate: RowLayout {
required property var modelData
Kirigami.Icon {
source: modelData.error ? "dialog-error" : "checkmark"
implicitWidth: Kirigami.Units.iconSizes.small
implicitHeight: implicitWidth
}
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 24
wrapMode: Text.Wrap
text: modelData.text
}
Controls.ToolButton {
visible: modelData.link !== ""
icon.name: "internet-services"
text: "Open"
onClicked: Qt.openUrlExternally(modelData.link)
}
}
}
Controls.Button {
visible: backend.doneLog.length > 0
text: "Clear"
icon.name: "edit-clear-history"
flat: true
onClicked: backend.clearDoneLog()
}
}
footer: Controls.Label {
@@ -1427,8 +1613,9 @@ Kirigami.ApplicationWindow {
opacity: 0.7
text: "Export leaves out the ranges you deleted, encrypts the eye images and any health answers to "
+ "the dataset's keys (only its maintainers can read them), and adds a manifest and checksums, "
+ "in " + backend.exportsDir + ". It runs at the lowest priority; the headset stays awake "
+ "until it's done. Nothing is uploaded until you press Upload on the Upload page."
+ "in " + backend.exportsDir + ". It runs at the lowest priority, one at a time; the headset "
+ "stays awake until they're done. Nothing is uploaded unless you press Export and upload here, "
+ "or Upload on the Upload page."
}
}
}
@@ -1454,7 +1641,7 @@ Kirigami.ApplicationWindow {
readonly property string blocked: {
if (session === "") return "Choose an export"
if (chosen && chosen.export_stale) return "Export the round again first"
if (backend.exporting) return "Wait for the export to finish"
if (root.exportPending(session)) return "Wait for its export to finish"
if (backend.uploading) return ""
if (backend.hubDryRun) return ""
if (!backend.uploadAllowed) return "Contributions aren't open yet"
@@ -1467,6 +1654,14 @@ Kirigami.ApplicationWindow {
Connections {
target: backend
function onSessionsChanged() { uploadView.refreshInfo() }
// An upload started by itself (queued on the Export page): show it.
function onUploadChanged() {
const i = backend.uploading ? exportBox.indexOfValue(backend.upload.session) : -1
if (i >= 0 && i !== exportBox.currentIndex) {
exportBox.currentIndex = i
exportBox.keep = backend.upload.session
}
}
}
Component.onCompleted: {
refreshInfo()
@@ -1547,8 +1742,19 @@ Kirigami.ApplicationWindow {
textRole: "label"
valueRole: "id"
enabled: !backend.uploading
Component.onCompleted: currentIndex = Math.max(0, indexOfValue(root.chosenSession))
onActivated: root.chosenSession = currentValue
// An export that finishes meanwhile adds a row (newest first): stay on the
// same export, the uploading one while an upload runs.
property string keep: ""
Component.onCompleted: {
currentIndex = Math.max(0, indexOfValue(root.chosenSession))
keep = currentValue || ""
}
onActivated: root.chosenSession = keep = currentValue
onModelChanged: Qt.callLater(() => {
const i = indexOfValue(backend.uploading ? backend.upload.session : keep)
currentIndex = i >= 0 ? i : (count > 0 ? Math.max(0, currentIndex) : -1)
keep = currentValue || ""
})
}
Controls.Label {
opacity: 0.7
@@ -1661,6 +1867,14 @@ Kirigami.ApplicationWindow {
+ "\">its page</a>. Upload checks the export, opens your pull request, then sends the files."
onLinkActivated: link => Qt.openUrlExternally(link)
}
Controls.Label {
Layout.fillWidth: true
visible: backend.uploadQueue.length + backend.uploadAfter.length > 0
wrapMode: Text.Wrap
opacity: 0.7
text: "Queued on the Export page to upload after this one, by themselves: "
+ root.plural(backend.uploadQueue.length + backend.uploadAfter.length, "export") + "."
}
RowLayout {
Controls.Button {
text: backend.uploading ? "Uploading…" : backend.hubDryRun ? "Upload (dry run)" : "Upload"
+3 -1
View File
@@ -1,4 +1,6 @@
# The eye dataset's recipients (gaze/rec/DESIGN.md "Files"): exports encrypt each chunk and health.json
# to all of them (recorder/age.py). Key 1 is kept on the maintainer's training machine, key 2 offline.
# to all of them (recorder/age.py). Key 1 is kept on the 7i, key 2 offline,
# key 3 on dody (the GPU training machine, which decrypts pull requests itself).
age1a0tc98pj2s7gdfgrv4gam6ah0qv69ymf3t2cc25hcwepmats69hst35z2e
age1r0zlq7kcyg4dz4y3vkl9j36jw3zc6ly6reddh9ag5kfy6aw0fy6s5hepat
age1e5vwgc03s9mk3dj5j39lcgxg6qyvkml77qh7g8svygar49r2kd7svldt70
+152
View File
@@ -0,0 +1,152 @@
# Building an eye bounty
A bounty is two things:
- **A board entry** in `gaze/rec/bounties.json`: id, title, blurb, minutes, priority, the script it runs,
and what it needs from the checklist (`recorder/bounties.py` has the format).
- **A round script**, `gaze/rec/rounds/<script>.json`: the steps (`session.py`'s docstring has the
format of every step kind).
Scripts ship with the recorder. The dataset's copy of the board (`DeeJanuz/frametop-eyes`,
`bounties.json`) can only describe, reorder, reprioritise or retire bounties, so a new bounty reaches
people only with a Frametop release that carries its script.
`load_round` enforces the rules marked **(checked)**: a script that breaks one doesn't load, and the
tests fail.
## What the round is for
Every recorded second becomes training data, labelled by where the dot was. So:
- **Know what each step labels.** In most steps the eyes are on a dot, and the dot's position is the
label. Where they aren't (eyes shut, reading, looking around freely), the cue words or the step's
kind say so, and training leaves those frames out or treats them as their own case.
- **One edge case per bounty.** Say in the board's blurb and the DESIGN.md table which case it covers
and why the trackers fail there.
- **Rounds open the same way.** The recorder adds putting the headset on and the nine dots before every
round. Don't repeat them. The group `dots` is theirs **(checked)**.
## What the person sees and hears, step by step
A round moves through the same states every step. Each one has its own way of telling the person what
to do and when, and the step's text and cues have to fit it.
| State | Recorded | What they do | How they know when |
|---|---|---|---|
| Ready screen | no | read the text, press Next | the title, the text, the keys line |
| Countdown 3-2-1 | yes | look at the dot it shows | the numbers |
| Eyes on dots | yes | look at each dot | its ring fills, then the dot moves; the bar along the bottom |
| Eyes shut | yes | close, then open | beeps (below) |
| Moving the headset | yes | push, slide or tilt it by hand, hold, let go | a cue word and a beep each way (below) |
| Head or reading | yes | the one action the text says, until the bar runs out | the bar along the bottom |
- **The ready screen says it all, the action first.** It's the only time they read: what to do, what
starts it ("when it says Push up and beeps"), and what ends it ("until its ring fills", "until the bar
runs out", "until it says Let go"). A step with no ready screen (`"ready": false`) can only add to that
in a few words, as they're doing something already.
- **Nothing starts in the countdown.** It points the eyes at the first dot. A one-off action (a blink, a
push) starts on a cue word after it, never on "go" from the numbers.
- **One-off actions get a cue word; ongoing ones get the bar.** Turning the head, reading or looking
around run until the bar runs out. A push, a blink or closing the eyes happens on its cue.
- **Redo does a chain again.** R or two presses during a step start it again, from the first step of its
chain; at a ready screen they go back to the step before. Make a chain one thing to do over: a nudge,
the dots done holding it, and letting go.
## Cues the person can't see get a beep
The panel can't cue someone whose eyes are shut. A cue that asks for that plays a beep
(`gaze/rec/sounds.py`) as its third field: `["Close your eyes", 2.5, "close"]`.
- **Blink:** a cue whose word asks for a blink plays `blink` **(checked)**.
- **Both eyes shut:** a cue asking to close or shut both eyes plays `close` **(checked)**, and the next
cue plays `open` **(checked)**, so the person hears when to open them. A step never ends with the eyes
shut.
- **Say it in the step's text**, which is read before Next: "Close your eyes when you hear a beep, and
then open your eyes when you hear the beep again." The words still show on the panel too.
- **One eye or half-closed** (winks, squints, half-closed lids) keep words alone: the open eye reads
them. They may beep too.
- **No voice.** Beeps only; generated speech sounds odd in the headset. A new kind of cue sound goes
into `sounds.SOUNDS` and the tests.
## Moving the headset by hand
A hand on the headset can't press Next, and the moment it moves is what the data is about. So:
- **A cue word and a beep start it:** a cue word starting with push, pull, slide, tilt, tip or nudge plays
`go` **(checked)**: `["Push up", 2.5, "go"]`. Say the direction in the word, the same way each time
("to your left", "left side up").
- **Eyes first:** a cue on the dot comes before it, so there's a still moment to compare with
**(checked)**: `["Look at the dot", 1.5]`.
- **Letting go has its own cue and beep:** "Let go" plays `done` **(checked)** and shows 2 s or more
**(checked)**, long enough to put the headset back and look at the dot again.
- **In the same chain:** the `done` cue comes before the next ready screen, a new group or a reseat
**(checked)**, so a redo does the push, the dots and the letting go again.
- **Steps done while holding say so:** their text starts "Keep holding" **(checked)**.
- **Say exactly how, in the ready screen's text:** where the hand goes (the same grip each time), which way
it moves, how far, what stays still ("Keep it level", "Its middle stays where it is"), and that the head
and eyes don't move. "Push the headset up" alone leaves people guessing.
- **A few millimetres, one hand, gently.** Say it in the round's intro, with the kinds of move to come, and
that a little blur while it's moved is fine. Never press it into the face.
## Cue words are labels
Training reads the cue words in `prompts.jsonl` (`frame-eyes/train/data/ingest.py`, `shut_cams`):
- A word with **blink, close, shut, wink or yawn** marks the eyes as shut while it shows. **Left** or
**right** limits that to one eye, and **half** means the lids are partly open, so the frames keep
their labels.
- Use those words only for that. "Open", "Look", "Relax" and "Normal" are the words for going back.
- Keep words short (40 characters at most): they show big on the panel.
## Timing
- **Dots:** a hold of 1.2 s or more after a settle of about 0.6 s (the defaults). The eyes take that
long to land on a dot and stay.
- **Shut eyes:** 2 s or more closed, then 2 s or more open before anything else, so the eyes settle back
on the dot.
- **Blinks on a beep:** uneven gaps, 1.2 to 2 s, so the beep can't be predicted.
- **Length:** about 4 to 6 minutes recorded. Check with `python3 gaze/rec/session.py <script> --plan`, and
put that, rounded, in the board's `minutes`.
## Where things can go
- **Dots and paths stay on the panel** **(checked)**: about 30 degrees to the sides, 23 up and down, less
in the corners (`off_panel`).
- **Distance:** 0.25 to 10 m **(checked)**. 1.5 m unless distance is what the bounty is about.
- **Head still,** unless the bounty is about moving it: say "Keep your head still" in the text of dot steps.
## Takes
- Steps of the same `group` share one take, and a group's steps must follow each other **(checked)**.
Group steps that a reviewer would delete together.
- A `reseat` step isn't recorded, and a round with one gets no shift check.
- Every step waits for Next unless it has `"ready": false`. Only chain steps that follow on naturally.
## Comfort, safety and privacy
- Comfortable eye turns only: "as far as is comfortable", never "as far as you can".
- No flashing: brightness changes are steps of a second or more (`light` levels), never a flicker.
- Nothing that asks for sound, the outward cameras, or anything about the person beyond the checklist.
Talking is fine (it moves the face), but the microphone is never recorded.
- Text to read is neutral and made up: no names, places or questions about the person.
## Writing the text
- Plain words, one instruction per step, the action first: "Look at each dot until its ring fills."
- `|` starts a new line on the panel, `||` leaves a blank line.
- Say what to do, not why. The why goes in the round's `intro` or the board's blurb.
## Before it ships
1. `python3 gaze/rec/session.py <script> --plan`: the steps and their length.
2. The tests, in the dev container: `cd gaze/rec && for t in tests/test_*.py; do python3 $t; done`.
`test_shipped_rounds_load` loads every script; update its count.
3. A dry run: `python3 gaze/rec/session.py <script> --dry-run --speed 20 --next-after 0.1`.
4. A replay end to end: `frame-eyes/tools/eyerec-dryrun <bounty>` (record, export, validate), and the
export through `eyes_hub.py check` and `ingest.py` on dody. Delete those rounds afterwards: their
labels are wrong.
5. A real round in the headset, with a note in `HEADSET-TESTS.md`: is every cue clear, and is the
length right?
6. Add the bounty to `gaze/rec/bounties.json` and a row to DESIGN.md's bounty table. After a release
carries the script, put it on the dataset's board too (`frame-eyes/dataset-repo/bounties.json`,
uploaded to `DeeJanuz/frametop-eyes`).
+31 -19
View File
@@ -2,12 +2,12 @@
"schema": 1,
"id": "blinks",
"title": "Blinks and lids",
"intro": "Blinks, closed and half-closed eyes. Keep looking at the dot whenever your eyes are open.",
"intro": "Blinks, closed and half-closed eyes, some of them on a beep. Keep looking at the dot whenever your eyes are open.",
"steps": [
{
"id": "blink-cue",
"kind": "prompt",
"text": "Look at the dot and blink each time the word says so.",
"text": "Look at the dot and blink once on each beep.",
"dot": [
0,
0
@@ -15,42 +15,50 @@
"cues": [
[
"Blink",
1.5
1.6,
"blink"
],
[
"Blink",
1.5
1.3,
"blink"
],
[
"Blink",
1.5
2.0,
"blink"
],
[
"Blink",
1.5
1.5,
"blink"
],
[
"Blink",
1.5
1.8,
"blink"
],
[
"Blink",
1.5
1.2,
"blink"
],
[
"Blink",
1.5
1.7,
"blink"
],
[
"Blink",
1.5
1.5,
"blink"
]
]
},
{
"id": "closed",
"kind": "prompt",
"text": "Look at the dot, then close both eyes when it says so.",
"text": "Close your eyes when you hear a beep, and then open your eyes when you hear the beep again.",
"dot": [
0,
0
@@ -61,20 +69,24 @@
2
],
[
"Close both eyes",
2.5
"Close your eyes",
2.5,
"close"
],
[
"Open",
2
"Open your eyes",
2,
"open"
],
[
"Close both eyes",
2.5
"Close your eyes",
2.5,
"close"
],
[
"Open",
2
"Open your eyes",
2,
"open"
]
]
},
+298 -20
View File
@@ -2,25 +2,35 @@
"schema": 1,
"id": "slips",
"title": "Headset slips",
"intro": "You'll nudge the headset on your face on purpose, then put it back. Use one hand, gently: a few millimetres is enough.",
"intro": "You'll move the headset on your face on purpose, seven ways: up, down, to your left, to your right, tilted, tipped forward and away from your face. Each time it's one hand on the front and a few millimetres, no more; each step says how. The picture may blur a little while it's moved: that's fine. Then you take it off and put it back on.",
"steps": [
{
"id": "nudge-up",
"kind": "prompt",
"group": "nudges",
"text": "Keep looking at the dot and push the headset up a little. Hold it there.",
"text": "Look at the dot. When it says Push up and beeps:|Hold the front with one hand, thumb under the bottom edge, and push the headset straight up a few millimetres, so it sits higher on your face. Keep it level: both sides go up together.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
"dot": [
0,
0
],
"seconds": 5
"cues": [
[
"Look at the dot",
1.5
],
[
"Push up",
2.5,
"go"
]
]
},
{
"id": "after-up",
"kind": "dots",
"group": "nudges",
"ready": false,
"text": "Hold the headset there. Look at each dot until its ring fills. Keep your head still.",
"text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
"dots": [
[
0,
@@ -46,22 +56,50 @@
"hold": 1.0
},
{
"id": "nudge-down",
"id": "back-up",
"kind": "prompt",
"group": "nudges",
"text": "Let go, then pull the headset down a little. Hold it there.",
"ready": false,
"text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
"dot": [
0,
0
],
"seconds": 5
"cues": [
[
"Let go, put it back",
3,
"done"
]
]
},
{
"id": "nudge-down",
"kind": "prompt",
"group": "nudges",
"text": "Look at the dot. When it says Pull down and beeps:|Hold the front with one hand, thumb under the bottom edge, and pull the headset straight down a few millimetres, the way a loose headset sags. Keep it level: both sides go down together.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
"dot": [
0,
0
],
"cues": [
[
"Look at the dot",
1.5
],
[
"Pull down",
2.5,
"go"
]
]
},
{
"id": "after-down",
"kind": "dots",
"group": "nudges",
"ready": false,
"text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
"text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
"dots": [
[
0,
@@ -87,22 +125,50 @@
"hold": 1.0
},
{
"id": "nudge-left",
"id": "back-down",
"kind": "prompt",
"group": "nudges",
"text": "Let go, then slide the headset a little to your left.",
"ready": false,
"text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
"dot": [
0,
0
],
"seconds": 5
"cues": [
[
"Let go, put it back",
3,
"done"
]
]
},
{
"id": "nudge-left",
"kind": "prompt",
"group": "nudges",
"text": "Look at the dot. When it says Slide to your left and beeps:|Hold the front with one hand, thumb under the bottom edge, and slide the headset a few millimetres to your left, across your face. Keep it level and at the same height.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
"dot": [
0,
0
],
"cues": [
[
"Look at the dot",
1.5
],
[
"Slide to your left",
2.5,
"go"
]
]
},
{
"id": "after-left",
"kind": "dots",
"group": "nudges",
"ready": false,
"text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
"text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
"dots": [
[
0,
@@ -128,22 +194,50 @@
"hold": 1.0
},
{
"id": "nudge-right",
"id": "back-left",
"kind": "prompt",
"group": "nudges",
"text": "Let go, then slide it a little to your right.",
"ready": false,
"text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
"dot": [
0,
0
],
"seconds": 5
"cues": [
[
"Let go, put it back",
3,
"done"
]
]
},
{
"id": "nudge-right",
"kind": "prompt",
"group": "nudges",
"text": "Look at the dot. When it says Slide to your right and beeps:|Hold the front with one hand, thumb under the bottom edge, and slide the headset a few millimetres to your right, across your face. Keep it level and at the same height.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
"dot": [
0,
0
],
"cues": [
[
"Look at the dot",
1.5
],
[
"Slide to your right",
2.5,
"go"
]
]
},
{
"id": "after-right",
"kind": "dots",
"group": "nudges",
"ready": false,
"text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
"text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
"dots": [
[
0,
@@ -169,22 +263,50 @@
"hold": 1.0
},
{
"id": "nudge-tilt",
"id": "back-right",
"kind": "prompt",
"group": "nudges",
"text": "Let go, then tilt the headset a little: one side up.",
"ready": false,
"text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
"dot": [
0,
0
],
"seconds": 5
"cues": [
[
"Let go, put it back",
3,
"done"
]
]
},
{
"id": "nudge-tilt",
"kind": "prompt",
"group": "nudges",
"text": "Look at the dot. When it says Tilt: left side up and beeps:|Hold the front with one hand, thumb under the bottom edge, and turn your wrist so the headset's left side goes up a few millimetres and its right side down, so it sits crooked on your face. Its middle stays where it is.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
"dot": [
0,
0
],
"cues": [
[
"Look at the dot",
1.5
],
[
"Tilt: left side up",
2.5,
"go"
]
]
},
{
"id": "after-tilt",
"kind": "dots",
"group": "nudges",
"ready": false,
"text": "Hold it there. Look at each dot until its ring fills. Keep your head still.",
"text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
"dots": [
[
0,
@@ -209,6 +331,162 @@
],
"hold": 1.0
},
{
"id": "back-tilt",
"kind": "prompt",
"group": "nudges",
"ready": false,
"text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
"dot": [
0,
0
],
"cues": [
[
"Let go, put it back",
3,
"done"
]
]
},
{
"id": "nudge-tip",
"kind": "prompt",
"group": "nudges",
"text": "Look at the dot. When it says Tip the front down and beeps:|Hold the front with one hand, thumb under the bottom edge, and tip the front down a little, the way a heavy headset droops: its bottom edge goes down a few millimetres and its top stays against your forehead. Keep it level side to side.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
"dot": [
0,
0
],
"cues": [
[
"Look at the dot",
1.5
],
[
"Tip the front down",
2.5,
"go"
]
]
},
{
"id": "after-tip",
"kind": "dots",
"group": "nudges",
"ready": false,
"text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
"dots": [
[
0,
0
],
[
15,
12
],
[
-15,
-12
],
[
15,
-12
],
[
-15,
12
]
],
"hold": 1.0
},
{
"id": "back-tip",
"kind": "prompt",
"group": "nudges",
"ready": false,
"text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
"dot": [
0,
0
],
"cues": [
[
"Let go, put it back",
3,
"done"
]
]
},
{
"id": "nudge-away",
"kind": "prompt",
"group": "nudges",
"text": "Look at the dot. When it says Pull away from your face and beeps:|Hold the front with one hand, thumb under the bottom edge, and pull the headset a few millimetres straight out, away from your face, so there's a small gap at your cheeks. Gently: the strap pulls back. Keep it level.|Only the headset moves: your head stays still and your eyes stay on the dot. Hold it there until it says Let go and beeps low.",
"dot": [
0,
0
],
"cues": [
[
"Look at the dot",
1.5
],
[
"Pull away from your face",
2.5,
"go"
]
]
},
{
"id": "after-away",
"kind": "dots",
"group": "nudges",
"ready": false,
"text": "Keep holding the headset there. Look at each dot until its ring fills. Keep your head still.",
"dots": [
[
0,
0
],
[
15,
12
],
[
-15,
-12
],
[
15,
-12
],
[
-15,
12
]
],
"hold": 1.0
},
{
"id": "back-away",
"kind": "prompt",
"group": "nudges",
"ready": false,
"text": "Keep holding it until it says Let go. Then put the headset back as it was, and look at the dot.",
"dot": [
0,
0
],
"cues": [
[
"Let go, put it back",
3,
"done"
]
]
},
{
"id": "reseat",
"kind": "reseat",
+252 -61
View File
@@ -8,11 +8,13 @@ Then, in step mode:
1. Fitment, not a take: the fit coach (gaze/fitcoach.py) on a centre dot until the headset
sits well and the person presses Next; Next before that warns, a second press goes on and
the round records it was skipped. Its readings go into fit.json.
2. On the contributor's first round, or when SteamVR's IPD setting changed since their last,
an IPD sweep: one take, the wheel turned end to end while looking at a centre dot.
3. Nine dots, a take: a 3x3 grid to 20 degrees on the mid background, the round's reference.
4. The bounty's steps, from its round script (rounds/<script>.json, below).
the round records it was skipped. Its readings go into fit.json. The coach is off for now
(FIT_COACH): then this only asks for a comfortable fit, Next goes on, and its readings are
kept without being shown. Either way it shows the IPD setting, read from SteamVR when the
round starts: it can't be changed with the headset on, so there's no IPD step, and a wrong
setting means stopping and setting it first.
2. Nine dots, a take: a 3x3 grid to 20 degrees on the mid background, the round's reference.
3. The bounty's steps, from its round script (rounds/<script>.json, below).
Each step that's "ready" shows first and waits for Next; then a 3-2-1 countdown, recorded, then
the step. Steps of the same group share a take (takes/NN-group): the eye frames (eyerec.py),
@@ -22,9 +24,12 @@ steps are recorded; the frame writer pauses while a step waits. During the round
looks again at the end of each centre dot, and a headset that moved since the fitment gets a
note (fit.json "slips").
Controls, as the hand recorder's: Next, pause, redo (the step running, or at a ready screen the
one before; the range is marked with a "redo" event), skip (the rest of the take), stop. The
headset button: press = next or pause, two presses = redo, hold = stop.
Controls, as the hand recorder's: Next, pause, redo, skip (the rest of the take), stop. Redo in
the countdown or the step starts it again; at a ready screen it goes back to the step before,
into the take before too (Back). A step that follows on without a ready screen goes again with
the one it follows. The range done before is marked with a "redo" event. The headset button:
press = next or pause, two presses = redo, hold = stop. The panel's keys line says what redo
does at that moment.
Round scripts, rounds/<id>.json: {"schema": 1, "id", "title", "intro", "shift_check": true,
"steps": [step]}. A step: {"id", "kind", "text" ("|" starts a line), "group" (default its id),
@@ -34,7 +39,10 @@ Round scripts, rounds/<id>.json: {"schema": 1, "id", "title", "intro", "shift_ch
path "path": "circle|line|eight <numbers>" (the panel's path command), "seconds"
jumps "count", "range": [yaw, pitch] (a box), "each": [shortest, longest] s
prompt "dot": [yaw, pitch] or null (none), and "seconds" or "cues": [[word, s]...] (shown
big in turn), "cue_distance": {word: metres}
big in turn; [word, s, sound] also plays a beep as it starts, for steps done with the
eyes shut or moving the headset by hand: sound blink, close, open, go or done,
sounds.py; rounds/README.md has the rules, and load_round enforces them),
"cue_distance": {word: metres}
light "dot", "levels": [[background, s]...]
world "point": [x, y, z] (metres from the head when it starts, -z ahead), "seconds"
reseat not recorded: the text, Next, then a fit reading
@@ -76,6 +84,7 @@ sys.path.insert(0, GAZE)
sys.path.insert(0, HERE)
import eyerec # noqa: E402 (gaze/rec/eyerec.py: the frame writer)
import fitcoach # noqa: E402 (gaze/fitcoach.py)
import sounds # noqa: E402 (gaze/rec/sounds.py: the beeps)
from recorder import bounties # noqa: E402
from recorder.button import BUTTON_DEVICE, ButtonReader, find_button, mouse_connected, read_input_devices # noqa: E402
from recorder.common import (clean_text, clock_sample, in_container, mono_ns, os_version, # noqa: E402
@@ -110,25 +119,35 @@ FIT_ASK_S = 0.5 # the fit coach's reading, this often during the fitme
FIT_GOOD_S = 2.0 # OK this long: the fit is good
READING_WAIT_S = 6.0 # a reseat's fit reading waits this long for both eyes
CENTRE_DEG = 1.0 # a dot this near the middle gets the shift check after its hold
SPLIT_IPD_M = 0.0005 # the IPD setting changed: an IPD sweep
VERSION = "ft-eyerec 0.1"
READY_TEXT = "Ready? Press Space or click Next"
READY_BUTTON = "Ready? Press the button on the right side of the headset"
READY_BUTTON_MOUSE = "Ready? Press Space, click Next, or press the headset button"
RESUME_HINT = "Paused. Resume: P in the Eye recorder window"
RESUME_HINT_BUTTON = "Paused. Resume: press the headset button, or P in the Eye recorder window"
KEYS = "Eye recorder window: Space next · P pause · R redo · S skip · Esc stop"
KEYS_BUTTON = ("Headset button: press for next or pause · press twice to redo · hold to stop "
"· Window: Space P R S Esc")
# What R (two presses of the headset button) does now, for the panel's keys line (_keys): during
# the countdown and the step, it starts again; at a ready screen, the step before goes again.
REDO_RUNNING = "start this step again"
REDO_BACK = "back to the last step"
NOTHING_BACK = "Nothing to go back to from here"
FIT_TITLE = "Headset fit"
FIT_TEXT = "Look at the dot. Adjust the headset until this says good."
FIT_ADJUST = "Adjust the headset until it says good"
FIT_SKIP_NOTE = "The fit isn't good yet: recordings may be less useful. Press again to start anyway."
IPD_STEP = {"id": "ipd", "kind": "prompt", "group": "ipd", "dot": [0, 0], "seconds": 25,
"text": "Keep looking at the dot and turn the IPD wheel slowly: all the way narrow, then all the "
"way wide, then back to where the picture is sharpest."}
# The fit coach is off until it's trained on more people's fits (2026-10-16: its hints jumped
# around and "good" was out of reach). Off, the fitment only asks for a comfortable fit and a look
# at the dot, Next goes straight on, and the coach's readings still go into fit.json, silently:
# they're what it will be trained on. No baseline then, so no slip notes either.
# FT_EYEREC_FIT_COACH=1 (or session.py --fit-coach) turns it back on.
FIT_COACH = os.environ.get("FT_EYEREC_FIT_COACH", "") == "1"
FIT_OFF_TITLE = "Put the headset on"
FIT_OFF_TEXT = "Put the headset on so it's comfortable and the picture is sharp. Then look at the dot."
# The IPD setting is the headset's (SteamOS, read through SteamVR), shown at the fitment. It can't be
# changed with the headset on, so a wrong one means stopping the round.
IPD_TEXT = "Your IPD setting: {mm:.1f} mm. If that's not right for you, stop, take the headset off and set it."
IPD_UNKNOWN = "Couldn't read the IPD setting from SteamVR."
NINE_DOTS_TEXT = "Look at each dot until its ring fills. Keep your head still."
DONE_SCREEN = {"title": "All done", "text": "Thank you! You can take off the headset.|Review the round in the Eye "
DONE_SCREEN = {"title": "All done", "text": "Thank you! This recording session will close. Continue in the application.|Review the round in the Eye "
"recorder window.", "seconds": 5}
STOPPED_SCREEN = {"title": "Stopped", "text": "What was recorded so far is saved. Review it in the Eye recorder window.",
"seconds": 4}
@@ -203,6 +222,74 @@ def _dot(v, what):
return d
HAND_WORDS = ("push", "pull", "slide", "tilt", "tip", "nudge") # a cue word starting so moves the headset
HOLD_TEXT = "Keep holding" # the text of a step done while holding the headset moved
def cue_needs(word):
"""The sound a cue must have because the person can't see the panel then, or the timing is
the point (rounds/README.md): "blink" for a blink, "close" for shutting both eyes, "go" for
moving the headset by hand, "done" for letting go of it, else None. Half-closed eyes and one
eye (a wink: the other one reads the panel) don't need one."""
w = word.lower()
if w.split()[:1] and w.split()[0].strip(":,") in HAND_WORDS:
return "go"
if "let go" in w:
return "done"
if "half" in w or "left" in w or "right" in w:
return None
if "blink" in w:
return "blink"
if "close" in w or "shut" in w:
return "close"
return None
def _check_sounds(cues, sid):
"""Eyes-shut cues beep, and the cue after a "close" plays "open": with the eyes shut, the
words can't tell anyone when to open them. A push on the headset ("go") comes after a cue on
the dot, and letting go ("done") shows long enough to settle."""
for k, (word, _secs, *sound) in enumerate(cues):
need = cue_needs(word)
if need and sound != [need]:
raise ValueError('%s: cue "%s" needs the sound "%s" (the eyes are shut then; rounds/README.md)'
% (sid, word, need))
if sound == ["close"] and (k + 1 == len(cues) or cues[k + 1][2:] != ["open"]):
raise ValueError('%s: after "%s" comes a cue with the sound "open" (rounds/README.md)' % (sid, word))
if sound == ["go"] and k == 0:
raise ValueError('%s: "%s" comes after a cue on the dot, so the eyes are still first (rounds/README.md)'
% (sid, word))
if sound == ["done"] and _secs < 2:
raise ValueError('%s: "%s" shows 2 s or more, so the eyes settle (rounds/README.md)' % (sid, word))
def _check_hands(steps, path):
"""A headset moved by hand is let go of in the same chain (the steps that follow on without a
ready screen), and the steps done while holding it say so: the person can't take a hand off
the headset to press Next, and a redo does the whole chain again."""
holding = None # the step whose "go" cue moved the headset
for k, st in enumerate(steps):
if holding and (st["ready"] or st["group"] != holding["group"] or st["kind"] == "reseat"):
raise ValueError('%s: %s moves the headset; a "done" cue ("Let go...") follows in its chain '
'(rounds/README.md)' % (path, holding["id"]))
sounds_ = [c[2] for c in st.get("cues") or [] if len(c) > 2]
if holding and not st["text"].startswith(HOLD_TEXT):
raise ValueError('%s: %s is done holding the headset; its text starts "%s" (rounds/README.md)'
% (path, st["id"], HOLD_TEXT))
for snd in sounds_:
if snd == "go":
if holding:
raise ValueError('%s: %s moves the headset again before letting go' % (path, st["id"]))
holding = st
elif snd == "done":
if not holding:
raise ValueError('%s: %s lets go of a headset nothing moved' % (path, st["id"]))
holding = None
if holding:
raise ValueError('%s: %s moves the headset; a "done" cue ("Let go...") follows in its chain '
'(rounds/README.md)' % (path, holding["id"]))
def _step(raw, i):
"""One step, checked and filled in."""
if not isinstance(raw, dict):
@@ -263,12 +350,16 @@ def _step(raw, i):
st["dot"] = None if "dot" in raw and raw["dot"] is None else _dot(raw.get("dot", [0, 0]), sid + " dot")
cues = raw.get("cues")
if cues is not None:
want = "%s: cues want [[word, seconds]...] or [word, seconds, sound] (sound: %s)" % (
sid, ", ".join(sounds.SOUNDS))
if not isinstance(cues, list) or not cues:
raise ValueError("%s: cues want [[word, seconds]...]" % sid)
st["cues"] = [[clean_text(c[0])[:40], _num(c[1], 0.2, 60, sid + " cue")] for c in cues
if isinstance(c, list) and len(c) == 2 and isinstance(c[0], str)]
raise ValueError(want)
st["cues"] = [[clean_text(c[0])[:40], _num(c[1], 0.2, 60, sid + " cue")] + c[2:] for c in cues
if isinstance(c, list) and len(c) in (2, 3) and isinstance(c[0], str)
and (len(c) == 2 or c[2] in sounds.SOUNDS)]
if len(st["cues"]) != len(cues):
raise ValueError("%s: cues want [[word, seconds]...]" % sid)
raise ValueError(want)
_check_sounds(st["cues"], sid)
cd = raw.get("cue_distance") or {}
st["cue_distance"] = {str(k): _num(v, 0.25, 10, sid + " cue distance") for k, v in cd.items()}
else:
@@ -311,8 +402,9 @@ def load_round(path):
raise ValueError("%s: group %s comes back after another" % (path, s["group"]))
groups.add(s["group"])
last = s["group"]
if groups & {"dots", "ipd"}:
raise ValueError("%s: the groups dots and ipd are the opener's" % path)
if "dots" in groups:
raise ValueError("%s: the group dots is the opener's" % path)
_check_hands(steps, path)
rid = raw.get("id")
return {"id": rid if isinstance(rid, str) else os.path.basename(path)[:-5], "title": str(raw.get("title", rid)),
"intro": str(raw.get("intro", "")), "shift_check": raw.get("shift_check", True) is not False and
@@ -346,13 +438,10 @@ def nine_dots(rng):
"shuffle": False}
def build_plan(rnd, seed, ipd_sweep):
def build_plan(rnd, seed):
"""The round's steps in order, the opener's first, shuffles and jumps made from seed."""
rng = random.Random(seed)
steps = []
if ipd_sweep:
steps.append(_step(IPD_STEP, 0))
steps.append(nine_dots(rng))
steps = [nine_dots(rng)]
for s in rnd["steps"]:
s = json.loads(json.dumps(s))
if s["kind"] == "dots" and s["shuffle"]:
@@ -561,7 +650,7 @@ class Session:
def __init__(self, base_dir, script, *, bounty=None, profile=None, checklist=None, health=None, dry_run=False,
speed=1.0, on_status=None, next_after=None, cams=None, panel_bin=None, panel_no_vr=False,
steamvr_gaze=True, button=True, button_device=None, seed=None, rounds_dir=None,
profile_path=None, eyes_python=None):
profile_path=None, eyes_python=None, fit_coach=None):
self.base_dir = os.path.abspath(os.path.expanduser(base_dir or BASE_DIR))
self.script = script
self.round = load_round(round_path(script, rounds_dir))
@@ -575,6 +664,7 @@ class Session:
self.steamvr_gaze = steamvr_gaze and not dry_run
self.button, self.button_device = button, button_device
self.eyes_python = eyes_python or EYES_PYTHON
self.fit_coach = FIT_COACH if fit_coach is None else bool(fit_coach)
self.seed = seed
self.print = print
self.quiet_units = () if dry_run or cams else QUIET_UNITS
@@ -589,10 +679,11 @@ class Session:
self._status = {"state": "starting", "bounty": self.bounty["id"], "title": "", "step": "", "step_index": 0,
"step_count": 0, "prompt": "", "note": "", "take": None, "error": "", "waiting": False,
"countdown": 0, "big": "", "can_redo": False, "fit": None, "fit_words": "", "fit_card": None,
"ipd": None, "ready_text": READY_TEXT, "button": False, "seconds_left": 0.0}
"ipd": None, "ready_text": READY_TEXT, "button": False, "seconds_left": 0.0,
"fit_coach": self.fit_coach}
self._last_emit = 0.0
self._log_file = None
self._panel = self._eyes = None
self._panel = self._eyes = self._sounds = None
self._procs = {} # name: Popen
self._gaze_thread = None
self._gaze_lock = threading.Lock()
@@ -605,6 +696,7 @@ class Session:
self._paused = self._paused_recording = False
self._waiting = False
self._redo_ok = False
self._lingering = False
self._button_reader = None
self._ipd = None
self._step_t0 = None
@@ -739,6 +831,7 @@ class Session:
def _setup(self):
self._panel = Socket(PANEL_SOCKET, self.dry_run, fake_panel, self._log, self.print)
self._eyes = Socket(EYES_SOCKET, self.dry_run, fake_eyes(self.ref), self._log)
self._sounds = sounds.Player(on=not self.dry_run, log=self._log)
if not self.dry_run:
for path, build in ((self.panel_bin, "gaze/build.sh"), (FT_EYES, "gaze/tracker/build.sh")):
if not os.path.exists(path):
@@ -755,9 +848,8 @@ class Session:
r = self._panel.cmd("ipd", reply=True)
self._ipd = float(r.split()[1]) if r and r.startswith("ok ") else None
self._status["ipd"] = self._ipd
last = self.profile.get("eyes_ipd")
sweep = self._ipd is not None and (not isinstance(last, (int, float)) or abs(self._ipd - last) >= SPLIT_IPD_M)
self.plan = build_plan(self.round, self.seed, sweep)
self._log("IPD setting: %s" % ("%.1f mm" % (self._ipd * 1000) if self._ipd is not None else "unknown"))
self.plan = build_plan(self.round, self.seed)
self._fit["reference"] = {"source": self.ref.source, "faces": self.ref.faces, "made": self.ref.made,
"provisional": self.ref.provisional} if self.ref else None
self._session_json = {
@@ -765,7 +857,7 @@ class Session:
"title": self.bounty.get("title") or self.round["title"], "board_updated": self.bounty.get("updated"),
"checklist": self.checklist, "recorder": VERSION, "os": os_version(), "steamvr": steamvr_version(),
"started": time.strftime("%Y-%m-%dT%H:%M:%S%z"), "ended": None, "status": "running", "seed": self.seed,
"dry_run": self.dry_run, "replay": self.cams or None, "ipd_start": self._ipd, "ipd_sweep": sweep,
"dry_run": self.dry_run, "replay": self.cams or None, "ipd_start": self._ipd,
"fit": None, "eye_fps": None, "takes": [],
"plan": [{"id": s["id"], "kind": s["kind"], "group": s["group"]} for s in self.plan]}
if self.health:
@@ -855,7 +947,7 @@ class Session:
def _finish(self, state, error=""):
if self._take:
try:
self._end_take("complete" if state == "done" else "stopped")
self._end_take("complete" if state == "done" or self._lingering else "stopped")
except Exception:
self._log(traceback.format_exc())
if self._session_json is not None:
@@ -921,6 +1013,9 @@ class Session:
raise _Skip()
if want["redo"] and self._redo_ok:
raise _Redo()
if want["redo"] and self._waiting:
self._panel.set("note", "note " + NOTHING_BACK)
self._emit(note=NOTHING_BACK)
if not want["pause"]:
return False
self._pause()
@@ -1022,13 +1117,28 @@ class Session:
text = (READY_BUTTON_MOUSE if mouse_connected(self.input_devices) else READY_BUTTON) if button else READY_TEXT
if action:
self._panel.set("action", "action " + text)
self._panel.set("keys", "keys " + (KEYS_BUTTON if button else KEYS))
self._keys(back=self._redo_ok)
self._status.update(ready_text=text, button=button)
def _keys(self, running=False, back=False):
"""The panel's bottom line: the controls, saying what R does now (or leaving it out)."""
redo = REDO_RUNNING if running else REDO_BACK if back else ""
if self._button_ok():
parts = ["Headset button: press for " + ("pause" if running else "next")]
parts += ["press twice: " + redo] if redo else []
text = " · ".join(parts + ["hold to stop", "Window: Space P R S Esc"])
else:
parts = ["Eye recorder window: Space next", "P pause"] + (["R " + redo] if redo else [])
text = " · ".join(parts + ["S skip", "Esc stop"])
self._panel.set("keys", "keys " + text)
def _show(self, title, text):
self._panel.set("title", "title " + clean_text(title))
self._panel.set("text", "text " + clean_text(text))
def _ipd_text(self):
return IPD_UNKNOWN if self._ipd is None else IPD_TEXT.format(mm=self._ipd * 1000)
# --- the fit coach
def _reading(self, previous=None, baseline=None):
"""The fit coach's reading of ft-eyes' last second, and the finds it's from (or None, None)."""
@@ -1039,10 +1149,13 @@ class Session:
def _fitment(self):
"""The fit coach on a centre dot until the fit is good and Next is pressed; Next before
that warns, and a second press goes on (fit.json records it was skipped)."""
that warns, and a second press goes on (fit.json records it was skipped). With the coach
off (FIT_COACH), _fitment_quiet instead."""
if not self.fit_coach:
return self._fitment_quiet()
title = "%s: %s" % (FIT_TITLE, self._session_json["title"])
intro = self.round["intro"]
self._show(title, FIT_TEXT + ("||" + intro if intro else ""))
self._show(title, FIT_TEXT + "||" + self._ipd_text() + ("||" + intro if intro else ""))
self._panel.cmd("dot 0 0 look")
f = {"reading": None, "summary": None, "since": 0.0, "good": False, "ask": 0.0, "warned": False}
started = mono_ns()
@@ -1069,7 +1182,7 @@ class Session:
self._panel.set("note", "note ")
self._panel.cmd("dot 0 0 %s" % ("done" if good else "look"))
self._panel.set("action", "action " + (self._status["ready_text"] if good or f["warned"] else FIT_ADJUST))
self._show(title, "Headset position: " + words + ("||" + intro if intro else ""))
self._show(title, "Headset position: " + words + "||" + self._ipd_text() + ("||" + intro if intro else ""))
self._emit(force=False, fit=r["state"] if r else None, fit_words=words, fit_card=fit_card(r))
self._redo_ok = False
@@ -1098,6 +1211,37 @@ class Session:
self._panel.cmd("dot 0 0 off")
self._emit(note="", fit=None, fit_words="", fit_card=None)
def _fitment_quiet(self):
"""The fitment with the coach off: a comfortable fit and the centre dot, Next goes on.
The coach's readings are kept in fit.json, but nothing of them is shown."""
intro = self.round["intro"]
self._show("%s: %s" % (FIT_OFF_TITLE, self._session_json["title"]),
FIT_OFF_TEXT + "||" + self._ipd_text() + ("||" + intro if intro else ""))
self._panel.cmd("dot 0 0 look")
self._panel.set("action", "action " + self._status["ready_text"])
f = {"reading": None, "ask": 0.0}
started = mono_ns()
def tick(dt):
f["ask"] -= dt
if f["ask"] > 0:
return
f["ask"] = FIT_ASK_S
r, _ = self._reading(f["reading"])
if r is not None:
f["reading"] = r
self._fit["readings"].append({"t": mono_ns(), "phase": "fitment", **compact(r)})
self._redo_ok = False
self._await_next(tick)
self._fit["fitment"] = {"from": started, "to": mono_ns(), "result": "off", "ipd": self._ipd,
"last": compact(f["reading"]) if f["reading"] else None}
self._session_json["fit"] = "off"
self._save_session()
self._save_fit()
self._log("fitment: the coach is off%s" % ((", it read " + coach_words(f["reading"])) if f["reading"] else ""))
self._panel.cmd("dot 0 0 off")
def _shift_check(self, step):
"""After a centre dot's hold: has the headset moved since the fitment?"""
if not self.round["shift_check"] or not self._fit["baseline"]:
@@ -1134,7 +1278,8 @@ class Session:
if r:
self._fit["readings"].append({"t": mono_ns(), "phase": "reseat", "step": step["id"], **compact(r)})
words = coach_words(r)
self._panel.set("note", "note Headset position: " + words if r["state"] != "ok" else "note ")
if self.fit_coach:
self._panel.set("note", "note Headset position: " + words if r["state"] != "ok" else "note ")
self._log(" reseat: %s" % words)
self._save_fit()
@@ -1212,15 +1357,29 @@ class Session:
self._log("take %s %s" % (t["id"], status))
# --- the steps
def _chain_head(self, k):
"""The first step of k's chain: the steps that follow on without a ready screen ("ready":
false) go again with the one they follow (a nudge and the dots done while holding it)."""
plan, h = self.plan, k
while h > 0 and not plan[h]["ready"] and plan[h - 1]["group"] == plan[k]["group"]:
h -= 1
return h
def _steps(self):
"""The plan's steps. Each "ready" one (and one to do again) shows first and waits for
Next, then a recorded countdown; the recording pauses after a step unless the next one
follows straight on. A new group starts a new take. R during a step: it starts again;
at a ready screen: the step before (in the same take) goes again."""
follows straight on. A new group starts a new take, once its first step's Next is
pressed: until then the take before stays open, so Back can reach its last step.
R during the countdown or the step: it starts again, from the start of its chain (a step
that follows on goes again with the one it follows). R at a ready screen (Back): the
step before goes again, with its chain. Either way the range done before is marked with
a "redo" event. Nothing goes back over a reseat: the headset sits differently after it."""
plan = self.plan
done = [] # steps finished in this take: (index, id, from_ns, to_ns)
done = [] # steps finished in the open take: (index, id, from_ns, to_ns)
again = set()
k = 0
self._lingering = False # the open take's steps are done; it ends at the next step's Next
while k < len(plan):
st = plan[k]
where = "%s · step %d of %d" % (self._session_json["title"], k + 1, len(plan))
@@ -1230,23 +1389,31 @@ class Session:
if self._take:
self._end_take("complete")
done = []
self._lingering = False
self._reseat(st)
k += 1
continue
if self._take and self._take["group"] != st["group"]:
self._end_take("complete")
ready = st["ready"] or k in again
if self._take and self._take["group"] != st["group"] and not ready:
self._end_take("complete") # follows on into a new group: no screen to go back from
done = []
self._lingering = bool(self._take and self._take["group"] != st["group"])
started = self._step_t0 = None
try:
if st["ready"] or k in again:
if ready:
self._redo_ok = bool(done)
self._ready(st, can_redo=bool(done))
self._await_next()
if self._take and self._take["group"] != st["group"]:
self._end_take("complete")
done = []
self._lingering = False
self._redo_ok = True
started = self._countdown(st)
else:
self._redo_ok = True
self._show(self._title(st), st["text"])
self._keys(running=True)
self._emit(can_redo=True, prompt=st["text"].replace("|", "\n"))
self._step_t0 = started = self._event("step", id=st["id"], kind=st["kind"])
self._take["json"]["steps"].append(st["id"]) if st["id"] not in self._take["json"]["steps"] else None
@@ -1260,29 +1427,43 @@ class Session:
k += 1
except _Redo:
self._redo_ok = False
if started is not None:
self._event("redo", id=st["id"], **{"from": started, "to": mono_ns()})
self._log(" redo %s" % st["id"])
if started is None:
started = self._step_t0 # R in the countdown: its "ready" event
if started is not None: # this step again, from the start of its chain
h = self._chain_head(k)
while done and done[-1][0] >= h:
started = done.pop()[2]
self._event("redo", id=plan[h]["id"], **{"from": started, "to": mono_ns()})
self._log(" redo %s" % plan[h]["id"])
self._step_end()
again.add(k)
elif done:
k = h
again = {x for x in again if x < k} | {k}
elif done: # Back: the step before, with its chain
k, sid, a, b = done.pop()
h = self._chain_head(k)
while done and done[-1][0] >= h:
k, sid, a, _ = done.pop()
self._event("redo", id=sid, **{"from": a, "to": b})
self._log(" redo %s" % sid)
again.add(k)
self._log(" back to %s" % sid)
self._lingering = False
again = {x for x in again if x < k} | {k}
except _Skip:
group = st["group"]
self._log("skipped the rest of %s" % group)
if self._take:
if self._take and self._take["group"] != group:
self._end_take("complete") # at the first step's ready screen: the take before is done
elif self._take:
self._step_end()
self._end_take("skipped")
done = []
self._lingering = False
while k < len(plan) and plan[k]["group"] == group and plan[k]["kind"] != "reseat":
k += 1
finally:
self._redo_ok = False
if self._take:
self._end_take("complete")
self._lingering = False
def _ready(self, st, can_redo):
self._show(self._title(st), st["text"])
@@ -1295,7 +1476,7 @@ class Session:
countdown=0, big="", seconds_left=round(step_seconds(st), 1))
def _title(self, st):
return {"ipd": "IPD sweep", "dots": "Nine dots"}.get(st["group"], self.round["title"])
return {"dots": "Nine dots"}.get(st["group"], self.round["title"])
def _first_dot(self, st):
"""Where the eyes should be when the step starts: shown during its countdown."""
@@ -1314,6 +1495,9 @@ class Session:
self._start_take(st["group"])
t = self._step_t0 = self._event("ready", id=st["id"], seconds=COUNTDOWN_S)
self._record(True)
self._keys(running=True)
self._panel.set("note", "note ")
self._emit(note="")
first = self._first_dot(st)
if first:
self._panel.cmd("dot %.3f %.3f look" % tuple(first))
@@ -1390,12 +1574,17 @@ class Session:
self._wait(st["seconds"])
return
total, gone = sum(c[1] for c in st["cues"]), 0.0
for word, secs in st["cues"]:
for word, secs, *sound in st["cues"]:
self._panel.set("big", "big " + word)
dist = st["cue_distance"].get(word)
if dist:
self._panel.cmd("distance %.3f" % dist)
self._event("cue", id=st["id"], word=word, **({"distance": dist} if dist else {}))
self._event("cue", id=st["id"], word=word, **({"distance": dist} if dist else {}),
**({"sound": sound[0]} if sound else {}))
if sound:
self._sounds.play(sound[0])
if self.dry_run:
self.print("sound: " + sound[0])
self._emit(big=word)
self._wait(secs, bar=False)
gone += secs
@@ -1468,6 +1657,7 @@ def main():
ap.add_argument("--no-headset-button", action="store_true", help="don't read the headset's button")
ap.add_argument("--button-device", metavar="PATH", help="test: read the button from this device or FIFO")
ap.add_argument("--seed", type=int, help="the shuffles and jumps (default: from the round's id)")
ap.add_argument("--fit-coach", action="store_true", help="the fit coach on (off by default for now: FIT_COACH)")
ap.add_argument("--glasses", choices=("none", "glasses", "inserts", "contacts"), help="the checklist's answer")
a = ap.parse_args()
@@ -1504,8 +1694,8 @@ def main():
print("round %s: %s" % (script, e))
return 2
if a.plan:
plan = build_plan(rnd, a.seed or 0, ipd_sweep=False)
print("%s: %d steps, about %.1f min recorded (plus the fitment, and the IPD sweep on a first round)"
plan = build_plan(rnd, a.seed or 0)
print("%s: %d steps, about %.1f min recorded (plus the fitment)"
% (rnd["title"], len(plan), plan_seconds(plan) / 60))
for s in plan:
extra = len(s["dots"]) if "dots" in s else ""
@@ -1536,7 +1726,8 @@ def main():
s = Session(base, script, bounty=entry, profile=profile, checklist=checklist, dry_run=a.dry_run, speed=a.speed,
on_status=on_status, next_after=a.next_after, cams=a.cams, panel_bin=a.panel, panel_no_vr=a.panel_no_vr,
steamvr_gaze=not a.no_steamvr_gaze, button=not a.no_headset_button and (not a.dry_run or a.button_device),
button_device=a.button_device, seed=a.seed, profile_path=profile_path, eyes_python=a.eyes_python)
button_device=a.button_device, seed=a.seed, profile_path=profile_path, eyes_python=a.eyes_python,
fit_coach=True if a.fit_coach else None)
def on_signal(*_):
print("stopping", flush=True)
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
"""The eye recorder's sound cues: plain beeps for steps done with the eyes shut, where the panel's
words can't be seen, and for moving the headset by hand, where the timing matters. A round script names one as a cue's third field (session.py: "cues":
[[word, seconds, sound]...]); it plays as the cue starts, and prompts.jsonl's "cue" line says which.
blink a short high beep: blink once
close a beep: close your eyes
open the same beep again: open them
go a higher beep: move the headset now (the cue word says how)
done a low, longer one: let go of it
No voice, no sound files: the beeps are made here (a sine with a short fade, so it doesn't click)
into the run folder, and played with PipeWire's pw-play on the default output (the headset's
speakers). A beep starts some tens of ms after its cue line (pw-play's start and its 20 ms
latency); a person's reaction to it takes several times that.
sounds.py play each one, a second apart
sounds.py NAME... play those
"""
import math
import os
import shutil
import struct
import subprocess
import sys
import time
import wave
RATE = 48000
VOLUME = 0.6 # pw-play's stream volume; the beep itself is at half scale
# name: (pitch Hz, seconds)
SOUNDS = {
"blink": (1000.0, 0.09),
"close": (660.0, 0.18),
"open": (660.0, 0.18),
"go": (880.0, 0.14),
"done": (440.0, 0.3),
}
FADE_S = 0.005
def _dir():
d = os.path.join(os.environ.get("XDG_RUNTIME_DIR") or "/run/user/%d" % os.getuid(), "frametop", "eyerec-sounds")
os.makedirs(d, exist_ok=True)
return d
def wav_path(name):
"""The sound's WAV file, made the first time (and again when its pitch or length changed)."""
hz, secs = SOUNDS[name]
path = os.path.join(_dir(), "%s-%d-%d.wav" % (name, hz, secs * 1000))
if not os.path.exists(path):
n, fade = int(RATE * secs), int(RATE * FADE_S)
frames = bytearray()
for i in range(n):
env = min(1.0, i / fade, (n - 1 - i) / fade)
frames += struct.pack("<h", int(0.5 * 32767 * env * math.sin(2 * math.pi * hz * i / RATE)))
with wave.open(path + ".tmp", "wb") as w:
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(RATE)
w.writeframes(bytes(frames))
os.replace(path + ".tmp", path)
return path
class Player:
"""Plays sounds without waiting for them. Off (dry runs, tests): it only says what it would play.
Without pw-play, or when a sound fails, it logs that once and the round goes on silent."""
def __init__(self, on=True, log=None):
self.log = log or (lambda text: None)
self.pw_play = shutil.which("pw-play") if on else None
self.procs = []
self.warned = False
if on and not self.pw_play:
self._warn("no pw-play: the sound cues are off")
if self.pw_play:
for name in SOUNDS: # made now, not while a step runs
wav_path(name)
def _warn(self, text):
if not self.warned:
self.warned = True
self.log(text)
def play(self, name):
if name not in SOUNDS:
raise ValueError("no sound %r" % name)
if not self.pw_play:
return False
self.procs = [p for p in self.procs if p.poll() is None]
env = dict(os.environ, XDG_RUNTIME_DIR=os.environ.get("XDG_RUNTIME_DIR") or "/run/user/%d" % os.getuid())
try:
self.procs.append(subprocess.Popen(
[self.pw_play, "--latency", "20ms", "--volume", str(VOLUME), "--media-role", "Notification",
wav_path(name)], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
env=env))
return True
except OSError as e:
self._warn("can't play the sound cues: %s" % e)
return False
def close(self):
for p in self.procs:
try:
p.wait(timeout=1)
except subprocess.TimeoutExpired:
p.kill()
self.procs = []
def main(argv):
names = argv or list(SOUNDS)
bad = [n for n in names if n not in SOUNDS]
if bad:
sys.exit("no sound %s; there are: %s" % (", ".join(bad), ", ".join(SOUNDS)))
p = Player(log=lambda text: print(text, file=sys.stderr))
for i, name in enumerate(names):
if i:
time.sleep(1.0)
print(name)
p.play(name)
p.close()
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+28 -9
View File
@@ -63,16 +63,35 @@ class HubTest(unittest.TestCase):
self.assertEqual(hub.rhub.uploads(hub.meta_path(self.store, RID)), [])
def test_gate(self):
self.assertFalse(hub.texts_ready(), "the texts are drafts")
with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: ""}):
self.assertFalse(hub.upload_allowed(hub.dataset_id()), "never the real dataset before the review")
with self.assertRaises(hub.HubError) as cm:
hub.upload(self.store, RID)
self.assertEqual(cm.exception.kind, "closed")
with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: "a/test"}):
self.assertTrue(hub.texts_ready(), "the texts are reviewed (2026-10-16)")
with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "", hub.DATASET_ENV: ""}):
self.assertTrue(hub.upload_allowed(hub.dataset_id()))
with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "", hub.DATASET_ENV: "a/test"}):
self.assertFalse(hub.upload_allowed(hub.dataset_id()))
# While a text is a draft: never the real dataset, a rehearsal only with ALLOW_ENV.
with mock.patch.object(hub, "texts_ready", return_value=False):
with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: ""}):
self.assertFalse(hub.upload_allowed(hub.dataset_id()), "never the real dataset before the review")
with self.assertRaises(hub.HubError) as cm:
hub.upload(self.store, RID)
self.assertEqual(cm.exception.kind, "closed")
with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "1", hub.DATASET_ENV: "a/test"}):
self.assertTrue(hub.upload_allowed(hub.dataset_id()))
with mock.patch.dict(os.environ, {hub.ALLOW_ENV: "", hub.DATASET_ENV: "a/test"}):
self.assertFalse(hub.upload_allowed(hub.dataset_id()))
def test_draft_consent_blocks(self):
# An export made under a draft consent text never goes to the real dataset.
prof = os.path.join(self.store.base, "profile.json")
with open(prof) as f:
p = json.load(f)
p["consent"]["version"] = "2026-10-14-draft"
with open(prof, "w") as f:
json.dump(p, f)
shutil.rmtree(self.store.export_dir(RID))
self.store.export(RID, low_priority=False)
with mock.patch.dict(os.environ, {hub.DATASET_ENV: ""}):
with self.assertRaises(hub.HubError) as cm:
hub.upload(self.store, RID, dry_run=True)
self.assertEqual(cm.exception.kind, "draft-consent")
def test_invalid_blocks(self):
with open(os.path.join(self.store.export_dir(RID), "health.json"), "w") as f:
+185 -11
View File
@@ -69,6 +69,39 @@ class RoundTest(unittest.TestCase):
self.bad([{"id": "a", "kind": "dots", "group": "g", "dots": [[0, 0]]}, {"id": "b", "kind": "dots", "dots": [[0, 0]]},
{"id": "c", "kind": "dots", "group": "g", "dots": [[0, 0]]}], "comes back")
self.bad([{"id": "a", "kind": "dots", "group": "dots", "dots": [[0, 0]]}], "opener")
self.bad([{"id": "a", "kind": "prompt", "cues": [["Blink", 1, "honk"]]}], "sound: blink")
self.bad([{"id": "a", "kind": "prompt", "cues": [["Blink", 1, "blink", 2]]}], "cues want")
# The eyes-shut rules (rounds/README.md)
self.bad([{"id": "a", "kind": "prompt", "cues": [["Blink", 1]]}], 'needs the sound "blink"')
self.bad([{"id": "a", "kind": "prompt", "cues": [["Close both eyes", 2], ["Open", 2]]}], 'needs the sound "close"')
self.bad([{"id": "a", "kind": "prompt", "cues": [["Shut your eyes", 2, "close"]]}], 'comes a cue with the sound "open"')
self.bad([{"id": "a", "kind": "prompt", "cues": [["Close your eyes", 2, "close"], ["Open", 2]]}], '"open"')
# Moving the headset by hand
look = ["Look at the dot", 1.5]
self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2]]}], 'needs the sound "go"')
self.bad([{"id": "a", "kind": "prompt", "cues": [["Push up", 2, "go"], ["Let go", 2, "done"]]}],
"the eyes are still first")
self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2, "go"], ["Let go", 1, "done"]]}], "2 s or more")
self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2, "go"]]}], 'a "done" cue')
self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Push up", 2, "go"]]},
{"id": "b", "kind": "prompt", "cues": [["Let go", 2, "done"]]}], 'a "done" cue')
self.bad([{"id": "a", "kind": "prompt", "cues": [look, ["Tilt: left side up", 2, "go"]]},
{"id": "b", "kind": "dots", "ready": False, "group": "a", "dots": [[0, 0]], "text": "Look"},
{"id": "c", "kind": "prompt", "ready": False, "group": "a", "text": "Keep holding",
"cues": [["Let go", 2, "done"]]}], 'starts "Keep holding"')
session.load_round(write_round(self.tmp, [
{"id": "a", "kind": "prompt", "cues": [look, ["Slide to your left", 2, "go"]]},
{"id": "b", "kind": "dots", "ready": False, "group": "a", "dots": [[0, 0]], "text": "Keep holding it"},
{"id": "c", "kind": "prompt", "ready": False, "group": "a", "text": "Keep holding",
"cues": [["Let go, put it back", 2, "done"]]}]))
for word in ("Close your left eye", "Half closed", "Squint", "Open"):
session.load_round(write_round(self.tmp, [{"id": "a", "kind": "prompt", "cues": [[word, 1]]}]))
def test_sound_cues(self):
r = session.load_round(write_round(self.tmp, [{"id": "a", "kind": "prompt",
"cues": [["Look", 1], ["Blink", 1.5, "blink"]]}]))
self.assertEqual(r["steps"][0]["cues"], [["Look", 1.0], ["Blink", 1.5, "blink"]])
self.assertEqual(session.step_seconds(r["steps"][0]), 2.5)
def test_grid_and_defaults(self):
r = session.load_round(write_round(self.tmp, [{"id": "a", "kind": "dots", "grid": [3, 2, 10, 5], "ready": False},
@@ -81,18 +114,17 @@ class RoundTest(unittest.TestCase):
def test_plan(self):
rnd = session.load_round(os.path.join(session.ROUNDS_DIR, "jumps.json"))
a = session.build_plan(rnd, 7, ipd_sweep=True)
self.assertEqual([s["group"] for s in a[:3]], ["ipd", "dots", "jumps"])
nine = a[1]["dots"]
a = session.build_plan(rnd, 7)
self.assertEqual([s["group"] for s in a[:2]], ["dots", "jumps"])
nine = a[0]["dots"]
self.assertEqual((nine[0], nine[-1], len(nine)), ([0.0, 0.0], [0.0, 0.0], 10))
self.assertEqual(sorted(map(tuple, nine[1:-1])), sorted(map(tuple, (d for d in session.grid(3, 3, 20, 20) if d != [0, 0]))))
jumps = a[2]
jumps = a[1]
self.assertEqual(len(jumps["dots"]), 40)
self.assertTrue(all(0.5 <= t <= 1.5 for t in jumps["times"]))
self.assertTrue(all(not session.off_panel(*d) for d in jumps["dots"]))
self.assertEqual(a, session.build_plan(rnd, 7, ipd_sweep=True), "the same seed, the same round")
self.assertNotEqual(a[1]["dots"], session.build_plan(rnd, 8, ipd_sweep=True)[1]["dots"])
self.assertEqual(session.build_plan(rnd, 7, ipd_sweep=False)[0]["group"], "dots")
self.assertEqual(a, session.build_plan(rnd, 7), "the same seed, the same round")
self.assertNotEqual(a[0]["dots"], session.build_plan(rnd, 8)[0]["dots"])
self.assertGreater(session.plan_seconds(a), 60)
@@ -171,6 +203,7 @@ class DryRunTest(unittest.TestCase):
with open(path, "w") as f:
json.dump(profile or {}, f)
states = []
kw.setdefault("fit_coach", True) # the coach's own tests; test_fitment_coach_off has it off
s = session.Session(self.tmp, script, profile=profile, dry_run=True, speed=kw.pop("speed", 40),
next_after=kw.pop("next_after", 0.25), profile_path=path, button=False,
on_status=lambda st: states.append(dict(st)), **kw)
@@ -198,8 +231,11 @@ class DryRunTest(unittest.TestCase):
s, states, profile = self.run_round("depth")
self.assertEqual(s.state, "done")
sj, fit = self.files(s)
self.assertEqual(sj["takes"], ["01-ipd", "02-dots", "03-near", "04-mid", "05-far", "06-near-far"])
self.assertEqual((sj["status"], sj["fit"], sj["ipd_sweep"], sj["ipd_start"]), ("done", "good", True, 0.063))
# No IPD step: the setting is read and shown at the fitment.
self.assertEqual(sj["takes"], ["01-dots", "02-near", "03-mid", "04-far", "05-near-far"])
self.assertEqual((sj["status"], sj["fit"], sj["ipd_start"]), ("done", "good", 0.063))
self.assertNotIn("ipd_sweep", sj)
self.assertIn(session.IPD_TEXT.format(mm=63.0), "\n".join(self.out))
self.assertEqual(fit["fitment"]["result"], "good")
self.assertIsNotNone(fit["baseline"])
self.assertEqual(profile["bounties"]["depth"]["rounds"], 1)
@@ -207,16 +243,29 @@ class DryRunTest(unittest.TestCase):
for take in sj["takes"]:
with open(os.path.join(s.session_dir, "takes", take, "take.json")) as f:
self.assertEqual(json.load(f)["status"], "complete", take)
events = [e["event"] for e in self.prompts(s, "03-near")]
events = [e["event"] for e in self.prompts(s, "02-near")]
self.assertEqual(events[:4], ["take", "ready", "rec", "step"])
self.assertEqual(events[-3:], ["wait", "rec", "end"])
self.assertIn("ft_eyerecpanel: distance 0.500", self.out)
self.assertIn("ft_eyerecpanel: distance 5.000", self.out)
# The IPD is the same next time: no sweep.
s2, _, profile = self.run_round("depth", profile=profile)
self.assertEqual(self.files(s2)[0]["takes"][0], "01-dots")
self.assertEqual(profile["bounties"]["depth"]["rounds"], 2)
def test_blink_beeps(self):
# The blinks round's beeps: each in prompts.jsonl's cue line, played as the cue starts.
s, _, _ = self.run_round("blinks", profile={"eyes_ipd": 0.063})
self.assertEqual(s.state, "done")
played = [line[len("sound: "):] for line in self.out if line.startswith("sound: ")]
self.assertEqual(played, ["blink"] * 8 + ["close", "open", "close", "open"])
sj, _ = self.files(s)
[take] = [t for t in sj["takes"] if t.endswith("blink-cue")]
cues = [e for e in self.prompts(s, take) if e["event"] == "cue"]
self.assertEqual([(e["word"], e.get("sound")) for e in cues], [("Blink", "blink")] * 8)
[take] = [t for t in sj["takes"] if t.endswith("closed")]
cues = [e.get("sound") for e in self.prompts(s, take) if e["event"] == "cue"]
self.assertEqual(cues, [None, "close", "open", "close", "open"])
def test_fitment_warns_then_skips(self):
# Next before the fit has been good for FIT_GOOD_S: a warning, then the round goes on.
s, states, _ = self.run_round("reading", next_after=0.01, speed=20)
@@ -225,6 +274,21 @@ class DryRunTest(unittest.TestCase):
self.assertTrue(any(st["note"] == session.FIT_SKIP_NOTE for st in states))
self.assertIsNone(fit["baseline"])
def test_fitment_coach_off(self):
# The default for now: no coaching or warning, the first Next goes on, and the coach's
# readings are still kept; no baseline, so no slip checks.
self.assertFalse(session.FIT_COACH)
s, states, _ = self.run_round("reading", next_after=0.6, speed=20, fit_coach=False)
sj, fit = self.files(s)
self.assertEqual(s.state, "done")
self.assertEqual((sj["fit"], fit["fitment"]["result"]), ("off", "off"))
self.assertTrue(fit["readings"])
self.assertIsNone(fit["baseline"])
self.assertFalse(fit["slips"])
self.assertFalse(any(st["note"] for st in states))
self.assertFalse(any(st["fit_words"] or st["fit_card"] for st in states))
self.assertTrue(all(st["fit_coach"] is False for st in states))
def test_skip_redo_stop(self):
def steer(s, states):
def wait_for(cond):
@@ -254,6 +318,116 @@ class DryRunTest(unittest.TestCase):
self.assertLess(redo[0]["from"], redo[0]["to"])
self.assertNotIn("bounties", profile, "a stopped round isn't counted")
# A round of two takes: a nudge with dots done while holding it (a chain), then one prompt.
TWO = [{"id": "nudge", "kind": "prompt", "group": "one", "dot": [0, 0], "seconds": 1, "text": "Push"},
{"id": "hold", "kind": "dots", "group": "one", "dots": [[0, 0], [5, 0]], "hold": 0.5, "ready": False,
"text": "Keep holding"},
{"id": "other", "kind": "prompt", "group": "two", "dot": [0, 0], "seconds": 1, "text": "Other"}]
def run_two(self, steer, **kw):
write_round(self.tmp, self.TWO)
return self.run_round("t", steer=steer, speed=10, next_after=0.6, rounds_dir=self.tmp, fit_coach=False,
profile={"eyes_ipd": 0.063}, **kw)
@staticmethod
def at(s, step, state):
end = time.monotonic() + 30
while time.monotonic() < end and not (s._status["step"] == step and s._status["state"] == state):
time.sleep(0.005)
return s._status["step"] == step and s._status["state"] == state
def take_status(self, s, take):
with open(os.path.join(s.session_dir, "takes", take, "take.json")) as f:
return json.load(f)["status"]
def redos(self, s, take):
return [e["id"] for e in self.prompts(s, take) if e["event"] == "redo"]
def test_redo_in_the_countdown(self):
# R in the countdown starts this step again (it used to go back to the one before).
seen = {}
def steer(s, states):
self.assertTrue(self.at(s, "other", "countdown"))
s.redo()
seen["ready"] = self.at(s, "other", "ready")
seen["recording"] = s._recording
s, _, _ = self.run_two(steer)
self.assertEqual(s.state, "done")
self.assertTrue(seen["ready"])
self.assertFalse(seen["recording"], "the ready screen isn't recorded")
takes = self.files(s)[0]["takes"]
self.assertEqual(self.redos(s, takes[-1]), ["other"])
self.assertEqual(self.redos(s, takes[-2]), [])
def test_back_into_the_take_before(self):
# Back at a new take's first ready screen: the take before stays open until its Next,
# so its last step (with the nudge it follows on from) goes again in the same take.
seen = {}
def steer(s, states):
self.assertTrue(self.at(s, "other", "ready"))
seen["can_redo"] = s._status["can_redo"]
s.redo()
seen["back"] = self.at(s, "nudge", "ready")
s, _, _ = self.run_two(steer)
self.assertEqual(s.state, "done")
self.assertTrue(seen["can_redo"])
self.assertTrue(seen["back"], "back to the chain's head, not to hold")
takes = self.files(s)[0]["takes"]
self.assertEqual(len(takes), 3, takes) # the opener's dots, one, two: no extra take
one = self.prompts(s, takes[1])
self.assertEqual(self.redos(s, takes[1]), ["nudge"])
self.assertEqual([e["id"] for e in one if e["event"] == "step"], ["nudge", "hold", "nudge", "hold"])
self.assertEqual(self.take_status(s, takes[1]), "complete")
def test_redo_a_step_that_follows_on(self):
# R while holding: the nudge goes again with it.
def steer(s, states):
self.assertTrue(self.at(s, "hold", "running"))
s.redo()
self.assertTrue(self.at(s, "nudge", "ready"))
s, _, _ = self.run_two(steer)
self.assertEqual(s.state, "done")
takes = self.files(s)[0]["takes"]
steps = [e["id"] for e in self.prompts(s, takes[1]) if e["event"] == "step"]
self.assertEqual(self.redos(s, takes[1]), ["nudge"])
self.assertEqual(steps, ["nudge", "hold", "nudge", "hold"])
def test_stop_or_skip_at_the_next_ready_screen(self):
# The take before is finished, not stopped or skipped, when the next take's ready screen
# is where it ends.
def stop(s, states):
self.assertTrue(self.at(s, "other", "ready"))
s.stop(wait=0)
s, _, _ = self.run_two(stop)
self.assertEqual(s.state, "stopped")
takes = self.files(s)[0]["takes"]
self.assertEqual(len(takes), 2)
self.assertEqual(self.take_status(s, takes[1]), "complete")
def skip(s, states):
self.assertTrue(self.at(s, "other", "ready"))
s.skip()
s, _, _ = self.run_two(skip)
self.assertEqual(s.state, "done")
takes = self.files(s)[0]["takes"]
self.assertEqual(len(takes), 2, "two was skipped before it started")
self.assertEqual(self.take_status(s, takes[1]), "complete")
def test_nothing_to_go_back_to(self):
# After a reseat Back is off: the headset sits differently now.
rnd = [{"id": "a", "kind": "prompt", "dot": [0, 0], "seconds": 1, "text": "A"},
{"id": "off", "kind": "reseat", "text": "off and on"},
{"id": "b", "kind": "prompt", "dot": [0, 0], "seconds": 1, "text": "B"}]
write_round(self.tmp, rnd)
seen = {}
def steer(s, states):
self.assertTrue(self.at(s, "b", "ready"))
seen["can_redo"] = s._status["can_redo"]
s, states, _ = self.run_round("t", steer=steer, speed=10, next_after=0.4, rounds_dir=self.tmp,
fit_coach=False, profile={"eyes_ipd": 0.063})
self.assertEqual(s.state, "done")
self.assertFalse(seen["can_redo"], "nothing goes back over a reseat")
def test_pause_stops_the_recording(self):
def steer(s, states):
end = time.monotonic() + 30
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""sounds.py: the beeps' WAV files, and a player that never stops a round."""
import os
import shutil
import sys
import tempfile
import unittest
import wave
from unittest import mock
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, ".."))
import sounds # noqa: E402
class SoundsTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.mkdtemp(prefix="eyerec-sounds-")
self.env = mock.patch.dict(os.environ, {"XDG_RUNTIME_DIR": self.tmp})
self.env.start()
def tearDown(self):
self.env.stop()
shutil.rmtree(self.tmp, ignore_errors=True)
def test_wavs(self):
self.assertEqual(set(sounds.SOUNDS), {"blink", "close", "open", "go", "done"})
self.assertEqual(len({sounds.SOUNDS[n] for n in ("blink", "close", "go", "done")}), 4, "told apart by ear")
self.assertEqual(sounds.SOUNDS["close"], sounds.SOUNDS["open"], "open is the same beep again")
for name, (hz, secs) in sounds.SOUNDS.items():
path = sounds.wav_path(name)
self.assertTrue(path.startswith(self.tmp))
with wave.open(path) as w:
self.assertEqual((w.getnchannels(), w.getsampwidth(), w.getframerate()), (1, 2, sounds.RATE))
self.assertEqual(w.getnframes(), int(sounds.RATE * secs))
frames = w.readframes(w.getnframes())
first, last = int.from_bytes(frames[:2], "little", signed=True), int.from_bytes(frames[-2:], "little", signed=True)
self.assertEqual((first, last), (0, 0), "faded in and out: no click")
def test_player_off_and_missing(self):
logged = []
self.assertFalse(sounds.Player(on=False, log=logged.append).play("blink"))
self.assertEqual(logged, [])
with mock.patch.object(sounds.shutil, "which", return_value=None):
p = sounds.Player(log=logged.append)
self.assertFalse(p.play("close"))
self.assertEqual(logged, ["no pw-play: the sound cues are off"])
with self.assertRaises(ValueError):
p.play("honk")
if __name__ == "__main__":
unittest.main()
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env python3
"""Offline pupil detection: shared preprocessing preserves both closing attempts.
Synthetic images exercise normal pupils, glints, the small-kernel retry, lost eyes,
and windowed searches. No camera, calibration, or SteamVR connection is needed.
"""
import sys
import unittest
from pathlib import Path
from unittest.mock import patch
import cv2
import numpy as np
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tracker"))
import eyes_pupil as pupil
def frame(retry=False):
img = np.full((400, 512), 100, np.uint8)
centre = (44, 200) if retry else (256, 200)
if retry:
img[:, :11] = 0 # large closing joins the pupil to this edge region
cv2.ellipse(img, centre, (30, 24), 0, 0, 360, 0, -1)
return img
def separate_attempts(img, ox, oy):
"""The old preprocessing order: each attempt gets a newly calculated mask."""
for close in (pupil.CLOSE, pupil.CLOSE_TIGHT):
dark = (cv2.GaussianBlur(img, (5, 5), 0) < pupil.DARK).astype(np.uint8)
p = pupil._find(img, dark, ox, oy, close)
if p is not None:
return p
return None
class PupilTest(unittest.TestCase):
def test_first_attempt(self):
img = frame()
with patch.object(pupil.cv2, "GaussianBlur", wraps=cv2.GaussianBlur) as blur, \
patch.object(pupil, "_find", wraps=pupil._find) as find:
p = pupil.find_pupil(img)
self.assertEqual((blur.call_count, find.call_count), (1, 1))
self.assertAlmostEqual(p["x"], 256, delta=0.1)
self.assertAlmostEqual(p["y"], 200, delta=0.1)
def test_retry_preserves_mask_and_kernels(self):
img = frame(retry=True)
dark = (cv2.GaussianBlur(img, (5, 5), 0) < pupil.DARK).astype(np.uint8)
self.assertIsNone(pupil._find(img, dark, 0, 0))
expected = pupil._find(img, dark, 0, 0, pupil.CLOSE_TIGHT)
self.assertIsNotNone(expected)
kernels = {k: v.copy() for k, v in pupil._KERNELS.items()}
with patch.object(pupil.cv2, "GaussianBlur", wraps=cv2.GaussianBlur) as blur, \
patch.object(pupil, "_find", wraps=pupil._find) as find:
self.assertEqual(pupil._find_either(img, 0, 0), expected)
self.assertEqual((blur.call_count, find.call_count), (1, 2))
first, second = (call.args[1] for call in find.call_args_list)
self.assertIs(first, second)
np.testing.assert_array_equal(first, dark)
for k, v in kernels.items():
np.testing.assert_array_equal(pupil._KERNELS[k], v)
def test_exact_results_for_full_and_cropped_images(self):
rng = np.random.default_rng(512400)
images = [np.zeros((400, 512), np.uint8), np.full((400, 512), 255, np.uint8),
frame(), frame(retry=True)]
glints = frame()
for y in (194, 207):
cv2.circle(glints, (256, y), 2, 255, -1)
images.append(glints)
for _ in range(12):
img = rng.integers(35, 180, (400, 512), np.uint8)
cv2.ellipse(img, (int(rng.integers(30, 480)), int(rng.integers(25, 375))),
(int(rng.integers(10, 45)), int(rng.integers(8, 25))),
int(rng.integers(0, 180)), 0, 360, 0, -1)
images.append(img)
for i, img in enumerate(images):
for x, y in ((0, 0), (11, 15), (200, 120)):
roi = img if (x, y) == (0, 0) else img[y:y + 180, x:x + 200]
before = roi.copy()
with self.subTest(image=i, offset=(x, y)):
self.assertEqual(pupil._find_either(roi, x, y), separate_attempts(roi, x, y))
np.testing.assert_array_equal(roi, before)
def test_window_hint_and_full_frame_fallback(self):
img = frame()
expected = pupil.find_pupil(img)
self.assertEqual(pupil.find_pupil(img, expected), expected)
self.assertEqual(pupil.find_pupil(img, {"x": 50, "y": 50, "a": 5}), expected)
self.assertEqual(pupil.find_pupil(np.zeros_like(img), expected), None)
if __name__ == "__main__":
unittest.main()
+7
View File
@@ -13,6 +13,7 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
"$root/scripts/sync.sh" >/dev/null
exec "$root/scripts/frame.sh" -C gaze/tracker 'set -e; mkdir -p build
gcc -std=gnu11 -O2 -Wall -Wextra -pthread -o build/ft-eyegrab ft-eyegrab.c
strip --strip-unneeded build/ft-eyegrab
max=$(objdump -T build/ft-eyegrab | grep -oE "GLIBC_[0-9.]+" | sort -uV | tail -1)
echo "built build/ft-eyegrab, newest glibc symbol: $max"
[ "$(printf "%s\n" "$max" GLIBC_2.39 | sort -V | tail -1)" = GLIBC_2.39 ] || { echo "needs newer glibc than the host has" >&2; exit 1; }
@@ -21,6 +22,12 @@ if [ "${FRAME_IN_BOX:-0}" = 1 ] && [ -x /opt/frametop/venv/bin/python ]; then
python3 -m venv --without-pip build/venv
/opt/frametop/venv/bin/python -c "import sysconfig; print(sysconfig.get_path(\"purelib\"))" \
>"$(build/venv/bin/python -c "import sysconfig; print(sysconfig.get_path(\"purelib\"))")/frametop-image.pth"
# The eye net (eyes_net.py) needs ncnn, which the image does not have yet: add it here. It needs
# the network; without it ft-eyes uses the pupil finder.
if ! build/venv/bin/python -c "import ncnn" 2>/dev/null; then
{ build/venv/bin/python -m ensurepip >/dev/null && build/venv/bin/python -m pip install -q --disable-pip-version-check \
--no-deps "$(grep -E "^ncnn==" requirements.txt)"; } || echo "build/venv: no ncnn (the eye net needs it)" >&2
fi
elif ! cmp -s requirements.txt build/venv/requirements.done; then
rm -rf build/venv
python3 -m venv build/venv
+56
View File
@@ -0,0 +1,56 @@
"""The eye net: one eye camera's frame in, that eye's raw gaze out (frame-eyes' train/gaze).
A model folder holds model.json and the ncnn files it names:
{"name": "eyes-v0", "variant": "s100", "input": [100, 128], "gaze_scale": 30, "out": 6,
"param": "model.ncnn.param", "bin": "model.ncnn.bin", "in": "in0", "outname": "out0", ...}
The net's output holds both cameras' heads (OUT numbers each, camera 0's first): gaze yaw and
pitch / gaze_scale, pupil x / 512 and y / 400, eye open (logit), pupil visible (logit).
The gaze is raw: each person's own offset is left to the calibration (eyes_model.Calibration,
with this gaze where the pupil centre would be). So a calibration belongs to one model: ft-eyes
keeps each model's in its own folder.
ncnn fp16 on one thread: 1.4 ms a frame for s100 on an A720 core (frame-eyes notes/e0-runtime.md).
"""
import json
import math
import os
import numpy as np
W, H = 512, 400
class Net:
def __init__(self, folder):
import cv2
import ncnn
self.cv2 = cv2
self.folder = folder
with open(os.path.join(folder, "model.json")) as f:
self.info = json.load(f)
self.name = str(self.info["name"])
self.in_h, self.in_w = self.info["input"]
self.scale = float(self.info.get("gaze_scale", 30.0))
self.out = int(self.info.get("out", 6))
self.in_name, self.out_name = self.info.get("in", "in0"), self.info.get("outname", "out0")
net = ncnn.Net()
net.opt.num_threads = 1
net.opt.use_vulkan_compute = False
net.opt.use_fp16_storage = net.opt.use_fp16_arithmetic = net.opt.use_fp16_packed = True
for key, load in (("param", net.load_param), ("bin", net.load_model)):
if load(os.path.join(folder, self.info[key])) != 0:
raise RuntimeError(f"{folder}: can't load {self.info[key]}")
self.ncnn, self.net = ncnn, net
def __call__(self, img, cam):
"""img: one camera's 400x512 uint8 frame -> dict(gaze=(yaw, pitch), x, y, open, visible),
open and visible as probabilities."""
x = self.cv2.resize(img, (self.in_w, self.in_h), interpolation=self.cv2.INTER_AREA)
x = x.astype(np.float32)[None] * (1.0 / 255.0)
ex = self.net.create_extractor()
ex.input(self.in_name, self.ncnn.Mat(x))
o = np.array(ex.extract(self.out_name)[1]).reshape(-1)[cam * self.out:(cam + 1) * self.out]
sig = lambda v: 1.0 / (1.0 + math.exp(-float(v))) # noqa: E731
return {"gaze": (float(o[0]) * self.scale, float(o[1]) * self.scale), "x": float(o[2]) * W,
"y": float(o[3]) * H, "open": sig(o[4]), "visible": sig(o[5])}
+9 -8
View File
@@ -22,6 +22,7 @@ MAX_ASPECT = 3.0 # long / short axis; the steep camera sees a squashed pupil
GLINT = 200 # glints are near-saturated spots on or by the pupil
CLOSE = 7 # px: closes the glints' holes in the pupil (the right eye's need this much)
CLOSE_TIGHT = 3 # px: the retry, keeps a pupil near the dark background apart from it
_KERNELS = {size: np.ones((size, size), np.uint8) for size in {3, CLOSE, CLOSE_TIGHT}}
NEAR = 70 # the windowed search: this many pixels, or 3 pupil radii, around a hint
@@ -46,18 +47,18 @@ def find_pupil(frame, near=None):
def _find_either(img, ox, oy):
p = _find(img, ox, oy)
return p if p is not None else _find(img, ox, oy, CLOSE_TIGHT)
# Both closing sizes use the same input mask; morphologyEx returns a new one.
dark = (cv2.GaussianBlur(img, (5, 5), 0) < DARK).astype(np.uint8)
p = _find(img, dark, ox, oy)
return p if p is not None else _find(img, dark, ox, oy, CLOSE_TIGHT)
def _find(img, ox, oy, close=CLOSE):
"""The pupil in `img` (a window at ox, oy of the frame), with frame coordinates. A dark
def _find(img, dark, ox, oy, close=CLOSE):
"""The pupil in `img` (a window at ox, oy), using its prepared `dark` mask. A dark
region touching the window's edge doesn't count: it's background, rim, or cut off."""
f = cv2.GaussianBlur(img, (5, 5), 0)
dark = (f < DARK).astype(np.uint8)
# Glints punch bright holes in the pupil; close them so the blob stays whole.
dark = cv2.morphologyEx(dark, cv2.MORPH_CLOSE, np.ones((close, close), np.uint8))
dark = cv2.morphologyEx(dark, cv2.MORPH_OPEN, np.ones((3, 3), np.uint8))
dark = cv2.morphologyEx(dark, cv2.MORPH_CLOSE, _KERNELS[close])
dark = cv2.morphologyEx(dark, cv2.MORPH_OPEN, _KERNELS[3])
n, lab, stats, _ = cv2.connectedComponentsWithStats(dark, connectivity=8)
h, w = img.shape
best = None
+94 -8
View File
@@ -38,10 +38,23 @@ Control socket: abstract datagram "@ft_eyes"; each command gets one reply line.
append every frame looked at to PATH as JSON lines (the eye
recorder's own.jsonl): {"t", "cam", "x", "y", "a", "b", "angle",
"glints": [[x, y], ...], "gaze": [yaw, pitch] once calibrated},
or {"t", "cam", "lost": true}. "finds stop" replies "ok ROWS"
or {"t", "cam", "lost": true}; with the net also "net": {"gaze",
"x", "y", "open", "visible"}. "finds stop" replies "ok ROWS"
The eye net (eyes_net.py) instead of the pupil finder: EYES_MODEL in ~/.config/frametop.conf (or
FT_EYES_MODEL), picked on Input Settings' Gaze page: latest, a version's name (gaze/tracker/models.json,
downloaded into gaze/tracker/models/NAME by models/catalog.py), or a model folder from frame-eyes'
training (anything with a "/"); classic or unset is the pupil finder. Each eye's raw gaze from the net
then stands where the pupil centre would: the calibration, the shifts clicks teach and the dots'
spread are in its degrees, and there are no glints (no slip estimate or jump notice). Each model
has its own state folder, net-NAME/ (calibration.json, state.json, clicks.jsonl), so switching
asks for a calibration once per model and the classic one is kept. Read at start: the gaze
service restarts ft-eyes when EYES_MODEL changes. If the model can't load, ft-eyes says
so and uses the pupil finder. While the fit coach asks ("fit") or finds are logged, the pupil
finder runs too, for them.
Environment, for replays (lab/ft-eyes-e2e): FT_EYES_CAMS, FT_EYES_GAZE, FT_EYES_STATE (the
state folder), FT_EYES_SOCKET (the socket's name). With FT_EYES_CAMS set, the want file isn't
state folder), FT_EYES_SOCKET (the socket's name), FT_EYES_MODEL. With FT_EYES_CAMS set, the want file isn't
touched.
/dev/shm/frametop-eyes-gaze, 128 bytes, little-endian (mirrored in ft-gaze.cpp):
@@ -81,11 +94,40 @@ import eyes_model # noqa: E402
import eyes_pupil # noqa: E402
import fitcoach # noqa: E402
def load_net():
"""The eye net EYES_MODEL picks (FT_EYES_MODEL wins), or None (the pupil finder). models/catalog.py
resolves it: classic (or unset: the catalog's default), latest (the newest version, once downloaded),
a downloaded version's name, or a folder of one's own (anything with a "/")."""
try:
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "models"))
import catalog
r = catalog.resolve("eyes")
except Exception as e: # noqa: BLE001 (a broken catalog mustn't stop the tracker)
print(f"ft-eyes: the model catalog didn't load ({type(e).__name__}: {e}); using the pupil finder",
file=sys.stderr, flush=True)
return None
if not r["folder"]:
if r["setting"] not in ("classic", "steamvr"):
print(f"ft-eyes: EYES_MODEL={r['setting']}: {r['why']}; using the pupil finder", file=sys.stderr, flush=True)
return None
try:
import eyes_net
return eyes_net.Net(r["folder"])
except Exception as e: # noqa: BLE001 (any failure: say so and track with the pupil finder)
print(f"ft-eyes: the eye net in {r['folder']} didn't load ({type(e).__name__}: {e}); using the pupil finder",
file=sys.stderr, flush=True)
return None
NET = load_net()
CAMS = os.environ.get("FT_EYES_CAMS", "/dev/shm/frametop-eyes-cams") # overrides for replays
OUT = os.environ.get("FT_EYES_GAZE", "/dev/shm/frametop-eyes-gaze")
WANT = None if "FT_EYES_CAMS" in os.environ else Path("/dev/shm/frametop-eyes-want")
OUT_SIZE = 128
CALIBRATION = Path(os.environ.get("FT_EYES_STATE", Path.home() / ".local/state/frametop/gaze/eyes")) / "calibration.json"
STATE_BASE = Path(os.environ.get("FT_EYES_STATE", Path.home() / ".local/state/frametop/gaze/eyes"))
FEATURE = "net:" + NET.name if NET else "pupil" # what the calibration maps: each model has its own
CALIBRATION = (STATE_BASE / ("net-" + NET.name) if NET else STATE_BASE) / "calibration.json"
W, H = 512, 400
FRESH = 0.03 # an eye's reading counts toward the output for this long (s)
LOST_EVERY = 3 # while an eye is lost, search the whole frame only every 3rd frame
@@ -98,7 +140,8 @@ HISTORY = 12.0 # seconds of pupil positions kept per eye, for dots and cl
GAP = 3.0 # s without frames: the headset was off, and may sit differently now
CLICK_BEFORE = 0.3 # a click's frames: the 300 ms before it (like the probe's fixation)
CALIB_MIN = 15 # frames an eye needs in a calibration dot's window
CALIB_SPREAD = 4.0 # px: more than this and the eye moved during the dot
CALIB_SPREAD = 2.5 if NET else 4.0 # px (the net: degrees): more than this and the eye moved during the dot
CLASSIC_FOR = 3.0 # s: with the net, the pupil finder runs this long after the fit coach asks
# Waiting for frames. They come only as a counter in shared memory, so there's nothing to block
# on: sleep until a camera's next frame is due, then look every POLL. ft-eyegrab passes each one
# on within a ms or two of when its camera starts the one after, so they arrive 11.1 ms apart,
@@ -187,6 +230,7 @@ class Eye:
self.obs = None # the fit coach's window (fitcoach.Observations), shared by both eyes
self.fed = None # (frame time, CLOCK_MONOTONIC_RAW then) of the newest frame fed
self.finds = None # the finds log (Tracker.finds), shared by both eyes
self.coach = [0.0] # with the net: until when the fit coach wants pupils (monotonic), shared
def use(self, cal, shift=None):
self.cal = cal if cal is not None and cal.has("pupil", self.eye) else None
@@ -199,6 +243,8 @@ class Eye:
if self.last_t is not None and t - self.last_t > GAP:
self.shift.reseat()
self.last_t = t
if NET:
return self.feed_net(t, img)
if self.last is None:
self.lost += 1
if self.lost % LOST_EVERY:
@@ -227,6 +273,31 @@ class Eye:
self.finds.write(self, t, p)
self.work += time.perf_counter() - t0
def feed_net(self, t, img):
"""The net's raw gaze is the feature: history, calibration and shifts as for a pupil."""
t0 = time.perf_counter()
r = NET(img, self.eye)
classic = self.finds is not None or time.monotonic() < self.coach[0]
p = None
if classic:
p = eyes_pupil.find_pupil(img, self.last)
self.last = p
if self.obs is not None:
self.obs.add(self.eye, t, p)
self.fed = (t, time.clock_gettime(time.CLOCK_MONOTONIC_RAW))
if r["visible"] >= 0.5:
self.found += 1
fx, fy = r["gaze"]
self.history.append((t, fx, fy, math.nan, math.nan))
while self.history and self.history[0][0] < t - HISTORY:
self.history.popleft()
if self.cal:
g = self.cal.gaze(self.eye, fx, fy, self.shift.value)
self.gaze = (t, float(g[0]), float(g[1]), r["x"], r["y"])
if self.finds is not None:
self.finds.write(self, t, p, r)
self.work += time.perf_counter() - t0
def window(self, t0, t1):
"""Median pupil and glint midpoint over [t0, t1], the frame count, and the spread."""
rows = np.array([r for r in self.history if t0 <= r[0] <= t1]).reshape(-1, 5)
@@ -246,13 +317,18 @@ class Finds:
self.f = open(path, "a", buffering=1 << 16)
self.rows = 0
def write(self, eye, t, p):
def write(self, eye, t, p, net=None):
if p is None:
row = {"t": round(t, 6), "cam": eye.eye, "lost": True}
else:
row = {"t": round(t, 6), "cam": eye.eye, "x": round(float(p["x"]), 2), "y": round(float(p["y"]), 2),
"a": round(float(p["a"]), 2), "b": round(float(p["b"]), 2), "angle": round(float(p["angle"]), 1),
"glints": [[round(float(x), 1), round(float(y), 1)] for x, y in p.get("glints") or []]}
if eye.gaze and eye.gaze[0] == t and not NET:
row["gaze"] = [round(eye.gaze[1], 3), round(eye.gaze[2], 3)]
if net is not None:
row["net"] = {"gaze": [round(v, 3) for v in net["gaze"]], "x": round(net["x"], 1), "y": round(net["y"], 1),
"open": round(net["open"], 3), "visible": round(net["visible"], 3)}
if eye.gaze and eye.gaze[0] == t:
row["gaze"] = [round(eye.gaze[1], 3), round(eye.gaze[2], 3)]
self.f.write(json.dumps(row) + "\n")
@@ -270,10 +346,16 @@ class Tracker:
self.dots = [] # calibration dots so far, in ft-eyes-score's click form
self.calibrating = False
self.fit = fitcoach.Observations()
coach = [0.0]
for e in self.eyes:
e.obs = self.fit
e.obs, e.coach = self.fit, coach
if CALIBRATION.exists():
self.cal = eyes_model.Calibration.load(CALIBRATION)
if self.cal.info.get("feature", "pupil") != FEATURE:
print(f"ft-eyes: {CALIBRATION} is for {self.cal.info.get('feature', 'pupil')}, not {FEATURE}:"
" calibrate again", file=sys.stderr, flush=True)
self.cal = None
if self.cal:
if not self.cal.spread:
self.cal.spread = spread_from_dots(self.cal)
shifts = {}
@@ -319,6 +401,7 @@ class Tracker:
return "ok"
return "fail finds start PATH | finds stop"
if w[0] == "fit":
self.eyes[0].coach[0] = time.monotonic() + CLASSIC_FOR
# In frame time, which a replay's frames keep from the recording: the newest frame's,
# plus how long ago it came (frames stopping, the headset off, empty the window).
fed = [e.fed for e in self.eyes if e.fed]
@@ -345,7 +428,8 @@ class Tracker:
if len(self.dots) < eyes_model.MIN_CLICKS:
return f"fail only {len(self.dots)} dots (need {eyes_model.MIN_CLICKS})"
cal = eyes_model.Calibration.fit(self.dots, {"made": time.strftime("%Y-%m-%d %H:%M:%S"),
"dots": len(self.dots), "from": "probe calibration"})
"dots": len(self.dots), "from": "probe calibration",
"feature": FEATURE})
if not all(cal.has(n, c) for n in ("pupil", "where") for c in (0, 1)):
return "fail not enough dots with both eyes"
errs = [float(np.hypot(*(np.mean([cal.gaze(c, *k["eye"][c]["pupil"]) for c in (0, 1)], axis=0)
@@ -389,6 +473,7 @@ class Tracker:
def status(self):
return {"calibration": self.cal.info if self.cal else None, "calibrating": self.calibrating,
"feature": FEATURE, "model": NET.folder if NET else None,
"dots": len(self.dots),
"eyes": {EYES[e.eye]: {"shift": e.shift.value.tolist(), "clicks": len(e.shift.meas),
"jump": bool(np.any(e.shift.jump)), "jump_px": e.shift.jump.tolist(),
@@ -493,7 +578,8 @@ def main():
sock.bind(SOCKET)
sock.setblocking(False)
cal = tracker.cal
print("ft-eyes: " + (f"calibration from {cal.info.get('made')} ({cal.info.get('from', cal.info.get('capture'))})"
print("ft-eyes: " + (f"the eye net {NET.name} ({NET.folder}); " if NET else "")
+ (f"calibration from {cal.info.get('made')} ({cal.info.get('from', cal.info.get('capture'))})"
if cal else "not calibrated: run the probe's calibration with the Own tracker")
+ f"; waiting for {CAMS}", file=sys.stderr, flush=True)
cams = wait_for_cams(sock, tracker, want)
+6 -3
View File
@@ -2,10 +2,12 @@
"""ft-eyes-replay: play a recording into the shared-frame layout, as ft-eyegrab --share
would, at the recorded pace, to test ft-eyes without the headset.
Usage: lab/py lab/ft-eyes-replay NAME [PATH] [--from S] [--for S]
Usage: lab/py lab/ft-eyes-replay NAME [PATH] [--from S] [--for S] [--now]
NAME is a recording (a bare name is in eyes_lab.CAPTURES). PATH defaults to
/dev/shm/frametop-eyes-cams-replay; run ft-eyes with FT_EYES_CAMS=PATH (and FT_EYES_GAZE=...
so it doesn't overwrite the live output).
so it doesn't overwrite the live output). Frames keep the recording's times, unless --now: then
they're stamped with today's CLOCK_MONOTONIC_RAW as they're written, as ft-eyegrab's are (for
tools that line them up with something live, such as the eye recorder's dots).
"""
import mmap
import os
@@ -42,6 +44,7 @@ def main(argv):
count = [0, 0]
t0 = idx[0, 3] + start
wall0 = time.monotonic()
shift = time.clock_gettime(time.CLOCK_MONOTONIC_RAW) - t0 if "--now" in argv else 0.0
try:
for i in range(len(frames)):
t = idx[i, 3]
@@ -58,7 +61,7 @@ def main(argv):
seq = struct.unpack_from("<Q", mm, off)[0]
struct.pack_into("<Q", mm, off, seq + 1)
mm[off + 64:off + 64 + W * H] = frames[i].tobytes()
struct.pack_into("<dQII", mm, off + 8, t, n, cam, int(idx[i, 1]))
struct.pack_into("<dQII", mm, off + 8, t + shift, n, cam, int(idx[i, 1]))
struct.pack_into("<Q", mm, off, seq + 2)
count[cam] = n + 1
struct.pack_into("<Q", mm, 24 + 8 * cam, n + 1)
+59
View File
@@ -0,0 +1,59 @@
{
"catalog": 1,
"kind": "eyes",
"latest": "eyes-v0",
"default": "classic",
"baseline": "steamvr",
"metrics": [
{"key": "fresh", "label": "Right after calibrating, median", "unit": "°", "better": "lower",
"note": "One person, a later recording after taking the headset off and on again, 124 click targets."},
{"key": "fresh_p90", "label": "Right after calibrating, 90% of looks within", "unit": "°", "better": "lower"},
{"key": "reseat_clicks", "label": "After a re-seat, old calibration + click corrections, median", "unit": "°", "better": "lower",
"note": "The headset was taken off and put back on; the calibration is the one from before, and the pointer's clicks correct it."},
{"key": "reseat_clicks_p90", "label": "After a re-seat, with click corrections, 90% within", "unit": "°", "better": "lower"},
{"key": "reseat", "label": "After a re-seat, old calibration alone, median", "unit": "°", "better": "lower"}
],
"models": [
{
"name": "classic",
"title": "Frametop classic (pupil and glints)",
"kind": "classic",
"status": "builtin",
"summary": "Our hand-written tracker: finds each pupil and its glints. Best once your clicks have corrected it.",
"notes": "No learned model: it finds the pupil and the two glints in each eye camera's frame and maps them with your calibration. The glints show when the headset slips, so the pointer's clicks correct it quickly after a re-seat. Right after a calibration it's less precise than SteamVR's.",
"cost": "CPU, in Frametop's container, while the gaze is in use. Not yet measured side by side with the nets.",
"bench": {"fresh": 2.83, "fresh_p90": 4.71, "reseat_clicks": 1.25, "reseat_clicks_p90": 3.35, "reseat": 14.03}
},
{
"name": "steamvr",
"title": "SteamVR (Valve's tracker)",
"kind": "valve",
"status": "builtin",
"summary": "Valve's own eye tracking, as SteamVR reports it, with Frametop's calibration on top.",
"notes": "The most precise right after calibrating, and it needs nothing installed. It drifts when the headset moves, and our click corrections help it less than the classic tracker.",
"cost": "None for Frametop: Valve's tracker runs on the headset's own processors.",
"bench": {"fresh": 1.18, "fresh_p90": 2.32, "reseat_clicks": 3.11, "reseat_clicks_p90": 5.43, "reseat": 3.73}
},
{
"name": "eyes-v0",
"title": "Frametop eye net v0",
"kind": "custom",
"status": "testing",
"date": "2026-10-16",
"summary": "Experimental: our first deep-learning eye net. Better than classic right after calibrating, worse after the headset moves.",
"notes": "A small network that reads each eye camera's frame and gives a raw gaze, which your calibration fits (each net keeps its own calibration, so calibrate once after switching).\n\nTrained on one person in one headset seating, so it doesn't carry over well when the headset sits differently: after a re-seat, the classic tracker with click corrections is more than twice as precise. Try it to help test the pipeline; later versions trained on more people should do better.",
"cost": "CPU, while the gaze is in use: 1.4 ms per pair of eye frames in a benchmark (12% of one big core at 90 frames a second), about 1.2 ms per frame inside ft-eyes.",
"bench": {"fresh": 1.61, "fresh_p90": 3.23, "reseat_clicks": 2.87, "reseat_clicks_p90": 6.41, "reseat": 4.0},
"format": 1,
"rev": "5f4341cc53d698f622368c22bf4bae1addbe20a0",
"files": {
"NOTICE": "2894eeb7d9269890ea54a5546b3fd7ee1c7e4cba0d0c1c64f8207dab68de7acf",
"model.json": "9d7e252137348148597b1a951066053868d4504229520459bb31e95795cd2101",
"model.ncnn.bin": "230cac3489333257ad99ad1ce5ce6fa76fe60a80f4fa92fb8bc78c7f50a9a44a",
"model.ncnn.param": "ec82393354ff522bbd17be57d96a68ebdd1337915f48dfa86ac989f4756f984b",
"LICENSE": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
"README.md": "2625af9abfe4bd350c268f39c0f7ee315cd40f83ada2f93253a0abf68c2f78c2"
}
}
]
}
+3
View File
@@ -3,3 +3,6 @@
# VTK, GDAL, and over a gigabyte of map data; these wheels are about 165 MB.
numpy==2.5.3
opencv-python-headless==5.0.0.93
# The eye net (eyes_net.py, EYES_MODEL): ncnn's own wheel. Not in the image yet: build.sh adds
# it to build/venv there from this line.
ncnn==1.0.20260526
+49 -33
View File
@@ -4,64 +4,80 @@ title: Install the Frametop Hand Recorder
# Install the Frametop Hand Recorder
The Hand Recorder records your hands with the Steam Frame's tracking cameras for Frametop's open hand dataset ([DeeJanuz/frametop-hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) on Hugging Face). These are the Konsole commands to install it.
The Hand Recorder records your hands with the Steam Frame's tracking cameras for Frametop's open hand dataset ([DeeJanuz/frametop-hands](https://huggingface.co/datasets/DeeJanuz/frametop-hands) on Hugging Face), which trains Frametop's hand tracking.
> **Fixed in Frametop 0.2.1 (October 5, 2026):** the recorder now works on headsets without the Arcturus color passthrough module too. If you installed it earlier and the camera check stopped with "Not all of the headset's tracking cameras are running (ft-camd publishes only 2 of 4 mono cameras ...)", run the commands under [Update](#update).
You pick a **bounty**: a short round, about 4 minutes, of what the hand tracker needs most right now. Windows appear in front of you in the headset, and you poke, tap, pinch and grab them as you would real ones while the cameras record your hands. Afterwards you watch what was recorded, delete anything you don't want to share, and upload it from your own Hugging Face account. Nothing leaves the headset until you press Upload.
With Frametop's hand tracking installed there's also a **testing session**: up to 3 minutes of using Frametop as you always do. When the tracking gets something wrong, press the headset button to mark the moment. New versions of the hand tracking are tested on these recordings before they ship. Testing sessions are never used for training and aren't published: they're encrypted on your headset so that only the maintainer can open them.
Join the [Frametop Discord](https://discord.gg/W3X9f7z3Bc) for questions and help with recording.
These steps install Frametop first, and the recorder runs inside Frametop's desktop. To record **bounty rounds** (short rounds of taps, pinches and grabs, about 4 minutes each) without installing Frametop, use the standalone [Frametop Hand Recorder](https://github.com/Frametop/frametop-hand-recorder) instead: it runs from the SteamVR dashboard and installs with one command. Bounty rounds go to the same public dataset. In Frametop itself they need the experimental version (`--experimental` instead of `--stable` below).
There are two ways to install it. Both record the same rounds for the same dataset.
Installing a Frametop release (Frametop.zip, FrameDrop, or `get.sh --release`) instead? Its installer offers the Hand Recorder, built, so there's nothing to compile: leave Hand Recorder ticked (or answer yes) and skip step 2. To add it to a release later, run `~/.local/share/frametop/releases/current/hands/rec/install.sh`.
- [**Without Frametop**](#without-frametop): the standalone Hand Recorder runs from SteamVR's dashboard. One command installs it.
- [**With Frametop**](#with-frametop): Frametop's experimental release brings the recorder already built. It runs on Frametop's desktop.
## Before you start
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/Frametop/frametop/blob/main/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first session.
- You must be 18 or older, and for now you can't take part if you live in Illinois, Texas or Washington (USA). The [consent text](https://github.com/Frametop/frametop/blob/experimental/hands/rec/CONSENT.md) explains what's recorded and what you agree to. The recorder shows it again before your first round.
- You need a Steam Frame on the stable SteamOS release (not the beta), an internet connection, and a keyboard (Bluetooth, or the on-screen one).
- You need a `sudo` password. If you've never set one, run `passwd` in Konsole first.
- Recordings are several gigabytes per round, and uploading one needs about the same again free while it runs. `df -h ~` shows your free space.
- You need a `sudo` password: the camera broker needs it once to read the cameras. If you've never set one, run `passwd` in Konsole first.
- A round and its export take up to about 30 GB. You can delete both once the round is uploaded. `df -h ~` shows your free space.
## Open Konsole
## Without Frametop
1. In the headset, open the dashboard, then **Launch a program** → **Konsole**.
2. Run:
```
curl -fsSL https://raw.githubusercontent.com/Frametop/frametop-hand-recorder/main/get-hand-recorder.sh | bash
```
It asks for your `sudo` password once.
3. Open the dashboard, then **Launch a program** → **Frametop Hand Recorder**.
To update, run the command again: it installs the newest release. To uninstall:
```
~/.local/share/frametop-hand-recorder/current/get-hand-recorder.sh --uninstall
```
The [standalone recorder's README](https://github.com/Frametop/frametop-hand-recorder#readme) says what the installer changes.
## With Frametop
1. In the launcher, choose Launch a program → Desktop.
2. In the application menu, open System → Konsole.
3. Run:
## 1. Install Frametop
```
curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --release --experimental
```
This downloads Frametop's newest experimental release (about 1.4 GB) and installs it without compiling anything. The installer asks what to install: say yes to the Hand Recorder, and to hand tracking if you want to record testing sessions. It asks for your `sudo` password for those. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, the same command updates it, and the Hand Recorder with it.
4. Choose Launch a program → Desktop again, and open Frametop Hand Recorder from the application menu.
Installing `Frametop.zip` with [FrameDrop](https://framedropvr.com), or by hand from the [releases](https://github.com/Frametop/frametop/releases), works too: leave Hand Recorder ticked in the installer's window.
To add the recorder to a release you've already installed:
```
curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
~/.local/share/frametop/releases/current/hands/rec/install.sh
```
This clones Frametop into `~/frametop` and runs its installer. The first run downloads 1–2 GB. The installer asks a few questions (gaze mode, the eye tracker, the Bluetooth fixes); the defaults are fine. At the end SteamVR restarts, which closes Konsole. If Frametop is already installed, this updates it.
From a clone of the experimental branch (`get.sh --experimental`, without `--release`), `~/frametop/hands/rec/install.sh` builds it, which takes a few minutes. Frametop's stable version (0.2.2) has an older recorder without bounty rounds.
## 2. Install the Hand Recorder
## Record
After SteamVR restarts, choose Launch a program → Desktop again, open Konsole, and run:
Open the recorder. It starts on **Bounties**, with the rounds needed most at the top. Press **Record** on one. The first time, it shows the consent text. Then a short checklist checks the cameras, and the round starts in the headset.
```
~/frametop/hands/rec/install.sh
```
While you record, use the button on the right side of the headset: press it to start each step, press it twice to record a step again, and hold it to stop. In a testing session, a press marks a moment and a hold finishes.
This builds the camera broker, the hand tracker and the headset panel (the first build takes a few minutes), asks for your `sudo` password once to let the camera broker read the cameras, and adds Frametop Hand Recorder to the application menu.
## 3. Record
Open Frametop Hand Recorder from the desktop's application menu. It walks you through the consent text, a short checklist, the recording, a review of what you recorded, and the upload to Hugging Face. Nothing leaves the headset until you press Upload.
## Update
```
curl -fsSL https://frametop.github.io/frametop/get.sh | bash -s -- --stable
~/frametop/hands/rec/install.sh
```
Run the second command after SteamVR has restarted, as in the install.
After the round, **Export and upload** prepares it and opens the upload page, where you log in to Hugging Face. Your upload arrives as a pull request on the dataset, and the maintainer checks it before it's merged.
## Uninstall
```
~/frametop/hands/rec/install.sh uninstall
```
For the standalone recorder, see [Without Frametop](#without-frametop). For Frametop's, run its installer with `uninstall`: `~/.local/share/frametop/releases/current/hands/rec/install.sh uninstall` for a release, or `~/frametop/hands/rec/install.sh uninstall` for a clone.
This removes the menu entry. With your `sudo` password, it also takes back the camera broker's permission to read the cameras. If you also installed Frametop's live hand tracking, the camera broker keeps that permission, because live hand tracking still uses it. Your recordings stay in `~/.local/share/frametop/hands/contrib`; delete that folder to remove them. To remove Frametop as well, follow [Uninstall](https://github.com/Frametop/frametop#uninstall) in the README.
+2
View File
@@ -13,3 +13,5 @@ taps-out/
tools/taps-*.sh
# recorded demo clips (hands/rec/demo/camgifs.py) show the room around the recorder: never shipped
rec/demos-recorded/
# hand model versions picked in Input Settings (models/catalog.py downloads them; CC BY-NC 4.0)
models/versions/
+3 -2
View File
@@ -50,7 +50,8 @@ Settings in `~/.config/frametop.conf` (`FT_<name>` in the environment overrides
- `HANDS_SWAP_SIDES=auto` (the default): ft-hands tells from the hands which side camera is which, and corrects ft-camd's names when they're backwards (see "Which camera is which" below). `1` forces them exchanged and `0` forces ft-camd's names; ft-hands still checks, and if the hands disagree it logs a warning and publishes the hands' answer as the truth (`sides.json`), so recordings are labelled right. The example config said `0` until 2026-10-05; `scripts/conf-migrate.sh` (run by `install.sh` and `hands/rec/install.sh`) turns that untouched line into `auto`.
- `HANDS_CPUS=5,6,7`: the CPUs the model threads run on (below).
- `HANDS_CAMERAS` (`auto`), `HANDS_BRIGHT` (`all`), `HANDS_BRIGHT_ON` (40), `HANDS_BRIGHT_OFF` (25): which cameras ft-hands tracks with, as `--cams`, `--bright`, `--bright-on` and `--bright-off` (see ft-hands). `HANDS_CAMERAS=mono` also keeps ft-camd off the colour cameras.
- `HANDS_MODELS` (unset: `hands/models/student`, the fine-tuned models, when `hands/models.sh` fetched them, else `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to try other models without passing options to every launcher. The fine-tuned folders lack the `*-int8` files, so `--int8` without `--models` uses the stock ones.
- `HANDS_MODEL=latest`: which hand models to run, picked on Input Settings' Hands page with notes and benchmarks: `latest` (`hands/models/student`, kept current by `models.sh`), `stock` (`hands/models/ncnn`, Google's MediaPipe models), or a version from `hands/models.json`, downloaded into `hands/models/versions/NAME` (see `models/README.md`). A version that isn't downloaded falls back to latest.
- `HANDS_MODELS` (unset: what `HANDS_MODEL` picks; by default `hands/models/student`, the fine-tuned models, when `hands/models.sh` fetched them, else `hands/models/ncnn`, the stock MediaPipe models): a folder holding `palm.ncnn.*` and `hand.ncnn.*`, as `--models`. Use it to try other models without passing options to every launcher. The fine-tuned folders lack the `*-int8` files, so `--int8` without `--models` uses the stock ones.
- `HANDS_MODELS_UPDATE=1`: when hand tracking starts, fetch the fine-tuned models GitHub's branch pins if they changed, without a new release (`models.sh` below). `0` keeps the ones it came with.
- `HANDS_COLOR_LEFT` (`color_video0`), `HANDS_COLOR_CROP` (`subtract`): how the colour module's calibration maps onto its images, as `--color-left` and `--color-crop`.
- `HANDS_SKELETON=1`: fit a skeleton with your bone lengths and publish it (see "Skeleton" below); `0` publishes the filtered landmarks as before. As `--skeleton`.
@@ -227,7 +228,7 @@ ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gest
With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/ft-pointer.cpp`) reads the gestures file every frame. It's off by default.
**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`).
**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. The ray comes up as the thumb nears the index or middle tip (`POINTER_RAY_READY`). With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), a three-finger pinch (thumb, index and middle together) scrolls as the hand moves, and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`).
- **Pinch to click, in gaze mode.** Your eyes aim and your hands click, Vision Pro style, and the hand ray stays off (`POINTER_GAZE_HAND_RAY`). A pinch locks the pointer where you were looking just before the fingers began to close (75 ms before ft-hands saw the pinch, since the eyes often move on as the hand closes), and a quick tap clicks there. Pinch, hold and move the hand to nudge the pointer: nothing moves until the hand has moved 4 degrees (about 3 cm at arm's length), then the pointer follows at 0.3 times the hand's movement. Letting go clicks where the pointer is, and the correction is a lesson for the gaze tracker, as with the mouse. Held still for `POINTER_GAZE_HOLD` (0.5 s) instead, the pinch is a real press at that point, and the hand drags at `POINTER_GRIP_GAIN`. A middle pinch is a right click the same way. Hold a pinch with one hand and pinch with the other: both buttons are down, left for the first hand and right for the second, each until its own pinch opens (a first pinch not yet pressed becomes a press at once, where the pointer is), as when you hold both of a mouse's buttons. The rules and timings are in `pointer/helper/gazehand.h`.
- **Without gaze mode,** a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is.
+2
View File
@@ -14,6 +14,8 @@ PartOf=steamvr.service
# every 15 minutes, never more than a minute, offline or failing it starts with the ones it has).
ExecStartPre=-/usr/bin/pkill -x ft-hands
ExecStartPre=-/usr/bin/timeout 60 @REPO@/hands/models.sh auto
# A version picked by name (HANDS_MODEL, Input Settings' Hands page) that isn't downloaded yet.
ExecStartPre=-/usr/bin/timeout 60 /usr/bin/python3 @REPO@/models/ft-models auto hands
ExecStart=@REPO@/scripts/in-box @REPO@/hands/build/ft-hands --status 60
ExecStopPost=-/usr/bin/pkill -x ft-hands
Restart=always
+11 -7
View File
@@ -70,10 +70,12 @@
// not near not near a panel (ft-screens' near near: at once (near wins)
// claim, or the fingertip at an overlay)
// not a fist no grip down a grip down
// ready the thumb and index tips within readyHide (0.10 m) apart: the laser is
// readyShow (0.06 m) (user, 2026-10-11: for pinching, not a hand held up (0: off;
// "the aiming laser only activates when frames without the gap pass)
// your fingers get within a distance")
// ready the thumb tip within readyShow (0.06 readyHide (0.10 m) apart: the laser is
// m) of the index or the middle tip for pinching, not a hand held up (0: off;
// (user, 2026-10-11: "the aiming laser frames without either gap pass). The
// only activates when your fingers get middle counts for its right click and the
// within a distance"; 2026-10-15: also three-finger scroll
// for the middle and three-finger taps)
// open not a fist: HandFrame::curl (the more fistCurl - 0.05: a hand closing into a
// open of the index and the other three fist drops its laser, so the fist's
// fingers) at least fistCurl (1.1) thumb-to-index touch isn't a click
@@ -147,7 +149,7 @@ struct Params {
double forearmShow = -35, forearmHide = -50, elbowConfMin = 0.5;
double pitchMin = -45, pitchMax = 60, pitchMinHide = -60, pitchMaxHide = 75;
double yawShow = 70, yawHide = 80;
double readyShow = 0.06, readyHide = 0.10; // the thumb and index tips apart (0: no "ready" test)
double readyShow = 0.06, readyHide = 0.10; // the thumb to the nearer of the index and middle tips (0: no "ready" test)
double fistCurl = 1.1; // HandFrame::curl under this: a fist (0: no "open" test)
// filter (see the top)
double minCutoff = 0.8, beta = 0.06, dCutoff = 1.0, originMinCutoff = 1.5, originBeta = 0.5;
@@ -276,7 +278,8 @@ struct HandFrame {
double elbowConf = 0; // 0: the arm model's guess
bool grip = false; // a grip (closed hand) is down
bool near = false; // near a panel (see "not near" at the top)
double gap = -1; // the thumb and index tips apart (m); < 0 unknown (the "ready" test passes)
double gap = -1; // the thumb and index tips apart (m); < 0 unknown
double gap2 = -1; // the thumb and middle tips apart (m); < 0 unknown (both: the "ready" test passes)
double curl = -1; // how open the hand is (FistCurl): about 1.9 open, under 1.1 a fist; < 0 unknown
};
@@ -328,7 +331,8 @@ inline Tests Evaluate(const Params &p, const Body &body, const HandFrame &f, con
test(r.pitch >= p.pitchMin && r.pitch <= p.pitchMax, r.pitch >= p.pitchMinHide && r.pitch <= p.pitchMaxHide, "aim");
test(r.yawOff <= p.yawShow, r.yawOff <= p.yawHide, "yaw");
test(!f.grip, !f.grip, "grip");
if (p.readyShow > 0 && f.gap >= 0) test(f.gap <= p.readyShow, f.gap <= std::max(p.readyShow, p.readyHide), "ready");
const double gap = f.gap < 0 ? f.gap2 : f.gap2 < 0 ? f.gap : std::min(f.gap, f.gap2);
if (p.readyShow > 0 && gap >= 0) test(gap <= p.readyShow, gap <= std::max(p.readyShow, p.readyHide), "ready");
if (p.fistCurl > 0 && f.curl >= 0) test(f.curl >= p.fistCurl, f.curl >= p.fistCurl - 0.05, "fist");
r.show = show, r.keep = keep;
return r;
+80
View File
@@ -0,0 +1,80 @@
{
"catalog": 1,
"kind": "hands",
"latest": "v16b",
"default": "latest",
"baseline": "stock",
"metrics": [
{"key": "replay_bright", "label": "Hands tracked, bright room (90 s replay)", "unit": "", "better": "higher",
"note": "The maintainer's recordings replayed through ft-hands with its default settings."},
{"key": "replay_dim", "label": "Hands tracked, dim room (120 s replay)", "unit": "", "better": "higher"},
{"key": "replay_lit", "label": "Hands tracked, lit room (120 s replay)", "unit": "", "better": "higher"},
{"key": "found", "label": "Hand found in ft-hands' crops (held-out people)", "unit": "%", "better": "higher",
"note": "Contributors the model never trained on (Frametop Hands #4, #7, #8)."},
{"key": "landmark_px", "label": "Finger point error, median (held-out people)", "unit": "px", "better": "lower"},
{"key": "within5", "label": "Points within 5% of hand size (held-out people)", "unit": "%", "better": "higher"},
{"key": "shape_mm", "label": "3D hand shape error, median (held-out people)", "unit": "mm", "better": "lower"},
{"key": "side_ok", "label": "Left or right hand told correctly (held-out people)", "unit": "%", "better": "higher"},
{"key": "pinch_closed", "label": "Thumb-index pinch read closed (held-out people)", "unit": "%", "better": "higher",
"note": "Of the held-out views where the thumb and index tips touch, the share the model reads as touching."}
],
"models": [
{
"name": "v16b",
"title": "Frametop v16b",
"kind": "custom",
"status": "recommended",
"date": "2026-10-16",
"summary": "v13b-p1 plus the pinch and tap bounty rounds: pinches and taps read closed more often.",
"notes": "v13b-p1's landmark model trained further with Frametop Hands contributions #12 (index taps), #13 (pinches) and #14. The palm detector is v13b-p1's, unchanged. Same networks as stock, so the same CPU cost.\n\nWhat to expect against v13b-p1: a thumb touching the index finger is read as touching more often (80% of held-out touches against 71%), so pinches and taps register more reliably. Finger points and hand shape are as close as before, and the replays track the same hands.\n\nKnown weak spots: a thumb touching the middle finger is still often read as a near miss, and empty views are taken for a hand slightly more often than with v13b-p1 (0.37% against 0.22% of empty crops).",
"cost": "Same as stock: two networks, fp16, on the cores hand tracking already uses.",
"bench": {"replay_bright": 960, "replay_dim": 1225, "replay_lit": 1434, "found": 98.7, "landmark_px": 3.8,
"within5": 94.7, "shape_mm": 8.1, "side_ok": 96.6, "pinch_closed": 80.0},
"format": 1,
"rev": "d96e56599b0af18fc05af2a0a4489985914b754a",
"files": {
"NOTICE": "0d31ad8d108fc01e9fc4c5612eb5a9b16d90f9a971e6677cafe4969c2388fc06",
"hand.ncnn.bin": "c602ad2344bd1676a628fc8ab4fca2f53574b9f12bc97c705f47762d23f5f941",
"hand.ncnn.param": "54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd",
"palm.ncnn.bin": "9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909",
"palm.ncnn.param": "c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e",
"LICENSE": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
"README.md": "322fc0b146d301865592958ba4dd512bf249836816b578f0919a2f671f3f014c"
}
},
{
"name": "v13b-p1",
"title": "Frametop v13b-p1",
"kind": "custom",
"status": "older",
"date": "2026-10-14",
"summary": "Retrained on the Steam Frame's own cameras: about twice as many hands found in bright rooms as stock, and steadier fingers.",
"notes": "MediaPipe's palm detector and hand landmark model, retrained on the Frametop Hands dataset (contributions #3 to #7), the maintainer's recordings and flagged tracking failures. Same networks as stock, so the same CPU cost.\n\nWhat to expect against stock: hands are found far more often, most of all in bright rooms (twice as many tracked in the bright replay, 1.3 to 1.8 times in dim and lit ones), the finger points sit about a quarter closer, and the 3D hand shape is about 40% closer.\n\nKnown weak spot: a thumb touching the middle finger is often read as a near miss, so middle-finger pinches (right click) can need a firmer touch.",
"cost": "Same as stock: two networks, fp16, on the cores hand tracking already uses.",
"bench": {"replay_bright": 960, "replay_dim": 1224, "replay_lit": 1433, "found": 98.7, "landmark_px": 3.8,
"within5": 94.6, "shape_mm": 8.3, "side_ok": 96.5, "pinch_closed": 71.3},
"format": 1,
"rev": "41afd78fbd4527456fc4cbd840ffc9f39fdc939a",
"files": {
"NOTICE": "0e6b4fa8ea37665cb699abc36fcf1ee0ffaa63a78dfee788be6613dea6607f84",
"hand.ncnn.bin": "d6a6f00d845fb9ee490276d84b1e978da884361c372614546ed2b1b4d2f5c449",
"hand.ncnn.param": "54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd",
"palm.ncnn.bin": "9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909",
"palm.ncnn.param": "c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e",
"LICENSE": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
"README.md": "29c6c5460e47412a9a37b22d6a7c8a9ed266a872bbcce93d3867f9e56cfe0ca1"
}
},
{
"name": "stock",
"title": "MediaPipe (stock, off the shelf)",
"kind": "stock",
"status": "builtin",
"summary": "Google's original palm and hand models, unchanged. Comes with Frametop.",
"notes": "Trained on colour photos from ordinary lenses. The Frame's tracking cameras are infrared, monochrome and fisheye, so these lose hands often, most of all in bright rooms. Useful to compare, or to go back to if a retrained version tracks worse for you.",
"cost": "Two networks, fp16, on the cores hand tracking already uses.",
"bench": {"replay_bright": 442, "replay_dim": 912, "replay_lit": 782, "found": 94.8, "landmark_px": 5.2,
"within5": 88.5, "shape_mm": 14.3, "side_ok": 93.8, "pinch_closed": 7.8}
}
]
}
+5 -5
View File
@@ -5,12 +5,12 @@
# needs to run them: raise it when a model needs a newer ft-hands, and older installs keep theirs.
# Made by frame-hands' model-repo/pin.py.
format 1
rev 41afd78fbd4527456fc4cbd840ffc9f39fdc939a
model v13b-p1
file NOTICE 0e6b4fa8ea37665cb699abc36fcf1ee0ffaa63a78dfee788be6613dea6607f84
file hand.ncnn.bin d6a6f00d845fb9ee490276d84b1e978da884361c372614546ed2b1b4d2f5c449
rev d96e56599b0af18fc05af2a0a4489985914b754a
model v16b
file NOTICE 0d31ad8d108fc01e9fc4c5612eb5a9b16d90f9a971e6677cafe4969c2388fc06
file hand.ncnn.bin c602ad2344bd1676a628fc8ab4fca2f53574b9f12bc97c705f47762d23f5f941
file hand.ncnn.param 54e08f284ff6f121ff00f4a9c65afc38c0dbb22a32f4af44fc4e0fde6c4a1bbd
file palm.ncnn.bin 9865f5fa3c6bc885cb0b50f201c4ac1bd83b2c1ad64af4e122efa3fc56d61909
file palm.ncnn.param c61df2dde5b221a55c35c9693e530845a95ce2732949a9b3d3c583e3b2bae92e
file LICENSE 41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0
file README.md 29c6c5460e47412a9a37b22d6a7c8a9ed266a872bbcce93d3867f9e56cfe0ca1
file README.md 322fc0b146d301865592958ba4dd512bf249836816b578f0919a2f671f3f014c
+1 -1
View File
@@ -43,7 +43,7 @@ The recorder doesn't record sound, your name, your email address or your Steam a
- Recordings stay on your headset, in `~/.local/share/frametop/hands/contrib`. Nothing is uploaded automatically.
- Before you share anything, you can watch every recording in the Review page. You can delete any stretch of a recording, a whole take or a whole session. Export leaves out what you deleted.
- You upload the export yourself, from the Upload page. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
- You upload the export yourself: from the Upload page, or with Export and upload, which sends it once it's exported. That opens a pull request on the dataset. The maintainer checks it before it becomes part of the dataset, and may decline it. Until it's merged, you can close the pull request yourself.
## Where it goes
+9 -2
View File
@@ -65,6 +65,7 @@ Test hooks:
```
~/.local/share/frametop/hands/contrib/
profile.json consent and contributor id (below)
contributions.json every finished upload, kept when its session is deleted (Review and export, "Freeing space")
sessions/<YYYYMMDD-HHMMSS>/ (a second session started in the same second gets -2, and so on)
session.json the session: checklist answers, lighting, versions, mode, takes
calibration.json /persist/xrservice.json with identifying fields removed (below)
@@ -401,7 +402,7 @@ Short rounds (about 3 min recorded, 4 with reading) of what the hand tracker nee
- `bounties.json` is the board: `{"schema": 1, "updated", "bounties": [{"id", "title", "blurb", "minutes", "priority", "script"}]}`. A newer copy fetched from the dataset (`hub.py board`, the window's "Get the newest board", into `BASE/board-cache.json`) can only describe, reorder, reprioritise or retire bounties: each one's steps come from `bounties/<script>.json`, shipped with the recorder. A bounty whose script this recorder lacks shows "Needs a newer recorder".
- A round script is a normal script (above) with `"record_hz": 30`: hand size first (the no-hands check), then one scene section, and at most 30 s of plain steps after it when a gesture needs no window (the palm up). At 30 sets a second a round is about 2.5 min recorded, about 2.1 MB a second exported.
- The window opens on the Bounties page: the board by priority, then least done first, then in the board's own order (a fetched board ranks bounties of one priority by listing them; before 2026-10-15 the title broke ties), each with Record (Record again once done), "Get the newest board", and below them the full session and the quick round. Picking a round goes to the consent page if this contributor hasn't agreed to the current version, else straight to the checklist; the checklist's Round section shows the pick (or the full/quick choice) with "Pick another…" back to the board, and hides what a bounty doesn't use (objects within reach, controllers, guidance). Its camera check and Start then run the script (`session.py --bounty ID`; `--board` lists them). When the round is done, the Session page's "Export and upload" exports it and opens the Upload page for that export once the export finishes (`exportDone`). session.json gets `"bounty"`, the upload's title and description name it, and Review shows its title by the session. A round that ends `done` (not a dry run) counts in profile.json's `bounties`.
- The window opens on the Bounties page: the board by priority, then least done first, then in the board's own order (a fetched board ranks bounties of one priority by listing them; before 2026-10-15 the title broke ties), each with Record (Record again once done), "Get the newest board", and below them the full session and the quick round. Picking a round goes to the consent page if this contributor hasn't agreed to the current version, else straight to the checklist; the checklist's Round section shows the pick (or the full/quick choice) with "Pick another…" back to the board, and hides what a bounty doesn't use (objects within reach, controllers, guidance). Its camera check and Start then run the script (`session.py --bounty ID`; `--board` lists them). When the round is done, the Session page's "Export and upload" queues its export to upload by itself once exported (logged in), or else exports it and opens the Upload page for that export once the export finishes (`exportDone`). session.json gets `"bounty"`, the upload's title and description name it, and Review shows its title by the session. A round that ends `done` (not a dry run) counts in profile.json's `bounties`.
The first four (2026-10-14), each on the gestures Frametop uses:
@@ -450,6 +451,10 @@ The board can list one script under two bounties: `daylight-taps` is the index-t
Compression runs at nice 19. While it runs with the headset worn, the window notes that VR may stutter a little (exports are done in the headset). Worn is judged the way `frame-job` does: `vrcompositor` runs and a `/sys/class/backlight/*/brightness` reads over 0 (SteamVR turns the panel off 5 s after the headset comes off). CPU work while in VR causes stutter. The proximity sensor is no use here: it read 9-43 with the headset sitting unworn.
Free space and zstd's pipe (2026-10-15). An export is about three quarters of the recording (2026-10-14's 15 GB sessions made 11 GB exports), so a round that fit can leave too little room for its export. Export refuses to start with less than half the kept recording free, and a disk that fills up as it goes ends it with what's free, about what it needs, and the old export it would replace (that one stays until the new one is done), not with "Broken pipe" (`takes.DiskFull`). zstd's messages go to a temporary file, so a zstd that ends early (a full disk, a signal) leaves its own words as the error (`takes.ZstdStopped`), and one that writes a lot can't stall the export. A testing session's encryption drains zstd a piece at a time when it fails. The window reads hub.py's stderr as it comes for the same reason: read only at the end, retry warnings on a bad network could fill the pipe and stall the upload. `tests/test_export_pipes.py` and `recorder/tests/test_window_pipes.py` cover these.
Room to export (2026-10-17). A user's middle-taps export kept failing with "Broken pipe" on rc3: the round passed the 16 GB check before recording, then filled the disk while exporting, with earlier rounds still kept. The Frame's five exports measured 0.72 to 0.76 of their recordings. Now a round needs room for its recording and its export before it starts (`ft_handrec.ROUND_BYTES`: 16 GB plus `takes.export_need` of it, about 31 GB), export needs 0.8 of the kept recording plus 2 GB to start (`takes.export_need`), and it stops itself, removing its half, when the disk gets down to 1.5 GB (`takes.check_room`, as the session stops recording at 1.5 GB), so a full home disk never takes SteamVR and Steam down with it. `tests/test_free_space.py` covers these.
Freeing space (2026-10-17). Nothing is deleted after an upload by itself, and a session's uploads were recorded only in its session.json, so deleting it lost the record. `contributions.json` (`recorder/contributions.py`) now keeps one entry per finished upload: session, bounty, consent kind, dataset, pull request, upload time, export size, and `local_deleted` once its local copy is gone. hub.py adds the entry when an upload finishes, and the window copies in what older recorders recorded only in session.json when it opens. A session is freeable when a finished upload exists and its recordings haven't changed since (an upload without a readable time: when the export there now is the one uploaded); `Store.free_session` writes the log entry first, then deletes the recordings and export, so a log that can't be written deletes nothing. The window offers it per session on Review ("Free up space"), for all of them on Review, on Before you start when there isn't room for a round, on Export after a failed export, and on Upload right after an upload. Review lists "Your uploads" from the log, including the ones deleted here; the board's round counts were already in profile.json, and the testing session's count adds the log's.
- **Upload.** The Upload page uploads an export from the window, logging in included: no terminal. Below its steps it shows `UPLOAD.md` ("About uploading"), with the export's path, size and contributor id filled in.
- **`hands/rec/validate.py`** (standard library) checks an export. The window runs it before an upload, and the maintainer runs it on each submission (`validate.py DIR [--json]`, exit status 1 on errors). It checks:
- `SHA256SUMS`: every file listed and matching.
@@ -485,7 +490,9 @@ The board can list one script under two bounties: `daylight-taps` is the index-t
2. Log in to Hugging Face. The page shows whether someone is logged in (`whoami`). Log in runs `hub.py login`, opens the link in the browser, and shows the code large, with Copy code and Cancel. "Use another account" logs in again. A classic read-only token counts as not logged in.
3. Upload, with a note to accept the dataset's terms the first time.
Upload has Cancel, the phase with a progress bar (a share while the export is checked, a sweep while it's sent, as `huggingface_hub` reports no progress), the pull request's link once it's open, with "plug in the headset and leave it plugged in until this says Uploaded", and Uploaded at the end. If this export was uploaded before, the page says so, and uploading it again asks first. A stale export can't be uploaded.
Upload has Cancel, the phase with a progress bar (a share while the export is checked, a sweep while it's sent, as `huggingface_hub` reports no progress), the pull request's link once it's open, with "plug in the headset and leave it plugged in until this says Uploaded", and Uploaded at the end. If this export was uploaded before, the page says so, and uploading it again asks first. A stale export can't be uploaded. An export and an upload of different sessions run together (2026-10-17: Upload used to wait for any export); a session's own export and upload wait for each other, since an export replaces its old one when it ends. One export and one upload at a time. `tests/test_upload_during_export.py` covers it.
Exports queue (2026-10-17, "so they don't have to keep coming back to the headset"): Export, or Export and upload, while another export runs puts it in a queue on the Export page, run in order. Export and upload (logged in, and the dataset takes contributions) uploads it by itself once its export is done: uploads wait their turn too, one at a time, and an export uploaded before (the same SHA256SUMS) isn't sent again. Each waiting export has a "then upload" box and Remove; Cancel stops only the running export (and its upload), and the next one starts. A failed export or upload doesn't hold up the rest. The page's Done list says how each went, with the pull request's link, for whoever comes back to the headset. The queue lives in the window: closing it asks, then cancels them all. A session waiting to export or upload can't be deleted, nor its export. The awake unit's workers now bring it to the reasons current when they run, so an export's end and its upload's start can't leave it stopped (`recorder/tests/test_stay_awake.py`). `tests/test_export_queue.py` covers the queue.
- **Staying awake:** while an export or an upload runs, the window holds a host unit, `frametop-handrec-awake.service`, running `systemd-inhibit --what=sleep:idle --mode=block ... sleep infinity`, and stops it when the last of them ends (or on quit). It's meant to keep Steam from putting the Frame to sleep with the headset off; whether Steam's sleep honours a logind block inhibitor is still to be checked on the device. Exports and uploads are done in the headset: the export page only notes that VR may stutter a little while it compresses.
- **While the texts are drafts**, Upload stays off unless `FT_HANDREC_ALLOW_UPLOAD=1`, so the maintainer can rehearse against a private test repo. `FT_HANDREC_DATASET` overrides `HF_DATASET`. `ft-handrec --hub-dry-run` makes Upload a dry run: no network, so it isn't held back by the drafts.
- **Rehearsal: `hands/rec/rehearse.sh [--repo ID]`** runs it all without the headset, in the dev container, in one `frame-job --local` scope when frame-job is installed. `ft-ringplay` plays 30 s of a recording into a ring in `/run/user/UID`. A tracking ft-hands that's already running is used, or one is started on that ring. `ft-handpanel --no-vr` stands in for the panel. `session.py --no-start --next-after 0.3` records a three-section test script (a prompt, a two-cue sweep, no hands) in step mode, three parts of 6 s (countdown and hold), about 360 MB once exported. Then `takes.py` exports, `validate.py` checks, and `hub.py` uploads: a dry run by default, or for real to `--repo ID` with `FT_HANDREC_ALLOW_UPLOAD=1`. It prints a summary, deletes its temporary folders (camera images of a room) and stops everything it started, Ctrl+C included. The `--no-vr` panel logs no poses, so `poses.jsonl` is missing there (a warning).
+1 -1
View File
@@ -12,7 +12,7 @@
"done": {
"title": "All done",
"seconds": 6,
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
"text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
},
"stopped": {
"title": "Stopped",
+1 -1
View File
@@ -12,7 +12,7 @@
"done": {
"title": "All done",
"seconds": 6,
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
"text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
},
"stopped": {
"title": "Stopped",
+1 -1
View File
@@ -12,7 +12,7 @@
"done": {
"title": "All done",
"seconds": 6,
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
"text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
},
"stopped": {
"title": "Stopped",
+1 -1
View File
@@ -12,7 +12,7 @@
"done": {
"title": "All done",
"seconds": 6,
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
"text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
},
"stopped": {
"title": "Stopped",
+1 -1
View File
@@ -12,7 +12,7 @@
"done": {
"title": "All done",
"seconds": 6,
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
"text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
},
"stopped": {
"title": "Stopped",
+1 -1
View File
@@ -12,7 +12,7 @@
"done": {
"title": "All done",
"seconds": 6,
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."
"text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."
},
"stopped": {
"title": "Stopped",
+5 -2
View File
@@ -1,6 +1,8 @@
#!/bin/bash
# Launch the Frametop Hand Recorder from a Plasma session on the Frame host.
# The app runs in Frametop's container (PySide6, Kirigami and zstd come from Fedora there).
# The app runs in Frametop's container (PySide6, Kirigami and zstd come from Fedora there; a
# release's image has the recorders' cryptography and huggingface_hub in /opt/frametop/venv, which
# distrobox leaves off the front of PATH, so it goes first here).
# podman needs the real XDG_RUNTIME_DIR and the real user bus (to reach systemd for
# the container's cgroup; the Frametop session runs on a private bus from
# dbus-run-session). The session's Wayland socket and bus go to the app itself.
@@ -14,7 +16,8 @@ export DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/bus
# From the home folder: the app's host commands (distrobox-host-exec) run in its folder on the
# host, and a folder only the container has, such as /run/host/tmp, makes them all fail.
cd "$HOME" || exit 1
exec "$here/../../scripts/in-box" env WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
exec "$here/../../scripts/in-box" env PATH="/opt/frametop/venv/bin:/opt/frametop/bin:$PATH:/usr/sbin:/sbin" \
WAYLAND_DISPLAY="$wl" DISPLAY="${DISPLAY:-}" \
XAUTHORITY="${XAUTHORITY:-}" DBUS_SESSION_BUS_ADDRESS="$session_bus" \
QT_QPA_PLATFORM="wayland;xcb" \
python3 "$here/ft_handrec.py" "$@"
+104 -6
View File
@@ -21,7 +21,8 @@ A Kirigami (QML) app with a Python backend. It runs in the dev container:
focus; the button on the headset's right side is Next, pause and resume). The prompts appear
in the headset. Each step waits for Next unless "Advance by itself" was ticked.
- Review: sessions, their takes, and a viewer for one frame set at a time, where ranges,
takes and sessions can be deleted (takes.py).
takes and sessions can be deleted (takes.py). Uploaded sessions can be deleted to free space
while the record of their uploads stays (contributions.json), listed under "Your uploads".
- Export: compress what's kept into exports/<session>/ at nice 19 (takes.py), with a warning
when the headset is worn.
- Upload: the Hugging Face login status, and Upload: hub.py checks the export (validate.py)
@@ -85,7 +86,11 @@ DEBUG_ROW = {"id": DEBUG_ID, "title": "Testing session", "priority": "", "minute
LIVE_HANDS_UNIT = "frametop-hands.service"
# One window at a time, Frametop's or the standalone one (recorder/one_window.py)
WINDOW_NAME = "ft_handrec_window"
ROUND_BYTES = 16 * 1000 ** 3 # about what one round of recording takes (a bounty: 2.5-3 min at 30 sets/s, ~85 MB/s)
RECORD_BYTES = 16 * 1000 ** 3 # about what one round of recording takes (a bounty: 2.5-3 min at 30 sets/s, ~85 MB/s)
# The free space a round needs before it starts: the recording, then its export next to it
# (takes.export_need). Until 2026-10-17 only the recording was asked for, so a round could be
# recorded that couldn't be exported.
ROUND_BYTES = RECORD_BYTES + takes.export_need(RECORD_BYTES)
# The checklist's choices; the keys are what session.json stores.
OBJECTS = [("pencil", "Pencil or pen"), ("phone", "Phone"), ("cup", "Cup or mug (empty)"),
("keyboard", "Keyboard"), ("mouse", "Mouse"), ("gamepad", "Gamepad"),
@@ -239,6 +244,10 @@ class Backend(RecorderBackend):
self._ready_panel = None # the ft-handpanel getReady started: ends on quit
self._readyPart.connect(self._on_ready_part)
self._readyDone.connect(self._on_ready_done)
try: # uploads older recorders kept only in session.json join the contributions log
store.log_uploads()
except (OSError, ValueError) as e:
print(f"contributions log: {e}", file=sys.stderr, flush=True)
def _texts_draft(self):
return hub.texts_draft(self._consent_kind)
@@ -257,6 +266,9 @@ class Backend(RecorderBackend):
except hub.HubError:
return hub.HF_TEST_DATASET if self._consent_kind == "testing" else hub.HF_DATASET
def _session_label(self, session):
return session_label(session)
def _session_dataset(self, session):
try:
return hub.dataset_id(hub.session_kind(self.store, session))
@@ -378,9 +390,12 @@ class Backend(RecorderBackend):
return self._frametop_hands
def _debug_row(self):
"""The testing session as a row of the board (bounties.view's keys)."""
done = [s for s in self.store.sessions() if s.get("debug") and s["status"] == "done" and not s["dry_run"]]
return dict(DEBUG_ROW, done=len(done), last=done[0]["id"] if done else "", available="")
"""The testing session as a row of the board (bounties.view's keys). Uploaded ones whose
local copy was deleted count from the contributions log."""
done = {s["id"] for s in self.store.sessions() if s.get("debug") and s["status"] == "done" and not s["dry_run"]}
done |= {c["session"] for c in self.store.contributions() if c.get("debug")}
last = max(done) if done else ""
return dict(DEBUG_ROW, done=len(done), last=last, available="")
@Property(str, notify=boardChanged)
def boardUpdated(self):
@@ -704,6 +719,8 @@ class Backend(RecorderBackend):
return "exporting a recording"
if self.uploading:
return "uploading a recording"
if self._upload_queue:
return "waiting to upload a recording"
return ""
def shutdown(self):
@@ -800,6 +817,9 @@ class Backend(RecorderBackend):
s["active"] = self.sessionActive and s["id"] == self._session_id
s["statusText"] = {"recording": "" if s["active"] else "interrupted", "error": "ended with an error",
"stopped": "stopped early"}.get(s["status"], "")
s["freeable"] = s["freeable"] and not s["active"]
s["freeText"] = takes.human_bytes(s["bytes"] + s["export_bytes"])
s["uploadText"] = ("uploaded" if s["freeable"] else "uploaded, changed since") if s["uploaded"] else ""
out.append(s)
return out
@@ -872,7 +892,7 @@ class Backend(RecorderBackend):
@Slot(str)
def deleteSession(self, session):
if self._active_guard(session):
if self._active_guard(session) or self._busy_guard(session):
return
try:
self.store.delete_session(session)
@@ -881,6 +901,84 @@ class Backend(RecorderBackend):
self.message.emit(f"Couldn't delete the session: {e}", True)
self.refreshSessions()
# --- freeing space: uploaded rounds' local copies go, the contributions log keeps the record
def _freeable(self):
busy = {self._session_id if self.sessionActive else "", self._export_session if self.exporting else "",
(self._upload or {}).get("session", "") if self.uploading else ""}
return [f for f in self.store.freeable() if f["id"] not in busy and not self._queued(f["id"])]
@Property(int, notify=sessionsRelay)
def freeableCount(self):
"""Uploaded rounds whose local copies can be deleted (nothing changed since the upload)."""
return len(self._freeable())
@Property(str, notify=sessionsRelay)
def freeableText(self):
"""What deleting them all frees, "" when nothing can go."""
rows = self._freeable()
return takes.human_bytes(sum(f["bytes"] for f in rows)) if rows else ""
@Property(str, constant=True)
def roundNeedText(self):
return gigabytes(self.ROUND_BYTES)
@Property("QVariantList", notify=sessionsRelay)
def contributions(self):
"""The contributions log for the Review page, newest first: {session, label, title,
uploadedText, prUrl, prText, local, testing}."""
out = []
for c in self.store.contributions():
title = (c.get("bounty") or {}).get("title") or ("Testing session" if c.get("debug") else "Session")
when = (c.get("uploaded") or "")[:10]
out.append({"session": c["session"], "label": session_label(c["session"]), "title": title,
"uploadedText": f"uploaded {when}" if when else "uploaded",
"prUrl": c.get("pr_url", ""), "prText": f"pull request #{c['pr_num']}" if c.get("pr_num") else "",
"local": c["local"], "testing": c.get("kind") == "testing",
"deletedText": "deleted here, still in your record" if not c["local"] else ""})
return out
def _free(self, sids):
freed, failed = 0, []
for sid in sids:
try:
freed += self.store.free_session(sid)
except (OSError, ValueError) as e:
failed.append(f"{session_label(sid)}: {e}")
return freed, failed
@Slot(str)
def freeSession(self, session):
"""Delete one uploaded session's recordings and export; its record stays."""
if self._active_guard(session):
return
if session not in {f["id"] for f in self._freeable()}:
self.message.emit("Only an uploaded round, unchanged since its upload, can be deleted this way", True)
return
freed, failed = self._free([session])
if failed:
self.message.emit("Couldn't delete it: " + failed[0], True)
else:
self.message.emit(f"Deleted the local copy of {session_label(session)}: {takes.human_bytes(freed)} freed. "
"Your record of the upload stays.", False)
self.refreshSessions()
@Slot()
def freeAllUploaded(self):
"""Delete every uploaded, unchanged round's local copy; their records stay."""
sids = [f["id"] for f in self._freeable()]
if not sids:
self.message.emit("No uploaded rounds to delete", False)
return
freed, failed = self._free(sids)
done = len(sids) - len(failed)
if failed:
self.message.emit(f"Deleted {done} of {len(sids)} rounds ({takes.human_bytes(freed)} freed). "
"Couldn't delete " + "; ".join(failed), True)
else:
self.message.emit(f"Deleted {done} uploaded round{'s' if done != 1 else ''}: {takes.human_bytes(freed)} "
"freed. Your record of the uploads stays.", False)
self.refreshSessions()
# --- export
@Property(bool, constant=True)
def zstdFound(self):
+9 -1
View File
@@ -21,7 +21,9 @@ An upload:
and records it under "uploads" in session.json with "status": "started";
6. upload_folder(..., revision="refs/pr/N") to contributions/<contributor>/<session>; a retry of
the same export goes on in the same pull request while it's still open;
7. marks the record {"repo", "pr_url", "pr_num", "uploaded", "export_sha", "status": "done"}.
7. marks the record {"repo", "pr_url", "pr_num", "uploaded", "export_sha", "status": "done"},
and adds it to BASE/contributions.json (recorder/contributions.py), which outlives the
session: deleting the local copy to free space keeps it.
--dry-run does all of it except the network calls (4 to 6) and recording (5, 7), and says what
it would upload.
@@ -53,6 +55,7 @@ sys.path.insert(0, HERE)
import takes # noqa: E402 (next to this file)
sys.path.insert(0, os.path.dirname(os.path.dirname(HERE)))
from recorder import bounties # noqa: E402
from recorder import contributions # noqa: E402
from recorder import hub as rhub # noqa: E402
from recorder.hub import HF_ENV, REPO_RE, TOKENS_URL, HubError, explain, export_sha, is_draft, now_iso # noqa: E402,F401
@@ -275,6 +278,11 @@ def upload(store, session, dry_run=False, again=False, progress=None, log=None):
pr, who, record = rhub.push(_hf(), repo, export, path_in_repo, message, description, meta_path(store, session), sha,
len(files), report.bytes, tell, takes.write_json, takes.human_bytes)
try: # the log is for the contributor's own record: a failure to write it doesn't undo the upload
contributions.add(store.base, [contributions.entry(session, takes.read_json(meta_path(store, session)), record,
report.bytes, summary.get("minutes"))])
except OSError as e:
say(f"couldn't add the upload to {contributions.FILE}: {e}")
tell("done", "Uploaded", 1.0)
return dict(plan, dry_run=False, pr_url=record["pr_url"], pr_num=pr.num, user=who.get("name", ""), record=record)
+353 -24
View File
@@ -85,6 +85,24 @@ Kirigami.ApplicationWindow {
return n + " " + word + (n === 1 ? "" : "s")
}
// A session's label, from its id (the Export page's queue).
function sessionLabel(id) {
for (const s of backend.sessions)
if (s.id === id)
return s.label
return id
}
// The session's export can upload by itself once done: logged in, and its dataset takes
// contributions (or a dry run).
function canUploadAfter(id) {
return backend.canUploadAfter && (backend.hubDryRun || backend.uploadAllowedFor(id))
}
// Waiting to export, or exporting now.
function exportPending(id) {
return backend.exportSessionId === id && backend.exporting
|| backend.exportQueue.some(q => q.session === id)
}
function show(page) {
pageStack.clear()
pageStack.push(page)
@@ -137,6 +155,23 @@ Kirigami.ApplicationWindow {
}
}
// Freeing space: an uploaded round's recordings and export go, the record of its upload
// stays (contributions.json, listed on the Review page).
function askFreeAll() {
confirm.ask("Delete the uploaded rounds here?",
root.plural(backend.freeableCount, "uploaded round") + " (" + backend.freeableText + ") are deleted "
+ "from the headset: their recordings and exports. They're already in your pull requests, and "
+ "your record of each upload stays on the Review page.",
() => backend.freeAllUploaded())
}
function askFree(row, after) {
const sid = row.id
confirm.ask("Delete the local copy?",
"The session of " + row.label + " is uploaded (" + row.pr_url + "). Its recordings and export ("
+ row.freeText + ") are deleted from the headset; your record of the upload stays on the Review page.",
() => { if (after) after(); backend.freeSession(sid) })
}
// Restart SteamVR (the camera check's fix). What it does to Frametop, read from the code
// (hands/README.md, "Camera check"): ft-screens quits when SteamVR does, which ends the
// desktop; its unit doesn't restart, and stopping it ends what was started in it.
@@ -180,9 +215,11 @@ Kirigami.ApplicationWindow {
close.accepted = false
confirm.ask("Stop the session?", "A session is recording. Closing stops it; what's recorded so far is kept.",
() => { root.quitting = true; Qt.quit() })
} else if (backend.uploading && !quitting) {
} else if ((backend.uploading || backend.exporting || backend.uploadQueue.length > 0) && !quitting) {
close.accepted = false
confirm.ask("Cancel the upload?", "An upload is running. Closing cancels it; you can start it again later.",
confirm.ask("Cancel the exports and uploads?",
"An export or upload is running or waiting (the Export page lists them). Closing cancels "
+ "them all; an export cancelled keeps nothing. You can start them again later.",
() => { root.quitting = true; Qt.quit() })
}
}
@@ -661,15 +698,26 @@ Kirigami.ApplicationWindow {
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Space" }
Controls.Label {
Kirigami.FormData.label: "Free space:"
text: backend.freeText + " (a round takes about 15 GB, and its export about 12 GB more)"
text: backend.freeText + " (a round needs about " + backend.roundNeedText
+ ": about 16 GB to record and 15 GB to export it)"
}
Kirigami.InlineMessage {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
Layout.fillWidth: true
visible: !backend.diskOk
type: Kirigami.MessageType.Error
text: "Not enough free space for a round. Export and delete earlier sessions (Review), "
+ "or free up space, first."
text: "Not enough free space for a round and its export. "
+ (backend.freeableCount > 0
? root.plural(backend.freeableCount, "uploaded round") + " can go to free "
+ backend.freeableText + ": your record of each upload stays."
: "Delete sessions you don't need (Review), or free up space, first.")
actions: backend.freeableCount > 0 ? [freeAllBefore] : []
Kirigami.Action {
id: freeAllBefore
text: "Delete uploaded rounds"
icon.name: "edit-delete"
onTriggered: root.askFreeAll()
}
}
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Cameras" }
@@ -1328,15 +1376,22 @@ Kirigami.ApplicationWindow {
icon.name: "media-playback-stop"
onClicked: backend.stopSession()
}
// Logged in: the export uploads by itself once done (queued behind any other).
// Not yet: the Upload page follows the export, to log in there.
Controls.Button {
visible: !backend.sessionActive && backend.sessionId !== "" && !backend.exporting
visible: !backend.sessionActive && backend.sessionId !== ""
enabled: !root.exportPending(backend.sessionId)
focusPolicy: Qt.NoFocus
text: "Export and upload"
icon.name: "cloud-upload"
onClicked: {
root.chosenSession = backend.sessionId
root.sendAfterExport = true
backend.exportSession(backend.sessionId, false)
if (root.canUploadAfter(backend.sessionId)) {
backend.queueExport(backend.sessionId, false, true)
} else {
root.sendAfterExport = true
backend.exportSession(backend.sessionId, false)
}
root.show(exportPage)
}
}
@@ -1368,10 +1423,58 @@ Kirigami.ApplicationWindow {
model: backend.sessions
spacing: Kirigami.Units.smallSpacing
header: ColumnLayout {
width: ListView.view.width
Kirigami.InlineMessage {
Layout.fillWidth: true
Layout.bottomMargin: visible ? Kirigami.Units.smallSpacing : 0
visible: backend.freeableCount > 0
type: Kirigami.MessageType.Information
text: root.plural(backend.freeableCount, "uploaded round") + " can be deleted here to free "
+ backend.freeableText + ". Your record of each upload stays, below."
actions: [
Kirigami.Action {
text: "Delete uploaded rounds"
icon.name: "edit-delete"
onTriggered: root.askFreeAll()
}
]
}
}
// The contributions log: every upload, kept when its local copy is deleted.
footer: ColumnLayout {
width: ListView.view.width
visible: backend.contributions.length > 0
spacing: Kirigami.Units.smallSpacing
Kirigami.Heading {
Layout.topMargin: Kirigami.Units.largeSpacing
Layout.leftMargin: Kirigami.Units.largeSpacing
level: 3
text: "Your uploads"
}
Repeater {
model: backend.contributions
delegate: Controls.Label {
required property var modelData
Layout.fillWidth: true
Layout.leftMargin: Kirigami.Units.largeSpacing
Layout.rightMargin: Kirigami.Units.largeSpacing
wrapMode: Text.Wrap
textFormat: Text.StyledText
text: "<b>" + modelData.title + "</b> · " + modelData.label + " · " + modelData.uploadedText
+ (modelData.prUrl ? " · <a href=\"" + modelData.prUrl + "\">" + (modelData.prText || "pull request") + "</a>" : "")
+ (modelData.testing ? " · testing" : "")
+ (modelData.deletedText ? " · " + modelData.deletedText : "")
onLinkActivated: link => Qt.openUrlExternally(link)
}
}
}
Kirigami.PlaceholderMessage {
anchors.centerIn: parent
width: parent.width - Kirigami.Units.gridUnit * 4
visible: parent.count === 0
visible: parent.count === 0 && backend.contributions.length === 0
text: "No sessions yet"
explanation: "Recorded sessions appear here, to watch and trim before you export them."
}
@@ -1407,8 +1510,18 @@ Kirigami.ApplicationWindow {
+ (sessionRow.modelData.lightingText ? " · " + sessionRow.modelData.lightingText : "")
+ (sessionRow.modelData.export_stale ? " · changed since its export"
: sessionRow.modelData.exported ? " · exported (" + sessionRow.modelData.exportText + ")" : "")
+ (sessionRow.modelData.uploadText ? " · " + sessionRow.modelData.uploadText : "")
}
}
Controls.ToolButton {
icon.name: "edit-clear-all"
text: "Free up space: delete the local copy, keep the record of the upload"
display: Controls.AbstractButton.IconOnly
visible: sessionRow.modelData.freeable
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: root.askFree(sessionRow.modelData, () => pageStack.pop(pageStack.items[0]))
}
Controls.ToolButton {
icon.name: "edit-delete"
text: "Delete session"
@@ -1722,6 +1835,12 @@ Kirigami.ApplicationWindow {
return s
return null
}
// The chosen session is exporting, or waiting to.
readonly property bool pending: chosen !== null && root.exportPending(chosen.id)
readonly property bool canExport: chosen !== null && !pending && backend.zstdFound && !chosen.active
&& !(backend.uploading && backend.upload.session === chosen.id)
&& backend.uploadQueue.indexOf(chosen.id) < 0
Component.onCompleted: if (backend.login.state === "unknown") backend.checkLogin()
Timer {
interval: 5000
@@ -1740,6 +1859,15 @@ Kirigami.ApplicationWindow {
type: Kirigami.MessageType.Information
text: "Exporting your session: the Upload page follows by itself once it's done."
}
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: backend.uploadAfter.length + backend.uploadQueue.length > 0
position: Kirigami.InlineMessage.Position.Header
type: Kirigami.MessageType.Positive
text: "You can take the headset off and leave it plugged in: it stays awake until the "
+ "exports and uploads below are done, and they show here how they went. Keep this "
+ "window open: closing it cancels them."
}
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: exportView.worn && backend.exporting
@@ -1764,7 +1892,6 @@ Kirigami.ApplicationWindow {
model: backend.sessions
textRole: "label"
valueRole: "id"
enabled: !backend.exporting
Component.onCompleted: {
const i = indexOfValue(root.chosenSession)
currentIndex = i >= 0 ? i : (count > 0 ? 0 : -1)
@@ -1780,32 +1907,73 @@ Kirigami.ApplicationWindow {
Kirigami.FormData.label: "Export:"
visible: exportView.chosen !== null
text: !exportView.chosen ? ""
: exportView.pending ? (backend.exportSessionId === exportView.chosen.id && backend.exporting
? "exporting now" : "waiting its turn, below")
: exportView.chosen.export_stale ? "changed since it was exported: export it again"
: exportView.chosen.exported ? "ready, " + exportView.chosen.exportText : "not exported yet"
}
// Exports run one at a time; one asked for meanwhile waits its turn. "Export and
// upload" sends it by itself once it's exported.
RowLayout {
Controls.Button {
text: backend.exporting ? "Exporting…"
: exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
text: exportView.chosen && exportView.chosen.exported ? "Export again" : "Export"
icon.name: "document-export"
enabled: exportView.chosen !== null && !backend.exporting && backend.zstdFound
&& !exportView.chosen.active
onClicked: backend.exportSession(exportView.chosen.id, false)
enabled: exportView.canExport
onClicked: backend.queueExport(exportView.chosen.id, false, false)
}
Controls.Button {
text: "Export and upload"
icon.name: "cloud-upload"
enabled: exportView.canExport && root.canUploadAfter(exportView.chosen.id)
onClicked: backend.queueExport(exportView.chosen.id, false, true)
}
}
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
visible: exportView.chosen !== null && !backend.hubDryRun
&& (!backend.canUploadAfter || !backend.uploadAllowedFor(exportView.chosen.id))
wrapMode: Text.Wrap
opacity: 0.7
text: !backend.canUploadAfter ? "To upload by itself once exported, log in on the Upload page first."
: "Contributions aren't open yet, so it can't upload by itself."
}
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
visible: backend.exporting && exportView.chosen !== null && !exportView.pending
wrapMode: Text.Wrap
opacity: 0.7
text: "Another export is running: this one waits its turn."
}
// The running export
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Exporting now"
visible: backend.exporting
}
RowLayout {
Kirigami.FormData.label: "Session:"
visible: backend.exporting
Controls.Label { text: root.sessionLabel(backend.exportSessionId) }
// Shows the moment Export is pressed: the first progress can take a few seconds.
Controls.BusyIndicator {
visible: backend.exporting
running: visible
running: backend.exporting
implicitWidth: Kirigami.Units.gridUnit * 1.5
implicitHeight: implicitWidth
}
Controls.Button {
visible: backend.exporting
text: "Cancel"
icon.name: "dialog-cancel"
onClicked: { root.sendAfterExport = false; backend.cancelExport() }
}
}
Controls.CheckBox {
visible: backend.exporting
text: "Upload it once exported"
checked: backend.uploadAfter.indexOf(backend.exportSessionId) >= 0
enabled: checked || root.canUploadAfter(backend.exportSessionId)
onToggled: backend.setUploadAfter(backend.exportSessionId, checked)
}
// Drawn here: the desktop style's ProgressBar paints nothing without a QApplication.
Rectangle {
Kirigami.FormData.label: "Progress:"
@@ -1827,8 +1995,23 @@ Kirigami.ApplicationWindow {
wrapMode: Text.WrapAnywhere
text: backend.exportText
}
Kirigami.InlineMessage {
Layout.maximumWidth: Kirigami.Units.gridUnit * 26
Layout.fillWidth: true
visible: !backend.exporting && backend.exportText.indexOf("Failed") === 0 && backend.freeableCount > 0
type: Kirigami.MessageType.Information
text: root.plural(backend.freeableCount, "uploaded round") + " can be deleted to free "
+ backend.freeableText + ". Your record of each upload stays."
actions: [
Kirigami.Action {
text: "Delete uploaded rounds"
icon.name: "edit-delete"
onTriggered: root.askFreeAll()
}
]
}
RowLayout {
visible: exportView.chosen !== null && exportView.chosen.exported && !backend.exporting
visible: exportView.chosen !== null && exportView.chosen.exported && !exportView.pending
Controls.Button {
text: "Upload"
icon.name: "go-next"
@@ -1837,6 +2020,8 @@ Kirigami.ApplicationWindow {
Controls.Button {
text: "Delete export"
icon.name: "edit-delete"
enabled: !(backend.uploading && backend.upload.session === exportView.chosen.id)
&& backend.uploadQueue.indexOf(exportView.chosen.id) < 0
onClicked: {
const sid = exportView.chosen.id
confirm.ask("Delete this export?", "The export is deleted; the session's recordings stay.",
@@ -1844,6 +2029,107 @@ Kirigami.ApplicationWindow {
}
}
}
// Waiting: exports, then uploads
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Waiting to export"
visible: backend.exportQueue.length > 0
}
Repeater {
model: backend.exportQueue
delegate: RowLayout {
required property var modelData
required property int index
Kirigami.FormData.label: (index + 1) + "."
Controls.Label { text: root.sessionLabel(modelData.session) }
Controls.CheckBox {
text: "then upload"
checked: modelData.upload
enabled: checked || root.canUploadAfter(modelData.session)
onToggled: backend.setUploadAfter(modelData.session, checked)
}
Controls.ToolButton {
icon.name: "list-remove"
text: "Remove"
display: Controls.AbstractButton.IconOnly
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: backend.unqueueExport(modelData.session)
}
}
}
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Uploading"
visible: backend.uploading || backend.uploadQueue.length > 0
}
RowLayout {
Kirigami.FormData.label: "Now:"
visible: backend.uploading
Controls.Label {
text: root.sessionLabel(backend.upload.session || "") + (backend.upload.text ? ": " + backend.upload.text : "")
}
Controls.Button {
text: "Show"
icon.name: "go-next"
flat: true
onClicked: { root.chosenSession = backend.upload.session; root.show(uploadPage) }
}
}
Repeater {
model: backend.uploadQueue
delegate: RowLayout {
required property var modelData
required property int index
Kirigami.FormData.label: "Then:"
Controls.Label { text: root.sessionLabel(modelData) }
Controls.ToolButton {
icon.name: "list-remove"
text: "Don't upload it"
display: Controls.AbstractButton.IconOnly
Controls.ToolTip.text: text
Controls.ToolTip.visible: hovered
onClicked: backend.unqueueUpload(modelData)
}
}
}
// How they went, for whoever comes back to the headset
Kirigami.Separator {
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Done"
visible: backend.doneLog.length > 0
}
Repeater {
model: backend.doneLog
delegate: RowLayout {
required property var modelData
Kirigami.Icon {
source: modelData.error ? "dialog-error" : "checkmark"
implicitWidth: Kirigami.Units.iconSizes.small
implicitHeight: implicitWidth
}
Controls.Label {
Layout.maximumWidth: Kirigami.Units.gridUnit * 24
wrapMode: Text.Wrap
text: modelData.text
}
Controls.ToolButton {
visible: modelData.link !== ""
icon.name: "internet-services"
text: "Open"
onClicked: Qt.openUrlExternally(modelData.link)
}
}
}
Controls.Button {
visible: backend.doneLog.length > 0
text: "Clear"
icon.name: "edit-clear-history"
flat: true
onClicked: backend.clearDoneLog()
}
}
footer: Controls.Label {
@@ -1852,8 +2138,8 @@ Kirigami.ApplicationWindow {
opacity: 0.7
text: "Export leaves out the ranges you deleted, compresses the rest and adds a manifest and "
+ "checksums, in " + backend.exportsDir + ". It runs at the lowest priority "
+ "and takes a few minutes per round; the headset stays awake until it's done. Nothing is "
+ "uploaded until you press Upload on the Upload page."
+ "and takes a few minutes per round, one at a time; the headset stays awake until they're "
+ "done. Nothing is uploaded unless you press Export and upload here, or Upload on the Upload page."
}
}
}
@@ -1879,7 +2165,7 @@ Kirigami.ApplicationWindow {
readonly property string blocked: {
if (session === "") return "Choose an export"
if (chosen && chosen.export_stale) return "Export the session again first"
if (backend.exporting) return "Wait for the export to finish"
if (root.exportPending(session)) return "Wait for its export to finish"
if (backend.uploading) return ""
if (backend.hubDryRun) return ""
if (!backend.uploadAllowedFor(session)) return "Contributions aren't open yet"
@@ -1892,6 +2178,14 @@ Kirigami.ApplicationWindow {
Connections {
target: backend
function onSessionsChanged() { uploadView.refreshInfo() }
// An upload started by itself (queued on the Export page): show it.
function onUploadChanged() {
const i = backend.uploading ? exportBox.indexOfValue(backend.upload.session) : -1
if (i >= 0 && i !== exportBox.currentIndex) {
exportBox.currentIndex = i
exportBox.keep = backend.upload.session
}
}
}
Component.onCompleted: {
refreshInfo()
@@ -1971,8 +2265,19 @@ Kirigami.ApplicationWindow {
textRole: "label"
valueRole: "id"
enabled: !backend.uploading
Component.onCompleted: currentIndex = Math.max(0, indexOfValue(root.chosenSession))
onActivated: root.chosenSession = currentValue
// An export that finishes meanwhile adds a row (newest first): stay on the
// same export, the uploading one while an upload runs.
property string keep: ""
Component.onCompleted: {
currentIndex = Math.max(0, indexOfValue(root.chosenSession))
keep = currentValue || ""
}
onActivated: root.chosenSession = keep = currentValue
onModelChanged: Qt.callLater(() => {
const i = indexOfValue(backend.uploading ? backend.upload.session : keep)
currentIndex = i >= 0 ? i : (count > 0 ? Math.max(0, currentIndex) : -1)
keep = currentValue || ""
})
}
Controls.Label {
opacity: 0.7
@@ -2086,6 +2391,14 @@ Kirigami.ApplicationWindow {
+ "\">its page</a>. Upload checks the export, opens your pull request, then sends the files."
onLinkActivated: link => Qt.openUrlExternally(link)
}
Controls.Label {
Layout.fillWidth: true
visible: backend.uploadQueue.length + backend.uploadAfter.length > 0
wrapMode: Text.Wrap
opacity: 0.7
text: "Queued on the Export page to upload after this one, by themselves: "
+ root.plural(backend.uploadQueue.length + backend.uploadAfter.length, "export") + "."
}
RowLayout {
Controls.Button {
text: backend.uploading ? "Uploading…" : backend.hubDryRun ? "Upload (dry run)" : "Upload"
@@ -2192,6 +2505,22 @@ Kirigami.ApplicationWindow {
onTriggered: backend.copy(uploadView.upResult.pr_url)
}
}
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: !backend.uploading && uploadView.mine && uploadView.up.phase === "done"
&& !uploadView.upResult.dry_run && uploadView.chosen !== null && uploadView.chosen.freeable
type: Kirigami.MessageType.Information
text: "This round doesn't need to stay on the headset now. Deleting its local copy frees "
+ (uploadView.chosen ? uploadView.chosen.freeText : "") + "; your record of the upload "
+ "stays on the Review page."
actions: [
Kirigami.Action {
text: "Delete local copy"
icon.name: "edit-clear-all"
onTriggered: root.askFree(uploadView.chosen, null)
}
]
}
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: !backend.uploading && uploadView.mine && uploadView.up.phase === "failed"
+1 -1
View File
@@ -4,7 +4,7 @@
"intro_s": 4,
"between_s": 3,
"welcome": {"title": "Hand recorder", "seconds": 8, "text": "Follow the steps on this panel. Each one plays a short clip of what to do: copy it. In the clips the palm is light and the back of the hand is dark.|The button on the right side of the headset goes on when a step waits for you, and pauses while recording."},
"done": {"title": "All done", "seconds": 6, "text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."},
"done": {"title": "All done", "seconds": 6, "text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."},
"stopped": {"title": "Stopped", "seconds": 4, "text": "What was recorded so far is saved. Review it in the Hand recorder window."},
"objects": {"pencil": "pencil or pen", "phone": "phone", "cup": "cup", "keyboard": "keyboard", "mouse": "mouse", "gamepad": "gamepad", "small": "small thing"},
"sections": [
+1 -1
View File
@@ -5,7 +5,7 @@
"welcome": {"title": "Hand recorder", "seconds": 7,
"text": "Follow the steps on this panel. Each one shows what to do with your hands, and where. A bar shows the time left.|The button on the right side of the headset goes on when a step waits for you, and pauses while recording."},
"done": {"title": "All done", "seconds": 6,
"text": "Thank you! You can take off the headset.|Review your recordings in the Hand recorder window."},
"text": "Thank you! This recording session will close. Continue in the application.|Review your recordings in the Hand recorder window."},
"stopped": {"title": "Stopped", "seconds": 4,
"text": "What was recorded so far is saved. Review it in the Hand recorder window."},
"cue_names": {"flat": "Flat", "flat-back": "Backs", "spread": "Spread", "open": "Open", "fist": "Fist",
+133 -7
View File
@@ -4,6 +4,7 @@
Standard library only (no Qt, no NumPy), so the window and the command line share it.
The layout is hands/rec/DESIGN.md's "Files":
BASE/profile.json
BASE/contributions.json (finished uploads, kept after their sessions go: recorder/contributions.py)
BASE/sessions/<YYYYMMDD-HHMMSS>/session.json, calibration.json
BASE/sessions/<id>/takes/<NN>-<section>/sets.bin (sets-2.bin, ... after pauses),
prompts.jsonl, poses.jsonl, take.json
@@ -26,6 +27,7 @@ usage: takes.py [--base DIR] list | takes SESSION | export SESSION | sides SESSI
import argparse
import datetime
import errno
import hashlib
import json
import os
import re
@@ -35,11 +37,13 @@ import subprocess
import sys
import tempfile
import threading
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import sides # noqa: E402 (hands/rec/sides.py, next to this file)
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
from recorder import age # noqa: E402 (a debug session's recordings are encrypted)
from recorder import contributions # noqa: E402 (the log of uploads that outlives the sessions)
from recorder.common import tree_version # noqa: E402
import eventlog # noqa: E402 (hands/rec/eventlog.py: a debug session's panel and pointer decisions)
from recorder.files import (Cancelled, free_bytes, human_bytes, in_ranges, merge_ranges, read_json, # noqa: E402,F401
@@ -85,10 +89,16 @@ EXPORT_SCHEMA = 1
# someone is in VR makes the headset stutter).
ZSTD_ARGS = ["-10", "-T2", "-q", "-c"]
ZSTD_PATHS = ("/usr/bin/zstd", "/usr/local/bin/zstd", "/run/host/usr/bin/zstd")
# An export is about three quarters of the recording's size (2026-10-14's: 15 GB sessions, 11 GB
# exports). Under half of it free, it can't fit: export says so before it starts.
# An export is about three quarters of the recording's size (2026-10-17, the Frame's five
# exports: 0.72 to 0.76). Export starts only with room for EXPORT_RATIO_NEED of it plus
# EXPORT_RESERVE, and stops when the disk gets down to EXPORT_MIN_FREE: a full home disk breaks
# more than the export (SteamVR, Steam and the desktop write there too). Until 2026-10-17 it
# started with half free and ran until zstd failed on a full disk.
EXPORT_RATIO = 0.75
EXPORT_RATIO_MIN = 0.5
EXPORT_RATIO_NEED = 0.8
EXPORT_RESERVE = 2 * 1000 ** 3
EXPORT_MIN_FREE = 1.5e9 # as session.py's MIN_FREE for recording
ROOM_CHECK_S = 1.0
class ZstdStopped(RuntimeError):
@@ -100,6 +110,33 @@ class DiskFull(RuntimeError):
"""Not enough free space for the export, before it started or as it went."""
class NearlyFull(OSError):
"""The export stopped itself: the disk got down to EXPORT_MIN_FREE."""
def __init__(self, free):
super().__init__(errno.ENOSPC, f"only {human_bytes(free)} left on the disk")
def export_need(kept):
"""The free space an export of kept bytes of recording needs to start."""
return int(kept * EXPORT_RATIO_NEED + EXPORT_RESERVE)
_room_checked = [0.0] # when check_room last looked (monotonic s)
def check_room(path):
"""Raise NearlyFull when the disk holding path is down to EXPORT_MIN_FREE. Looks at most once
every ROOM_CHECK_S (a statvfs per set would cost more than it saves)."""
now = time.monotonic()
if now - _room_checked[0] < ROOM_CHECK_S:
return
_room_checked[0] = now
free = free_bytes(path)
if 0 < free < EXPORT_MIN_FREE:
raise NearlyFull(free)
def disk_full(e):
"""True if e (an export's error) or what it came from is a full disk or a quota."""
seen = set()
@@ -433,6 +470,41 @@ def take_index(take_dir):
return index
def export_sha(export_dir):
"""The export's identity, as hub.py records it: the SHA256 of its SHA256SUMS ("" if none)."""
try:
with open(os.path.join(export_dir, "SHA256SUMS"), "rb") as f:
return hashlib.sha256(f.read()).hexdigest()
except OSError:
return ""
def _epoch(iso):
try:
return datetime.datetime.fromisoformat(iso).timestamp()
except (TypeError, ValueError):
return None
def upload_state(meta, changed, export_dir):
"""A session's uploads, from its session.json (meta) and its files' newest change (changed,
tree_stat's): {"uploaded": a finished upload exists, "pr_url": the last one's, "freeable":
deleting the local copy loses nothing that wasn't uploaded}. That's when the recordings haven't
changed since the last finished upload, or (an upload without a readable time) the export
there now is the one uploaded."""
done = contributions.done_uploads(meta)
if not done:
return {"uploaded": False, "pr_url": "", "freeable": False}
last = done[-1]
when = _epoch(last.get("uploaded", ""))
if when is not None:
freeable = changed <= when
else:
sha = export_sha(export_dir)
freeable = bool(sha) and any(u.get("export_sha") == sha for u in done)
return {"uploaded": True, "pr_url": last.get("pr_url", ""), "freeable": freeable}
# ---------------------------------------------------------------- the store
class Store:
@@ -463,7 +535,8 @@ class Store:
# --- listing
def sessions(self):
"""Newest first: [{id, started, takes, bytes, lighting, status, dry_run, contributor, exported,
export_stale (changed since), export_bytes}]."""
export_stale (changed since), export_bytes, uploaded, pr_url, freeable}] (upload_state's
last three)."""
out = []
try:
names = os.listdir(self.sessions_dir)
@@ -484,7 +557,8 @@ class Store:
"bounty": ((meta.get("bounty") or {}).get("title") or "") if isinstance(meta.get("bounty"), dict) else "",
"debug": bool(meta.get("debug")), "testing": consent_kind(meta) == "testing",
"exported": bool(exported), "export_stale": bool(exported) and changed > exported,
"export_bytes": tree_bytes(self.export_dir(sid)) if exported else 0})
"export_bytes": tree_bytes(self.export_dir(sid)) if exported else 0,
**upload_state(meta, changed, self.export_dir(sid))})
return sorted(out, key=lambda s: s["id"], reverse=True)
def take_ids(self, session):
@@ -559,6 +633,53 @@ class Store:
if os.path.isdir(path):
shutil.rmtree(path)
# --- freeing space: uploaded sessions go, their record stays (recorder/contributions.py)
def log_uploads(self, session=None):
"""Copy finished uploads from session.json into the contributions log: one session's, or
every session's (uploads older recorders recorded only there). Returns how many were new."""
items = []
for sid in [session] if session else [s for s in os.listdir(self.sessions_dir)
if SESSION_RE.match(s)] if os.path.isdir(self.sessions_dir) else []:
meta = read_json(os.path.join(self.session_dir(sid), "session.json"))
export = self.export_dir(sid)
size = tree_bytes(export) if os.path.isdir(export) else 0
items += [contributions.entry(sid, meta, u, size if u.get("export_sha") == export_sha(export) else 0)
for u in contributions.done_uploads(meta)]
return contributions.add(self.base, items) if items else 0
def free_session(self, session):
"""Delete an uploaded session's recordings and export, keeping its uploads in the
contributions log. Refuses (ValueError) a session that isn't uploaded as it is now.
Returns the bytes freed."""
path = self.session_dir(session)
meta = read_json(os.path.join(path, "session.json"))
if not meta:
raise ValueError(f"no session {session}")
_, changed = tree_stat(path)
state = upload_state(meta, changed, self.export_dir(session))
if not state["freeable"]:
raise ValueError("This session isn't uploaded as it is now: upload it first, or delete it")
freed = tree_bytes(path) + (tree_bytes(self.export_dir(session)) if os.path.isdir(self.export_dir(session)) else 0)
self.log_uploads(session) # the record first: if it can't be written, nothing is deleted
if session not in contributions.sessions(self.base):
raise OSError(f"couldn't keep the record of {session} in {contributions.FILE}")
self.delete_session(session)
contributions.mark_deleted(self.base, session)
return freed
def freeable(self):
"""The sessions free_session would take: [{id, bytes (recordings + export)}], newest first."""
return [{"id": s["id"], "bytes": s["bytes"] + s["export_bytes"]} for s in self.sessions()
if s["freeable"] and s["status"] != "recording"]
def contributions(self):
"""The contributions log, newest first, each with "local": its session is still here."""
out = []
for c in reversed(contributions.load(self.base)):
local = bool(SESSION_RE.match(c["session"])) and os.path.isdir(os.path.join(self.sessions_dir, c["session"]))
out.append(dict(c, local=local))
return out
# --- export
def sides(self, session, swapped=None):
"""session.json's side camera decision (sides.py) plus each take's parts; with swapped
@@ -609,7 +730,7 @@ class Store:
shutil.rmtree(work)
kept = self.kept_bytes(session)
free = free_bytes(self.exports_dir) # 0: couldn't tell, so no check
if 0 < free < kept * EXPORT_RATIO_MIN:
if 0 < free < export_need(kept):
raise DiskFull(self.space_text(session, kept, "Not enough free space to export this session"))
os.makedirs(work)
try:
@@ -619,6 +740,9 @@ class Store:
os.replace(work, final)
except BaseException as e:
shutil.rmtree(work, ignore_errors=True)
if isinstance(e, NearlyFull):
raise DiskFull(self.space_text(session, kept, "The disk got nearly full, so the export "
"stopped and removed its half")) from e
if isinstance(e, Exception) and not isinstance(e, DiskFull) and disk_full(e):
raise DiskFull(self.space_text(session, kept, "The disk filled up while exporting")) from e
raise
@@ -632,7 +756,8 @@ class Store:
def space_text(self, session, kept, first):
"""The disk-full message: what's free, about what the export needs, and what to do."""
text = (f"{first}: {human_bytes(free_bytes(self.exports_dir))} free, and the export needs about "
f"{human_bytes(kept * EXPORT_RATIO)}. Free some space and export again.")
f"{human_bytes(export_need(kept))} to start. Free some space and export again. Rounds that "
"are already uploaded can be deleted on the Review page, and your record of them stays.")
old = tree_bytes(self.export_dir(session)) if os.path.isdir(self.export_dir(session)) else 0
if old:
text += f" Its old export ({human_bytes(old)}) stays until the new one is done: deleting it frees that too."
@@ -768,6 +893,7 @@ class Store:
for i in keep:
if cancelled():
raise Cancelled()
check_room(dest)
record = index.read_raw(i)
proc.stdin.write(record)
raw += len(record)
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""Exports queue up and run one at a time; one set to upload uploads by itself once it's
exported, after any upload before it (recorder/window.py queueExport, 2026-10-17: "users can
queue exports, and trigger an auto upload when the export is finished"). Needs PySide6 (the dev
container); skipped without it.
python3 hands/rec/tests/test_export_queue.py
"""
import os
import shutil
import sys
import tempfile
import unittest
from unittest import mock
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.dirname(HERE))
import takes # noqa: E402
from test_validate import make_session # noqa: E402
A, B, C = "20261003-184200", "20261003-190000", "20261003-193000"
class ExportQueueTest(unittest.TestCase):
def setUp(self):
try:
import ft_handrec
from recorder import window
except ImportError as e:
self.skipTest(f"no PySide6: {e}")
self.base = tempfile.mkdtemp(prefix="handrec-export-queue-test-")
for sid in (A, B, C):
make_session(self.base, sid=sid)
self.store = takes.Store(self.base)
self.backend = ft_handrec.Backend(self.store, {"dry_run": True}, hub_dry_run=True, ignore_cameras=True)
self.backend._stay_awake = lambda why, on: None
# Nothing runs: exports start and end by hand (finish), uploads start a fake hub.py.
self.started = [] # sessions whose export started, in order
self.sent = [] # sessions whose upload started, in order
real_start = self.backend._start_export
def start_export(session, keep_notes):
real_start(session, keep_notes)
self.started.append(session)
self.backend._start_export = start_export
popen = mock.patch.object(window.subprocess, "Popen", side_effect=self.popen)
popen.start()
self.addCleanup(popen.stop)
thread = mock.patch.object(self.backend, "_thread")
thread.start()
self.addCleanup(thread.stop)
def tearDown(self):
shutil.rmtree(self.base, ignore_errors=True)
def popen(self, argv, **kw):
self.sent.append(argv[argv.index("upload") + 1])
return mock.Mock()
def finish_export(self, error=""):
self.backend._on_export_finished("" if error else self.base, error)
def finish_upload(self, ok=True):
self.backend._on_upload_finished({"done": {"dry_run": True}} if ok else {"error": {"kind": "hub", "text": "no"}})
def test_exports_run_in_turn(self):
for sid in (A, B, C):
self.backend.queueExport(sid, False, False)
self.assertEqual(self.started, [A])
self.assertEqual([q["session"] for q in self.backend.exportQueue], [B, C])
self.finish_export()
self.assertEqual(self.started, [A, B])
self.finish_export("disk full") # a failed one doesn't hold up the rest
self.assertEqual(self.started, [A, B, C])
self.finish_export()
self.assertFalse(self.backend.exporting)
self.assertEqual(self.sent, []) # none was set to upload
def test_queued_once(self):
self.backend.queueExport(A, False, False)
self.backend.queueExport(B, False, False)
self.backend.queueExport(B, False, True) # again, now to upload: the same place in the queue
self.backend.queueExport(A, False, False)
self.assertEqual(self.backend.exportQueue, [{"session": B, "upload": True}])
self.assertEqual(self.started, [A])
def test_upload_after_export(self):
self.backend.queueExport(A, False, True)
self.assertEqual(self.sent, [])
self.finish_export()
self.assertEqual(self.sent, [A])
self.assertEqual(self.backend.upload["session"], A)
def test_no_upload_after_a_failed_or_cancelled_export(self):
self.backend.queueExport(A, False, True)
self.finish_export("disk full")
self.backend.queueExport(B, False, True)
self.backend.cancelExport()
self.finish_export("cancelled")
self.assertEqual(self.sent, [])
self.assertTrue(self.backend.doneLog[0]["error"])
def test_uploads_wait_their_turn(self):
for sid in (A, B, C):
self.backend.queueExport(sid, False, sid != B)
self.finish_export() # A exported: it uploads
self.finish_export() # B exported: not set to upload
self.finish_export() # C exported: waits for A's upload
self.assertEqual(self.sent, [A])
self.assertEqual(self.backend.uploadQueue, [C])
self.finish_upload(ok=False) # a failed upload doesn't hold up the next
self.assertEqual(self.sent, [A, C])
self.finish_upload()
self.assertFalse(self.backend.uploading)
self.assertEqual(self.backend.uploadQueue, [])
def test_upload_after_can_change(self):
self.backend.queueExport(A, False, False)
self.backend.queueExport(B, False, True)
self.backend.setUploadAfter(A, True)
self.backend.setUploadAfter(B, False)
self.finish_export()
self.finish_upload()
self.finish_export()
self.assertEqual(self.sent, [A])
def test_unqueue(self):
self.backend.queueExport(A, False, False)
self.backend.queueExport(B, False, True)
self.backend.unqueueExport(B)
self.finish_export()
self.assertEqual(self.started, [A])
self.assertEqual(self.sent, [])
def test_queued_session_isnt_deleted(self):
self.backend.queueExport(A, False, False)
self.backend.queueExport(B, False, False)
self.backend.deleteSession(B)
self.assertTrue(os.path.isdir(os.path.join(self.base, "sessions", B)))
def test_no_second_upload_of_the_same_export(self):
self.backend.queueExport(A, False, True)
with mock.patch.object(self.backend, "uploadInfo", return_value={"previous": {"export_sha": "x"}, "uploads": 1}):
self.finish_export()
self.assertEqual(self.sent, [])
self.assertIn("uploaded before", self.backend.doneLog[-1]["text"])
if __name__ == "__main__":
unittest.main()
+308
View File
@@ -0,0 +1,308 @@
#!/usr/bin/env python3
"""Tests for freeing space (2026-10-17): the contributions log (recorder/contributions.py) that
outlives deleted sessions, Store.free_session and its refusals, the export's tighter space check
and its stop before the disk fills, and hub.py adding each finished upload to the log.
python3 hands/rec/tests/test_free_space.py (anywhere with zstd)
"""
import datetime
import os
import shutil
import sys
import tempfile
import time
import unittest
from unittest import mock
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.dirname(HERE))
import takes # noqa: E402
import hub # noqa: E402
from recorder import contributions # noqa: E402
from test_validate import make_session, SESSION # noqa: E402
PR = "https://huggingface.co/datasets/DeeJanuz/frametop-hands/discussions/12"
def iso(t):
return datetime.datetime.fromtimestamp(t).astimezone().isoformat(timespec="seconds")
class LogTest(unittest.TestCase):
def setUp(self):
self.base = tempfile.mkdtemp(prefix="handrec-log-test-")
def tearDown(self):
shutil.rmtree(self.base, ignore_errors=True)
def entry(self, **kw):
c = {"session": SESSION, "repo": "a/b", "pr_url": PR, "pr_num": 12, "uploaded": "2026-10-15T13:04:38+00:00",
"export_sha": "s", "export_bytes": 0, "bounty": {}, "kind": "public", "debug": False, "started": "",
"minutes": None, "local_deleted": ""}
c.update(kw)
return c
def test_add_once(self):
self.assertEqual(contributions.add(self.base, [self.entry()]), 1)
self.assertEqual(contributions.add(self.base, [self.entry()]), 0)
self.assertEqual(len(contributions.load(self.base)), 1)
def test_add_fills_gaps_only(self):
contributions.add(self.base, [self.entry()])
contributions.add(self.base, [self.entry(export_bytes=99, pr_url="other")])
c, = contributions.load(self.base)
self.assertEqual(c["export_bytes"], 99)
self.assertEqual(c["pr_url"], PR)
def test_two_pull_requests_two_entries(self):
contributions.add(self.base, [self.entry(), self.entry(pr_num=13, pr_url=PR[:-2] + "13")])
self.assertEqual(len(contributions.load(self.base)), 2)
def test_mark_deleted(self):
contributions.add(self.base, [self.entry()])
self.assertTrue(contributions.mark_deleted(self.base, SESSION, "then"))
self.assertEqual(contributions.load(self.base)[0]["local_deleted"], "then")
self.assertFalse(contributions.mark_deleted(self.base, "20990101-000000"))
def test_unreadable_log(self):
with open(contributions.path(self.base), "w") as f:
f.write("{not json")
self.assertEqual(contributions.load(self.base), [])
def test_done_uploads(self):
meta = {"uploads": [{"status": "started", "pr_url": PR}, {"status": "done", "pr_url": PR},
{"pr_url": PR, "uploaded": "old"}, {"status": "done", "pr_url": ""}, "junk"]}
self.assertEqual(len(contributions.done_uploads(meta)), 2)
class FreeTest(unittest.TestCase):
def setUp(self):
self.base = tempfile.mkdtemp(prefix="handrec-free-test-")
make_session(self.base)
self.store = takes.Store(self.base)
self.sdir = self.store.session_dir(SESSION)
self.meta_path = os.path.join(self.sdir, "session.json")
def tearDown(self):
shutil.rmtree(self.base, ignore_errors=True)
def upload(self, **kw):
record = {"repo": "DeeJanuz/frametop-hands", "pr_url": PR, "pr_num": 12, "uploaded": iso(time.time() + 5),
"export_sha": "x", "status": "done"}
record.update(kw)
meta = takes.read_json(self.meta_path)
meta.setdefault("uploads", []).append(record)
meta["bounty"] = {"id": "index-taps", "title": "Index taps"}
st = os.stat(self.meta_path)
takes.write_json(self.meta_path, meta)
os.utime(self.meta_path, ns=(st.st_atime_ns, st.st_mtime_ns)) # as record_upload keeps it
def row(self):
return next(s for s in self.store.sessions() if s["id"] == SESSION)
def test_not_uploaded(self):
self.assertFalse(self.row()["freeable"])
with self.assertRaises(ValueError):
self.store.free_session(SESSION)
self.assertTrue(os.path.isdir(self.sdir))
def test_started_upload_isnt_enough(self):
self.upload(status="started")
self.assertFalse(self.row()["uploaded"])
self.assertFalse(self.row()["freeable"])
def test_free_uploaded(self):
self.upload()
os.makedirs(self.store.export_dir(SESSION))
row = self.row()
self.assertTrue(row["uploaded"] and row["freeable"])
self.assertEqual(row["pr_url"], PR)
self.assertEqual([f["id"] for f in self.store.freeable()], [SESSION])
self.assertGreater(self.store.free_session(SESSION), 0)
self.assertFalse(os.path.exists(self.sdir))
self.assertFalse(os.path.exists(self.store.export_dir(SESSION)))
c, = self.store.contributions()
self.assertEqual((c["session"], c["pr_url"], c["bounty"]["id"], c["local"]), (SESSION, PR, "index-taps", False))
self.assertTrue(c["local_deleted"])
self.assertEqual(self.store.freeable(), [])
def test_changed_since_upload(self):
self.upload(uploaded=iso(time.time() - 3600))
self.assertTrue(self.row()["uploaded"])
self.assertFalse(self.row()["freeable"])
with self.assertRaises(ValueError):
self.store.free_session(SESSION)
self.assertTrue(os.path.isdir(self.sdir))
def test_no_upload_time_uses_the_export(self):
path = self.store.export(SESSION, low_priority=False)
sha = takes.export_sha(path)
self.upload(uploaded="", export_sha="other")
self.assertFalse(self.row()["freeable"])
self.upload(uploaded="", export_sha=sha, pr_num=13)
self.assertTrue(self.row()["freeable"])
def test_record_first(self):
"""The log can't be written: nothing is deleted."""
self.upload()
with mock.patch.object(contributions, "add", side_effect=OSError(28, "No space left on device")):
with self.assertRaises(OSError):
self.store.free_session(SESSION)
self.assertTrue(os.path.isdir(self.sdir))
def test_backfill(self):
self.upload()
self.assertEqual(self.store.log_uploads(), 1)
self.assertEqual(self.store.log_uploads(), 0)
self.assertTrue(self.store.contributions()[0]["local"])
def test_bad_ids(self):
for sid in ("../x", "", "20261002-101500/.."):
with self.assertRaises(ValueError):
self.store.free_session(sid)
class ExportRoomTest(unittest.TestCase):
def setUp(self):
if not takes.find_zstd():
self.skipTest("no zstd")
self.base = tempfile.mkdtemp(prefix="handrec-room-test-")
make_session(self.base)
self.store = takes.Store(self.base)
self.partial = self.store.export_dir(SESSION) + ".partial"
def tearDown(self):
shutil.rmtree(self.base, ignore_errors=True)
takes._room_checked[0] = 0.0
def test_needs_the_export_and_a_reserve(self):
kept = self.store.kept_bytes(SESSION)
self.assertEqual(takes.export_need(kept), int(kept * takes.EXPORT_RATIO_NEED + takes.EXPORT_RESERVE))
with mock.patch.object(takes, "free_bytes", return_value=takes.export_need(kept) - 1):
with self.assertRaises(takes.DiskFull) as cm:
self.store.export(SESSION, low_priority=False)
self.assertIn("already uploaded can be deleted", str(cm.exception))
self.assertFalse(os.path.exists(self.partial))
def test_stops_before_full(self):
"""The disk gets down to EXPORT_MIN_FREE as it goes: the export stops and cleans up."""
real = takes.free_bytes
takes._room_checked[0] = 0.0
def free(path):
return 10 ** 12 if path == self.store.exports_dir else int(takes.EXPORT_MIN_FREE) - 1
with mock.patch.object(takes, "free_bytes", side_effect=free):
with self.assertRaises(takes.DiskFull) as cm:
self.store.export(SESSION, low_priority=False)
self.assertTrue(str(cm.exception).startswith("The disk got nearly full, so the export stopped"))
self.assertFalse(os.path.exists(self.partial))
self.assertIs(takes.free_bytes, real)
class HubLogTest(unittest.TestCase):
"""hub.upload adds the finished upload to the contributions log."""
def setUp(self):
if not takes.find_zstd():
self.skipTest("no zstd")
self.base = tempfile.mkdtemp(prefix="handrec-hublog-test-")
make_session(self.base)
self.store = takes.Store(self.base)
self.store.export(SESSION, low_priority=False)
def tearDown(self):
shutil.rmtree(self.base, ignore_errors=True)
def fake_push(self, hf, repo, export, path_in_repo, message, description, meta_path, sha, nfiles, nbytes, tell,
write_json, human_bytes):
record = {"repo": repo, "pr_url": PR, "pr_num": 12, "started": hub.now_iso(), "export_sha": sha,
"uploaded": hub.now_iso(), "status": "done"}
hub.rhub.record_upload(meta_path, record, write_json)
return mock.Mock(num=12), {"name": "someone"}, record
def test_logged(self):
with mock.patch.object(hub, "upload_allowed", return_value=True), \
mock.patch.object(hub, "_hf", return_value=None), \
mock.patch.object(hub.rhub, "push", side_effect=self.fake_push):
result = hub.upload(self.store, SESSION)
self.assertEqual(result["pr_url"], PR)
c, = contributions.load(self.base)
self.assertEqual((c["session"], c["pr_num"], c["kind"]), (SESSION, 12, "public"))
self.assertGreater(c["export_bytes"], 0)
def test_dry_run_not_logged(self):
hub.upload(self.store, SESSION, dry_run=True)
self.assertEqual(contributions.load(self.base), [])
class BackendTest(unittest.TestCase):
"""The window's side: what can be freed, the free slots, the testing session's count from the
log, and the backfill when the window opens. Needs PySide6 (the dev container)."""
def setUp(self):
try:
import ft_handrec
except ImportError as e:
self.skipTest(f"no PySide6: {e}")
self.ft = ft_handrec
self.base = tempfile.mkdtemp(prefix="handrec-free-backend-test-")
make_session(self.base)
self.store = takes.Store(self.base)
meta_path = os.path.join(self.store.session_dir(SESSION), "session.json")
meta = takes.read_json(meta_path)
meta["debug"] = {"use": "testing"}
meta["status"] = "done"
meta["uploads"] = [{"repo": "DeeJanuz/frametop-hands-test", "pr_url": PR, "pr_num": 3, "status": "done",
"uploaded": iso(time.time() + 5), "export_sha": "x"}]
st = os.stat(meta_path)
takes.write_json(meta_path, meta)
os.utime(meta_path, ns=(st.st_atime_ns, st.st_mtime_ns))
os.environ["FT_HANDREC_FRAMETOP_HANDS"] = "1"
self.addCleanup(os.environ.pop, "FT_HANDREC_FRAMETOP_HANDS", None)
self.backend = ft_handrec.Backend(self.store, {"dry_run": True}, ignore_cameras=True)
self.messages = []
self.backend.message.connect(lambda text, error: self.messages.append((text, error)))
def tearDown(self):
shutil.rmtree(self.base, ignore_errors=True)
def test_backfilled_on_open(self):
self.assertEqual([c["session"] for c in contributions.load(self.base)], [SESSION])
def test_free_all(self):
self.assertEqual(self.backend.freeableCount, 1)
self.assertTrue(self.backend.freeableText)
row, = self.backend.sessions
self.assertTrue(row["freeable"])
self.assertEqual(row["uploadText"], "uploaded")
self.backend.freeAllUploaded()
self.assertFalse(os.path.exists(self.store.session_dir(SESSION)))
self.assertEqual(self.backend.freeableCount, 0)
self.assertEqual(self.backend.freeableText, "")
self.assertFalse(self.messages[-1][1], self.messages)
c, = self.backend.contributions
self.assertEqual((c["title"], c["local"], c["testing"]), ("Testing session", False, True))
self.assertTrue(c["deletedText"])
debug = next(b for b in self.backend.bounties if b["id"] == self.ft.DEBUG_ID)
self.assertEqual(debug["done"], 1) # counted from the log once the session is gone
def test_free_one_refuses_unuploaded(self):
make_session(self.base, sid="20261003-184200")
self.backend.freeSession("20261003-184200")
self.assertTrue(self.messages[-1][1])
self.assertTrue(os.path.isdir(self.store.session_dir("20261003-184200")))
def test_busy_session_not_offered(self):
self.backend._upload_proc = object()
self.backend._upload = {"session": SESSION}
self.assertEqual(self.backend.freeableCount, 0)
self.backend.freeSession(SESSION)
self.assertTrue(os.path.isdir(self.store.session_dir(SESSION)))
self.backend._upload_proc = None
def test_round_needs_room_for_its_export(self):
self.assertGreaterEqual(self.ft.ROUND_BYTES, self.ft.RECORD_BYTES + takes.export_need(self.ft.RECORD_BYTES))
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""An export and an upload of different sessions go on together; the same session waits
(recorder/window.py exportSession and startUpload, 2026-10-17: an upload was refused while
any other session exported). Needs PySide6 (the dev container); skipped without it.
python3 hands/rec/tests/test_upload_during_export.py
"""
import os
import shutil
import sys
import tempfile
import threading
import unittest
from unittest import mock
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.dirname(HERE))
import takes # noqa: E402
from test_validate import make_session # noqa: E402
A, B = "20261003-184200", "20261003-190000"
class UploadDuringExportTest(unittest.TestCase):
def setUp(self):
try:
import ft_handrec
from recorder import window
except ImportError as e:
self.skipTest(f"no PySide6: {e}")
self.window = window
self.base = tempfile.mkdtemp(prefix="handrec-upload-export-test-")
make_session(self.base, sid=A)
make_session(self.base, sid=B)
self.store = takes.Store(self.base)
self.backend = ft_handrec.Backend(self.store, {"dry_run": True}, hub_dry_run=True, ignore_cameras=True)
self.backend._stay_awake = lambda why, on: None
def tearDown(self):
shutil.rmtree(self.base, ignore_errors=True)
def exporting(self, session):
self.backend._export_cancel = threading.Event()
self.backend._export_session = session
def uploading(self, session):
self.backend._upload_proc = object()
self.backend._upload = {"session": session}
def upload(self, session):
"""startUpload(session): whether it started hub.py."""
with mock.patch.object(self.window.subprocess, "Popen") as popen, \
mock.patch.object(self.backend, "_thread"):
self.backend.startUpload(session, False)
return popen.called
def export(self, session):
"""exportSession(session): whether it started the export."""
with mock.patch.object(self.backend, "_thread") as thread:
self.backend.exportSession(session, False)
return thread.called
def test_upload_while_another_exports(self):
self.exporting(A)
self.assertTrue(self.upload(B))
self.assertTrue(self.backend.uploading)
def test_upload_waits_for_its_own_export(self):
self.exporting(A)
self.assertFalse(self.upload(A))
def test_export_while_another_uploads(self):
self.uploading(B)
self.assertTrue(self.export(A))
self.assertEqual(self.backend.exportSessionId, A)
def test_no_export_of_the_uploading_session(self):
self.uploading(B)
self.assertFalse(self.export(B))
def test_one_export_at_a_time(self):
self.exporting(A)
self.assertFalse(self.export(B))
self.assertEqual(self.backend.exportQueue, [{"session": B, "upload": False}]) # it waits its turn
if __name__ == "__main__":
unittest.main()
+27 -6
View File
@@ -45,8 +45,9 @@
// HANDS_CAMERAS, HANDS_BRIGHT, HANDS_BRIGHT_ON, HANDS_BRIGHT_OFF, HANDS_COLOR_LEFT (which
// colour camera is passthrough_left: color_video0 or color_video3), HANDS_COLOR_CROP
// (subtract or none: tools/check_color.py tells both), HANDS_MISREAD_GUARD (0 or 1: the
// tracker's guards for fine-tuned landmark models, Tracker::set_misread_guard), HANDS_MODELS (as
// --models: a folder with palm.ncnn.* and hand.ncnn.*, e.g. fine-tuned ones), HANDS_REACQUIRE (seconds
// tracker's guards for fine-tuned landmark models, Tracker::set_misread_guard), HANDS_MODEL (latest,
// stock, or a version's name: models/catalog.py), HANDS_MODELS (as --models: a folder with palm.ncnn.*
// and hand.ncnn.*, used instead of HANDS_MODEL), HANDS_REACQUIRE (seconds
// the palm search aims at a lost hand, 0 off: Tracker::set_reacquire), HANDS_PAIR_SIDES (0 or 1: two
// hands are one left and one right, Tracker::set_pair_sides), HANDS_PINCH_DIP (0 or 1: pinches also
// begin on a sharp dip of the thumb-index distance, PinchParams::dip), HANDS_PINCH_FIST_CURL (1.2; 0 off:
@@ -69,6 +70,7 @@
#include <unistd.h>
#include <algorithm>
#include <cctype>
#include <cmath>
#include <ctime>
#include <memory>
@@ -456,10 +458,29 @@ int main(int argc, char **argv) {
std::fflush(stdout);
return true;
};
if (models.empty())
models = model_root + (!int8 && access((model_root + "student/hand.ncnn.param").c_str(), R_OK) == 0 &&
access((model_root + "student/palm.ncnn.param").c_str(), R_OK) == 0
? "student" : "ncnn");
// HANDS_MODEL (Input Settings' Hands page, models/catalog.py): latest (models/student, which
// hands/models.sh keeps current), stock (models/ncnn), or a downloaded version (models/versions/NAME).
// None of the fine-tuned ones has 8-bit files, so --int8 always takes the stock ones.
auto has_models = [](const std::string &dir) {
return access((dir + "/hand.ncnn.param").c_str(), R_OK) == 0 && access((dir + "/palm.ncnn.param").c_str(), R_OK) == 0;
};
if (models.empty()) {
const std::string pick = setting("HANDS_MODEL");
const bool name_ok = !pick.empty() && pick.size() <= 64 && std::isalnum((unsigned char)pick[0]) &&
pick.find_first_not_of("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._-") ==
std::string::npos;
if (pick == "stock" || pick == "off" || pick == "none") {
models = model_root + "ncnn";
} else if (name_ok && pick != "latest") {
if (!int8 && has_models(model_root + "versions/" + pick)) models = model_root + "versions/" + pick;
else if (!int8)
std::fprintf(stderr, "HANDS_MODEL=%s isn't downloaded (Frametop Input Settings > Hands, or "
"models/ft-models fetch hands %s): using the latest\n", pick.c_str(), pick.c_str());
} else if (!pick.empty() && pick != "latest") {
std::fprintf(stderr, "HANDS_MODEL=%s isn't a model name: using the latest\n", pick.c_str());
}
if (models.empty()) models = model_root + (!int8 && has_models(model_root + "student") ? "student" : "ncnn");
}
if (!load_calibration(calib, err) || !ring.open(ring_path.c_str(), err) || !nets.load(models, int8, err) ||
(publish && (!pub.open(err) || (gestures_on && !gestures.open(pinch, grip, err)))) ||
(!record.empty() && !start_recording(record, err))) {
+182 -3
View File
@@ -23,14 +23,17 @@ to the input relay over its control socket (@frametop_relay):
go to ~/.config/frametop-voice.json (voice/voice_settings.py); ft-voiced reads them again
within 2 s, and answers "status" on its control socket.
- Gaze: the pointer's gaze mode (@ft_pointer_helper "gaze") and the gaze service
(gaze/ft-gazed, @ft_gazed: status, forget, reload), with its eye tracker (SteamVR's or
our own) and eye bias (GAZE_TRACKER, GAZE_EYE in frametop.conf).
(gaze/ft-gazed, @ft_gazed: status, forget, reload), with its eye tracker and eye bias
(GAZE_TRACKER, GAZE_EYE in frametop.conf). The tracker is picked from a catalog (models/catalog.py,
gaze/tracker/models.json): SteamVR's, our classic one, or one of our eye nets (EYES_MODEL: latest,
which follows new versions, or a version), each with notes and benchmarks, downloaded on use.
- Hands: when hand tracking runs (HANDS_MODE: with the Frametop desktop, always with SteamVR,
or off; hands/handsmode.py on the host starts and stops its services), the hand cutouts'
margin (HANDS_MATTE_MARGIN, sent live to ft-screens: "cutouts margin DEG"), and three hand
inputs, each on its own (handsmode.METHODS): clicks at the gaze (POINTER_HANDS,
POINTER_GAZE_PINCH), the hand ray with gaze off (POINTER_HAND_RAY), and direct touch on
Frametop's panels (HANDS_NEAR, sent live to ft-screens: "handinput on|off").
Frametop's panels (HANDS_NEAR, sent live to ft-screens: "handinput on|off"), and the hand
models (HANDS_MODEL, from hands/models.json: latest, a version, or stock MediaPipe), the same way.
- Bluetooth: paired devices, and re-applying the Bluetooth LE fixes after pairing.
- A warning on every page when SteamVR won't load the ft_pointer driver (blocked after a
crash, disabled, or SteamVR in safe mode) or hasn't loaded it (@ft_pointer doesn't answer
@@ -71,6 +74,12 @@ GAZE_PROBE = os.path.join(REPO, "gaze", "probe", "ft-gazeprobe")
sys.path.insert(0, os.path.join(REPO, "hands"))
import handsmode # noqa: E402
HANDSMODE = os.path.join(REPO, "hands", "handsmode.py")
# The hand and eye model catalogs: versions, notes, benchmarks, downloads (standard library only).
sys.path.insert(0, os.path.join(REPO, "models"))
import catalog as model_catalog # noqa: E402
# How the pickers tag each entry: by status, and builtins by what they are.
MODEL_TAGS = {"recommended": "Recommended", "testing": "Testing", "older": "Older"}
MODEL_KIND_TAGS = {"stock": "Off the shelf", "valve": "Valve", "classic": "Built in", "custom": "Frametop"}
SCREENS = "\0ft_screens"
HANDS_MODES = {"frametop": "With Frametop: while the Frametop desktop runs",
"always": "Always: whenever SteamVR runs, in games and the SteamVR dashboard too",
@@ -454,6 +463,7 @@ class Backend(QObject):
captured = Signal(int, str) # code, name
capturedController = Signal(str, str) # button, label
shortcutCaptureChanged = Signal()
modelsChanged = Signal()
message = Signal(str, bool) # text, is error
def __init__(self):
@@ -485,6 +495,8 @@ class Backend(QObject):
self._hands = {} # handsmode.py's status, {} until it answers
self._hands_lock = threading.Lock() # one handsmode.py at a time
self._hands_refreshing = False
self._model_busy = {"hands": "", "eyes": ""} # what a model download is doing, per kind
self._model_views = {}
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
self.sock.bind("") # autobind an abstract address the relay can reply to
self.sock.setblocking(False)
@@ -1594,6 +1606,173 @@ class Backend(QObject):
self._send(f"cutouts margin {value}", SCREENS)
self.handsChanged.emit()
# --- models (hands and eyes: models/catalog.py) ---
def _model_view(self, kind):
"""The picker's view of a kind: the catalog's entries as options, a "latest" one first (it
follows new versions), which one the settings pick ("choice") and which version runs ("running")."""
try:
v = model_catalog.view(kind)
except (OSError, ValueError, model_catalog.CatalogError) as e:
return {"error": f"The {kind} model list didn't load: {e}", "options": [], "metrics": []}
conf = read_conf()
by = {m["name"]: m for m in v["models"]}
choice = v["setting"]
note = ""
if kind == "eyes" and self.gazeTracker == "steam":
choice = "steamvr"
elif "/" in choice:
note = f"EYES_MODEL is a folder of your own ({choice}); pick one here to replace it."
choice = ""
if kind == "hands" and (os.environ.get("FT_HANDS_MODELS") or conf.get("HANDS_MODELS")):
note = ("HANDS_MODELS in frametop.conf names a folder of its own, and ft-hands uses that instead "
"of what's picked here. Remove it to use this choice.")
running = v["latest"] if choice == "latest" else choice
if kind == "hands" and choice == "latest":
# hands/models.sh keeps latest in hands/models/student, on its own pin.
here = ""
try:
with open(os.path.join(REPO, "hands", "models", "student", "PIN")) as f:
here = next((ln.split()[1] for ln in f if ln.startswith("model ")), "")
except (OSError, IndexError):
pass
running = here or "stock"
if not here:
note = note or (f"This install has no fine-tuned models yet, so stock runs: {v['latest']} comes "
"when hand tracking next starts online (hands/models.sh).")
elif here != v["latest"]:
note = note or f"This install still has {here}: {v['latest']} comes when hand tracking next starts."
elif kind == "eyes" and choice == "latest":
running = v["resolved"]["name"] or "classic"
if not v["resolved"]["folder"]:
note = note or f"{v['latest']} isn't downloaded yet: the classic tracker runs until it is."
# One row a version. The latest version's row is "latest": picking it follows updates. Set by
# name in frametop.conf, it shows on that row as pinned (not following) until picked here.
pinned = choice == v["latest"]
if pinned:
choice = "latest"
setting = "HANDS_MODEL" if kind == "hands" else "EYES_MODEL"
note = note or (f"{setting} in frametop.conf names {v['latest']}, so it stays on {v['latest']} when a "
"newer version comes. Use this to follow updates.")
opts = []
for m in v["models"]:
tag = MODEL_TAGS.get(m["status"]) or MODEL_KIND_TAGS.get(m["kind"], "")
if m["name"] == v["latest"]:
opts.append(dict(m, name="latest", target=m["name"], follows=True,
tag=(tag + " · " if tag else "") + "Latest",
installed=True if kind == "hands" else m["installed"]))
else:
opts.append(dict(m, target=m["name"], follows=False, tag=tag))
for o in opts:
o["bench"] = {k: (None if val is None else float(val)) for k, val in o["bench"].items()}
checked = time.strftime("%H:%M", time.localtime(v["checked"])) if v["checked"] else ""
return {"options": opts, "metrics": v["metrics"], "byName": by, "latest": v["latest"],
"baseline": v["baseline"], "choice": choice, "pinned": pinned, "running": running, "note": note,
"source": v["source"], "checked": checked, "error": ""}
def _models_update(self):
self._model_views = {k: self._model_view(k) for k in ("hands", "eyes")}
self.modelsChanged.emit()
@Property("QVariantMap", notify=modelsChanged)
def handModels(self):
if "hands" not in self._model_views: # (no signal from inside a getter: a binding loop)
self._model_views["hands"] = self._model_view("hands")
return self._model_views["hands"]
@Property("QVariantMap", notify=modelsChanged)
def eyeModels(self):
if "eyes" not in self._model_views: # (no signal from inside a getter: a binding loop)
self._model_views["eyes"] = self._model_view("eyes")
return self._model_views["eyes"]
@Property("QVariantMap", notify=modelsChanged)
def modelBusy(self):
return self._model_busy
def _model_job(self, kind, work):
"""Run a download (or a catalog check) off the UI thread, one per kind at a time."""
if self._model_busy[kind]:
return
def job():
try:
work()
except (OSError, ValueError, model_catalog.CatalogError) as e:
self.message.emit(str(e), True)
finally:
self._model_busy[kind] = ""
self._models_update()
self._model_busy[kind] = "working…"
self.modelsChanged.emit()
threading.Thread(target=job, daemon=True).start()
def _busy(self, kind, text):
self._model_busy[kind] = text
self.modelsChanged.emit()
@Slot()
def reloadModels(self):
"""Read the catalogs and the downloads again (the pages call it when they open)."""
self._models_update()
@Slot(str)
def refreshModels(self, kind):
"""The kind's catalog from GitHub (this install's branch): new versions and notes."""
if kind not in model_catalog.SPECS:
return
def work():
self._busy(kind, "checking for new versions…")
ok, msg = model_catalog.refresh(kind)
self.message.emit(("Hand" if kind == "hands" else "Eye") + " models: " + msg, not ok)
self._model_job(kind, work)
@Slot(str, str)
def useModel(self, kind, name):
"""Pick a model: download it first if it needs that, then write the setting and apply it.
hands: HANDS_MODEL (latest, stock or a version), and hand tracking restarts to load it.
eyes: steamvr is GAZE_TRACKER=steam; anything else is our tracker (GAZE_TRACKER=own) with
EYES_MODEL (classic, latest or a version), which the gaze service applies (it restarts ft-eyes)."""
if kind not in model_catalog.SPECS or not (name == "latest" or model_catalog.NAME.fullmatch(name)):
return
view = self._model_views.get(kind) or self._model_view(kind)
target = view["latest"] if name == "latest" else name
e = view["byName"].get(target)
if e is None:
return
def work():
if not e["builtin"] and not (kind == "hands" and name == "latest"):
if not e["usable"]:
raise model_catalog.CatalogError(f"{target} needs a newer Frametop")
if not e["installed"]:
model_catalog.fetch(kind, target, progress=lambda t: self._busy(kind, t + "…"))
if kind == "hands":
write_conf_value("HANDS_MODEL", name)
r = host("systemctl", "--user", "try-restart", "frametop-hands.service")
running = r.returncode == 0 and host("systemctl", "--user", "is-active", "--quiet",
"frametop-hands.service").returncode == 0
self.message.emit(f"Hand model: {e['title']}" + (", hand tracking restarted" if running
else " (used when hand tracking starts)"), False)
self.handsChanged.emit()
elif name == "steamvr":
self._set_gaze("GAZE_TRACKER", "steam", "Eye tracker: SteamVR's")
else:
write_conf_value("EYES_MODEL", name)
self._set_gaze("GAZE_TRACKER", "own", f"Eye tracker: {e['title']}"
+ ("" if e["builtin"] else " (calibrate it once: each net keeps its own calibration)"))
self._model_job(kind, work)
@Slot(str, str)
def removeModel(self, kind, name):
"""Delete a downloaded version (not the one in use)."""
view = self._model_views.get(kind) or self._model_view(kind)
if kind not in model_catalog.SPECS or name in (view["running"], view["choice"]):
return
if model_catalog.remove(kind, name):
self.message.emit(f"Removed the download of {name}", False)
self._models_update()
# --- bluetooth ---
@Property("QVariantList", notify=bluetoothChanged)
def bluetooth(self):
+268 -56
View File
@@ -54,6 +54,212 @@ Kirigami.ApplicationWindow {
})[backend.driverBlock] || ""
}
// A model picker (Hands and Gaze pages): the catalog's versions (models/catalog.py) with their notes
// and benchmarks, set against the one running (viewing that one: against the baseline, stock or
// SteamVR). Choosing a version only shows it; "Use" downloads it if needed and switches to it.
// The catalog's texts come from GitHub, so they're shown as plain text.
component ModelPicker: ColumnLayout {
id: picker
property string kind
property var view: ({})
readonly property string busy: (backend.modelBusy || {})[kind] || ""
property string viewed: ""
readonly property var options: view.options || []
readonly property var opt: options.find(o => o.name === (viewed || view.choice)) || options[0] || ({})
readonly property string running: view.running || ""
readonly property string otherName: opt.target !== running ? running
: opt.target !== view.baseline ? (view.baseline || "") : ""
readonly property var other: otherName ? ((view.byName || {})[otherName] || null) : null
readonly property bool inUse: opt.name === view.choice && !view.pinned
spacing: Kirigami.Units.smallSpacing
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
function fmt(v, unit) {
if (v === undefined || v === null) return "–"
const n = Math.abs(v) >= 100 ? Math.round(v) : Math.round(v * 100) / 100
return n + (unit === "" || unit === "%" || unit === "°" ? unit : " " + unit)
}
function better(m, v, o) {
if (v === undefined || v === null || o === undefined || o === null || v === o) return 0
return (m.better === "lower" ? v < o : v > o) ? 1 : -1
}
Kirigami.InlineMessage {
Layout.fillWidth: true
visible: !!picker.view.error
type: Kirigami.MessageType.Error
text: picker.view.error || ""
}
Repeater {
model: picker.options
delegate: Controls.RadioButton {
required property var modelData
text: modelData.title + (modelData.tag ? " · " + modelData.tag : "")
+ (modelData.name === picker.view.choice ? (picker.view.pinned ? " · in use, not following updates"
: " · in use") : "")
+ (!modelData.usable ? " · needs a newer Frametop"
: !modelData.installed ? " · not downloaded" : "")
font.bold: modelData.name === picker.view.choice
checked: picker.opt.name === modelData.name
onToggled: if (checked) picker.viewed = modelData.name
}
}
Controls.Label {
visible: !!picker.view.note
text: picker.view.note || ""
textFormat: Text.PlainText
Layout.fillWidth: true
wrapMode: Text.WordWrap
color: Kirigami.Theme.neutralTextColor
font: Kirigami.Theme.smallFont
}
// The one being looked at: what it is, what to expect, what it costs, and its numbers.
Kirigami.Separator { Layout.fillWidth: true; visible: !!picker.opt.name }
Controls.Label {
text: (picker.opt.title || "") + (picker.opt.date ? " (" + picker.opt.date + ")" : "")
textFormat: Text.PlainText
font.bold: true
}
Controls.Label {
visible: picker.opt.follows === true
text: "The newest version. Picked, it follows updates: when a newer version ships, Frametop moves to it without asking. Any other version stays put when picked."
Layout.fillWidth: true
wrapMode: Text.WordWrap
opacity: 0.7
font: Kirigami.Theme.smallFont
}
Controls.Label {
text: picker.opt.summary || ""
visible: text !== ""
textFormat: Text.PlainText
Layout.fillWidth: true
wrapMode: Text.WordWrap
}
Controls.Label {
text: picker.opt.notes || ""
visible: text !== ""
textFormat: Text.PlainText
Layout.fillWidth: true
wrapMode: Text.WordWrap
opacity: 0.85
}
Controls.Label {
text: picker.opt.cost ? "Cost: " + picker.opt.cost : ""
visible: text !== ""
textFormat: Text.PlainText
Layout.fillWidth: true
wrapMode: Text.WordWrap
opacity: 0.7
font: Kirigami.Theme.smallFont
}
GridLayout {
visible: (picker.view.metrics || []).some(m => picker.opt.bench && picker.opt.bench[m.key] !== undefined)
columns: 3
columnSpacing: Kirigami.Units.largeSpacing
rowSpacing: 2
Controls.Label { text: "Benchmark"; font.bold: true; Layout.fillWidth: true }
Controls.Label { text: picker.opt.target || ""; textFormat: Text.PlainText; font.bold: true }
Controls.Label {
text: picker.other ? picker.otherName + (picker.otherName === picker.running ? " (in use)" : "") : ""
textFormat: Text.PlainText
font.bold: true
opacity: 0.7
}
Repeater {
model: picker.view.metrics || []
delegate: Controls.Label {
required property var modelData
required property int index
text: modelData.label
textFormat: Text.PlainText
Layout.row: index + 1
Layout.column: 0
Layout.fillWidth: true
wrapMode: Text.WordWrap
font: Kirigami.Theme.smallFont
Controls.ToolTip.visible: !!modelData.note && mhover.hovered
Controls.ToolTip.text: modelData.note || ""
HoverHandler { id: mhover }
}
}
Repeater {
model: picker.view.metrics || []
delegate: Controls.Label {
required property var modelData
required property int index
readonly property var v: (picker.opt.bench || {})[modelData.key]
readonly property int b: picker.other ? picker.better(modelData, v, (picker.other.bench || {})[modelData.key]) : 0
text: picker.fmt(v, modelData.unit)
Layout.row: index + 1
Layout.column: 1
color: b > 0 ? Kirigami.Theme.positiveTextColor : b < 0 ? Kirigami.Theme.negativeTextColor
: Kirigami.Theme.textColor
}
}
Repeater {
model: picker.view.metrics || []
delegate: Controls.Label {
required property var modelData
required property int index
text: picker.other ? picker.fmt((picker.other.bench || {})[modelData.key], modelData.unit) : ""
Layout.row: index + 1
Layout.column: 2
opacity: 0.7
}
}
}
Controls.Label {
visible: (picker.view.metrics || []).some(m => picker.opt.bench && picker.opt.bench[m.key] !== undefined)
text: "Green: better than " + (picker.otherName || "the other") + ", red: worse. Hover a row for how it was measured."
textFormat: Text.PlainText
opacity: 0.7
font: Kirigami.Theme.smallFont
}
RowLayout {
Controls.Button {
text: picker.inUse ? "In use"
: !picker.opt.installed ? "Download and use"
: "Use this"
icon.name: picker.inUse ? "checkmark" : !picker.opt.installed ? "download" : "dialog-ok-apply"
enabled: !picker.inUse && picker.opt.usable !== false && picker.busy === "" && !!picker.opt.name
onClicked: backend.useModel(picker.kind, picker.opt.name)
}
Controls.Button {
text: "Remove download"
icon.name: "edit-delete"
visible: !picker.opt.builtin && !picker.opt.follows && picker.opt.installed === true
&& picker.opt.target !== picker.running && picker.opt.name !== picker.view.choice
enabled: picker.busy === ""
onClicked: backend.removeModel(picker.kind, picker.opt.name)
}
Controls.Button {
text: "Check for new versions"
icon.name: "view-refresh"
enabled: picker.busy === ""
onClicked: backend.refreshModels(picker.kind)
}
Controls.BusyIndicator {
visible: picker.busy !== ""
running: visible
Layout.preferredHeight: Kirigami.Units.iconSizes.medium
Layout.preferredWidth: Layout.preferredHeight
}
Controls.Label {
visible: picker.busy !== ""
text: picker.busy
opacity: 0.7
font: Kirigami.Theme.smallFont
}
}
Controls.Label {
text: "List from " + (picker.view.source || "this install")
+ (picker.view.checked ? ", checked " + picker.view.checked : "")
opacity: 0.6
font: Kirigami.Theme.smallFont
}
}
function show(page) {
pageStack.clear()
pageStack.push(page)
@@ -282,49 +488,49 @@ Kirigami.ApplicationWindow {
onClicked: { backend.setMapping(bpage.deviceId, bpage.capturedCode, newAction.currentValue); bpage.capturedCode = -1 }
}
}
}
Kirigami.Heading {
visible: bpage.rows.length > 0
level: 3
text: "Current mappings"
}
Kirigami.Separator {
visible: bpage.rows.length > 0
Kirigami.FormData.isSection: true
Kirigami.FormData.label: "Current mappings"
}
Repeater {
model: bpage.rows
delegate: RowLayout {
required property var modelData
Layout.fillWidth: true
spacing: Kirigami.Units.largeSpacing
Controls.Label {
text: modelData.name
font.family: "monospace"
Layout.preferredWidth: Kirigami.Units.gridUnit * 10
Repeater {
model: bpage.rows
delegate: RowLayout {
required property var modelData
Layout.fillWidth: true
spacing: Kirigami.Units.largeSpacing
Controls.Label {
text: modelData.name
font.family: "monospace"
Layout.preferredWidth: Kirigami.Units.gridUnit * 10
}
Controls.ComboBox {
model: backend.actions
textRole: "text"
valueRole: "value"
Layout.preferredWidth: Kirigami.Units.gridUnit * 13
Component.onCompleted: currentIndex = indexOfValue(modelData.action)
onActivated: backend.setMapping(bpage.deviceId, modelData.code, currentValue)
}
Controls.Label {
text: modelData.custom ? (modelData.isDefault ? "changed" : "custom") : "default"
opacity: 0.6
Layout.preferredWidth: Kirigami.Units.gridUnit * 4
}
// Remove: your own binding goes away (the button passes through again).
// Reset: a changed built-in binding goes back to its default.
// Unbind: an untouched built-in binding is set to "Do nothing".
Controls.Button {
text: modelData.custom ? (modelData.isDefault ? "Reset" : "Remove") : "Unbind"
icon.name: modelData.custom ? (modelData.isDefault ? "edit-undo" : "edit-delete-remove")
: "list-remove"
onClicked: modelData.custom ? backend.removeMapping(bpage.deviceId, modelData.code)
: backend.unbind(bpage.deviceId, modelData.code)
}
Item { Layout.fillWidth: true }
}
Controls.ComboBox {
model: backend.actions
textRole: "text"
valueRole: "value"
Layout.preferredWidth: Kirigami.Units.gridUnit * 13
Component.onCompleted: currentIndex = indexOfValue(modelData.action)
onActivated: backend.setMapping(bpage.deviceId, modelData.code, currentValue)
}
Controls.Label {
text: modelData.custom ? (modelData.isDefault ? "changed" : "custom") : "default"
opacity: 0.6
Layout.preferredWidth: Kirigami.Units.gridUnit * 4
}
// Remove: your own binding goes away (the button passes through again).
// Reset: a changed built-in binding goes back to its default.
// Unbind: an untouched built-in binding is set to "Do nothing".
Controls.Button {
text: modelData.custom ? (modelData.isDefault ? "Reset" : "Remove") : "Unbind"
icon.name: modelData.custom ? (modelData.isDefault ? "edit-undo" : "edit-delete-remove")
: "list-remove"
onClicked: modelData.custom ? backend.removeMapping(bpage.deviceId, modelData.code)
: backend.unbind(bpage.deviceId, modelData.code)
}
Item { Layout.fillWidth: true }
}
}
@@ -1285,26 +1491,22 @@ Kirigami.ApplicationWindow {
opacity: 0.7
font: Kirigami.Theme.smallFont
}
RowLayout {
Kirigami.FormData.label: "Eye tracker:"
Controls.RadioButton {
text: "Own tracker (recommended)"
checked: backend.gazeTracker === "own"
onToggled: if (checked) backend.setGazeTracker("own")
}
Controls.RadioButton {
text: "SteamVR"
checked: backend.gazeTracker === "steam"
onToggled: if (checked) backend.setGazeTracker("steam")
}
// Whose eye tracking: SteamVR's (GAZE_TRACKER=steam), or ours (own) with the classic
// pupil finder or an eye net (EYES_MODEL: latest or a version).
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Eye tracker" }
ModelPicker {
Kirigami.FormData.label: "Tracker:"
kind: "eyes"
view: backend.eyeModels
Component.onCompleted: backend.reloadModels()
}
Controls.Label {
// Ours is the default once it's installed (GAZE_TRACKER=auto, gaze/ft-gazed).
visible: backend.gazeTracker === "steam" && !backend.gazeOwnInstalled
text: "Our own tracker is more accurate. Install it with gaze/tracker/install.sh in the "
+ "Frametop folder, in a terminal (asks for sudo)"
text: "Our own tracker (the classic one and the eye nets) needs its frame grabber. Install it "
+ "with gaze/tracker/install.sh in the Frametop folder, in a terminal (asks for sudo)"
+ (backend.gazeTrackerAuto ? "; gaze then uses it, and you calibrate it once."
: ", then pick Own tracker here and calibrate it once.")
: ", then pick it here and calibrate it once.")
Layout.maximumWidth: Kirigami.Units.gridUnit * 30
wrapMode: Text.WordWrap
opacity: 0.7
@@ -1354,6 +1556,7 @@ Kirigami.ApplicationWindow {
opacity: 0.7
font: Kirigami.Theme.smallFont
}
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Gaze pointer tuning" }
Repeater {
model: backend.gazeSettings
delegate: RowLayout {
@@ -1454,7 +1657,7 @@ Kirigami.ApplicationWindow {
+ "a pinch clicks where you look; pinch, hold and move the hand to nudge the pointer first. Pinch nudge "
+ "starts after: how far the hand moves before the pointer does. Pinch nudge speed: how far the pointer "
+ "moves for each degree of hand movement after that. Eye tracker: SteamVR's, or our own "
+ "(gaze/tracker), which keeps its own calibration. Eye bias: the gaze "
+ "(gaze/tracker), the classic pupil finder or an eye net; each keeps its own calibration. Eye bias: the gaze "
+ "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as "
+ "much, and Auto weights each by how far off it was at your recent nudges."
}
@@ -1532,6 +1735,15 @@ Kirigami.ApplicationWindow {
}
}
// Which hand models ft-hands runs (HANDS_MODEL): latest, a version, or stock MediaPipe.
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Hand model" }
ModelPicker {
Kirigami.FormData.label: "Model:"
kind: "hands"
view: backend.handModels
Component.onCompleted: backend.reloadModels()
}
// Three ways the hands work the desktop, each on its own (handsmode.METHODS).
Kirigami.Separator { Kirigami.FormData.isSection: true; Kirigami.FormData.label: "Hand input" }
Controls.Switch {
+7 -1
View File
@@ -28,6 +28,9 @@ relay_path = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "inp
tag = f"ft_keys_test_{os.getpid()}"
with open(relay_path) as f:
src = f.read().replace('"\\0frametop_relay"', f'"\\0{tag}_relay"')
screens_socket = "screens_sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM | socket.SOCK_NONBLOCK)"
assert src.count(screens_socket) == 1, "relay's screens socket constructor changed"
src = src.replace(screens_socket, "screens_sock = _test_screens_sender")
relay = importlib.util.module_from_spec(importlib.util.spec_from_loader("relay", loader=None))
relay.__file__ = os.path.abspath(relay_path)
sys.path.insert(0, os.path.dirname(os.path.abspath(relay_path))) # its game_pause and vrws
@@ -174,7 +177,10 @@ def probe(path):
relay.probe = probe
screens = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
# A connected pair holds all 16 startup releases even when the kernel limits unconnected
# datagram queues to 10 messages. The relay still uses real sendto(), including other peers.
relay._test_screens_sender, screens = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)
relay._test_screens_sender.setblocking(False)
screens.bind(relay.SCREENS)
screens.settimeout(0.05)
voice = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) # Frametop Voice's socket
+32 -8
View File
@@ -8,15 +8,17 @@
# distrobox the release brings, makes the release's container from its image, and builds nothing.
#
# Usage: ./install.sh [--yes] [--no-voice] [--no-hands] [--no-hand-recorder] [--no-eye-tracker]
# [--no-bluetooth | --bluetooth]
# --yes don't ask; installs voice dictation and gaze mode, and hand tracking, the Hand
# Recorder and our eye tracker if sudo can run without a password prompt; skips
# the Bluetooth fixes and the SteamVR restart
# [--no-eye-recorder] [--no-bluetooth | --bluetooth]
# --yes don't ask; installs voice dictation, gaze mode and the Eye Recorder, and hand
# tracking, the Hand Recorder and our eye tracker if sudo can run without a
# password prompt; skips the Bluetooth fixes and the SteamVR restart
# --no-voice don't offer voice dictation (Frametop Voice: it downloads about 320 MB)
# --no-hands don't offer hand tracking or the Hand Recorder
# --no-hand-recorder don't offer the Hand Recorder (a release's; a clone builds it with
# hands/rec/install.sh)
# --no-eye-tracker don't offer our own eye tracker
# --no-eye-tracker don't offer our own eye tracker (or the Eye Recorder, which needs it)
# --no-eye-recorder don't offer the Eye Recorder (a release's; a clone builds it with
# gaze/rec/install.sh)
# --no-bluetooth don't offer the Bluetooth fixes
# --bluetooth install the Bluetooth fixes without asking (with --yes: if sudo can run
# without a password prompt)
@@ -25,17 +27,18 @@ set -euo pipefail
root=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
. "$root/scripts/_env.sh"
assume_yes=0 bluetooth=1 eye_tracker=1 hands=1 hand_recorder=1 voice=1
assume_yes=0 bluetooth=1 eye_tracker=1 eye_recorder=1 hands=1 hand_recorder=1 voice=1
for arg in "$@"; do
case $arg in
--yes) assume_yes=1 ;;
--no-bluetooth) bluetooth=0 ;;
--bluetooth) bluetooth=2 ;;
--no-eye-tracker) eye_tracker=0 ;;
--no-eye-tracker) eye_tracker=0 eye_recorder=0 ;;
--no-eye-recorder) eye_recorder=0 ;;
--no-voice) voice=0 ;;
--no-hands) hands=0 hand_recorder=0 ;;
--no-hand-recorder) hand_recorder=0 ;;
-h|--help) sed -n '2,22p' "$0"; exit 0 ;;
-h|--help) sed -n '2,24p' "$0"; exit 0 ;;
*) echo "unknown option: $arg" >&2; exit 2 ;;
esac
done
@@ -224,6 +227,27 @@ else
echo "skipped: gaze mode uses SteamVR's eye tracker. Install ours later with: gaze/tracker/install.sh"
fi
# The Eye Recorder (gaze/rec), as the Hand Recorder: a release brings it built, so it's only the menu
# entry, which needs our eye tracker's frame grabber (step 11). An entry from the release before
# points into that release's folder: it's renewed without asking. A clone builds it with
# gaze/rec/install.sh (a few minutes), so it isn't offered here.
if [ "$FRAME_RELEASE" = 1 ]; then
recorder="the Eye Recorder (records your eyes for Frametop's eye tracking dataset)"
if [ "$eye_recorder" = 0 ]; then
echo "$recorder: skipped. Add it later with: gaze/rec/install.sh"
elif on_frame '[ -e ~/.local/share/applications/frametop-eyerec.desktop ]'; then
"$root/gaze/rec/install.sh" install >/dev/null && echo "$recorder: updated" ||
echo "$recorder: couldn't update it (above). Try again with: gaze/rec/install.sh"
elif ! on_frame 'systemctl is-enabled -q frametop-eyegrab.service 2>/dev/null'; then
echo "$recorder: skipped: it needs our eye tracker. Install both later with: gaze/tracker/install.sh, then gaze/rec/install.sh"
elif ask "Add the Eye Recorder to the app menu? It records your eyes for Frametop's eye tracking dataset, which trains our eye tracker; nothing leaves the headset until you press Upload, and uploads are encrypted." y; then
"$root/gaze/rec/install.sh" install ||
echo "$recorder: not installed (above). Try again later with: gaze/rec/install.sh"
else
echo "$recorder: skipped. Add it later with: gaze/rec/install.sh"
fi
fi
step "12/12 Bluetooth fixes (optional; they let LE mice and keyboards like the Swiftpoint Z3 reconnect)"
if [ "$bluetooth" = 2 ] && [ "$assume_yes" = 1 ] && ! sudo_quiet; then
echo "skipped: they need your password (sudo), and --yes doesn't ask. Install later with: setup/bluetooth/install.sh install"
+4 -1
View File
@@ -20,7 +20,7 @@ test-python:
set -uo pipefail
python3 -c 'import PySide6, numpy, cv2, sherpa_onnx' || { echo "python3 can't import PySide6, numpy, cv2, and sherpa_onnx"; exit 1; }
status=0
for t in input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py recorder/tests/*.py gaze/test/*.py gaze/rec/tests/*.py pack/test/*.py voice/test/*.py spotwatch/test/*.py; do
for t in models/test/*.py input/test/*.py hands/tests/test_*.py hands/rec/tests/*.py recorder/tests/*.py gaze/test/*.py gaze/rec/tests/*.py pack/test/*.py voice/test/*.py spotwatch/test/*.py; do
[ -f "$t" ] || continue
if python3 "$t" >/dev/null; then echo "$t: ok"; else echo "$t: FAILED"; status=1; fi
done
@@ -53,6 +53,9 @@ test-c:
g++ -std=c++17 -Wall -Wextra -Werror -I. "$t" -o "build/tests/$n"
"build/tests/$n" >/dev/null && echo "$t: ok"
done
g++ -std=c++17 -O2 -Wall -Wextra -Werror screens/tests/rounded-test.cpp \
-o build/tests/rounded-test
build/tests/rounded-test && echo "screens/tests/rounded-test.cpp: ok"
# shell scripts: syntax-gate everything that ships. No git: the image has
# none, and a mounted checkout can trip "dubious ownership" — find lists
+50
View File
@@ -0,0 +1,50 @@
# Model versions
Users pick which hand and eye models Frametop runs on Frametop Input Settings' Hands and Gaze pages.
Each page lists the versions in a catalog, with notes on what to expect and benchmark numbers set
against the one in use. `catalog.py` reads the catalogs, downloads versions and says what a setting
runs; `ft-models` is the same from a terminal.
| Kind | Catalog | Setting (frametop.conf) | Downloads go to | Files from |
|---|---|---|---|---|
| hands | `hands/models.json` | `HANDS_MODEL`: `latest`, `stock`, a version | `hands/models/versions/NAME` | `DeeJanuz/frametop-hands-models` |
| eyes | `gaze/tracker/models.json` | `EYES_MODEL`: `classic`, `latest`, a version; `GAZE_TRACKER=steam` for SteamVR's | `gaze/tracker/models/NAME` | `DeeJanuz/frametop-eyes-models` |
- **Latest** follows the catalog's `latest`, so it moves when a new version ships. A version picked
by name stays put. For hands, latest is `hands/models/student`, which `hands/models.sh` keeps on
`hands/models.pin` (older installs only read that pin). The catalog's latest must be the same
files: `models/test/test_catalog.py` checks it.
- **Where the list comes from.** Installs read their branch's catalog from GitHub (`main` for a
stable release, `experimental` for an experimental one or a clone on that branch), so a push
reaches them without a release: Input Settings' "Check for new versions", and the services when
they start (`ft-models auto`, at most every 15 minutes; `HANDS_MODELS_UPDATE=0` /
`EYES_MODELS_UPDATE=0` turn that off). Offline, or when GitHub's copy isn't valid, the copy in
the install is used.
- **Trust.** A catalog only names files. They come from the kind's Hugging Face repo, which is
fixed in `catalog.py` and never taken from a catalog. Each download is pinned to a commit and must
match its sha256, or it isn't kept. Catalog checks are strict: one bad entry and the whole
catalog is ignored. The texts are shown as plain text.
- **Who applies it.** ft-hands reads `HANDS_MODEL` at start, and Input Settings restarts hand
tracking after a change. ft-eyes reads `EYES_MODEL` at start, and the gaze service restarts it
when the setting changes. Each eye net keeps its own calibration (`net-NAME/`), so a switch asks
for one calibration.
## Shipping a version
1. Upload the version's folder to the kind's Hugging Face repo, next to the earlier ones (hands:
frame-hands' `model-repo/`, eyes: frame-eyes' `model-repo/`), with a NOTICE: how it was trained,
what it was tested on, the numbers.
2. Print its files' hashes from that staging folder:
`models/ft-models hashes eyes ~/Desktop/Projects/frame-eyes/model-repo eyes-v1`
3. Add an entry to the catalog: name, title, kind (`custom`), status (`testing` until it's the one
to recommend), date, summary, notes (what to expect against the current latest and the
baseline), cost, `bench` (the catalog's metrics, measured the same way as the other entries),
`format`, `rev` (the Hugging Face commit) and `files`. Set the old latest's status to `older`
when a new one is recommended.
4. To make it latest, set `latest`. For hands, also re-pin `hands/models.pin` with frame-hands'
`model-repo/pin.py VERSION REV CHECKOUT`.
5. Run `python3 models/test/test_catalog.py` and `models/ft-models fetch KIND NAME` (a real
download, checked), then push to experimental. Stable installs see it once it's on main.
Raise `format` (and `SPECS[kind]["format"]` in `catalog.py`) when a version needs a newer runtime.
Older installs then list the version as needing a newer Frametop instead of running it.
+465
View File
@@ -0,0 +1,465 @@
"""The model catalogs: which hand and eye models there are, their notes and benchmarks, and the
versions someone picked, downloaded and checked.
Two catalogs, one per kind, in the repo and on GitHub:
hands hands/models.json ft-hands' palm detector + hand landmark model pairs
eyes gaze/tracker/models.json the eye tracker: SteamVR's, our classic pupil finder, our eye nets
An install reads its branch's catalog from GitHub (main for a stable release, experimental for an
experimental one or a clone on that branch: models.sh's rule), so a new version and its notes reach
installs on a push, without a release. Offline, or when GitHub's copy isn't valid, the copy that came
with the install is used. A catalog only names versions: the files come from the kind's own Hugging
Face repo (HF_REPO here, never from the catalog), at the commit the entry pins, and every file must
match the sha256 it lists.
A catalog (JSON):
{"catalog": 1, "kind": "hands", "latest": "v13b-p1", "default": "latest", "baseline": "stock",
"metrics": [{"key", "label", "unit", "better": "higher"|"lower", "note"}],
"models": [{"name", "title", "kind", "status", "date", "summary", "notes", "cost", "bench": {key: n},
"format": 1, "rev": "<40 hex>", "files": {"name": "<sha256>"}}]}
kind custom (ours, trained on Frametop data), stock (off the shelf, unchanged), valve (SteamVR's),
classic (our hand-written tracker, no model)
status recommended, testing, older, builtin
latest the custom version "Latest" follows: it changes as new versions ship
default what an unset setting uses ("latest", or a builtin's name)
format what that version needs of the runtime (FORMATS here); newer ones show, but can't be used
Builtin entries (no rev or files) ship with Frametop or come from SteamVR.
Downloaded versions live in the kind's store (hands/models/versions/NAME, gaze/tracker/models/NAME),
with PIN.json, the entry they were fetched by. hands' "latest" is hands/models/student, which
hands/models.sh keeps on its own pin (models.pin); the catalog's latest must match it (the tests check).
Standard library only: the input settings, the gaze service (on the host) and ft-eyes (in the box)
import it. CLI: models/ft-models.
"""
import fcntl
import hashlib
import json
import os
import re
import shutil
import time
import urllib.error
import urllib.request
from pathlib import Path
REPO = Path(__file__).resolve().parents[1]
GITHUB = "Frametop/frametop"
CACHE = Path(os.environ.get("XDG_CACHE_HOME") or Path.home() / ".cache") / "frametop" / "models"
CATALOG_VERSION = 1 # the catalog layout this code reads
MAX_CATALOG = 128 * 1024 # bytes
MAX_FILE = 200 * 1024 * 1024 # bytes, per model file
NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,63}")
SHA = re.compile(r"[0-9a-f]{64}")
REV = re.compile(r"[0-9a-f]{40}")
KINDS_OF = ("custom", "stock", "valve", "classic")
STATUSES = ("recommended", "testing", "older", "builtin")
TEXT_MAX = {"title": 80, "date": 20, "summary": 400, "notes": 4000, "cost": 300}
METRIC_TEXT_MAX = {"key": 40, "label": 120, "unit": 20, "note": 300}
# Per kind: the catalog, the Hugging Face repo files come from, where versions go, the files a
# version may have (and must), files at the repo's top, and the newest version format we can run.
SPECS = {
"hands": {
"catalog": "hands/models.json",
"hf": "DeeJanuz/frametop-hands-models",
"store": "hands/models/versions",
"files": {"NOTICE", "hand.ncnn.bin", "hand.ncnn.param", "palm.ncnn.bin", "palm.ncnn.param"},
"required": {"hand.ncnn.bin", "hand.ncnn.param", "palm.ncnn.bin", "palm.ncnn.param"},
"top": {"LICENSE", "README.md"},
"format": 1,
"setting": "HANDS_MODEL",
"update": "HANDS_MODELS_UPDATE",
},
"eyes": {
"catalog": "gaze/tracker/models.json",
"hf": "DeeJanuz/frametop-eyes-models",
"store": "gaze/tracker/models",
"files": {"NOTICE", "model.json", "model.ncnn.bin", "model.ncnn.param"},
"required": {"model.json", "model.ncnn.bin", "model.ncnn.param"},
"top": {"LICENSE", "README.md"},
"format": 1,
"setting": "EYES_MODEL",
"update": "EYES_MODELS_UPDATE",
},
}
class CatalogError(ValueError):
pass
def _check_text(d, limits, where):
for key, n in limits.items():
v = d.get(key, "")
if not isinstance(v, str) or len(v) > n:
raise CatalogError(f"{where}: {key} must be text of at most {n} characters")
def validate(cat, kind):
"""Check a catalog strictly; return it, or raise CatalogError. Anything unexpected fails the
whole catalog: GitHub's copy is then ignored for the one that came with the install."""
spec = SPECS[kind]
if not isinstance(cat, dict):
raise CatalogError("not an object")
if cat.get("catalog") != CATALOG_VERSION:
raise CatalogError(f"catalog version {cat.get('catalog')!r}, this Frametop reads {CATALOG_VERSION}")
if cat.get("kind") != kind:
raise CatalogError(f"a catalog of {cat.get('kind')!r}, not {kind}")
metrics = cat.get("metrics", [])
if not isinstance(metrics, list) or len(metrics) > 40:
raise CatalogError("metrics must be a list (at most 40)")
keys = set()
for m in metrics:
if not isinstance(m, dict):
raise CatalogError("a metric isn't an object")
_check_text(m, METRIC_TEXT_MAX, "metric")
if not m.get("key") or not m.get("label") or m.get("better", "higher") not in ("higher", "lower"):
raise CatalogError(f"metric {m.get('key')!r}: needs key, label, better higher|lower")
keys.add(m["key"])
models = cat.get("models")
if not isinstance(models, list) or not models or len(models) > 200:
raise CatalogError("models must be a non-empty list (at most 200)")
names = set()
for e in models:
if not isinstance(e, dict):
raise CatalogError("a model isn't an object")
name = e.get("name")
if not isinstance(name, str) or not NAME.fullmatch(name) or name in ("latest",):
raise CatalogError(f"bad model name {name!r}")
if name in names:
raise CatalogError(f"{name} listed twice")
names.add(name)
_check_text(e, TEXT_MAX, name)
if e.get("kind") not in KINDS_OF or e.get("status") not in STATUSES:
raise CatalogError(f"{name}: kind must be one of {KINDS_OF}, status one of {STATUSES}")
bench = e.get("bench", {})
if not isinstance(bench, dict) or any(k not in keys or not (v is None or isinstance(v, (int, float)))
or isinstance(v, bool) for k, v in bench.items()):
raise CatalogError(f"{name}: bench must map listed metrics to numbers")
if e["status"] == "builtin":
if "rev" in e or "files" in e:
raise CatalogError(f"{name}: a builtin has no rev or files")
continue
if not isinstance(e.get("format"), int) or e["format"] < 1:
raise CatalogError(f"{name}: format must be a positive number")
if not isinstance(e.get("rev"), str) or not REV.fullmatch(e["rev"]):
raise CatalogError(f"{name}: rev must be a full commit hash")
files = e.get("files")
if not isinstance(files, dict) or any(not isinstance(v, str) or not SHA.fullmatch(v) for v in files.values()):
raise CatalogError(f"{name}: files must map names to sha256s")
allowed = spec["files"] | spec["top"]
if set(files) - allowed or spec["required"] - set(files):
raise CatalogError(f"{name}: files must be among {sorted(allowed)} and include {sorted(spec['required'])}")
for key in ("latest", "baseline"):
v = cat.get(key)
if v is not None and v not in names:
raise CatalogError(f"{key} {v!r} isn't listed")
latest = cat.get("latest")
if latest is not None and next(e for e in models if e["name"] == latest)["status"] == "builtin":
raise CatalogError("latest must be a downloadable version")
default = cat.get("default", "latest")
if default != "latest" and default not in names:
raise CatalogError(f"default {default!r} isn't listed")
if default == "latest" and latest is None:
raise CatalogError("default is latest, but there's no latest")
return cat
# --- where and which ---
def store(kind, tree=REPO):
return Path(tree) / SPECS[kind]["store"]
def channel(tree=REPO):
"""The branch whose catalog an install follows: main or experimental, or None (any other branch:
only the copy in the checkout)."""
tree = Path(tree)
rel = tree / ".frametop-release"
if rel.exists():
ch = ""
for line in rel.read_text().splitlines():
if line.startswith("CHANNEL="):
ch = line.split("=", 1)[1].strip()
return "experimental" if ch == "experimental" else "main"
git = tree / ".git"
try:
if git.is_file(): # a worktree: "gitdir: PATH"
git = Path(git.read_text().split(":", 1)[1].strip())
head = (git / "HEAD").read_text().strip()
except (OSError, IndexError):
return None
branch = head.removeprefix("ref: refs/heads/") if head.startswith("ref: refs/heads/") else None
return branch if branch in ("main", "experimental") else None
def read_conf(path=None):
path = Path(path) if path else Path.home() / ".config" / "frametop.conf"
conf = {}
try:
for line in path.read_text().splitlines():
line = line.split("#", 1)[0].strip()
if "=" in line:
k, v = line.split("=", 1)
conf[k.strip()] = v.strip().strip("\"'")
except OSError:
pass
return conf
def bundled(kind, tree=REPO):
path = Path(tree) / SPECS[kind]["catalog"]
return validate(json.loads(path.read_text()), kind)
def cached(kind):
"""GitHub's copy from the last refresh, if it's still valid: (catalog, branch, fetched unix)."""
try:
d = json.loads((CACHE / f"{kind}.json").read_text())
return validate(d["catalog"], kind), d["branch"], float(d["fetched"])
except (OSError, ValueError, KeyError, TypeError):
return None
def load(kind, tree=REPO):
"""The catalog to show: GitHub's copy for this install's branch (from the last refresh), else the
one in the checkout. Returns (catalog, source), source "github BRANCH" or "this install"."""
ch = channel(tree)
c = cached(kind)
if c and ch and c[1] == ch:
return c[0], f"github {ch}"
return bundled(kind, tree), "this install"
def refresh(kind, tree=REPO, timeout=15):
"""Fetch the branch's catalog from GitHub into the cache. Returns (ok, message)."""
ch = channel(tree)
if not ch:
return False, "this checkout isn't on main or experimental: only its own catalog"
url = f"https://raw.githubusercontent.com/{GITHUB}/{ch}/{SPECS[kind]['catalog']}"
try:
with urllib.request.urlopen(url, timeout=timeout) as r:
data = r.read(MAX_CATALOG + 1)
except (urllib.error.URLError, OSError) as e:
return False, f"couldn't reach GitHub ({getattr(e, 'reason', e)})"
if len(data) > MAX_CATALOG:
return False, "GitHub's catalog is too big; keeping this one"
try:
cat = validate(json.loads(data), kind)
except (ValueError, CatalogError) as e:
return False, f"GitHub's catalog isn't valid for this Frametop ({e}); keeping this one"
CACHE.mkdir(parents=True, exist_ok=True)
tmp = CACHE / f".{kind}.json.tmp"
tmp.write_text(json.dumps({"branch": ch, "fetched": time.time(), "catalog": cat}))
os.replace(tmp, CACHE / f"{kind}.json")
return True, f"up to date with {ch}"
def entry(cat, name):
return next((e for e in cat["models"] if e["name"] == name), None)
def installed(kind, name, tree=REPO):
"""A downloaded version's folder, if it has every file it needs and its pin, else None."""
if not NAME.fullmatch(name or ""):
return None
d = store(kind, tree) / name
if (d / "PIN.json").is_file() and all((d / f).is_file() for f in SPECS[kind]["required"]):
return d
return None
def installed_names(kind, tree=REPO):
s = store(kind, tree)
try:
return sorted(p.name for p in s.iterdir() if not p.name.startswith(".") and installed(kind, p.name, tree))
except OSError:
return []
def usable(kind, e):
"""Can this Frametop run the entry: builtins, and versions whose format it knows."""
return e["status"] == "builtin" or e["format"] <= SPECS[kind]["format"]
def setting(kind, conf=None):
"""The kind's setting from frametop.conf (FT_ in the environment wins), or the catalog's default."""
key = SPECS[kind]["setting"]
v = os.environ.get("FT_" + key)
if v is None:
v = (conf if conf is not None else read_conf()).get(key, "")
return v.strip()
# --- downloading ---
def _sha(path):
h = hashlib.sha256()
with open(path, "rb") as f:
for block in iter(lambda: f.read(1 << 20), b""):
h.update(block)
return h.hexdigest()
def verify(kind, name, tree=REPO):
"""Check a downloaded version's files against its pin: [] or the names that don't match."""
d = installed(kind, name, tree)
if not d:
return ["(not downloaded)"]
pin = json.loads((d / "PIN.json").read_text())
return [f for f, sha in pin["files"].items() if not (d / f).is_file() or _sha(d / f) != sha]
def fetch(kind, name, tree=REPO, cat=None, progress=None, timeout=30):
"""Download a version into the store, every file checked, then swapped in whole. Returns the
folder. Raises CatalogError (or OSError) when it can't."""
spec = SPECS[kind]
cat = cat or load(kind, tree)[0]
e = entry(cat, name)
if e is None:
raise CatalogError(f"{name} isn't in the catalog")
if e["status"] == "builtin":
raise CatalogError(f"{name} comes with Frametop: nothing to download")
if not usable(kind, e):
raise CatalogError(f"{name} needs a newer Frametop")
s = store(kind, tree)
s.mkdir(parents=True, exist_ok=True)
with open(s / ".lock", "w") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
d = s / name
if installed(kind, name, tree) and json.loads((d / "PIN.json").read_text()) == e and not verify(kind, name, tree):
return d
new = s / f".{name}.new"
new.mkdir(exist_ok=True)
for f, sha in sorted(e["files"].items()):
path = new / f
if path.is_file() and _sha(path) == sha:
continue
sub = f if f in spec["top"] else f"{name}/{f}"
url = f"https://huggingface.co/{spec['hf']}/resolve/{e['rev']}/{sub}"
if progress:
progress(f"downloading {f}")
part = new / (f + ".part")
try:
with urllib.request.urlopen(url, timeout=timeout) as r, open(part, "wb") as out:
total = 0
while block := r.read(1 << 20):
total += len(block)
if total > MAX_FILE:
raise CatalogError(f"{f} is bigger than {MAX_FILE >> 20} MB")
out.write(block)
except (urllib.error.URLError, OSError) as err:
part.unlink(missing_ok=True)
raise CatalogError(f"couldn't download {f} ({getattr(err, 'reason', err)})") from err
except CatalogError:
part.unlink(missing_ok=True)
raise
if _sha(part) != sha:
part.unlink()
raise CatalogError(f"{f}'s sha256 doesn't match the catalog; not kept")
os.replace(part, path)
(new / "PIN.json").write_text(json.dumps(e, indent=1))
old = s / f".{name}.old"
shutil.rmtree(old, ignore_errors=True)
if d.exists():
os.replace(d, old)
os.replace(new, d)
shutil.rmtree(old, ignore_errors=True)
return d
def remove(kind, name, tree=REPO):
d = installed(kind, name, tree)
if d:
shutil.rmtree(d)
return bool(d)
# --- what a setting means ---
def resolve(kind, value=None, tree=REPO, cat=None):
"""What the kind's setting runs now: {"setting", "name", "folder", "why"}. name is the version
(a catalog name, "latest"'s target), folder its downloaded folder or None (a builtin, or not
downloaded: then the runtime falls back and "why" says so). A value with a "/" is a folder of
one's own (for development), used as is."""
cat = cat or load(kind, tree)[0]
value = value if value is not None else setting(kind)
if not value:
value = cat.get("default", "latest")
if "/" in value:
return {"setting": value, "name": None, "folder": os.path.expanduser(value), "why": "a folder of your own"}
if value.lower() in ("off", "none"):
value = "classic" if kind == "eyes" else "stock"
if value == "latest":
name = cat.get("latest")
if kind == "hands": # hands/models.sh keeps latest in hands/models/student
return {"setting": "latest", "name": name, "folder": None, "why": "latest (hands/models/student)"}
folder = installed(kind, name, tree) if name else None
if folder:
return {"setting": "latest", "name": name, "folder": str(folder), "why": "latest"}
# Not fetched yet: the newest custom one downloaded, while the auto update fetches it.
have = [e["name"] for e in cat["models"] if e["kind"] == "custom" and installed(kind, e["name"], tree)]
if have:
return {"setting": "latest", "name": have[0], "folder": str(installed(kind, have[0], tree)),
"why": f"latest is {name}, not downloaded yet: using {have[0]}"}
return {"setting": "latest", "name": None, "folder": None, "why": f"latest is {name}, not downloaded yet"}
e = entry(cat, value)
if e is not None and e["status"] == "builtin":
return {"setting": value, "name": value, "folder": None, "why": e.get("title") or value}
folder = installed(kind, value, tree)
if folder:
return {"setting": value, "name": value, "folder": str(folder), "why": value}
return {"setting": value, "name": None, "folder": None, "why": f"{value} isn't downloaded"}
def auto(kind, tree=REPO, every=15 * 60):
"""What the services run when they start: refresh the catalog (at most every 15 minutes), then
fetch what the setting needs if it's missing. Not with KIND_MODELS_UPDATE=0. Returns a message."""
conf = read_conf()
upd = os.environ.get("FT_" + SPECS[kind]["update"], conf.get(SPECS[kind]["update"], "1"))
if upd.strip() == "0":
return f"{kind} models: not updating ({SPECS[kind]['update']}=0)"
c = cached(kind)
msg = ""
if not c or time.time() - c[2] > every:
msg = refresh(kind, tree)[1]
cat = load(kind, tree)[0]
value = setting(kind, conf) or cat.get("default", "latest")
want = cat.get("latest") if value == "latest" else value
if kind == "hands" and value == "latest":
return f"hands models: latest is hands/models.sh's ({msg})" if msg else "hands models: latest is hands/models.sh's"
e = entry(cat, want) if want and NAME.fullmatch(want) else None
if e is None or e["status"] == "builtin" or not usable(kind, e):
return f"{kind} models: nothing to fetch for {value!r}" + (f" ({msg})" if msg else "")
have = installed(kind, want, tree)
if have and json.loads((have / "PIN.json").read_text()) == e:
return f"{kind} models: {want} is here" + (f" ({msg})" if msg else "")
fetch(kind, want, tree, cat)
return f"{kind} models: fetched {want}"
def view(kind, tree=REPO):
"""Everything the input settings show for a kind: the catalog with each entry's state."""
cat, source = load(kind, tree)
c = cached(kind)
value = setting(kind) or cat.get("default", "latest")
have = set(installed_names(kind, tree))
models = []
for e in cat["models"]:
m = {k: e.get(k, "") for k in ("name", "title", "kind", "status", "date", "summary", "notes", "cost")}
m["bench"] = e.get("bench", {})
m["builtin"] = e["status"] == "builtin"
m["installed"] = m["builtin"] or e["name"] in have
m["usable"] = usable(kind, e)
m["size_mb"] = None
models.append(m)
listed = {e["name"] for e in cat["models"]}
for name in sorted(have - listed): # downloaded once, no longer listed: still usable
models.append({"name": name, "title": name, "kind": "custom", "status": "older", "date": "",
"summary": "No longer in the catalog; downloaded here.", "notes": "", "cost": "",
"bench": {}, "builtin": False, "installed": True, "usable": True, "size_mb": None})
return {"kind": kind, "latest": cat.get("latest"), "default": cat.get("default", "latest"),
"baseline": cat.get("baseline"), "metrics": cat.get("metrics", []), "models": models,
"setting": value, "resolved": resolve(kind, value, tree, cat), "source": source,
"checked": c[2] if c else None}
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env python3
"""ft-models: the hand and eye model catalogs (models/catalog.py) from a terminal.
ft-models list hands|eyes [--json] the versions, their notes, and what's in use
ft-models refresh hands|eyes fetch this install's branch's catalog from GitHub
ft-models fetch hands|eyes NAME download a version (every file checked against the catalog)
ft-models verify hands|eyes NAME check a downloaded version's files again
ft-models remove hands|eyes NAME delete a downloaded version
ft-models resolve hands|eyes [VALUE] what the setting (or VALUE) runs now, as JSON
ft-models auto hands|eyes what the services run at start: refresh at most every
15 minutes, fetch what the setting needs if it's missing
ft-models hashes hands|eyes STAGING VERSION
for a new catalog entry: the "files" of STAGING/VERSION and
STAGING's LICENSE and README.md, as uploaded (models/README.md)
Pick a model in Frametop Input Settings (Hands and Gaze pages), or set HANDS_MODEL / EYES_MODEL in
~/.config/frametop.conf: latest (follows new versions), a version's name, or a builtin (stock for
hands; classic for eyes, and GAZE_TRACKER=steam for SteamVR's).
"""
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import catalog # noqa: E402
def main(argv):
if len(argv) < 2 or argv[0] not in ("list", "refresh", "fetch", "verify", "remove", "resolve", "auto", "hashes") \
or argv[1] not in catalog.SPECS:
print(__doc__.strip(), file=sys.stderr)
return 2
cmd, kind, rest = argv[0], argv[1], argv[2:]
try:
if cmd == "list":
v = catalog.view(kind)
if "--json" in rest:
print(json.dumps(v, indent=1))
return 0
print(f"{kind} models ({v['source']}); setting {v['setting']!r} -> {v['resolved']['why']}")
for m in v["models"]:
mark = "*" if m["name"] == v["resolved"]["name"] else " "
latest = " (latest)" if m["name"] == v["latest"] else ""
state = "builtin" if m["builtin"] else "downloaded" if m["installed"] else \
"needs a newer Frametop" if not m["usable"] else "not downloaded"
print(f"{mark} {m['name']:<16} {m['status']:<12} {state:<15} {m['title']}{latest}")
if m["summary"]:
print(f" {'':<16} {m['summary']}")
return 0
if cmd == "refresh":
ok, msg = catalog.refresh(kind)
print(f"{kind} catalog: {msg}")
return 0 if ok else 1
if cmd == "resolve":
print(json.dumps(catalog.resolve(kind, rest[0] if rest else None)))
return 0
if cmd == "auto":
print(catalog.auto(kind))
return 0
if cmd == "hashes" and len(rest) == 2:
staging, version = Path(rest[0]), rest[1]
spec = catalog.SPECS[kind]
files = {p.name: catalog._sha(p) for p in sorted((staging / version).iterdir()) if p.is_file()}
files.update({n: catalog._sha(staging / n) for n in sorted(spec["top"]) if (staging / n).is_file()})
if set(files) - spec["files"] - spec["top"] or spec["required"] - set(files):
print(f"ft-models: a {kind} version has {sorted(spec['files'])} (needs {sorted(spec['required'])}),"
f" plus {sorted(spec['top'])} at the top; found {sorted(files)}", file=sys.stderr)
return 1
print(json.dumps({"files": files}, indent=1))
return 0
if not rest:
print(f"ft-models {cmd} {kind} NAME", file=sys.stderr)
return 2
name = rest[0]
if cmd == "fetch":
d = catalog.fetch(kind, name, progress=lambda s: print(s, flush=True))
print(f"{kind} model {name} in {d}")
elif cmd == "verify":
bad = catalog.verify(kind, name)
print(f"{kind} model {name}: " + ("all files match" if not bad else "doesn't match: " + ", ".join(bad)))
return 1 if bad else 0
elif cmd == "remove":
print(f"{kind} model {name}: " + ("removed" if catalog.remove(kind, name) else "wasn't downloaded"))
except (catalog.CatalogError, OSError, ValueError) as e:
print(f"ft-models: {e}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+236
View File
@@ -0,0 +1,236 @@
#!/usr/bin/env python3
"""models/catalog.py: the catalogs in the repo are valid and agree with hands/models.pin, a bad catalog
is refused whole, downloads keep only files that match, and settings resolve as the runtimes expect.
No network: downloads go to a fake urlopen. Run: python3 models/test/test_catalog.py"""
import copy
import hashlib
import io
import json
import os
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
import catalog # noqa: E402
REPO = HERE.parents[1]
def sha(b):
return hashlib.sha256(b).hexdigest()
FILES = {"model.json": b'{"name": "eyes-t1"}', "model.ncnn.param": b"param", "model.ncnn.bin": b"weights",
"NOTICE": b"notice", "LICENSE": b"licence", "README.md": b"card"}
REV = "a" * 40
def eye_catalog(**over):
cat = {"catalog": 1, "kind": "eyes", "latest": "eyes-t1", "default": "classic", "baseline": "steamvr",
"metrics": [{"key": "fresh", "label": "Fresh", "unit": "°", "better": "lower"}],
"models": [
{"name": "classic", "title": "Classic", "kind": "classic", "status": "builtin", "bench": {"fresh": 2.8}},
{"name": "steamvr", "title": "SteamVR", "kind": "valve", "status": "builtin"},
{"name": "eyes-t1", "title": "T1", "kind": "custom", "status": "testing", "format": 1, "rev": REV,
"files": {k: sha(v) for k, v in FILES.items()}, "bench": {"fresh": 1.6}},
{"name": "eyes-t0", "title": "T0", "kind": "custom", "status": "older", "format": 1, "rev": REV,
"files": {k: sha(v) for k, v in FILES.items()}},
]}
cat.update(over)
return cat
class FakeHF:
"""urlopen for huggingface.co/DeeJanuz/frametop-eyes-models/resolve/REV/...: serves FILES."""
def __init__(self, files=FILES):
self.files, self.urls = dict(files), []
def __call__(self, url, timeout=None):
self.urls.append(url)
prefix = f"https://huggingface.co/{catalog.SPECS['eyes']['hf']}/resolve/{REV}/"
assert url.startswith(prefix), url
name = url[len(prefix):].split("/")[-1]
return io.BytesIO(self.files[name])
class RepoCatalogs(unittest.TestCase):
def test_bundled_valid(self):
for kind in catalog.SPECS:
cat = catalog.bundled(kind)
self.assertEqual(cat["kind"], kind)
def test_hands_latest_matches_models_pin(self):
"""hands/models.sh fetches latest by models.pin; the catalog's latest must be the same files."""
pin = {"files": {}}
for line in (REPO / "hands" / "models.pin").read_text().splitlines():
parts = line.split()
if not parts or parts[0].startswith("#"):
continue
if parts[0] == "file":
pin["files"][parts[1]] = parts[2]
else:
pin[parts[0]] = parts[1]
cat = catalog.bundled("hands")
e = catalog.entry(cat, cat["latest"])
self.assertEqual((e["name"], e["rev"], e["files"]), (pin["model"], pin["rev"], pin["files"]))
def test_texts_have_no_markup(self):
"""Shown as plain text, but keep the catalogs free of markup anyway."""
for kind in catalog.SPECS:
for e in catalog.bundled(kind)["models"]:
for k in ("title", "summary", "notes", "cost"):
self.assertNotIn("<", e.get(k, ""), (kind, e["name"], k))
class Validate(unittest.TestCase):
def bad(self, cat):
with self.assertRaises(catalog.CatalogError):
catalog.validate(cat, "eyes")
def test_good(self):
catalog.validate(eye_catalog(), "eyes")
def test_refusals(self):
c = eye_catalog()
self.bad(dict(c, kind="hands"))
self.bad(dict(c, catalog=2))
self.bad(dict(c, latest="classic")) # latest must be downloadable
self.bad(dict(c, latest="nope"))
self.bad(dict(c, default="nope"))
for change in (
lambda m: m.update(name="../x"),
lambda m: m.update(name="a/b"),
lambda m: m.update(rev="abc"),
lambda m: m["files"].update({"run.sh": "0" * 64}), # not a model file
lambda m: m["files"].update({"model.json": "xyz"}), # not a sha256
lambda m: m["files"].pop("model.ncnn.bin"), # a required file missing
lambda m: m.update(bench={"unknown": 1}),
lambda m: m.update(bench={"fresh": "1"}),
lambda m: m.update(status="best"),
lambda m: m.update(notes="x" * 5000),
lambda m: m.update(format=0),
):
c2 = copy.deepcopy(c)
change(c2["models"][2])
self.bad(c2)
c2 = copy.deepcopy(c)
c2["models"][0]["rev"] = REV # a builtin has nothing to fetch
self.bad(c2)
c2 = copy.deepcopy(c)
c2["models"].append(copy.deepcopy(c2["models"][2]))
self.bad(c2) # listed twice
class Store(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.tree = Path(self.tmp.name)
self.cat = eye_catalog()
env = {"XDG_CACHE_HOME": str(self.tree / "cache"), "FT_EYES_MODEL": "", "FT_EYES_MODELS_UPDATE": ""}
self.env = mock.patch.dict(os.environ, env)
self.env.start()
self.cache = mock.patch.object(catalog, "CACHE", self.tree / "cache" / "frametop" / "models")
self.cache.start()
def tearDown(self):
self.cache.stop()
self.env.stop()
self.tmp.cleanup()
def fetch(self, name="eyes-t1", hf=None):
hf = hf or FakeHF()
with mock.patch.object(catalog.urllib.request, "urlopen", hf):
return catalog.fetch("eyes", name, tree=self.tree, cat=self.cat), hf
def test_fetch_and_verify(self):
d, hf = self.fetch()
self.assertEqual(d, self.tree / "gaze/tracker/models/eyes-t1")
self.assertEqual({p.name for p in d.iterdir()}, set(FILES) | {"PIN.json"})
self.assertEqual(catalog.verify("eyes", "eyes-t1", self.tree), [])
self.assertTrue(any(u.endswith(f"/{REV}/LICENSE") for u in hf.urls)) # top files from the top
self.assertTrue(any(u.endswith(f"/{REV}/eyes-t1/model.ncnn.bin") for u in hf.urls))
_, again = self.fetch() # already there and matching: nothing downloaded
self.assertEqual(again.urls, [])
(d / "model.ncnn.bin").write_bytes(b"changed")
self.assertEqual(catalog.verify("eyes", "eyes-t1", self.tree), ["model.ncnn.bin"])
def test_bad_file_not_kept(self):
with self.assertRaises(catalog.CatalogError):
self.fetch(hf=FakeHF(dict(FILES, **{"model.ncnn.bin": b"tampered"})))
self.assertIsNone(catalog.installed("eyes", "eyes-t1", self.tree))
new = self.tree / "gaze/tracker/models/.eyes-t1.new"
self.assertFalse(any(p.name.endswith(".part") or p.name == "model.ncnn.bin" for p in new.iterdir()))
def test_builtin_and_unknown_not_fetched(self):
for name in ("classic", "nope"):
with self.assertRaises(catalog.CatalogError):
self.fetch(name)
def test_newer_format_not_fetched(self):
self.cat["models"][2]["format"] = catalog.SPECS["eyes"]["format"] + 1
with self.assertRaises(catalog.CatalogError):
self.fetch()
def test_resolve(self):
r = lambda v: catalog.resolve("eyes", v, self.tree, self.cat) # noqa: E731
self.assertEqual(r("")["name"], "classic") # unset: the catalog's default
self.assertIsNone(r("classic")["folder"])
self.assertIsNone(r("latest")["folder"]) # nothing downloaded: the pupil finder
self.assertIsNone(r("eyes-t1")["folder"])
self.assertEqual(r("/some/folder")["folder"], "/some/folder")
self.fetch("eyes-t0")
self.assertEqual(r("latest")["name"], "eyes-t0") # latest missing: a downloaded custom one
self.fetch("eyes-t1")
self.assertEqual(r("latest")["folder"], str(self.tree / "gaze/tracker/models/eyes-t1"))
self.assertEqual(r("eyes-t0")["name"], "eyes-t0")
def test_hands_latest_is_student(self):
cat = catalog.bundled("hands")
self.assertIsNone(catalog.resolve("hands", "latest", self.tree, cat)["folder"])
self.assertIsNone(catalog.resolve("hands", "stock", self.tree, cat)["folder"])
def test_channel(self):
self.assertIsNone(catalog.channel(self.tree))
(self.tree / ".git").mkdir()
(self.tree / ".git" / "HEAD").write_text("ref: refs/heads/experimental\n")
self.assertEqual(catalog.channel(self.tree), "experimental")
(self.tree / ".git" / "HEAD").write_text("ref: refs/heads/my-branch\n")
self.assertIsNone(catalog.channel(self.tree))
(self.tree / ".frametop-release").write_text("VERSION=x\nCHANNEL=stable\n")
self.assertEqual(catalog.channel(self.tree), "main")
def test_auto_off(self):
with mock.patch.dict(os.environ, {"FT_EYES_MODELS_UPDATE": "0"}):
self.assertIn("not updating", catalog.auto("eyes", self.tree))
def test_auto_fetches_setting(self):
(self.tree / "gaze/tracker").mkdir(parents=True)
(self.tree / "gaze/tracker/models.json").write_text(json.dumps(self.cat))
with mock.patch.dict(os.environ, {"FT_EYES_MODEL": "latest"}), \
mock.patch.object(catalog, "refresh", lambda kind, tree: (False, "offline")), \
mock.patch.object(catalog.urllib.request, "urlopen", FakeHF()):
self.assertIn("fetched eyes-t1", catalog.auto("eyes", self.tree))
self.assertIn("is here", catalog.auto("eyes", self.tree))
def test_cached_catalog_only_for_its_branch(self):
(self.tree / "gaze/tracker").mkdir(parents=True)
(self.tree / "gaze/tracker/models.json").write_text(json.dumps(self.cat))
newer = eye_catalog(latest="eyes-t0")
catalog.CACHE.mkdir(parents=True)
(catalog.CACHE / "eyes.json").write_text(json.dumps({"branch": "experimental", "fetched": 0, "catalog": newer}))
self.assertEqual(catalog.load("eyes", self.tree)[0]["latest"], "eyes-t1") # not on a branch: its own
(self.tree / ".frametop-release").write_text("CHANNEL=experimental\n")
cat, source = catalog.load("eyes", self.tree)
self.assertEqual((cat["latest"], source), ("eyes-t0", "github experimental"))
(catalog.CACHE / "eyes.json").write_text(json.dumps({"branch": "experimental", "fetched": 0,
"catalog": dict(newer, latest="classic")}))
self.assertEqual(catalog.load("eyes", self.tree)[0]["latest"], "eyes-t1") # invalid cache: ignored
if __name__ == "__main__":
unittest.main()
+58 -34
View File
@@ -8,7 +8,7 @@
# Frame's dev container; the list below freezes the runtime-relevant subset of
# it plus the toolchain (the binaries are built in this image). Keep the two in
# sync on purpose: dev-container.sh documents *why* each package is there.
FROM registry.fedoraproject.org/fedora-toolbox:44@sha256:034cb7c472038e2d879ddc19568106a1342aa24403f0d641a0f73dda638707ad
FROM registry.fedoraproject.org/fedora-toolbox:44@sha256:034cb7c472038e2d879ddc19568106a1342aa24403f0d641a0f73dda638707ad AS runtime-common
# Toolchain (the binaries are built in here) and the libraries they link.
RUN dnf -y install --setopt=install_weak_deps=False \
@@ -25,8 +25,9 @@ RUN dnf -y install --setopt=install_weak_deps=False \
# Vulkan backend; ft-voiced records with pw-record and plays its sounds with pw-play
glslc spirv-headers-devel pipewire-utils \
# Frametop Input Settings (Kirigami, PySide6) and remote desktop (krdp):
# the KDE/Qt stack has no PyPI wheels, so it stays a dnf package
python3-pyside6 kf6-kirigami kf6-qqc2-desktop-style qt6-qtwayland \
# keep the native KDE/Qt stack in matching Fedora packages
python3-shiboken6 kf6-kirigami kf6-qqc2-desktop-style \
qt6-qtwayland qt6-qtsvg qt6-qtimageformats qt6-qtpdf \
breeze-icon-theme plasma-breeze krdp freerdp tigervnc-x11-server xrandr \
&& dnf -y remove 'glibc-langpack-*' \
&& dnf -y install glibc-langpack-en \
@@ -34,14 +35,6 @@ RUN dnf -y install --setopt=install_weak_deps=False \
&& rm -rf /usr/share/doc/* /usr/share/man/* /usr/share/info/* \
/usr/share/wallpapers
# uv, pinned and checked. Python dependencies come from the committed uv.lock,
# not from whatever PyPI serves on build day.
RUN curl -fsSL https://github.com/astral-sh/uv/releases/download/0.12.23/uv-aarch64-unknown-linux-gnu.tar.gz \
-o /tmp/uv.tar.gz \
&& echo "6524bd338177ed50d035d39354e12545e993bbeba2ecbddf0480c5b3a81d313f /tmp/uv.tar.gz" | sha256sum -c \
&& tar -xzf /tmp/uv.tar.gz --strip-components=1 -C /usr/local/bin \
&& rm /tmp/uv.tar.gz
# OpenVR's API library for linking, from the SDK tag the build scripts pin
# (scripts/openvr.sh fetches the headers). The image has no SteamVR, so the
# binaries link this copy; their rpath names SteamVR's folder first, so on the
@@ -53,47 +46,75 @@ RUN install -d /opt/frametop/bin /opt/frametop/lib \
# Python: Fedora's own python3 (the digest-pinned base freezes it), with the
# locked packages in a venv on top. The venv sees the system site-packages,
# because PySide6 and Kirigami come from dnf for this python3: one interpreter
# because PySide6 and Kirigami come from Fedora for this python3: one interpreter
# imports both, as in the dev container on the Frame. The check names the
# recorders' huggingface_hub (uploads) and cryptography (encrypted exports) too:
# the tests that need them skip without them. Rebuilt only when the lockfile
# changes, so source edits below reuse this layer.
# the tests that need them skip without them. PySide6 itself is checked in the
# runtime stage at the end, where the builder's export lands. Rebuilt only when
# the lockfile changes, so source edits below reuse this layer.
# uv sync --frozen --group tracker --group voice: the default dev group (pytest/ruff
# for CI's in-image tests) plus the tracker stack and Frametop Voice's (sherpa-onnx);
# the types group (mypy) stays out.
# --no-cache keeps uv's download cache out of the image.
# uv is a pinned, checked build tool, not a runtime dependency. Extract it,
# create the environment, and remove it in one layer so neither uv nor uvx
# takes space in an installed image.
ENV UV_PROJECT_ENVIRONMENT=/opt/frametop/venv UV_PYTHON=/usr/bin/python3 UV_PYTHON_DOWNLOADS=never
COPY pyproject.toml uv.lock .python-version /src/frametop/
RUN uv venv --system-site-packages /opt/frametop/venv \
&& cd /src/frametop && uv sync --frozen --group tracker --group voice --no-cache \
&& /opt/frametop/venv/bin/python -c 'import PySide6, numpy, cv2, sherpa_onnx, sentencepiece, huggingface_hub, cryptography'
RUN uvdir=$(mktemp -d) \
&& curl -fsSL https://github.com/astral-sh/uv/releases/download/0.12.23/uv-aarch64-unknown-linux-gnu.tar.gz \
-o "$uvdir/uv.tar.gz" \
&& echo "6524bd338177ed50d035d39354e12545e993bbeba2ecbddf0480c5b3a81d313f $uvdir/uv.tar.gz" | sha256sum -c \
&& tar -xzf "$uvdir/uv.tar.gz" --strip-components=1 -C "$uvdir" uv-aarch64-unknown-linux-gnu/uv \
&& "$uvdir/uv" venv --system-site-packages /opt/frametop/venv \
&& cd /src/frametop && "$uvdir/uv" sync --frozen --group tracker --group voice --no-cache \
&& /opt/frametop/venv/bin/python -c 'import numpy, cv2, sherpa_onnx, sentencepiece, huggingface_hub, cryptography' \
&& rm -rf "$uvdir"
# Preserve separate base/environment layers for cached builds and rollback.
# Keep the toolchain and headers: ft dev shell/test still work in this image.
# PySide's full Fedora RPM also requires WebEngine/3D/etc. The builder exports
# the unchanged bindings Frametop uses, against these same system Qt libraries.
WORKDIR /src/frametop
ENV FRAME_IN_BOX=1 OPENVR_LIB=/opt/frametop/lib
ENV PATH=/opt/frametop/venv/bin:/opt/frametop/bin:/usr/local/bin:/usr/bin:/usr/sbin
RUN ln -s /run/host/opt/steamvr /opt/steamvr
CMD ["sleep", "infinity"]
FROM runtime-common AS builder
RUN rpm -qa --qf '%{NEVRA}\n' | sort > /tmp/runtime-rpms \
&& dnf -y install --setopt=install_weak_deps=False python3-pyside6 \
&& rpm -qa --qf '%{NEVRA}\n' | sort > /tmp/builder-rpms \
&& test -z "$(comm -23 /tmp/runtime-rpms /tmp/builder-rpms)" \
&& dnf clean all \
&& install -d /runtime/opt/frametop/bin /runtime/opt/frametop/lib
# The sources, then the binaries, built by the same build scripts as on the
# Frame (FRAME_IN_BOX=1: they run here instead of entering the dev container).
# Each program stays in its build/ folder in /src/frametop and is copied to
# /opt/frametop.
COPY . /src/frametop
WORKDIR /src/frametop
ENV FRAME_IN_BOX=1 OPENVR_LIB=/opt/frametop/lib
RUN screens/build.sh \
&& install -t /opt/frametop/bin screens/build/ft-screens screens/build/ft-handtest screens/build/ft-spotpanel
&& install -t /runtime/opt/frametop/bin screens/build/ft-screens screens/build/ft-handtest screens/build/ft-spotpanel \
&& rm -f screens/build/*.o
RUN pointer/helper/build.sh && pointer/driver/build.sh \
&& install -t /opt/frametop/bin pointer/helper/build/ft-pointer \
&& install -t /opt/frametop/lib pointer/driver/build/driver_ft_pointer.so
&& install -t /runtime/opt/frametop/bin pointer/helper/build/ft-pointer \
&& install -t /runtime/opt/frametop/lib pointer/driver/build/driver_ft_pointer.so
RUN gaze/build.sh \
&& install -t /opt/frametop/bin gaze/build/ft-gaze gaze/build/ft-gazepanel
&& install -t /runtime/opt/frametop/bin gaze/build/ft-gaze gaze/build/ft-gazepanel
RUN power/build.sh \
&& install -t /opt/frametop/bin power/build/ft-powerd
&& install -t /runtime/opt/frametop/bin power/build/ft-powerd
RUN stream/build.sh \
&& install -t /opt/frametop/bin stream/build/ft-stream
&& install -t /runtime/opt/frametop/bin stream/build/ft-stream \
&& rm -rf stream/build/mlc stream/build/gamestream
# Our own eye tracker: ft-eyegrab (it runs on the host, as root) and ft-eyes'
# build/venv, which here only points at the venv above.
RUN gaze/tracker/build.sh \
&& install -t /opt/frametop/bin gaze/tracker/build/ft-eyegrab
&& install -t /runtime/opt/frametop/bin gaze/tracker/build/ft-eyegrab
# Frametop Voice: whisper.cpp's server (Vulkan), and build/venv pointing at the venv
# above. Its speech models aren't in the image: voice/models.sh fetches them on install.
RUN voice/build.sh \
&& install -t /opt/frametop/bin voice/build/bin/whisper-server
&& install -t /runtime/opt/frametop/bin voice/build/bin/whisper-server
# Hand tracking: ft-camd (static: it runs on the host, with the file
# capabilities hands/run.sh install sets) and ft-hands, which builds ncnn at the
# tag hands/Makefile pins. Both stay in hands/build/, where their units and
@@ -115,14 +136,17 @@ RUN git init -q pack/build/distrobox \
&& git fetch -q --depth 1 https://github.com/89luca89/distrobox.git 40c3cd724faa434aeb0a23e28776665b92de68bd \
&& git checkout -q FETCH_HEAD \
&& rm -rf .git
# Programs built against OpenVR look for SteamVR's library here first. In the
# container on the Frame, distrobox mounts the host at /run/host; elsewhere the
# link leads nowhere and they use /opt/frametop/lib's.
RUN ln -s /run/host/opt/steamvr /opt/steamvr
# Export Fedora's selected Python bindings without changing their native code.
RUN python3 pack/pyside-runtime.py --repo /src/frametop --destination /runtime
# The default environment: Frametop's binaries and the venv's python3 (Fedora's
# interpreter, with the locked packages and the dnf Qt stack) first.
# interpreter, with the locked packages and the Fedora Qt stack) first.
# sleep infinity as the default command keeps the image usable with distrobox
# create, like the toolbox base.
ENV PATH=/opt/frametop/venv/bin:/opt/frametop/bin:/usr/local/bin:/usr/bin:/usr/sbin
CMD ["sleep", "infinity"]
FROM runtime-common AS runtime
COPY --from=builder /runtime/usr/ /usr/
COPY --from=builder /runtime/opt/ /opt/
COPY --from=builder /src/frametop/ /src/frametop/
RUN python3 pack/pyside-runtime.py --verify /usr/share/frametop/pyside-runtime.json \
&& /usr/bin/python3 -c 'from PySide6 import QtCore, QtGui, QtQml, QtQuick, QtQuickControls2' \
&& python3 -c 'import numpy, cv2'
+28 -1
View File
@@ -19,7 +19,9 @@ different glibc. This directory turns the container from a recipe into an
versions with aarch64 wheels, installed into `/opt/frametop/venv`. The venv
sits on Fedora's own `python3`, which the base digest freezes, and sees the
system site-packages: PySide6 and Kirigami come from dnf for that
interpreter, so one `python3` imports both, as in the dev container.
interpreter, so one `python3` imports both, as in the dev container. The
checked uv executable is temporary: only the environment it creates stays
in the image.
3. **OpenVR from the pinned public tag** — `v2.15.6`, in
`scripts/openvr.sh`. The build scripts fetch its headers and check their
sha256. The image has no SteamVR, so it links the SDK's prebuilt
@@ -36,6 +38,31 @@ tracking's `ft-camd` and `ft-hands` (with ncnn, built from the tag
`hands/Makefile` pins) stay in `hands/build/`, where their units and
`ft-camd`'s file capabilities point.
Component builds strip non-runtime symbols before the binaries are copied into
the image. The pointer driver and gaze panel also discard unused sections;
they keep `-O2`, the existing floating-point flags, and the host glibc checks.
The image has a shared Fedora/toolchain stage, a builder, and the final runtime.
The builder installs Fedora's full PySide6 RPM, then `pyside-runtime.py` exports
the bindings imported by the four Frametop settings/recording applications.
Their native binaries and support libraries are copied unchanged to the same
Fedora paths, so both the system Python used by distrobox and the locked venv
can import them. A manifest records their source RPMs and file hashes; the
image build and offline tests check imports, hashes and shared-library resolution.
The available-module metadata describes the exported subset rather than the
full SDK. Keep `BACKENDS` in that script current when adding another Qt app.
The native Qt/Kirigami libraries, desktop style, Breeze icons, platform plugins
and image formats (including SVG and PDF) remain distro packages. Qt WebEngine,
3D and other bindings unused by Frametop stay in the builder. This is Frametop's
private application runtime; the separate `dev` container still installs the
full PySide SDK. The runtime also keeps its compilers, headers and test tools,
so the native build recipes and `ft dev test` still work. Base/environment
layers stay separate from the source payload: the pinned toolbox base is
shared with the retained rollback release, and cached builds can also reuse
the added environment layers. Clean builds on independent runners need not
produce identical environment layers, even with identical RPM versions.
What deliberately stays outside the image for now: the **host payload** — the
SteamVR driver registration (`vrpathreg`), ft-camd's file capabilities
(setcap), systemd units, the KWin script. Those need the SteamOS host, and
+2 -1
View File
@@ -20,6 +20,7 @@
#
# Usage: install-release.sh [--yes] [--any-steamos] [--unpack-only] [--dir DIR]
# [--no-voice] [--no-hands] [--no-hand-recorder] [--no-eye-tracker]
# [--no-eye-recorder]
# [--no-bluetooth | --bluetooth]
# --unpack-only stop after step 2 and the copy: don't run install.sh
# --dir DIR where the releases go (default ~/.local/share/frametop/releases)
@@ -113,7 +114,7 @@ main() {
--any-steamos) any=1 ;;
--unpack-only) unpack_only=1 ;;
--dir) base=${2:?--dir needs a folder}; shift ;;
--no-voice|--no-hands|--no-hand-recorder|--no-eye-tracker|--no-bluetooth|--bluetooth) pass+=("$1") ;;
--no-voice|--no-hands|--no-hand-recorder|--no-eye-tracker|--no-eye-recorder|--no-bluetooth|--bluetooth) pass+=("$1") ;;
-h|--help) sed -n '2,27p' "$0"; return 0 ;;
*) echo "unknown option: $1" >&2; return 2 ;;
esac
+160
View File
@@ -0,0 +1,160 @@
#!/usr/bin/env python3
"""Export Frametop's Fedora PySide bindings, or verify the installed app runtime.
Native Qt, QML, platform/style plugins and python3-shiboken6 remain dnf packages.
Only the unused Python Qt bindings and SDK tool links are omitted; ELF files are
copied unchanged to their original Fedora paths. This is a private app runtime.
"""
import argparse
import ast
import compileall
import hashlib
import importlib
import importlib.util
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
DIRECT = ("QtCore", "QtGui", "QtQml", "QtQuick", "QtQuickControls2")
BACKENDS = ("display-settings/ft_display_settings.py", "input-settings/ft_input_settings.py",
"remote-displays/ft_remote_displays.py", "hands/rec/ft_handrec.py", "gaze/rec/ft_eyerec.py")
MANIFEST = Path("usr/share/frametop/pyside-runtime.json")
def run(*args, **kwargs):
return subprocess.check_output(args, text=True, **kwargs).strip()
def imports(repo):
for name in DIRECT:
importlib.import_module("PySide6." + name)
for i, name in enumerate(BACKENDS):
path = repo / name
sys.path.insert(0, str(path.parent))
spec = importlib.util.spec_from_file_location(f"_ft_pyside_{i}", path)
spec.loader.exec_module(importlib.util.module_from_spec(spec))
sys.path.pop(0)
def check_elf(paths):
for path in paths:
if "not found" in run("ldd", str(path)):
raise RuntimeError(f"unresolved ELF dependency: {path}")
def export(dest, repo):
imports(repo)
import PySide6
from PySide6 import _config
from PySide6.QtCore import qVersion
package = Path(PySide6.__file__).parent.resolve()
modules = {name.split(".")[-1]: Path(m.__file__).resolve()
for name, m in sys.modules.items() if name.startswith("PySide6.Qt")
and isinstance(getattr(m, "__file__", None), str) and m.__file__.endswith(".so")}
if not set(DIRECT) <= modules.keys():
raise RuntimeError("a required binding was not imported")
if (dest / package.relative_to("/")).exists() or (dest / MANIFEST).exists():
raise RuntimeError(f"PySide was already exported into {dest}")
copied, elf = [], []
for name in run("rpm", "-ql", "python3-pyside6").splitlines():
src = Path(name)
if not src.is_file() and not src.is_symlink():
continue
if src.name == "entry_points.txt":
continue # this private runtime does not advertise the omitted SDK commands
if src.is_relative_to(package):
rel = src.relative_to(package)
if "__pycache__" in rel.parts or rel.parts[0] in ("Qt", "scripts"):
continue
if len(rel.parts) == 1 and rel.name.startswith("Qt") and rel.name.split(".")[0] not in modules:
continue
if src.is_symlink() and ".so" not in src.name:
continue # SDK command links, not the package's Python/private helpers
elif not ((src.name.startswith("libpyside6") and ".so" in src.name)
or any(part.startswith("PySide6-") and part.endswith((".dist-info", ".egg-info"))
for part in src.parts)
or name.startswith("/usr/share/licenses/python3-pyside6/")):
continue
out = dest / src.relative_to("/")
if out.exists() or out.is_symlink():
raise RuntimeError(f"export would replace a file: {out}")
out.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(src, out, follow_symlinks=False)
copied.append(name)
if ".so" in src.name:
elf.append(name)
if not src.is_symlink() and src.read_bytes() != out.read_bytes():
raise RuntimeError(f"ELF changed during export: {src}")
# __all__ discovers the available extension files. The build metadata must agree.
cfg = dest / (package / "_config.py").relative_to("/")
cfg_stat = cfg.stat()
text = cfg.read_text()
nodes = [n for n in ast.parse(text).body if isinstance(n, ast.Assign)
and any(isinstance(t, ast.Name) and t.id == "built_modules" for t in n.targets)]
if len(nodes) != 1 or nodes[0].col_offset:
raise RuntimeError("unrecognized PySide built_modules metadata")
available = [n for n in _config.built_modules if (n if n.startswith("Qt") else "Qt" + n) in modules]
if len(available) != len(modules):
raise RuntimeError("PySide module metadata and imported bindings disagree")
node, lines = nodes[0], text.splitlines(keepends=True)
cfg.write_text("".join(lines[:node.lineno - 1]) + f"built_modules = {available!r}\n"
+ "".join(lines[node.end_lineno:]))
os.utime(cfg, ns=(cfg_stat.st_atime_ns, cfg_stat.st_mtime_ns))
staged = dest / package.relative_to("/")
check_elf([dest / n.lstrip("/") for n in elf])
probe = ("import PySide6; from PySide6 import _config; "
f"assert set(PySide6.__all__) == {set(modules)!r}; "
f"assert _config.built_modules == {available!r}")
run(sys.executable, "-c", probe, env=dict(os.environ, PYTHONPATH=str(staged.parent),
PYTHONDONTWRITEBYTECODE="1"))
if not compileall.compile_dir(str(staged), ddir=str(package), quiet=1, optimize=[0, 1]):
raise RuntimeError("PySide bytecode compilation failed")
for pyc in staged.rglob("*.pyc"):
stat = Path(importlib.util.source_from_cache(str(pyc))).stat()
os.utime(pyc, ns=(stat.st_atime_ns, stat.st_mtime_ns))
manifest = {"private_app_runtime": True, "pyside": PySide6.__version__, "qt": qVersion(), "modules": sorted(modules),
"source_rpms": run("rpm", "-q", "python3-pyside6", "python3-shiboken6").splitlines(),
"metadata_change": "_config.built_modules narrowed; omitted SDK scripts and entry_points.txt",
"elf": elf, "sha256": {n: hashlib.sha256((dest / n.lstrip("/")).read_bytes()).hexdigest()
for n in copied if not (dest / n.lstrip("/")).is_symlink()}}
target = dest / MANIFEST
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(json.dumps(manifest, indent=2) + "\n")
print(json.dumps({"modules": sorted(modules), "files": len(copied), "destination": str(dest)}))
def main():
p = argparse.ArgumentParser(description=__doc__)
action = p.add_mutually_exclusive_group(required=True)
action.add_argument("--destination", "--export", dest="export", type=Path,
help="staging directory, later copied to / (may already contain /opt)")
action.add_argument("--verify", type=Path, help="installed pyside-runtime.json")
p.add_argument("--repo", type=Path, default=Path(__file__).resolve().parent.parent)
a = p.parse_args()
if a.export:
export(a.export.resolve(), a.repo.resolve())
else:
manifest = json.loads(a.verify.read_text())
for path, sha in manifest["sha256"].items():
if hashlib.sha256(Path(path).read_bytes()).hexdigest() != sha:
raise RuntimeError(f"runtime file changed: {path}")
check_elf(manifest["elf"])
for name in manifest["modules"]:
importlib.import_module("PySide6." + name)
imports(a.repo.resolve())
import PySide6
from PySide6.QtCore import qVersion
if qVersion() != manifest["qt"] or run("rpm", "-q", "python3-shiboken6") not in manifest["source_rpms"]:
raise RuntimeError("runtime Qt/Shiboken version differs from the builder")
if set(PySide6.__all__) != set(manifest["modules"]):
raise RuntimeError("runtime module advertisement differs")
print("PySide runtime: all checks passed")
if __name__ == "__main__":
main()
+1 -1
View File
@@ -1,6 +1,6 @@
## Install
On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1.1 GB), then either:
On a Steam Frame (SteamOS, VR variant). Download **Frametop.zip** below (about 1.4 GB), then either:
- **With [FrameDrop](https://framedropvr.com) on a PC:** drop Frametop.zip on your paired headset. In the headset, open Frametop in your library and press Play.
- **On the headset:** unpack Frametop.zip (in the desktop's Dolphin or Konsole) and run `Frametop/frametop-install.sh`.
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env python3
"""Check the GUI bindings under both Python paths used by the image and distrobox."""
import importlib.util
import os
from pathlib import Path
import subprocess
import sys
REPO = Path(__file__).resolve().parents[2]
HELPER = REPO / "pack/pyside-runtime.py"
def probe():
spec = importlib.util.spec_from_file_location("pyside_runtime", HELPER)
helper = importlib.util.module_from_spec(spec)
spec.loader.exec_module(helper)
manifest = Path("/") / helper.MANIFEST
if manifest.exists():
# The helper verifies hashes, ELF dependencies, advertised modules, versions,
# required Qt modules and the four real backend imports together.
subprocess.run([sys.executable, str(HELPER), "--verify", str(manifest),
"--repo", str(REPO)], check=True, timeout=90)
else:
# Full Fedora SDK/source-development containers need no subset manifest.
helper.imports(REPO)
print(f"{sys.executable}: Frametop GUI bindings passed", flush=True)
def main():
env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1")
env.pop("PYTHONPATH", None)
env.pop("PYTHONHOME", None)
seen = set()
for python in (Path(sys.executable), Path("/usr/bin/python3"),
Path("/opt/frametop/venv/bin/python")):
if not python.is_file():
continue
# Keep system and venv paths distinct even when their binaries are symlinks
# to the same executable. Only aliases within one bin directory are skipped.
directory = python.parent.resolve()
if directory in seen:
continue
seen.add(directory)
subprocess.run([str(python), str(Path(__file__).resolve()), "--probe"],
env=env, check=True, timeout=120)
if __name__ == "__main__":
if sys.argv[1:] == ["--probe"]:
probe()
elif len(sys.argv) == 1:
main()
else:
raise SystemExit("usage: pyside-runtime-test.py")
+3 -2
View File
@@ -10,10 +10,11 @@ root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)
exec "$root/scripts/frame.sh" -C pointer/driver 'set -e
mkdir -p build
. ../../scripts/openvr.sh
g++ -std=c++17 -O2 -fPIC -shared -fvisibility=hidden -fno-math-errno -Wall -Wno-unused-parameter \
-static-libstdc++ -static-libgcc -Wl,--exclude-libs,ALL \
g++ -std=c++17 -O2 -ffunction-sections -fdata-sections -fPIC -shared -fvisibility=hidden -fno-math-errno -Wall -Wno-unused-parameter \
-static-libstdc++ -static-libgcc -Wl,--exclude-libs,ALL,--gc-sections \
$OPENVR_CFLAGS \
-o build/driver_ft_pointer.so driver_ft_pointer.cpp -lpthread
strip --strip-unneeded build/driver_ft_pointer.so
max=$(objdump -T build/driver_ft_pointer.so | grep -oE "GLIBC_[0-9.]+" | sort -uV | tail -1)
echo "built build/driver_ft_pointer.so, newest glibc symbol: $max"
[ "$(printf "%s\n" "$max" GLIBC_2.39 | sort -V | tail -1)" = GLIBC_2.39 ] || { echo "needs newer glibc than the host has" >&2; exit 1; }'
+1
View File
@@ -7,4 +7,5 @@ exec "$root/scripts/frame.sh" -C pointer/helper 'set -e; mkdir -p build
. ../../scripts/openvr.sh
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter $OPENVR_CFLAGS -I../common \
-o build/ft-pointer ft-pointer.cpp $OPENVR_LIBS -lpthread
strip --strip-unneeded build/ft-pointer
echo "built build/ft-pointer"'
+60 -18
View File
@@ -319,9 +319,13 @@
// right: clicked on frametop.screen.0 (right ray, aim 100 ms before, held 0.15 s)") or why not. A middle
// pinch (thumb to middle finger, fh_gestures v3) is a right click on a tap; held and moved, it
// scrolls: the virtual stick deflects with the hand's displacement from the begin (0.5 cm dead zone,
// full at 4 cm; up and down, and across the ray), times POINTER_RAY_SCROLL (1; -1 turns it round).
// A grip with the ray up does nothing. With the ray down, pinches and grips are gaze gestures, as
// below ("Hands"). A ray stays up while its pinch is held, and 250 ms after.
// full at 4 cm; up and down, and across the ray), times POINTER_RAY_SCROLL (1; -1 turns it round);
// in its first POINTER_RAY_TAP (0.3 s) only past 1.5 cm, so a right click that wobbles still clicks.
// A three-finger pinch (the thumb on the index and middle tips, both within POINTER_GAZE_SCROLL_GAP,
// as in gaze mode; whichever finger ft-hands calls it, at the begin or joining before anything went
// out) scrolls the same way and never clicks (user, 2026-10-15). A middle pinch's aim lock follows
// the thumb to middle distance. A grip with the ray up does nothing. With the ray down, pinches and
// grips are gaze gestures, as below ("Hands"). A ray stays up while its pinch is held, and 250 ms after.
// Smoothing (fh_ray.h, fh_smooth.h): the ray is read at this loop's time (At), not at the tracker's
// frames: its One Euro filters run in continuous time, so the pose sent to the driver moves every
// loop (about 100 Hz) instead of jumping at 15-30 Hz, which the dashboard's dragger turned into jerks.
@@ -1297,8 +1301,11 @@ int main() {
// ray that follows the pinching hand; side is the gesture's). released: the pinch ended before
// its press went out, which then clicks (released at releaseAt). For the log: took (it took
// the ray from the other hand), lockAge (the aim's age at the begin, s; < 0 none), the hit.
// rayThree: a three-finger pinch (the thumb on the index and middle tips, either kind): it
// scrolls as a middle pinch does, and never clicks.
int rayKind = 0, raySlot = -1;
bool pendingPress = false, scrolling = false, pushOn = false, aimSent = false, released = false, took = false;
bool rayThree = false;
Clock::time_point pressAt{}, releaseAt{}, beganAt{};
double lockAge = -1;
std::string pressHit;
@@ -2543,6 +2550,7 @@ int main() {
f.elbow = fv(h->elbow), f.elbowConf = h->elbowConf;
f.grip = gripDown[s];
f.gap = h->pinchD[0] >= 0 ? h->pinchD[0] : Length(h->pts[4] - h->pts[8]);
f.gap2 = h->pinchD[1] >= 0 ? h->pinchD[1] : Length(h->pts[4] - h->pts[12]);
f.curl = fhray::FistCurl(h->pts, [](Vec3 v) { return Length(v); });
// near: ft-screens says so (a fresh report), or the fingertip is at an overlay: a cast
// from 20 cm behind the index tip along the finger meets one within 25 cm (the tip
@@ -2564,7 +2572,7 @@ int main() {
}
}
rays[s].Frame(body, f, mayPoint(s), secsOf(tnow));
rayAims[s].Add(tc, rays[s].Current(), rays[s].Shown(), h->pinchD[0]);
rayAims[s].Add(tc, rays[s].Current(), rays[s].Shown(), h->pinchD[0], h->pinchD[1]);
}
}
for (int s = 0; s < 2; ++s) {
@@ -2891,6 +2899,12 @@ int main() {
out = hin::ToRoom(head, p);
return true;
};
// The thumb on both the index and middle tips (POINTER_GAZE_SCROLL_GAP): a three-finger pinch,
// whichever finger ft-hands calls it (it needs the thumb nearer the middle by 1 cm for a middle pinch).
auto threeFingers = [&](int side) {
return sideGap[side][0] >= 0 && sideGap[side][0] < gazeScrollGap && sideGap[side][1] >= 0 &&
sideGap[side][1] < gazeScrollGap;
};
// A ray pinch (see "Hand ray" at the top). side: the gesture's; rs: the ray's (the one that follows
// the pinching hand). The aim is locked where the ray pointed just before the fingers began to
// close (rayowner.h), and the dead zone counts from the ray now.
@@ -2914,7 +2928,8 @@ int main() {
};
angles(rays[rs].Current().dir, h.refYaw, h.refPitch);
fhray::Ray lock;
if (rayAims[rs].Lock(double(g.begin_ns) / 1e9, handOk && hg.end_m > 0 ? hg.end_m : 0.035, lock, h.lockAge))
if (rayAims[rs].Lock(double(g.begin_ns) / 1e9, handOk && hg.end_m > 0 ? hg.end_m : 0.035, lock, h.lockAge,
kind == 2 ? 1 : 0))
angles(lock.dir, yaw, pitch);
else
yaw = h.refYaw, pitch = h.refPitch, h.lockAge = -1;
@@ -2924,11 +2939,13 @@ int main() {
rayInactive = false;
const bool ours_ = laserOurs();
if (!ours_ && !claimPending && !claimHeld) claimPending = true, claimAt = tnow;
h.pendingPress = kind == 0;
h.rayThree = rayScroll != 0 && threeFingers(side); // a scroll, not a click (user, 2026-10-15)
h.pendingPress = kind == 0 && !h.rayThree;
h.pressAt = tnow + std::chrono::milliseconds(ours_ ? 0 : 30 + (claimHeld ? 0 : 8));
hold = h;
rays[rs].Hold(true, secsOf(tnow));
if (debug) std::printf("ray %s %s began\n", rs ? "right" : "left", kind ? "middle pinch" : "pinch");
if (debug)
std::printf("ray %s %s began\n", rs ? "right" : "left", h.rayThree ? "three-finger pinch" : kind ? "middle pinch" : "pinch");
if (debug) std::fflush(stdout);
};
// What a ray pinch did, for its log line.
@@ -2950,16 +2967,23 @@ int main() {
// A ray pinch ended (lost: by losing the hand, a grip, or the ray; why says which).
auto endRayHold = [&](bool lost, const char *why) {
const int side = hold.side, kind = hold.rayKind;
if (!lost && !hold.pressed && (kind == 0 || !hold.scrolling)) {
if (!lost && !hold.pressed && !hold.rayThree && (kind == 0 || !hold.scrolling)) {
// a tap before its press went out (or a middle tap): it clicks once the press can go
// (the press block below), along the locked aim
hold.released = hold.pendingPress = true;
return;
}
if (kind == 0 && hold.pressed && leftHeld) releaseLeft();
if (kind == 2 && hold.scrolling) SendTo(out, "ft_pointer", "scroll 0 0");
const bool scrolls = kind == 2 || hold.rayThree;
if (scrolls && hold.scrolling) SendTo(out, "ft_pointer", "scroll 0 0");
std::string what;
if (kind == 2 && hold.scrolling) what = "scrolled";
if (hold.rayThree) {
char b[96];
std::snprintf(b, sizeof b, "three-finger pinch: %s (stick up to %.2f)%s%s",
hold.scrolling ? "scrolled" : "no scroll, it didn't move", hold.scrollMax, lost ? ": " : "",
lost ? why : "");
what = b;
} else if (scrolls && hold.scrolling) what = "scrolled";
else if (!hold.pressed) what = std::string("no click: ") + why;
else if (lost) what = "pressed on " + hitName(hold.pressHit) + ", let go: " + why;
else what = (hold.engaged ? "dragged from " : "clicked on ") + hitName(hold.pressHit);
@@ -3095,7 +3119,7 @@ int main() {
// no laser up: no press (POINTER_PINCH_NO_RAY); the first one wakes the pointer
handUsed = tnow;
if (!active) wake(tnow);
if (kind == 0) pinchLog(s, kind, "ignored (no ray up: bring thumb and index together to raise the laser)");
if (kind == 0) pinchLog(s, kind, "ignored (no ray up: bring the thumb to the index or middle tip to raise the laser)");
} else {
beginHold(s, kind == 1, g);
}
@@ -3151,6 +3175,8 @@ int main() {
const int other = k == 0 ? 2 : k == 2 ? 0 : -1;
if (hold.gazeScroll && other >= 0 && (slots[other][s].flags & FH_PINCH_DOWN)) {
hold.right = other == 2; // a three-finger pinch: ft-hands calls the other finger now; it goes on
} else if (hold.src == Src::Ray && hold.rayThree && other >= 0 && (slots[other][s].flags & FH_PINCH_DOWN)) {
hold.rayKind = other; // (the same on the ray: both kinds' point is the knuckles)
} else if (hold.pressed && hold.second >= 0 && hold.second != s) {
// the two-hand press (see gazeTwoHand): the other hand's pinch still holds it, and drags now
hold.side = hold.second, hold.right = hold.secondKind == 2, hold.second = -1;
@@ -3164,6 +3190,7 @@ int main() {
for (PendingBegin &p : pendingBegins)
if (p.side == s && p.kind == k && ended) p.ended = true, p.lost = lost;
if (!began) continue;
if (k != 1 && hold.src == Src::Ray && hold.rayThree && hold.side == s) continue; // its other finger
if (!allowed) {
if (k != 1) pinchLog(s, k, "ignored (hands not allowed: a VR game, or the headset off)");
continue;
@@ -3271,16 +3298,30 @@ int main() {
}
}
}
// A middle pinch held and moved: scrolling, the stick deflected by the hand's displacement
// from the begin (0.5 cm dead zone, full at 4 cm): up and down, and across the ray.
if (hold.src == Src::Ray && hold.rayKind == 2) {
const fh_pinch_t &g = hg.pinch2[hold.side];
// The other finger joining a ray pinch just begun, before anything went out (no press, no
// scroll, not a tap already over): a three-finger scroll from here.
if (hold.src == Src::Ray && !hold.rayThree && !hold.pressed && !hold.scrolling && !hold.released &&
(hold.rayKind == 2 || hold.pendingPress) && rayScroll != 0 &&
tnow - hold.beganAt < std::chrono::milliseconds(400) && threeFingers(hold.side)) {
const fh_pinch_t &g = hold.rayKind == 2 ? hg.pinch2[hold.side] : hg.pinch[hold.side];
if ((g.flags & FH_PINCH_TRACKED) && rayRoom(g.point, hg.capture_ns, hold.beginPoint)) {
hold.rayThree = true, hold.pendingPress = false;
if (debug) std::printf("ray %s: the other finger joined, a three-finger pinch\n", hold.raySlot ? "right" : "left");
}
}
// A middle or three-finger pinch held and moved: scrolling, the stick deflected by the hand's
// displacement from the begin (0.5 cm dead zone, full at 4 cm): up and down, and across the ray.
// A middle pinch's first rayTap (0.3 s) is a tap's: it scrolls only past 1.5 cm then, so a
// right click that wobbles still clicks (2026-10-15: two of the user's turned into scrolls).
if (hold.src == Src::Ray && (hold.rayKind == 2 || hold.rayThree)) {
const fh_pinch_t &g = hold.rayKind == 2 ? hg.pinch2[hold.side] : hg.pinch[hold.side];
Vec3 now;
if ((g.flags & FH_PINCH_TRACKED) && rayRoom(g.point, hg.capture_ns, now)) {
const Vec3 d = now - hold.beginPoint;
const Vec3 right = Normalize(Cross(rayDir, {0, 1, 0}));
const double dx = Dot(d, right), dy = d.y;
if (!hold.scrolling && std::hypot(dx, dy) > 0.005) hold.scrolling = true;
const bool tap = !hold.rayThree && tnow - hold.beganAt < std::chrono::duration<double>(rayTap);
if (!hold.scrolling && std::hypot(dx, dy) > (tap ? 0.015 : 0.005)) hold.scrolling = true;
if (hold.scrolling) {
auto axis = [&](double v) {
const double a = std::clamp((std::fabs(v) - 0.005) / 0.035, 0.0, 1.0) * rayScroll;
@@ -3289,6 +3330,7 @@ int main() {
const double sx = std::clamp(axis(dx), -1.0, 1.0), sy = std::clamp(axis(dy), -1.0, 1.0);
if (sx != hold.scrollX || sy != hold.scrollY) {
hold.scrollX = sx, hold.scrollY = sy;
hold.scrollMax = std::max(hold.scrollMax, std::max(std::fabs(sx), std::fabs(sy)));
char msg[64];
std::snprintf(msg, sizeof msg, "scroll %.2f %.2f", sx, sy);
SendTo(out, "ft_pointer", msg);
@@ -3376,7 +3418,7 @@ int main() {
// The driving ray (see "Hand ray" at the top): the pointer's origin and direction. During a ray
// pinch the direction holds still until the ray has turned rayDeadzone, then follows (steer);
// a middle pinch holds it. Push and pull move the origin along the ray while a carry is held.
// a middle or three-finger pinch holds it. Push and pull move the origin along the ray while a carry is held.
// rayMode: a hand's ray has the pointer, driving it or (its owner's grace, rayowner.h) holding
// its last pose.
const bool rayDrives = active && raySide >= 0 && rays[raySide].Shown();
@@ -3390,7 +3432,7 @@ int main() {
if (hold.src == Src::Ray && hold.raySlot == raySide) {
// a tap's dead zone first: a quick pinch whose hand turned a little still clicks
const bool tap = tnow - hold.beganAt < std::chrono::duration<double>(rayTap);
if (hold.rayKind == 0) steer(ry, rp, tap ? std::max(rayDeadzone, rayTapDeadzone) : rayDeadzone, 1.0);
if (hold.rayKind == 0 && !hold.rayThree) steer(ry, rp, tap ? std::max(rayDeadzone, rayTapDeadzone) : rayDeadzone, 1.0);
} else {
yaw = ry, pitch = std::clamp(rp, -85.0, 85.0);
}
+12 -10
View File
@@ -28,8 +28,8 @@
// ft-hands has called it the other side for flipHold (0.3 s), or it's gone; then it takes the hand
// called its side that the other ray doesn't follow.
//
// Aim lock: where a ray pinch clicks. ft-hands sees a pinch once the thumb and index tips are
// within begin_m, and closing them turns the hand (1.3-1.6 deg median in the first 150 ms after the
// Aim lock: where a ray pinch clicks. ft-hands sees a pinch once the thumb and index tips (the
// middle's for a middle pinch) are within begin_m, and closing them turns the hand (1.3-1.6 deg median in the first 150 ms after the
// begin, 2.4-3.5 at the 90th percentile, on recordings; more before it). So the press goes out
// along the ray as it was just before the fingers began to close: the newest frame, within lockBack
// (0.35 s) before the begin, with the tips still lockOpen (1 cm) further apart than the pinch's
@@ -217,22 +217,24 @@ private:
class AimHistory {
public:
explicit AimHistory(Params p = {}) : p_(p) {}
// A hands frame at capture time t: the ray (filtered), whether it's up, and the thumb to index
// distance (m; < 0 unknown).
void Add(double t, const fhray::Ray &ray, bool shown, double pinchD) {
h_.push_back({t, ray, shown, pinchD});
// A hands frame at capture time t: the ray (filtered), whether it's up, and the thumb to index and
// thumb to middle distances (m; < 0 unknown).
void Add(double t, const fhray::Ray &ray, bool shown, double pinchD, double pinchD2 = -1) {
h_.push_back({t, ray, shown, {pinchD, pinchD2}});
while (!h_.empty() && h_.front().t < t - 1.0) h_.pop_front();
}
// The ray to press along for a pinch that began at tBegin (see the top); endD: the pinch's release
// distance (end_m). False: no frame with the ray up within lockBack. age: how long before the begin.
bool Lock(double tBegin, double endD, fhray::Ray &out, double &age) const {
// distance (end_m); finger: 0 the index pinched, 1 the middle. False: no frame with the ray up
// within lockBack. age: how long before the begin.
bool Lock(double tBegin, double endD, fhray::Ray &out, double &age, int finger = 0) const {
const double openD = endD + p_.lockOpen;
const int fi = finger == 1 ? 1 : 0;
const S *open = nullptr, *lead = nullptr;
for (auto it = h_.rbegin(); it != h_.rend(); ++it) {
if (it->t > tBegin + 1e-6) continue;
if (it->t < tBegin - p_.lockBack - 1e-6) break;
if (!it->shown) continue;
if (!open && it->d >= openD) open = &*it;
if (!open && it->d[fi] >= openD) open = &*it;
if (!lead && it->t <= tBegin - p_.lockLead + 1e-6) lead = &*it;
}
const S *pick = open ? open : lead;
@@ -254,7 +256,7 @@ private:
double t;
fhray::Ray ray;
bool shown;
double d;
double d[2];
};
Params p_;
std::deque<S> h_;
+20
View File
@@ -450,6 +450,26 @@ void TestReadyAndFist() {
double t = 0;
CHECK(RunUntil(r, b, t, 1.0, true, Pointing) >= 0, "no gap or curl known: the old gate");
}
{ // the middle counts too (user, 2026-10-15): its right click and the three-finger scroll
auto middle = [](double gap, double gap2) {
return [gap, gap2](double tt) {
HandFrame f = Pointing(tt);
f.gap = gap, f.gap2 = gap2, f.curl = 1.8;
return f;
};
};
Body b;
HandRay r(1);
double t = 0;
CHECK(RunUntil(r, b, t, 1.0, true, middle(0.12, 0.13)) < 0, "both 12-13 cm away: no laser");
CHECK(RunUntil(r, b, t, 1.0, true, middle(0.12, 0.04)) >= 0, "the thumb 4 cm from the middle tip: the laser");
CHECK(RunUntil(r, b, t, 0.6, false, middle(0.12, 0.08)) < 0, "8 cm from it (under readyHide): it stays");
CHECK(RunUntil(r, b, t, 1.0, false, middle(0.14, 0.14)) >= 0, "both past readyHide: it goes");
Body b2;
HandRay o(1);
t = 0;
CHECK(RunUntil(o, b2, t, 1.0, true, middle(-1, 0.04)) >= 0, "only the middle's gap known: it counts");
}
}
int main() {
+10
View File
@@ -290,6 +290,16 @@ void TestAimLock() {
AimHistory f;
for (double t = 1.0; t <= begin + 1e-9; t += kFrame) f.Add(t, RayAt(t), t > begin - 0.07, 0.01);
CHECK(f.Lock(begin, 0.035, r, age) && age <= 0.07 + 1e-9 && age > 0.03, "a ray just up: its oldest frame (%.3f)", age);
// a middle pinch: the thumb closing on the middle tip with the index held out (its distance
// never closes): the lock follows the middle's distance, as the index's for a pinch
AimHistory m;
for (double t = 1.0; t <= begin + 1e-9; t += kFrame) {
const double closing = std::clamp((t - (begin - 0.15)) / 0.15, 0.0, 1.0);
m.Add(t, RayAt(-3 * closing), true, 0.07, 0.06 - 0.04 * closing);
}
CHECK(m.Lock(begin, 0.035, r, age, 1) && std::fabs(YawDeg(r)) < 1.3 && age > 0.05 && age < 0.15,
"a middle pinch: from before the middle closed (%.2f deg, %.3f s)", YawDeg(r), age);
CHECK(m.Lock(begin, 0.035, r, age, 0) && age < kFrame, "by the index's distance: the begin's own frame (%.3f s)", age);
}
// fh_ray's ShowForPinch: a pinch brings back a ray that just dropped, not one sent away.
+1
View File
@@ -7,4 +7,5 @@ exec "$root/scripts/frame.sh" -C power 'set -e; mkdir -p build
. ../scripts/openvr.sh
g++ -std=c++17 -O2 -Wall -Wno-unused-parameter $OPENVR_CFLAGS \
-o build/ft-powerd ft-powerd.cpp $OPENVR_LIBS
strip --strip-unneeded build/ft-powerd
echo "built build/ft-powerd"'
+118
View File
@@ -0,0 +1,118 @@
"""The contributions log: what a contributor has uploaded, kept after the recordings are deleted.
A session's uploads are recorded in its own session.json (recorder/hub.py record_upload), so
deleting the session to free space would lose them. This log keeps one entry per finished upload
in BASE/contributions.json, next to profile.json:
{"schema": 1, "contributions": [
{"session", "repo", "pr_url", "pr_num", "uploaded", "export_sha", "export_bytes",
"bounty": {"id", "title"} | {}, "kind": "public" | "testing", "debug": bool,
"started", "minutes", "local_deleted": "<time>" | ""}]}
An entry is keyed by (repo, pr_num), or (repo, session, export_sha) for one without a pull request
number. hub.py adds the entry when an upload finishes; the window backfills uploads that older
recorders recorded only in session.json, and marks an entry when its local copy is deleted.
Nothing here leaves the Frame.
Standard library only."""
import datetime
import json
import os
SCHEMA = 1
FILE = "contributions.json"
def path(base):
return os.path.join(base, FILE)
def now_iso():
return datetime.datetime.now().astimezone().isoformat(timespec="seconds")
def load(base):
"""The log's entries, oldest first ([] when there's none or it can't be read)."""
try:
with open(path(base), encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError):
return []
items = data.get("contributions") if isinstance(data, dict) else None
return [c for c in items if isinstance(c, dict) and c.get("session")] if isinstance(items, list) else []
def _save(base, items):
os.makedirs(base, mode=0o700, exist_ok=True)
p = path(base)
tmp = p + ".tmp"
with open(tmp, "w", encoding="utf-8") as f:
json.dump({"schema": SCHEMA, "contributions": items}, f, indent=2)
f.write("\n")
os.replace(tmp, p)
def _key(c):
if c.get("pr_num") not in (None, ""):
return (c.get("repo", ""), "pr", str(c["pr_num"]))
return (c.get("repo", ""), c.get("session", ""), c.get("export_sha", ""))
def entry(session, meta, upload, export_bytes=0, minutes=None):
"""A log entry for a finished upload: session id, its session.json (meta), the upload's
record (session.json "uploads", status done), and what the export held when known."""
bounty = meta.get("bounty") if isinstance(meta.get("bounty"), dict) else {}
consent = meta.get("consent") if isinstance(meta.get("consent"), dict) else {}
debug = bool(meta.get("debug"))
return {"session": session, "repo": upload.get("repo", ""), "pr_url": upload.get("pr_url", ""),
"pr_num": upload.get("pr_num"), "uploaded": upload.get("uploaded", ""),
"export_sha": upload.get("export_sha", ""), "export_bytes": int(export_bytes or 0),
"bounty": {"id": bounty.get("id", ""), "title": bounty.get("title", "")} if bounty else {},
"kind": "testing" if consent.get("kind") == "testing" or debug else "public", "debug": debug,
"started": meta.get("started", ""), "minutes": minutes, "local_deleted": ""}
def done_uploads(meta):
"""session.json's finished uploads (records from before 2026-10-03 have no status: done)."""
return [u for u in meta.get("uploads") or [] if isinstance(u, dict)
and u.get("status", "done") == "done" and u.get("pr_url")]
def add(base, items):
"""Add entries (a list) the log doesn't have yet, and fill in what an existing entry lacks.
Returns how many were new."""
log = load(base)
known = {_key(c): c for c in log}
new = 0
for c in items:
old = known.get(_key(c))
if old is None:
log.append(dict(c))
known[_key(c)] = log[-1]
new += 1
else:
for k, v in c.items():
if k != "local_deleted" and v not in (None, "", {}, 0) and old.get(k) in (None, "", {}, 0):
old[k] = v
if items:
_save(base, log)
return new
def mark_deleted(base, session, when=None):
"""Note that a session's local copy was deleted (its entries stay)."""
log = load(base)
when = when or now_iso()
hit = False
for c in log:
if c.get("session") == session and not c.get("local_deleted"):
c["local_deleted"] = when
hit = True
if hit:
_save(base, log)
return hit
def sessions(base):
"""The session ids the log has a finished upload of."""
return {c["session"] for c in load(base)}
+1 -1
View File
@@ -76,7 +76,7 @@ class AgeTest(unittest.TestCase):
def test_files_and_recipients(self):
rec = age.read_recipients(os.path.join(HERE, "..", "..", "gaze", "rec", "recipients.txt"))
self.assertEqual(len(rec), 2)
self.assertEqual(len(rec), 3) # keys 1 (7i), 2 (offline), 3 (dody)
with tempfile.TemporaryDirectory() as d:
src, dst = os.path.join(d, "a"), os.path.join(d, "a.age")
with open(src, "wb") as f:
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/python3
"""recorder/window.py's _stay_awake: the awake unit runs while any export or upload does, even
when its workers run out of order (2026-10-17: a queued export's end and its upload's start come
together, and an export's stop could land after the upload's start). Needs PySide6 (the
standalone recorder's venv, or the dev container).
python3 recorder/tests/test_stay_awake.py
"""
import os
import sys
import tempfile
import types
import unittest
from unittest import mock
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, os.path.join(HERE, "..", ".."))
try:
from recorder import window
except ImportError: # no PySide6
window = None
@unittest.skipIf(window is None, "no PySide6")
class StayAwakeTest(unittest.TestCase):
def setUp(self):
class Backend(window.RecorderBackend):
AWAKE_UNIT = "ft-test-awake"
self.base = tempfile.mkdtemp(prefix="recorder-awake-test-")
self.backend = Backend(types.SimpleNamespace(base=self.base), types.SimpleNamespace())
self.workers = [] # held back, to run in any order
self.backend._thread = self.workers.append
self.calls = []
for name in ("start_unit", "stop_unit"):
p = mock.patch.object(window.units, name, side_effect=lambda *a, n=name, **k: self.calls.append(n))
p.start()
self.addCleanup(p.stop)
def tearDown(self):
os.rmdir(self.base)
def run_workers(self, order):
for i in order:
self.workers[i]()
self.workers.clear()
def test_on_then_off(self):
self.backend._stay_awake("export", True)
self.run_workers([0])
self.backend._stay_awake("export", False)
self.run_workers([0])
self.assertEqual(self.calls, ["start_unit", "stop_unit"])
def test_export_ends_as_its_upload_starts(self):
self.backend._stay_awake("export", True)
self.run_workers([0])
self.backend._stay_awake("upload", True)
self.backend._stay_awake("export", False)
self.run_workers([1, 0]) # the export's stop runs first
self.assertEqual(self.calls, ["start_unit"]) # never stopped: the upload holds it
def test_out_of_order_off(self):
self.backend._stay_awake("export", True)
self.backend._stay_awake("export", False)
self.run_workers([1, 0]) # the stop's worker runs before the start's
self.assertEqual(self.calls, []) # nothing to keep awake by then: never started
if __name__ == "__main__":
unittest.main()
+189 -17
View File
@@ -1,7 +1,7 @@
"""What the recorders' windows share (hands/rec/ft_handrec.py, gaze/rec/ft_eyerec.py): a QObject
backend with the consent and profile, the session's controls, export on a thread, the Hugging Face
login and upload in hub.py child processes, the Frame kept awake while they run, and small
helpers. Each recorder's Backend inherits it, adds its checklist, session and review, and keeps
backend with the consent and profile, the session's controls, export on a thread (one at a time,
the rest queued), the Hugging Face login and upload in hub.py child processes (an export can upload
by itself once it's done), the Frame kept awake while they run, and small helpers. Each recorder's Backend inherits it, adds its checklist, session and review, and keeps
these slot and property names, which its main.qml calls.
A "session" here is what the recorder records in one go: a hand session, or an eye round.
@@ -145,12 +145,21 @@ class RecorderBackend(QObject):
self._session_id = ""
self._status = {}
self._awake = set() # what keeps the Frame awake now: "export", "upload"
self._awake_on = False # the awake unit runs (set under _awake_lock)
self._awake_lock = threading.Lock()
self._export_cancel = None
self._export_thread = None
self._export_fraction = 0.0
self._export_text = ""
self._export_session = ""
# Waiting their turn: exports [{"session", "keep_notes"}], then uploads [session]. The
# sessions in _upload_after upload by themselves once exported (queued or running).
self._export_queue = []
self._upload_after = set()
self._upload_queue = []
# What the exports and uploads came to, newest last, for whoever comes back to the
# headset: [{"text", "error", "link"}].
self._done_log = []
self._statusArrived.connect(self._on_status)
self._exportProgress.connect(self._on_export_progress)
self._exportFinished.connect(self._on_export_finished)
@@ -197,6 +206,10 @@ class RecorderBackend(QObject):
the dataset."""
return {"SESSION": session}
def _session_label(self, session):
"""How the window names a session in a message."""
return session
def _keep_awake(self):
"""False: don't start the awake unit (a dry run)."""
return not self._hub_dry_run
@@ -369,6 +382,8 @@ class RecorderBackend(QObject):
self._session.stop()
except Exception:
pass
self._export_queue, self._upload_queue = [], []
self._upload_after.clear()
if self._export_cancel:
self._export_cancel.set()
self._export_thread.join(15)
@@ -376,7 +391,7 @@ class RecorderBackend(QObject):
if self._upload_proc:
self.cancelUpload()
self._upload_thread.join(10)
if self._awake and self.AWAKE_UNIT:
if (self._awake or self._awake_on) and self.AWAKE_UNIT:
try:
units.stop_unit(self.AWAKE_UNIT)
except Exception:
@@ -410,30 +425,130 @@ class RecorderBackend(QObject):
def _stay_awake(self, why, on):
"""Hold off sleep while an export or upload runs, so the headset can be taken off and
left plugged in: a host unit running systemd-inhibit, started with the first reason
and stopped with the last."""
and stopped with the last. The reasons change here, on this thread; each worker brings
the unit to what they are when it runs, so workers finishing out of order (an export's
end, then its upload's start) leave it right."""
if not self.AWAKE_UNIT or not self._keep_awake():
return
(self._awake.add if on else self._awake.discard)(why)
def run():
with self._awake_lock:
had = bool(self._awake)
(self._awake.add if on else self._awake.discard)(why)
want = bool(self._awake)
if want == self._awake_on:
return
try:
if self._awake and not had:
if want:
units.start_unit(self.AWAKE_UNIT, "keeps the Frame awake while exporting or uploading",
["systemd-inhibit", "--what=sleep:idle", "--mode=block",
"--who=" + self.APP, "--why=" + self.AWAKE_WHY, "sleep", "infinity"],
app=self.APP)
elif had and not self._awake:
else:
units.stop_unit(self.AWAKE_UNIT)
self._awake_on = want
except Exception as e:
print(f"keeping the Frame awake: {e}", file=sys.stderr, flush=True)
self._thread(run)
def _exporting(self, session):
return self.exporting and self._export_session == session
def _uploading(self, session):
return self.uploading and self._upload.get("session") == session
def _queued(self, session):
"""Waiting to export, or to upload."""
return (any(q["session"] == session for q in self._export_queue)
or session in self._upload_queue)
def _busy_guard(self, session):
"""Its export or upload runs or waits: its recordings and export stay until then."""
if self._exporting(session) or self._uploading(session) or self._queued(session):
self.message.emit("This session is exporting or uploading, or waiting to: "
"cancel that first (on the Export page)", True)
return True
return False
@Property("QVariantList", notify=exportChanged)
def exportQueue(self):
"""The exports waiting their turn, in order: [{"session", "upload"}]."""
return [{"session": q["session"], "upload": q["session"] in self._upload_after} for q in self._export_queue]
@Property("QVariantList", notify=exportChanged)
def uploadAfter(self):
"""The sessions that upload by themselves once exported (the running export's too)."""
return sorted(self._upload_after)
@Property("QVariantList", notify=uploadChanged)
def uploadQueue(self):
"""The exported sessions waiting to upload, in order."""
return list(self._upload_queue)
@Property("QVariantList", notify=exportChanged)
def doneLog(self):
return list(self._done_log)
def _log_done(self, text, error=False, link=""):
self._done_log = (self._done_log + [{"text": text, "error": error, "link": link}])[-20:]
self.exportChanged.emit()
@Slot()
def clearDoneLog(self):
self._done_log = []
self.exportChanged.emit()
@Slot(str, bool)
def exportSession(self, session, keep_notes):
if self.exporting or self._active_guard(session):
self.queueExport(session, keep_notes, False)
@Slot(str, bool, bool)
def queueExport(self, session, keep_notes, upload):
"""Export it now, or after the exports already waiting (one at a time). upload: once it's
exported, it uploads by itself (it waits for any upload before it)."""
# Another session's upload can go on: the export writes beside its own export (.partial)
# and only replaces that one at the end, so not while it's uploading or waiting to.
if self._uploading(session) or session in self._upload_queue or self._active_guard(session):
return
if upload:
self._upload_after.add(session)
if self._exporting(session) or any(q["session"] == session for q in self._export_queue):
self.exportChanged.emit() # already on its way: only whether it uploads changed
return
self._export_queue.append({"session": session, "keep_notes": keep_notes})
if self.exporting:
self.exportChanged.emit()
else:
self._next_export()
@Slot(str, bool)
def setUploadAfter(self, session, on):
"""Whether a queued or running export uploads by itself once it's done."""
if not (self._exporting(session) or any(q["session"] == session for q in self._export_queue)):
return
(self._upload_after.add if on else self._upload_after.discard)(session)
self.exportChanged.emit()
@Slot(str)
def unqueueExport(self, session):
"""Take a waiting export out of the queue (the running one: cancelExport)."""
self._export_queue = [q for q in self._export_queue if q["session"] != session]
self._upload_after.discard(session)
self.exportChanged.emit()
@Slot(str)
def unqueueUpload(self, session):
if session in self._upload_queue:
self._upload_queue.remove(session)
self.uploadChanged.emit()
def _next_export(self):
"""Start the first waiting export, if none runs."""
if self.exporting or not self._export_queue:
return
q = self._export_queue.pop(0)
self._start_export(q["session"], q["keep_notes"])
def _start_export(self, session, keep_notes):
cancel = threading.Event()
self._export_cancel = cancel
self._export_session = session
@@ -455,7 +570,9 @@ class RecorderBackend(QObject):
@Slot()
def cancelExport(self):
"""Cancel the running export (and its upload after); the next waiting one starts."""
if self._export_cancel:
self._upload_after.discard(self._export_session)
self._export_cancel.set()
self._export_text = "Cancelling"
self.exportChanged.emit()
@@ -466,23 +583,54 @@ class RecorderBackend(QObject):
self.exportChanged.emit()
def _on_export_finished(self, path, error):
session = self._export_session
send = session in self._upload_after
self._upload_after.discard(session)
self._export_cancel = None
self._stay_awake("export", False)
label = self._session_label(session)
if error == "cancelled":
self._export_text = "Cancelled: nothing was kept"
self._log_done(f"{label}: export cancelled")
elif error:
self._export_text = "Failed: " + error
self.message.emit("Export failed: " + error, True)
self._log_done(f"{label}: export failed: {error}", True)
else:
self._export_fraction = 1.0
self._export_text = f"Exported to {path} ({human_bytes(tree_bytes(path))})"
self.message.emit("Export ready", False)
self._log_done(f"{label}: exported ({human_bytes(tree_bytes(path))})")
self.exportChanged.emit()
self.refreshSessions()
self.exportDone.emit(self._export_session, not error)
self.exportDone.emit(session, not error)
if send and not error:
self._queue_upload(session)
self._next_export()
if not self.exporting:
self._stay_awake("export", False)
def _queue_upload(self, session):
"""An export done that uploads by itself: now, or after the upload running."""
if session not in self._upload_queue and not self._uploading(session):
self._upload_queue.append(session)
self.uploadChanged.emit()
self._next_upload()
def _next_upload(self):
"""Start the first waiting upload that still can, if none runs. One already uploaded
(the same export) isn't sent again; one that can't go now says why and leaves the queue."""
while not self.uploading and self._upload_queue:
session = self._upload_queue.pop(0)
if self.uploadInfo(session)["previous"].get("export_sha"):
self._log_done(f"{self._session_label(session)}: this export was uploaded before, so it wasn't sent again")
elif self._start_upload(session, False):
return
self.uploadChanged.emit()
@Slot(str)
def deleteExport(self, session):
if self._busy_guard(session):
return
try:
self.store.delete_export(session)
self.message.emit("Export deleted; the recordings stay", False)
@@ -674,13 +822,28 @@ class RecorderBackend(QObject):
return {"previous": {}, "uploads": 0}
return {"previous": before or {}, "uploads": count}
@Property(bool, notify=loginChanged)
def canUploadAfter(self):
"""An export can be set to upload by itself: logged in (or a dry run). Whether its
dataset takes contributions yet: uploadAllowedFor."""
return self._hub_dry_run or self._login.get("state") == "ok"
@Slot(str, bool)
def startUpload(self, session, again):
if self.uploading or self.exporting or self._active_guard(session):
return
if session in self._upload_queue:
self._upload_queue.remove(session) # pressed by hand: it goes now
self._start_upload(session, again)
self.uploadChanged.emit()
def _start_upload(self, session, again):
"""Start hub.py upload; False if it couldn't (and it said why)."""
# Not while this session exports again (its export is replaced at the end); another's is fine.
if (self.uploading or self._exporting(session) or any(q["session"] == session for q in self._export_queue)
or self._active_guard(session)):
return False
if not self._hub_dry_run and not self._upload_allowed_for(session):
self.message.emit("Contributions aren't open yet", True)
return
return False
argv = self._hub_argv("upload", session, "--json")
if self._hub_dry_run:
argv.append("--dry-run")
@@ -691,7 +854,7 @@ class RecorderBackend(QObject):
text=True, bufsize=1)
except OSError as e:
self.message.emit(f"Couldn't start the upload: {e}", True)
return
return False
self._upload_proc = proc
self._upload_cancelled = False
self._upload = {"session": session, "phase": "check", "text": "Starting", "fraction": 0.0, "log": "",
@@ -703,6 +866,7 @@ class RecorderBackend(QObject):
last = self._read_lines(proc, self._uploadLine, lambda: self._upload_cancelled, "The upload stopped")
self._uploadFinished.emit(last or {"error": {"kind": "cancelled", "text": "Cancelled"}})
self._upload_thread = self._thread(run)
return True
@Slot()
def cancelUpload(self):
@@ -726,7 +890,6 @@ class RecorderBackend(QObject):
def _on_upload_finished(self, last):
self._upload_proc = None
self._stay_awake("upload", False)
up = dict(self._upload)
if "done" in last:
result = last["done"]
@@ -734,6 +897,9 @@ class RecorderBackend(QObject):
text="Dry run: nothing was uploaded" if result.get("dry_run") else "Uploaded")
if not result.get("dry_run"):
self.message.emit("Uploaded: the pull request is open", False)
self._log_done(f"{self._session_label(up.get('session', ''))}: "
+ ("dry run, nothing sent" if result.get("dry_run") else "uploaded"),
link=result.get("pr_url", ""))
else:
e = last.get("error") or {}
cancelled = e.get("kind") == "cancelled"
@@ -742,9 +908,15 @@ class RecorderBackend(QObject):
if up.get("pr_url") else "Cancelled") if cancelled else e.get("text", ""))
if not cancelled:
self.message.emit("Upload failed", True)
self._log_done(f"{self._session_label(up.get('session', ''))}: upload "
+ ("cancelled" if cancelled else "failed: " + e.get("text", "")),
not cancelled, up.get("pr_url", "") or e.get("link", ""))
self._upload = up
self.uploadChanged.emit()
self.sessionsChanged.emit()
self._next_upload()
if not self.uploading:
self._stay_awake("upload", False)
@Slot(str, result=str)
def uploadText(self, session):
+3 -1
View File
@@ -25,6 +25,7 @@ $cc -c -o build/remote.o remote.c
cxx="g++ -std=c++17 -O2 -Wall -Wno-missing-field-initializers -Ibuild/include $(pkg-config --cflags egl glesv2 gbm libdrm)"
$cxx -c -o build/vr.o vr.cpp
$cxx -c -o build/keyboard.o keyboard.cpp
$cxx -c -o build/overlaytex.o overlaytex.cpp
$cxx -c -o build/handcut.o handcut.cpp
$cxx -c -o build/handrender.o handrender.cpp
$cxx -c -o build/handinput.o handinput.cpp
@@ -37,9 +38,10 @@ $cxx -c -o build/handskin.o handskin.cpp
$cxx -c -o build/handrobot.o handrobot.cpp
$cxx -c -o build/handview.o handview.cpp
vrlibs="$(pkg-config --libs egl glesv2 gbm) $OPENVR_LIBS"
g++ -o build/ft-screens build/compositor.o build/remote.o build/vr.o build/keyboard.o build/handcut.o build/handrender.o build/handinput.o build/palmmenu.o build/grabthat.o build/steampanels.o \
g++ -o build/ft-screens build/compositor.o build/remote.o build/vr.o build/keyboard.o build/overlaytex.o build/handcut.o build/handrender.o build/handinput.o build/palmmenu.o build/grabthat.o build/steampanels.o \
$(pkg-config --libs wlroots-0.20 wayland-server xkbcommon pixman-1) $vrlibs
g++ -o build/ft-handtest build/handtest.o build/handcut.o build/handrender.o $vrlibs
g++ -o build/ft-handview build/handview.o build/handmesh.o build/handskin.o build/handrobot.o build/handcut.o build/handrender.o $vrlibs
$cxx -o build/ft-spotpanel ../spotwatch/ft-spotpanel.cpp $(pkg-config --libs gbm libdrm) $OPENVR_LIBS -lpthread
strip --strip-unneeded build/ft-screens build/ft-handtest build/ft-handview build/ft-spotpanel
echo "built build/ft-screens build/ft-handtest build/ft-handview build/ft-spotpanel"'
+29 -1
View File
@@ -227,6 +227,7 @@ static void track_buffer(struct server *s, struct wlr_buffer *buffer) {
static bool relay_can_press(struct server *s);
static void release_relay_buttons(struct server *s, const char *why);
static void release_seat_buttons(struct server *s, const char *why);
// A video (or anything moving over a large area) on a screen you don't look at keeps the
// full frame rate (see frame_interval): its commits keep redrawing much of it, and keep
@@ -530,7 +531,15 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
// pointer until the next move: a press right after crossing onto another
// screen landed where the pointer had been. Entering one unit off makes the
// motion below go through.
wlr_seat_pointer_notify_enter(s->seat, surface, x + 1, y);
// With a button held (a window or a drag carried onto another screen), enter
// without wlroots' notify_enter: on a new surface it forgets the held buttons
// (reset_buttons), so their release was dropped and KWin kept the window moving
// until the next click. KWin is one client with a surface per screen, so the
// release belongs on the new one.
if (s->seat->pointer_state.button_count && !wlr_seat_pointer_has_grab(s->seat))
wlr_seat_pointer_enter(s->seat, surface, x + 1, y);
else
wlr_seat_pointer_notify_enter(s->seat, surface, x + 1, y);
s->pointer_focus = sc;
}
wlr_seat_pointer_notify_motion(s->seat, t, x, y);
@@ -572,6 +581,10 @@ static void handle_vr_event(const struct ft_event *e, void *data) {
case FT_LEAVE:
if (s->pointer_focus == sc) {
release_relay_buttons(s, "the pointer left the screens");
// Any other button still held (the 3D mouse's or a laser's press) is released
// here too: clearing the focus forgets it, its release would be dropped, and
// KWin would keep a move or drag going until the next click.
release_seat_buttons(s, "the pointer left the screens");
wlr_seat_pointer_notify_clear_focus(s->seat);
s->pointer_focus = NULL;
}
@@ -755,6 +768,21 @@ static void release_relay_buttons(struct server *s, const char *why) {
wlr_seat_pointer_notify_frame(s->seat);
}
// Release every button the seat still holds, to the surface it was pressed on. wlroots
// counts presses of the same button (n_pressed) and sends the release with the last one,
// so release the first button until the seat's list is empty.
static void release_seat_buttons(struct server *s, const char *why) {
struct wlr_seat_pointer_state *ps = &s->seat->pointer_state;
if (!ps->button_count || !ps->focused_surface) return;
while (ps->button_count) {
const uint32_t code = ps->buttons[0].button;
const bool last = ps->buttons[0].n_pressed == 1;
wlr_seat_pointer_notify_button(s->seat, now_ms(), code, WL_POINTER_BUTTON_STATE_RELEASED);
if (last) wlr_log(WLR_INFO, "button %u released (%s)", code, why);
}
wlr_seat_pointer_notify_frame(s->seat);
}
// Keys from the input relay (physical keyboards): "key <evdev code> <1 press|0 release>".
// They go to the screen KWin has keyboard focus on (the last one clicked), while typing
// goes to the desktop (keys_update). The release of a key the desktop got the press for
+1 -1
View File
@@ -136,7 +136,7 @@ void Detector::Frame(int64_t t, const std::vector<Hand> &hands, const Mat &head,
if (h->id != sd.handId) sd.recent.clear();
sd.lastSeen = t, sd.handId = h->id, sd.head = head;
const Features f = Measure(*h, head);
sd.thumb = Dist3(h->p[4], h->p[8]);
sd.thumb = std::min(Dist3(h->p[4], h->p[8]), Dist3(h->p[4], h->p[12])); // (the nearer: either pinch)
sd.recent.push_back({t, f.fwd, f.palmFwd});
while (!sd.recent.empty() && t - sd.recent.front().t > int64_t(p_.pullMs * 1e6)) sd.recent.erase(sd.recent.begin());
double palmNow[3];
+7 -5
View File
@@ -37,13 +37,15 @@
// pulling back never comes back far enough, a pinch keeps the other fingers open, and typing or a
// hand in the lap is never held out with the palm forward.
//
// The claim: in Closing, on frames where the thumb tip is claimThumb or more from the index tip,
// and for claimMs after a fire, the hand is ours. vr.cpp reports it near ("near ... <L> <R>" to the
// The claim: in Closing, on frames where the thumb tip is claimThumb or more from the index and
// the middle tips, and for claimMs after a fire, the hand is ours. vr.cpp reports it near ("near ... <L> <R>" to the
// pointer helper, as the palm menu does), so the grip ft-hands sees as the fist closes isn't a
// gaze grip (a press and drag at the pointer). ft-hands' grip begins with every finger under 1.2,
// after the three fingers' mean is under claimCurl (1.5), and with the thumb 3 cm or more from the
// index tip; a pinch begins under 2 cm, so a pinch made from a hand held out (a ray click with the
// other fingers curled) stays the helper's. On recordings without the gesture (probe-moves, the
// other fingers curled) stays the helper's, and so does a middle pinch (its right click: the ring and
// little curl with the middle; 2026-10-15, the user's right clicks were left to ft-screens). The
// user's flag keeps the thumb 3.5 cm or more from the middle tip as the fist closes. On recordings without the gesture (probe-moves, the
// 2026-09-29 and lit recordings, the PR #3 takes: 15 minutes), 1 of 143 pinches and 3 of 12 grips
// began on a claimed hand (11 pinches without the thumb rule), and nothing fired (one did before
// the palm rule above: an open palm turned over, then closed and pulled in); nor in the user's
@@ -76,7 +78,7 @@ struct Params {
double armMs = 200;
double closeMs = 450;
double claimCurl = 1.5; // the middle, ring and little fingers' mean extension
double claimThumb = 0.025; // ...with the thumb tip this far (m) from the index tip: not a pinch
double claimThumb = 0.025; // ...with the thumb tip this far (m) from the index and middle tips: not a pinch
double closePalm = 0.05, closePalmMs = 100;
double fist = 1.0;
double pull = 0.08, pullMs = 500; // m, within ms
@@ -183,7 +185,7 @@ private:
int64_t since = 0, lastSeen = 0, lastOpen = 0, quietUntil = 0, claimUntil = 0;
double eye[3] = {0, 0, 0}, palm[3] = {0, 0, 0}, normal[3] = {0, 0, 0};
Mat head{}; // the head on the last frame
double thumb = 0; // thumb tip to index tip now (m)
double thumb = 0; // thumb tip to the nearer of the index and middle tips now (m)
double holdFrom[3] = {0, 0, 0}; // the arming hold: where the palm was as it began,
double facingSum = 0; // ...its facing off the aim summed over its frames
int facingN = 0;
+2 -13
View File
@@ -16,6 +16,7 @@
// uploads a new texture each time, left the panel blank for a moment on every change;
// it's only the fallback if the buffers can't be made.
#include "keyboard.h"
#include "rounded.h"
#define STB_TRUETYPE_IMPLEMENTATION
#include "stb_truetype.h"
@@ -236,19 +237,7 @@ void Layout() {
// A rounded rectangle over what's there (straight alpha, like the other panels).
void FillRounded(int x0, int y0, int x1, int y1, double radius, const uint8_t rgba[4]) {
for (int y = y0; y < y1; ++y)
for (int x = x0; x < x1; ++x) {
const double cx = std::clamp(x + 0.5, x0 + radius, x1 - radius);
const double cy = std::clamp(y + 0.5, y0 + radius, y1 - radius);
const double cover = std::clamp(radius - std::hypot(x + 0.5 - cx, y + 0.5 - cy) + 0.5, 0.0, 1.0);
if (cover <= 0) continue;
uint8_t *p = &g_px[(size_t(y) * kW + x) * 4];
const double a = rgba[3] / 255.0 * cover, below = p[3] / 255.0;
const double out = a + below * (1 - a);
for (int c = 0; c < 3; ++c)
p[c] = uint8_t(std::lround(out > 0 ? (rgba[c] * a + p[c] * below * (1 - a)) / out : 0));
p[3] = uint8_t(std::lround(out * 255));
}
rounded::Fill(g_px.data(), kW, x0, y0, x1, y1, radius, rgba);
}
// ---------------------------------------------------------------- buffers
+141
View File
@@ -0,0 +1,141 @@
// See overlaytex.h.
#include "overlaytex.h"
#include <drm_fourcc.h>
#include <fcntl.h>
#include <gbm.h>
#include <unistd.h>
#include <cstdio>
#include <cstring>
#include <map>
#include <tuple>
namespace overlaytex {
namespace {
struct Image {
gbm_bo *bo = nullptr;
int fd = -1;
vr::SharedTextureHandle_t handle = 0;
int users = 0; // overlays showing it
uint64_t used = 0; // when last put on one (a counter)
};
using Key = std::tuple<int, int, uint64_t>; // w, h, the pixels' hash
int g_drm = -1;
gbm_device *g_gbm = nullptr;
int g_state = 0; // 0: not tried yet, 1: buffers, -1: SetOverlayRaw
std::map<Key, Image> g_images;
std::map<vr::VROverlayHandle_t, Key> g_on; // what each overlay shows
uint64_t g_clock = 0;
uint64_t Hash(const uint8_t *px, size_t n) { // FNV-1a, 8 bytes at a time
uint64_t h = 1469598103934665603ull;
size_t i = 0;
for (; i + 8 <= n; i += 8) {
uint64_t w;
std::memcpy(&w, px + i, 8);
h = (h ^ w) * 1099511628211ull;
}
for (; i < n; ++i) h = (h ^ px[i]) * 1099511628211ull;
return h;
}
void Drop(Image &im) {
if (im.handle) vr::VRIPCResourceManager()->UnrefResource(im.handle);
if (im.fd >= 0) close(im.fd);
if (im.bo) gbm_bo_destroy(im.bo);
im = Image{};
}
bool Ready() {
if (g_state) return g_state > 0;
g_drm = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC);
if (g_drm >= 0) g_gbm = gbm_create_device(g_drm);
g_state = g_gbm ? 1 : -1;
if (g_state < 0) std::printf("overlay textures: no render node; using SetOverlayRaw (at most 200 changes)\n");
return g_state > 0;
}
bool Make(Image &im, const uint8_t *px, int w, int h) {
im.bo = gbm_bo_create(g_gbm, uint32_t(w), uint32_t(h), GBM_FORMAT_ABGR8888, GBM_BO_USE_RENDERING | GBM_BO_USE_LINEAR);
if (!im.bo || (im.fd = gbm_bo_get_fd(im.bo)) < 0) return false;
uint32_t stride = 0;
void *mapping = nullptr;
auto *dst = static_cast<uint8_t *>(gbm_bo_map(im.bo, 0, 0, uint32_t(w), uint32_t(h), GBM_BO_TRANSFER_WRITE, &stride, &mapping));
if (!dst) return false;
for (int y = 0; y < h; ++y) std::memcpy(dst + size_t(y) * stride, px + size_t(y) * w * 4, size_t(w) * 4); // (ABGR8888: R, G, B, A in memory)
gbm_bo_unmap(im.bo, mapping);
vr::DmabufAttributes_t a{};
a.unWidth = uint32_t(w), a.unHeight = uint32_t(h);
a.unDepth = a.unMipLevels = a.unArrayLayers = a.unSampleCount = 1;
a.unFormat = DRM_FORMAT_ABGR8888;
a.ulModifier = DRM_FORMAT_MOD_LINEAR;
a.unPlaneCount = 1;
a.plane[0].unOffset = gbm_bo_get_offset(im.bo, 0);
a.plane[0].unStride = gbm_bo_get_stride(im.bo);
a.plane[0].nFd = im.fd;
return vr::VRIPCResourceManager()->ImportDmabuf(vr::VRApplication_Overlay, &a, &im.handle) && im.handle;
}
void Release(vr::VROverlayHandle_t o) {
auto on = g_on.find(o);
if (on == g_on.end()) return;
auto it = g_images.find(on->second);
if (it != g_images.end() && it->second.users > 0) --it->second.users;
g_on.erase(on);
}
// Past kKeep images: drop the least recently used no overlay shows.
void Trim() {
while (int(g_images.size()) > kKeep) {
auto old = g_images.end();
for (auto it = g_images.begin(); it != g_images.end(); ++it)
if (!it->second.users && (old == g_images.end() || it->second.used < old->second.used)) old = it;
if (old == g_images.end()) return;
Drop(old->second);
g_images.erase(old);
}
}
} // namespace
vr::EVROverlayError Set(vr::VROverlayHandle_t o, const uint8_t *px, int w, int h) {
if (o == vr::k_ulOverlayHandleInvalid) return vr::VROverlayError_UnknownOverlay;
if (!Ready()) return vr::VROverlay()->SetOverlayRaw(o, const_cast<uint8_t *>(px), uint32_t(w), uint32_t(h), 4);
const Key key{w, h, Hash(px, size_t(w) * h * 4)};
auto it = g_images.find(key);
if (it == g_images.end()) {
Image im;
if (!Make(im, px, w, h)) {
std::fprintf(stderr, "overlay textures: a %dx%d buffer failed; SetOverlayRaw this once\n", w, h);
Drop(im);
Release(o);
return vr::VROverlay()->SetOverlayRaw(o, const_cast<uint8_t *>(px), uint32_t(w), uint32_t(h), 4);
}
it = g_images.emplace(key, im).first;
}
const auto on = g_on.find(o);
if (on == g_on.end() || on->second != key) {
Release(o);
g_on[o] = key, ++it->second.users;
}
it->second.used = ++g_clock;
vr::Texture_t tex = {&it->second.handle, vr::TextureType_SharedTextureHandle, vr::ColorSpace_Gamma};
const vr::EVROverlayError err = vr::VROverlay()->SetOverlayTexture(o, &tex);
Trim();
return err;
}
void Forget(vr::VROverlayHandle_t o) { Release(o); }
void Shutdown() {
for (auto &[k, im] : g_images) Drop(im);
g_images.clear(), g_on.clear();
if (g_gbm) gbm_device_destroy(g_gbm);
if (g_drm >= 0) close(g_drm);
g_gbm = nullptr, g_drm = -1, g_state = 0;
}
} // namespace overlaytex
+31
View File
@@ -0,0 +1,31 @@
// Small CPU-drawn textures on ft-screens' overlays (the controls, the glows, the carousel) without
// SetOverlayRaw. SteamVR never gives back the memory block each SetOverlayRaw call takes: the 201st
// call in a process fails ("Refusing to create memory block because 201 blocks are already
// outstanding"), and the overlay keeps its old texture at its new size and place. That was the
// glow bug of 2026-10-15: after about 40 minutes of hand use, a corner's glow showed as the
// fingertip dot's disc, as big as the corner's glow, and "grab that"'s halo kept the last panel's
// shape. Measured with a probe: hidden, shown or transparent, destroyed or not, 20 s apart or
// not, every call keeps its block.
// Here each distinct image goes into a linear DMA-BUF SteamVR imports once (as the keyboard's
// buffers, keyboard.cpp), kept by its content, and the overlay switches to it with
// SetOverlayTexture, which has no such limit (600 switches, and 600 imports dropped again, fine).
// Images no overlay shows are dropped, the least recently used first, past kKeep of them.
#pragma once
#include <openvr.h>
#include <cstdint>
namespace overlaytex {
constexpr int kKeep = 64;
// w x h pixels, RGBA, straight alpha (as SetOverlayRaw took them), onto overlay o. Falls back to
// SetOverlayRaw if the buffers can't be made.
vr::EVROverlayError Set(vr::VROverlayHandle_t o, const uint8_t *px, int w, int h);
// The overlay is going (before DestroyOverlay): its image may be dropped.
void Forget(vr::VROverlayHandle_t o);
// Every image dropped (at shutdown, after the overlays are gone).
void Shutdown();
} // namespace overlaytex
+39
View File
@@ -0,0 +1,39 @@
// The keyboard's straight-alpha rounded rectangles, without SteamVR or GBM dependencies.
#pragma once
#include <algorithm>
#include <cmath>
#include <cstddef>
#include <cstdint>
namespace rounded {
inline void Fill(uint8_t *px, int width, int x0, int y0, int x1, int y1, double radius, const uint8_t rgba[4]) {
for (int y = y0; y < y1; ++y) {
const double cy = std::clamp(y + 0.5, y0 + radius, y1 - radius);
const double dy = y + 0.5 - cy;
for (int x = x0; x < x1; ++x) {
const double cx = std::clamp(x + 0.5, x0 + radius, x1 - radius);
const double dx = x + 0.5 - cx;
// Most pixels lie on a straight strip or in the centre. Keep the coverage
// arithmetic even there: fractional radii can round just below full coverage.
const double distance = dx == 0 ? std::abs(dy) : dy == 0 ? std::abs(dx) : std::hypot(dx, dy);
const double cover = std::clamp(radius - distance + 0.5, 0.0, 1.0);
if (cover <= 0) continue;
uint8_t *p = &px[(size_t(y) * width + x) * 4];
// A fully covered opaque key replaces its background. The empty background
// likewise takes a nonzero source unchanged, including its straight RGB.
if (cover == 1 && rgba[3] > 0 && (rgba[3] == 255 || p[3] == 0)) {
std::copy_n(rgba, 4, p);
continue;
}
const double a = rgba[3] / 255.0 * cover, below = p[3] / 255.0;
const double out = a + below * (1 - a);
for (int c = 0; c < 3; ++c)
p[c] = uint8_t(std::lround(out > 0 ? (rgba[c] * a + p[c] * below * (1 - a)) / out : 0));
p[3] = uint8_t(std::lround(out * 255));
}
}
}
} // namespace rounded
+19
View File
@@ -154,6 +154,7 @@ struct SynthHand {
double tiltDeg = 20; // fingertips up: the palm turns toward the head behind it
int curl = 0; // middle, ring and little: 0 open, 1 half curled, 2 a fist
int curlIndex = 0; // ...and the index, likewise (grab that's fist)
bool thumbOnMiddle = false; // ...the thumb tip on the middle's (a middle pinch)
double curlT = -1, curlIndexT = -1; // ...or in between (0..2: open, half, fist), a hand closing
// a pointing hand (PointingHand) whose middle finger the tracker has nearly straight: its tip
// this much deeper (toward where the index points) than the index tip (NaN: curled)
@@ -192,6 +193,7 @@ void PalmHand(const SynthHand &h, float out[21][3]) {
if (h.curlT >= 0)
for (int f = 0; f < 3; ++f) mix(&base[10 + 4 * f], half[f], fist[f], std::clamp(h.curlT, 0.0, 2.0), &out[10 + 4 * f]);
if (h.curlIndexT >= 0) mix(&base[6], halfIndex, fistIndex, std::clamp(h.curlIndexT, 0.0, 2.0), &out[6]);
if (h.thumbOnMiddle) out[4][0] = out[12][0] + 0.008f, out[4][1] = out[12][1], out[4][2] = out[12][2];
const double r = h.rollDeg * M_PI / 180, a = h.tiltDeg * M_PI / 180;
double c[3] = {0, 0, 0};
for (int j = 0; j < 21; ++j) {
@@ -3160,6 +3162,23 @@ void TestGrabThatGesture() {
for (int k = 1; k <= 6; ++k) q.Frame({StopHand(0.1, -0.15, -0.42 + 0.025 * k, k / 6.0, 2)});
CHECK(q.fires.empty());
}
{ // a middle pinch from a hand held out (a ray's right click, the other two curled with it): the
// pointer helper's, not claimed (2026-10-15: the user's right clicks were "left to ft-screens")
auto closing = [](bool middlePinch) {
GrabRig r;
for (int k = 0; k < 10; ++k) r.Frame({StopHand(0.1, -0.15, -0.42)});
for (int k = 0; k < 6; ++k) {
SynthHand h = StopHand(0.1, -0.15, -0.42, 0, 1);
h.thumbOnMiddle = middlePinch;
r.Frame({h});
}
return r.claimed[1];
};
const int fist = closing(false), pinch = closing(true);
std::printf("grab that: the three curled from a hand held out claim %d frames, with the thumb on the middle %d\n",
fist, pinch);
CHECK(fist >= 2 && pinch == 0);
}
{ // a wave: held out open, palm forward, swinging 15 cm either way at 1.5 Hz for 3 s
GrabRig r;
for (int k = 0; k < 90; ++k) {
+136
View File
@@ -0,0 +1,136 @@
// Compare every byte with the original keyboard renderer, without a headset or graphics stack.
#include "../rounded.h"
#include <chrono>
#include <cstdio>
#include <cstring>
#include <random>
#include <vector>
namespace {
void Original(uint8_t *px, int width, int x0, int y0, int x1, int y1, double radius, const uint8_t rgba[4]) {
for (int y = y0; y < y1; ++y)
for (int x = x0; x < x1; ++x) {
const double cx = std::clamp(x + 0.5, x0 + radius, x1 - radius);
const double cy = std::clamp(y + 0.5, y0 + radius, y1 - radius);
const double cover = std::clamp(radius - std::hypot(x + 0.5 - cx, y + 0.5 - cy) + 0.5, 0.0, 1.0);
if (cover <= 0) continue;
uint8_t *p = &px[(size_t(y) * width + x) * 4];
const double a = rgba[3] / 255.0 * cover, below = p[3] / 255.0;
const double out = a + below * (1 - a);
for (int c = 0; c < 3; ++c)
p[c] = uint8_t(std::lround(out > 0 ? (rgba[c] * a + p[c] * below * (1 - a)) / out : 0));
p[3] = uint8_t(std::lround(out * 255));
}
}
using Fill = decltype(&Original);
constexpr int kW = 1280, kH = 470;
uint64_t Hash(const std::vector<uint8_t> &px) {
uint64_t hash = 14695981039346656037ull;
for (uint8_t b : px) hash = (hash ^ b) * 1099511628211ull;
return hash;
}
// The real keyboard's rounded shapes, including hovered/pressed/latched colours and grip.
// Text and arrows use unchanged renderers; this fixture isolates the rounded rectangles.
void Keyboard(std::vector<uint8_t> &px, Fill fill, int state) {
std::fill(px.begin(), px.end(), uint8_t(0));
const uint8_t back[4] = {20, 22, 26, 225}, bar[4] = {220, 222, 228, uint8_t(state % 2 ? 235 : 120)};
fill(px.data(), kW, 0, 0, kW, kH, 22, back);
fill(px.data(), kW, kW / 2 - 130, 15, kW / 2 + 130, 27, 6, bar);
const std::vector<std::vector<double>> rows = {
{1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2},
{1.5, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1.5},
{1.75, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2.25},
{1.25, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1.75, 1, 1},
{1.5, 1.5, 7, 2, 1, 1, 1},
};
const double unit = double(kW - 24) / 15, row = double(kH - 40 - 12) / 5;
int index = 0;
for (size_t r = 0; r < rows.size(); ++r) {
double x = 12;
for (double units : rows[r]) {
uint8_t face[4] = {58, 62, 70, 255};
if (index == state - 1) face[0] = 26, face[1] = 115, face[2] = 232;
else if (index == state) for (int c = 0; c < 3; ++c) face[c] = uint8_t(face[c] + 38);
fill(px.data(), kW, int(std::lround(x + 3.5)), int(std::lround(40 + r * row + 3.5)),
int(std::lround(x + units * unit - 3.5)), int(std::lround(40 + (r + 1) * row - 3.5)), 10, face);
x += units * unit;
++index;
}
}
}
bool Check() {
// Every source channel/alpha pair over transparent pixels, including nonzero RGB at A=0.
for (double radius : {0., 0.5})
for (int a = 0; a < 256; ++a)
for (int c = 0; c < 256; ++c) {
uint8_t old[4] = {43, 129, 255, 0}, got[4];
std::copy_n(old, 4, got);
const uint8_t rgba[4] = {uint8_t(c), uint8_t(255 - c), uint8_t(c ^ 85), uint8_t(a)};
Original(old, 1, 0, 0, 1, 1, radius, rgba);
rounded::Fill(got, 1, 0, 0, 1, 1, radius, rgba);
if (std::memcmp(old, got, 4)) return false;
}
std::mt19937 rng(42);
std::vector<uint8_t> old(64 * 48 * 4), got(old.size());
for (int i = 0; i < 4000; ++i) {
for (auto &v : old) v = uint8_t(rng());
if (i % 3 == 0) for (size_t p = 3; p < old.size(); p += 4) old[p] = 0;
if (i % 3 == 1) for (size_t p = 3; p < old.size(); p += 4) old[p] = 255;
got = old;
const uint8_t color[4] = {uint8_t(rng()), uint8_t(rng()), uint8_t(rng()), uint8_t(i % 256)};
const int x = int(rng() % 40), y = int(rng() % 24), w = 1 + int(rng() % 24), h = 1 + int(rng() % 24);
const double radius = std::min(w, h) * (rng() % 501) / 1000.;
Original(old.data(), 64, x, y, x + w, y + h, radius, color);
rounded::Fill(got.data(), 64, x, y, x + w, y + h, radius, color);
if (old != got) return false; // compares untouched pixels outside the rectangle too
}
old.resize(kW * kH * 4), got.resize(old.size());
// Coverage near 1 must keep the original subtraction's rounding at larger coordinates.
for (auto &v : old) v = uint8_t(rng());
got = old;
const uint8_t color[4] = {191, 73, 225, 50};
Original(old.data(), kW, 31, 267, 51, 298, 2.84, color);
rounded::Fill(got.data(), kW, 31, 267, 51, 298, 2.84, color);
if (old != got) return false;
uint64_t hash = 0;
for (int state = 0; state <= 62; ++state) {
Keyboard(old, Original, state), Keyboard(got, rounded::Fill, state);
if (old != got) return false;
hash ^= Hash(got);
}
std::printf("65536 channel/alpha pairs at two coverages, 4001 varied rectangles and 63 keyboard states identical; hash %016llx\n",
static_cast<unsigned long long>(hash));
return true;
}
void Benchmark() {
std::vector<uint8_t> px(kW * kH * 4);
// Alternate order across rounds; checksum work stays outside the timed region.
for (int round = 0; round < 5; ++round)
for (int order = 0; order < 2; ++order) {
const bool original = (order + round) % 2 == 0;
const Fill fill = original ? Original : rounded::Fill;
Keyboard(px, fill, 0);
const auto start = std::chrono::steady_clock::now();
for (int i = 0; i < 20; ++i) Keyboard(px, fill, i);
const double ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - start).count() / 20;
std::printf("round %d %s %.6f ms hash %016llx\n", round, original ? "original" : "optimized", ms,
static_cast<unsigned long long>(Hash(px)));
}
}
} // namespace
int main(int argc, char **argv) {
if (!Check()) {
std::fputs("rounded renderer differs from the original\n", stderr);
return 1;
}
if (argc == 2 && std::strcmp(argv[1], "--benchmark") == 0) Benchmark();
}
+17 -13
View File
@@ -107,10 +107,12 @@
#include "grabthat.h"
#include "handinput.h"
#include "softglow.h"
#include "overlaytex.h"
#include "palmmenu.h"
#include "keyboard.h"
#include "roomview.h"
#include "steampanels.h"
#include "../spotwatch/reserve.h"
#include <drm_fourcc.h>
#include <openvr.h>
@@ -584,7 +586,7 @@ vr::VROverlayHandle_t MakeChrome(const char *key, const char *name, const std::v
std::fprintf(stderr, "openvr: can't create overlay %s: %s\n", key, vr::VROverlay()->GetOverlayErrorNameFromEnum(err));
return o;
}
vr::VROverlay()->SetOverlayRaw(o, const_cast<uint8_t *>(px.data()), uint32_t(w), uint32_t(h), 4);
overlaytex::Set(o, px.data(), w, h);
vr::VROverlay()->SetOverlayInputMethod(o, vr::VROverlayInputMethod_Mouse);
// SteamVR's laser and ComputeOverlayIntersection size the hit area from the mouse scale,
// not the texture: at the default 1x1 it's a square as tall as the control is wide, so the
@@ -599,7 +601,7 @@ void LightBar(Screen &s, bool lit) {
if (s.barLit == lit) return;
s.barLit = lit;
const auto &px = BarTexture(lit);
vr::VROverlay()->SetOverlayRaw(s.bar, const_cast<uint8_t *>(px.data()), 256, 24, 4);
overlaytex::Set(s.bar, px.data(), 256, 24);
}
// ---------------------------------------------------------------- wrist guides
@@ -646,7 +648,7 @@ struct GuidePart {
void Light(bool on, const std::vector<uint8_t> &px, int n) {
if (int(on) == lit) return;
lit = on;
vr::VROverlay()->SetOverlayRaw(overlay, const_cast<uint8_t *>(px.data()), uint32_t(n), uint32_t(n), 4);
overlaytex::Set(overlay, px.data(), n, n);
}
};
struct Guide { GuidePart ring, dot; };
@@ -1172,7 +1174,7 @@ void UpdateCorner(Screen &s, const Mat &p, const std::vector<Mat> &lasers) {
if (lit != s.cornerLit && s.handle != vr::k_ulOverlayHandleInvalid) {
s.cornerLit = lit;
const auto &px = HandleTexture(lit);
vr::VROverlay()->SetOverlayRaw(s.handle, const_cast<uint8_t *>(px.data()), 64, 64, 4);
overlaytex::Set(s.handle, px.data(), 64, 64);
}
if (std::fabs(glow - s.cornerGlow) > 0.02f || (glow == 0) != (s.cornerGlow == 0)) {
s.cornerGlow = glow;
@@ -1653,7 +1655,7 @@ bool EnsureFloatPanel(Screen &s) {
void DropFloatPanel(Screen &s) {
for (auto o : s.All())
if (o != vr::k_ulOverlayHandleInvalid) vr::VROverlay()->DestroyOverlay(o);
if (o != vr::k_ulOverlayHandleInvalid) overlaytex::Forget(o), vr::VROverlay()->DestroyOverlay(o);
s.overlay = s.bar = s.handle = s.curveButton = s.rollButton = s.dockButton = s.closeButton = s.resetButton =
vr::k_ulOverlayHandleInvalid;
s.visible = s.controlsUp = s.barLit = s.cornerLit = false;
@@ -2814,7 +2816,7 @@ void StartGrabMove(int index, Screen &s, const Mat &to, double width, double ms
// ---------------------------------------------------------------- the front slot
// One place in front of you for the lazy susan (user, 2026-10-11: "make the system a bit more logical
// with the lazy susan"; the proposals they took): a stroke (or Meta+Alt+Tab) brings the next screen
// there, 1 m ahead (g_frontReading), as big as its resolution needs for its text to read
// there, 1 m ahead (g_frontReading; never nearer than behind spotwatch's reserved spot), as big as its resolution needs for its text to read
// (grabthat::ReadableSize), one at a time: the one there goes home first. Turned away from, it goes
// home, unless you used it a while (g_frontKeepMs of input on it: kept). Resized there, it comes that
// size next time (g_frontDeg). Its home: where it lived before it came.
@@ -3440,7 +3442,7 @@ void ShowCarousel() {
vr::VROverlay()->CreateOverlay("frametop.carousel", "Frametop: screens", &g_carousel.overlay) != vr::VROverlayError_None)
return;
const vr::VROverlayHandle_t o = g_carousel.overlay;
vr::VROverlay()->SetOverlayRaw(o, px.data(), uint32_t(W), uint32_t(H), 4);
overlaytex::Set(o, px.data(), W, H);
// under the screen in the slot, past its bar: 0.45 mm a pixel at 1 m (30 px text: about 0.8 deg)
const Screen &s = g_screens[g_front.panel];
const double d = std::max(0.3, Apart(head, g_front.to));
@@ -3520,7 +3522,7 @@ void ShowResetAsk(int panel) {
if (g_resetAsk.overlay == vr::k_ulOverlayHandleInvalid &&
vr::VROverlay()->CreateOverlay("frametop.resetask", "Frametop: reset the layout?", &g_resetAsk.overlay) != vr::VROverlayError_None)
return;
vr::VROverlay()->SetOverlayRaw(g_resetAsk.overlay, px.data(), uint32_t(W), uint32_t(H), 4);
overlaytex::Set(g_resetAsk.overlay, px.data(), W, H);
vr::VROverlay()->SetOverlaySortOrder(g_resetAsk.overlay, 10);
g_resetAsk.panel = panel, g_resetAsk.shownMs = NowMs(), g_resetAsk.px = W;
g_resetAsk.tipZ[0] = g_resetAsk.tipZ[1] = 1;
@@ -3727,7 +3729,7 @@ void UpdateTargetGlow() {
const int h = std::clamp(int(std::lround((kTargetW - 2 * m) * H / std::max(1e-6, W))) + 2 * m, 2 * m + 4, 2048);
if (kTargetW != g_target.texW || h != g_target.texH || m != g_target.texM) {
const std::vector<uint8_t> px = softglow::TargetTexture(kTargetW, h, m);
vr::VROverlay()->SetOverlayRaw(g_target.overlay, const_cast<uint8_t *>(px.data()), kTargetW, uint32_t(h), 4);
overlaytex::Set(g_target.overlay, px.data(), kTargetW, h);
g_target.texW = kTargetW, g_target.texH = h, g_target.texM = m;
static int logged = -1;
if (logged != g_target.panel)
@@ -3844,7 +3846,7 @@ void UpdateHandGuides() {
if (textures.size() > 48) textures.clear();
t = textures.emplace(key, softglow::GuideTexture(hg.g, tw, th, wm, halo, arm, r, edge)).first;
}
vr::VROverlay()->SetOverlayRaw(hg.overlay, const_cast<uint8_t *>(t->second.data()), uint32_t(tw), uint32_t(th), 4);
overlaytex::Set(hg.overlay, t->second.data(), tw, th);
hg.key = key;
}
vr::VROverlay()->SetOverlayWidthInMeters(hg.overlay, float(wm));
@@ -4270,7 +4272,8 @@ bool ft_vr_init(void) {
g_curveMaxArc = deg * M_PI / 180;
}
g_susanBring = handcut::Setting("SCREEN_SPIN_BRING", "1") != "0"; // the lazy susan brings panels (0: the room turns)
num("SCREEN_SPIN_DISTANCE", g_frontReading, 0.4, 3); // the front slot's reading distance (m)
num("SCREEN_SPIN_DISTANCE", g_frontReading, 0.4, 3); // the front slot's reading distance (m)...
g_frontReading = std::max(g_frontReading, spotreserve::kDistance + spotreserve::kFrontGap); // ...behind spotwatch's spot
num("SCREEN_SPIN_PPD", g_frontRead.ppd, 8, 80); // ...as big as its pixels at this many per degree need
num("SCREEN_SPIN_MAX_DEG", g_frontRead.maxW, 30, 100); // ...at most this wide (deg; wider ones curve round you)
num("SCREEN_SPIN_MAX_TALL_DEG", g_frontRead.maxH, 20, 90);
@@ -4351,7 +4354,7 @@ bool ft_vr_init(void) {
if (vr::VROverlay()->CreateOverlay("frametop.catcher", "Frametop: release catcher", &g_catcher) ==
vr::VROverlayError_None) {
static std::vector<uint8_t> clear(4 * 4 * 4, 0);
vr::VROverlay()->SetOverlayRaw(g_catcher, clear.data(), 4, 4, 4);
overlaytex::Set(g_catcher, clear.data(), 4, 4);
vr::VROverlay()->SetOverlayInputMethod(g_catcher, vr::VROverlayInputMethod_Mouse);
vr::VROverlay()->SetOverlayAlpha(g_catcher, 0);
vr::VROverlay()->SetOverlayFlag(g_catcher, vr::VROverlayFlags_MakeOverlaysInteractiveIfVisible, true);
@@ -4385,6 +4388,7 @@ void ft_vr_shutdown(void) {
keyboard::Destroy();
std::this_thread::sleep_for(std::chrono::milliseconds(200));
for (auto &[k, h] : g_imports) vr::VRIPCResourceManager()->UnrefResource(h);
overlaytex::Shutdown();
g_guides.clear();
g_screens.clear();
g_imports.clear();
@@ -4580,7 +4584,7 @@ void ft_vr_screen_destroy(int index) {
const Screen &s = it->second;
if (s.remote && ScreenPose(s, &p)) g_parked[index] = {p, s.metres, s.curve, s.pinned, s.pinRel, s.alone};
for (auto o : it->second.All())
if (o != vr::k_ulOverlayHandleInvalid) vr::VROverlay()->DestroyOverlay(o);
if (o != vr::k_ulOverlayHandleInvalid) overlaytex::Forget(o), vr::VROverlay()->DestroyOverlay(o);
for (auto &[k, sub] : it->second.subs) vr::VROverlay()->DestroyOverlay(sub.overlay);
g_screens.erase(it);
}
+2
View File
@@ -206,6 +206,7 @@ fi
# looks wrong first. It wakes an idle gaze service for about 20 s, to see the tracker send.
if has gaze; then
python3 "$repo/scripts/gaze-report.py" </dev/null 2>&1 || echo "scripts/gaze-report.py failed"
python3 "$repo/models/ft-models" list eyes 2>&1 | head -n 12
fi
# Hand tracking: whether it runs, which models ft-hands loaded and whether they're the pinned ones,
@@ -220,6 +221,7 @@ if has hands; then
[ -d "$student" ] && "$repo/hands/models.sh" check 2>&1 | tail -n 3
echo "last checked for newer ones: $(date -r "$repo/hands/models/.student-checked" '+%F %T' 2>/dev/null || echo never)"
grep -E '^\s*(FT_)?HANDS_MODELS(_UPDATE)?=' ~/.config/frametop.conf 2>/dev/null | sed 's/\s*#.*//; s/^/override (ft-hands loads these instead): /'
python3 "$repo/models/ft-models" list hands 2>&1 | head -n 12
python3 "$repo/hands/camcheck.py" 2>&1 | head -n 30
echo "sides: $(cat "$XDG_RUNTIME_DIR/frametop-hands/sides.json" 2>/dev/null || echo 'no sides.json')"
for c in 5 6 7; do # ft-hands' CPUs (HANDS_CPUS)
Loaded 100 of 108 files, more files were not shown because too many files have changed in this diff. Show more