The search opened 1 m ahead, where the front slot puts a spun panel, so the two shared a
plane. It now opens 0.75 m ahead at the same angular size (0.42 m wide), and ft-screens
keeps the front slot at least 0.15 m behind it (SCREEN_SPIN_DISTANCE is raised to 0.9 if
set nearer). Its sort order (14) stays over the panels', so a panel nearer than it still
doesn't cover it. The numbers live in spotwatch/reserve.h, shared by both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SteamVR never gives back the memory block each SetOverlayRaw call takes,
so ft-screens' 201st call failed and overlays kept stale textures at new
sizes: corner glows showed as big fingertip-dot discs and "grab that"'s
halo kept another panel's shape (2026-10-15 recording, about 40 minutes
in). Each distinct image now goes into a linear DMA-BUF imported once,
shared by content, dropped least recently used past 64.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user, 2026-10-15: right clicks were hard and the dot kept going away.
The laser's "ready" gate now opens with the thumb near the index or the
middle tip, so a middle pinch (right click) or a three-finger pinch has a
laser to aim with. A middle pinch's aim lock follows the thumb to middle
distance.
A three-finger pinch (thumb on the index and middle tips, the gaze
scroll's POINTER_GAZE_SCROLL_GAP) on the ray scrolls as the hand moves and
never clicks, whichever finger ft-hands calls it, at the begin or joining
before the press went out. A middle tap's first POINTER_RAY_TAP scrolls
only past 1.5 cm, so a right click that wobbles still clicks (two of the
day's turned into scrolls).
Grab that's claim skipped a thumb on the middle tip, so a middle pinch
from a hand held out (the ray's pose) looked like the fist closing and
was left to ft-screens (13 of the day's middle pinches). It now needs the
thumb off both tips; the user's flag keeps it 3.5 cm or more from the
middle tip as the fist closes, and still fires twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KWin 6.2.5 (SteamOS 0.4.5) draws each output straight onto the toplevel,
which note_layer_buffer skipped, so every panel stayed empty there. The
toplevel's own DMA-BUF is now noted in screen_commit and competes on size
like a child's; KWin 6.6's 1x1 single-pixel buffer isn't a DMA-BUF, so it
never wins. The surface watcher leaves toplevels to screen_commit, so the
order of the two commit listeners doesn't matter.
A child surface destroyed between its commit and its toplevel's no longer
leaves latest_surface pointing at freed memory (a desynchronized child can).
update-check: libkwin before 6.6 has no host-protocol checks, so that's
ok, not a warning on every current Frame; each libkwin file is read once
(the two symlinks named the same 8.8 MB file).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Looking for ksmserver with pgrep every 3 seconds for the whole session cost
about 38 ms a look, 1.3% of a core. The watcher now looks only until Plasma
has started it (at most 2 minutes), then waitpid sleeps until it exits
(a /proc check every 3 seconds where util-linux has no waitpid). Under the
script's set -e a failed waitpid no longer ends the watcher.
The KWin wrapper is matched from the start of its command line: pgrep -f
kwin_wayland_wrapper also matched any shell whose command named it.
docs/design.md: the watcher, under Other session behavior.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Off by default until it's been tried in the headset. The fit check keeps
a good fit as yours (fit-baseline.json), as the eye recorder's fitment
does. With GAZE_FIT_WATCH=1:
- The quick check that opens when the headset goes on first holds its
dot while the fit coach reads the fit against the saved one. Moved
past 5 degrees, the panel widens and says which way to move it, live,
until it's back; a click or 30 s takes the dot as it sits.
- With our tracker in use and gaze mode on, ft-eyes' glint slip (status
now has jump_px) past 5 degrees for 2 s shows a small strip low in
view ("show note" in ft-gazepanel), for 8 s or until the slip is gone,
at most once every 2 minutes.
gaze/test/fit-watch-test.py tests both offline. The eye recorder's
redo test waited on a state the step before left, and missed the redo
about 1 run in 8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
install.sh offers hand tracking (step 9/12, yes by default; --no-hands skips it,
and --yes installs it only when sudo runs without a prompt). Installed, it runs
with the Frametop desktop (HANDS_MODE=frametop), unless the settings already say
always. get.sh, pack/install-release.sh and the FrameDrop window pass --no-hands,
and the window has a hand tracking box that needs the password like the eye
tracker's.
The release image builds ft-camd and ft-hands (with ncnn at the tag
hands/Makefile pins), and hands/run.sh install doesn't build in a release, so
it only sets ft-camd's file capabilities and installs the units. The release
workflow's smoke test checks both binaries. Docs drop "deferred" and describe
the gestures and the Hands page.
The monitor's closest listed mode, and HDR off before the resolution
switch. frametop-2.0.0-1 joins the older builds the setup replaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
host/macos: an instance follows its monitor by maker, model and serial,
waits while it's unplugged, and a new monitor needs the setup again.
host/windows: new monitors show up in Add displays; how long a virtual
monitor lives. remote-displays.md: the Mac runs one instance per display
now. hands/README: SteamVR's panels take taps, not grabs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reverts 797e5c7. With pointer/dashpatch, the treadmill moves SteamVR's
windows by their grab bars, so the default goes back to the role that
leaves both hands to the controllers. docs/design.md explains the cause and
the patch; reference.md lists the patch; update-check.py checks that the
dashboard still has the function the patch wraps and that its debugger is
reachable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As a treadmill (the default since 564bb4b), the 3D mouse's laser clicks and
resizes SteamVR's windows but can't move them by their grab bars: the
compositor's window grab follows only /user/hand/left or /user/hand/right,
and logs nothing. With the right hand's role the same laser moves them
(headset test 2026-10-13, SteamVR 2.18.2). The driver's default role is 2
again and POINTER_ROLE defaults to right; treadmill stays a setting.
update-check warns when the device is at /user/treadmill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Remote displays section in the README with the three host kits; host/windows/README.md
and host/linux/README.md beside the Mac's; reference.md names frametop-2.0.0-2 for the
resolution switch; remote-displays.md is the design notes now. No machine names in hands/README.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
X11 windows get the layout's coordinates times its largest scale, which wrap
past 32767. The 4th spare at x 27000 and scale 1.25 put Brave at x -31506:
KWin focused it on a click, but the click never reached it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spotwatch's "spin to N" still turned the whole ring after the merge, the panel in the
front slot with it, while next and prev bring one panel to the slot. With
SCREEN_SPIN_BRING on (the default) it now brings panel N there too; the ring turn stays
for SCREEN_SPIN_BRING=0 and for degrees. Docs: reference.md's spin, spotwatch.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hands/rec/demo: ft-handdemo, sheets.py and gifs.py recorded the demo clips in the
headset, dropped on 2026-10-12 for generated skeleton clips. camgifs.py keeps its
gifsicle lookup; the README covers what's left.
- hands/.gitignore: rec/demos-recorded/, the recorded clips show the recorder's room.
- spotwatch/test/render.py -> spotwatch/tools/render.py: it needs an output path, and
just test-python ran every spotwatch/test/*.py with no arguments.
- voice/README.md: the advanced example's hook is the repo's hooks/on-command, not a
path in one person's home.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frametop Streams lists the Mac's displays (its Sunshine instances) and turns
each one, or all of them, on or off. ft-sunshine off|on|status does the same
in Terminal. Off disables the instance's LaunchAgent, so it stays off through
logins until on. The setup compiles the app from Frametop Streams.applescript
(AppleScriptObjC, no compiler) and starts it at login; --undo removes it.
README.md is the Mac setup guide, also the notes of the kit's release.
Fifth live test (2026-10-10 22:55): grab that brought the desktop in, it was
carried and thrown 3 m out, then five gestures at it or the remote displays
logged "no panel there". The open palm is held out below the line of sight,
so the line from the eyes through it runs 12-15 deg under where the head
points (the user's flag 150115). Since the fourth live test the line had to
meet a panel exactly, and a thrown 1.35 m tall desktop 3 m out is only
12.7 deg each way, so the line passed just under it. One of the flag's two
fires misses such a panel that way.
Pick now takes the panel the line meets (unchanged, nearest wins), else the
one whose outline it passes nearest within HANDS_GRAB_THAT_MARGIN_DEG
(6 deg seen from the eyes; 0 = the old rule). The gesture itself is
unchanged, so it stays as hard to trigger, and held, pinned, pressed and
near panels are still never taken. "no panel there" now logs where the line
went relative to the head and the nearest panel it may take, with its angle.
Test: TestGrabThatAfterThrow runs the user's sequence with synthetic hands:
grab that brings the desktop in, a bar carry is thrown out and up at
1.3 m/s, then grab that at it again (missed by 0.8 deg with the old rule,
taken now), a remote display after that, precision cases, and the flag's
two fires (2 of 2 now, 1 with the old rule).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The example config gets HANDS_FIST_REACH, HANDS_JOIN_REACH, HANDS_BUTTON_REACH,
HANDS_PINCH_BUTTONS, HANDS_EDGE_GLOW, HANDS_GRAB_THAT_FROM, HANDS_GRAB_THAT_MARGIN and
HANDS_GRAB_THAT_FACING; hands/README.md and docs/reference.md describe them, the white
halo, the corner, edge and button glows, and that near panels go away by push, flick or
throw now, not by grab that. `handinput state` counts joins, pinches that waited to be
fists, and buttons pinched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/README.md (near input, the palm menu), docs/reference.md and session/frametop.conf.example:
HANDS_PUSH, HANDS_PUSH_SPEED, HANDS_STRETCH_TURN, HANDS_TRACK_GATE, HANDS_TARGET_GLOW,
HANDS_SEND_AWAY_WIDTH; fists ft-hands calls pinches, the pinch slack, holds through the helper's
word, the near-the-eyes carry, the stricter palm menu, with the replays' numbers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
session/frametop.conf.example: HANDS_POKE_TRAVEL=0.017, the card
pinch's fingers, and the new HANDS_HOLD_LOST_MS, HANDS_FLICK,
HANDS_FLICK_SPEED, HANDS_FLICK_OPENING, HANDS_THROW, HANDS_THROW_SPEED,
HANDS_THROW_DISTANCE and POINTER_GAZE_CONTROL_HOLD. scripts/conf-migrate.sh
moves the example's exact HANDS_POKE_TRAVEL=0.025 line (hands-touch,
never released) to 0.017. hands/README.md's near input section,
docs/reference.md's .resize, settings and handinput state, and the
gesture tables at the top of handinput.h and vr.cpp say what each
gesture does now and what wins. The README notes that the hand glows
are drawn into the cutout buffer, so they don't show with the cutouts
off (the default since the drawn hands).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
session/frametop.conf.example: HANDS_NEAR_REACH=0.075, the new
HANDS_POKE_TRAVEL, HANDS_GRAB_BAND, HANDS_PINCH_CLICK, HANDS_CARD_PINCH and
HANDS_FIST_GRAB. scripts/conf-migrate.sh moves the example's exact 0.30
line (hands-reach, never released) to 0.075. hands/README.md's near input
section, docs/reference.md's settings and handinput state, and the
gesture table at the top of handinput.h and vr.cpp say what each gesture
does near a panel and what wins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- ft-pointer.cpp, the gaze pinch comment: typing's "the rules for all
gestures hold (..., typing, ...)" kept, gazechord's chord and
POINTER_GAZE_PINCH paragraphs added after it, and a line that a hand
typing holds back makes no chord.
- ft-pointer.cpp, the hands settings: gazechord's gazePinchOn, nearOn,
handsTrack kept; typingHold dropped for typing's typing::Params.
- ft-pointer.cpp, mayPoint: both conditions (handsOn, and not a side
typing holds back).
- ft-pointer.cpp, beginGazePinch: gazechord's busy line (a chord isn't
busy), without the old sinceKey/typingHold lines. The chord's pinch
goes through typing.h's Why in the gesture loop before startGesture,
like any pinch, so a hand typing holds back can't make one.
- docs/reference.md settings line: typing's and reach's keys, plus
POINTER_GAZE_PINCH.
- hands/README.md near input: typing's hold-back sentence and
gazechord's HANDS_NEAR=on.
- session/frametop.conf.example: reach's HANDS_GRAB_ROTATE* and
HANDS_GRAB_THAT* lines, then gazechord's HANDS_NEAR line.
Reconciled: gazehand's Situation loses sinceKey and typingHold, and
Refused its "ignored (typing)": the helper had stopped setting them on
hands-typing (typing.h holds hands back first), so that guard could
never fire. gazehand-test's key checks now drive typing.h: keys hold a
hand back (2 s now, was 1), POINTER_PINCH_TYPING=0 is no guard, and a
chord's pinch by a hand on the keys is held back while one lifted off
them goes ahead. The chord's own pinch counts as a gesture held for
typing's "busy" (free if out of the keyboard zone when keys come), as
the left's hold does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict: docs/reference.md's hands settings line (typing's POINTER_TYPING_*
keys next to reach's HANDS_NEAR_REACH, HANDS_GRAB_ROTATE* and
HANDS_GRAB_THAT*): both kept.
Typing and grab that: a hand typing holds back (the helper's "typing" word)
doesn't arm or fire "grab that" either, so it never claims a fist as near;
vr.cpp's UpdateGrabThat leaves it out, as Poke and Grab already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hold an open hand out to a screen, remote display or floating window,
palm to it, then pull the arm back making a fist: the panel comes over
250 ms to 45 cm ahead of the eyes, at the fist's height, facing you
(user, 2026-10-10). Screens wider than HANDS_GRAB_THAT_WIDTH (0.6 m)
come at that width. The same gesture at a panel that came, still where
it came to, sends it back; the reset button sends floating windows back
(the layout reset places screens and remote displays itself).
screens/grabthat.{h,cpp}: the detector (pure logic): open fingers, palm
ahead and the wrist 28 cm out for 0.2 s arm it; the fingers closing with
the palm still to the panel, then on one frame a fist, the wrist 8 cm
back within 0.5 s and the palm turned over fire it. The panel is the one
the eye-through-palm line meets, else the one the head points at. From
the closing (thumb clear of the index: not a pinch) to 0.4 s after the
fire the hand counts as near, so the helper doesn't take the fist as a
gaze grip. Hands frames carry the head pose at capture now
(Hands::captureHead).
Tests replay the user's flag (fires twice, right hand, at -11.35 and
-5.89 s; expected -11.4 and -5.9), and the other 12 flags and 15 minutes
of recordings fire nothing (1 of 143 pinches, 3 of 12 grips begin on a
claimed hand); synthetic waves, pushes, fists without the pull, slow
pulls, a hand turned over first, held-out pinches, a drop to the lap and
typing don't fire. handinput-test --grabthat FLAG_DIR... scans flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without Accessibility, Sunshine on macOS still moves the pointer (it
warps it) but clicks, scrolling and typing are dropped. Sunshine asks
only once, so a lost grant, or one given to a Homebrew Sunshine, never
gets asked for again.
The setup now reports Screen Recording and Accessibility from what
Sunshine noted at its last start, at the end of a run and with --check.
If Accessibility is still missing after the first instance restarts, it
offers to clear Sunshine's TCC records and restart it so macOS asks
again. Documented in remote-displays.md and reference.md.
While one hand carries a panel, the hand's turn since the grab turns it
too, 1:1 about yaw, pitch and roll (user, 2026-10-10: line displays up).
The turn comes from the wrist and the index and little knuckles, which
hold still while the fingers pinch, smoothed with fh_smooth's Direction
filter, relative to facing the head across the floor from the hand, and
is applied on top of the panel facing the head: carrying a panel round
you still only keeps it facing you. Nothing turns until the hand has
turned HANDS_GRAB_ROTATE_DEADZONE (5 deg), then it turns from there
without a jump; within 2.5 deg of level it snaps level. Let go, it keeps
the turn, and the next grab starts from it. Corner and tab grabs only
resize; a stretch is unchanged and the hand left holding starts afresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper feeds typing.h the relay's keys and each hands frame. A hand it
holds back makes no pinch, middle pinch or grip (ray, gaze or plain), and has
no ray: one that drove the pointer drops it (to the mouse or the gaze, not
the controllers) and can't claim it again until lifted off the keys. A hand
pointing or holding a gesture out of the keyboard zone isn't held back by
the other hand's keys. ft-screens gets "typing <L> <R>" on a change and every
second. The old 1 s guards (beginHold, gazehand's) give way to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The near claim reaches 0.30 m (was 0.15): within it the pointer helper
hides that hand's ray and leaves its pinches alone, and the index pokes
(user, 2026-10-10: "near = poke within a foot"). A hand near a panel
stays near until 5 cm farther out, then 250 ms on, so the ray doesn't
flicker at the zone's edge. A hand resting low by the body (the poke's
resting rule) is near only within 0.15 m, so a low panel within reach
doesn't take gaze pinches from the lap. The poke's rules are unchanged:
on the PR #3 desk and touch replays the presses are the same as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Spotlight-like search pill (docs/spotwatch.md). Meta+Space (the input
relay's new spotwatch action, bindable to mouse and controller buttons
too) opens a faux-glass pill straight ahead of the head. Typing finds
apps and remote displays ({host} - {display}). Enter brings a running
app's most recent window to the front with the lazy susan, launches an
app that isn't running as a floating window, or connects and spins to a
remote display. Shift+Enter or a right click opens the item's menu: the
app's desktop actions, Open new window, Open docked on this screen, and
its windows; a display's Go to and Disconnect.
- spotwatch/ft-spotwatch (Python, in the Plasma session's autostart):
the index, ranking (spotsearch.py), keys via libxkbcommon, usage.
- spotwatch/ft-spotpanel.cpp (SteamVR overlay, built by
screens/build.sh): the pill, drawn on the CPU; --render for a look
without a headset.
- ft-screens: "spin to N", "front N", and "spotwatch on|off", which
routes every key to the search while it's open (a lease, renewed
every second) and dismisses it on a click elsewhere or a pause.
- ft-floatd: "spotlist", "activate ID", and launch action=/screen=;
the KWin script tracks when each window was last active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Mac runs a Sunshine instance per display, and each display's client is
paired with its own. An ft-layout from before ports (release 0.2.2's Reset
Screen Layout, Display Settings and profile launchers) restarted the
Mac's streams at the bare address, the instance on 47989: the other
displays froze, unpaired, and their pointer went to the main display.
ft-stream now looks its port up by its client id, as it reads the route.
Checked: pair with the bare address finds 48089 and the paired client.
host/macos/frametop-host-setup.sh makes a Mac a host: the pinned Sunshine
pre-release (v2026.1010.141104, the first with the mouse fix for displays
other than the main one), and one instance per real display on ports
47989 + 100*N, each started at login by a LaunchAgent. An instance follows
its monitor (vendor, model, serial), looks its display id up at every start,
and waits while the monitor is unplugged. One certificate and one Web UI
login for all of them. No virtual displays on the Mac.
On the Frame: a display can have a port, and ft-stream takes HOST:PORT.
Plain Sunshine has no tokens, so Remote Displays checks the Web UI login at
sign-in, keeps it in memory until the picked displays are paired, and
passes it on stdin (ft-layout remote add --port --login, ft-stream pair
--login), where ft-stream approves its own pairing request by id. The
Mac's instances show up as one computer.
Tested against the M3 Max with throwaway instances: pairing, remote add,
three streams at once, sound, and the pointer on a non-main display. Not
yet tested in the headset or the live desktop.
Frametop's Vibepollo build now switches a streamed monitor to the stream's
resolution while it streams, and back afterwards, instead of scaling the
monitor's own resolution into the stream. Nothing changes on the Frame:
ft-stream already asks for the size set in Remote Displays. The note under a
monitor's stream settings, reference.md and remote-displays.md say so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
voice/ft-voiced (was frame_voice.py) runs in Frametop's container as
frametop-voice.service. It keeps the mic, the wake word and live text,
and hands the keyboard side to the input relay:
- push-to-talk keys are key combinations (dictate, dictate_hold); the
relay sends "toggle|start|stop" to @frametop_voice, taken only from
this user
- text goes to the relay as "type TEXT" (ASCII, in pieces of 2000
bytes), so it lands where typing goes; the relay holds it back while
a modifier is held
- controller buttons are read from vrserver's web socket with
input/vrws.py, in a thread, so the websockets package goes
No own uinput keyboard, fd store, evdev, or controller service any more.
whisper-server runs only while the wake word is on, started and stopped
by ft-voiced. Voice commands and hooks run on the host through
distrobox-host-exec (env passed through env(1)); their FRAME_VOICE_*
variables keep frame-voice's names.
Settings: ~/.config/frametop-voice.json (voice_settings.py, standard
library), wake word off by default, "advanced" for the old config.toml
tunables. voice/run.sh install carries frame-voice's settings over once
(its keys become key combinations) and turns the standalone units off.
Install: install.sh step 8/11 asks (yes by default; --no-voice through
get.sh, install-release.sh and a FrameDrop checkbox). voice/build.sh
builds whisper-server (whisper.cpp d09f61a, Vulkan, static) and
build/venv; voice/models.sh fetches the 322 MB of models into
~/.local/share/frametop/voice by fixed URL and sha256. The image gets
glslc, spirv-headers-devel and pipewire-utils, uv.lock a voice group (sherpa-onnx with its
core pinned: uv dropped it), CI smoke whisper-server and the venv.
uninstall.sh asks about the models; update-check and report.sh know the
service (report.sh leaves out the lines with what was said).
Tests: voice/test (settings, carrying over, typing through a stand-in
relay, push-to-talk, controller buttons from a stand-in web socket) in
just test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
