A Remote displays section in the README with the three host kits; host/windows/README.md
and host/linux/README.md beside the Mac's; reference.md names frametop-2.0.0-2 for the
resolution switch; remote-displays.md is the design notes now. No machine names in hands/README.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gap_report: per side, for gaps where the full search lost the hand for a while
(back in view) and the rest, the time from the oracle's first sample finding it
again to the tracker having it. replay_gaps.sh: REPLAY=BINARY replays with
another ft-handreplay build, for a baseline.
On 2026-10-13 they showed student v13b losing hands v11c keeps (bright desk
recording: 12.8% of findable hands missed against 3.3%, mostly palms the
landmark model rejected).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User, 2026-10-13: "implement interpolation for the rendered hands so that they
feel smoother", and "if the hand is lost it's preferable the rendered hand
display freeze instead of make impossible movements, spaz out, or vanish until a
few seconds have passed".
screens/handhold.h, per side in ft-handview:
- each tracking frame blends in joint by joint from the drawn pose over about
the frame interval (15-70 ms), drawn every 15 ms meanwhile;
- a lost hand stays frozen where it was drawn for HANDS_VIEW_HOLD s (3);
- a frame 15 cm / 6 m/s off the last one taken, or with the palm turned over,
waits for two agreeing frames (or 0.25 s) while the drawn hand holds still;
a first-seen or returning hand waits for two agreeing frames too.
HANDS_VIEW_SMOOTH=0 gives the old behaviour. Tests in handview_test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User, 2026-10-13: "make the lazy susan just slightly more difficult to trigger".
One hand: edge 0.88, lead 0.16, 20 cm, quick at 0.75 m/s somewhere along it;
3 frames off shape ridden out. The flags replayed: all 47 practice strokes still
count (each peaked at 0.8 m/s or more), other use 3 false strokes, down from 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ray places it; the wrist's turn no longer tilts or turns it. Controllers
and the mouse still carry it rigidly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The handle (the laser's hit area, all outside the corner) is 1.8 buttons
wide, and the laser's corner glow reaches as far off the edges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The laser's aim at a corner gets the same white L a hand's pinch gets
(a third guide overlay, frametop.guide.laser), brightest while resizing.
The handle stays just outside the corner as the laser's hit area.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live misses were palm-led strokes with the palm's lead under 0.25, tilted
0.81-0.88 toward flat, or drifting 21-22 cm up. Lead 0.3 (one hand 0.12),
edge 0.85 (one hand 0.9), drift 0.3, 4 frames off shape ridden out. The
flags: 51 strokes (49), 4 in other use (2). A refused stroke's log now says
which place rule refused it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The line from the eyes through the palm lit panels off to the side of a palm
that wasn't facing them. The line is now the palm's own, along its normal,
with the margins seen from the palm. HANDS_GRAB_THAT_AIM=eyes brings the old
line back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
X11 windows get the layout's coordinates times its largest scale, which wrap
past 32767. The 4th spare at x 27000 and scale 1.25 put Brave at x -31506:
KWin focused it on a click, but the click never reached it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brave's Flatpak opens an X11 window with no app id, class Brave-browser, and a pid inside
its sandbox, so neither the process nor the desktop file name matched and it opened docked
(2026-10-13). desktop_name now finds a window's desktop file by its app id, else by its class:
StartupWMClass, the desktop file's name or its last dotted part, or the program it runs
(Flatpak's --command=). As a last resort, a window no desktop file claims floats as the
launch's when exactly one launch is waiting. The same name keys floating places, spotlist
and profiles (a profile now relaunches such an app by its desktop file, not a sandbox pid's
command line). Answers are cached for a minute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-floatd's spotlist used the profile filter, which leaves out every ft-* window, so Enter
on an open Input Settings, Display Settings or Remote Displays launched another copy
(2026-10-13). spotlist now has its own filter (listable) that keeps them; profiles still
leave them out. ft-spotwatch logs each click and menu with the row, to tell a right click
from a left one in the headset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A width typed as 1921 in Remote Displays was saved and asked for as is. The host's encoder
(NVENC, 4:2:0) sent no video for it, the host ended the stream about 11 s in, and the
display reconnected forever (2026-10-13). remote set, remote add --size and each stream's
start round to even, so a saved odd size from before works too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spotwatch's "spin to N" still turned the whole ring after the merge, the panel in the
front slot with it, while next and prev bring one panel to the slot. With
SCREEN_SPIN_BRING on (the default) it now brings panel N there too; the ring turn stays
for SCREEN_SPIN_BRING=0 and for degrees. Docs: reference.md's spin, spotwatch.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hands/rec/demo: ft-handdemo, sheets.py and gifs.py recorded the demo clips in the
headset, dropped on 2026-10-12 for generated skeleton clips. camgifs.py keeps its
gifsicle lookup; the README covers what's left.
- hands/.gitignore: rec/demos-recorded/, the recorded clips show the recorder's room.
- spotwatch/test/render.py -> spotwatch/tools/render.py: it needs an output path, and
just test-python ran every spotwatch/test/*.py with no arguments.
- voice/README.md: the advanced example's hook is the repo's hooks/on-command, not a
path in one person's home.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Far stretch: both palms held out at the same far panel, then two fists
within 0.6 s, scales it round its centre (a floating window by pixels)
until a hand opens. Aims at the panel both hands' lines meet, or the
line through the palms' midpoint. Grab that holds its fire meanwhile, and
both hands are claimed from the pointer helper so a fist passing through
a pinch doesn't click or drag under a raised hand ray. Near misses log why.
- Reset button: a hand's tap shows "Reset the layout? Tap" under it for
5 s; a tap on it (or the button again) resets. A laser click resets at once.
- Lazy susan: with the other hand still or unseen, a looser shape, 18 cm
and 0.5 m/s count; any stroke rides out 2 frames off its shape; misses
log open/tilt/lead against their limits.
- Pointer helper: the two-hand gaze drag no longer logs "with the right
button" (it is the left alone).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Throws: a carry's let-go takes the fastest outward stretch in the last
150 ms before the fingers opened (a hand slows as it lets go), at 0.45 m/s
within 70 deg (was 0.55 within 60, the last 100 ms only). Every let-go
moving 0.3 m/s or more is logged.
Sent away (thrown, flicked, pushed), a panel goes along that line out as
far and as big as it was before a hand or "grab that" took it, not back
to its old spot (layouts are for that). Put down past 1.2 m, that's its
distance from then on; never taken, it flies to 3 m as before. The front
slot's screen leaves the slot the same way.
The bar's buttons sit a button's width apart, and the reset button
resets the layout on a tap (HANDS_RESET_HOLD_MS=0; 500 before).
Grab that's glow: ft_pointer's laser sits on the panel with the cursor
all the time (the 3D mouse, the gaze), so "no glow under a laser" kept
the glow off that panel. The pointer helper now tells ft-screens which
hand rays are up ("ray L R", on change and every second); our laser
counts as one only then. Controllers' lasers count as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The "no glow under a laser" rule counted any overlay event as a laser, and
SteamVR sends panels events from the headset and from no device at all, so
the glow stayed off nearly everywhere and only the last one drawn (the
32:9 remote) was ever seen. Only a controller's events (the pointer
helper's 3D mouse and hand ray are ft_pointer controllers) count now. The
log says when a laser holds the glow back, and the glow's size per panel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 5120x1440 remote in the front slot was held to 100 deg wide (51 px/deg,
hard to read). Panels 3:1 or wider may now go to SCREEN_SPIN_MAX_WIDE_DEG
(150) instead, and their curve may wrap that far round you.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plugging the colour module in with passthrough working moves the side
cameras from the ISP nodes (video0/3) to video9/13 and puts the colour
cameras on video0/3. Those nodes keep streaming, so silent_camera never
fired and ft-hands was fed 2464x2464 NV12 frames as side cameras.
Check each node's format once a second on a handle opened at setup;
any change exits with status 4 so systemd restarts on the new layout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Headset off/on 2026-10-13: one camera started 0.1 s, then over 3 s, after
the others, and ft-camd exited as if XRService had moved the cameras (about
5 s without hands each time). For 10 s after the cameras start again a
silent camera is allowed; one moved while the headset was off is caught
after that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2 script's demo clips come in two kinds now, picked on the start page
("Guidance") and kept in settings.json: recorded clips of real hands
(demos-recorded/) or the drawn skeleton ones (demos/). Recorded is the
default when there are any; a part with no clip of the chosen kind shows the
other kind. session.json records the choice. The panel's demo command takes
"nokey" so a recorded clip shows without the skeleton's colour key; the
window does the same.
demo/camgifs.py makes the recorded clips from a recorder session: each
part's right-hand (or both-hands) step, the two side fisheyes redrawn as a
plain view from between the eyes using the session's calibration.json,
denoised so the GIFs compress (10-15 MB a clip without it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without the colour module the side cameras come through the ISP and their
near-black exposures are all zeros. ft-camd learned once which V4L2 indices
carry them and never read those indices again. Measured 2026-10-13 with the
headset worn, that split moves every few seconds: bright frames landing on a
skipped index were dropped, and find_fresh published them at the next dequeue
under its timestamp, 8 ms late. The side cameras matched the upper ones in
only ~55-60% of sets and live tracking fell from ~28 to 10-18 sets/s with 2-3x
the stereo error.
A camera bound by VIDIOC_QUERYBUF now maps every index to its own buffer and
judges each frame by its pixels, like the upper cameras; a repeated all-zero
frame counts as dark, never as a reason to grab another buffer. Live: 30.0
matched sets/s, offsets 0.3 ms, tracker 27.2 sets/s, 1.5 mm.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hand overlay covered what the person watches, and the pointer turned the steps' pinches into
clicks and opened Steam's menu. A session stops the running ones after the camera check and starts
them again when it ends; a list in the run dir lets the window or the next session start them if
the recorder was killed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ft-camd exits when a mono camera goes silent for 3 s while the others
run (XRService moved it), so its service starts it on the new devices
- a step whose recording exits, lacks the side pair or writes no set for
3 s waits for the camera check (with repair, up to 120 s) and goes
again; it fails the session if they don't come back or after 3 losses
- the session's preflight repairs an ft-camd left on moved cameras
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 08:10 the colour module disconnected and XRService moved the side
cameras from video9/13 to video0/3. ft-camd, started the day before, kept
publishing the old devices; ft-hands named cameras by the new log and left
both side cameras out, so a full v2 session recorded only the upper pair.
The camera check counted 4 cameras and said ok.
- camcheck: ft-camd's devices must be the ones XRService runs now
- camera_check(repair): also restarts the live frametop-camd (then
frametop-hands, which names cameras when it starts)
- a part whose first set has no slam_left/slam_right stops the session
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
script-v2.json has a step per part of demo/parts.json, each playing that
part's GIF from hands/rec/demos: one-hand parts run left (the clip
flipped), right, then both. About 12 min recorded; the quick round is
hand size, index taps, palm up and no hands. ft-handrec --script and
session.py --script pick it; script.json stays v1 for now.
ft-handpanel: "demo <path> [mirror] | restart | off" plays a GIF in a
16-degree left column with a key under it (light: palm, dark: back of
hand), the text beside it left-aligned. Frames are decoded one at a time
with stb_image's GIF decoder, so a clip holds three frames in memory, and
each frame redraws and copies only the clip's rectangle. The session
restarts the clip when a hold begins; prompt events carry "demo" and
session.json the script's name. The window plays the same clip
(AnimatedImage) with the same key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The demo recorder keeps the part and its motions on ft-handpanel, fixed to
the head and raised above the recorded headset view (new --up and --width
options); gifs.py crops off the strip where it dips in. SteamVR opens
Steam's menu on the same side button, so the app closes it after each press
and as a take starts. 2a and 3a say to look down at the low hand, and the
spoken-number check takes "to a" for 2a.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frametop Streams lists the Mac's displays (its Sunshine instances) and turns
each one, or all of them, on or off. ft-sunshine off|on|status does the same
in Terminal. Off disables the instance's LaunchAgent, so it stays off through
logins until on. The setup compiles the app from Frametop Streams.applescript
(AppleScriptObjC, no compiler) and starts it at login; --undo removes it.
README.md is the Mac setup guide, also the notes of the kit's release.
hands/rec/demo: ft-handdemo (GTK 4) walks parts.json, one take per part from
the headset's right-side button, recording the headset view (Room View) with
vr-record while panels hide, hand tracking parks and gaze is off. sheets.py
makes contact sheets and checks the spoken part number; gifs.py turns the
hand-picked trims into one GIF per part.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11, the second gaze test:
- "hand tracking stopped" ends were an unsigned age: ft-hands publishing just after the
loop's tnow made it negative, wrapped to 584 years, and ended held pinches, nudges,
drags and scrolls. Signed now.
- The two-hand drag mirrors the keys: hold one pinch (aim, nudge), pinch with the other:
the left button goes down where the dot is (as Meta+K with Meta+J held); the drag
lasts while either pinch holds, and the hand still pinching drags. A second pinch
during a drag joins it.
- The fast-hand filter refused dozens of still pinches (frame-to-frame palm speed is
noisy): smoothed, 1.0 m/s.
- A three-finger scroll goes on when ft-hands swaps which finger it calls pinched.
- Lazy susan: counted strokes came at 25-30 cm in 230-470 ms, at both limits: 22 cm,
quick somewhere along it (0.7 m/s peak) within 0.9 s; looser hand shape. The flags
replayed: 2 strokes in 45 flags of other use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11: "the lazy susan can be considered to be in use and remain active until I
move it out of its spot manually through either the eye tracking or grabbing it with
hand-tracking". Looking away no longer sends it home (SCREEN_FRONT_AWAY=1 brings that
back, with the keep-by-use); a gaze-pinch drag, a hand grab, a push or grab that
releases it, and the next stroke sends it home.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HANDS_PINCH_END, HANDS_PINCH_DRAG_OPEN, HANDS_PINCH_DRAG_OPEN_S, HANDS_GRIP_TAKEOVER_S
(user 2026-10-11, gaze + hands: pinches released at 4-6 cm, and grips took pinches
130 ms in, while the fingers were still together).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user's 2026-10-11 test of gaze + hands:
- Two-hand drag as they meant it: hold one hand's pinch (nudge it into place), tap
with the other: index presses the left button where the pointer is, middle the
right; the first hand drags (POINTER_GAZE_TWO_HAND). The tap-first right-drag is
off by default (it could catch a tap followed by a nudge).
- 9 held pinches ended as "hand tracking stopped (1.5 s)" while ft-hands still held
them: 4 s with nothing pressed, 2.5 s with a press, and the line now says the file's
seq and ages to find the cause.
- A gaze pinch held back ends without a click when the hand closes into a fist
(ft-hands' grip may not take it past 0.1 s now); a pinch begun while the palm
moves faster than 0.6 m/s (a stroke) doesn't click (POINTER_PINCH_MAX_SPEED).
- Three-finger scroll: 3.5 cm and 0.4 s to join.
- Gaze mode hides the dot and collapses its laser for 1.5 s after each key
(POINTER_TYPING_HIDE): it sat big and bright on the text being typed.
- The front slot's keep counts clicks, scrolls, drags and keys, not the gaze pointer
passing over it (every brought screen was kept 3 s in).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11, gaze + hands becoming the main input:
- Right-drag: an index tap on one hand, then an index pinch on the other within 1 s,
holds the right button at the gaze lock; that hand drags until it ends
(POINTER_GAZE_RIGHT_DRAG). It replaces the chord (POINTER_GAZE_CHORD=0).
- Three-finger scroll: the thumb on both the index and middle tips (3 cm) scrolls where
you looked as the hand moves, like the ray's middle-pinch stick (POINTER_GAZE_SCROLL).
- The fist guard (a hand closing into a fist: no click) now covers gaze and chord
pinches too, from each hand's own frame (not the ray's).
- A hand's laser shows only its dot: the beam starts just short of it, as the eye's
does, held still through a carry (POINTER_RAY_BEAM=0; SteamVR's beam width is one
global setting it reads only at start).
- The front slot gives a screen up to manual controls: grab that, a push, flick or
throw, or a carry by hand or laser releases it (it moves like any other and stays
where you put it); the next stroke puts it home and brings the next.
- The lazy susan counts only palm-led strokes (the return stroke does nothing; the
other hand palms the other way; HANDS_SWEEP_BACK=1 for both).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user's proposals (2026-10-11): the lazy susan and grab that were two systems that
fought over a screen (grab that took one the susan brought: 2.3 m wide at 1.5 m).
Now one front slot, one screen at a time (the one there goes home first):
- reading distance (the lazy susan, 1 m) and arm's reach (grab that), both readable
(ReadableSize, the same view at either); grab that at the one there pulls it closer;
- a push, flick or throw steps it back: arm's reach -> reading -> home;
- one home per screen: every dismissal returns it there; moved by hand or laser it
lives where you put it; resized in the slot it keeps that view next time;
- turned away from it goes home, unless used 3 s (kept; SCREEN_FRONT_KEEP_MS);
- a strip of the screens' names, left to right as they live, for 2.5 s after each
stroke, the one in the slot lit and the next each way marked (keyboard.cpp's font,
now DrawLabel for other panels too);
- SCREEN_UI_SCALE for a remote display's own UI scale (a Mac's Retina: 2);
- the reset button needs a 0.5 s hold, laser or finger (a stray poke reset the layout;
HANDS_RESET_HOLD_MS).
grab that's remembered widths, send-away widths and view fit go (the slot sizes it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lazy susan (the third live test): a brought screen comes 1 m ahead, as big as its
resolution needs for text to read: its logical pixels (KWin's scale undone, now passed
to vr.cpp) at 26 px/deg, at most 100x55 deg keeping its shape, at least 40 deg wide,
curved round you past 60 deg (grabthat::ReadableSize; SCREEN_SPIN_DISTANCE, _PPD,
_MAX_DEG, _MAX_TALL_DEG). Its curve comes back on the way home. The strokes cycled
104, 1, 104 with three screens up: a screen still flying home read as straight ahead;
each one is now taken where it lives (home, or where it's going).
Accidental clicks: most were pinches with no laser up pressing where the pointer was
last left ("no ray up: a press where the pointer is", a dozen in two minutes). With
gaze off they now only wake the pointer (POINTER_PINCH_NO_RAY=1: the old air mouse).
The hand's laser shows only with the thumb and index within 6 cm (gone past 10;
POINTER_RAY_READY), and a fist drops it: fh_ray's "ready" and "open" tests, the fist
read as the index and the other three both curled under 1.1 (FistCurl; measured on 47k
flagged hands: the other three alone read as a fist in a fifth of the pinches). A ray
pinch made while the hand closes into a fist doesn't click. Input Settings switches for
all three (POINTER_ ones reload the helper).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lazy susan (the sweep, Meta+Alt+Tab) no longer turns the room: the next screen
round, by where each one lives, comes in front of you at a distance it spans at most
60 deg of the view at, and takes the typing. The next stroke sends it home and brings
the one after; pushed, flicked or thrown it goes home; turned away from for 0.6 s it
goes home; moved by hand or laser it stays. SCREEN_SPIN_BRING=0 turns the room again
(Input Settings switch too).
Sweeps were hard to make (counted strokes came at 30-35 cm, just over the 30 asked):
25 cm in 0.5 s, the hand open 1.4, tilted up to ~45 deg off its edge, and hovering near
a screen counts (the brought one sits right there). The flags replayed: one stroke in
49 flags of other use.
Grab that brought a remote display back at the 1.62 m it was resized to, 0.45 m ahead:
it filled the view and every poke went through it (both flags of the second live test).
A wide one now comes as far out as it spans at most 75 deg (HANDS_GRAB_THAT_VIEW_DEG).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Starting the desktop with the headset off failed to arrange the screens (no head
pose) and returned before telling ft-screens KWin's scales. With the native screen
at 125%, every laser and tap there landed at 1.25x its distance from the top-left
corner, hundreds of pixels off mid-screen. Arranging that fails at start is logged
and the scales are sent anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User, 2026-10-11: "make every interaction able to be enabled or disabled".
The Hands page gets a "Hand interactions" section: 24 switches in four
groups (tapping, moving screens, gestures, feedback), each writing its
frametop.conf key and telling ft-screens at once ("handsetting KEY 0|1",
SetHandSetting; number-valued ones go back to their defaults when switched
on). They follow the direct touch switch above.
The command is checked before the helper's "hands L R allowed" word: that
pattern matched "handsetting X 0" too and scrambled the hands word.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
