User, 2026-10-13: "implement interpolation for the rendered hands so that they
feel smoother", and "if the hand is lost it's preferable the rendered hand
display freeze instead of make impossible movements, spaz out, or vanish until a
few seconds have passed".
screens/handhold.h, per side in ft-handview:
- each tracking frame blends in joint by joint from the drawn pose over about
the frame interval (15-70 ms), drawn every 15 ms meanwhile;
- a lost hand stays frozen where it was drawn for HANDS_VIEW_HOLD s (3);
- a frame 15 cm / 6 m/s off the last one taken, or with the palm turned over,
waits for two agreeing frames (or 0.25 s) while the drawn hand holds still;
a first-seen or returning hand waits for two agreeing frames too.
HANDS_VIEW_SMOOTH=0 gives the old behaviour. Tests in handview_test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User, 2026-10-13: "make the lazy susan just slightly more difficult to trigger".
One hand: edge 0.88, lead 0.16, 20 cm, quick at 0.75 m/s somewhere along it;
3 frames off shape ridden out. The flags replayed: all 47 practice strokes still
count (each peaked at 0.8 m/s or more), other use 3 false strokes, down from 4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The line from the eyes through the palm lit panels off to the side of a palm
that wasn't facing them. The line is now the palm's own, along its normal,
with the margins seen from the palm. HANDS_GRAB_THAT_AIM=eyes brings the old
line back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Throws: a carry's let-go takes the fastest outward stretch in the last
150 ms before the fingers opened (a hand slows as it lets go), at 0.45 m/s
within 70 deg (was 0.55 within 60, the last 100 ms only). Every let-go
moving 0.3 m/s or more is logged.
Sent away (thrown, flicked, pushed), a panel goes along that line out as
far and as big as it was before a hand or "grab that" took it, not back
to its old spot (layouts are for that). Put down past 1.2 m, that's its
distance from then on; never taken, it flies to 3 m as before. The front
slot's screen leaves the slot the same way.
The bar's buttons sit a button's width apart, and the reset button
resets the layout on a tap (HANDS_RESET_HOLD_MS=0; 500 before).
Grab that's glow: ft_pointer's laser sits on the panel with the cursor
all the time (the 3D mouse, the gaze), so "no glow under a laser" kept
the glow off that panel. The pointer helper now tells ft-screens which
hand rays are up ("ray L R", on change and every second); our laser
counts as one only then. Controllers' lasers count as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 5120x1440 remote in the front slot was held to 100 deg wide (51 px/deg,
hard to read). Panels 3:1 or wider may now go to SCREEN_SPIN_MAX_WIDE_DEG
(150) instead, and their curve may wrap that far round you.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11, the second gaze test:
- "hand tracking stopped" ends were an unsigned age: ft-hands publishing just after the
loop's tnow made it negative, wrapped to 584 years, and ended held pinches, nudges,
drags and scrolls. Signed now.
- The two-hand drag mirrors the keys: hold one pinch (aim, nudge), pinch with the other:
the left button goes down where the dot is (as Meta+K with Meta+J held); the drag
lasts while either pinch holds, and the hand still pinching drags. A second pinch
during a drag joins it.
- The fast-hand filter refused dozens of still pinches (frame-to-frame palm speed is
noisy): smoothed, 1.0 m/s.
- A three-finger scroll goes on when ft-hands swaps which finger it calls pinched.
- Lazy susan: counted strokes came at 25-30 cm in 230-470 ms, at both limits: 22 cm,
quick somewhere along it (0.7 m/s peak) within 0.9 s; looser hand shape. The flags
replayed: 2 strokes in 45 flags of other use.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11: "the lazy susan can be considered to be in use and remain active until I
move it out of its spot manually through either the eye tracking or grabbing it with
hand-tracking". Looking away no longer sends it home (SCREEN_FRONT_AWAY=1 brings that
back, with the keep-by-use); a gaze-pinch drag, a hand grab, a push or grab that
releases it, and the next stroke sends it home.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HANDS_PINCH_END, HANDS_PINCH_DRAG_OPEN, HANDS_PINCH_DRAG_OPEN_S, HANDS_GRIP_TAKEOVER_S
(user 2026-10-11, gaze + hands: pinches released at 4-6 cm, and grips took pinches
130 ms in, while the fingers were still together).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user's 2026-10-11 test of gaze + hands:
- Two-hand drag as they meant it: hold one hand's pinch (nudge it into place), tap
with the other: index presses the left button where the pointer is, middle the
right; the first hand drags (POINTER_GAZE_TWO_HAND). The tap-first right-drag is
off by default (it could catch a tap followed by a nudge).
- 9 held pinches ended as "hand tracking stopped (1.5 s)" while ft-hands still held
them: 4 s with nothing pressed, 2.5 s with a press, and the line now says the file's
seq and ages to find the cause.
- A gaze pinch held back ends without a click when the hand closes into a fist
(ft-hands' grip may not take it past 0.1 s now); a pinch begun while the palm
moves faster than 0.6 m/s (a stroke) doesn't click (POINTER_PINCH_MAX_SPEED).
- Three-finger scroll: 3.5 cm and 0.4 s to join.
- Gaze mode hides the dot and collapses its laser for 1.5 s after each key
(POINTER_TYPING_HIDE): it sat big and bright on the text being typed.
- The front slot's keep counts clicks, scrolls, drags and keys, not the gaze pointer
passing over it (every brought screen was kept 3 s in).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User 2026-10-11, gaze + hands becoming the main input:
- Right-drag: an index tap on one hand, then an index pinch on the other within 1 s,
holds the right button at the gaze lock; that hand drags until it ends
(POINTER_GAZE_RIGHT_DRAG). It replaces the chord (POINTER_GAZE_CHORD=0).
- Three-finger scroll: the thumb on both the index and middle tips (3 cm) scrolls where
you looked as the hand moves, like the ray's middle-pinch stick (POINTER_GAZE_SCROLL).
- The fist guard (a hand closing into a fist: no click) now covers gaze and chord
pinches too, from each hand's own frame (not the ray's).
- A hand's laser shows only its dot: the beam starts just short of it, as the eye's
does, held still through a carry (POINTER_RAY_BEAM=0; SteamVR's beam width is one
global setting it reads only at start).
- The front slot gives a screen up to manual controls: grab that, a push, flick or
throw, or a carry by hand or laser releases it (it moves like any other and stays
where you put it); the next stroke puts it home and brings the next.
- The lazy susan counts only palm-led strokes (the return stroke does nothing; the
other hand palms the other way; HANDS_SWEEP_BACK=1 for both).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user's proposals (2026-10-11): the lazy susan and grab that were two systems that
fought over a screen (grab that took one the susan brought: 2.3 m wide at 1.5 m).
Now one front slot, one screen at a time (the one there goes home first):
- reading distance (the lazy susan, 1 m) and arm's reach (grab that), both readable
(ReadableSize, the same view at either); grab that at the one there pulls it closer;
- a push, flick or throw steps it back: arm's reach -> reading -> home;
- one home per screen: every dismissal returns it there; moved by hand or laser it
lives where you put it; resized in the slot it keeps that view next time;
- turned away from it goes home, unless used 3 s (kept; SCREEN_FRONT_KEEP_MS);
- a strip of the screens' names, left to right as they live, for 2.5 s after each
stroke, the one in the slot lit and the next each way marked (keyboard.cpp's font,
now DrawLabel for other panels too);
- SCREEN_UI_SCALE for a remote display's own UI scale (a Mac's Retina: 2);
- the reset button needs a 0.5 s hold, laser or finger (a stray poke reset the layout;
HANDS_RESET_HOLD_MS).
grab that's remembered widths, send-away widths and view fit go (the slot sizes it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lazy susan (the third live test): a brought screen comes 1 m ahead, as big as its
resolution needs for text to read: its logical pixels (KWin's scale undone, now passed
to vr.cpp) at 26 px/deg, at most 100x55 deg keeping its shape, at least 40 deg wide,
curved round you past 60 deg (grabthat::ReadableSize; SCREEN_SPIN_DISTANCE, _PPD,
_MAX_DEG, _MAX_TALL_DEG). Its curve comes back on the way home. The strokes cycled
104, 1, 104 with three screens up: a screen still flying home read as straight ahead;
each one is now taken where it lives (home, or where it's going).
Accidental clicks: most were pinches with no laser up pressing where the pointer was
last left ("no ray up: a press where the pointer is", a dozen in two minutes). With
gaze off they now only wake the pointer (POINTER_PINCH_NO_RAY=1: the old air mouse).
The hand's laser shows only with the thumb and index within 6 cm (gone past 10;
POINTER_RAY_READY), and a fist drops it: fh_ray's "ready" and "open" tests, the fist
read as the index and the other three both curled under 1.1 (FistCurl; measured on 47k
flagged hands: the other three alone read as a fist in a fifth of the pinches). A ray
pinch made while the hand closes into a fist doesn't click. Input Settings switches for
all three (POINTER_ ones reload the helper).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lazy susan (the sweep, Meta+Alt+Tab) no longer turns the room: the next screen
round, by where each one lives, comes in front of you at a distance it spans at most
60 deg of the view at, and takes the typing. The next stroke sends it home and brings
the one after; pushed, flicked or thrown it goes home; turned away from for 0.6 s it
goes home; moved by hand or laser it stays. SCREEN_SPIN_BRING=0 turns the room again
(Input Settings switch too).
Sweeps were hard to make (counted strokes came at 30-35 cm, just over the 30 asked):
25 cm in 0.5 s, the hand open 1.4, tilted up to ~45 deg off its edge, and hovering near
a screen counts (the brought one sits right there). The flags replayed: one stroke in
49 flags of other use.
Grab that brought a remote display back at the 1.62 m it was resized to, 0.45 m ahead:
it filled the view and every poke went through it (both flags of the second live test).
A wide one now comes as far out as it spans at most 75 deg (HANDS_GRAB_THAT_VIEW_DEG).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fifth live test (2026-10-10 22:55): grab that brought the desktop in, it was
carried and thrown 3 m out, then five gestures at it or the remote displays
logged "no panel there". The open palm is held out below the line of sight,
so the line from the eyes through it runs 12-15 deg under where the head
points (the user's flag 150115). Since the fourth live test the line had to
meet a panel exactly, and a thrown 1.35 m tall desktop 3 m out is only
12.7 deg each way, so the line passed just under it. One of the flag's two
fires misses such a panel that way.
Pick now takes the panel the line meets (unchanged, nearest wins), else the
one whose outline it passes nearest within HANDS_GRAB_THAT_MARGIN_DEG
(6 deg seen from the eyes; 0 = the old rule). The gesture itself is
unchanged, so it stays as hard to trigger, and held, pinned, pressed and
near panels are still never taken. "no panel there" now logs where the line
went relative to the head and the nearest panel it may take, with its angle.
Test: TestGrabThatAfterThrow runs the user's sequence with synthetic hands:
grab that brings the desktop in, a bar carry is thrown out and up at
1.3 m/s, then grab that at it again (missed by 0.8 deg with the old rule,
taken now), a remote display after that, precision cases, and the flag's
two fires (2 of 2 now, 1 with the old rule).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The example config gets HANDS_FIST_REACH, HANDS_JOIN_REACH, HANDS_BUTTON_REACH,
HANDS_PINCH_BUTTONS, HANDS_EDGE_GLOW, HANDS_GRAB_THAT_FROM, HANDS_GRAB_THAT_MARGIN and
HANDS_GRAB_THAT_FACING; hands/README.md and docs/reference.md describe them, the white
halo, the corner, edge and button glows, and that near panels go away by push, flick or
throw now, not by grab that. `handinput state` counts joins, pinches that waited to be
fists, and buttons pinched.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/README.md (near input, the palm menu), docs/reference.md and session/frametop.conf.example:
HANDS_PUSH, HANDS_PUSH_SPEED, HANDS_STRETCH_TURN, HANDS_TRACK_GATE, HANDS_TARGET_GLOW,
HANDS_SEND_AWAY_WIDTH; fists ft-hands calls pinches, the pinch slack, holds through the helper's
word, the near-the-eyes carry, the stricter palm menu, with the replays' numbers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
session/frametop.conf.example: HANDS_POKE_TRAVEL=0.017, the card
pinch's fingers, and the new HANDS_HOLD_LOST_MS, HANDS_FLICK,
HANDS_FLICK_SPEED, HANDS_FLICK_OPENING, HANDS_THROW, HANDS_THROW_SPEED,
HANDS_THROW_DISTANCE and POINTER_GAZE_CONTROL_HOLD. scripts/conf-migrate.sh
moves the example's exact HANDS_POKE_TRAVEL=0.025 line (hands-touch,
never released) to 0.017. hands/README.md's near input section,
docs/reference.md's .resize, settings and handinput state, and the
gesture tables at the top of handinput.h and vr.cpp say what each
gesture does now and what wins. The README notes that the hand glows
are drawn into the cutout buffer, so they don't show with the cutouts
off (the default since the drawn hands).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
session/frametop.conf.example: HANDS_NEAR_REACH=0.075, the new
HANDS_POKE_TRAVEL, HANDS_GRAB_BAND, HANDS_PINCH_CLICK, HANDS_CARD_PINCH and
HANDS_FIST_GRAB. scripts/conf-migrate.sh moves the example's exact 0.30
line (hands-reach, never released) to 0.075. hands/README.md's near input
section, docs/reference.md's settings and handinput state, and the
gesture table at the top of handinput.h and vr.cpp say what each gesture
does near a panel and what wins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts:
- ft-pointer.cpp, the gaze pinch comment: typing's "the rules for all
gestures hold (..., typing, ...)" kept, gazechord's chord and
POINTER_GAZE_PINCH paragraphs added after it, and a line that a hand
typing holds back makes no chord.
- ft-pointer.cpp, the hands settings: gazechord's gazePinchOn, nearOn,
handsTrack kept; typingHold dropped for typing's typing::Params.
- ft-pointer.cpp, mayPoint: both conditions (handsOn, and not a side
typing holds back).
- ft-pointer.cpp, beginGazePinch: gazechord's busy line (a chord isn't
busy), without the old sinceKey/typingHold lines. The chord's pinch
goes through typing.h's Why in the gesture loop before startGesture,
like any pinch, so a hand typing holds back can't make one.
- docs/reference.md settings line: typing's and reach's keys, plus
POINTER_GAZE_PINCH.
- hands/README.md near input: typing's hold-back sentence and
gazechord's HANDS_NEAR=on.
- session/frametop.conf.example: reach's HANDS_GRAB_ROTATE* and
HANDS_GRAB_THAT* lines, then gazechord's HANDS_NEAR line.
Reconciled: gazehand's Situation loses sinceKey and typingHold, and
Refused its "ignored (typing)": the helper had stopped setting them on
hands-typing (typing.h holds hands back first), so that guard could
never fire. gazehand-test's key checks now drive typing.h: keys hold a
hand back (2 s now, was 1), POINTER_PINCH_TYPING=0 is no guard, and a
chord's pinch by a hand on the keys is held back while one lifted off
them goes ahead. The chord's own pinch counts as a gesture held for
typing's "busy" (free if out of the keyboard zone when keys come), as
the left's hold does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict: docs/reference.md's hands settings line (typing's POINTER_TYPING_*
keys next to reach's HANDS_NEAR_REACH, HANDS_GRAB_ROTATE* and
HANDS_GRAB_THAT*): both kept.
Typing and grab that: a hand typing holds back (the helper's "typing" word)
doesn't arm or fire "grab that" either, so it never claims a fist as near;
vr.cpp's UpdateGrabThat leaves it out, as Poke and Grab already do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hold an open hand out to a screen, remote display or floating window,
palm to it, then pull the arm back making a fist: the panel comes over
250 ms to 45 cm ahead of the eyes, at the fist's height, facing you
(user, 2026-10-10). Screens wider than HANDS_GRAB_THAT_WIDTH (0.6 m)
come at that width. The same gesture at a panel that came, still where
it came to, sends it back; the reset button sends floating windows back
(the layout reset places screens and remote displays itself).
screens/grabthat.{h,cpp}: the detector (pure logic): open fingers, palm
ahead and the wrist 28 cm out for 0.2 s arm it; the fingers closing with
the palm still to the panel, then on one frame a fist, the wrist 8 cm
back within 0.5 s and the palm turned over fire it. The panel is the one
the eye-through-palm line meets, else the one the head points at. From
the closing (thumb clear of the index: not a pinch) to 0.4 s after the
fire the hand counts as near, so the helper doesn't take the fist as a
gaze grip. Hands frames carry the head pose at capture now
(Hands::captureHead).
Tests replay the user's flag (fires twice, right hand, at -11.35 and
-5.89 s; expected -11.4 and -5.9), and the other 12 flags and 15 minutes
of recordings fire nothing (1 of 143 pinches, 3 of 12 grips begin on a
claimed hand); synthetic waves, pushes, fists without the pull, slow
pulls, a hand turned over first, held-out pinches, a drop to the lap and
typing don't fire. handinput-test --grabthat FLAG_DIR... scans flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper sends the chord's right button (btn b) apart from the left's
press, so both reach the panel; the chord ends with its pinch, its hand
lost, the pointer going, a controller on its side, or the calibration
panel. POINTER_GAZE_PINCH (1) turns clicks at the gaze off on their own.
HANDS_NEAR=on runs the handoff for ft-screens' direct touch without
POINTER_HANDS (HANDS_NEAR=1 keeps its meaning), and the relay feeds it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While one hand carries a panel, the hand's turn since the grab turns it
too, 1:1 about yaw, pitch and roll (user, 2026-10-10: line displays up).
The turn comes from the wrist and the index and little knuckles, which
hold still while the fingers pinch, smoothed with fh_smooth's Direction
filter, relative to facing the head across the floor from the hand, and
is applied on top of the panel facing the head: carrying a panel round
you still only keeps it facing you. Nothing turns until the hand has
turned HANDS_GRAB_ROTATE_DEADZONE (5 deg), then it turns from there
without a jump; within 2.5 deg of level it snaps level. Let go, it keeps
the turn, and the next grab starts from it. Corner and tab grabs only
resize; a stretch is unchanged and the hand left holding starts afresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper feeds typing.h the relay's keys and each hands frame. A hand it
holds back makes no pinch, middle pinch or grip (ray, gaze or plain), and has
no ray: one that drove the pointer drops it (to the mouse or the gaze, not
the controllers) and can't claim it again until lifted off the keys. A hand
pointing or holding a gesture out of the keyboard zone isn't held back by
the other hand's keys. ft-screens gets "typing <L> <R>" on a change and every
second. The old 1 s guards (beginHold, gazehand's) give way to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The near claim reaches 0.30 m (was 0.15): within it the pointer helper
hides that hand's ray and leaves its pinches alone, and the index pokes
(user, 2026-10-10: "near = poke within a foot"). A hand near a panel
stays near until 5 cm farther out, then 250 ms on, so the ray doesn't
flicker at the zone's edge. A hand resting low by the body (the poke's
resting rule) is near only within 0.15 m, so a low panel within reach
doesn't take gaze pinches from the lap. The poke's rules are unchanged:
on the PR #3 desk and touch replays the presses are the same as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Spotlight-like search pill (docs/spotwatch.md). Meta+Space (the input
relay's new spotwatch action, bindable to mouse and controller buttons
too) opens a faux-glass pill straight ahead of the head. Typing finds
apps and remote displays ({host} - {display}). Enter brings a running
app's most recent window to the front with the lazy susan, launches an
app that isn't running as a floating window, or connects and spins to a
remote display. Shift+Enter or a right click opens the item's menu: the
app's desktop actions, Open new window, Open docked on this screen, and
its windows; a display's Go to and Disconnect.
- spotwatch/ft-spotwatch (Python, in the Plasma session's autostart):
the index, ranking (spotsearch.py), keys via libxkbcommon, usage.
- spotwatch/ft-spotpanel.cpp (SteamVR overlay, built by
screens/build.sh): the pill, drawn on the CPU; --render for a look
without a headset.
- ft-screens: "spin to N", "front N", and "spotwatch on|off", which
routes every key to the search while it's open (a lease, renewed
every second) and dismisses it on a click elsewhere or a pause.
- ft-floatd: "spotlist", "activate ID", and launch action=/screen=;
the KWin script tracks when each window was last active.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For Frametop Voice (voice/, next commits), the input relay does the
keyboard side of dictation:
- "type TEXT" on its control socket types TEXT where typing goes: to
ft-screens for the desktop, else on the virtual keyboard SteamVR
passes to Steam and games. US layout, one key every 4 ms, and a
character waits while a modifier is held. Only from the relay's own
user: the socket now reads each sender's credentials (SO_PASSCRED).
- New actions dictate (a press starts or stops) and dictate_hold
(records while held) send "toggle|start|stop <source>" to
@frametop_voice. They work without pointer mode and while paused.
- A modifier tap bound by its right-hand code ("97") is that key's
alone and comes before the folded one, so dictation's default, a
Right Ctrl tap, doesn't fire on Left Ctrl (crouch in many games).
It's in the defaults once frametop-voice.service is installed.
SHARE_KEYS and @frametop_keys go: they existed only so frame-voice
could see keyboards the relay grabs, and sent every key typed into the
desktop to a socket any local process could bind. conf-migrate.sh drops
the example's line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ideas from StereoKit (MIT, 07b9f0d), credited where a formula comes from it:
- The torso faces the head's forward counted twice plus the direction to
each wrist seen (POINTER_RAY_TORSO_WRISTS, on), so looking aside while
pointing swings the shoulders, and the ray, less. probe-moves replay: the
shoulder's share of a still hand's ray motion 20 -> 18%, p90 2.0-3.7 ->
1.7-3.6 deg/s; the tap capture: nothing worse.
- The ray can turn toward the hand's own direction, from the palm's axes
(wrist, middle, ring and little knuckles) tilted -30/8 deg
(POINTER_RAY_HAND_WEIGHT, 0). Off: every weight measured worse (at 0.3:
judder 0.07-0.14 -> 0.11-0.33 deg rms per 90 Hz tick, raw jumps over
3 deg 159 -> 330). The gate still tests the shoulder's ray.
- The knuckle point can be pinned to the palm's axes during a pinch, easing
back on the steady shoulder's schedule (POINTER_RAY_ANCHOR, 0). Off: our
pinch point is the knuckles already, and pinned it shook as much or more
and kicked no less on release, fitted or not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands keeps the last 12 s of hand crops (each view's landmark crop, 1.6x, at most
224 px), its hands, gestures and landmarks in RAM (track/flagbuf.h, HANDS_FLAG_BUFFER=1).
`ft-handsctl flag [note]` asks for them through run_dir/flag; ft-hands writes them and the
next 2 s to HANDS_FLAG_DIR/flag-<time> from a low-priority thread, for frame-hands'
offline tools. ft-handsctl also saves a /dev/video99 frame and ft-handview's head poses,
and the drawn hands pulse cyan when a flag is taken (ft-handview's new flash and poses
commands). ft-handreplay --flag-at/--flag-every makes the same flags from recordings.
Measured on a replayed ring: +58 MB peak RSS, 0.1 ms a crop on an A720 (about 1% of a
core). Unit tests: tests/flagbuf_test.cpp (make check), tests/test_handsflag.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stylized robot hand made in code, no asset: rigid pieces, each fixed to
one bone, so nothing bends or stretches out of shape. A light shell per
finger bone (a tapered box with rounded edges and ends), dark balls at the
knuckles and the thumb's base, dark hinges at the other joints, a palm of a
dark core between a palm plate and a back plate with panel lines, and a
short closed cuff past the wrist. Cyan accents glow: fingertip pads, a line
across the knuckles, a dot in the palm, a ring round the cuff. solid is
ceramic white over dark grey metal; ghost is pale and see-through.
screens/handrobot.cpp builds one mesh (2373 vertices, 4360 triangles) once,
a right hand on DemoPose's open hand, in 16 segments' frames. The frames are
handskin's rule (new handskin::TrackedFrames); a left hand's are a right
one's mirrored, so the same mesh draws it mirrored. Each vertex is fixed to
its bone's start or end: shells keep their rounded ends, the straight middle
takes up the tracked length, fingertips end at the tracked tips. The vertex
shader applies one segment's matrix per vertex; no cap pass (it's closed).
Cost (--snapshot, frame-job 4 cores, same run): robot 0.21-0.24 ms CPU and
0.55-0.57 ms GPU wait per draw, against asset 0.28/0.66, babylon 0.26/0.66,
mrtk3 0.25/0.66.
Tests: segments start at their joints along their bones, no vertex past its
bone's length, fingertips at the tracked tips, left = right mirrored vertex
for vertex, rigid segments across poses, only the middle stretches with a
longer bone, accents on the right sides.
No free robot hand asset fit; screens/assets/hands/README.md says what was
looked at.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The look: no more drawn outline. The old rim pass lit a hard line wherever the
hand met the background or a part of it further back, so a finger over the palm
got scratchy white lines. Now the body pass keeps only the surface nearest the
eye (depth) and writes how squarely it faces the eye, the fade, a light from
above and how near a fingertip; the second pass turns that into a soft fresnel
rim, a slight fingertip glow and an antialiased outline. ghost glows white
toward its edges; solid is lit from above with a soft cool rim.
The wrist: the model is cut open where the fade ends, so a hand pointing away
showed its inside (a hollow tube, a fist's fingertips through the palm). A
stencil pass counts front faces up and back faces down along each view (on its
own, with the depth test off: early depth would drop hidden ones); where a view
went in through the cut, a disc where the fade begins covers it.
The models: HANDS_VIEW_MODEL=asset (generic-hand, still the default) | babylon
(Babylon.js, CC BY 4.0) | mrtk3 (BSD-3-Clause, right only, mirrored) | godot
(Godot XR Tools, CC0) | mrtk2 (MIT) | capsules. Each is in
screens/assets/hands/<name>/ with its licence and a README giving the source
commit, hashes and conversion (the MRTK ones were FBX: assimp, then the new
screens/tools/hand-glb-clean.py). hand-asset.py maps each skeleton onto WebXR's
25 joints by name and writes them all into handasset.h; handskin takes the
model's index.
Cost (--snapshot, asset): CPU 0.34 ms per draw (was 0.32-0.33), GPU wait
0.61-0.62 ms per draw (was 0.57). --snapshot now prints the GPU wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rendered hands are now the WebXR Input Profiles' generic hand (MIT),
posed by the tracked skeleton and skinned on the GPU: per joint a frame
from the bone and the palm, scaled to the hand and each bone's length.
Two looks, HANDS_VIEW_STYLE=ghost|solid; HANDS_VIEW_MODEL=capsules keeps
the tube hand. With HANDS_VIEW_ROOMVIEW=1 (the default) the hands also
float over the cutout with Room View on. --snapshot draws demo poses
into a PNG without SteamVR, and the cost (0.4 ms a draw, the tubes 0.6).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A near grab carried the pinch point moved ahead along the hand's velocity, which is mostly
noise times the lead, so the panel shook. It now carries the pinch point as the cameras saw
it, smoothed (fh_smooth.h, read every frame ft-screens draws), and moves nothing until the
point is 1 cm from where the pinch began, then carries on from there without a jump. The
head the panel turns to face is smoothed too. On the probe-moves replay the carried point's
second difference per 90 Hz tick went from 13-17 mm rms to 2.3-2.6 mm; it now trails the
capture by 85 ms while moving.
Settings: HANDS_GRAB_SMOOTH_MINCUTOFF, HANDS_GRAB_SMOOTH_BETA, HANDS_GRAB_DEADZONE.
handinput_test --replay reports the grab input's smoothness.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ray's One Euro filters (new hands/include/fh_smooth.h) run in continuous time and the
helper reads the ray every loop, so the pose sent to ft_pointer moves at about 100 Hz
instead of jumping at the tracker's 15-30 Hz (the dashboard's dragger made those jumps
jerky). The direction is filtered on the sphere. On the probe-moves replay the pose's
second difference per 90 Hz tick went from 0.90-0.96 deg rms to 0.24-0.26 at the same lag.
While a ray pinch is held the ray pivots on the shoulder as it was at the press, so looking
around doesn't move what's dragged; the reach for push and pull is filtered. A pinch's
first 0.3 s takes 6 deg of turn before it drags. One failed read of the gestures file no
longer ends a drag, and the log says why a drag ended or the ray went off.
Settings: POINTER_RAY_SMOOTH_MINCUTOFF, POINTER_RAY_SMOOTH_BETA, POINTER_RAY_SMOOTH_HOLD,
POINTER_RAY_STEADY, POINTER_RAY_TAP, POINTER_RAY_TAP_DEADZONE. rayeval reads publish dumps
and reports the smoothness at the display's rate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With gaze mode and POINTER_HANDS on, the gaze is the pointer and the hand ray
stays off (POINTER_GAZE_HAND_RAY=1 brings it back). A pinch clicks where you
looked 75 ms before ft-hands saw it; held and moved past 4 deg of hand
movement (POINTER_GAZE_PINCH_DEADZONE) it nudges the pointer at 0.3x
(POINTER_GAZE_PINCH_GAIN), and the correction is a gaze lesson. Held still for
POINTER_GAZE_HOLD it's a press the hand drags. A middle pinch right-clicks at
the gaze. One log line per pinch. Rules in pointer/helper/gazehand.h, tested
in pointer/tests/gazehand-test.cpp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Until the tracking is good enough for a hand-shaped hole, each hand cuts a
disk facing the eye, wide enough for the whole hand plus the margin, drawn
as the stadium around the ellipse each eye sees on the screen with a soft
edge. mesh and capsules stay selectable (HANDS_MATTE, cutouts matte).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live test 2026-10-09: taps took several tries and the laser jumped between
hands. The owner's ray dropping for a moment (closing the fingers turns the
hand) handed the laser to the other raised hand at once, for good; a pinch
by the other hand pressed where the old owner's ray pointed; a pinch whose
ray had just dropped became a press at a stale spot, or only woke the
pointer.
- The owner keeps the laser through POINTER_RAY_GRACE (0.5 s) of its ray
being down; the other hand takes it by pinching, by raising its ray while
the owner's is down, or after the grace (pointer/helper/rayowner.h).
- A pinch on a ray that dropped within the grace, or is on its way up,
brings it back (fh_ray.h ShowForPinch) and is a ray pinch.
- Aim lock: the press goes along the ray from just before the fingers
began to close, after a pose along it has gone out; a tap that ends
before its press clicks there too.
- Each ray follows one tracked hand through a brief flip of ft-hands'
left/right call.
- Log: owner changes with the reason, the owner's ray down and back, and
one line per pinch saying what it clicked or why not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/handsmode.py starts and stops ft-camd, ft-hands and the hand view
(frametop-handview.service, when installed) by HANDS_MODE: frametop =
while the desktop runs (session hooks), always = wanted by
steamvr.service (and kept running through game pauses), off = the
default, only ft-handsctl on as before. ft-handsctl mode sets it;
missing ft-camd capabilities start nothing and say what to run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turn a bare hand palm up toward your face and look at it: after 0.3 s a
small ring shows above the palm and follows it, like the Vision Pro's
home indicator. A thumb-index pinch tap by that hand runs
`ft-steam menu` (opens the Steam menu, or closes the dashboard); the
ring is blue while the pinch is held. ft-screens owns it
(screens/palmmenu.cpp) and reports that hand near while the ring shows,
so the pointer helper doesn't also gaze-click the pinch. The helper is
unchanged. HANDS_PALM_MENU=0 turns it off; `palmmenu on|off|state`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
