Hold an open hand out to a screen, remote display or floating window,
palm to it, then pull the arm back making a fist: the panel comes over
250 ms to 45 cm ahead of the eyes, at the fist's height, facing you
(user, 2026-10-10). Screens wider than HANDS_GRAB_THAT_WIDTH (0.6 m)
come at that width. The same gesture at a panel that came, still where
it came to, sends it back; the reset button sends floating windows back
(the layout reset places screens and remote displays itself).
screens/grabthat.{h,cpp}: the detector (pure logic): open fingers, palm
ahead and the wrist 28 cm out for 0.2 s arm it; the fingers closing with
the palm still to the panel, then on one frame a fist, the wrist 8 cm
back within 0.5 s and the palm turned over fire it. The panel is the one
the eye-through-palm line meets, else the one the head points at. From
the closing (thumb clear of the index: not a pinch) to 0.4 s after the
fire the hand counts as near, so the helper doesn't take the fist as a
gaze grip. Hands frames carry the head pose at capture now
(Hands::captureHead).
Tests replay the user's flag (fires twice, right hand, at -11.35 and
-5.89 s; expected -11.4 and -5.9), and the other 12 flags and 15 minutes
of recordings fire nothing (1 of 143 pinches, 3 of 12 grips begin on a
claimed hand); synthetic waves, pushes, fists without the pull, slow
pulls, a hand turned over first, held-out pinches, a drop to the lap and
typing don't fire. handinput-test --grabthat FLAG_DIR... scans flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Until the tracking is good enough for a hand-shaped hole, each hand cuts a
disk facing the eye, wide enough for the whole hand plus the margin, drawn
as the stadium around the ellipse each eye sees on the screen with a soft
edge. mesh and capsules stay selectable (HANDS_MATTE, cutouts matte).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fh_gestures version 3 (the middle pinch) is longer; ft-screens mapped exactly its own struct's
size, so an older ft-hands' file wasn't read at all, and the dumps written by one weren't either.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hull shader ran over each piece's box; a forearm across the panel covered a fraction of
it. Now the polygon grown by the margin and the feather is drawn, and fragments outside a
piece are discarded (no blend). The forearm is a stub toward the elbow (18 cm at the average
hand size, as the capsules' 16 cm): the elbow is a guess and a longer piece costs a bigger
redraw. tests/matte_bench.cpp (scripts/bench-matte.sh) times the composite offline on the GPU,
mesh against capsules, frame by frame in turn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- handcut: reads fh_hands version 2 (the skeleton's joints, radii, elbow, pinch point; version 1
still works) and the gestures file; the matte is hand-shaped: a tapered capsule per finger
bone, the palm and the thumb's web as convex slabs, the forearm toward the elbow, drawn with
a soft-edged convex polygon shader; its margin is an angle from the eyes plus motion
(HANDS_MATTE_MARGIN, _MOTION, _MOTION_MAX), the capsules stay selectable (HANDS_MATTE), and a
per-distance correction hook is off by default (HANDS_MATTE_SHIFT). The CPU part is in
handcut.cpp (no GPU or SteamVR calls), the renderer in handrender.cpp.
- handinput: the near claim, poke to click (index straight and nearest, two frames at the
surface, an approach speed or 1 cm deep, release 1.5 cm out or 1 cm back, the click held
still until a 1.5 cm slide, re-press guard, through-the-panel release), pokes on the curve,
reset, dock and close buttons, pinch on the bar, edges, corners, tab or title bar to move
(facing the head) or resize, two hands to stretch and turn, two hands to one, lost hands.
- vr.cpp: hands read once per tick, "hands <L> <R> <allowed>" from the pointer helper, "near
<capture_ns> <L> <R>" to it per hands frame, want-hz while near, glows on the cutout buffer,
hand holders for KWin's button and carried panels, the handinput and cutouts commands.
- tests/handinput_test.cpp: unit tests, and replays of ft-handreplay --publish-dump
(scripts/test-handinput.sh, justfile test-c).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A see-through test panel with dots on the wrist, middle knuckle and fingertips of each
tracked hand, one colour per timing: where the cameras saw the hand, moved ahead to now,
and moved ahead to now + the cutouts' lead. Recorded together with the headset view, they
show how far the cutouts land from the hand Room View shows, still and moving, and which
lead fits. Each tick goes to a JSON-lines log. Renderer::Marks draws the dots; Hands::Points
gives the landmarks. ft-screens' cutouts are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cutout composite drew each screen's whole client buffer into a side-by-side buffer on
every tick a hand was in front of it, then waited for the GPU with glFinish (1-6 ms, the
likely cause of the VR frame drops on 2026-10-01). Now:
- A drawn buffer gets a fence and is shown from a later tick once the fence has passed, so
ft-screens never waits for the GPU (except for a panel's first buffer after a pause, so a
stale one never shows). The prediction lead goes from 25 to 36 ms for that tick.
- Nothing is drawn when the client frame and the cutouts haven't changed, and SteamVR
isn't handed the same buffer again.
- When only the cutouts moved, a buffer that holds the same client frame is drawn again
only around the old and new cutouts (scissored).
- "cutouts state" reports draws, partial draws, unchanged ticks, busy ticks, waits and CPU
time per second since the last state; ft-handtest prints the same counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The runtime files move to /run/user/UID/frametop-hands/: the desktop
session deletes /run/user/UID/frametop at every start.
- The cutout copy shader runs at highp: mediump (16-bit on Adreno)
stepped 1.7 texels across a 3440-pixel screen.
- One hand no longer pinches both sides after its left/right call flips
mid-pinch, and --pinch-palm-down (0.6) holds back pinches with the palm
facing down (typing on a lap keyboard).
- ft-camd judges a colour frame fresh by its luma rows only, and logs
per-buffer changes at stale colour frames with FT_CAMD_DEBUG=1.
- hands/run.sh caps skips the setcap when ft-camd already has them.
- The replay tool dumps poses (--poses), and its pinch events carry the
hand id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-screens' hand cutouts read /run/user/UID/frametop/hands through
hands/include/fh_hands.h instead of their own copy of its offsets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tracked hands arrive 30-60 ms after the cameras saw them and reach the
displays later still, so holes trailed moving hands. Track each hand's palm
velocity in the room and move its capsules ahead to about when the frame is
on the displays, every tick, so the holes also move smoothly between tracker
updates. Slow hands aren't moved (their velocity is noise). The control
socket gets cutouts predict on|off and cutouts lead <ms>.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capsule end near the eyes' plane projects far across a screen with a huge
radius, so one bad hand estimate there tore a hole through the screens for a
moment. Clip capsules 12 cm in front of the eye, as the tracker now does too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Where frame-hands tracks a hand between an eye and a screen, that eye sees the
room through the screen. handcut.cpp draws the screen's buffer side by side
(one half per eye) with the hands cut out, only while a hand is in front of it.
The cutouts command turns it on or off. ft-handtest tries it on a test panel.
Not yet tested in the headset.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
