Lazy susan (the third live test): a brought screen comes 1 m ahead, as big as its
resolution needs for text to read: its logical pixels (KWin's scale undone, now passed
to vr.cpp) at 26 px/deg, at most 100x55 deg keeping its shape, at least 40 deg wide,
curved round you past 60 deg (grabthat::ReadableSize; SCREEN_SPIN_DISTANCE, _PPD,
_MAX_DEG, _MAX_TALL_DEG). Its curve comes back on the way home. The strokes cycled
104, 1, 104 with three screens up: a screen still flying home read as straight ahead;
each one is now taken where it lives (home, or where it's going).
Accidental clicks: most were pinches with no laser up pressing where the pointer was
last left ("no ray up: a press where the pointer is", a dozen in two minutes). With
gaze off they now only wake the pointer (POINTER_PINCH_NO_RAY=1: the old air mouse).
The hand's laser shows only with the thumb and index within 6 cm (gone past 10;
POINTER_RAY_READY), and a fist drops it: fh_ray's "ready" and "open" tests, the fist
read as the index and the other three both curled under 1.1 (FistCurl; measured on 47k
flagged hands: the other three alone read as a fist in a fifth of the pinches). A ray
pinch made while the hand closes into a fist doesn't click. Input Settings switches for
all three (POINTER_ ones reload the helper).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fh_smooth.h's Params get maxCutoff (0, the default: no limit, as before for the ray and the
pointer helper). With the tracking gate on, a carried panel's turn (the wrist turn, a card's room
turn) is smoothed at most at 4 Hz however fast the hand's frame turns, so a flip the gate takes
after three agreeing frames is followed over about 40 ms instead of in one tick. On the third live
test's flags, carried panels' turn per 90 Hz tick: second difference rms 1.87 -> 1.60 deg, 99th
percentile 6.2 -> 5.2, the centre's 11.5 -> 9.7 mm (hands-throw: 33.3 deg and 159 mm).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Losing the hand for 250 ms (or ft-hands ending its pinch as lost) dropped
whatever it held. Now a pinch, card or fist grab, and a poke's or a
pinch's press in KWin, last until a clear release: the hand seen with
its gesture up and its fingers open (thumb to index tip over 3.5 cm; a
fist's fingers over 1.35 extended) on 2 hands frames in a row, or the
gesture up for 6 frames whatever the fingers say. A pinch that
flickers up for a frame or two in fast motion, or that ft-hands ended
as lost while the hand comes back still pinched, holds on. Unseen for
1.75 s (HANDS_HOLD_LOST_MS), it ends where it is.
While unseen (or on a frame whose point jumped more than 12 cm, up to
2 in a row: a glitch), the carried point coasts on the hand's last
velocity (a fit over 110 ms), slowing to a stop over 150 ms. When the
hand is back, the smoothing restarts there moving at the gap's average
velocity (fh_smooth Point::Seed, new) and the panel blends from the
coast to it (smoothstep over 100 ms, longer for more than 0.6 m/s on
average). The raw track starts afresh, so a throw's release velocity
never sees the gap.
Synthetic carries at 0.6 m/s losing the hand for 3, 6 and 10 frames:
held, the panel's largest step 12-17 mm a tick (6.7 moving; 13-24 with
3 mm of noise), back under the hand after. Two glitch frames 30 cm off:
11 mm. Release now comes 2 frames (67 ms) after the fingers open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ideas from StereoKit (MIT, 07b9f0d), credited where a formula comes from it:
- The torso faces the head's forward counted twice plus the direction to
each wrist seen (POINTER_RAY_TORSO_WRISTS, on), so looking aside while
pointing swings the shoulders, and the ray, less. probe-moves replay: the
shoulder's share of a still hand's ray motion 20 -> 18%, p90 2.0-3.7 ->
1.7-3.6 deg/s; the tap capture: nothing worse.
- The ray can turn toward the hand's own direction, from the palm's axes
(wrist, middle, ring and little knuckles) tilted -30/8 deg
(POINTER_RAY_HAND_WEIGHT, 0). Off: every weight measured worse (at 0.3:
judder 0.07-0.14 -> 0.11-0.33 deg rms per 90 Hz tick, raw jumps over
3 deg 159 -> 330). The gate still tests the shoulder's ray.
- The knuckle point can be pinned to the palm's axes during a pinch, easing
back on the steady shoulder's schedule (POINTER_RAY_ANCHOR, 0). Off: our
pinch point is the knuckles already, and pinned it shook as much or more
and kicked no less on release, fitted or not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ray's One Euro filters (new hands/include/fh_smooth.h) run in continuous time and the
helper reads the ray every loop, so the pose sent to ft_pointer moves at about 100 Hz
instead of jumping at the tracker's 15-30 Hz (the dashboard's dragger made those jumps
jerky). The direction is filtered on the sphere. On the probe-moves replay the pose's
second difference per 90 Hz tick went from 0.90-0.96 deg rms to 0.24-0.26 at the same lag.
While a ray pinch is held the ray pivots on the shoulder as it was at the press, so looking
around doesn't move what's dragged; the reach for push and pull is filtered. A pinch's
first 0.3 s takes 6 deg of turn before it drags. One failed read of the gestures file no
longer ends a drag, and the log says why a drag ended or the ray went off.
Settings: POINTER_RAY_SMOOTH_MINCUTOFF, POINTER_RAY_SMOOTH_BETA, POINTER_RAY_SMOOTH_HOLD,
POINTER_RAY_STEADY, POINTER_RAY_TAP, POINTER_RAY_TAP_DEADZONE. rayeval reads publish dumps
and reports the smoothness at the display's rate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live test 2026-10-09: taps took several tries and the laser jumped between
hands. The owner's ray dropping for a moment (closing the fingers turns the
hand) handed the laser to the other raised hand at once, for good; a pinch
by the other hand pressed where the old owner's ray pointed; a pinch whose
ray had just dropped became a press at a stale spot, or only woke the
pointer.
- The owner keeps the laser through POINTER_RAY_GRACE (0.5 s) of its ray
being down; the other hand takes it by pinching, by raising its ray while
the owner's is down, or after the grace (pointer/helper/rayowner.h).
- A pinch on a ray that dropped within the grace, or is on its way up,
brings it back (fh_ray.h ShowForPinch) and is a ray pinch.
- Aim lock: the press goes along the ray from just before the fingers
began to close, after a pose along it has gone out; a tap that ends
before its press clicks there too.
- Each ray follows one tracked hand through a brief flip of ft-hands'
left/right call.
- Log: owner changes with the reason, the owner's ray down and back, and
one line per pinch saying what it clicked or why not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/include/fh_ray.h, shared by the pointer helper and ft-screens (design
sections 2.4-2.5): a body model (torso yaw = head yaw low-passed, at most
30 deg behind; shoulders from it), the ray from the shoulder through the
knuckles (20 ms prediction), the pointing gate (show tests for 120 ms,
hide after 250 ms of a failed keep test; near and "off" hide at once;
held through a ray pinch), and One Euro filters on yaw, pitch and origin.
- pointer/tests/handray-test.cpp: body, gate timing and tests, filter
(just test-c).
- ft-handreplay --ray FILE: per set and hand, what the helper reads.
- pointer/tools/rayeval: the ray over a replay, with the head poses from the
ft-handtest probe's log of the same session (clock matched by the
published capture times): how often it shows, why not, jitter, the ray's
shift into a pinch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per side, a controller used recently wins, else the bare hand
(pointer/helper/handoff.h, design section 1): touch and presses from the
input relay, motion, charging and tracking from SteamVR, and the tracked
hands from ft-hands' hands file (handsin.h, which also takes the gestures
reader and the head pose history out of ft-pointer.cpp).
- The relay's web socket reader (game_pause.py TouchTracker) sends
"ctl <side> touch 0|1", "ctl <side> press", "ctl <side> gone" and
"ctl ws 0|1" with POINTER_HANDS=1, and the pause state as "paused 1|0".
- The helper logs transitions and sends ft-screens "hands <L> <R> <allowed>"
(hand|controller|off, 0|1) on a change and every second. Nothing acts
on it yet.
- fh_gestures v3 adds the middle pinch slots (written as zero for now).
- Tests: pointer/tests/handoff-test.cpp (just test-c), input/test/touch-test.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands fits a skeleton to each hand's 21 landmarks after the 3D step
(track/skeleton.{h,cpp}):
- Bone lengths and palm shape are learned per side from clean stereo
frames (medians, then a slow running median), with the model's average
hand times Hand::scale, or the other side, as the prior.
- Each frame: the palm as a rigid body, then each finger and the thumb as
a chain with joint limits (Gauss-Newton), the chain bases allowed a few
mm off the rigid palm. Landmarks are weighted by their triangulation
residual and down-weighted along the line of sight.
- The pose (palm centre, palm rotation, joint angles) is filtered, not the
points; each finger's cutoff follows its own speed.
- An elbow from the arm model in frame-hands' notes/forearm.md.
The hands file goes to version 2: version 1's struct is unchanged and an
fh_hand_fit_t per hand follows it (joints as position + quaternion,
bone lengths, angles, elbow, pinch point, fit quality, head-frame
velocities). Version 1's points and capsules come from the skeleton
(HANDS_SKELETON=0 or --skeleton 0 for the old ones). The hand
recorder's reader accepts version 2.
ft-handreplay reports the fit (track/skeval.h): time, residual,
calibration, bone-length spread, jitter, tip lag, elbow, and with
--labels the distance to frame-hands' auto-labels. SetReader reads a
hand-recorder take's later parts (sets-2.bin, ...).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The palm-down filter is off by default: the user's deliberate pinches,
hand raised, read 0.90-0.99, like typing.
- Typing is told apart by the keyboard instead: the input relay sends the
pointer helper "typing" on key presses, and it takes no pinch within
POINTER_PINCH_TYPING (1 s) of one.
- Grips are still held to hands raised (POINTER_GRIP_BELOW, 0.35 m below
the eyes); pinches aren't, since the user's own sat 0.35-0.45 m below,
elbow resting.
- A grip doesn't begin with the thumb on the index tip: that's a pinch
with the other fingers curled, which was taken for a grip.
- A pinch's point is the index and middle knuckles: the tips' midpoint
moved 1-2 cm as the pinch opened, dragging every release off its press.
- ft-hands --gesture-log prints what the detectors measure, 10 times a
second.
Pinches still aren't reliable enough to use; hand tracking is parked for
now in favour of the controllers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands tracks with the mono IR cameras in dim light and with every
camera (or the colour pair, HANDS_BRIGHT) in bright light, going by the
colour frames' mean brightness with hysteresis and a 2 s hold
(HANDS_CAMERAS=auto, the default; mono, color and all fix it). Colour
frames are placed on the mono cameras' clock by their dequeue time, and a
view in a camera a step lacks waits for that camera's next frame.
A grip (a closed hand) is a second gesture next to the pinch, in version 2
of the gestures file: every fingertip curled toward the wrist, beginning
only on a hand seen open within a second and held up in front. On the
2026-09-30 lit recording that leaves 6 false grips of 14, all with the
hands on the desk; pinch counts are unchanged. ft-handreplay logs grips
and finger curl, watch_gestures.py shows them, and tools/cut_sets.py
copies a few sets out of a recording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The runtime files move to /run/user/UID/frametop-hands/: the desktop
session deletes /run/user/UID/frametop at every start.
- The cutout copy shader runs at highp: mediump (16-bit on Adreno)
stepped 1.7 texels across a 3440-pixel screen.
- One hand no longer pinches both sides after its left/right call flips
mid-pinch, and --pinch-palm-down (0.6) holds back pinches with the palm
facing down (typing on a lap keyboard).
- ft-camd judges a colour frame fresh by its luma rows only, and logs
per-buffer changes at stale colour frames with FT_CAMD_DEBUG=1.
- hands/run.sh caps skips the setcap when ft-camd already has them.
- The replay tool dumps poses (--poses), and its pinch events carry the
hand id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Programs: ft-camd (the camera broker), ft-hands (the tracker), and
ft-handreplay and ft-ringplay for recordings, built by hands/build.sh
into hands/build/ with one Makefile. The first build fetches ncnn at
frame-hands' pinned tag and builds it with the same options.
- ft-camd gets its privileges from file capabilities (CAP_SYS_PTRACE,
CAP_PERFMON, CAP_DAC_READ_SEARCH) that hands/run.sh install sets with
sudo, and drops them once set up. It still works under sudo. It runs
on the host, linked statically, as frametop-camd.service. ft-hands
runs in the dev container as frametop-hands.service. Both start and
stop with SteamVR.
- Files move to /run/user/UID/frametop/ (cam-ring, hands, gestures),
not $XDG_RUNTIME_DIR, which a terminal in the Frametop desktop has
its own of. SIGUSR1 recordings go to ~/.local/share/frametop/hands.
- The calibration is read through /run/host in the container.
- Settings: HANDS_SWAP_SIDES and HANDS_CPUS in frametop.conf.
- install.sh offers hand tracking as an optional last step.
- The container gets jsoncpp-devel, glibc-static, and NumPy and OpenCV
for the Python tools.
- tools/ring.py reads the ring, and models/NOTICE credits the
Apache-2.0 models.
Checked: ft-handreplay gives identical summaries and byte-identical
depth dumps to frame-hands' fh-replay on both 2026-09-29 recordings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The history of frame-hands (~/Desktop/Projects/frame-hands on the
Frame), filtered to what moves: the camera broker (camd/), the tracker
and its offline tools (trackd/), the shared file layouts (include/), the
ncnn models, the analysis tools, and the calibration and model helpers
they import from the Python prototype. The reverse-engineering notes,
probes, camprobe, and the rest of the prototype stay in frame-hands.
Unchanged here: the renames to ft- names and Frametop paths follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
