HANDS_PINCH_END, HANDS_PINCH_DRAG_OPEN, HANDS_PINCH_DRAG_OPEN_S, HANDS_GRIP_TAKEOVER_S
(user 2026-10-11, gaze + hands: pinches released at 4-6 cm, and grips took pinches
130 ms in, while the fingers were still together).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No pinch begins while the index and middle are both curled (tip-to-wrist
under fist_curl times knuckle-to-wrist, Grip's curl measure), and a close
held back so has to open past end_m before one can begin. A pinch already
down carries on. A fist puts the thumb tip on the curled index, and live
ft-hands read 96% of fist frames in the flags as pinched.
1.2 rather than the flag analysis' 1.3: the model's world landmarks read
pinching fingers more curled. Replays (student v11c) against the tap
labels: clear non-fist pinches lost 2 of 204 at 1.2 (4 at 1.3), labelled
fists read as pinches 8 -> 5.
ft-handreplay: --pinch-fist-curl, the guard's count in the summary, the
finger curls in the timeline. The flag manifest's settings gain
pinch_dip and pinch_fist_curl.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quick taps with the hand raised often read 25-30 mm at contact and spring
back without reaching the 2 cm begin, so they never click. With
HANDS_PINCH_DIP=1 (ft-hands --pinch-dip, ft-handreplay --pinch-dip) a pinch
also begins on a sharp dip: within 150 ms the distance falls by 40% of its
recent highest to under 3.2 cm, with the palm moving under 0.3 m/s and the
thumb at least 4 cm from the middle fingertip. It ends 6 mm over its lowest,
or becomes an ordinary pinch once the tips get within 2 cm. Off by default;
the middle pinch doesn't use it.
Guided tap capture replay (student v4): quick taps on a 2 Hz click 10 -> 25
of 30, clear teacher taps 0.62 -> 0.74 at precision 0.93 -> 0.91. Near
misses (fingers ~1 cm apart) now click: 5 -> 16 pinches in that segment.
Unit tests in tests/pinch_test.cpp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands keeps the last 12 s of hand crops (each view's landmark crop, 1.6x, at most
224 px), its hands, gestures and landmarks in RAM (track/flagbuf.h, HANDS_FLAG_BUFFER=1).
`ft-handsctl flag [note]` asks for them through run_dir/flag; ft-hands writes them and the
next 2 s to HANDS_FLAG_DIR/flag-<time> from a low-priority thread, for frame-hands'
offline tools. ft-handsctl also saves a /dev/video99 frame and ft-handview's head poses,
and the drawn hands pulse cyan when a flag is taken (ft-handview's new flash and poses
commands). ft-handreplay --flag-at/--flag-every makes the same flags from recordings.
Measured on a replayed ring: +58 MB peak RSS, 0.1 ms a crop on an A720 (about 1% of a
core). Unit tests: tests/flagbuf_test.cpp (make check), tests/test_handsflag.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pinch held 0.3 s or longer (a drag) ends only once the tips are 5 cm apart, or 0.12 s
after they first went past 3.5 cm without closing back under 3 cm. Drags in the 2026-10-09
headset test ended with the tips 3.5-4.0 cm apart. A grip takes a pinch only within 0.3 s
of its begin (closing the hand into a fist); after that no grip begins on that hand, so
curling the other fingers around a held pinch doesn't drop it (3 of 28 pinches in the same
test). Taps end as before.
On the lit recording's replayed distances, pinches held 0.3 s or more ended 33-134 ms later
than before, 200 at most. tests/pinch_test.cpp (make check).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Joins with hands-screens: ft-hands keeps 30 Hz for a middle pinch as well
as for a want-hz reader; GesturePublisher keeps both use() (replay dumps)
and write()'s middle-pinch slots; both test groups run in test-c.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With hand interaction on (POINTER_HANDS=1), the hands have to be there
when the dashboard opens during a game (design section 5.5):
- ft-hands: while /run/user/UID/frametop-hands/park exists it processes
no frames (models stay loaded), publishes no hands, ends gestures as
lost, and writes 0 to hands-want every second.
- ft-camd: stops copying the mono frames while hands-want is fresh and 0
(a dead ft-hands' stale file doesn't starve other ring readers).
- The input relay (game_pause.py) leaves the hand services running when
it pauses for a game and parks them instead; it unparks while the
dashboard is open (the helper's new "vrdash 1|0") and on resume.
- hands/README.md: gestures v3, parking, and the hand ray.
- pause-test.py: parking, and "paused" kept off the live helper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands publishes a middle pinch per side in the gestures file's new
pinch2 slots (version 3; older readers copy the size they know). The
pointer helper's hand ray makes it a right click or scrolling.
- Pinch takes the fingertip (8 or 12), the other one it must be clearly
nearer than (1 cm for the middle pinch), and the two pinches exclude
each other on a hand. A middle pinch also wants the index extended and
the hand tracked 5 frames: closing the hand into a grab brought the
middle tip to the thumb (PR #8 gestures take: 6 false middle pinches
before, 1 after; the 2026-09-30 lit recording: none, and the same 48
pinches).
- ft-handreplay runs it too (timeline: pinch2); watch_gestures.py reads v3.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--publish-dump writes what ft-hands would have published after each processed set (the hands
and gestures files' bytes), for ft-screens' offline hand input tests. want-hz: a reader
(ft-screens with a hand near a panel) asks for up to 30 Hz by keeping the file fresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts were side by side (options, tracker setters, replay flags). Two
joins: Hand::right() keeps the pairing's side and the skeleton's fields,
and the skeleton now fits after the left/right pairing, so it fits the
side ft-hands publishes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fine-tuned models (the hand dataset's student models) can't ship in the
repo yet, so a headset that has them sets HANDS_MODELS in
~/.config/frametop.conf instead of passing --models to every launcher
(service, ft-cutouts, recorder, probes). --models still overrides it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live (2026-10-08 probe) a hand that dropped out of tracking stayed gone for
2-4 s while plainly in view. Cause: the palm search is the only way back once
a hand is forgotten (0.3 s), and it tries 2 of 42 tiles per round, so even the
likeliest tile comes round every 10-11 rounds: ~2 s at 5 rounds/s, 4-5 s live
(a round costs ~52 ms beside SteamVR). A quarter of the probe's per-side
"missing" time was a labelling error instead: both hands published, both
called the same side.
Tracker (on in ft-hands by default; off in ft-handreplay unless asked):
- --reacquire S (1.5): from the first step a hand (3+ frames) is missing
until S after it's forgotten, the palm search aims up to 2 tiles at its
predicted position in the cameras that see it best, every processed set;
the rotation keeps at least one tile. Held hands' hand-over crops follow
their velocity.
- --pair-sides (1): two hands seen together are one left and one right, by
fading log-odds of the model's calls (wrist order breaks close calls).
- --hold S (0.6): an established hand (30+ frames) is held 0.6 s, not 0.3.
Replay tooling: tracker trace events in --timeline, --cost-hand/--cost-palm
(live costs 14/52), per-side gap counts, label-free oracle miss rate,
--drop-every (blind a hand 0.35 s, time the re-find), tools/gap_report.py
(gap causes), tools/replay_gaps.sh. Results: frame-hands notes/reacquire.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands fits a skeleton to each hand's 21 landmarks after the 3D step
(track/skeleton.{h,cpp}):
- Bone lengths and palm shape are learned per side from clean stereo
frames (medians, then a slow running median), with the model's average
hand times Hand::scale, or the other side, as the prior.
- Each frame: the palm as a rigid body, then each finger and the thumb as
a chain with joint limits (Gauss-Newton), the chain bases allowed a few
mm off the rigid palm. Landmarks are weighted by their triangulation
residual and down-weighted along the line of sight.
- The pose (palm centre, palm rotation, joint angles) is filtered, not the
points; each finger's cutoff follows its own speed.
- An elbow from the arm model in frame-hands' notes/forearm.md.
The hands file goes to version 2: version 1's struct is unchanged and an
fh_hand_fit_t per hand follows it (joints as position + quaternion,
bone lengths, angles, elbow, pinch point, fit quality, head-frame
velocities). Version 1's points and capsules come from the skeleton
(HANDS_SKELETON=0 or --skeleton 0 for the old ones). The hand
recorder's reader accepts version 2.
ft-handreplay reports the fit (track/skeval.h): time, residual,
calibration, bone-length spread, jitter, tip lag, elbow, and with
--labels the distance to frame-hands' auto-labels. SetReader reads a
hand-recorder take's later parts (sets-2.bin, ...).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands --misread-guard 1 (HANDS_MISREAD_GUARD=1) and ft-handreplay --misread-guard. Fine-tuned
landmark models stay sure of a hand when two hands touch, and can read the held hand as the
other side: the two cameras' views then miss by 3-4 cm, the tracker splits one off as a new
hand, the hand-over puts the old hand back onto it, the duplicate check drops the new one, and
round it goes (a contributed session, holding a card with both hands: 51 duplicates and 40
splits in 59 s with the fine-tuned models, 5 and 9 with the stock ones). With the guard, a
reading whose side is more than 0.7 off its established hand's is dropped, and a split-off view
that sits where its hand already is in that camera is dropped instead of starting a new hand:
21 duplicates and 10 splits, both hands tracked as often. Off by default: the stock model's side
calls are noisier, and the first guard costs it tracking on some recordings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The side cameras came out swapped on most SteamVR starts, and right on some.
Two guesses combined:
- ft-hands and camcheck.py named each video device by its TrackingCameraInit
index (0 = slam_left). The index is only the order XRService opens the
cameras in: on every start logged since 2026-10-04, index 0 was slam_right.
XRService names each sensor subdev itself ("Found camera 'slam_left': ...
v4l_subdev=/dev/v4l-subdev30"), and each TrackingCameraInit line says which
subdev it opened. Name the devices by that; the index order stays the
fallback for a log without those lines. The capture-pipe fallback had vfe3
and vfe4 backwards too.
- ft-camd bound each run of XRService's buffers to a device by the sensor
subdev opened just before it in XRService's fd table. That order changes
when XRService restarts its cameras (17:09 today: the run after
og01a1b 4-0060's subdev was video9's). VIDIOC_QUERYBUF on the device names
the descriptor XRService queued at each index, from any handle, so bind runs
by that, and split a run holding two cameras' buffers (the upper pair and the
colour pair came as 32-buffer runs). The order stays the fallback.
Checked on the live XRService: all four tracking cameras "bound by
VIDIOC_QUERYBUF" with 16 buffers each, and ft-hands now reads
slam_left=video13 slam_right=video9. ft-hands' hands-based side check stays
as the safety net.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With HANDS_SWAP_SIDES set to 0 or 1, ft-hands kept the forced naming as the
published truth even after the hands showed it was backwards. The hand recorder
took that as the session's decision, so the export labelled slam_left and
slam_right the wrong way round (dataset PR #4: every take swapped).
ft-hands now publishes the hands' answer once they disagree (state "forced,
disagrees"); tracking keeps the forced names. sides.read_live corrects the same
case from an ft-hands built before this, and the recorder's log says to use auto.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without the Arcturus module, XRService runs the side cameras through the
ISP ("ISP enabled for tracking cameras (main VFE available)"): NV12 on
vfe0 and vfe1, pitch 1152. ft-camd published only grey cameras, so it
dropped them, and the recorder stopped at "ft-camd publishes only 2 of 4
mono cameras" whatever was restarted.
- ft-camd reads a tracking camera's NV12 luma as its grey image.
- ft-hands and check_sides name the cameras by XRService's numbering in
its log (TrackingCameraInit index 0-3), not by capture pipe, which
moves with the module; a camera whose size isn't its calibration's is
left out. sides.json says which device each camera was.
- camcheck reports the camera map, the ISP routing and which cameras
ft-camd is missing; the recorder says so instead of "restart it",
restarts its own ft-camd once when that's the only problem, and keeps
the map in session.json.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 4db5266cf7)
ft-camd tells the side cameras apart by XRService's buffer allocation order,
which some XRService starts reverse; both of 2026-10-02's starts did, so the
cutouts missed the hands. HANDS_SWAP_SIDES=auto (the default) has ft-hands
vote from hands seen in both side cameras: the landmark rays meet in front
of both cameras only under the right naming. While undecided it probes the
exchanged naming with the landmark model. It decides in about 2 s of hands
(right on all 7 recordings replayed), swaps the views in place, and publishes
sides.json. 0 and 1 still force it, with a warning when the hands disagree.
Recordings carry each part's naming and the session's decision; review,
export, validate and ft-handreplay put the names right, and takes.py sides
records a decision by hand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands --record-hz N records at most N frame sets a second, for the hand recorder (10).
DESIGN.md lays out the recorder: the headset panel, the session runner and its script,
the files, review and export, consent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/ft-cutouts on|off|status starts ft-camd and ft-hands as transient
user units with ft-hands' new --no-gestures: hands are published for
ft-screens' cutouts, but no pinch or grip is detected, so nothing clicks
or drags and a closing hand doesn't raise the tracking rate. It needs a
build and ft-camd's capabilities, not hands/run.sh install. Its units
conflict with ft-handsctl's, so each stops the other, and they stop with
SteamVR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- gaze/tracker/lab/eyes_track.py: nothing used it
- Input Settings: the pointer role backend, whose page was removed
- ft-screens: no log line for every floating-window resize
- hands: ft-hands --help gives the palm-down default (1: off), the
uninstall removes ft-handsctl's link, .frame-job is ignored
- Display Settings: "Save as profile…", not "Save current arrangement"
- two stale comments (ft-pointer's grabprobe, ft-gaze)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The palm-down filter is off by default: the user's deliberate pinches,
hand raised, read 0.90-0.99, like typing.
- Typing is told apart by the keyboard instead: the input relay sends the
pointer helper "typing" on key presses, and it takes no pinch within
POINTER_PINCH_TYPING (1 s) of one.
- Grips are still held to hands raised (POINTER_GRIP_BELOW, 0.35 m below
the eyes); pinches aren't, since the user's own sat 0.35-0.45 m below,
elbow resting.
- A grip doesn't begin with the thumb on the index tip: that's a pinch
with the other fingers curled, which was taken for a grip.
- A pinch's point is the index and middle knuckles: the tips' midpoint
moved 1-2 cm as the pinch opened, dragging every release off its press.
- ft-hands --gesture-log prints what the detectors measure, 10 times a
second.
Pinches still aren't reliable enough to use; hand tracking is parked for
now in favour of the controllers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands tracks with the mono IR cameras in dim light and with every
camera (or the colour pair, HANDS_BRIGHT) in bright light, going by the
colour frames' mean brightness with hysteresis and a 2 s hold
(HANDS_CAMERAS=auto, the default; mono, color and all fix it). Colour
frames are placed on the mono cameras' clock by their dequeue time, and a
view in a camera a step lacks waits for that camera's next frame.
A grip (a closed hand) is a second gesture next to the pinch, in version 2
of the gestures file: every fingertip curled toward the wrist, beginning
only on a hand seen open within a second and held up in front. On the
2026-09-30 lit recording that leaves 6 false grips of 14, all with the
hands on the desk; pinch counts are unchanged. ft-handreplay logs grips
and finger curl, watch_gestures.py shows them, and tools/cut_sets.py
copies a few sets out of a recording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The runtime files move to /run/user/UID/frametop-hands/: the desktop
session deletes /run/user/UID/frametop at every start.
- The cutout copy shader runs at highp: mediump (16-bit on Adreno)
stepped 1.7 texels across a 3440-pixel screen.
- One hand no longer pinches both sides after its left/right call flips
mid-pinch, and --pinch-palm-down (0.6) holds back pinches with the palm
facing down (typing on a lap keyboard).
- ft-camd judges a colour frame fresh by its luma rows only, and logs
per-buffer changes at stale colour frames with FT_CAMD_DEBUG=1.
- hands/run.sh caps skips the setcap when ft-camd already has them.
- The replay tool dumps poses (--poses), and its pinch events carry the
hand id.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Programs: ft-camd (the camera broker), ft-hands (the tracker), and
ft-handreplay and ft-ringplay for recordings, built by hands/build.sh
into hands/build/ with one Makefile. The first build fetches ncnn at
frame-hands' pinned tag and builds it with the same options.
- ft-camd gets its privileges from file capabilities (CAP_SYS_PTRACE,
CAP_PERFMON, CAP_DAC_READ_SEARCH) that hands/run.sh install sets with
sudo, and drops them once set up. It still works under sudo. It runs
on the host, linked statically, as frametop-camd.service. ft-hands
runs in the dev container as frametop-hands.service. Both start and
stop with SteamVR.
- Files move to /run/user/UID/frametop/ (cam-ring, hands, gestures),
not $XDG_RUNTIME_DIR, which a terminal in the Frametop desktop has
its own of. SIGUSR1 recordings go to ~/.local/share/frametop/hands.
- The calibration is read through /run/host in the container.
- Settings: HANDS_SWAP_SIDES and HANDS_CPUS in frametop.conf.
- install.sh offers hand tracking as an optional last step.
- The container gets jsoncpp-devel, glibc-static, and NumPy and OpenCV
for the Python tools.
- tools/ring.py reads the ring, and models/NOTICE credits the
Apache-2.0 models.
Checked: ft-handreplay gives identical summaries and byte-identical
depth dumps to frame-hands' fh-replay on both 2026-09-29 recordings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trackd/ becomes track/, and the calibration helper the Python tools
import moves from the prototype's folder into tools/. The model
development tools (nettest and the scripts that compare it with the
Python models or cut int8 calibration crops) stay in frame-hands with
the prototype they need.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
