Quest-style: a see-through light hand from the tracked skeleton with a bright
rim, fading out behind the wrist, over everything. Each hand is a placed,
side-by-side overlay facing the head, so it stays put as the head turns and
redraws only as the hand moves. Hidden while a hand holds a controller (the
pointer helper's word, now also on @ft_handview). frametop-handview.service
runs it; nothing installs it yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live test 2026-10-09: taps took several tries and the laser jumped between
hands. The owner's ray dropping for a moment (closing the fingers turns the
hand) handed the laser to the other raised hand at once, for good; a pinch
by the other hand pressed where the old owner's ray pointed; a pinch whose
ray had just dropped became a press at a stale spot, or only woke the
pointer.
- The owner keeps the laser through POINTER_RAY_GRACE (0.5 s) of its ray
being down; the other hand takes it by pinching, by raising its ray while
the owner's is down, or after the grace (pointer/helper/rayowner.h).
- A pinch on a ray that dropped within the grace, or is on its way up,
brings it back (fh_ray.h ShowForPinch) and is a ray pinch.
- Aim lock: the press goes along the ray from just before the fingers
began to close, after a pose along it has gone out; a tap that ends
before its press clicks there too.
- Each ray follows one tracked hand through a brief flip of ft-hands'
left/right call.
- Log: owner changes with the reason, the owner's ray down and back, and
one line per pinch saying what it clicked or why not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A gesture that waited for ft-screens' near report is a ray gesture only if
its ray is still up when it starts; a ray pinch's press waits until a
pointer the ray just woke is bound (350 ms). keys-test's pause stub knows
the relay's new calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With hand interaction on (POINTER_HANDS=1), the hands have to be there
when the dashboard opens during a game (design section 5.5):
- ft-hands: while /run/user/UID/frametop-hands/park exists it processes
no frames (models stay loaded), publishes no hands, ends gestures as
lost, and writes 0 to hands-want every second.
- ft-camd: stops copying the mono frames while hands-want is fresh and 0
(a dead ft-hands' stale file doesn't starve other ring readers).
- The input relay (game_pause.py) leaves the hand services running when
it pauses for a game and parks them instead; it unparks while the
dashboard is open (the helper's new "vrdash 1|0") and on resume.
- hands/README.md: gestures v3, parking, and the hand ray.
- pause-test.py: parking, and "paused" kept off the live helper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With POINTER_HANDS=1 and POINTER_HAND_RAY=1 (design sections 2, 5):
- Per side, each hands frame goes through fh_ray's gate; the first ray up
drives the pointer (a pinch by the other hand's ray takes it). It wakes
the pointer and claims the laser unless a controller pressed within 2 s.
The mouse takes the pointer back; the ray returns after 1 s of no mouse.
- The pose is the ray's origin and direction (a visible beam); our dot only
where it meets a panel, no catcher in free space. A controller taking
the laser hides our dot until a ray pinch claims it back.
- A ray pinch is the left button, 30 ms after a claim it had to make;
the direction holds until POINTER_RAY_DEADZONE (2.5 deg), then drags.
A middle pinch (fh_gestures v3) is a right click, or scrolling when
held and moved. Gaze pinches and grips are unchanged with the ray down.
- Push and pull while carrying (POINTER_RAY_PUSH): the origin moves along
the ray, distance d0 (h/h0)^3.
- ft-screens' "near <capture_ns> <L> <R>": a gesture waits for the report
on its frame (60 ms at most) and is left to ft-screens if near.
- The ray lets go of a pointer it woke POINTER_RAY_RELEASE after it went;
at once when its side's controller is in use or in a game.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/include/fh_ray.h, shared by the pointer helper and ft-screens (design
sections 2.4-2.5): a body model (torso yaw = head yaw low-passed, at most
30 deg behind; shoulders from it), the ray from the shoulder through the
knuckles (20 ms prediction), the pointing gate (show tests for 120 ms,
hide after 250 ms of a failed keep test; near and "off" hide at once;
held through a ray pinch), and One Euro filters on yaw, pitch and origin.
- pointer/tests/handray-test.cpp: body, gate timing and tests, filter
(just test-c).
- ft-handreplay --ray FILE: per set and hand, what the helper reads.
- pointer/tools/rayeval: the ray over a replay, with the head poses from the
ft-handtest probe's log of the same session (clock matched by the
published capture times): how often it shows, why not, jitter, the ray's
shift into a pinch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The driver's default role is the treadmill (4), hinted from Activate;
the helper's POINTER_ROLE defaults to treadmill. Needs the driver
reinstalled and a SteamVR restart to take effect.
- The "no hand role" release skips a device whose hint is the treadmill.
- Hand gestures: none begins on a side whose controller is in use, or by a
hand at a controller (by position); a held one ends, as lost, when its
side turns controller.
- The moving-controller release no longer applies while a bare hand's ray
drives the pointer (that ray goes only when its own side turns controller).
- update-check.py: the virtual controller's path, and the touch components.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per side, a controller used recently wins, else the bare hand
(pointer/helper/handoff.h, design section 1): touch and presses from the
input relay, motion, charging and tracking from SteamVR, and the tracked
hands from ft-hands' hands file (handsin.h, which also takes the gestures
reader and the head pose history out of ft-pointer.cpp).
- The relay's web socket reader (game_pause.py TouchTracker) sends
"ctl <side> touch 0|1", "ctl <side> press", "ctl <side> gone" and
"ctl ws 0|1" with POINTER_HANDS=1, and the pause state as "paused 1|0".
- The helper logs transitions and sends ft-screens "hands <L> <R> <allowed>"
(hand|controller|off, 0|1) on a change and every second. Nothing acts
on it yet.
- fh_gestures v3 adds the middle pinch slots (written as zero for now).
- Tests: pointer/tests/handoff-test.cpp (just test-c), input/test/touch-test.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The driver takes "role treadmill" and hints its role from Activate, so a
treadmill gets /user/treadmill (from gaze-first 87a6393); the compositor
binding has the /user/treadmill entries. The default role stays the right
hand until the helper side is ready. docs/design.md says why (Treadmill role).
Probes for frame-hands/notes/hand-interaction.md section 7:
- pointer/probe/lasertest (from gaze-first): who has the laser, claim time
- pointer/probe/ctlnoise: controller pose noise, activity level, charging
- vrprobe prints each device's activity level and charging
- input/vrws.py --touch prints touch and value components
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every place an installed Frametop started a program with `distrobox enter dev --` now goes
through scripts/in-box: the pointer and power units, ft-gazed (ft-gaze, ft-eyes), the gaze
check's panel, the desktop's ft-screens, the remote desktop's krdp and VNC, and the three
settings apps. in-box starts the container in a scope of its own (container-up.sh, which now
takes the container's name) and becomes `distrobox enter BOX --`, so callers behave as before.
BOX is "dev" for a source install. A release names its own container in .frametop-release,
so the same tree runs from an image without editing nine call sites. FRAMETOP_BOX overrides
both. Development tools (frame.sh, the headless test, the gaze probe and lab) and hand
tracking, which install.sh doesn't install, keep using the dev container.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
screens/build.sh conflicted: experimental added remote.c to ft-screens, and this branch
links OpenVR through scripts/openvr.sh. Both kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A remote display is a screen of ft-screens whose picture comes from another computer,
streamed by Vibepollo with Moonlight's protocol. ft-screens (screens/remote.c) runs one
ft-stream per remote screen (101 and up) over a socket pair: ft-stream (stream/, GPLv3:
moonlight-common-c and moonlight-embedded's libgamestream, pinned) decodes on the iris
decoder into a ring of three RGBA buffers that ft-screens shows as a panel with a
screen's controls, layout, profiles, curve, pins and hand cutouts. Mouse, keys and scroll
go back to the host; a button's release goes through the stream it was pressed on, as
Vibepollo wants, and a press carried onto another remote panel moves the pointer there.
A panel out of sight gets 10 fps; the host's sound plays from one stream per host.
ft-stream signs in with an API token and the host's pinned public key
(stream/host.cpp), and chooses its path: a Steam Link dongle on the Frame's hotspot when
it answers, or the network (the layout's "route": auto, network or dongle). ft-layout
gets "remote" commands (list, add, connect, disconnect, host route and direct), starts
the connected displays with the desktop and keeps disconnected ones parked. ft-gaze and
ft-pointer know the remote panels. ft-screens now shuts its VR side down before its
streams and waits for SteamVR before exiting (a vrcompositor crash at quit, in standby).
docs/remote-displays.md has the design, the spikes and the test results.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/openvr.sh is the one place for the OpenVR SDK the programs build
against: the v2.15.6 headers, fetched into build/include and checked by
sha256, and the API library. ft-pointer, the ft_pointer driver, and
ft-powerd now build against those headers like ft-screens and ft-gaze,
instead of the 2.1.0 copy in SteamVR's samples, which has no public tag
an image could pin. The library stays SteamVR's own; OPENVR_LIB links
another copy (the image has no SteamVR), and the rpath still names
SteamVR's folder first.
FRAME_IN_BOX=1 tells the scripts they already run in the build container,
so pack/Containerfile can run them instead of keeping a second copy of
every compile line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In gaze mode a mouse press is held back while you aim, and its release
clicks. A release of the right button sets pressRight with the click, so
the click goes out as a right one. When the pointer goes off in the same
loop (the relay's stand_down sends the release, then "hide", as Frametop
pauses), the click is dropped, but pressRight stayed set. The next press
after resume, from the mouse, Meta+J or a pinch, then went out as a
right click.
Dropping the click now forgets pressRight with it. A "hide" read a loop
after the release still comes too late to stop the click; that needs the
helper's socket to be full as the pause starts, and is noted in the
relay's stand_down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any "btn" or "scroll" from the relay woke a pointer that was off. When
Frametop pauses for a game, the relay's stand_down sends the releases of
the clicks held into the pause, and with its pointer already off (the
idle timeout or pointer_toggle during a press) it used to send them with
no "hide" after. The helper woke on them, connected the virtual
controller and took the right hand role during the game.
Now a button release or a scroll back to 0 0 doesn't wake it. The
release is still handled as before (it ends its press, or goes to the
driver), so the driver doesn't reconnect later with the button down.
Outside a pause this changes one case: a mouse button let go after a
moving controller released the pointer mid-press. That release used to
reconnect the controller, recenter it and release there; now it updates
the disconnected controller, and ft-screens still gets its "up".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The desktop has its own XDG_CONFIG_HOME (~/.config/frametop), and SteamVR's tools and OpenVR
read their path registry from there. So from a terminal in the desktop:
- pointer/driver/install.sh (install.sh step 4, every reinstall or update from Konsole) wrote
a new registry, ~/.config/frametop/openvr/openvrpaths.vrpath, with only our driver and
"runtime": null, and never registered the driver with SteamVR. That stray file then hid
SteamVR from every OpenVR program started in the desktop.
- scripts/update-check.py (and so doctor.sh and report.sh) reported OpenVR "can't connect to
SteamVR as a background app" (VRInitError_Init_PathRegistryNotFound, or
InstallationNotFound with the stray file) and "vrcmd --overlays lists no overlays". A user's
report of 2026-10-05 showed both, with SteamVR and Frametop's services fine.
- ft-layout's vrcmd calls (the gamescope backend) found no overlays.
They now run with XDG_CONFIG_HOME=~/.config: the driver installer (install, uninstall, probe,
aimhere), update-check.py (for all its checks: nothing there reads the desktop's own config),
report.sh's vrpathreg show, and ft-layout's vrcmd. report.sh doesn't set it for the whole
report, since session/fix-panels.py --check reads the desktop's Plasma config through it.
The driver installer also removes the stray registry (only vrpathreg's, with no runtime), and
update-check.py warns about one. And vrpathreg runs `xdg-open vrmonitor://driverinstalled` to
tell SteamVR's desktop monitor, which the Frame doesn't have: in the desktop that showed "could
not read file vrmonitor://driverinstalled" on every install. A stand-in xdg-open on its PATH
takes it, so install.sh no longer filters the step's output.
Tested in a fake HOME with a copy of SteamVR's registry, a stray one, and the desktop's
XDG_CONFIG_HOME: the new installer removes the stray file, registers the driver in the copy,
and never reaches xdg-open; the old one wrote the stray file, left the copy alone, and ran
xdg-open. update-check.py from that environment: OpenVR and vrcmd ok (the old one fails both);
its stray-registry warning fires on a seeded file. ft-layout's vrcmd: 119 overlays (was 0).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper now reads SteamVR's list of panels every 20 s instead of every second, so a panel
made in between (a floating window's menu, frametop.float.N.sub.K, or the Frametop keyboard
the first time it opens) couldn't be clicked with the mouse until the next read. ft-screens
now sends "overlay <key>" to @ft_pointer_helper right after it makes one, and the helper adds
it to its list at once (only frametop.* keys). An older helper ignores it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The panel-edge test read visible[edgeKey], and when the last panel the cursor touched was
gone from the overlay list, that added it back as hidden. The map then had more entries than
there are handles, which made the 50 ms visibility poll run every frame, and since the last
commit it also counted as a visibility change each time, so unchanged frames were never
reused. The edge test now looks the key up without adding it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay sent the helper one "move" per SYN_REPORT, so a 1000 Hz mouse sent 1000 datagrams
a second to a helper whose loop runs every 8 ms, and each one went through a dozen sscanf
and strncmp tests in the helper before reaching the move handler. In a 200 ms test at
1000 Hz, 149 reports now make 45 moves with the same total.
flush() on a report now sends only once 4 ms have passed since the last move; tick() sends
the rest when due, and the select timeout shrinks to match. Buttons and the gaze
keys still flush first, unconditionally, so a click lands where the pointer was. In the
helper, "move" is now tested first in the command dispatch, and its handling is one lambda.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Handle() held the state lock through a chain of up to a dozen sscanf calls per command, and
RunFrame, which vrserver calls every frame, takes the same lock, so a burst of commands
(about 116 poses a second, plus moves and buttons) could hold up vrserver's frame. Commands
are now parsed into locals first, and the lock is held only to store the result.
RunFrame also called UpdateBooleanComponent six times and UpdateScalarComponent twice every
frame, and TrackedDevicePoseUpdated every frame even while disconnected. Components now go to
SteamVR only when they change (all of them on the first frame). The pose still goes out every
frame while the device is connected, as a tracked device's should; the disconnected pose goes
out once. The helper now sends a pose only when it changes, so the comment says the driver
keeps the last one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every frame (about 116 a second) the helper tested the cursor ray against every visible
overlay twice with ComputeOverlayIntersection, set the dot's alpha, width, transform and
visibility (five calls into SteamVR), and sent the driver a pose datagram, even with the
mouse and the head still.
Now a frame reuses the last collision result when the mouse, the anchor (1 mm) and the
eye (5 mm) haven't moved and no overlay showed, hid, or changed handle. The passes still run
at least every 100 ms, since overlays move on their own (a floating window's controls follow
it), and always while dragging. The dots' setters go to SteamVR only when their value changes:
the placement when the dot moved 0.2 mm or the eye 5 mm, which turns or resizes it by well
under 1%, and the width on a 0.5% change. The plain pose goes to the driver only when the
laser's origin moved 0.2 mm or its direction 0.04 deg (0.1 mm where it lands, 15 cm on), and
at least every 100 ms; the driver keeps the last pose and reports it every frame. A tilt's
pose, a placement, or waking sends the next one regardless.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The main loop slept a fixed 8 ms, about 116 wakeups a second, whether the pointer was awake
or not, and every second it looked up every overlay's handle and read a string property from
all 64 device slots to find its own device.
With the pointer off and hand gestures off, the loop now waits in poll() on its command
socket for up to 250 ms, or 20 ms while mapped Frame controller buttons are being read
(SteamVR input has no event to wait for). A mouse command ends the wait at once. The
headset's activity level, the game check, and the "vrgame" and "gazeawake" repeats keep
going at that pace. The 50 ms visibility poll and the 1 s handle lookups run only while the
pointer is awake, and waking forces both. The device index is looked for only while it's
unknown, and again after SteamVR activates or deactivates a device. The HMD pose history is
kept only with hand gestures on, its one user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The helper ran `vrcmd --overlays` once a second while the pointer was awake, and in gaze
mode the pointer never sleeps. Each run is a shell plus vrcmd, a new SteamVR client, about
26 to 30 ms of CPU, so about 3% of a core all the time.
The list is now read every 20 seconds, and at once (at most once a second) when it may have
changed: the pointer waking, the dashboard opening or closing or creating an overlay, the
scene app changing, an "overlays" request, and a left click that hit nothing, which may be
on a panel that came up since. The thread waits on a condition variable instead of waking
every 100 ms, so it sleeps while paused. The main loop looks the keys up again as soon as a
new list is in, rather than at its next 1 s tick. Overlays already on the list still show
and hide within 50 ms, from the IsOverlayVisible poll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gaze service ran ft-gaze and our own eye tracker all the time: with gaze mode off, ft-eyes
still took about 60% of a core, and ft-eyegrab, ft-gaze and ft-gazed 3 to 4% each. Now ft-gaze
and our tracker run only while gaze mode is on and someone wears the headset, while a check or
the calibration is open or asked for, or under a "wake" lease, which the Gaze page of Frametop
Input Settings renews while it's open. 30 s after the last use they stop, and the frame grabber
idles with our tracker.
- The pointer helper answers "gaze ? headset" with worn|away (SteamVR's activity level for the
headset); an older helper answers it as before, and the service then goes by gaze mode alone.
- A quick check, calibration, or fit check asked for while idle wakes the tracker and opens once
it sends; the automatic calibration waits quietly while it starts.
- Status has "awake" and "idle" (why), and the Gaze page shows it.
- gaze/test/idle-test.py runs the service with a fake helper and ft-gaze, offline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frametop kept using the headset during games: with gaze mode off, our eye tracker still took
about 60% of a core, remote desktop about 2 cores while on, and KWin kept drawing hidden
screens because ft-screens sent their frame callbacks at 90 Hz. Pausing gives that back, and
resuming brings back only what pausing stopped. It's also a way to keep the gaze service and
our eye tracker off during games, which PR #13 asked for.
Paused (input/game_pause.py, run by the input relay):
- frametop-gaze stops (ft-eyegrab then idles by itself), and hand tracking and remote desktop
stop if they run
- the desktop hides and slows down: ft-screens "pause on" hides every panel and sends KWin a
frame callback once a second; or, with pause_desktop "close", the desktop closes and starts
again on resume
- the relay lets go of the 3D mouse, typing goes to Steam, and mapped buttons and key
combinations do only pause_toggle, steam_menu and commands
Toggled by both thumbsticks clicked together twice (configurable), read passively from
vrserver's web socket (input/vrws.py) so it works in games and takes nothing from them; by the
new pause_toggle action; by input/ft-pause; and, with pause_auto (default on), by a VR game
starting and ending, which the pointer helper now reports ("vrgame 1|0"). Frametop Input
Settings has a Games page for it. update-check.py checks the web socket, and doesn't count a
paused gaze service as failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- frame_sudo (scripts/_env.sh) replaces three copies of sudo_run. On
the Frame it asks in the terminal even when stdin isn't one, which
install.sh's steps never had: saying yes to the Bluetooth fixes or
hand tracking stopped the install with "no terminal for sudo".
From a PC it asks through ssh -t when .env has no password.
- start_with_steamvr: the pointer, power, and gaze services restart
when SteamVR runs (a re-install runs the new code) and are left to
start with it when it doesn't, instead of failing the install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- gaze/tracker/lab/eyes_track.py: nothing used it
- Input Settings: the pointer role backend, whose page was removed
- ft-screens: no log line for every floating-window resize
- hands: ft-hands --help gives the palm-down default (1: off), the
uninstall removes ft-handsctl's link, .frame-job is ignored
- Display Settings: "Save as profile…", not "Save current arrangement"
- two stale comments (ft-pointer's grabprobe, ft-gaze)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drag begun by right during the left's held-back press (or Meta+K during
Meta+J's) now lasts while either button or key is held, so pressing the right
one again is free: it tilts, as a right press does during any mouse drag. Meta+K
during a keyboard drag (held still into one, or the second Meta+K) tilts while
held: the head turns the panel, and the mouse can too; let go and the head drags
again from there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The right button's press is held back like the left's: hold it, the pointer
stops where you look, move onto the target, and the right click comes on the
release (held still, it's a real right press). Right during the left's held-back
press now starts a drag where the pointer is, as Meta+J then Meta+K does, in
place of a right click there; letting go of either drops it. So once you've
moved, the left alone only clicks, and the right starts a drag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POINTER_GAZE_MOUSE_MOVE=held, the default (the Gaze page's Mouse movement
switch, with why it's on): while the gaze has the pointer, moving the mouse
does nothing; it moves the pointer only while a button is held, as a
correction. A bumped or drifting mouse can't pull the pointer off what you're
looking at, and every mouse move is a correction, so lessons aren't polluted by
mouse moves to somewhere else. With the gaze stale for a second, in a game, or
with the headset off, the mouse moves the pointer as usual. "free" is the old
behaviour.
Pressing the right button while the left one's press is held back right-clicks
where the pointer is instead (correct with the left, then right-click); both
releases are then nothing. Meta+J then Meta+K stays a drag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the quick check's second step (3629681): the user didn't want it, and it
re-entered itself every tick, re-placing and redrawing the panel, which
flickered in the headset.
POINTER_GAZE_NUDGE_MAX is the one learning limit, now 55 degrees by default:
half of the 109 the Frame shows across. The helper measures the correction
itself (raw gaze to click), not the mouse's path; past the limit it isn't
learned and the helper sends "recheck" for the quick check, for the mouse and
keyboard alike. ft-gazed's own limits (8 and 25) follow the setting.
Test, our tracker's 409 clicks since its Sep 29 calibration: a one-dot check
set from any one of them puts the next 2 minutes' clicks within 15 degrees (99%
within 4.2) and the next 10 minutes' within 25, so it gets well under 55.
The panel's dot is still and its capture ring fills in quarters, so it's drawn
again only a few times per dot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the quick check's capture, in gaze mode, the dot stays where it is in the
room (the panel's new "lock") and the gaze has the pointer again. You put the
pointer on the dot as you'd correct a click, with the mouse or Meta+J and the
head. That click clicks nothing: the helper sends it as "calverify", learned
whatever its size. Over POINTER_GAZE_NUDGE_MAX, the capture runs again, up to
3 times. A right click or Meta+K skips it.
POINTER_GAZE_NUDGE_MAX is now the one limit: 15 degrees by default (was 8),
and ft-gazed's own limits (8 for SteamVR's tracker, 25 for ours) follow it. A
keyboard correction past it isn't learned but opens a quick check ("recheck"),
in place of the 30 degree keyboard limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Meta+J hold's correction is always meant, but it went through the mouse's
8 degree nudge limit. Live, our tracker was 12 degrees off and every correction
was dropped without a word. Debug output now says when one is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-gazepanel (gaze/panel) is a SteamVR overlay that stays put in your view and
shows dots at known head-relative directions; the gaze service runs it and
drives it (gaze/gazecheck.py):
- a one-dot quick check 3 s after the headset goes on, when our tracker asks
for a click (reseat), at most every 2 minutes, and from Quick check on the
Gaze page; five dots follow if the next 3 lessons are still over 2 degrees off
- the full calibration (the probe's three rounds, dark to bright) when gaze mode
comes on without one, or from Calibrate; quitting it with still no calibration
turns gaze mode off
Each dot takes the gaze once it has held still for 0.6 s; a left click or Meta+J
takes it at once, a right click or Meta+K closes the panel (the pointer helper
hides its dot and passes those presses on while "calpanel" lasts). Our tracker
gets clicks and its own calibration; SteamVR's gets lessons per eye, or a new
calibration.json.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gaze_left and gaze_right (Meta+J and Meta+K by default) click where you look.
A tap clicks where the dot was at the press and tells the gaze tracker it was
right there. Held, the pointer stays put in your view, so turning your head
carries it onto the target; the release clicks there and the correction is a
lesson (judged by the net correction, not the head's path). Held still for
POINTER_GAZE_HOLD it's a real press that the head drags, and Meta+K while
Meta+J aims presses where the dot is now, to correct and then drag.
A Meta combination now also sends the desktop an F24 press and Meta's release
at once: Meta's release no longer opens Plasma's launcher, and the click isn't
Meta+click (KWin's window move and resize, which swallowed right clicks). The
desktop gets Meta back for the next key if it's still held.
Also fixes e16b758, which had taken the gaze actions off the key combination
list along with the controllers'; and gaze_quickcal (the gaze service's
one-dot check) joins the actions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Steam reads the Frame controllers itself, outside SteamVR's bindings, and every
press and release it sees takes SteamVR out of laser mode, so controller clicks
at the gaze can't be done cleanly (docs/gaze-controllers.md, from the gaze-first
branch's tests).
- Gaze precision and Gaze drag take a mouse button or a key combination only;
the controller source, its aim steering, and POINTER_PRECISION_GAIN,
POINTER_PRECISION_DEADZONE and POINTER_GAZE_DRAG_GAIN are gone.
- The gaze actions (including Gaze pointer on/off) can't be mapped to controller
buttons: the relay ignores them there and doesn't ask the helper for those
buttons, and Input Settings no longer offers them.
- Last used wins in gaze mode too: picking up a controller hands it the laser,
as docs/design.md already said.
- The gaze dot shows all the time (POINTER_GAZE_DOT=always, the default;
moving brings back the old behaviour), with a switch on the Gaze page.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local RDP login between krdp and the VNC bridge uses the account's
own name instead of "steamos", its certificate the hostname instead of
"steam-frame", and the ft_pointer driver installs under the user's home
instead of /home/steamos. The remote-access address already came from
the tailnet (tailscale0 and tailscaled's local API).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two new actions for mouse buttons, controller buttons, and key
combinations: gaze_precision (hold: the pointer stops where you look and
the button's device steers it, a controller by where it points at
POINTER_PRECISION_GAIN, the mouse by its moves; release: click there) and
gaze_drag (the same with a real press at once, dragging until the
release). The relay sends "precision|gazedrag <source> 1|0" to the
helper, which runs them through the same holds as pinches and grips.
- Key combinations on any keyboard ("key_bindings" in the rules, e.g.
Ctrl+Alt+G for gaze on/off): the last key isn't typed.
- POINTER_GAZE_MOUSE: in gaze mode the left button is a precision button
(the default, as before) or clicks right away (direct).
- In gaze mode a moving controller no longer takes the pointer away.
- POINTER_ROLE (right, left, stylus): the driver takes a "role" command,
so the pointer can stay off the hand holding the precision controller,
which would otherwise take the role back. Needs the rebuilt driver.
- Input Settings: the actions on the Controllers and Buttons pages; on the
Gaze page the mouse choice, the role, the precision sliders, and key
combinations (captured from any keyboard).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hand tracking no longer starts with SteamVR: hands/run.sh install leaves
the units disabled and links hands/ft-handsctl into ~/.local/bin, which
turns it on and off (on | off | status | log | cutouts on|off | gestures).
With it on, hands show through the screens; pinches and grips move the
pointer only with POINTER_HANDS=1, now off by default. ft-camd's service
runs the mono cameras only: while the headset is worn, the colour module
writes just a half-size image into the top-left quarter of its buffers,
which ft-camd can't use yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The palm-down filter is off by default: the user's deliberate pinches,
hand raised, read 0.90-0.99, like typing.
- Typing is told apart by the keyboard instead: the input relay sends the
pointer helper "typing" on key presses, and it takes no pinch within
POINTER_PINCH_TYPING (1 s) of one.
- Grips are still held to hands raised (POINTER_GRIP_BELOW, 0.35 m below
the eyes); pinches aren't, since the user's own sat 0.35-0.45 m below,
elbow resting.
- A grip doesn't begin with the thumb on the index tip: that's a pinch
with the other fingers curled, which was taken for a grip.
- A pinch's point is the index and middle knuckles: the tips' midpoint
moved 1-2 cm as the pinch opened, dragging every release off its press.
- ft-hands --gesture-log prints what the detectors measure, 10 times a
second.
Pinches still aren't reliable enough to use; hand tracking is parked for
now in favour of the controllers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pressed when the pinch closes, released when it opens, its hand dragging
the pointer in between (at the pinch gain, past the dead zone), like the
mouse's button. That's what the gaze probe's Click practice needs: it
freezes its own gaze dot at the press and drags it by the pointer's
movement until the release. With gaze mode on, the pinch still holds the
press back and clicks on the release.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pointer helper reads ft-hands' gestures. A pinch holds the pointer
where the gaze put it and clicks on the release; held, the hand nudges
the pointer at half its angle past a 1.5 degree dead zone, which also
teaches the gaze tracker. A grip presses where the pointer is, drags with
the hand, and releases when the hand opens, unless it began more than
30 cm below the eyes (hands on a desk). Hand movement is taken in the
room with the head pose at capture time. Hand use keeps the pointer from
the relay's idle release, as gaze mode does. POINTER_HANDS and friends in
frametop.conf.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Floating windows join the keyboard and the hand cutouts. Floating panels
don't get cutouts yet: their panel and popups show crops of the client
buffer (texture bounds), which the side-by-side cutout buffer doesn't
match. KWin gets both the spare outputs and our input method, and the
pointer helper's frametop. prefix already covers the float panels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes#7: SteamVR's keyboard never came up for the desktop's apps, and
opening it for our panels doesn't work well on the Frame (it's Steam's own
panel, mounted in the dashboard's scene, it follows the laser between
panels, and it takes the controllers over to SteamVR's laser).
- KWin starts input/ft-textinput as the desktop's input method. It tells
the input relay when a text field gains or loses focus, and the relay
asks ft-screens to open or close the keyboard. The session drops the
QT_IM_MODULE=xim and GTK_IM_MODULE=xim that the gamescope session sets,
or Qt and GTK apps never report text fields.
- The keyboard is ft-screens' own panel (screens/keyboard.cpp): a US laptop
layout, typed with a controller's laser or the 3D mouse. It opens 0.7 m
in front of you, below your eyes and facing you. It has a grab bar to
move it, a Close key, latching Shift, Ctrl and Alt, and repeat on a held
key. It's drawn into shared DMA-BUFs, so it doesn't flicker. Its keys
reach the focused screen as key presses, so every app takes them.
- It steps aside while the Steam menu or Steam's own keyboard is up and
comes back after. A layout reset closes it, and it doesn't open without
a head pose.
- Frametop Input Settings has a Keyboard page: open it for every text
field, only while no keyboard is connected (the default), only from a
mapped button (the new Open/close keyboard action, for mice and
controllers), or never. A switch keeps it open until you close it.
- The pointer helper treats every frametop.* overlay as a real panel. The
keyboard's shared texture reports 0x0 like SteamVR's scene-graph
controls, and the helper had given it their wide catch radius.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
The pointer and gaze services need steamvr.service to be running (Requisite=), and ft-pointer, ft-screens, and ft-gaze connect as a background app before switching to overlay, so they never start a vrserver of their own. One started from the dev container never finds the headset, which left a reboot stuck in a loop.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 51b4e79318)
Full screen fills the window's own panel (the margin drops to zero).
Meta+scroll over a floating window changes its scale at the same size in
pixels. Spare outputs are sized to a multiple of KWin's buffer scale, and
screens to even sizes: an odd buffer at a fractional scale is a protocol
error that disconnected KWin. The 3D mouse's drag lock now crosses onto
other Frametop panels unless the pressed one is being carried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-screens makes a panel for each spare output (frametop.float.N), hidden
until ft-floatd floats a window on it. The panel shows only the window's
rectangle of the buffer at the density of the screen it came from, its
popups and dialogs get small panels over it, pressing its title bar
carries it while KWin's pointer stays put, the corner tab resizes the
window in pixels, and two more buttons close it and put it back on the
desktop. The session adds FLOAT_SLOTS spare outputs to KWin and starts
ft-floatd; ft-layout arranges only the screens' outputs, and the pointer
helper treats the new panels like screens. Not yet tried in the headset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While a button is held on a screen, the laser leaving it no longer takes
KWin's pointer, and an invisible catcher overlay sits on the laser whenever
it's off every panel, so the release reaches KWin at the pointer's last
spot. The pointer helper also reports the mouse's left release as a
backstop. Not yet tested in the headset.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
