host/macos/frametop-host-setup.sh makes a Mac a host: the pinned Sunshine
pre-release (v2026.1010.141104, the first with the mouse fix for displays
other than the main one), and one instance per real display on ports
47989 + 100*N, each started at login by a LaunchAgent. An instance follows
its monitor (vendor, model, serial), looks its display id up at every start,
and waits while the monitor is unplugged. One certificate and one Web UI
login for all of them. No virtual displays on the Mac.
On the Frame: a display can have a port, and ft-stream takes HOST:PORT.
Plain Sunshine has no tokens, so Remote Displays checks the Web UI login at
sign-in, keeps it in memory until the picked displays are paired, and
passes it on stdin (ft-layout remote add --port --login, ft-stream pair
--login), where ft-stream approves its own pairing request by id. The
Mac's instances show up as one computer.
Tested against the M3 Max with throwaway instances: pairing, remote add,
three streams at once, sound, and the pointer on a non-main display. Not
yet tested in the headset or the live desktop.
Frametop's Vibepollo build now switches a streamed monitor to the stream's
resolution while it streams, and back afterwards, instead of scaling the
monitor's own resolution into the stream. Nothing changes on the Frame:
ft-stream already asks for the size set in Remote Displays. The note under a
monitor's stream settings, reference.md and remote-displays.md say so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
voice/ft-voiced (was frame_voice.py) runs in Frametop's container as
frametop-voice.service. It keeps the mic, the wake word and live text,
and hands the keyboard side to the input relay:
- push-to-talk keys are key combinations (dictate, dictate_hold); the
relay sends "toggle|start|stop" to @frametop_voice, taken only from
this user
- text goes to the relay as "type TEXT" (ASCII, in pieces of 2000
bytes), so it lands where typing goes; the relay holds it back while
a modifier is held
- controller buttons are read from vrserver's web socket with
input/vrws.py, in a thread, so the websockets package goes
No own uinput keyboard, fd store, evdev, or controller service any more.
whisper-server runs only while the wake word is on, started and stopped
by ft-voiced. Voice commands and hooks run on the host through
distrobox-host-exec (env passed through env(1)); their FRAME_VOICE_*
variables keep frame-voice's names.
Settings: ~/.config/frametop-voice.json (voice_settings.py, standard
library), wake word off by default, "advanced" for the old config.toml
tunables. voice/run.sh install carries frame-voice's settings over once
(its keys become key combinations) and turns the standalone units off.
Install: install.sh step 8/11 asks (yes by default; --no-voice through
get.sh, install-release.sh and a FrameDrop checkbox). voice/build.sh
builds whisper-server (whisper.cpp d09f61a, Vulkan, static) and
build/venv; voice/models.sh fetches the 322 MB of models into
~/.local/share/frametop/voice by fixed URL and sha256. The image gets
glslc, spirv-headers-devel and pipewire-utils, uv.lock a voice group (sherpa-onnx with its
core pinned: uv dropped it), CI smoke whisper-server and the venv.
uninstall.sh asks about the models; update-check and report.sh know the
service (report.sh leaves out the lines with what was said).
Tests: voice/test (settings, carrying over, typing through a stand-in
relay, push-to-talk, controller buttons from a stand-in web socket) in
just test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For Frametop Voice (voice/, next commits), the input relay does the
keyboard side of dictation:
- "type TEXT" on its control socket types TEXT where typing goes: to
ft-screens for the desktop, else on the virtual keyboard SteamVR
passes to Steam and games. US layout, one key every 4 ms, and a
character waits while a modifier is held. Only from the relay's own
user: the socket now reads each sender's credentials (SO_PASSCRED).
- New actions dictate (a press starts or stops) and dictate_hold
(records while held) send "toggle|start|stop <source>" to
@frametop_voice. They work without pointer mode and while paused.
- A modifier tap bound by its right-hand code ("97") is that key's
alone and comes before the folded one, so dictation's default, a
Right Ctrl tap, doesn't fire on Left Ctrl (crouch in many games).
It's in the defaults once frametop-voice.service is installed.
SHARE_KEYS and @frametop_keys go: they existed only so frame-voice
could see keyboards the relay grabs, and sent every key typed into the
desktop to a socket any local process could bind. conf-migrate.sh drops
the example's line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Display Settings read the log's last line, so an error logged after
"forwarding" (a failed Notify) showed "In the desktop" while forwarding.
It takes the last state line now.
- "on" also forwards to SteamOS's steam_notif_daemon, where notifications are
lost while Plasma's popups are off. The settings app shows it only while
frametop.conf has it, with that warning.
- update-check reads libnotificationmanager.so.1 (its soname) first; the
unversioned .so is a dev symlink.
- justfile: why session/test/test_notifyfwd.py isn't in test-python.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ideas from StereoKit (MIT, 07b9f0d), credited where a formula comes from it:
- The torso faces the head's forward counted twice plus the direction to
each wrist seen (POINTER_RAY_TORSO_WRISTS, on), so looking aside while
pointing swings the shoulders, and the ray, less. probe-moves replay: the
shoulder's share of a still hand's ray motion 20 -> 18%, p90 2.0-3.7 ->
1.7-3.6 deg/s; the tap capture: nothing worse.
- The ray can turn toward the hand's own direction, from the palm's axes
(wrist, middle, ring and little knuckles) tilted -30/8 deg
(POINTER_RAY_HAND_WEIGHT, 0). Off: every weight measured worse (at 0.3:
judder 0.07-0.14 -> 0.11-0.33 deg rms per 90 Hz tick, raw jumps over
3 deg 159 -> 330). The gate still tests the shoulder's ray.
- The knuckle point can be pinned to the palm's axes during a pinch, easing
back on the steady shoulder's schedule (POINTER_RAY_ANCHOR, 0). Off: our
pinch point is the knuckles already, and pinned it shook as much or more
and kicked no less on release, fitted or not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-hands keeps the last 12 s of hand crops (each view's landmark crop, 1.6x, at most
224 px), its hands, gestures and landmarks in RAM (track/flagbuf.h, HANDS_FLAG_BUFFER=1).
`ft-handsctl flag [note]` asks for them through run_dir/flag; ft-hands writes them and the
next 2 s to HANDS_FLAG_DIR/flag-<time> from a low-priority thread, for frame-hands'
offline tools. ft-handsctl also saves a /dev/video99 frame and ft-handview's head poses,
and the drawn hands pulse cyan when a flag is taken (ft-handview's new flash and poses
commands). ft-handreplay --flag-at/--flag-every makes the same flags from recordings.
Measured on a replayed ring: +58 MB peak RSS, 0.1 ms a crop on an A720 (about 1% of a
core). Unit tests: tests/flagbuf_test.cpp (make check), tests/test_handsflag.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stylized robot hand made in code, no asset: rigid pieces, each fixed to
one bone, so nothing bends or stretches out of shape. A light shell per
finger bone (a tapered box with rounded edges and ends), dark balls at the
knuckles and the thumb's base, dark hinges at the other joints, a palm of a
dark core between a palm plate and a back plate with panel lines, and a
short closed cuff past the wrist. Cyan accents glow: fingertip pads, a line
across the knuckles, a dot in the palm, a ring round the cuff. solid is
ceramic white over dark grey metal; ghost is pale and see-through.
screens/handrobot.cpp builds one mesh (2373 vertices, 4360 triangles) once,
a right hand on DemoPose's open hand, in 16 segments' frames. The frames are
handskin's rule (new handskin::TrackedFrames); a left hand's are a right
one's mirrored, so the same mesh draws it mirrored. Each vertex is fixed to
its bone's start or end: shells keep their rounded ends, the straight middle
takes up the tracked length, fingertips end at the tracked tips. The vertex
shader applies one segment's matrix per vertex; no cap pass (it's closed).
Cost (--snapshot, frame-job 4 cores, same run): robot 0.21-0.24 ms CPU and
0.55-0.57 ms GPU wait per draw, against asset 0.28/0.66, babylon 0.26/0.66,
mrtk3 0.25/0.66.
Tests: segments start at their joints along their bones, no vertex past its
bone's length, fingertips at the tracked tips, left = right mirrored vertex
for vertex, rigid segments across poses, only the middle stretches with a
longer bone, accents on the right sides.
No free robot hand asset fit; screens/assets/hands/README.md says what was
looked at.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The look: no more drawn outline. The old rim pass lit a hard line wherever the
hand met the background or a part of it further back, so a finger over the palm
got scratchy white lines. Now the body pass keeps only the surface nearest the
eye (depth) and writes how squarely it faces the eye, the fade, a light from
above and how near a fingertip; the second pass turns that into a soft fresnel
rim, a slight fingertip glow and an antialiased outline. ghost glows white
toward its edges; solid is lit from above with a soft cool rim.
The wrist: the model is cut open where the fade ends, so a hand pointing away
showed its inside (a hollow tube, a fist's fingertips through the palm). A
stencil pass counts front faces up and back faces down along each view (on its
own, with the depth test off: early depth would drop hidden ones); where a view
went in through the cut, a disc where the fade begins covers it.
The models: HANDS_VIEW_MODEL=asset (generic-hand, still the default) | babylon
(Babylon.js, CC BY 4.0) | mrtk3 (BSD-3-Clause, right only, mirrored) | godot
(Godot XR Tools, CC0) | mrtk2 (MIT) | capsules. Each is in
screens/assets/hands/<name>/ with its licence and a README giving the source
commit, hashes and conversion (the MRTK ones were FBX: assimp, then the new
screens/tools/hand-glb-clean.py). hand-asset.py maps each skeleton onto WebXR's
25 joints by name and writes them all into handasset.h; handskin takes the
model's index.
Cost (--snapshot, asset): CPU 0.34 ms per draw (was 0.32-0.33), GPU wait
0.61-0.62 ms per draw (was 0.57). --snapshot now prints the GPU wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rendered hands are now the WebXR Input Profiles' generic hand (MIT),
posed by the tracked skeleton and skinned on the GPU: per joint a frame
from the bone and the palm, scaled to the hand and each bone's length.
Two looks, HANDS_VIEW_STYLE=ghost|solid; HANDS_VIEW_MODEL=capsules keeps
the tube hand. With HANDS_VIEW_ROOMVIEW=1 (the default) the hands also
float over the cutout with Room View on. --snapshot draws demo poses
into a PNG without SteamVR, and the cost (0.4 ms a draw, the tubes 0.6).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A near grab carried the pinch point moved ahead along the hand's velocity, which is mostly
noise times the lead, so the panel shook. It now carries the pinch point as the cameras saw
it, smoothed (fh_smooth.h, read every frame ft-screens draws), and moves nothing until the
point is 1 cm from where the pinch began, then carries on from there without a jump. The
head the panel turns to face is smoothed too. On the probe-moves replay the carried point's
second difference per 90 Hz tick went from 13-17 mm rms to 2.3-2.6 mm; it now trails the
capture by 85 ms while moving.
Settings: HANDS_GRAB_SMOOTH_MINCUTOFF, HANDS_GRAB_SMOOTH_BETA, HANDS_GRAB_DEADZONE.
handinput_test --replay reports the grab input's smoothness.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With gaze mode and POINTER_HANDS on, the gaze is the pointer and the hand ray
stays off (POINTER_GAZE_HAND_RAY=1 brings it back). A pinch clicks where you
looked 75 ms before ft-hands saw it; held and moved past 4 deg of hand
movement (POINTER_GAZE_PINCH_DEADZONE) it nudges the pointer at 0.3x
(POINTER_GAZE_PINCH_GAIN), and the correction is a gaze lesson. Held still for
POINTER_GAZE_HOLD it's a press the hand drags. A middle pinch right-clicks at
the gaze. One log line per pinch. Rules in pointer/helper/gazehand.h, tested
in pointer/tests/gazehand-test.cpp.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Until the tracking is good enough for a hand-shaped hole, each hand cuts a
disk facing the eye, wide enough for the whole hand plus the margin, drawn
as the stadium around the ellipse each eye sees on the screen with a soft
edge. mesh and capsules stay selectable (HANDS_MATTE, cutouts matte).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
screens/roomview.h tells whether Room View is on: forced by HANDS_ROOMVIEW, else
off when the passthrough shared state says disabled, else the last
RoomViewShown/Hidden event in a game, else off in a game and the construct's
setting outside one. `cutouts state` says which; `cutouts roomview auto|on|off`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live test 2026-10-09: taps took several tries and the laser jumped between
hands. The owner's ray dropping for a moment (closing the fingers turns the
hand) handed the laser to the other raised hand at once, for good; a pinch
by the other hand pressed where the old owner's ray pointed; a pinch whose
ray had just dropped became a press at a stale spot, or only woke the
pointer.
- The owner keeps the laser through POINTER_RAY_GRACE (0.5 s) of its ray
being down; the other hand takes it by pinching, by raising its ray while
the owner's is down, or after the grace (pointer/helper/rayowner.h).
- A pinch on a ray that dropped within the grace, or is on its way up,
brings it back (fh_ray.h ShowForPinch) and is a ray pinch.
- Aim lock: the press goes along the ray from just before the fingers
began to close, after a pose along it has gone out; a tap that ends
before its press clicks there too.
- Each ray follows one tracked hand through a brief flip of ft-hands'
left/right call.
- Log: owner changes with the reason, the owner's ray down and back, and
one line per pinch saying what it clicked or why not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hands/handsmode.py starts and stops ft-camd, ft-hands and the hand view
(frametop-handview.service, when installed) by HANDS_MODE: frametop =
while the desktop runs (session hooks), always = wanted by
steamvr.service (and kept running through game pauses), off = the
default, only ft-handsctl on as before. ft-handsctl mode sets it;
missing ft-camd capabilities start nothing and say what to run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Turn a bare hand palm up toward your face and look at it: after 0.3 s a
small ring shows above the palm and follows it, like the Vision Pro's
home indicator. A thumb-index pinch tap by that hand runs
`ft-steam menu` (opens the Steam menu, or closes the dashboard); the
ring is blue while the pinch is held. ft-screens owns it
(screens/palmmenu.cpp) and reports that hand near while the ring shows,
so the pointer helper doesn't also gaze-click the pinch. The helper is
unchanged. HANDS_PALM_MENU=0 turns it off; `palmmenu on|off|state`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The desktop is a KDE session, so it ran /etc/xdg/autostart/steam.desktop.
Steam is already running (the desktop starts from it), so `steam -silent`
only reached that client as a command line it ran (ExecCommandLine in its
console log). SteamOS 0.4 adds -vrdisable -deckard to that entry, for
Desktop Mode, where Plasma starts Steam itself. The session now hides it
with the other two. The marker goes to 2, so desktops that hid the first
two get Steam's copy once, and an entry someone brought back stays.
The Steam client's XCURSOR_THEME=steam reaches the desktop through its
environment, and KWin and the apps prefer it to the desktop's setting.
There was no such theme, so they fell back to Breeze; SteamOS 0.4's
holo-cursors adds one. The session sets the desktop's own theme (Breeze
unless changed in its System Settings) and leaves the size as it was.
ft-screens doesn't draw KWin's cursor, so this shows over remote access
and with the gamescope backend.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Joins with hands-screens: ft-hands keeps 30 Hz for a middle pinch as well
as for a want-hz reader; GesturePublisher keeps both use() (replay dumps)
and write()'s middle-pinch slots; both test groups run in test-c.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The driver's default role is the treadmill (4), hinted from Activate;
the helper's POINTER_ROLE defaults to treadmill. Needs the driver
reinstalled and a SteamVR restart to take effect.
- The "no hand role" release skips a device whose hint is the treadmill.
- Hand gestures: none begins on a side whose controller is in use, or by a
hand at a controller (by position); a held one ends, as lost, when its
side turns controller.
- The moving-controller release no longer applies while a bare hand's ray
drives the pointer (that ray goes only when its own side turns controller).
- update-check.py: the virtual controller's path, and the touch components.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ft-layout reset reopens the profile in use (the custom arrangement with an
active profile) with use NAME, so its screens, hidden screens, remote
displays and apps all go back as the profile has them. Without one it runs
apply, as before. Meta+Shift+R and the Reset Screen Layout entry
(ft-layout-reset), the input relay's layout_reset action (a mapped mouse or
controller button), and the reset button on a screen's bar now run it.
Display Settings' Arrange now still only arranges.
Before, every quick reset ran apply, which put the desktop screens back but
left the profile's hidden screens, remote displays and apps alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
install.sh's desktop step builds ft-stream and installs Remote Displays' menu entry.
uninstall.sh removes the entry and offers to delete the remote displays' pairings and
sign-ins with the other settings. docs/reference.md describes the app and the PC setup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reference.md's ft-screens keys paragraph, the relay's top docstring and
hazards.md's key releases section now say that a mouse button passed
through as a key in pointer mode goes to the screen the pointer is on,
not where typing goes, only while one shows and Frametop isn't paused,
that ft-screens lets its release through and releases it itself when
the pointer leaves the screens, its screen hides, or Frametop pauses,
and that a relay that starts releases mouse buttons too. hazards.md
adds that the button still reaches the virtual mouse, which gamescope
reads, and that a keyboard's media keys come from its Consumer Control
node, which stays ungrabbed, so they reach gamescope as well as the
desktop even while typing goes to the desktop. design.md says why the
buttons follow the pointer: as keys, a release was lost whenever typing
moved, and wlroots counts presses per button. reference.md names
scripts/test-relay-buttons.sh, and keys-test's wider scope.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frametop.conf.example said HANDS_SWAP_SIDES=0, and desktops.sh copies it on a
fresh install, so every new install forced ft-camd's side camera names and
turned the hand tracker's own side check off. Some SteamVR restarts swap those
names, and then hands land beside their cutouts and recordings are mislabelled.
The example now says auto. scripts/conf-migrate.sh, run by install.sh and
hands/rec/install.sh, replaces the old line only where it's still exactly as
the example wrote it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pause section of reference.md now says the relay releases a click
still held as it lets go of the 3D mouse, and lists
input/test/pause-buttons-test.py with the other pause test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two installers give ft-camd its capabilities: hands/run.sh install and the
Hand Recorder's (through hands/run.sh caps). #30 made hands/run.sh uninstall
take them back unconditionally, before it removed the units. With the
recorder installed too, its next session then stopped at "ft-camd needs its
capabilities", and so did ft-cutouts. The recorder's own uninstall, the one
hand-recorder.md points to, still left them on the binary.
One rule now: hands/run.sh uncaps takes them back only while neither the
Hand Recorder's menu entry nor the frametop-camd unit is installed, and says
which one keeps them otherwise. It checks through on_frame, in one call, so
it works from a PC too. hands/run.sh uninstall runs it after removing the
units, and hands/rec/install.sh uninstall after removing the entry, so each
stops counting itself. uninstall.sh removes both and always takes them back.
The docs say which commands ask for the password, and AGENTS.md says
uninstall.sh also has to undo file capabilities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After 9c66475 and the previous commit, ft-layout's log is
$XDG_RUNTIME_DIR/frametop-layout.log, but report.sh still collected
/tmp/frametop-layout.log, so bug reports lost it, and reference.md still
named the /tmp path.
report.sh reads the new path (it already sets XDG_RUNTIME_DIR to
/run/user/<uid>). reference.md and the comment on RunLayout name the host's
runtime directory, which ft-screens and the desktop start share. The nested
desktop has its own, /run/user/<uid>/frametop, removed at every desktop start
and stop, and the log isn't there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts with experimental's pause_toggle, steam_menu and command: actions and its
AnnounceOverlay: both kept. The spin bindings join the Meta tap in the defaults, spinning
doesn't need pointer mode, and like other actions it does nothing while Frametop is paused.
AnnounceOverlay now sends through the PR's SendPointer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SteamVR's own floating windows take the laser while a controller points
at them in a game and give it back when it points away. Frametop's
panels didn't: with the controllers left to the game (outside_games, the
default, or dashboard), they couldn't be clicked without the dashboard.
ft-screens now sets MakeOverlaysInteractiveIfVisible on a screen or
floating window while a hand controller's laser pose meets it, its
controls, or its popups (UpdateAim; curved screens hit on their
cylinder), and clears it 0.3 s after the aim leaves a wider margin. A
drag or a held button keeps it on. The keyboard, one overlay, uses
ComputeOverlayIntersection and now follows the mode when a game starts
or ends while it's open. This replaces the reset button's own aim zone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each desktop screen gets a reset button left of its bar (a reticle). It
puts every screen back in its layout around where you are now, like
Meta+Shift+R (ft-layout apply).
In a VR game the screens leave the controllers to the game (the
outside_games and dashboard modes), so a controller couldn't click any
of their controls. Aiming a hand controller at the reset button now sets
MakeOverlaysInteractiveIfVisible on that button's overlay alone, so the
trigger clicks it; the flag clears half a second after the aim leaves a
zone twice as wide, and the game gets the controllers back. The aim
comes from the laser poses ft-screens already reads to show the controls.
The ft-layout spawn is now RunLayout(cmd), shared with the arrange.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ft-screens "spin next|prev|<degrees>": every unpinned screen and
floating window turns together about a vertical axis through your
head (0.3 s, eased), so the next panel on the right or left comes to
straight ahead; the arrangement stays as it is. Taps during a spin
add to it, from where the panels are headed; grabbing a panel or
placing it (ft-layout, ft-floatd) takes it out of the spin
- when a spin settles, the panel in front gets the pointer (recenter),
typing (as after a click), and KWin's active window: its floating
window, or the top window on a screen (ft-floatd "front N", the KWin
script's activate-output). KWin's outputs follow the screens'
new places (ft-layout scale), as after a move
- the input relay: spin_next and spin_prev actions, Meta+Alt+Tab and
Meta+Alt+Shift+Tab by default; Frametop Input Settings lists them.
Not Meta+Tab: that's Cmd+Tab on a Mac reached through a remote
desktop like RustDesk, and the relay would take the Mac's app
switcher. Meta+Alt+Tab (Cmd+Option+Tab) is unused on macOS,
Windows, and KDE
Used on the Frame (SteamOS 0.3.0 build 20260922) with one screen and
three or four floating windows, through RustDesk to a Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DeeJanuz