e3fcd3eef8 Use gamescope input for Frame trackpad and keyboard (#66)
* Live view: a Desktop view that stays still, and Control to tap on the Frame

The live view gets a second source and a way to use the Frame from it:

- Desktop: the app panel in use in the headset, streamed from its own window
  (x11grab of gamescope's redirected window), so it doesn't move as the
  wearer looks around. A picker shows any other panel, view only.
- Control: on the Desktop view a tap or click lands exactly where you put it;
  drag is a mouse drag, press and hold right-clicks, two fingers scroll, and
  on a computer the mouse, wheel and keyboard work directly. On the headset
  view the view is a trackpad. A text field and key row type from a phone.

Input goes through gamescope's own EIS socket (the way Steam feeds Remote
Play input) with the libei already on the image: ui/frame_touch.py, over
the same long-lived ssh machinery as the keyboard agent, nothing to
install. It reaches the panel that has focus on either X display, which
the KDE Connect route can't. Verified on the Frame and from the iPhone app
in the Simulator.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from review

- Keys held on the Frame are released with buttons when Control stops or
  the view loses focus; keys for the Frame no longer trigger Frame Control's
  own shortcuts.
- Taps only act when the picture on screen is the panel in use; positions,
  presses, keys, text and scrolls name their panel (display and window: ids
  repeat across :0 and :1, told apart by pid), and the Frame drops them if
  focus has moved on. Releases always go.
- While connecting, a tap keeps its position; on an error only releases wait
  and retries back off; trimming a long queue never drops a release.
- Lifting one of two scrolling fingers ends the scroll; a cancelled touch
  isn't a tap; clicks and holds on the bars around the picture do nothing.
- A capture loop from before a Live restart can't stop the new video.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: close the targeting gaps from the second review

- The focused panel's display comes from GAMESCOPE_FOCUS_DISPLAY (gamescope
  packs ":1" into the first value), so a window id repeated across :0 and :1
  can't be mistaken; the pid is only the fallback.
- Presses, keys, text and scrolls read focus afresh on the Frame; only moves
  use a reading up to a second old.
- A gesture remembers the panel it started on and does nothing more if that
  stops being the one in use; a press with no panel to aim at isn't sent.
- Opening a screenshot clears the panel Control would act on; switching to
  another app releases held keys and buttons.
- Trimming keeps a click with its position; the error backoff holds for new
  input too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: fixes from the SWE-2 Max review

- The Frame side tracks keys as well as buttons and lets go of both when the
  session ends.
- A stale tap tells the page, which re-reads the panels at once.
- A paused input device waits instead of ending the session; only a
  disconnect does. An OS error on one event skips it.
- Presses check focus with two property reads and do the full lookup only
  when it changed.
- Writes to an agent's stdin are serialized, so two devices sending at once
  can't tear a line (the keyboard agent too).
- Connecting gives up with a message after 15 s instead of hanging on
  "Connecting…"; text goes in 100-character pieces so releases don't wait
  behind a long paste; a cancelled mouse gesture releases what's held.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Control: stale clears, pauses reconverge, pastes split per request

- The Frame says it has caught up as soon as an aimed event lands after a
  stale one, so the page stops re-reading the panels.
- After a device pause it lets go of everything it holds (releases that
  arrived while paused were dropped), and waits for the device once per
  batch, not once per event.
- The quick focus check no longer freshens the panel geometry's age.
- Each request carries at most about 100 characters of text.
- Turning Control off while it connects doesn't report an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: build the socket path on the Frame, so Windows can import it for tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* frame_touch: any event that goes through clears the stale flag

A trackpad move names no panel, so waiting for an aimed event could leave
the page re-reading panels for the rest of the session; a release still
aimed at the old panel doesn't count.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Trackpad and key row go through gamescope's input; KDE Connect only for accents and emoji

The pad reaches whichever panel has focus (Steam's menus too), needs nothing installed, and no longer clamps to 1280x720. Characters that aren't on a US keyboard start KDE Connect the first time they're typed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Touch: a bare release no longer clears the stale-focus notice before the page has seen it

The page leaves the panel off releases, so a release says nothing about focus.
Found in review of the stale-latch fix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Pad: keep typing in order and acknowledge it once, with accents through one KDE Connect queue

Review found accent-only input repeated forever (sent was false), text split
across two transports out of order, accents stuck until the next keypress,
duplicated on retry, and racy. A batch with an accent now sends its whole
keyboard through KDE Connect; later keys follow while any are queued; the
server owns one ordered, bounded queue drained by a single thread; sent means
the whole batch is taken.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Pad: drain the KDE Connect queue without losing, stranding or reordering input

Second review found the queue could be trimmed under an in-flight send (deleting
unsent events), the worker could retire as a request enqueued, and ASCII could
overtake an accent still being typed (or the reverse). The worker now takes its
batch out while sending and puts it back on failure, retires under the lock, and
each way waits out the other's estimated typing time.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Pad: count gamescope's real typing time, with the settle margin added once

Review found the estimate used 12 ms a character where the agent takes 16 ms
(32 shifted), and added the margin on every request so it piled up.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Document gamescope pad verification and shared-device limits

* Prevent Unicode input startup from deadlocking a headset switch

* Record real focused-panel click and ASCII key delivery

* Retain accepted Unicode input under queue saturation and failure

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: SAPHID <saphid@users.noreply.github.com>
2026-10-24 11:54:36 +00:00
2026-10-24 10:53:48 +00:00

Frame Control icon

Frame Control

Manage your Valve Steam Frame from your computer.
See what the headset sees, install games and Android apps, move files and text across, and check battery and status, all over SSH.

Latest release Platforms Checks License: MIT

Website · Download · Trailer · Features · Set up the headset · Feedback · Docs


Frame Control's Games tab: installed games, sideloaded titles, and your Steam library with Frame ratings

Watch the Frame Control trailer

The trailer: 66 seconds, with sound. Downloads the MP4 from the trailer release.

Unofficial hobby project, not affiliated with Valve. Free and open source.


Features

👓 Headset view and Desktop
Live video of what the lenses show, or of the app panel in use, flat and still however the wearer looks around. Turn on Control and tap or click right on it to use the Frame from your phone or computer.

🔋 Battery and status
Charge, charging watts and time left, storage, memory, temperature, Wi-Fi, and what's running.

For repeatable read-only battery, power-rail, temperature and fan measurements, use the power and heat sampler.

🎮 Steam games
Everything you own with its Steam Frame rating. Install onto the headset with live progress, search the store, and force quit a game that stops responding.

🤖 Android apps
About 4,500 F-Droid apps rated for the Frame. One click installs each as its own app in your Steam library.

📁 Files and games
Drag files onto the window to send them. Drop a game's .zip, folder or .exe to add it to the Steam library, with Proton or the Linux runtime picked for you.

📸 Screenshots and recordings
Browse the shots you take in the headset and save them to your Pictures folder, or have new ones and your Steam clips copied over automatically. Record what the headset shows, with its sound, or export Steam's clips at any length, as .mp4 files in your Videos folder (not tested on a headset yet).

⌨️ Keyboard and trackpad
Type and point in the headset from your computer or phone, through Valve's own input path, with nothing to install. Accents and emoji use KDE Connect, which Frame Control brings along and sets up the first time you type one. Copy text to the headset's clipboard and back again. Press T to open it.

🧩 Flatpaks and display
Install desktop apps like Moonlight or VLC, and set each Android app's resolution and text size.

⚡ One-click tools
SSH, SFTP, Steam Link, remote desktop, volume, sleep, restart and shut down.

💾 Backup and recovery
Copy game saves, settings and Android app data to your computer, and restore them after a factory reset. Sign-in files and keys are never copied. A Return to Game Mode button helps when the headset is stuck in Desktop Mode. Not tested on a headset yet. How it works.

The optional Family and comfort card adds session limits, breaks, local alerts and one-click casting. A session copies a small Frame Control worker into your headset user account.

For the other features, nothing is installed on the Frame: the app uses what SteamOS already ships (sideloading a game copies Valve's own devkit scripts to ~/devkit-utils, as Valve's Devkit Client does). The optional performance HUD copies our own Python helpers into ~/.local/share/frame-control/vr/. How each feature works.

Install

Install Frame Control on your computer, not on the headset. The website is for downloads and instructions; it does not connect to your Frame in a browser. Choose the operating system and processor of the computer you will use.

Download Needs
macOS (Apple Silicon) Frame-Control-mac-arm64.dmg Nothing extra
Windows 10 / 11 (x64) Frame-Control-Setup-x64.exe · portable .zip OpenSSH Client (normally installed)
Linux (x64) AppImage · .deb ssh (most desktops have it)
Linux (arm64) AppImage · .deb ssh, and adb for Android apps (sudo apt install adb)

Which build? On a Mac, check Apple menu → About This Mac: the download is for Apple Silicon (M-series); no Intel Mac build is published. On Windows, check Settings → System → About → System type: the release is x64, not a native Windows ARM build. On Linux, uname -m reports x86_64 (x64) or aarch64 (arm64). Use .deb on Debian/Ubuntu, or AppImage on other distributions.

iPhone and iPad: no App Store, TestFlight or .ipa download is provided in the current release. It runs from your phone without a computer after setup. Build it from ios/ in Xcode on a Mac with your own signing team; see docs/iphone.md.

The app brings its own Python and adb. macOS includes SSH; Windows needs the OpenSSH Client optional feature (normally installed). From 0.4 it updates itself: when a new version is published, a banner offers Update and restart. It sends anonymous usage statistics, which you can turn off. Sharing compatibility results and error details is opt-in. See docs/privacy.md. Google doesn't publish adb for arm64 Linux, so that build uses your distribution's. If you already have adb, the app uses yours.

macOS: the app isn't notarized

There's no paid Apple developer account behind it, so macOS says the app is damaged or can't be checked. Drag it to Applications, then clear the download quarantine once, only for a download you trust from the releases linked above:

xattr -dr com.apple.quarantine "/Applications/Frame Control.app"

The first time, macOS also asks to allow local network access (for SSH) and control of Terminal (for the password prompts).

Windows: SmartScreen warning

The installer isn't code-signed, so Windows SmartScreen may say it protected your PC. Choose More info → Run anyway. The portable .zip avoids the installer: unzip it anywhere and run Frame Control.exe.

Linux: running the AppImage
chmod +x Frame-Control-linux-*.AppImage && ./Frame-Control-linux-*.AppImage

If it complains about FUSE, install libfuse2 (Ubuntu 24.04+: libfuse2t64), or run it with --appimage-extract-and-run.

Open the app

  • macOS: open the .dmg, drag Frame Control to Applications, then open it from Applications. For the unsigned-app warning, use the macOS steps above. Allow local network access and control of Terminal when prompted.
  • Windows: run the installer, then open Frame Control from Start. For the portable download, extract the entire .zip first and run Frame Control.exe inside it. See the SmartScreen note above.
  • Linux: install the .deb with your package installer and launch Frame Control from your applications menu; or make the AppImage executable and run it as described above. Install the OpenSSH client if ssh is missing.

Set up the headset

Keep the Frame awake and connect it and your computer to the same trusted local network. Guest Wi-Fi/client isolation can prevent discovery and connections.

  1. On the Frame: Steam Settings → System → Enable Developer Mode, then Developer → Set User Password. Choose a password that isn't easy to guess: it also protects sudo and remote desktop, even after SSH keys are set up.
  2. On your computer: open Frame Control and choose Set Up Connection…. If you dismissed the first-run prompt, use Frame → Set Up Connection…. Setup opens a terminal and tries to find the headset. If it can't, use its IP address from Quick Settings or Steam Settings → Internet. Windows/Linux setup prompts for it; macOS exits instead. On macOS, return to the app → Devices & connection → + Add a headset…, set SSH alias to frame for the default headset, fill in Address, then choose Set Up….
  3. Pair or enter the password: setup first tries Valve's devkit pairing. Open Steam Settings → Developer → Pair new host in the headset and approve only the request you just started. The service and pairing-mode step are verified on a Frame; approval and key installation through that flow remain unverified. If pairing fails or times out, setup falls back to asking for the Developer Mode password in the computer's terminal. Password typing may show no characters; press Enter when done. See SSH details.
  4. After setup succeeds, return to Frame Control. It uses SSH keys for later connections whenever the Frame is awake and reachable. If it stays offline, check Developer Mode, the address and network isolation, then run setup again. For connections away from home, see Tailscale.

What it changes: only what you click. Installs go to your user account on the Frame (--user Flatpaks, Lepton instances, Steam downloads, sideloaded games in ~/devkit-game), and nothing needs sudo except the power buttons. On your computer it adds a Host frame entry to ~/.ssh/config and keys at ~/.ssh/id_ed25519_frame and ~/.ssh/id_rsa_frame_devkit (the pairing service only takes RSA keys).

Feedback

This is a first public test, so reports are really useful, especially from Windows and Linux. The quickest way is Report a problem in the app (the speech-bubble button at the top, or Help → Report a Problem…). It adds diagnostics with personal details removed, shows you exactly what's included, and sends it privately to the maintainer; nothing is published. Without the app, use the feedback form. Please include:

  • what you tried and what happened
  • your computer's OS and your SteamOS build (Steam Settings → System)
  • the server log: Frame → Show Server Log in the app

Issues and PRs opened directly on GitHub by new contributors are auto-closed until a maintainer approves them; see CONTRIBUTING.md.

Going further

This repo also holds the scripts behind the app and field notes on how the Frame's software fits together, all checked against a real headset and labelled verified or inferred.

Frame Control in detail Every feature, how it works, per-platform notes, building
Scripts and headset setup The command-line helpers, minimum typing, streaming options, floating panels
How the Frame works SteamVR → gamescope → Plasma, verified facts, debugging
Android apps (Lepton) Sideloading, the rated F-Droid catalogue, per-app instances
Import from my Quest Copy games you own off your Quest by USB; what runs, and why licence-checked games may not
My Meta library Experimental: install and test Quest games your Meta account owns, with your own token
Sideloading Linux and Windows games A .zip, folder or .exe as a Steam Devkit Game, runtime detection
Install links for websites frame-control://install links and manifests, the rules, a button to paste
VR comfort and HUD · Steam games · VR video · WebXR in Chromium Installing and buying, watching VR180/360, the Chromium build
PC in the headset Experimental, Windows only, off until you turn it on: a whole PC screen as a panel in the Frame. Tested in a Windows VM; individual windows need a WGC-enabled build
Mac in the headset Mac windows and screens as panels in the Frame, with laser and keyboard input
VR mods and custom songs UEVR for Unreal games from Frame Control (install, play in VR, remove), the per-game table, Beat Saber blockers
SSH · Streaming · Files · Panels · Tailscale Topic notes
Frame Control for iPhone The iPhone and iPad app, how it runs the server on the Frame, pairing
Recovery and OS images Where to download the Frame's OS, what's inside, testing without the headset
AI agents and assistant Key-free MCP tools, human approvals, and an opt-in assistant panel
Power and heat measurements Read-only sampling and offline HTML reports with JSON/CSV downloads
Testing Unit tests, end-to-end tests against a fake Frame in Docker, the headset smoke test and a Windows test VM
Open questions What's still unchecked
Security notes
  • With Developer Mode on, sshd, ADB and xrdp are all reachable on your LAN. Each running Lepton (Android) instance opens its own ADB port in 5555–5599, listening on 0.0.0.0 rather than only loopback. This was seen on the device on 2026-09-25, so anyone on the network can reach it. Use trusted networks only, and turn Developer Mode off when you don't need it.
  • Frame Control reaches ADB and the Steam client's DevTools port (Frame loopback 127.0.0.1:8080) only through SSH tunnels. The compatibility database key (maintainer-only) is never written to the repo.
  • steamos has sudo, protected by the same Developer Mode password. Once you've switched to key auth, that password still protects sudo and RDP, so pick one that isn't trivially guessable.
  • Don't port-forward 22, 3389, or 5555–5599 from your router. For remote access, use Tailscale: scripts/tailscale-on-frame.sh (no sudo). In its userspace mode every Frame port is reachable from your tailnet, including Steam's DevTools on loopback 8080; see docs/tailscale.md.

Development

python3 -m unittest discover -s tests   # server tests; no headset needed
scripts/e2e.sh                          # end-to-end against a fake Frame (Linux with Docker)
cd app && npm install && npm start      # run the app from the checkout

The server is Python stdlib only; the app is Electron. GitHub Actions runs the tests on macOS, Windows and Linux, and a v* tag builds all three installers into a draft release, which reaches users once published. See building and releasing.

License

MIT. The apps also ship other people's software under its own licence, notably KDE Connect (GPL) for the keyboard and trackpad; see THIRD_PARTY_NOTICES.md. Steam, Steam Frame and SteamVR are trademarks of Valve Corporation. This project isn't affiliated with or endorsed by Valve.

See Game storage for read-only headset space and Steam install sizes, and the changelog for unreleased changes.

S
Description
GitHub pull mirror of saphid/frame-control. GitHub repository ID: 1388213277. Gitea import mode: github-full.
https://github.com/saphid/frame-control Readme MIT
47 MiB
0 Stars 1 Watchers 0 Forks
2026-10-24 15:32:38 +02:00
Languages
Python 43.5%
JavaScript 41.7%
HTML 9.2%
Swift 3.1%
Shell 1.3%
Other 1%