fix(input): cancel old headset input when switching routes

Windows test VM: a driver script and how to set one up

README.md

@@ -73,7 +73,7 @@ Drag files onto the window to send them. Drop a game's .zip, folder or .exe to a
 Browse the shots you take in the headset and save them to your Pictures folder.
 
 **⌨️ Keyboard and trackpad**<br>
-Type and point in the Frame's apps from your computer or phone, through KDE Connect, which Frame Control brings along and sets up on the Frame. Nothing else to install, anywhere.
+Type and point in the headset from your computer or phone, through Valve's own input path, with nothing to install. Accents and emoji use KDE Connect, which Frame Control brings along and sets up the first time you type one.
 
 </td>
 </tr>
@@ -224,7 +224,7 @@ Frame's software fits together, all checked against a real headset and labelled
 | [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing |
 | [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset |
 | [AI agents and assistant](docs/agents.md) | Key-free MCP tools, human approvals, and an opt-in assistant panel |
-| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, and the headset smoke test |
+| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, the headset smoke test and a Windows test VM |
 | [Open questions](docs/open-questions.md) | What's still unchecked |
 
 <details>

docs/streaming.md

@@ -21,6 +21,20 @@ The confidence labels are the same as in [ssh.md](ssh.md).
 | **ADB + scrcpy (Lepton only)** | A mirror of the Android container | **Guess** | `brew install scrcpy android-platform-tools`, then `adb connect frame.local:5555` while Lepton Development is running ([adb_lepton](https://partner.steamgames.com/doc/steamhardware/steamframe/adb_lepton)), then `scrcpy`. This only shows Android apps, not SteamOS. |
 | VNC server on the Frame (krfb / wayvnc) | A mirror of the Plasma desktop | **Inferred (SteamOS)** | Deck users run krfb in Desktop Mode ([one.vg](https://one.vg/blog/remote-control-your-steam-deck)). On the Frame, the in-headset desktop is a virtual screen, and krfb isn't known to be preinstalled. RDP and Steam Link cover this case, so it's not recommended. |
 
+**RDP from Windows, verified 2026-09-30:** Windows 11 25H2's Remote Desktop
+(`mstsc`) against BUILD_ID 20260925.6191901. xrdp picks TLS, not NLA, so
+Remote Desktop never asks for a user or password. It warns that the certificate
+(`www.xrdp.org`) can't be verified. After **Yes**, xrdp shows its own "Login to
+frame" box with the username blank. Any user but `steamos` gets "User does not
+exist, or could not be authenticated". Signing in as `steamos` with the
+Developer Mode password opens a Plasma (X11) desktop within about 6 seconds.
+It's a new session on display `:10`, separate from what the headset shows,
+and it uses about 1.3 GB of the Frame's memory. Closing Remote Desktop leaves
+it running, and the next sign-in reconnects to it. To end it, find its
+session with `loginctl list-sessions` over SSH (its leader is `xrdp-sesexec`)
+and run `loginctl terminate-session <id>`; the headset's own session keeps
+running.
+
 **Recommendation for A:** start with Steam Link for macOS, because Valve
 documents it. Use Windows App (RDP) when you want a proper Linux desktop on the
 Mac with keyboard, mouse, and clipboard.
@@ -137,10 +151,13 @@ two fingers to scroll, two-finger tap to right-click) plus a text field that
 types on the Frame. On a computer, clicking the pad passes your mouse and
 keyboard through to the Frame until you press Esc (⌘ is sent as Ctrl on a Mac).
 
-It goes through **KDE Connect**, the first-party route (KDE makes the Frame's
-desktop): Frame Control's server runs [`ui/frame_input_agent.py`](../ui/frame_input_agent.py)
-on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
-`kdeconnectd` as if it were a phone. KDE Connect does the typing and clicking.
+Since Control (below), the pad goes through **gamescope's own input socket**
+([`ui/frame_touch.py`](../ui/frame_touch.py)), so it reaches whichever panel has
+focus, Steam's own menus included, and needs nothing installed on the Frame.
+Only characters that aren't on a US keyboard (accents, emoji) go through
+**KDE Connect** as below; it starts the first time you type one. (Before
+Control, everything went through KDE Connect, which is why the notes below
+describe it in detail.)
 
 **Verified 2026-09-28** (SteamOS 0.4.1, build 20260925.6191901):
 

docs/testing.md

@@ -8,6 +8,7 @@ A fourth, a SteamOS VM, may come later ([issue #6](https://github.com/saphid/ste
 | Unit tests (`tests/*.py`) | `python3 -m unittest discover -s tests` | Nothing | Parsing, validation, request guards; SSH and HTTP are mocked |
 | Fake Frame (`tests/e2e`) | `scripts/e2e.sh` | Linux with Docker | The real server and scripts against a container that behaves like a Frame |
 | Headset smoke test | `scripts/frame-smoke.sh` | A Frame on the `frame` alias | Install, launch and remove on the real device, recorded with its BUILD_ID |
+| [Windows test VM](#windows-test-vm) | `scripts/windows-vm.sh` | A Linux machine with KVM and Docker | The Windows build on a real Windows desktop, against a real Frame when needed |
 
 ## Unit tests
 
@@ -186,6 +187,86 @@ refuses ids with a hyphen (`missing/invalid arguments`), which the fake had
 accepted. The fake now refuses them the same way, and Frame Control makes ids
 Steam accepts.
 
+## Windows test VM
+
+The unit tests run on Windows in CI, but the app itself doesn't. Anything that
+depends on the Windows desktop (Remote Desktop, the installer, the bundled
+Python, file dialogs) needs a real Windows machine. This is a Windows 11 VM in a
+[dockur/windows](https://github.com/dockur/windows) container on a Linux machine
+with KVM, driven from the Mac with `scripts/windows-vm.sh`.
+
+Setting it up, once, on the Linux machine:
+
+- **Windows comes from Microsoft.** The container downloads the Windows 11
+  image from Microsoft on first start. Windows runs unactivated, which is fine
+  for testing. Don't use activation workarounds or third-party Windows images.
+- **Publish its ports on loopback only.** Map `127.0.0.1:2222:22` (SSH),
+  `127.0.0.1:8006:8006` (the web console) and, if you need it,
+  `127.0.0.1:13389:3389`. The Mac reaches them through `ssh -J`. Use
+  `restart: "no"` and `stop_grace_period: 2m` so it only runs when someone is
+  testing, and a `docker stop` shuts Windows down cleanly.
+- **Give it SSH on first sign-in.** The container runs `/oem/install.bat`
+  once. Have it add the OpenSSH Server capability, start `sshd`, set PowerShell
+  as its default shell, and put a dedicated public key (for example
+  `~/.ssh/id_ed25519_winvm` on the Mac) in
+  `C:\ProgramData\ssh\administrators_authorized_keys`.
+- Give it 4 cores, 8 GB of memory and a 32 GB disk. That's enough for the app
+  and the tests.
+
+Using it, with `WINVM_HOST` set to the Linux machine's ssh alias:
+
+```sh
+scripts/windows-vm.sh up                        # start it and wait for SSH (1-2 minutes)
+scripts/windows-vm.sh put Frame-Control-Setup-x64.exe
+scripts/windows-vm.sh ps 'Start-Process "$env:USERPROFILE\Downloads\Frame-Control-Setup-x64.exe" /S -Wait'
+scripts/windows-vm.sh shot screen.png            # what's on its screen
+scripts/windows-vm.sh click 723 359              # screen pixels, as in the screenshot
+scripts/windows-vm.sh keys s t e a m o s ret     # QEMU key names
+scripts/windows-vm.sh down                       # shut Windows down
+```
+
+- **Clicks go through a scheduled task.** Commands over SSH run in a
+  session with no desktop, so `click` and `scroll` write the position to a
+  file, and a scheduled task running as the signed-in user replays it. The
+  VM's screen must be signed in; it is after `up`. Clicks and scrolls sent at
+  the same time run one after another. QEMU's own `mouse_move` is relative
+  and drifts, so the script doesn't use it.
+- **Screenshots may not show the pointer.** Check the result of a click (a
+  menu that opens, a button that changes) rather than the pointer's position.
+- **Windows' `ssh` waits for stdin.** The script closes it for every command.
+  Do the same if you run `ssh` in the VM by hand.
+- **Starting the app.** Run Frame Control in the signed-in session, not over
+  SSH. Use its Start-menu shortcut via `click`, or a scheduled task like the
+  one `click` uses.
+
+**Testing against a real Frame.** The VM reaches the headset on the LAN like
+any other computer. Run **Set Up Connection** in the VM's Frame Control once;
+that makes the VM's keys. So the VM doesn't keep access between test runs,
+afterwards take the lines it added out of the headset's
+`~/.ssh/authorized_keys`: the ones matching the VM's
+`~/.ssh/id_ed25519_frame.pub` and, if pairing made one,
+`~/.ssh/id_rsa_frame_devkit.pub`. Check that `ssh -o BatchMode=yes frame true`
+in the VM now fails. Then, around each run:
+
+```sh
+scripts/windows-vm.sh frame-key add       # let the VM's key into the headset
+# ... test ...
+scripts/windows-vm.sh frame-key remove    # and take it out again
+```
+
+`add` appends the VM's key tagged `windows-vm-test`, unless the headset
+already trusts that key through another line. `remove` deletes only the exact
+line `add` wrote, so it doesn't undo what Set Up Connection did, and it leaves
+the file alone if it can't rewrite it. Follow the shared-device
+procedure in [Headset smoke test](#headset-smoke-test) before installing or
+launching anything on the headset.
+
+**Verified 2026-09-30:** Windows 11 Pro 25H2 (build 26200) against a Frame on
+BUILD_ID 20260925.6191901. The script was used to install Frame Control 0.4.0,
+connect it to the Frame and follow Remote Desktop through to the Frame's
+desktop ([what it does](streaming.md#a-see-and-control-the-frame-from-the-mac)).
+`frame-key` left `authorized_keys` byte-for-byte as it was.
+
 ## Owned media player
 
 `tests/test_media.py` covers layout evidence and overrides, OU eye ordering,

scripts/windows-vm.sh

@@ -0,0 +1,161 @@
+#!/usr/bin/env zsh
+# Mac or Linux: drive a Windows 11 test VM for Frame Control's Windows build.
+# The VM is a dockur/windows container on another machine; this reaches it over
+# SSH through that machine. See docs/testing.md#windows-test-vm.
+#
+# Usage: scripts/windows-vm.sh <command> [args]
+#   up | down | status      start it (waits for SSH), shut Windows down cleanly, show state
+#   ps '<PowerShell>'       run PowerShell as the VM's user
+#   put <file> [<dir>]      copy a file in (default: the user's Downloads)
+#   shot <out.png>          save the VM's screen
+#   click <x> <y>           click screen pixel x,y in the signed-in session
+#   scroll <x> <y> <n>      turn the wheel n notches at x,y (negative scrolls down)
+#   keys <key>...           type QEMU key names: a, shift-a, ret, tab, esc, spc ...
+#   frame-key add|remove    let the VM's Frame Control key into the headset (`frame`
+#                           alias) for a test run, then take it out again
+#
+# Set WINVM_HOST to the ssh alias of the machine running the container. Optional:
+# WINVM_CONTAINER (frame-winvm), WINVM_USER (frame), WINVM_PORT (2222: the VM's
+# sshd, published on that machine's loopback), WINVM_KEY (~/.ssh/id_ed25519_winvm).
+set -euo pipefail
+setopt extendedglob
+
+host=${WINVM_HOST:?set WINVM_HOST to the ssh alias of the machine running the VM}
+ctr=${WINVM_CONTAINER:-frame-winvm}
+user=${WINVM_USER:-frame}
+port=${WINVM_PORT:-2222}
+key=${WINVM_KEY:-$HOME/.ssh/id_ed25519_winvm}
+opts=(-o ConnectTimeout=20 -o StrictHostKeyChecking=accept-new
+      -o UserKnownHostsFile=${TMPDIR:-/tmp}/windows-vm-known_hosts -i $key -J $host)
+TAG=windows-vm-test  # comment on the VM's key in the headset's authorized_keys
+
+die() { print -u2 "windows-vm: $*"; exit 1 }
+int() { [[ $1 == (-|)<-99999> ]] || die "not a whole number (up to 99999): $1" }
+# These go into commands run by a shell on the other machine, so keep them plain.
+for v in $host $ctr $user; do [[ $v == [A-Za-z0-9_.]##[A-Za-z0-9_.-]# ]] || die "not a plain name: $v"; done
+[[ $port == <1-65535> ]] || die "WINVM_PORT isn't a port: $port"
+
+# Windows' OpenSSH waits for stdin to close, so it always gets /dev/null. The
+# script travels UTF-16 base64-encoded, so no quoting survives two shells.
+vm_ps() {
+  local b64=$(print -rn -- "\$ProgressPreference = 'SilentlyContinue'"$'\n'"$1" |
+              iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
+  ssh $opts -p $port $user@127.0.0.1 "powershell -NoProfile -NonInteractive -OutputFormat Text -EncodedCommand $b64" </dev/null
+}
+
+# QEMU's monitor inside the container: one command per line on stdin.
+monitor() { ssh $host "docker exec -i $ctr nc -q 1 -U /run/shm/monitor.sock" >/dev/null }
+
+# Input has to come from the signed-in desktop session, not SSH's session 0, so
+# a scheduled task running as the user replays one click or wheel turn written
+# to input.txt, then deletes the file to say it's done. Any error stops it
+# before that, so the caller times out instead of reporting a click.
+INPUT_PS1='$ErrorActionPreference = "Stop"
+$a = (Get-Content "$PSScriptRoot\input.txt").Trim() -split "\s+"
+Add-Type -Namespace WinVm -Name Input -MemberDefinition @"
+[DllImport("user32.dll")] public static extern bool SetProcessDPIAware();
+[DllImport("user32.dll")] public static extern bool SetCursorPos(int x, int y);
+[DllImport("user32.dll")] public static extern void mouse_event(uint flags, int dx, int dy, int data, System.IntPtr extra);
+"@
+[WinVm.Input]::SetProcessDPIAware() | Out-Null
+if (-not [WinVm.Input]::SetCursorPos([int]$a[0], [int]$a[1])) { throw "SetCursorPos failed" }
+Start-Sleep -Milliseconds 150
+if ($a[2] -eq "click") {
+  [WinVm.Input]::mouse_event(0x2, 0, 0, 0, [IntPtr]::Zero); Start-Sleep -Milliseconds 60
+  [WinVm.Input]::mouse_event(0x4, 0, 0, 0, [IntPtr]::Zero)
+} else { [WinVm.Input]::mouse_event(0x800, 0, 0, 120 * [int]$a[3], [IntPtr]::Zero) }
+Remove-Item "$PSScriptRoot\input.txt"'
+
+pointer() {  # x y click|wheel [notches]
+  local b64=$(print -rn -- $INPUT_PS1 | base64 | tr -d '\n')
+  vm_ps '$ErrorActionPreference = "Stop"
+$d = Join-Path $env:LOCALAPPDATA "windows-vm"; $req = "$d\input.txt"; $mine = $false
+# Giving up: end the helper first, or it could wake up and act in a later request.
+function Abandon {
+  Stop-ScheduledTask -TaskName WindowsVmInput -ErrorAction SilentlyContinue
+  foreach ($i in 1..25) {
+    if ((Get-ScheduledTask -TaskName WindowsVmInput -ErrorAction SilentlyContinue).State -ne "Running") { break }
+    Start-Sleep -Milliseconds 200
+  }
+  Remove-Item $req -ErrorAction SilentlyContinue
+}
+trap { [Console]::Error.WriteLine("windows-vm: $_"); if ($mine) { Abandon }; exit 1 }
+# One request at a time: the helper, the task and input.txt are shared. Windows
+# frees the mutex when this process ends, however it ends.
+$lock = New-Object Threading.Mutex($false, "windows-vm-input")
+try { $mine = $lock.WaitOne(15000) } catch [Threading.AbandonedMutexException] { $mine = $true }
+if (-not $mine) { [Console]::Error.WriteLine("windows-vm: another click or scroll is still in progress"); exit 1 }
+New-Item -ItemType Directory -Force $d | Out-Null
+[IO.File]::WriteAllText($req, "'"$*"'")
+[IO.File]::WriteAllText("$d\input.ps1", [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String("'$b64'")))
+$act = New-ScheduledTaskAction -Execute powershell.exe -Argument "-NoProfile -WindowStyle Hidden -ExecutionPolicy Bypass -File `"$d\input.ps1`""
+$who = New-ScheduledTaskPrincipal -UserId $env:USERNAME -LogonType Interactive
+Register-ScheduledTask -TaskName WindowsVmInput -Action $act -Principal $who -Force | Out-Null
+Start-ScheduledTask -TaskName WindowsVmInput
+foreach ($i in 1..50) { if (-not (Test-Path $req)) { exit 0 }; Start-Sleep -Milliseconds 200 }
+Abandon
+[Console]::Error.WriteLine("windows-vm: no input after 10 s: is $env:USERNAME signed in on the VM screen, and is x,y on it?"); exit 1'
+}
+
+(( $# )) || die "usage: see the top of $0"
+cmd=$1; shift
+case $cmd in
+  up)
+    ssh $host "docker start $ctr" >/dev/null
+    for i in {1..60}; do
+      vm_ps 'exit 0' 2>/dev/null && { print "up"; exit 0 }
+      sleep 5
+    done
+    die "Windows didn't answer on SSH within 5 minutes" ;;
+  down)   # the container turns SIGTERM into an ACPI shutdown and waits for Windows
+    ssh $host "docker stop -t 150 $ctr" >/dev/null && print "down" ;;
+  status) ssh $host "docker ps -a --filter 'name=^$ctr\$' --format '{{.Names}}: {{.Status}}'" ;;
+  ps)     (( $# == 1 )) || die "usage: ps '<PowerShell>'"; vm_ps "$1" ;;
+  put)
+    [[ -f ${1:-} ]] || die "usage: put <file> [<dir>]"
+    scp -q $opts -P $port $1 "$user@127.0.0.1:${2:-C:/Users/$user/Downloads}/${1:t}" </dev/null ;;
+  shot)
+    (( $# == 1 )) || die "usage: shot <out.png>"
+    print "screendump /tmp/windows-vm-shot.ppm" | monitor
+    sleep 1
+    ssh $host "docker exec $ctr sh -c 'cat /tmp/windows-vm-shot.ppm && rm /tmp/windows-vm-shot.ppm'" | python3 -c '
+import re, struct, sys, zlib
+d = sys.stdin.buffer.read()
+m = re.match(rb"P6\s+(\d+)\s+(\d+)\s+255\s", d) or sys.exit("windows-vm: no screen dump")
+w, h = int(m[1]), int(m[2]); px = d[m.end():]
+raw = b"".join(b"\0" + px[y * w * 3:(y + 1) * w * 3] for y in range(h))
+chunk = lambda t, b: struct.pack(">I", len(b)) + t + b + struct.pack(">I", zlib.crc32(t + b))
+open(sys.argv[1], "wb").write(b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
+                              + chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))
+' $1
+    print $1 ;;
+  click)  (( $# == 2 )) || die "usage: click <x> <y>"; int $1; int $2; pointer $1 $2 click ;;
+  scroll) (( $# == 3 )) || die "usage: scroll <x> <y> <n>"; int $1; int $2; int $3; pointer $1 $2 wheel $3 ;;
+  keys)
+    (( $# )) || die "usage: keys <key>..."
+    for k; do [[ $k == [a-z0-9_.,/=-]## ]] || die "not a QEMU key name: $k"; done
+    for k; do print "sendkey $k"; done | monitor ;;
+  frame-key)
+    [[ ${1:-} == (add|remove) ]] || die "usage: frame-key add|remove"
+    pub=(${=$(vm_ps 'Get-Content (Join-Path $env:USERPROFILE ".ssh\id_ed25519_frame.pub")' | tr -d '\r')})
+    [[ ${pub[1]:-} == ssh-ed25519 && ${pub[2]:-} == [A-Za-z0-9+/=]## ]] ||
+      die "the VM has no Frame Control key yet: run Set Up Connection in the app first"
+    # $1: add|remove, $2: the key's base64, $3: the exact line this script owns.
+    ssh frame "sh -s -- $1 '$pub[2]' '$pub[1] $pub[2] $TAG'" <<'EOF'
+f=~/.ssh/authorized_keys
+if [ "$1" = add ]; then
+  if grep -qxF "$3" "$f"; then exit 0; fi
+  if grep -qF "$2" "$f"; then echo "the headset already trusts this key through another entry; left as it is"; exit 0; fi
+  [ -z "$(tail -c 1 "$f")" ] || echo >> "$f"   # a last line without a newline would swallow ours
+  echo "$3" >> "$f"
+else
+  t=$(mktemp "$f.XXXXXX") || exit 1
+  grep -vxF "$3" "$f" > "$t"   # 0: lines left, 1: none left, more: couldn't read or write
+  if [ $? -gt 1 ] || ! chmod 600 "$t" || ! mv "$t" "$f"; then
+    rm -f "$t"; echo "couldn't rewrite $f; it's unchanged" >&2; exit 1
+  fi
+fi
+EOF
+    print "frame-key $1: done" ;;
+  *) die "unknown command: $cmd (see the top of $0)" ;;
+esac

tests/test_input.py

@@ -18,6 +18,7 @@ import tempfile
 import threading
 import time
 import unittest
+from unittest import mock
 from pathlib import Path
 
 ROOT = Path(__file__).resolve().parent.parent
@@ -631,3 +632,257 @@ class PageQueue(unittest.TestCase):
 
 if __name__ == "__main__":
     unittest.main()
+
+
+class PadEvents(unittest.TestCase):
+    """The trackpad and key row go through gamescope; accents and emoji go the KDE Connect way."""
+
+    @classmethod
+    def setUpClass(cls):
+        import server
+        cls.pad = staticmethod(server.pad_events)
+
+    def test_moves_clicks_and_scroll(self):
+        out, extra = self.pad([{"dx": 3, "dy": -2}, {"singleclick": True}, {"singlehold": True},
+                               {"singlerelease": True}, {"scroll": True, "dy": 1}, {"rightclick": True}])
+        self.assertEqual(out[0], {"dx": 3, "dy": -2})
+        self.assertEqual(out[1:3], [{"button": "left", "down": True}, {"button": "left", "down": False}])
+        self.assertEqual(out[3:5], [{"button": "left", "down": True}, {"button": "left", "down": False}])
+        self.assertEqual(out[5], {"scroll": [0, -15]})  # up = content follows the finger
+        self.assertEqual(out[6:], [{"button": "right", "down": True}, {"button": "right", "down": False}])
+        self.assertEqual(extra, [])
+
+    def test_keys_and_modifiers(self):
+        out, _ = self.pad([{"specialKey": 12}, {"specialKey": 1, "ctrl": True}, {"key": "c", "ctrl": True}])
+        self.assertEqual(out[:2], [{"key": 28, "down": True}, {"key": 28, "down": False}])
+        self.assertEqual([e["key"] for e in out[2:6]], [29, 14, 14, 29])  # Ctrl+Backspace
+        self.assertEqual([(e["key"], e["down"]) for e in out[6:]],
+                         [(29, True), (46, True), (46, False), (29, False)])  # Ctrl+C
+
+    def test_text_stays_ascii_through_gamescope(self):
+        out, slow = self.pad([{"key": "Hi "}, {"key": "there"}])
+        self.assertEqual(out, [{"text": "Hi there"}])
+        self.assertEqual(slow, [])
+
+    def test_keyboard_goes_to_kde_whole_and_in_order(self):
+        events = [{"key": "a"}, {"key": "é"}, {"specialKey": 12}, {"key": "c", "ctrl": True}, {"dx": 2, "dy": 1}]
+        out, slow = self.pad(events, kde=True)
+        self.assertEqual(out, [{"dx": 2, "dy": 1}])  # the pointer still goes through gamescope
+        self.assertEqual(slow, events[:4])  # the keyboard as sent, in order
+
+    def test_shifted_key_with_modifier(self):
+        out, _ = self.pad([{"key": "A", "ctrl": True}])
+        self.assertEqual([e["key"] for e in out], [29, 42, 30, 30, 42, 29])
+
+
+class PadDelivery(unittest.TestCase):
+    """remote_input's promises: "sent" means the whole batch is taken; accents go once, in order."""
+
+    class Agent:
+        def __init__(self, sent=True, state="ready"):
+            self.sent, self.state, self.got = sent, state, []
+
+        def send(self, events):
+            if events and self.sent:
+                self.got.append(events)
+            return {"state": self.state, "sent": self.sent and bool(events)}
+
+    def setUp(self):
+        import server
+        self.s = server
+        self.saved = (server._touch, server._input, server.KDE_WAIT)
+        server._touch, server._input = self.Agent(), self.Agent()
+        self.reset()
+
+    def tearDown(self):
+        self.s._touch, self.s._input, self.s.KDE_WAIT = self.saved
+        self.settle()
+        self.reset()
+
+    def reset(self):
+        self.s._kde_queue.clear()
+        self.s._kde_until = self.s._typed_until = 0.0
+        self.s._kde_worker[0] = None
+
+    def settle(self):
+        worker = self.s._kde_worker[0]
+        if worker:
+            worker.join(5)
+
+    def typed(self):
+        return [e for batch in self.s._input.got for e in batch]
+
+    def test_accent_only_batch_is_acknowledged_and_typed_once(self):
+        status = self.s.remote_input({"events": [{"key": "é"}]})
+        self.assertTrue(status["sent"])  # so the page doesn't send it again
+        self.settle()
+        self.assertEqual(self.typed(), [{"key": "é"}])
+
+    def test_mixed_text_stays_in_one_ordered_stream(self):
+        self.s.remote_input({"events": [{"key": "aéb"}, {"specialKey": 12}]})
+        self.settle()
+        self.assertEqual(self.typed(), [{"key": "aéb"}, {"specialKey": 12}])
+        self.assertEqual(self.s._touch.got, [])  # none of it split off through gamescope
+
+    def test_batch_gamescope_didnt_take_is_not_queued(self):
+        self.s._touch = self.Agent(sent=False, state="starting")
+        status = self.s.remote_input({"events": [{"dx": 1, "dy": 1}, {"key": "é"}]})
+        self.assertFalse(status["sent"])  # the page tries the whole batch again...
+        self.assertEqual(self.s._kde_queue, [])  # ...so the accent isn't queued twice
+
+    def test_waits_for_kde_connect_then_sends_without_more_input(self):
+        self.s._input = self.Agent(sent=False, state="starting")
+        self.s.remote_input({"events": [{"key": "é"}]})
+        time.sleep(0.2)
+        self.assertEqual(self.s._kde_queue, [{"key": "é"}])
+        self.s._input.sent = True
+        self.settle()
+        self.assertEqual(self.typed(), [{"key": "é"}])
+        self.assertEqual(self.s._kde_queue, [])
+
+    def test_later_ascii_waits_behind_a_pending_accent(self):
+        self.s._input = self.Agent(sent=False, state="starting")
+        self.s.remote_input({"events": [{"key": "é"}]})
+        self.s.remote_input({"events": [{"key": "x"}]})
+        self.assertEqual(self.s._touch.got, [])  # "x" can't overtake the "é"
+        self.assertEqual(self.s._kde_queue, [{"key": "é"}, {"key": "x"}])
+        self.s._input.sent = True
+        self.settle()
+        self.assertEqual(self.typed(), [{"key": "é"}, {"key": "x"}])
+
+    def test_queue_is_bounded_and_dropped_on_error(self):
+        self.s._input = self.Agent(sent=False, state="error")
+        self.s.remote_input({"events": [{"key": "é"}] * 150})
+        self.s.remote_input({"events": [{"key": "é"}] * 150})
+        self.settle()
+        self.assertEqual(self.s._kde_queue, [])
+        self.s._input = self.Agent(sent=False, state="starting")
+        self.s.KDE_WAIT = 0.3
+        self.s.remote_input({"events": [{"key": "é"}] * 150})
+        self.s.remote_input({"events": [{"key": "é"}] * 150})
+        self.assertLessEqual(len(self.s._kde_queue), self.s.KDE_QUEUE_LIMIT)
+        self.settle()
+        self.assertEqual(self.s._kde_queue, [])
+
+    def test_ascii_just_after_an_accent_stays_behind_it(self):
+        self.s.remote_input({"events": [{"key": "é"}]})
+        self.settle()
+        self.s.remote_input({"events": [{"key": "x"}]})  # KDE Connect may still be typing the é
+        self.settle()
+        self.assertEqual(self.s._touch.got, [])
+        self.assertEqual(self.typed(), [{"key": "é"}, {"key": "x"}])
+
+    def test_accent_just_after_gamescope_text_waits_for_it(self):
+        self.s.remote_input({"events": [{"key": "x" * 50}]})
+        self.assertEqual(self.s._touch.got, [[{"text": "x" * 50}]])
+        start = time.time()
+        self.s.remote_input({"events": [{"key": "é"}]})
+        self.settle()
+        self.assertGreater(time.time() - start, 0.6)  # 50 keys at 16 ms, plus the margin
+        self.assertEqual(self.typed(), [{"key": "é"}])
+
+    def test_typing_time_counts_every_key_transition(self):
+        t = self.s.typing_seconds
+        self.assertAlmostEqual(t([{"text": "ab"}]), 4 * 0.008)
+        self.assertAlmostEqual(t([{"text": "A"}]), 4 * 0.008)  # shift down, key down, key up, shift up
+        self.assertAlmostEqual(t([{"text": "x" * 500}]), 8.0)  # a long paste
+        self.assertAlmostEqual(t([{"key": 29, "down": True}, {"dx": 3, "dy": 0}]), 0.008)
+
+    def test_the_margin_is_added_once_not_per_request(self):
+        for _ in range(10):
+            self.s.remote_input({"events": [{"key": "ab"}]})
+        # ten requests of 4 transitions each: the work queues up, with no margin added per request
+        self.assertLess(self.s._typed_until - time.time(), 10 * 4 * 0.008 + 0.05)
+
+    def test_trimming_while_sending_doesnt_drop_unsent_events(self):
+        slow = self.Agent()
+        gate = threading.Event()
+        send = slow.send
+
+        def held(events):
+            gate.wait(5)
+            return send(events)
+        slow.send = held
+        self.s._input = slow
+        self.s.remote_input({"events": [{"key": "é"}] * 150})
+        time.sleep(0.2)  # the worker has taken its 150 and is mid-send
+        self.s.remote_input({"events": [{"key": "ü"}] * 150})
+        gate.set()
+        self.settle()
+        sent = self.typed()
+        self.assertEqual(sent[:150], [{"key": "é"}] * 150)
+        self.assertEqual(sent[150:], [{"key": "ü"}] * 150)  # none of the new ones went missing
+
+    def test_input_queued_as_the_worker_finishes_is_still_sent(self):
+        for _ in range(40):
+            self.reset()
+            self.s._input.got.clear()
+            self.s.remote_input({"events": [{"key": "é"}]})
+            time.sleep(0.002)
+            self.s.remote_input({"events": [{"key": "ü"}]})
+            self.settle()
+            self.assertEqual(self.s._kde_queue, [])
+            self.assertEqual(self.typed(), [{"key": "é"}, {"key": "ü"}])
+
+
+class PadRoute(unittest.TestCase):
+    def setUp(self):
+        import server
+        self.s = server
+        self.agents = mock.patch.multiple(server, _touch=mock.Mock(), _input=mock.Mock())
+        self.agents.start()
+        self.addCleanup(self.agents.stop)
+        self.s._touch.send.return_value = {"state": "ready", "sent": True}
+        self.s._input.send.return_value = {"state": "ready", "sent": True}
+        self.s.reset_pad_route()
+        self.s._touch.reset_mock(); self.s._input.reset_mock()
+        self.addCleanup(self.s.reset_pad_route)
+
+    def switch(self):
+        with mock.patch.multiple(self.s, FRAME="frame-a", HOST_OPTS=[], MUX=[], SSH=[], macview=None), \
+             mock.patch.multiple(self.s.frame_android, FRAME="frame-a", SSH_OPTS=[]):
+            self.s.route("frame-b", [])
+
+    def test_switch_stops_both_input_agents_and_discards_old_queue(self):
+        self.s._kde_queue.append({"key": "old"})
+        self.s._kde_until = self.s._typed_until = time.time() + 60
+        cancel = self.s._kde_cancel
+        self.switch()
+        self.s._input.stop.assert_called_once()
+        self.s._touch.stop.assert_called_once()
+        self.assertEqual(self.s._kde_queue, [])
+        self.assertEqual((self.s._kde_until, self.s._typed_until), (0, 0))
+        self.assertTrue(cancel.is_set())
+
+    def test_switch_cancels_delayed_and_retrying_text_before_new_input(self):
+        for delaying in (True, False):
+            with self.subTest(delaying=delaying):
+                waiting, cancel = threading.Event(), threading.Event()
+                class ObservedCancel:
+                    def wait(self, timeout):
+                        waiting.set()
+                        return cancel.wait(timeout)
+                    def set(self):
+                        cancel.set()
+                self.s.reset_pad_route()
+                self.s._kde_cancel = ObservedCancel()
+                self.s._typed_until = time.time() + 60 if delaying else 0
+                self.s._input.reset_mock()
+                self.s._input.send.return_value = {"state": "starting", "sent": False}
+                self.s.remote_input({"events": [{"key": "é"}]})
+                worker = self.s._kde_worker[0]
+                self.assertTrue(waiting.wait(1), "worker never reached its cancellable delay")
+                self.switch()
+                worker.join(1)
+                self.assertFalse(worker.is_alive())
+                attempts = self.s._input.send.call_count
+                self.assertEqual(attempts, 0 if delaying else 1)
+                self.s._input.send.return_value = {"state": "ready", "sent": True}
+                self.s.remote_input({"events": [{"key": "ü"}]})
+                next_worker = self.s._kde_worker[0]
+                if next_worker:
+                    next_worker.join(1)
+                    self.assertFalse(next_worker.is_alive())
+                self.assertEqual(self.s._input.send.call_count, attempts + 1)
+                self.assertEqual(self.s._input.send.call_args.args[0], [{"key": "ü"}])
+                self.assertEqual(self.s._kde_queue, [])

tests/test_touch.py

@@ -150,6 +150,14 @@ class Apply(unittest.TestCase):
         self.assertFalse(self.t.STALE[0])
         self.assertEqual(self.said[-1][1].get("stale"), None)
 
+    def test_stale_survives_a_bare_release(self):
+        gs = FakeGamescope()
+        panel = self.t.apply(gs, {"button": "left", "down": True, "window": 99, "display": ":1"}, None)
+        self.assertTrue(self.t.STALE[0])
+        self.t.apply(gs, {"button": "left", "down": False}, panel)  # the page's release names no panel
+        self.assertTrue(self.t.STALE[0])
+        self.assertEqual(self.said[-1][1].get("stale"), True)
+
     def test_same_window_id_on_the_other_display_is_another_panel(self):
         gs = FakeGamescope()
         self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":0"}, None)

ui/frame_touch.py

@@ -506,8 +506,10 @@ def apply(gs, event, panel):
         say("ready", focus=panel.get("window"), display=panel.get("display"), stale=True)  # the page re-syncs
         STALE[0] = True
         return panel
-    if STALE[0] and not stale:
+    release = event.get("down") is False and ("button" in event or "key" in event)
+    if STALE[0] and not stale and not (release and "window" not in event):
         # Anything that goes through (a trackpad move names no panel) means caught up: stop re-syncing.
+        # A bare release doesn't: the page leaves the panel off releases, so it says nothing about focus.
         STALE[0] = False
         say("ready", focus=(panel or {}).get("window"), display=(panel or {}).get("display"))
     if "fx" in event and panel and panel.get("window"):

ui/index.html

@@ -932,10 +932,9 @@
         <button data-pad-key="12" title="Enter">Enter</button>
       </span>
     </div>
-    <div class="hint">Types and points in apps on the Frame, such as Chromium or the desktop's Linux apps; Steam's own VR menus
-      don't take it. It works through KDE Connect, which comes with Frame Control: the first time, it copies Valve's build
-      for the Frame (3.6 MB, 18 MB unpacked) into your home folder there and pairs with it. No internet needed, and nothing
-      else to install. <a href="#" data-about>Licences</a></div>
+    <div class="hint">Types and points in whichever panel has focus in the headset, through Valve's own input path: nothing to
+      install and no internet needed. Accents and emoji go through KDE Connect, which comes with Frame Control and starts
+      the first time you type one (it copies Valve's build for the Frame, 3.6 MB, into your home folder there). <a href="#" data-about>Licences</a></div>
   </section>
 
   <section id="shots">
@@ -4323,10 +4322,10 @@ const typesText = el => el?.matches?.('textarea, input:not([type=range], [type=c
 document.addEventListener("focusin", e => document.body.classList.toggle("typing", !!typesText(e.target)));
 document.addEventListener("focusout", () => document.body.classList.remove("typing"));
 
-// ---- keyboard and trackpad: events go to KDE Connect on the Frame (see ui/frame_input_agent.py) ----
+// ---- keyboard and trackpad: events go to the Frame through gamescope (ui/frame_touch.py); accents and emoji through KDE Connect ----
 const pad = { state: "off", queue: [], sending: false, poll: null, captured: false };
 const PAD_STATES = {
-  off: "", starting: "Connecting…", installing: "Setting up KDE Connect on the Frame (first time only)…",
+  off: "", starting: "Connecting…", installing: "Setting up (first time only)…",
   pairing: "Pairing with KDE Connect…", ready: "On",
 };
 // KDE Connect's specialKey numbers (plugins/mousepad in its source).
@@ -4344,7 +4343,7 @@ function padShow(status) {
   $("padOn").hidden = ready || busy;
   $("padOn").textContent = pad.state === "error" ? "Try again" : "Turn on";
   $("padArea").classList.toggle("off", !ready);
-  $("padHint").textContent = pad.state === "error" ? status.message || "Couldn't reach KDE Connect on the Frame."
+  $("padHint").textContent = pad.state === "error" ? status.message || "Couldn't reach the Frame."
     : !ready ? (busy ? PAD_STATES[pad.state] : "Turn on to use this as a trackpad for the Frame.")
     : TOUCH ? "Drag to move the pointer · tap to click · two fingers to scroll · two-finger tap to right-click"
     : pad.captured ? "Your mouse and keyboard now control the Frame. Press Esc to stop."

ui/server.py

@@ -63,6 +63,7 @@ import frame_report  # noqa: E402
 import frame_store  # noqa: E402
 import frame_telemetry  # noqa: E402
 import frame_titles  # noqa: E402
+import frame_touch  # noqa: E402
 import frame_webinstall  # noqa: E402
 import frame_vr  # noqa: E402
 import frame_utilities  # noqa: E402
@@ -115,6 +116,9 @@ def route(alias, host_opts):
     with _route_lock:
         moved = alias != FRAME
         if moved:
+            reset_pad = globals().get("reset_pad_route")
+            if reset_pad:
+                reset_pad()  # cancel old input before any agent can start on the new route
             frame_catalog._env.clear()  # the SteamOS and Lepton builds reports record are per headset
         FRAME = frame_android.FRAME = alias
         HOST_OPTS = list(host_opts)
@@ -125,9 +129,6 @@ def route(alias, host_opts):
     mv = globals().get("macview")
     if mv:
         mv.retarget(alias, host_opts)  # its tunnel is its own ssh: it must follow the headset too
-    agent = globals().get("_input")
-    if moved and agent:
-        agent.stop()  # typing and pointing mustn't go on reaching the headset switched away from
 
 
 LINK = None  # the connector (frame_link.Link); None on the Frame itself
@@ -1153,12 +1154,192 @@ def licenses():
     return notices
 
 
+# KDE Connect's specialKey numbers -> Linux key codes (KEY_BACKSPACE, KEY_TAB, ...).
+PAD_KEYS = {1: 14, 2: 15, 4: 105, 5: 103, 6: 106, 7: 108, 8: 104, 9: 109, 10: 102, 11: 107, 12: 28, 13: 111, 14: 1,
+            **{21 + i: code for i, code in enumerate([59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 87, 88])}}
+PAD_MODS = (("ctrl", 29), ("alt", 56), ("shift", 42), ("super", 125))
+PAD_CLICKS = {"singleclick": "left", "rightclick": "right", "middleclick": "middle"}
+PAD_NOTCH = 15  # scroll units for one wheel notch
+
+
+def typed_in_kde(events):
+    """Whether these events must go the KDE Connect way: they hold an accent or emoji, or
+    earlier ones are queued, being sent or still being typed there (typing must stay in
+    order, so it can't be split between the two ways)."""
+    with _kde_lock:
+        waiting = bool(_kde_queue) or time.time() < _kde_until
+    return waiting or any(ch not in frame_touch.ASCII for e in events for ch in e.get("key", ""))
+
+
+def pad_events(events, kde=False):
+    """The trackpad's and key row's events as (gamescope events, KDE Connect events).
+
+    Gamescope's own input reaches every panel and needs nothing installed, but types only
+    US-keyboard characters. With kde, the keyboard goes to KDE Connect as it was sent, so
+    text stays in order; the pointer always goes through gamescope.
+    """
+    ascii_keys = frame_touch.ASCII
+    out, slow = [], []
+
+    def tap(code, mods):
+        down = [{"key": c, "down": True} for c in mods]
+        out.extend(down + [{"key": code, "down": True}, {"key": code, "down": False}]
+                   + [{"key": c, "down": False} for c in reversed(mods)])
+    for e in events:
+        mods = [code for name, code in PAD_MODS if e.get(name)]
+        if "dx" in e or "dy" in e:
+            if not e.get("scroll"):
+                out.append({k: e[k] for k in ("dx", "dy") if k in e})
+        if e.get("scroll"):
+            out.append({"scroll": [0, -(e.get("dy") or 0) * PAD_NOTCH]})
+        for flag, button in PAD_CLICKS.items():
+            if e.get(flag):
+                out += [{"button": button, "down": True}, {"button": button, "down": False}]
+        if e.get("doubleclick"):
+            out += [{"button": "left", "down": d} for d in (True, False, True, False)]
+        if e.get("singlehold"):
+            out.append({"button": "left", "down": True})
+        if e.get("singlerelease"):
+            out.append({"button": "left", "down": False})
+        if kde and ("key" in e or "specialKey" in e):
+            slow.append({k: e[k] for k in ("key", "specialKey", "ctrl", "alt", "shift", "super") if k in e})
+            continue
+        if "specialKey" in e and e["specialKey"] in PAD_KEYS:
+            tap(PAD_KEYS[e["specialKey"]], mods)
+        if "key" in e:
+            for ch in e["key"]:
+                if ch not in ascii_keys:
+                    continue  # only reached when kde is wrong; the caller checks typed_in_kde first
+                elif mods:
+                    code, shifted = ascii_keys[ch]
+                    tap(code, mods + ([42] if shifted and 42 not in mods else []))
+                else:
+                    if out and "text" in out[-1] and len(out[-1]["text"]) < 100:
+                        out[-1]["text"] += ch
+                    else:
+                        out.append({"text": ch})
+    return out, slow
+
+
 def remote_input(body):
-    """{"events": [...]} sends keyboard and pointer events; {} (or none yet) just starts the agent."""
+    """{"events": [...]} sends keyboard and pointer events; {} (or none yet) just starts the agent.
+
+    "sent" in the answer means the whole batch is taken (the page keeps it and tries again
+    if not), so the keyboard's share is queued only once that's so.
+    """
     events = body.get("events", [])
     if not isinstance(events, list) or len(events) > INPUT_BATCH_LIMIT:
         raise Failure(f"events must be a list of at most {INPUT_BATCH_LIMIT}", 400)
-    return _input.send([input_event(e) for e in events])
+    events = [input_event(e) for e in events]
+    touch, slow = pad_events(events, typed_in_kde(events))
+    status = _touch.send(touch)
+    if touch and not status.get("sent"):
+        return status
+    if touch:
+        _note_typed(touch)
+    if slow:
+        _queue_for_kde(slow)
+    return {**status, "sent": True}
+
+
+# Typing KDE Connect is to do: it starts the first time it's needed, so this waits for it.
+# Nothing on the Frame says when typed text has landed, so each way holds the other back
+# for as long as what it was given should take (a short margin on a time estimate).
+_kde_queue = []
+_kde_lock = threading.Lock()
+_pad_send_lock = threading.Lock()
+_kde_worker = [None]
+_kde_until = 0.0     # keys typed through KDE Connect should have landed by then
+_typed_until = 0.0   # when gamescope's agent will have typed everything it's been given
+_kde_route = object()
+_kde_cancel = threading.Event()
+KDE_QUEUE_LIMIT = 200
+KDE_WAIT = 90  # seconds to wait for KDE Connect before giving up on what's queued
+TYPING_SETTLE = 0.6  # seconds past the estimate, once
+
+
+def reset_pad_route():
+    """Discard input for the headset being left, including batches waiting for an agent."""
+    global _kde_route, _kde_cancel, _kde_until, _typed_until
+    with _pad_send_lock, _kde_lock:
+        _kde_cancel.set()
+        _kde_cancel = threading.Event()
+        _kde_route = object()
+        _kde_queue.clear()
+        _kde_worker[0] = None
+        _kde_until = _typed_until = 0.0
+        _input.stop()
+        _touch.stop()
+
+
+def typing_seconds(touch):
+    """How long gamescope's agent takes over these: it sleeps 8 ms after every key transition."""
+    transitions = 0
+    for e in touch:
+        if "text" in e:
+            transitions += sum(2 + 2 * frame_touch.ASCII[ch][1] for ch in e["text"] if ch in frame_touch.ASCII)
+        elif "key" in e:
+            transitions += 1
+    return transitions * 0.008
+
+
+def _note_typed(touch):
+    global _typed_until
+    seconds = typing_seconds(touch)
+    if seconds:
+        with _kde_lock:  # the end of the work, which queues up; the margin is added once, where it's checked
+            _typed_until = max(_typed_until, time.time()) + seconds
+
+
+def _queue_for_kde(events):
+    with _kde_lock:
+        _kde_queue.extend(events)
+        del _kde_queue[:-KDE_QUEUE_LIMIT]
+        if _kde_worker[0] is None:
+            _kde_worker[0] = threading.Thread(target=_drain_kde, args=(_kde_route, _kde_cancel), daemon=True)
+            _kde_worker[0].start()
+
+
+def _drain_kde(route, cancel):
+    """The only sender, in order. A batch leaves the queue while it's being sent (the queue
+    can be trimmed meanwhile) and goes back to the front if it didn't go."""
+    global _kde_until
+    deadline = time.time() + KDE_WAIT
+    while True:
+        with _kde_lock:
+            if route is not _kde_route:
+                return
+            batch = _kde_queue[:INPUT_BATCH_LIMIT]
+            del _kde_queue[:len(batch)]
+            if not batch:
+                _kde_worker[0] = None  # retired under the lock, so a new request starts another
+                return
+            wait = _typed_until + TYPING_SETTLE - time.time()
+            _kde_until = time.time() + 3600  # held while in flight
+        if wait > 0 and cancel.wait(wait):
+            return  # a switch cancels the delay and its old-headset batch
+        # Serialize sending with route resets without blocking callers from queuing input.
+        with _pad_send_lock:
+            with _kde_lock:
+                if route is not _kde_route:
+                    return
+            status = _input.send(batch)
+        with _kde_lock:
+            if route is not _kde_route:
+                return
+            if status.get("sent"):
+                _kde_until = time.time() + len(batch) * 0.03 + TYPING_SETTLE
+                deadline = time.time() + KDE_WAIT
+                continue
+            _kde_until = 0.0
+            if status.get("state") == "error" or time.time() > deadline:
+                _kde_queue.clear()  # not coming; don't hold it forever
+                _kde_worker[0] = None
+                return
+            _kde_queue[:0] = batch
+            del _kde_queue[:-KDE_QUEUE_LIMIT]  # as when queuing: the oldest go first
+        if cancel.wait(0.4):
+            return
 
 
 # ---- touch: the headset's panels, through gamescope's own input (frame_touch.py) ----
@@ -2708,7 +2889,7 @@ class Handler(BaseHTTPRequestHandler):
             elif path == "/api/touch":
                 self.send_json(_touch.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_touch.status))
             elif path == "/api/input":
-                self.send_json(_input.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_input.status))
+                self.send_json(_touch.send([]) if parse_qs(url.query).get("start") == ["1"] else dict(_touch.status))
             elif path == "/api/job":
                 self.send_json(job_status(url.query))
             elif path == "/api/queue":