Compare commits

..
Author SHA1 Message Date
SAPHID 0c99722073 Document current Frame panel layout API limits 2026-10-24 10:09:49 +00:00
22 changed files with 59 additions and 1787 deletions

No files matched your search

-20
View File
@@ -2,26 +2,6 @@
## Unreleased
- Use gamescope's input for the keyboard and trackpad, following the focused
headset panel without installing an input service. Start KDE Connect only for
accents and emoji, keeping queued text ordered and cancelling it on headset
switches. Retain intended-headset checks before starting either control and
prevent Unicode helper startup from deadlocking with a headset switch.
- Keep accepted keyboard text when the Unicode queue is full: reject new batches
before sending their clicks or keys, and count in-flight text toward the limit.
Retain unsent text after helper failures/timeouts, report the error and resume
it with Try again. Clean up failed workers without touching a new headset's queue.
[Current verification and device limits](docs/evidence/pad-gamescope-2026-10-24.md).
- Keep media playback running when SteamVR refuses the optional theatre dark
surround, including a retry after standby. Log the refusal and play without
the surround. Overlay teardown attempts every owned handle and always calls
OpenVR shutdown even if a handle fails to close.
- Check the desktop server's single-instance lock before starting background
downloads, so a second server exits cleanly with the existing "already
running" message.
- Recheck saved spatial layouts on SteamOS 0.5.6 / SteamVR 2.18.2 (issue #31).
Read-only probes expose some panel geometry, but safe restoration of
Steam/gamescope docking and positions remains unverified. Record exact API
+2 -2
View File
@@ -75,8 +75,8 @@ Drag files onto the window to send them. Drop a game's .zip, folder or .exe to a
**📸 Screenshots and recordings**<br>
Browse the shots you take in the headset and save them to your Pictures folder, or have new ones and your Steam clips copied over automatically. Record what the headset shows, with its sound, or export Steam's clips at any length, as .mp4 files in your Videos folder ([not tested on a headset yet](docs/recording.md)).
**⌨️ Keyboard and trackpad**<br>
Type and point in the headset from your computer or phone, through Valve's own input path, with nothing to install. Accents and emoji use KDE Connect, which Frame Control brings along and sets up the first time you type one. Copy text to the headset's clipboard and back again. Press T to open it.
**⌨️ Type on the headset**<br>
Type and point in the Frame's apps from your computer or phone, through KDE Connect, which Frame Control brings along and sets up on the Frame. Copy text to the headset's clipboard and back again. Nothing else to install, anywhere. Press T to open it.
</td>
</tr>
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "frame-control",
"version": "0.4.18",
"version": "0.4.17",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "frame-control",
"version": "0.4.18",
"version": "0.4.17",
"license": "MIT",
"devDependencies": {
"electron": "^44.4.5",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "frame-control",
"productName": "Frame Control",
"version": "0.4.18",
"version": "0.4.17",
"description": "Desktop app for managing a Valve Steam Frame over SSH",
"private": true,
"main": "main.js",
-142
View File
@@ -1,142 +0,0 @@
# Media surround hardening: 2026-10-24
PR [#51](https://github.com/saphid/frame-control/pull/51), branch
`theatre-media-surround-hardening`, merged main `2ded4c5` without rebasing.
Both fixes remain necessary on that main: an optional-surround error escapes
`play()`, and a non-`RuntimeError` during overlay teardown skips the remaining
handles and OpenVR shutdown.
## Offline checks
- Focused media suite: **23 tests passed** on Linux / Python 3.12.3.
- Final full suite: **1,381 tests, OK (15 skipped)** in 145.384 s with
`PYTHONFAULTHANDLER=1 python3 -m unittest discover -s tests -v`.
The service-regression module passed **33 tests**. Player/server compilation
and `git diff --check` passed. Current macOS/Windows validation is left to CI.
- The startup/upload, creation/setup, deferred-surround refusal and teardown
regression tests were also run with main's player implementation: they
failed as expected (six errors across subtests). The screen-refusal test
passed, confirming that actual screen failures still propagate.
- The historical macOS failure was job
[109239733717](https://github.com/saphid/frame-control/actions/runs/36516462623/job/109239733717),
on `dac9dbc`, Python 3.12: 619 tests, three failures in artwork resolver
tests (`test_resolver_slot_released_when_thread_cannot_start`,
`test_timed_out_lookups_are_capped`,
`test_url_fetch_refuses_private_hosts_and_honours_deadline`). Media tests
passed. Late DNS workers retained semaphore slots; the first failed
assertion could also leak acquired slots into later tests.
- Main already contains `8051687`'s wait-for-idle and unconditional slot
cleanup, preserved by `d546568`. The current artwork module passed all
**40 tests** locally. No additional artwork changes were needed.
- The first full run completed **1,380 tests, 15 skipped**, with one failure:
`test_a_second_server_is_refused_until_the_first_exits` observed SIGSEGV
(`-11`) instead of exit 1. All 41 server tests passed on a focused rerun.
Main started daemon index/TLS workers before refusing a second server;
its existing shutdown notes document a race with OpenSSL interpreter exit.
This crash mechanism is inferred; the first run did not retain a native
fault trace. The subprocess assertion now retains stderr for diagnosis.
A deterministic regression now checks that rejection starts no listener,
cleanup sweep or background workers. It failed before moving the ownership
check ahead of startup work and passed afterwards.
The surround retry now logs a refusal before discarding that cosmetic work.
Creation, setup and first-upload failures also allow the screen to play;
standby remains retryable, and screen errors remain fatal. Teardown tests
exercise OpenVR errors, unexpected exceptions and a late `InterruptedError`,
checking that all handles are attempted and shutdown runs.
## Earlier blocked real Frame attempt
Read-only SSH preflight: SteamOS **0.5.6**, BUILD_ID **20261015.6141207**;
SteamVR server PID **297506**, compositor PID **297620**; existing media
service `inactive/dead`, MainPID 0; battery **99%**, fuel gauge `Not charging`,
charger `Full`. The shared lock was absent at preflight.
The subsequent atomic `mkdir /tmp/frame-test.lock` failed with
`FileExistsError(17, 'File exists')`: another session acquired it in between.
The probe stopped before uploading files, launching media, creating overlays,
or changing any state. The foreign lock was neither modified nor released.
All generated fixtures remained on the computer and were then removed; no
headset cleanup was necessary. Steam/SteamVR and the prior player state were
left alone.
## Follow-up real Frame smoke at the exact PR head
**Passed, 2026-10-24 10:35:02–10:35:23 UTC**, with byte-identical media modules
from **`5d551a81658d9ff14f49285c28609d506cbc20aa`**. This later run resolves
the earlier lack of fresh playback evidence. No production code changed.
The headset was unworn. SteamOS **0.5.6**, BUILD_ID **20261015.6141207**,
SteamVR **2.18.2**; battery **100%, Full** before and after.
Read-only preflight found Game Mode (`XDG_CURRENT_DESKTOP=gamescope`,
`start-gamescope` running, no `plasmashell`), Steam/SteamVR running, no media
overlays, and the installed media service inactive/not loaded. The test then
acquired `/tmp/frame-test.lock` with atomic `mkdir`; it did not take over a
foreign lock. Under the lock, those safety conditions were checked again.
Only generated media and the PR's four unchanged modules were uploaded, under
the test-owned, uid-1000 directory
`/tmp/pr51-5d551a8-4c0976a52564-utfbk_m0/`. The clip was a **12 s, 30 fps,
960×540 half-SBS H.264** MP4 with solid red/cyan eye images and **silent AAC**.
It was generated with FFmpeg's `color`, `hstack` and `anullsrc` sources; no
personal or external media was used.
The exact `frame_media_remote.run()` Play/status/Stop path was exercised by
importing its unmodified module and changing only its in-memory `ROOT`,
`RUNTIME`, `STATUS` and `UNIT` constants to the isolated test locations and
`frame-control-media-pr51-5d551a8-4c0976a52564.service`. The real
`frame_media_player.py` ran unchanged with `--layout auto --theatre`, using
`h264_v4l2m2m` and Pulse output. This checks the player and its systemd lifecycle;
it does **not** reverify the HTTP/UI upload or runtime installer, nor the
literal production unit name. The installed media runtime, log, status and
library were preserved.
- First Play: `starting` → `playing` → **`ended`**, **360 frames**, **160 dropped**,
**12.0397 s**, `running=false`. The drops were the normal unworn/standby
`SetOverlayRaw` retry path; no fault was forced. At 60 frames, dropped was 0.
`vrcmd --overlays` listed `framecontrol.media.surround` as **1×1 visible**
and `framecontrol.media.screen` as **visible**. These are SteamVR metadata,
not a wearer or eye-capture verification.
- After EOF: unit **inactive/dead, not-found, MainPID 0**, no test player or
decoder, and neither owned overlay remained. Stop after EOF succeeded and
retained `ended`.
- Second Play reached `playing`, 60 frames, 0 dropped. Explicit Stop returned
**`stopped`, 72 frames, 0 dropped, `running=false`**. Again the unit was
inactive/dead/not-found with MainPID 0, no player/decoder and no owned overlays.
- The test unit's `ExecStart` and cgroup identified the unique temporary
runtime. First-run player/decoder PIDs were **309650/309655**; second-run
PIDs were **309805/309810**. Stop controlled only this dedicated test unit.
Its player log was empty; no surround refusal or teardown error was logged.
### Uploaded SHA-256 values
```text
generated_SBS.mp4 1b39cac1d3b092b89657250c817019b68e87a474976ccd44aeb38e8eee44ffca
frame_media.py 001e3158f4c3931794846b21f1e4cb553349d9326b4db1454a01976d446ad97a
frame_media_player.py 86b0a140ebff857c2819c74d3ac45d1897735c585d44c7360ad6f18c2ecf2645
frame_media_remote.py b60386eb192d32594358246a4e665474c543600f2c564c2efd554ef38d7a631d
frame_splat.py f35750926f1149c41b95617a302012406d355c0dfdd4530438b0c42ee6d55daa
```
### Cleanup and limits
The temporary directory, including its fixture, code, bytecode, status and log,
was removed only after verifying that its player and decoder had exited. The
transient unit was collected. The test released only its own lock after cleanup.
All installed media runtime file hashes, including `player.log` and
`status.json`, were unchanged. The installed `frame-control-media.service`
remained inactive/not loaded. Steam **297826**, vrserver **297506**,
vrcompositor **297620**, vrdashboard **297721**, and start-gamescope **296997**
kept the same PIDs. The pre-existing overlay key/dimension/visibility/type
inventory matched the final inventory, and all dashboard main panels remained
`not_visible`. No dashboard command, wake request, global setting, reboot or
Steam/SteamVR stop was issued.
Cosmetic surround refusal and exceptional teardown remain **fake-tested**;
no real refusal or teardown fault was induced. This short unworn-headset
smoke does not verify appearance, eye isolation, comfort, audible sound/lip
sync (the AAC was silent), sustained display performance, or the issue #28
report of laggy 1080p video. It does not cover new H.265, photo or splat cases.
Historical checks remain in the earlier PR #51 description and
[media playback notes](../vr-video.md). The PR remains draft and unmerged
for parent review and CI.
-178
View File
@@ -1,178 +0,0 @@
# Keyboard and trackpad through gamescope, 2026-10-24
PR: [#66](https://github.com/saphid/frame-control/pull/66), `pad-gamescope`.
Parent/main: v0.4.18 commit `0da1fe5`. Merge commit: `1737366`, with parents
`cba23fb` (the existing PR) and `0da1fe5` (main). No history was rewritten.
## Merge and implementation
Both `/api/input` (keyboard/trackpad) and `/api/touch` (live-view Control)
start and poll the gamescope helper. KDE Connect starts only when Unicode
typing needs it. The existing ordered, bounded queue and switch cancellation
remain in place; cross-service typing order is an estimate, not an application
acknowledgement.
The merge keeps main's intended-device validation and work reservation before
a GET starts an input helper, immutable SSH routes, launch generations and
serialized writes. Switching headsets also stops the power report and resets
both input agents and queued typing before publishing the new route. The page
keeps main's Tools sections, clipboard controls and single instances of the
keyboard/trackpad and VR tools, with the PR's gamescope help text.
The start/status regression tests now exercise the gamescope helper for both
endpoints. An explicit `/api/input` test checks that start and poll use that
helper without starting KDE Connect. Existing stale/missing-headset and
route-reservation tests remain active.
## Real Frame: focused click and ASCII keys received
Host: lxso2, `ssh frame`. SteamOS **0.5.6**, BUILD_ID **20261015.6141207**,
SteamVR **2.18.2**. At **11:05 UTC**, Game Mode was active, full Plasma was
inactive, Steam and SteamVR were running, battery was **99%**, and gamescope's
focusable/focused window lists were empty. The shared test lock was absent.
Existing X11 tools and libX11 were available; Tk could not load its library.
At **11:09:23 UTC**, an initial read-only preflight saw no focused window
but activity level **0** (idle), rather than the required **3** (standby), and
aborted before lock acquisition or input. At **11:09:57 UTC**, another task
held the shared lock with a focused panel; that lock and panel were left alone.
At **11:20:22 UTC**, read-only checks found the lock absent, no focused or
focusable windows, activity **3** and battery **98%**. The bounded probe ran at
**11:20:37 UTC** against implementation commit `5641d3c`:
1. Rechecked empty focus and standby, acquired `/tmp/frame-test.lock` with
atomic `mkdir`, wrote a unique owner marker and rechecked both conditions.
2. Created only a disposable 640×400 X11 event-recorder window on `:0`, tagged
with test app ID `2000999066`. It had a 45-second alarm and closed on stdin
EOF. No existing application, full desktop or dashboard was started.
3. Waited for gamescope to report that exact window (`62914561`) focused.
Started the production server's `TouchAgent` over SSH; it reported ready.
4. Called production `server.remote_touch` for positioning, then
`server.remote_input` with a left click, `Frame66` and the Enter key button.
Both responses reported `sent: true`. More importantly, the **window itself
recorded ButtonPress/Release (button 1), the ASCII key events in order and
Return**. `XLookupString` decoded exactly `Frame66\r`. The events were
delivered through gamescope's EIS socket, not fabricated with XSendEvent.
5. Closed the helper by EOF and waited for exit, closed the recorder, checked
empty focus and standby, then removed only the owner-matching test lock.
[State and received events](pad-gamescope-2026-10-24/readback.json) and the
[probe](pad-gamescope-2026-10-24/probe.py)/[X11 recorder](pad-gamescope-2026-10-24/recorder.py)
are retained. The archived scripts use checkout-relative source paths; the
recorded run used the same harness from `/tmp`. Motion records omit an unused
field that only has meaning for key/button events.
Delayed readback at **11:21:46 UTC** confirmed no test lock or recorder process,
empty focused/focusable lists, Game Mode active, full Plasma inactive, activity
3 and battery 98%. The original Steam and vrserver PIDs were unchanged. No
install, security/global setting change, dashboard/session switch, reset or
reboot was performed. No headset helper or owned lock remained.
This establishes **focused-panel click and ASCII key delivery on `:0`**.
It does not establish wearer-visible presentation, Unicode delivery or GUI
interaction. The absolute-position setup reached the recorder's edge
(639,399), rather than its centre; pixel-accurate absolute pointing is **not**
proved by this small-window probe. The trackpad click and typed keys still
arrived in the owned focused window. The original PR's earlier acceptance-only
observations remain historical; this observation adds actual window events.
## Offline checks and remaining limits
- `python3 -m unittest discover -s tests -p test_input.py -v`: **58 passed**.
- `python3 -m unittest discover -s tests -p test_touch.py -v`: **43 passed**.
- `python3 -m unittest discover -s tests -p test_service_regressions.py -v`:
**34 passed**.
- `python3 -m unittest discover -s tests -v`: **1,403 ran, OK, 15 skipped**
(Linux x86_64, Python 3.12; 150.825 seconds).
- Full fake-Frame e2e in task-specific Docker project `frame-control-pr66`:
**47 passed** after the lock fix (192.270 seconds). Both task containers,
their network and all three volumes were removed afterward. The run emitted
an unclosed test-socket ResourceWarning; the suite returned OK.
- Node page suites: `ctrl_gestures.mjs`, `pad_queue.mjs`, `desk_panels.mjs`
passed, using the production page functions.
- `node --test app/test/*.test.js`: **3 test files passed**.
- `node --test site/test/*.test.mjs`: **2 test files passed**.
- Python compilation, desktop/site JavaScript syntax and `git diff --check`
passed. Fake-Frame e2e and exact-head CI results are recorded in the PR after completion.
The first full regression run executed 1,401 tests and found two tests still
expecting `/api/input` to start the old KDE Connect helper. They were updated
to the intended gamescope route. The final full run includes the lazy-KDE
regression above and the lock-order regression below. The initial restricted-sandbox input run could not open
loopback sockets; the focused rerun with managed network access passed.
Review of the merged code then found a lock inversion: the Unicode worker
held the pad-send lock while `InputAgent.start()` acquired the route lock,
whereas a headset switch held the route lock before resetting the pad. A
bounded subprocess regression reproduced the deadlock with the real
`InputAgent.send/start` methods and an offline launch stub. Unicode delivery
and resets now take the route lock before the pad lock; the route lock is
reentrant so an agent can capture its route inside that reservation. The new
regression passed after the fix, as did all **58 focused input tests** and
the **1,403-test full suite** (15 skipped). The local fake-Frame suite also
passed **47 tests after this fix**. Exact-head CI results are recorded in the PR.
Remaining device work requires an idle, available Frame and the shared lock:
verify Unicode in a real text field, ordering under real typing load and
precise absolute pointing for small panels. Worn-headset presentation,
packaged Electron pointer lock and Windows GUI execution remain separate
checks. Automated Windows/macOS CI does not establish those interactions.
## Review round 2: queue saturation and failure recovery
The coordinator identified silent eviction in both `_queue_for_kde()` and the
worker's retry path. Both sliced the oldest accepted events off at 200 entries,
even though the page had already received `sent: true` and discarded its copy.
Admission now serializes concurrent requests, checks capacity before any pointer
or keyboard send, and returns `sent: false, state: waiting` if the whole batch
will not fit. Text stays in the bounded queue while in flight, and is removed
only after the helper accepts it. Rejected mixed batches send no clicks; retrying
them cannot duplicate clicks already sent by this server. The page polls until
it can retry. Pointer input stays usable while accepted text is pending.
The old worker also left its worker marker and one-hour timing sentinel behind
if `send()` raised. Failure/timeout handling now retains unsent text, clears the
worker/timing state and exposes an error through input status. The page polls
accepted text without requiring another keypress. **Try again** explicitly
restarts the retained queue. Thread-start failures follow the same retention
rule. A stale worker's exception cannot change the new route's state.
The proposed send-to-enqueue headset-switch race was not reachable through the
normal HTTP switch path: `Handler.do_POST()` reserves `working(meant)` across
both operations; `devices_post()` shares `_work_lock` and refuses a moving
operation while work is active. Connector configuration/reconnect paths defer
changed headsets while work runs. A production-handler regression attempts a
switch at both send and enqueue and passes even on the previous implementation.
No separate connector change was needed. Atomic queue admission now uses the
existing route-before-pad lock order as well, because starting either input
agent can capture its route synchronously.
Three deterministic saturation regressions fail on `215db7b` and pass with the
fix: capacity during the typing delay, capacity during an unsuccessful-send
retry, and two concurrent producers competing for the last slots. They check
exact accepted event sequences, reject a mixed click/text batch before its
pointer effects, and verify delivery exactly once after retry. Additional tests
cover explicit recovery after timeout, helper errors, send exceptions and
thread-start failure, stale-worker isolation, HTTP status/start behavior, and
the production page's polling/error behavior.
No new active headset test was performed: this change affects host queue
admission/recovery and status handling, not Frame event translation or injection.
The earlier real click/ASCII evidence remains scoped to its recorded commit.
Real Unicode, packaged Electron pointer lock, Windows GUI, wearer-visible
presentation and precise small-window absolute positioning remain unverified.
`sent: true` means accepted by this server, not application delivery. The queue
is in memory; switching headsets cancels it and server exit loses it. A transport
failure after a partial write can still make delivery ambiguous; this is not an
end-to-end exactly-once protocol.
Round 2 local verification: **141 focused input/touch/service tests passed**
(62 input, 43 touch, 36 service; 20.061 seconds), and **1,409 server tests ran, OK, 15 skipped**
(Linux x86_64, Python 3.12, 153.198 seconds). All three production-page Node
suites passed; app tests passed (3 files), site tests passed (2 files), Python
compilation and `git diff --check` passed. The existing skipped tests require
zsh, macOS/Swift or built native PC libraries. Current-head CI is tracked in
PR #66; the previous 47-test local fake-Frame run above is historical.
@@ -1,129 +0,0 @@
"""Active real-Frame evidence probe: aborts unless idle, then owns the shared lock.
Run manually on an authorised host: python3 docs/evidence/pad-gamescope-2026-10-24/probe.py
The recorded run used /tmp paths; archived paths are relative to this checkout.
"""
import base64
import datetime
import json
import os
from pathlib import Path
import queue
import shlex
import subprocess
import sys
import tempfile
import threading
import time
import uuid
ROOT = Path(__file__).resolve().parents[3]
data = tempfile.TemporaryDirectory(prefix='pr66-live-')
os.environ.update(FRAME_CONTROL_DATA_DIR=data.name, FRAME_CONTROL_TELEMETRY='0', FRAME_PRIVATE_SSH='1')
sys.path.insert(0, str(ROOT / 'ui'))
import server
SSH = ['ssh', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=5', 'frame']
owner = 'pr66-' + uuid.uuid4().hex
result = {'time': datetime.datetime.now(datetime.timezone.utc).isoformat(), 'owner': owner,
'base': '0da1fe5', 'route': 'production server.remote_input -> TouchAgent SSH -> frame_touch.py -> gamescope EIS',
'events': []}
recorder = None
locked = False
rows = queue.Queue()
def ssh(cmd):
return subprocess.run([*SSH, cmd], capture_output=True, text=True, timeout=15, check=True).stdout
def focus():
return json.loads(server.ssh('python3 - focus', stdin=(ROOT/'ui/frame_touch.py').read_text(), timeout=15))
def activity():
return json.loads(ssh('/opt/steamvr/bin/linuxarm64/vrcmd --stats'))[0]['activity_level']
def read_rows():
for raw in recorder.stdout:
rows.put(json.loads(raw))
def collect(seconds):
end = time.monotonic() + seconds
while time.monotonic() < end:
try:
result['events'].append(rows.get(timeout=max(0.01, end-time.monotonic())))
except queue.Empty:
break
try:
result['initial_focus'] = focus()
result['initial_activity'] = activity()
assert not result['initial_focus'].get('window'), 'existing focused panel; do not interrupt'
assert result['initial_activity'] == 3, 'headset is not in standby'
ssh('mkdir /tmp/frame-test.lock && printf %s ' + shlex.quote(owner) + ' > /tmp/frame-test.lock/owner')
locked = True
result['lock_acquired'] = True
assert not focus().get('window') and activity() == 3, 'session changed during lock acquisition'
code = base64.b64encode(Path(__file__).with_name('recorder.py').read_bytes()).decode()
command = 'python3 -u -c ' + shlex.quote("import base64;exec(compile(base64.b64decode('" + code + "'),'pr66-recorder','exec'))")
recorder = subprocess.Popen([*SSH, command], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, text=True)
threading.Thread(target=read_rows, daemon=True).start()
initial = rows.get(timeout=8)
result['recorder'] = initial
assert initial.get('ready'), initial
window = initial['window']
for _ in range(20):
current = focus()
if current.get('window') == window:
break
time.sleep(0.2)
result['focused_panel'] = current
assert current.get('window') == window and current.get('display') == ':0', 'owned panel did not obtain focus; no input sent'
server.remote_input({})
for _ in range(50):
if server._touch.status.get('state') in ('ready', 'error'):
break
time.sleep(0.1)
result['agent_status'] = dict(server._touch.status)
assert result['agent_status'].get('state') == 'ready', result['agent_status']
target = {'window': window, 'display': ':0'}
assert focus().get('window') == window and activity() == 3
result['position_response'] = server.remote_touch({'events': [dict(target, fx=0.5, fy=0.5)]})
time.sleep(0.2)
assert focus().get('window') == window and activity() == 3
result['pad_response'] = server.remote_input({'events': [{'singleclick': True}, {'key': 'Frame66'}, {'specialKey': 12}]})
collect(2)
received = result['events']
result['received_text'] = ''.join(e.get('text', '') for e in received if e.get('type') == 2)
result['click_received'] = any(e.get('type') == 4 and e.get('code') == 1 and e.get('window') == window for e in received)
result['release_received'] = any(e.get('type') == 5 and e.get('code') == 1 and e.get('window') == window for e in received)
result['passed'] = result['click_received'] and result['release_received'] and result['received_text'] == 'Frame66\r'
assert result['passed'], 'event recorder did not observe expected click and keys'
except Exception as e:
result['error'] = str(e)
finally:
proc = server._touch.proc
server.reset_pad_route()
if proc:
try:
proc.wait(timeout=8)
result['input_helper_exited'] = True
except subprocess.TimeoutExpired:
proc.terminate()
proc.wait(timeout=5)
result['input_helper_terminated'] = True
if recorder:
recorder.stdin.close()
recorder.wait(timeout=8)
collect(0.2)
result['recorder_exit'] = recorder.returncode
result['recorder_stderr'] = recorder.stderr.read()[-1000:]
if locked:
result['final_focus'] = focus()
result['final_activity'] = activity()
ssh('test "$(cat /tmp/frame-test.lock/owner)" = ' + shlex.quote(owner)
+ ' && rm /tmp/frame-test.lock/owner && rmdir /tmp/frame-test.lock')
result['owned_lock_released'] = True
Path('/tmp/pr66-live.json').write_text(json.dumps(result, indent=2) + '\n')
print(json.dumps(result, indent=2))
data.cleanup()
sys.exit(0 if result.get('passed') else 1)
@@ -1,280 +0,0 @@
{
"time": "2026-10-24T11:20:37.392607+00:00",
"owner": "pr66-7c2a8e8851f449bcb0bb6a040be94d99",
"base": "0da1fe5",
"route": "production server.remote_input -> TouchAgent SSH -> frame_touch.py -> gamescope EIS",
"events": [
{
"type": 6,
"window": 62914561,
"x": 320,
"y": 200,
"synthetic": false
},
{
"type": 6,
"window": 62914561,
"x": 639,
"y": 399,
"synthetic": false
},
{
"type": 4,
"window": 62914561,
"x": 639,
"y": 399,
"code": 1,
"synthetic": false
},
{
"type": 5,
"window": 62914561,
"x": 639,
"y": 399,
"code": 1,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 50,
"synthetic": false,
"text": "",
"keysym": 65505
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 41,
"synthetic": false,
"text": "F",
"keysym": 70
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 41,
"synthetic": false
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 50,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 27,
"synthetic": false,
"text": "r",
"keysym": 114
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 27,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 38,
"synthetic": false,
"text": "a",
"keysym": 97
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 38,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 58,
"synthetic": false,
"text": "m",
"keysym": 109
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 58,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 26,
"synthetic": false,
"text": "e",
"keysym": 101
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 26,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 15,
"synthetic": false,
"text": "6",
"keysym": 54
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 15,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 15,
"synthetic": false,
"text": "6",
"keysym": 54
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 15,
"synthetic": false
},
{
"type": 2,
"window": 62914561,
"x": 639,
"y": 399,
"code": 36,
"synthetic": false,
"text": "\r",
"keysym": 65293
},
{
"type": 3,
"window": 62914561,
"x": 639,
"y": 399,
"code": 36,
"synthetic": false
},
{
"cleaned": true
}
],
"initial_focus": {
"window": null
},
"initial_activity": 3,
"lock_acquired": true,
"recorder": {
"ready": true,
"window": 62914561,
"pid": 317210,
"display": ":0",
"appid": 2000999066
},
"focused_panel": {
"window": 62914561,
"display": ":0",
"name": "Frame Control PR66 disposable input probe",
"x": 0,
"y": 0,
"width": 640,
"height": 400,
"root": [
1920,
1080
],
"app": 2000999066
},
"agent_status": {
"state": "ready",
"focus": 62914561
},
"position_response": {
"state": "ready",
"focus": 62914561,
"sent": true
},
"pad_response": {
"state": "ready",
"focus": 62914561,
"sent": true
},
"received_text": "Frame66\r",
"click_received": true,
"release_received": true,
"passed": true,
"input_helper_exited": true,
"recorder_exit": 0,
"recorder_stderr": "",
"final_focus": {
"window": null
},
"final_activity": 3,
"owned_lock_released": true,
"implementation_commit": "5641d3c316dfe225321f7233840cc83ff8c3f527",
"device": {
"steamos": "0.5.6",
"build_id": "20261015.6141207",
"steamvr": "2.18.2",
"battery_before": 98,
"battery_after": 98
},
"delayed_cleanup": {
"time": "2026-10-24T11:21:46Z",
"shared_lock_absent": true,
"owned_recorder_absent": true,
"game_mode": "active",
"full_plasma": "inactive",
"steam_pid": 297826,
"vrserver_pid": 297506,
"focusable_windows": [],
"focused_windows": [],
"activity_level": 3
},
"limits": [
"No wearer or packaged Electron/GUI interaction",
"ASCII only; Unicode and cross-service ordering not tested on device",
"The positioning setup reached the recorder edge (639,399); pixel-accurate absolute targeting is not established by this probe"
]
}
@@ -1,82 +0,0 @@
"""Disposable X11 event recorder used for the PR #66 focused-input observation."""
import ctypes as C
import json
import os
import select
import signal
import time
X = C.CDLL('libX11.so.6')
P, U = C.c_void_p, C.c_ulong
for name, restype, args in [
('XOpenDisplay', P, [C.c_char_p]), ('XDefaultRootWindow', U, [P]),
('XCreateSimpleWindow', U, [P, U, C.c_int, C.c_int, C.c_uint, C.c_uint, C.c_uint, U, U]),
('XInternAtom', U, [P, C.c_char_p, C.c_int]),
('XChangeProperty', C.c_int, [P, U, U, U, C.c_int, C.c_int, P, C.c_int]),
('XStoreName', C.c_int, [P, U, C.c_char_p]),
('XSelectInput', C.c_int, [P, U, C.c_long]), ('XMapWindow', C.c_int, [P, U]),
('XFlush', C.c_int, [P]), ('XPending', C.c_int, [P]),
('XConnectionNumber', C.c_int, [P]), ('XNextEvent', C.c_int, [P, P]),
('XLookupString', C.c_int, [P, P, C.c_int, P, P]),
('XDestroyWindow', C.c_int, [P, U]), ('XCloseDisplay', C.c_int, [P]),
]:
fn = getattr(X, name)
fn.restype, fn.argtypes = restype, args
class InputEvent(C.Structure):
_fields_ = [('type', C.c_int), ('serial', U), ('send_event', C.c_int),
('display', P), ('window', U), ('root', U), ('subwindow', U),
('time', U), ('x', C.c_int), ('y', C.c_int), ('x_root', C.c_int),
('y_root', C.c_int), ('state', C.c_uint), ('code', C.c_uint),
('same_screen', C.c_int)]
class Event(C.Union):
_fields_ = [('type', C.c_int), ('input', InputEvent), ('pad', C.c_long * 24)]
def say(**row):
print(json.dumps(row), flush=True)
def end(*_):
raise TimeoutError('probe ended')
signal.signal(signal.SIGTERM, end)
signal.signal(signal.SIGALRM, end)
signal.alarm(45)
d = X.XOpenDisplay(b':0')
assert d, 'X display unavailable'
w = 0
try:
w = X.XCreateSimpleWindow(d, X.XDefaultRootWindow(d), 0, 0, 640, 400, 0, 0, 0x224466)
for name, value in [(b'STEAM_GAME', 2000999066), (b'_NET_WM_PID', os.getpid())]:
val = U(value)
X.XChangeProperty(d, w, X.XInternAtom(d, name, 0), 6, 32, 0, C.byref(val), 1)
X.XStoreName(d, w, b'Frame Control PR66 disposable input probe')
X.XSelectInput(d, w, 1 | 2 | 4 | 8 | 64 | 32768 | 2097152)
X.XMapWindow(d, w)
X.XFlush(d)
say(ready=True, window=w, pid=os.getpid(), display=':0', appid=2000999066)
while True:
while X.XPending(d):
event = Event()
X.XNextEvent(d, C.byref(event))
if event.type in (2, 3, 4, 5, 6):
e = event.input
row = dict(type=event.type, window=e.window, x=e.x, y=e.y,
synthetic=bool(e.send_event))
if event.type != 6:
row['code'] = e.code
if event.type == 2:
buf, keysym = C.create_string_buffer(32), U()
n = X.XLookupString(C.byref(e), buf, 32, C.byref(keysym), None)
row.update(text=buf.raw[:n].decode('utf-8', 'replace'), keysym=keysym.value)
say(**row)
ready, _, _ = select.select([0, X.XConnectionNumber(d)], [], [], 0.2)
if 0 in ready and not os.read(0, 1024):
break
except TimeoutError:
pass
finally:
if w:
X.XDestroyWindow(d, w)
X.XCloseDisplay(d)
say(cleaned=True)
+4 -25
View File
@@ -154,31 +154,10 @@ two fingers to scroll, two-finger tap to right-click) plus a text field that
types on the Frame. On a computer, clicking the pad passes your mouse and
keyboard through to the Frame until you press Esc (⌘ is sent as Ctrl on a Mac).
Since Control (below), the pad goes through **gamescope's own input socket**
([`ui/frame_touch.py`](../ui/frame_touch.py)), so it reaches whichever panel has
focus, Steam's own menus included, and needs nothing installed on the Frame.
Only characters that aren't on a US keyboard (accents, emoji) go through
**KDE Connect** as below; it starts the first time you type one. (Before
Control, everything went through KDE Connect, which is why the notes below
describe it in detail.)
The server keyboard queue holds at most 200 events, including text being sent.
A batch with an accent sends its whole keyboard part through KDE Connect, and
later keys follow it there while typing is pending. If the batch will not fit,
the server rejects it before sending any clicks or keys; the page keeps it and
retries after polling. Accepted events are never evicted to make room. A timeout
or helper failure retains unsent text and shows **Try again**; that button resumes
the retained queue. The queue lives in memory and is lost if the server exits.
Switching headsets cancels queued text and stops both input helpers before the
new route is published. Both controls validate the intended headset before they
start. Ordering between the two input services uses a typing-time estimate;
neither service acknowledges when characters reach the application.
[PR #66 verification on 2026-10-24](evidence/pad-gamescope-2026-10-24.md)
records an owned focused X11 panel receiving a click and `Frame66` plus Enter
on a real Frame with SteamOS 0.5.6, after the shared headset became available.
The helper, window and owner-marked lock were cleaned up. This is ASCII input
proof, not wearer-visible, Unicode or packaged Electron GUI verification.
It goes through **KDE Connect**, the first-party route (KDE makes the Frame's
desktop): Frame Control's server runs [`ui/frame_input_agent.py`](../ui/frame_input_agent.py)
on the Frame, which talks KDE Connect's own LAN protocol to the Frame's
`kdeconnectd` as if it were a phone. KDE Connect does the typing and clicking.
**Verified 2026-09-28** (SteamOS 0.4.1, build 20260925.6191901):
+1 -15
View File
@@ -12,16 +12,6 @@ A fourth, a SteamOS VM, may come later ([issue #6](https://github.com/saphid/ste
## Unit tests
The gamescope keyboard/trackpad route has focused suites in `test_input.py`,
`test_touch.py` and `test_service_regressions.py`. They cover pad translation,
ASCII/modifier keys, Unicode queuing/retries, stale panel targets, release
handling, route cancellation, and intended-headset start/status guards.
`tests/page/pad_queue.mjs` and `tests/page/ctrl_gestures.mjs` execute the page's
real queue and gesture functions in Node. These are offline tests; their
acknowledgements do not prove that a real application received input.
See [the 2026-10-24 PR #66 evidence](evidence/pad-gamescope-2026-10-24.md)
for the real focused-panel click/key observation, cleanup and remaining limits.
The read-only power/heat sampler has fixture and transport tests in
`tests/test_power_sample.py`. Run
`python3 -m unittest discover -s tests -p test_power_sample.py -v`.
@@ -392,11 +382,7 @@ trees on the fake Frame. None of it has run on a headset.
`tests/test_media.py` covers layout evidence and overrides, OU eye ordering,
hardware-decoder command construction, malformed splats, stereo parallax and
fake-Frame library/process ownership. It also covers optional-surround refusal
at creation, setup, first upload and retry after standby, fatal screen refusal,
and best-effort overlay teardown with shutdown.
Run `python3 -m unittest discover -s tests -p test_media.py -v`.
`tests/e2e/test_media_transfer.py` checks the real
fake-Frame library/process ownership. `tests/e2e/test_media_transfer.py` checks the real
HTTP/SSH upload and library listing without pretending the fake renders VR.
Real decode timings and captured stereo output are recorded in
[vr-video.md](vr-video.md). Generated media only; no external player required.
-8
View File
@@ -101,14 +101,6 @@ Stills and the theatre surround are re-sent after waking. Five minutes
without an accepted frame is reported as an error. Headset-view captures
taken during standby show a flat dark frame, not our screen.
**Optional surround:** if SteamVR refuses the theatre dark surround at startup
or on a retry after standby, playback continues without it and the player log
records `Theatre surround unavailable`. Screen errors still fail playback.
On exit, the player attempts to destroy every owned overlay and calls OpenVR
shutdown even if an overlay's teardown raises an exception. Refusal and
teardown errors are covered by fake-OpenVR tests; a real surround refusal has
not been induced safely. See the [current hardening checks](evidence/media-surround-2026-10-24.md).
![Media panel in Frame Control while a photo plays on the Frame](img/media-ui-panel.png)
**Verified failed route:** GStreamer 1.24.2's `playbin` selected
+1 -27
View File
@@ -31,8 +31,7 @@ function page(api) {
// A request that fails, or reaches a link that had dropped, keeps its keys and releases.
for (const [label, api] of [["request failed", async () => { throw new Error("offline"); }],
["not sent", async () => ({ state: "starting", sent: false })],
["queue full", async () => ({ state: "waiting", sent: false })]]) {
["not sent", async () => ({ state: "starting", sent: false })]]) {
const { pad, padSend, shown } = page(api);
padSend({ dx: 5 }); // sent and lost: a move, fine to drop
await new Promise(r => setTimeout(r, 0));
@@ -46,29 +45,4 @@ for (const [label, api] of [["request failed", async () => { throw new Error("of
fail(label + ": " + JSON.stringify(q));
if (!shown.length || shown.at(-1).state === "ready") fail(label + ": state not shown");
}
// Acknowledged text is removed locally, but its pending status must reach padShow
// so failures reported after acceptance are polled without another keypress.
{
const { pad, padSend, shown } = page(async () => ({ state: "ready", sent: true, queued: 1 }));
padSend({ key: "é" });
await new Promise(r => setTimeout(r, 0));
if (pad.queue.length || shown.at(-1)?.queued !== 1) fail("accepted text is not polled");
}
// Exercise the real status/poll function: queued text leaves pointer controls
// usable, still polls, shows an eventual error, and does not silently retry it.
{
const pad = { state: "ready", queue: [] }, elements = {};
let poll = null;
const show = new Function("pad", "$", "PAD_STATES", "TOUCH", "padFlush", "devGen", "api",
"setTimeout", "clearTimeout", grab("padShow") + "; return padShow;")(
pad, id => elements[id] ||= { classList: { toggle() {} } }, {}, false, () => {}, 0,
async () => ({ state: "error", message: "Text is retained; try again." }),
fn => { poll = fn; return 1; }, () => { poll = null; });
show({ state: "ready", queued: 1 });
if (pad.state !== "ready" || !poll) fail("pending text disabled pointer or skipped poll");
await poll();
if (pad.state !== "error" || poll || !elements.padHint.textContent.includes("retained"))
fail("asynchronous error not surfaced");
}
console.log("queue rules ok");
-426
View File
@@ -18,7 +18,6 @@ import tempfile
import threading
import time
import unittest
from unittest import mock
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
@@ -632,428 +631,3 @@ class PageQueue(unittest.TestCase):
if __name__ == "__main__":
unittest.main()
class PadEvents(unittest.TestCase):
"""The trackpad and key row go through gamescope; accents and emoji go the KDE Connect way."""
@classmethod
def setUpClass(cls):
import server
cls.pad = staticmethod(server.pad_events)
def test_moves_clicks_and_scroll(self):
out, extra = self.pad([{"dx": 3, "dy": -2}, {"singleclick": True}, {"singlehold": True},
{"singlerelease": True}, {"scroll": True, "dy": 1}, {"rightclick": True}])
self.assertEqual(out[0], {"dx": 3, "dy": -2})
self.assertEqual(out[1:3], [{"button": "left", "down": True}, {"button": "left", "down": False}])
self.assertEqual(out[3:5], [{"button": "left", "down": True}, {"button": "left", "down": False}])
self.assertEqual(out[5], {"scroll": [0, -15]}) # up = content follows the finger
self.assertEqual(out[6:], [{"button": "right", "down": True}, {"button": "right", "down": False}])
self.assertEqual(extra, [])
def test_keys_and_modifiers(self):
out, _ = self.pad([{"specialKey": 12}, {"specialKey": 1, "ctrl": True}, {"key": "c", "ctrl": True}])
self.assertEqual(out[:2], [{"key": 28, "down": True}, {"key": 28, "down": False}])
self.assertEqual([e["key"] for e in out[2:6]], [29, 14, 14, 29]) # Ctrl+Backspace
self.assertEqual([(e["key"], e["down"]) for e in out[6:]],
[(29, True), (46, True), (46, False), (29, False)]) # Ctrl+C
def test_text_stays_ascii_through_gamescope(self):
out, slow = self.pad([{"key": "Hi "}, {"key": "there"}])
self.assertEqual(out, [{"text": "Hi there"}])
self.assertEqual(slow, [])
def test_keyboard_goes_to_kde_whole_and_in_order(self):
events = [{"key": "a"}, {"key": "é"}, {"specialKey": 12}, {"key": "c", "ctrl": True}, {"dx": 2, "dy": 1}]
out, slow = self.pad(events, kde=True)
self.assertEqual(out, [{"dx": 2, "dy": 1}]) # the pointer still goes through gamescope
self.assertEqual(slow, events[:4]) # the keyboard as sent, in order
def test_shifted_key_with_modifier(self):
out, _ = self.pad([{"key": "A", "ctrl": True}])
self.assertEqual([e["key"] for e in out], [29, 42, 30, 30, 42, 29])
class PadDelivery(unittest.TestCase):
"""remote_input's promises: "sent" means the whole batch is taken; accents go once, in order."""
class Agent:
def __init__(self, sent=True, state="ready"):
self.sent, self.state, self.got = sent, state, []
def send(self, events):
if events and self.sent:
self.got.append(events)
return {"state": self.state, "sent": self.sent and bool(events)}
def setUp(self):
import server
self.s = server
self.saved = (server._touch, server._input, server.KDE_WAIT)
server._touch, server._input = self.Agent(), self.Agent()
self.reset()
def tearDown(self):
self.s._touch, self.s._input, self.s.KDE_WAIT = self.saved
self.settle()
self.reset()
def reset(self):
self.s._kde_queue.clear()
self.s._kde_error = None
self.s._kde_until = self.s._typed_until = 0.0
self.s._kde_worker[0] = None
def settle(self):
worker = self.s._kde_worker[0]
if worker:
worker.join(5)
def typed(self):
return [e for batch in self.s._input.got for e in batch]
def test_accent_only_batch_is_acknowledged_and_typed_once(self):
status = self.s.remote_input({"events": [{"key": "é"}]})
self.assertTrue(status["sent"]) # so the page doesn't send it again
self.settle()
self.assertEqual(self.typed(), [{"key": "é"}])
def test_mixed_text_stays_in_one_ordered_stream(self):
self.s.remote_input({"events": [{"key": "aéb"}, {"specialKey": 12}]})
self.settle()
self.assertEqual(self.typed(), [{"key": "aéb"}, {"specialKey": 12}])
self.assertEqual(self.s._touch.got, []) # none of it split off through gamescope
def test_batch_gamescope_didnt_take_is_not_queued(self):
self.s._touch = self.Agent(sent=False, state="starting")
status = self.s.remote_input({"events": [{"dx": 1, "dy": 1}, {"key": "é"}]})
self.assertFalse(status["sent"]) # the page tries the whole batch again...
self.assertEqual(self.s._kde_queue, []) # ...so the accent isn't queued twice
def test_waits_for_kde_connect_then_sends_without_more_input(self):
self.s._input = self.Agent(sent=False, state="starting")
self.s.remote_input({"events": [{"key": "é"}]})
time.sleep(0.2)
self.assertEqual(self.s._kde_queue, [{"key": "é"}])
self.s._input.sent = True
self.settle()
self.assertEqual(self.typed(), [{"key": "é"}])
self.assertEqual(self.s._kde_queue, [])
def test_later_ascii_waits_behind_a_pending_accent(self):
self.s._input = self.Agent(sent=False, state="starting")
self.s.remote_input({"events": [{"key": "é"}]})
self.s.remote_input({"events": [{"key": "x"}]})
self.assertEqual(self.s._touch.got, []) # "x" can't overtake the "é"
self.assertEqual(self.s._kde_queue, [{"key": "é"}, {"key": "x"}])
self.s._input.sent = True
self.settle()
self.assertEqual(self.typed(), [{"key": "é"}, {"key": "x"}])
def test_errors_and_timeout_retain_accepted_text_for_explicit_retry(self):
events = [{"key": "é"}] * 150
for state in ("error", "starting"):
with self.subTest(state=state):
self.reset()
self.s._input = self.Agent(sent=False, state=state)
self.s.KDE_WAIT = 0
self.assertTrue(self.s.remote_input({"events": events})["sent"])
self.settle()
self.assertEqual(self.s._kde_queue, events)
self.assertIsNone(self.s._kde_worker[0])
self.assertEqual(self.s._kde_until, 0)
status = self.s.pad_status({"state": "ready"})
self.assertEqual(status["state"], "error")
self.assertIn("retained", status["message"])
self.assertFalse(self.s.remote_input({"events": [{"key": "x"}]})["sent"])
self.s._input = self.Agent()
self.s.start_pad()
self.settle()
self.assertEqual(self.typed(), events)
self.assertEqual(self.s._kde_queue, [])
self.assertIsNone(self.s._kde_error)
def test_worker_exception_and_start_failure_are_visible_and_retryable(self):
for start_failure in (False, True):
with self.subTest(start_failure=start_failure):
self.reset()
events = [{"key": "é"}]
if start_failure:
with mock.patch.object(threading.Thread, "start", side_effect=RuntimeError("no thread")):
self.assertTrue(self.s.remote_input({"events": events})["sent"])
else:
with mock.patch.object(self.s._input, "send", side_effect=OSError("send failed")):
self.assertTrue(self.s.remote_input({"events": events})["sent"])
self.settle()
self.assertEqual(self.s._kde_queue, events)
self.assertIsNone(self.s._kde_worker[0])
self.assertEqual(self.s._kde_until, 0)
self.assertEqual(self.s.pad_status({"state": "ready"})["state"], "error")
self.s._input = self.Agent()
self.s.start_pad()
self.settle()
self.assertEqual(self.typed(), events)
def test_ascii_just_after_an_accent_stays_behind_it(self):
self.s.remote_input({"events": [{"key": "é"}]})
self.settle()
self.s.remote_input({"events": [{"key": "x"}]}) # KDE Connect may still be typing the é
self.settle()
self.assertEqual(self.s._touch.got, [])
self.assertEqual(self.typed(), [{"key": "é"}, {"key": "x"}])
def test_accent_just_after_gamescope_text_waits_for_it(self):
self.s.remote_input({"events": [{"key": "x" * 50}]})
self.assertEqual(self.s._touch.got, [[{"text": "x" * 50}]])
start = time.time()
self.s.remote_input({"events": [{"key": "é"}]})
self.settle()
self.assertGreater(time.time() - start, 0.6) # 50 keys at 16 ms, plus the margin
self.assertEqual(self.typed(), [{"key": "é"}])
def test_typing_time_counts_every_key_transition(self):
t = self.s.typing_seconds
self.assertAlmostEqual(t([{"text": "ab"}]), 4 * 0.008)
self.assertAlmostEqual(t([{"text": "A"}]), 4 * 0.008) # shift down, key down, key up, shift up
self.assertAlmostEqual(t([{"text": "x" * 500}]), 8.0) # a long paste
self.assertAlmostEqual(t([{"key": 29, "down": True}, {"dx": 3, "dy": 0}]), 0.008)
def test_the_margin_is_added_once_not_per_request(self):
for _ in range(10):
self.s.remote_input({"events": [{"key": "ab"}]})
# ten requests of 4 transitions each: the work queues up, with no margin added per request
self.assertLess(self.s._typed_until - time.time(), 10 * 4 * 0.008 + 0.05)
def hold_worker(self):
entered, release = threading.Event(), threading.Event()
class Gate:
def wait(self, timeout):
entered.set()
if not release.wait(3):
raise RuntimeError("test did not release worker")
return False
patch = mock.patch.object(self.s, "_kde_cancel", Gate())
patch.start()
self.addCleanup(patch.stop)
self.addCleanup(release.set)
return entered, release
def test_saturation_rejects_whole_batch_and_retry_delivers_every_accepted_event_once(self):
entered, release = self.hold_worker()
self.s._typed_until = time.time() + 60
first = [{"key": "é"}] * 150
remainder = [{"key": "ü"}] * 50
overflow = [{"singleclick": True}, {"key": "ß"}] * 75
self.assertTrue(self.s.remote_input({"events": first})["sent"])
self.assertTrue(entered.wait(1)) # batch is in flight, waiting for prior ASCII
try:
self.assertTrue(self.s.remote_input({"events": remainder})["sent"])
status = self.s.remote_input({"events": overflow})
self.assertFalse(status["sent"])
self.assertEqual(status["state"], "waiting")
self.assertEqual(self.s._kde_queue, first + remainder)
self.assertEqual(len(self.s._kde_queue), self.s.KDE_QUEUE_LIMIT)
self.assertEqual(self.s._touch.got, []) # rejected clicks cannot be duplicated
finally:
self.s._typed_until = 0
release.set()
self.settle()
self.assertEqual(self.typed(), first + remainder)
self.assertTrue(self.s.remote_input({"events": overflow})["sent"])
self.settle()
self.assertEqual(self.typed(), first + remainder + [{"key": "ß"}] * 75)
self.assertEqual(self.s._touch.got, [[e for _ in range(75) for e in (
{"button": "left", "down": True}, {"button": "left", "down": False})]])
def test_retrying_batch_keeps_its_capacity_and_order(self):
entered, release = self.hold_worker()
self.s._input = self.Agent(sent=False, state="starting")
first, second = [{"key": "é"}] * 150, [{"key": "ü"}] * 150
self.assertTrue(self.s.remote_input({"events": first})["sent"])
self.assertTrue(entered.wait(1)) # first send failed; worker is in retry delay
try:
self.assertFalse(self.s.remote_input({"events": second})["sent"])
self.assertEqual(self.s._kde_queue, first)
finally:
self.s._input.sent = True
release.set()
self.settle()
self.assertTrue(self.s.remote_input({"events": second})["sent"])
self.settle()
self.assertEqual(self.typed(), first + second)
def test_concurrent_requests_cannot_both_claim_the_last_capacity(self):
entered, release = self.hold_worker()
self.s._typed_until = time.time() + 60
first = [{"key": "é"}]
self.s.remote_input({"events": first})
self.assertTrue(entered.wait(1))
barrier, results = threading.Barrier(3), []
def request(key):
barrier.wait(2)
batch = [{"key": key}] * 150
results.append((batch, self.s.remote_input({"events": batch})["sent"]))
threads = [threading.Thread(target=request, args=(key,)) for key in ("ü", "ß")]
try:
for thread in threads:
thread.start()
barrier.wait(2)
for thread in threads:
thread.join(2)
self.assertFalse(thread.is_alive())
self.assertEqual(sorted(sent for _, sent in results), [False, True])
finally:
self.s._typed_until = 0
release.set()
self.settle()
accepted = next(batch for batch, sent in results if sent)
rejected = next(batch for batch, sent in results if not sent)
self.assertEqual(self.typed(), first + accepted)
self.assertTrue(self.s.remote_input({"events": rejected})["sent"])
self.settle()
self.assertEqual(self.typed(), first + accepted + rejected)
def test_input_queued_as_the_worker_finishes_is_still_sent(self):
for _ in range(40):
self.reset()
self.s._input.got.clear()
self.s.remote_input({"events": [{"key": "é"}]})
time.sleep(0.002)
self.s.remote_input({"events": [{"key": "ü"}]})
self.settle()
self.assertEqual(self.s._kde_queue, [])
self.assertEqual(self.typed(), [{"key": "é"}, {"key": "ü"}])
class PadRoute(unittest.TestCase):
def test_unicode_start_and_headset_switch_do_not_deadlock(self):
# Real InputAgent.send/start takes the route lock. Race it with route(),
# which cancels the pad worker; mocks of send() hide this lock inversion.
# A child keeps a regression's blocked daemon threads out of this suite.
code = r'''
import sandbox
import server as s
import threading
from unittest import mock
entered, release, switching = threading.Event(), threading.Event(), threading.Event()
s._input = s.InputAgent(packages=[])
s._input._launch = lambda *args: None # exercise real start, without any SSH
s._touch = mock.Mock()
send = s._input.send
def held(events):
entered.set()
assert release.wait(2)
return send(events)
s._input.send = held
s._power.stop = lambda *args: switching.set()
s._queue_for_kde([{'key': 'é'}])
worker = s._kde_worker[0]
assert entered.wait(2), 'Unicode worker never reached send'
switch = threading.Thread(target=s.route, args=('frame-pr66-new', []), daemon=True)
switch.start()
switching.wait(0.2)
release.set()
worker.join(2)
switch.join(2)
assert not worker.is_alive(), 'Unicode worker deadlocked starting input during headset switch'
assert not switch.is_alive(), 'headset switch deadlocked cancelling Unicode input'
assert not s._kde_queue
'''
env = {**os.environ, "PYTHONPATH": os.pathsep.join([str(ROOT / 'tests'), str(ROOT / 'ui')])}
result = subprocess.run([sys.executable, '-c', code], env=env, capture_output=True,
text=True, timeout=8)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
def setUp(self):
import server
self.s = server
self.agents = mock.patch.multiple(server, _touch=mock.Mock(), _input=mock.Mock())
self.agents.start()
self.addCleanup(self.agents.stop)
self.s._touch.send.return_value = {"state": "ready", "sent": True}
self.s._input.send.return_value = {"state": "ready", "sent": True}
self.s.reset_pad_route()
self.s._touch.reset_mock(); self.s._input.reset_mock()
self.addCleanup(self.s.reset_pad_route)
def switch(self):
with mock.patch.multiple(self.s, FRAME="frame-a", HOST_OPTS=[], MUX=[], SSH=[], macview=None), \
mock.patch.multiple(self.s.frame_android, FRAME="frame-a", SSH_OPTS=[]):
self.s.route("frame-b", [])
def test_stale_worker_exception_cannot_damage_new_route(self):
entered, release = threading.Event(), threading.Event()
class BrokenWait:
def wait(self, timeout):
entered.set()
if not release.wait(2):
raise RuntimeError("test did not release old worker")
raise OSError("old route wait failed")
def set(self):
pass # hold its exception until the new route has been published
self.s._kde_cancel = BrokenWait()
self.s._typed_until = time.time() + 60
self.s.remote_input({"events": [{"key": "é"}]})
worker = self.s._kde_worker[0]
try:
self.assertTrue(entered.wait(1))
self.switch()
marker = object()
with self.s._kde_lock:
self.s._kde_queue.append({"key": "new"})
self.s._kde_worker[0] = marker
self.s._kde_until = 42
finally:
release.set()
worker.join(2)
self.assertFalse(worker.is_alive())
self.assertEqual(self.s._kde_queue, [{"key": "new"}])
self.assertIs(self.s._kde_worker[0], marker)
self.assertEqual(self.s._kde_until, 42)
self.assertIsNone(self.s._kde_error)
def test_switch_stops_both_input_agents_and_discards_old_queue(self):
self.s._kde_queue.append({"key": "old"})
self.s._kde_until = self.s._typed_until = time.time() + 60
cancel = self.s._kde_cancel
self.switch()
self.s._input.stop.assert_called_once()
self.s._touch.stop.assert_called_once()
self.assertEqual(self.s._kde_queue, [])
self.assertEqual((self.s._kde_until, self.s._typed_until), (0, 0))
self.assertTrue(cancel.is_set())
def test_switch_cancels_delayed_and_retrying_text_before_new_input(self):
for delaying in (True, False):
with self.subTest(delaying=delaying):
waiting, cancel = threading.Event(), threading.Event()
class ObservedCancel:
def wait(self, timeout):
waiting.set()
return cancel.wait(timeout)
def set(self):
cancel.set()
self.s.reset_pad_route()
self.s._kde_cancel = ObservedCancel()
self.s._typed_until = time.time() + 60 if delaying else 0
self.s._input.reset_mock()
self.s._input.send.return_value = {"state": "starting", "sent": False}
self.s.remote_input({"events": [{"key": "é"}]})
worker = self.s._kde_worker[0]
self.assertTrue(waiting.wait(1), "worker never reached its cancellable delay")
self.switch()
worker.join(1)
self.assertFalse(worker.is_alive())
attempts = self.s._input.send.call_count
self.assertEqual(attempts, 0 if delaying else 1)
self.s._input.send.return_value = {"state": "ready", "sent": True}
self.s.remote_input({"events": [{"key": "ü"}]})
next_worker = self.s._kde_worker[0]
if next_worker:
next_worker.join(1)
self.assertFalse(next_worker.is_alive())
self.assertEqual(self.s._input.send.call_count, attempts + 1)
self.assertEqual(self.s._input.send.call_args.args[0], [{"key": "ü"}])
self.assertEqual(self.s._kde_queue, [])
+3 -62
View File
@@ -68,7 +68,7 @@ class Media(unittest.TestCase):
self.assertIn('-frames:v', cmd)
def play_with(self, name, busy=0, on_pixels=None, sleeps=None, info=None,
fail=None, layout='auto', on_create=None, on_call=None):
fail=None, layout='auto'):
"""Run the player against a fake OpenVR; returns (status, pixels calls).
Handle 0 is the theatre surround and 1 the screen. `fail(handle, n)` makes
@@ -82,18 +82,11 @@ class Media(unittest.TestCase):
write_status(path, **values)
class FakeOverlay:
created = 0
def create(self, *a, **k):
# Handles in creation order: surround 0, then screen 1 (theatre).
FakeOverlay.created += 1
if on_create:
on_create(*a, **k)
return FakeOverlay.created - 1
return len(calls)
def call(self, *a):
if on_call:
on_call(*a)
pass
def pixels(self, handle, data, w, h):
calls.append((handle, w, h))
@@ -139,58 +132,6 @@ class Media(unittest.TestCase):
self.assertEqual(result['dropped'], 2)
self.assertIn((0, 1, 1), calls[3:])
def test_theatre_surround_failure_does_not_stop_playback(self):
def fail_surround(n):
if n == 1: # the surround's upload is the first pixels call
raise RuntimeError('OpenVR SetOverlayRaw failed: 11')
result, _ = self.play_with('clip_SBS.mp4', on_pixels=fail_surround)
self.assertEqual((result['state'], result['frames']), ('ended', 4))
def test_theatre_surround_creation_and_setup_failures_are_cosmetic(self):
def fail_create(key, *a, **k):
if key.endswith('.surround'):
raise RuntimeError('OpenVR CreateOverlay failed: 11')
def fail_alpha(name, *a):
if name == 'SetOverlayAlpha':
raise RuntimeError('OpenVR SetOverlayAlpha failed: 11')
for hooks in ({'on_create': fail_create}, {'on_call': fail_alpha}):
with self.subTest(hooks=hooks), patch('builtins.print') as log:
result, calls = self.play_with('clip_SBS.mp4', **hooks)
self.assertEqual((result['state'], result['frames']), ('ended', 4))
self.assertEqual(sum(h == 1 for h, w, hgt in calls), 4)
self.assertIn('Theatre surround unavailable:', log.call_args.args[0])
def test_theatre_surround_refusal_after_standby_is_not_retried(self):
def refuse_retry(n):
if n == 3: # surround busy, screen accepted, surround retry refused
raise RuntimeError('OpenVR SetOverlayRaw failed: 11')
with patch('builtins.print') as log:
result, calls = self.play_with('clip_SBS.mp4', busy=1, on_pixels=refuse_retry)
self.assertEqual((result['state'], result['frames'], result['dropped']), ('ended', 4, 0))
self.assertEqual(sum(h == 0 for h, w, hgt in calls), 2)
self.assertIn('Theatre surround unavailable:', log.call_args.args[0])
def test_screen_refusal_still_fails_playback(self):
def refuse_screen(n):
if n == 2:
raise RuntimeError('OpenVR SetOverlayRaw failed: 11')
with self.assertRaisesRegex(RuntimeError, 'SetOverlayRaw failed: 11'):
self.play_with('clip_SBS.mp4', on_pixels=refuse_screen)
def test_close_attempts_every_overlay_and_shuts_down_after_teardown_errors(self):
for error in (RuntimeError('OpenVR error'), ValueError('bad handle'),
InterruptedError('late Stop')):
with self.subTest(error=type(error).__name__):
overlay = player.Overlay.__new__(player.Overlay)
overlay.handles = [10, 20]
overlay.vr = unittest.mock.Mock()
overlay.call = unittest.mock.Mock(side_effect=[error, None])
overlay.close()
self.assertEqual(overlay.call.call_args_list, [
unittest.mock.call('DestroyOverlay', 20),
unittest.mock.call('DestroyOverlay', 10)])
overlay.vr.VR_ShutdownInternal.assert_called_once_with()
def test_stop_mid_video_reports_stopped(self):
result, _ = self.play_with('clip_SBS.mp4', on_pixels=lambda n: n == 3 and stop_now())
self.assertEqual(result['state'], 'stopped')
+1 -1
View File
@@ -296,7 +296,7 @@ class OneServer(unittest.TestCase):
self.assertIn("Frame Control on", first.stdout.readline())
second = subprocess.run([sys.executable, str(ROOT / "ui" / "server.py"), "--port", "0"], env=env,
capture_output=True, text=True, timeout=60)
self.assertEqual(second.returncode, 1, second.stderr)
self.assertEqual(second.returncode, 1)
self.assertIn("already running", second.stderr)
first.terminate()
first.wait(10)
+6 -95
View File
@@ -2,7 +2,6 @@
import sandbox # noqa: F401
import contextlib
import hashlib
import io
import json
import os
from pathlib import Path
@@ -20,28 +19,6 @@ import frame_agent as agent
import frame_android as android
class StartupTests(unittest.TestCase):
def test_rejected_server_starts_no_background_work(self):
# A losing process exits through Python teardown. Background TLS/index
# workers there can crash it, and it must not sweep the owner's files.
with mock.patch.object(server, 'LOCAL', False), \
mock.patch.object(server, 'PRIVATE', False), \
mock.patch.object(sys, 'argv', ['server.py', '--port', '0']), \
mock.patch.object(server, 'one_server', side_effect=SystemExit('already running')), \
mock.patch.object(server, 'LoopbackServer') as listener, \
mock.patch.object(server, 'sweep_tmp') as sweep, \
mock.patch.object(server.threading, 'Thread') as worker, \
mock.patch.object(server.frame_telemetry, 'start') as telemetry, \
mock.patch.object(server.frame_contact, 'start') as contact:
with self.assertRaisesRegex(SystemExit, 'already running'):
server.main()
listener.assert_not_called()
sweep.assert_not_called()
worker.assert_not_called()
telemetry.assert_not_called()
contact.assert_not_called()
class TargetTests(unittest.TestCase):
def setUp(self):
self.active = {'id': 'a', 'name': 'Headset A'}
@@ -55,7 +32,6 @@ class TargetTests(unittest.TestCase):
self.stack.enter_context(mock.patch.object(server, '_backfill', {'running': False, 'last': 0.0}))
self.stack.enter_context(mock.patch.object(server.macview, 'retarget'))
self.stack.enter_context(mock.patch.object(server._input, 'stop'))
self.real_devices_action = server.frame_link.devices_action
self.stack.enter_context(mock.patch.object(server.frame_link, 'devices_action', self.switch))
def switch(self, link, body, setup, busy):
@@ -114,39 +90,6 @@ class TargetTests(unittest.TestCase):
server._live_tunnels.discard(proc)
server.route(*before)
def test_input_post_reserves_headset_through_send_and_enqueue(self):
body = json.dumps({'events': [{'singleclick': True}, {'key': 'é'}]}).encode()
handler = server.Handler.__new__(server.Handler)
handler.path = '/api/input'
handler.local_request = lambda: True
handler.headers = {'Content-Length': str(len(body)), 'X-Frame-Device': 'a'}
handler.rfile = io.BytesIO(body)
handler.send_json, handler.send_error_json = mock.Mock(), mock.Mock()
stages = []
def check_switch(stage):
self.assertEqual(server._work[0], 1)
with self.assertRaises(server.Failure):
server.devices_post({'action': 'use', 'id': 'b'})
stages.append(stage)
def send(events):
check_switch('send')
return {'state': 'ready', 'sent': True}
def enqueue(events):
check_switch('enqueue')
self.assertEqual(events, [{'key': 'é'}])
return True
with mock.patch.object(server.frame_link, 'devices_action', self.real_devices_action), \
mock.patch.object(server._touch, 'send', side_effect=send), \
mock.patch.object(server, '_queue_for_kde', side_effect=enqueue):
handler.do_POST()
handler.send_error_json.assert_not_called()
self.assertTrue(handler.send_json.call_args.args[0]['sent'])
self.assertEqual(stages, ['send', 'enqueue'])
self.assertEqual(self.active['id'], 'a')
self.assertEqual(server._work[0], 0)
server.devices_post({'action': 'use', 'id': 'b'})
self.assertEqual(self.active['id'], 'b')
def test_backfill_counts_work_until_error_cleanup(self):
entered, release = threading.Event(), threading.Event()
writes = []
@@ -437,9 +380,9 @@ class FinalGetStartTests(unittest.TestCase):
self.active.update(id='b', name='Headset B')
for name in ('input', 'touch'):
with self.subTest(name=name):
fresh = server.TouchAgent()
fresh = server.InputAgent(packages=[]) if name == 'input' else server.TouchAgent()
handler = self.handler('/api/%s?start=1' % name, 'a')
with mock.patch.object(server, '_touch', fresh), mock.patch.object(fresh, 'start') as start:
with mock.patch.object(server, '_' + name, fresh), mock.patch.object(fresh, 'start') as start:
handler.do_GET()
start.assert_not_called()
self.assertEqual(handler.send_error_json.call_args.args[1], 409)
@@ -448,46 +391,14 @@ class FinalGetStartTests(unittest.TestCase):
def test_desktop_get_start_requires_an_intended_device(self):
for name in ('input', 'touch'):
with self.subTest(name=name):
fresh = server.TouchAgent()
fresh = server.InputAgent(packages=[]) if name == 'input' else server.TouchAgent()
handler = self.handler('/api/%s?start=1' % name, None)
with mock.patch.object(server, '_touch', fresh), mock.patch.object(fresh, 'start') as start:
with mock.patch.object(server, '_' + name, fresh), mock.patch.object(fresh, 'start') as start:
handler.do_GET()
start.assert_not_called()
self.assertEqual(handler.send_error_json.call_args.args[1], 400)
self.assertEqual(server._work[0], 0)
def test_input_get_starts_and_polls_gamescope_without_starting_kde_connect(self):
touch, kde = mock.Mock(), mock.Mock()
touch.status = {'state': 'ready', 'focus': 7}
touch.send.return_value = {'state': 'starting', 'sent': False}
with mock.patch.object(server, '_touch', touch), mock.patch.object(server, '_input', kde):
start = self.handler('/api/input?start=1', 'a')
start.do_GET()
touch.send.assert_called_once_with([])
start.send_json.assert_called_once_with(touch.send.return_value)
status = self.handler('/api/input', 'a')
status.do_GET()
status.send_json.assert_called_once_with(touch.status)
kde.send.assert_not_called()
kde.start.assert_not_called()
def test_input_poll_exposes_retained_text_error_and_start_retries(self):
touch = mock.Mock(status={'state': 'ready'})
touch.send.return_value = {'state': 'ready', 'sent': False}
with mock.patch.object(server, '_touch', touch), \
mock.patch.object(server, '_kde_queue', [{'key': 'é'}]), \
mock.patch.object(server, '_kde_error', 'Text is retained; try again.'), \
mock.patch.object(server, '_kde_worker', [None]), \
mock.patch.object(server, '_start_kde_locked') as retry:
status = self.handler('/api/input', 'a')
status.do_GET()
self.assertEqual(status.send_json.call_args.args[0]['state'], 'error')
retry.assert_not_called() # polling never retries typing on its own
start = self.handler('/api/input?start=1', 'a')
start.do_GET()
retry.assert_called_once_with()
touch.send.assert_called_once_with([])
def test_get_start_validation_and_thread_registration_are_one_reservation(self):
attempts = []
fresh = server.TouchAgent()
@@ -547,10 +458,10 @@ class FinalGetStartTests(unittest.TestCase):
(None, None, '?start=1')):
for name in ('input', 'touch'):
with self.subTest(name=name, query=query, local=link is None):
fresh = server.TouchAgent()
fresh = server.InputAgent(packages=[]) if name == 'input' else server.TouchAgent()
handler = self.handler('/api/' + name + query, meant)
with mock.patch.object(server, 'LINK', link), mock.patch.object(server, 'LOCAL', link is None), \
mock.patch.object(server, '_touch', fresh), mock.patch.object(fresh, 'start') as start:
mock.patch.object(server, '_' + name, fresh), mock.patch.object(fresh, 'start') as start:
handler.do_GET()
self.assertEqual(start.call_count, 1 if query else 0)
handler.send_error_json.assert_not_called()
-8
View File
@@ -150,14 +150,6 @@ class Apply(unittest.TestCase):
self.assertFalse(self.t.STALE[0])
self.assertEqual(self.said[-1][1].get("stale"), None)
def test_stale_survives_a_bare_release(self):
gs = FakeGamescope()
panel = self.t.apply(gs, {"button": "left", "down": True, "window": 99, "display": ":1"}, None)
self.assertTrue(self.t.STALE[0])
self.t.apply(gs, {"button": "left", "down": False}, panel) # the page's release names no panel
self.assertTrue(self.t.STALE[0])
self.assertEqual(self.said[-1][1].get("stale"), True)
def test_same_window_id_on_the_other_display_is_another_panel(self):
gs = FakeGamescope()
self.t.apply(gs, {"fx": 0.5, "fy": 0.5, "window": 7, "display": ":0"}, None)
+10 -19
View File
@@ -88,14 +88,12 @@ class Overlay:
def close(self):
# Best effort: SteamVR removes a disconnected client's overlays anyway,
# and a teardown error must not overwrite a finished playback's status.
try:
for h in reversed(self.handles):
try:
self.call('DestroyOverlay', h)
except Exception: # signals are already ignored here, so Stop isn't lost
pass
finally:
self.vr.VR_ShutdownInternal()
for h in reversed(self.handles):
try:
self.call('DestroyOverlay', h)
except RuntimeError:
pass
self.vr.VR_ShutdownInternal()
def probe(path):
@@ -185,9 +183,6 @@ def play(args):
except OverlayBusy:
pending.insert(0, item)
return
except RuntimeError as e:
# Only the cosmetic surround is queued here.
print('Theatre surround unavailable: %s' % e, flush=True)
def hold(handle, data, w, h):
"""Keep a still (photo or splat) up until Stop, retrying through standby."""
@@ -208,14 +203,10 @@ def play(args):
try:
vr = Overlay()
if args.theatre:
try:
surround = vr.create('framecontrol.media.surround', 40, 4, order=0)
vr.call('SetOverlayAlpha', surround, .85)
if not show(surround, b'\x00\x00\x00\xff', 1, 1):
pending.append((surround, b'\x00\x00\x00\xff', 1, 1))
except RuntimeError as e:
# Cosmetic: play without the dark surround rather than not at all.
print('Theatre surround unavailable: %s' % e, flush=True)
surround = vr.create('framecontrol.media.surround', 40, 4, order=0)
vr.call('SetOverlayAlpha', surround, .85)
if not show(surround, b'\x00\x00\x00\xff', 1, 1):
pending.append((surround, b'\x00\x00\x00\xff', 1, 1))
screen = vr.create('framecontrol.media.screen', 3 if args.theatre else 1.6, 2,
plan['layout'] != 'mono', aspect)
if splat:
+1 -3
View File
@@ -511,10 +511,8 @@ def apply(gs, event, panel):
say("ready", focus=panel.get("window"), display=panel.get("display"), stale=True) # the page re-syncs
STALE[0] = True
return panel
release = event.get("down") is False and ("button" in event or "key" in event)
if STALE[0] and not stale and not (release and "window" not in event):
if STALE[0] and not stale:
# Anything that goes through (a trackpad move names no panel) means caught up: stop re-syncing.
# A bare release doesn't: the page leaves the panel off releases, so it says nothing about focus.
STALE[0] = False
say("ready", focus=(panel or {}).get("window"), display=(panel or {}).get("display"))
if "fx" in event and panel and panel.get("window"):
+13 -12
View File
@@ -1270,7 +1270,6 @@
<div class="sub sync-note" id="clipsSyncNote" hidden></div>
<div class="hint">Clips and background recordings from Steam's game recording in the headset. Export joins one into a single <code>.mp4</code> in <code>~/Videos/SteamFrame</code>, however long it is, without re-encoding it. Recordings you make with <b>Record</b> go there too.</div>
</section>
</div>
<div class="page" data-page="games">
@@ -1499,7 +1498,8 @@
<div class="shelf-head"><h2>Keyboard and trackpad</h2><span class="sub" id="padState"></span><span class="spacer"></span>
<button class="small" id="padOn">Turn on</button>
</div>
<p class="sub" id="padIntro">Use this computer's keyboard and mouse in the panel you're using in the headset.</p>
<p class="sub" id="padIntro">Use this computer's keyboard and mouse in apps on the Frame, such as Chromium or the
desktop's Linux apps.</p>
<div id="padBody" hidden>
<div class="pad-area" id="padArea" tabindex="0" role="application" aria-label="Trackpad for the Frame">
<div class="pad-hint" id="padHint"></div>
@@ -1522,9 +1522,10 @@
<button data-pad-key="12" title="Enter">Enter</button>
</span>
</div>
<div class="hint">Types and points in whichever panel has focus in the headset, through Valve's own input path: nothing to
install and no internet needed. Accents and emoji go through KDE Connect, which comes with Frame Control and starts
the first time you type one (it copies Valve's build for the Frame, 3.6 MB, into your home folder there). <a href="#" data-about>Licences</a></div>
<div class="hint">Types and points in apps on the Frame, such as Chromium or the desktop's Linux apps; Steam's own VR menus
don't take it. It works through KDE Connect, which comes with Frame Control: the first time, it copies Valve's build
for the Frame (3.6 MB, 18 MB unpacked) into your home folder there and pairs with it. No internet needed, and nothing
else to install. <a href="#" data-about>Licences</a></div>
</div>
<div class="shelf-head" style="margin-top:18px"><h2>Clipboard</h2></div>
<textarea id="clipText" placeholder="Text to put on the Frame's clipboard…"></textarea>
@@ -6087,11 +6088,11 @@ const typesText = el => el?.matches?.('textarea, input:not([type=range], [type=c
document.addEventListener("focusin", e => document.body.classList.toggle("typing", !!typesText(e.target)));
document.addEventListener("focusout", () => document.body.classList.remove("typing"));
// ---- keyboard and trackpad: events go to the Frame through gamescope (ui/frame_touch.py); accents and emoji through KDE Connect ----
// ---- keyboard and trackpad: events go to KDE Connect on the Frame (see ui/frame_input_agent.py) ----
const pad = { state: "off", queue: [], sending: false, poll: null, captured: false };
const PAD_STATES = {
off: "", starting: "Connecting…", installing: "Setting up (first time only)…",
pairing: "Pairing with KDE Connect…", waiting: "Waiting for queued text…", ready: "On",
off: "", starting: "Connecting…", installing: "Setting up KDE Connect on the Frame (first time only)…",
pairing: "Pairing with KDE Connect…", ready: "On",
};
// KDE Connect's specialKey numbers (plugins/mousepad in its source).
const PAD_SPECIAL = { Backspace: 1, Tab: 2, ArrowLeft: 4, ArrowUp: 5, ArrowRight: 6, ArrowDown: 7, PageUp: 8,
@@ -6110,14 +6111,14 @@ function padShow(status) {
$("padArea").classList.toggle("off", !ready);
$("padBody").hidden = pad.state === "off";
$("padIntro").hidden = pad.state !== "off";
$("padHint").textContent = pad.state === "error" ? status.message || "Couldn't reach the Frame."
$("padHint").textContent = pad.state === "error" ? status.message || "Couldn't reach KDE Connect on the Frame."
: !ready ? (busy ? PAD_STATES[pad.state] : "Turn on to use this as a trackpad for the Frame.")
: TOUCH ? "Drag to move the pointer · tap to click · two fingers to scroll · two-finger tap to right-click"
: pad.captured ? "Your mouse and keyboard now control the Frame. Press Esc to stop."
: "Click here to use your mouse and keyboard on the Frame (Esc to stop), or drag like a trackpad.";
clearTimeout(pad.poll);
if (ready && pad.queue.length) padFlush();
if (busy || status.queued) {
if (busy) {
const gen = devGen;
pad.poll = setTimeout(async () => {
const status = await api('/api/input').catch(e => ({state:'error', message:e.message}));
@@ -6140,7 +6141,7 @@ function padSend(event) {
// While it reconnects, keys and clicks wait in the queue; pointer moves would be stale.
// A button release is always kept, so a drag never leaves the button held on the Frame.
const room = pad.queue.length < 200 || event.singlerelease;
const reconnecting = ["starting", "pairing", "installing", "waiting"].includes(pad.state) && room;
const reconnecting = ["starting", "pairing", "installing"].includes(pad.state) && room;
// Even after an error: a button pressed on the Frame must be let go once it's back.
const release = event.singlerelease && pad.state !== "off";
if (pad.state !== "ready" && !(reconnecting && !isMove(event)) && !release) return;
@@ -6173,7 +6174,7 @@ async function padFlush() {
const status = await api("/api/input", { events });
if (gen !== devGen) return;
if (!status.sent) padKeep(events);
if (status.state !== "ready" || status.queued) padShow(status);
if (status.state !== "ready") padShow(status);
} catch (e) {
if (gen !== devGen) return;
padKeep(events);
+14 -250
View File
@@ -69,7 +69,6 @@ import frame_report # noqa: E402
import frame_store # noqa: E402
import frame_telemetry # noqa: E402
import frame_titles # noqa: E402
import frame_touch # noqa: E402
import frame_webinstall # noqa: E402
import frame_vr # noqa: E402
import frame_utilities # noqa: E402
@@ -112,9 +111,7 @@ frame_android.SSH_OPTS = SSH[1:]
_power = frame_power.Observation()
# Pad delivery reserves the route before its own locks; InputAgent.start()
# captures that same route synchronously, so this lock must be reentrant.
_route_lock = threading.RLock()
_route_lock = threading.Lock()
def route(alias, host_opts):
@@ -126,10 +123,13 @@ def route(alias, host_opts):
moved = alias != FRAME
if moved:
_power.stop('Headset switched; received samples retained for the previous headset')
reset_pad = globals().get("reset_pad_route")
if reset_pad:
reset_pad() # cancel queued input and both agents before publishing the new route
frame_catalog._env.clear() # the SteamOS and Lepton builds reports record are per headset
# Invalidate launches before publishing the new route. send/stop share
# a writer lock, so an old process cannot receive events after this.
for name in ("_input", "_touch"):
agent = globals().get(name)
if agent:
agent.stop()
FRAME = frame_android.FRAME = alias
HOST_OPTS = list(host_opts)
MUX[:] = [*MUX_BASE, *HOST_OPTS]
@@ -1499,244 +1499,12 @@ def licenses():
return notices
# KDE Connect's specialKey numbers -> Linux key codes (KEY_BACKSPACE, KEY_TAB, ...).
PAD_KEYS = {1: 14, 2: 15, 4: 105, 5: 103, 6: 106, 7: 108, 8: 104, 9: 109, 10: 102, 11: 107, 12: 28, 13: 111, 14: 1,
**{21 + i: code for i, code in enumerate([59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 87, 88])}}
PAD_MODS = (("ctrl", 29), ("alt", 56), ("shift", 42), ("super", 125))
PAD_CLICKS = {"singleclick": "left", "rightclick": "right", "middleclick": "middle"}
PAD_NOTCH = 15 # scroll units for one wheel notch
def typed_in_kde(events):
"""Whether these events must go the KDE Connect way: they hold an accent or emoji, or
earlier ones are queued, being sent or still being typed there (typing must stay in
order, so it can't be split between the two ways)."""
with _kde_lock:
waiting = bool(_kde_queue) or time.time() < _kde_until
return waiting or any(ch not in frame_touch.ASCII for e in events for ch in e.get("key", ""))
def pad_events(events, kde=False):
"""The trackpad's and key row's events as (gamescope events, KDE Connect events).
Gamescope's own input reaches every panel and needs nothing installed, but types only
US-keyboard characters. With kde, the keyboard goes to KDE Connect as it was sent, so
text stays in order; the pointer always goes through gamescope.
"""
ascii_keys = frame_touch.ASCII
out, slow = [], []
def tap(code, mods):
down = [{"key": c, "down": True} for c in mods]
out.extend(down + [{"key": code, "down": True}, {"key": code, "down": False}]
+ [{"key": c, "down": False} for c in reversed(mods)])
for e in events:
mods = [code for name, code in PAD_MODS if e.get(name)]
if "dx" in e or "dy" in e:
if not e.get("scroll"):
out.append({k: e[k] for k in ("dx", "dy") if k in e})
if e.get("scroll"):
out.append({"scroll": [0, -(e.get("dy") or 0) * PAD_NOTCH]})
for flag, button in PAD_CLICKS.items():
if e.get(flag):
out += [{"button": button, "down": True}, {"button": button, "down": False}]
if e.get("doubleclick"):
out += [{"button": "left", "down": d} for d in (True, False, True, False)]
if e.get("singlehold"):
out.append({"button": "left", "down": True})
if e.get("singlerelease"):
out.append({"button": "left", "down": False})
if kde and ("key" in e or "specialKey" in e):
slow.append({k: e[k] for k in ("key", "specialKey", "ctrl", "alt", "shift", "super") if k in e})
continue
if "specialKey" in e and e["specialKey"] in PAD_KEYS:
tap(PAD_KEYS[e["specialKey"]], mods)
if "key" in e:
for ch in e["key"]:
if ch not in ascii_keys:
continue # only reached when kde is wrong; the caller checks typed_in_kde first
elif mods:
code, shifted = ascii_keys[ch]
tap(code, mods + ([42] if shifted and 42 not in mods else []))
else:
if out and "text" in out[-1] and len(out[-1]["text"]) < 100:
out[-1]["text"] += ch
else:
out.append({"text": ch})
return out, slow
def remote_input(body):
"""{"events": [...]} sends keyboard and pointer events; {} (or none yet) just starts the agent.
"sent" in the answer means the whole batch is taken (the page keeps it and tries again
if not), so the keyboard's share is queued only once that's so.
"""
"""{"events": [...]} sends keyboard and pointer events; {} (or none yet) just starts the agent."""
events = body.get("events", [])
if not isinstance(events, list) or len(events) > INPUT_BATCH_LIMIT:
raise Failure(f"events must be a list of at most {INPUT_BATCH_LIMIT}", 400)
events = [input_event(e) for e in events]
# Serialize admission with other requests and the sender. Keep route -> pad
# order: either agent may synchronously capture its route during send().
with _route_lock, _pad_send_lock:
touch, slow = pad_events(events, typed_in_kde(events))
with _kde_lock:
blocked = slow and (_kde_error or len(_kde_queue) + len(slow) > KDE_QUEUE_LIMIT)
if blocked:
# Refuse the WHOLE batch before sending its clicks/moves. The page
# retains it and polls, so a retry cannot duplicate pointer events.
return pad_status({"state": "waiting", "sent": False})
status = _touch.send(touch)
if touch and not status.get("sent"):
return status
if touch:
_note_typed(touch)
if slow:
_queue_for_kde(slow)
# Poll at least once after accepting text, even if its worker finished
# already, so an asynchronous error is visible without another keypress.
return pad_status({**status, "sent": True}, pending=bool(slow))
# Typing KDE Connect is to do: it starts the first time it's needed, so this waits for it.
# Nothing on the Frame says when typed text has landed, so each way holds the other back
# for as long as what it was given should take (a short margin on a time estimate).
_kde_queue = []
_kde_lock = threading.Lock()
_pad_send_lock = threading.Lock()
_kde_worker = [None]
_kde_error = None
_kde_until = 0.0 # keys typed through KDE Connect should have landed by then
_typed_until = 0.0 # when gamescope's agent will have typed everything it's been given
_kde_route = object()
_kde_cancel = threading.Event()
KDE_QUEUE_LIMIT = 200
KDE_WAIT = 90 # seconds to wait before pausing queued text for an explicit retry
TYPING_SETTLE = 0.6 # seconds past the estimate, once
def reset_pad_route():
"""Discard input for the headset being left, including batches waiting for an agent."""
global _kde_route, _kde_cancel, _kde_until, _typed_until, _kde_error
with _route_lock, _pad_send_lock, _kde_lock:
_kde_cancel.set()
_kde_cancel = threading.Event()
_kde_route = object()
_kde_queue.clear()
_kde_worker[0] = None
_kde_error = None
_kde_until = _typed_until = 0.0
_input.stop()
_touch.stop()
def typing_seconds(touch):
"""How long gamescope's agent takes over these: it sleeps 8 ms after every key transition."""
transitions = 0
for e in touch:
if "text" in e:
transitions += sum(2 + 2 * frame_touch.ASCII[ch][1] for ch in e["text"] if ch in frame_touch.ASCII)
elif "key" in e:
transitions += 1
return transitions * 0.008
def _note_typed(touch):
global _typed_until
seconds = typing_seconds(touch)
if seconds:
with _kde_lock: # the end of the work, which queues up; the margin is added once, where it's checked
_typed_until = max(_typed_until, time.time()) + seconds
def pad_status(status, pending=False):
with _kde_lock:
if _kde_error:
return {**status, "state": "error", "message": _kde_error}
if (_kde_queue or pending) and status.get("state") != "error":
return {**status, "queued": len(_kde_queue) or 1}
return status
def start_pad():
with _route_lock, _pad_send_lock:
status = _touch.send([])
with _kde_lock:
if _kde_queue and _kde_worker[0] is None:
_start_kde_locked() # explicit Try again retries retained text
return pad_status(status)
def _start_kde_locked():
global _kde_error
_kde_error = None
try:
worker = threading.Thread(target=_drain_kde, args=(_kde_route, _kde_cancel), daemon=True)
_kde_worker[0] = worker
worker.start()
except Exception:
_kde_worker[0] = None
_kde_error = "Couldn't start queued typing. Text is retained; try again."
def _queue_for_kde(events):
with _kde_lock:
if len(_kde_queue) + len(events) > KDE_QUEUE_LIMIT:
return False
_kde_queue.extend(events)
if _kde_worker[0] is None and not _kde_error:
_start_kde_locked()
return True
def _drain_kde(route, cancel):
"""One sender, in order. In-flight text stays in the bounded queue until sent.
Errors retain it for an explicit retry; only a headset switch cancels it."""
global _kde_until, _kde_error
deadline = time.time() + KDE_WAIT
try:
while True:
with _kde_lock:
if route is not _kde_route:
return
batch = _kde_queue[:INPUT_BATCH_LIMIT]
if not batch:
_kde_worker[0] = None
return
wait = _typed_until + TYPING_SETTLE - time.time()
_kde_until = time.time() + 3600 # held while in flight
if wait > 0 and cancel.wait(wait):
return
# Match route()/reset_pad_route()'s lock order. send() may start an
# agent and take _route_lock again to capture its immutable route.
with _route_lock, _pad_send_lock:
with _kde_lock:
if route is not _kde_route:
return
status = _input.send(batch)
with _kde_lock:
if route is not _kde_route:
return
if status.get("sent"):
del _kde_queue[:len(batch)]
_kde_until = time.time() + len(batch) * 0.03 + TYPING_SETTLE
deadline = time.time() + KDE_WAIT
continue
_kde_until = 0.0
if status.get("state") == "error" or time.time() > deadline:
_kde_error = "Queued typing didn't reach the input service. Text is retained; try again."
_kde_worker[0] = None
return
if cancel.wait(0.4):
return
except Exception:
# A stale worker must never retire the new route's worker or alter its
# queue/timing. Retain unsent text and let the page report/retry failure.
with _kde_lock:
if route is _kde_route:
_kde_until = 0.0
_kde_worker[0] = None
_kde_error = "Queued typing failed. Text is retained; try again."
return _input.send([input_event(e) for e in events])
# ---- touch: the headset's panels, through gamescope's own input (frame_touch.py) ----
@@ -3534,7 +3302,7 @@ class Handler(BaseHTTPRequestHandler):
elif path == "/api/panels":
self.send_json(panels())
elif path in ("/api/touch", "/api/input"):
agent = _touch # both controls use gamescope; KDE Connect starts only for Unicode
agent = _touch if path == "/api/touch" else _input
if parse_qs(url.query).get("start") == ["1"]:
# Validate and reserve before start() captures its immutable
# route and schedules a worker. Status-only polls stay free.
@@ -3542,11 +3310,9 @@ class Handler(BaseHTTPRequestHandler):
if LINK and not meant:
raise Failure("Control start needs the intended X-Frame-Device headset", 400)
with working(meant):
result = start_pad() if path == "/api/input" else agent.send([])
result = agent.send([])
else:
result = dict(agent.status)
if path == "/api/input":
result = pad_status(result)
self.send_json(result)
elif path == "/api/job":
self.send_json(job_status(url.query))
@@ -3871,11 +3637,6 @@ def main():
help="stop cleanly when stdin closes (the app closes it on quit; "
"Windows has no SIGTERM to catch)")
args = ap.parse_args()
global LINK, _ONE_SERVER
if not LOCAL and not PRIVATE:
# Refuse before starting daemon workers: the losing process exits through
# Python teardown, which can race TLS work and crash instead of exit 1.
_ONE_SERVER = one_server()
httpd = LoopbackServer(("127.0.0.1", args.port), Handler)
sweep_tmp()
if os.environ.get('FRAME_COMPAT_SEARCH_REPORTS') != '0':
@@ -3884,7 +3645,10 @@ def main():
threading.Thread(target=apk_search.warm, daemon=True).start() # big indexes download before the first search
frame_telemetry.start()
frame_contact.start()
global LINK, _ONE_SERVER
if not LOCAL:
if not PRIVATE: # a private server only uses the headsets (see one_server)
_ONE_SERVER = one_server()
LINK = frame_link.Link(frame_devices.Registry(), env_alias=FRAME if FRAME_FROM_ENV else None,
mux_base=MUX_BASE, control=CONTROL, apply=route, explain=unreachable)
LINK.work_lock, LINK.work = _work_lock, lambda: _work[0]