mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-24 06:01:39 +02:00
Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
eb125b59c7 | ||
|
|
08cb267ce4 | ||
|
|
0119544c3e | ||
|
|
2a9e7dcabf | ||
|
|
5a9099dddb | ||
|
|
ec4086db69 | ||
|
|
5a7c3541f8 | ||
|
|
525040162e | ||
|
|
21e771d16e |
No files matched your search
@@ -2,6 +2,20 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
- Keep the full host-key recovery explanation readable in Devices, including
|
||||
long addresses, and remove doubled punctuation from the connection tooltip
|
||||
(issue #87). Refresh stays within the header and switching headsets selects
|
||||
the new headset's details.
|
||||
|
||||
- Fix experimental PC screen capture on Windows builds whose streaming libraries
|
||||
lack Windows Graphics Capture. Whole screens use Desktop Duplication; individual
|
||||
windows are hidden with an explanation instead of failing when shown. A Windows
|
||||
VM streamed a real desktop to the Frame; physical input remains unverified.
|
||||
|
||||
- Add read-only battery health in Tools: charging state, temperature, cycle count,
|
||||
and available capacity with explicit unknown states. The parser was checked
|
||||
on a real Frame; the connected desktop UI was tested with synthetic readings.
|
||||
|
||||
- Add Tools → Game storage: read-only headset filesystem capacity and Steam
|
||||
library/game install-size breakdown, with unknown sizes, partial inventories
|
||||
and permission/missing-drive states. Includes instructions for Steam's own
|
||||
|
||||
@@ -226,7 +226,7 @@ Frame's software fits together, all checked against a real headset and labelled
|
||||
| [Sideloading Linux and Windows games](docs/sideloading.md) | A .zip, folder or .exe as a Steam Devkit Game, runtime detection |
|
||||
| [Install links for websites](docs/web-install.md) | `frame-control://install` links and manifests, the rules, a button to paste |
|
||||
| [VR comfort and HUD](docs/vr-utilities.md) · [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build |
|
||||
| [PC in the headset](docs/pc-in-headset.md) | Experimental, Windows only, off until you turn it on: your PC's windows and screens as panels in the Frame. Not yet tried on a real Windows desktop or headset |
|
||||
| [PC in the headset](docs/pc-in-headset.md) | Experimental, Windows only, off until you turn it on: a whole PC screen as a panel in the Frame. Tested in a Windows VM; individual windows need a WGC-enabled build |
|
||||
| [Mac in the headset](docs/mac-in-headset.md) | Mac windows and screens as panels in the Frame, with laser and keyboard input |
|
||||
| [VR mods and custom songs](docs/mods.md) | UEVR for Unreal games from Frame Control (install, play in VR, remove), the per-game table, Beat Saber blockers |
|
||||
| [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes |
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "frame-control",
|
||||
"version": "0.4.10",
|
||||
"version": "0.4.11",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "frame-control",
|
||||
"version": "0.4.10",
|
||||
"version": "0.4.11",
|
||||
"license": "MIT",
|
||||
"devDependencies": {
|
||||
"electron": "^44.4.5",
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "frame-control",
|
||||
"productName": "Frame Control",
|
||||
"version": "0.4.10",
|
||||
"version": "0.4.11",
|
||||
"description": "Desktop app for managing a Valve Steam Frame over SSH",
|
||||
"private": true,
|
||||
"main": "main.js",
|
||||
|
||||
@@ -22,6 +22,15 @@ FC_API int fc_has_element(const char *name) {
|
||||
GstElementFactory *f=gst_element_factory_find(name);
|
||||
if(!f)return 0;gst_object_unref(f);return 1;
|
||||
}
|
||||
FC_API int fc_has_property(const char *element, const char *property) {
|
||||
/* Probe the same initialized GStreamer instance that capture uses. The
|
||||
* bundled MSYS2 plugin can omit optional WGC properties entirely. */
|
||||
GstElement *source=gst_element_factory_make(element,NULL);
|
||||
if(!source)return 0;
|
||||
int found=g_object_class_find_property(G_OBJECT_GET_CLASS(source),property)!=NULL;
|
||||
gst_object_unref(source);
|
||||
return found;
|
||||
}
|
||||
static GstPadProbeReturn probe(GstPad *pad,GstPadProbeInfo *info,gpointer data) {
|
||||
Capture *c=data; GstBuffer *b=GST_PAD_PROBE_INFO_BUFFER(info);
|
||||
if(!b)return GST_PAD_PROBE_OK;
|
||||
|
||||
@@ -97,6 +97,12 @@ The app rereads the gateway every 5 seconds and Tailscale's state every
|
||||
|
||||
## The connection, stage by stage
|
||||
|
||||
When a connection fails, the header pill keeps its compact offline state and
|
||||
retry countdown; its tooltip and connection dialog contain the full reason.
|
||||
Devices shows a short reason in the headset row (with the full reason in its
|
||||
tooltip) and a wrapping recovery explanation in the selected headset's details.
|
||||
**Use this headset** also selects that headset's details.
|
||||
|
||||
The connector runs in the server (`frame_link.Link`) and moves through:
|
||||
|
||||
1. **Checking this computer's network**: gateway, Wi-Fi, this computer's IP, Tailscale.
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
# Windows VM and real Frame test, 2026-10-24
|
||||
|
||||
The v0.4.10 installer could enumerate windows/screens and encode a synthetic
|
||||
Media Foundation stream, but every desktop capture failed at pipeline parsing:
|
||||
`no property "capture-api" in element "d3d11screencapturesrc"`.
|
||||
MSYS2's GStreamer 1.28.8 package disables WGC. After the fix, the same installed
|
||||
bundle captured the actual Windows desktop with Desktop Duplication and sent
|
||||
H.264 to a real Frame: 219 sent frame records, 215 decoded and drawn before the
|
||||
snapshot. Individual windows are explicitly unavailable with this bundle.
|
||||
|
||||
The base was `e9cbb8fd5555f5cfceeba1babadf2d7d35dcd3be` (`v0.4.10`, origin/main,
|
||||
including merged #121), in isolated branch `t3/test-windows-vm-headset`.
|
||||
|
||||
## Evidence
|
||||
|
||||
- [Capture before the fix](windows-vm-2026-10-24/capture-before.json): five
|
||||
synthetic H.264 frames and five JPEG frames; zero desktop/window frames,
|
||||
with the missing-property error for both codecs.
|
||||
- [Actual bundled properties](windows-vm-2026-10-24/plugin-properties.json):
|
||||
`monitor-handle` exists; `capture-api`, `window-handle`, `show-border` do not.
|
||||
[MSYS2 package build](https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-gst-plugins-bad/PKGBUILD)
|
||||
sets `-Dd3d11-wgc=disabled`. Upstream
|
||||
[installs those properties conditionally](https://github.com/GStreamer/gstreamer/blob/main/subprojects/gst-plugins-bad/sys/d3d11/gstd3d11screencapturesrc.cpp).
|
||||
- [Capture after the fix](windows-vm-2026-10-24/capture-after.json): the real
|
||||
display produces five H.264 and five JPEG frames. Window sources are hidden
|
||||
with an explicit reason, and the display selects `capture_api=dxgi`.
|
||||
- [Raw locked headset session](windows-vm-2026-10-24/headset-session.json):
|
||||
Windows helper, Media Foundation encoder, reverse SSH tunnel, production
|
||||
panel launcher and shared viewer. The image below was captured from that
|
||||
viewer's own X window on the Frame, rather than the VM's console.
|
||||
- [Windows tests](windows-vm-2026-10-24/windows-tests.txt) and
|
||||
[guest system details](windows-vm-2026-10-24/windows-system.txt).
|
||||
|
||||

|
||||
|
||||
## Host and VM prerequisites
|
||||
|
||||
lxso2: Intel Core i7-8700, 12 logical CPUs, VT-x/KVM; Docker 29.1.3 and Compose
|
||||
2.40.3. Before starting: 31 GiB RAM, 15 GiB available; 147 GiB free disk.
|
||||
Swap was already full. No other VM or service workload was started or stopped.
|
||||
Docker listed no Windows VM, and ports 2222/8006/13389 were unused.
|
||||
|
||||
The execution sandbox hid KVM and denied Docker/network access. Checks outside
|
||||
it found `/dev/kvm`, `/dev/net/tun` and the Docker daemon. Direct user access to
|
||||
KVM was denied, but a disposable Docker container with `--device /dev/kvm`
|
||||
opened it successfully; no host permissions changed.
|
||||
|
||||
Created only `frame-winvm-priority3`, labelled
|
||||
`frame-control.task=priority3-windows-test`: 4 vCPUs, 8 GiB guest RAM, 32 GiB
|
||||
growable disk, no restart, 150-second stop timeout. SSH 2222 and console 8006
|
||||
bind only 127.0.0.1; RDP is unpublished. No GPU passthrough. Image:
|
||||
`dockurr/windows@sha256:16fd7c39a3e48bb946d29c59dc6015942e5b900a1fa5d7e5552053bfc48ee2ea`
|
||||
(Windows for Docker v6.06, QEMU v11.1.1).
|
||||
|
||||
The container downloaded Windows 11 from Microsoft, with no activation changes.
|
||||
It took about 16 minutes to reach the first desktop. Guest build 26200.8037,
|
||||
Red Hat VirtIO GPU DOD driver 100.103.104.30200. The old WindowsProductName
|
||||
registry field still says Windows 10 Pro; the actual desktop/setup is Windows
|
||||
11. After installing the app, the 32 GiB disk had about 9.3 GB free.
|
||||
|
||||
Official `Frame-Control-Setup-x64.exe` v0.4.10 SHA-256:
|
||||
`30165bc5aa39f941cacc697d195dc5c30b01ad0b9ffb472da8a18a5220549202`.
|
||||
Its installer exited 0. The VM stream used an intermediate Python/UI-only
|
||||
fix with the released DLL bundle and bundled Python 3.12.14. A later CI failure
|
||||
showed that probing GStreamer through a separately loaded DLL could see a
|
||||
different, uninitialized instance. The final fix probes optional properties
|
||||
through the capture DLL itself. Windows CI checks that native change; the VM
|
||||
stream was not repeated with the final DLL.
|
||||
|
||||
OEM setup added OpenSSH Server, PowerShell and a dedicated public login key;
|
||||
password authentication is disabled. Its config needed an explicit
|
||||
`AuthorizedKeysFile C:/ProgramData/ssh/administrators_authorized_keys` before
|
||||
key login worked. The default informational share was replaced with a private
|
||||
writable staging share inside this container. These were test-VM setup steps,
|
||||
not app defects. Host setup files and the stopped VM disk are retained under
|
||||
`/tmp/frame-winvm-priority3/`; no private keys are committed or printed.
|
||||
The Frame test key was generated inside Windows and its private key stayed
|
||||
there. Only its public key was temporarily authorized on the Frame. After
|
||||
testing, the guest key and scheduled task were removed; `docker stop -t 150`
|
||||
shut down the dedicated VM. Inspection confirmed `exited`, restart `no`
|
||||
(container exit 143; logs show the ACPI shutdown signal, waiting for Windows,
|
||||
and `Shutdown completed!`). The disk is retained
|
||||
for reuse; the VM is not running.
|
||||
|
||||
## Checks
|
||||
|
||||
- Baseline PC-focused suite: 24 tests passed, one native class skipped.
|
||||
- After fix: full Linux unittest suite **1260 tests, OK, 15 skipped**, 144.366 s;
|
||||
focused final PC suite **27 tests, OK, one native class skipped**. The C
|
||||
controller's frozen 600-state trace ran on Linux.
|
||||
- Actual Windows installer bundle with the intermediate Python/UI fix:
|
||||
**33 tests, OK, four skipped** (no C compiler, Linux PipeWire test, two
|
||||
CI-only x264 tests). Its GObject property probe, host/encoder check and
|
||||
x264-absence check ran. The final native probe is checked in the PR's
|
||||
Windows CI build, not by this VM run.
|
||||
- Browser check with the real production HTML/JavaScript and mocked Windows API
|
||||
responses: DXGI whole-screen Show remains available, the limitation appears
|
||||
once, WGC window sources retain Show, off hides controls, and the limitation
|
||||
text is escaped. This used a static asset server, without headset API access.
|
||||
- `node --test app/test/*.test.js`: passed, three test files.
|
||||
- `node --test site/test/*.test.mjs`: passed, one test file.
|
||||
- The first full Linux run inside the sandbox failed (11 failures, 139 errors),
|
||||
including loopback socket `PermissionError`. The successful full runs were
|
||||
outside that socket restriction.
|
||||
- An initial guest test archive omitted `apk-catalog`, causing one import
|
||||
error; the complete archive produced the 33-test passing Windows run.
|
||||
|
||||
Raw local logs: `/tmp/frame-windows-{pc,app,site}-tests.log`,
|
||||
`/tmp/frame-windows-full-tests-unrestricted.log`,
|
||||
`/tmp/frame-windows-full-fixed-tests.log`, and
|
||||
`/tmp/frame-winvm-priority3/`. The committed artifacts above preserve the
|
||||
capture, Windows test and headset evidence independently of those local files.
|
||||
|
||||
## Shared Frame session and cleanup
|
||||
|
||||
Read-only checks first: SteamOS 0.5.6, BUILD_ID `20261015.6141207`, battery 99%,
|
||||
Not charging (another supply reports Full). The lock was free.
|
||||
|
||||
Both attempts acquired `/tmp/frame-test.lock` atomically before any key or
|
||||
viewer changes, and removed only the lock acquired by this task. The first,
|
||||
35-second attempt failed because the test wrapper sent CRLF script input and
|
||||
left SSH stdin open. Production already sends script bytes and closes stdin;
|
||||
correcting the wrapper resolved both. Its key was removed and lock released.
|
||||
|
||||
The successful session held the lock **03:09:36–03:10:05 UTC (29 seconds)**.
|
||||
It used a dedicated `pc-view-priority3` profile and port range 47960–47964,
|
||||
leaving existing viewers/profiles alone. The app's Windows `PCView` captured
|
||||
`display:65537`, 1280×800, Light preset (30 fps ceiling), and opened panel
|
||||
`valve.steam.desktopgame.2001396898`. The viewer reported H.264 software decode,
|
||||
218 received frames, 215 decoded/drawn before the snapshot (219 sent records).
|
||||
|
||||
This mostly idle, roughly ten-second sample reported 19 fps and content delay
|
||||
p50/p95 51.6/75.2 ms. These are observations, not a performance-target pass.
|
||||
|
||||
Cleanup stopped the helper and tunnel, closed the own viewer, removed its two
|
||||
possible profile paths and log, then removed the exact public-key line. The
|
||||
`authorized_keys` SHA-256 was identical before/after. A follow-up login from
|
||||
Windows with that key failed with exit 255. Separate checks found no task-owned
|
||||
viewer processes, the profile absent, and the shared lock released. Battery
|
||||
remained 99%; the sampled gamescope focus properties matched before/after.
|
||||
No reboot, global settings changes, game launch or Steam/SteamVR stop occurred.
|
||||
|
||||
## Limits
|
||||
|
||||
This verifies a Windows VM's desktop capture, Media Foundation encoding and
|
||||
real Frame viewer decode/rendering. It does not verify physical-PC hardware
|
||||
encoding, WGC or individual-window capture, mixed DPI, multiple displays,
|
||||
physical laser alignment/clicks, keyboard layouts, resize/minimise, busy-scene
|
||||
performance, bitrate/encoder restarts, or worn-headset comfort. No input was
|
||||
injected during the real Frame session. The installed app’s Tools navigation was not exercised; its streaming
|
||||
controller, tunnel and viewer were exercised directly. No independent review was performed.
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"python": "3.12.14 (main, Sep 24 2026, 17:57:31) [MSC v.1944 64 bit (AMD64)]",
|
||||
"resources": "C:\\Users\\frame\\AppData\\Local\\Programs\\Frame Control\\resources",
|
||||
"elements": {
|
||||
"d3d11screencapturesrc": true,
|
||||
"mfh264enc": true,
|
||||
"jpegenc": true,
|
||||
"x264enc": false
|
||||
},
|
||||
"windowCaptureReason": "Individual windows are unavailable in this build. Choose a whole screen instead.",
|
||||
"windows": [],
|
||||
"displays": [
|
||||
{
|
||||
"src": "display:65537",
|
||||
"name": "Display 1",
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 1280,
|
||||
"h": 800,
|
||||
"capture_api": "dxgi"
|
||||
}
|
||||
],
|
||||
"runs": [
|
||||
{
|
||||
"source": "test",
|
||||
"codec": "h264",
|
||||
"frames": 5,
|
||||
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"last_bytes": 154
|
||||
},
|
||||
{
|
||||
"source": "test",
|
||||
"codec": "jpeg",
|
||||
"frames": 5,
|
||||
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"last_bytes": 7336
|
||||
},
|
||||
{
|
||||
"source": "display:65537",
|
||||
"codec": "h264",
|
||||
"frames": 5,
|
||||
"pipeline": "d3d11screencapturesrc show-cursor=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"last_bytes": 65
|
||||
},
|
||||
{
|
||||
"source": "display:65537",
|
||||
"codec": "jpeg",
|
||||
"frames": 5,
|
||||
"pipeline": "d3d11screencapturesrc show-cursor=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"last_bytes": 113339
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
{
|
||||
"python": "3.12.14 (main, Sep 24 2026, 17:57:31) [MSC v.1944 64 bit (AMD64)]",
|
||||
"resources": "C:\\Users\\frame\\AppData\\Local\\Programs\\Frame Control\\resources",
|
||||
"elements": {
|
||||
"d3d11screencapturesrc": true,
|
||||
"mfh264enc": true,
|
||||
"jpegenc": true,
|
||||
"x264enc": false
|
||||
},
|
||||
"windows": [
|
||||
{
|
||||
"src": "window:131836",
|
||||
"id": 131836,
|
||||
"title": "Administrator: C:\\WINDOWS\\system32\\cmd.exe - powershell -noprofile -executionpolicy bypass -file \\\\host.lan\\data\\bootstrap.ps1",
|
||||
"app": "Windows",
|
||||
"w": 1115,
|
||||
"h": 628,
|
||||
"x": 51,
|
||||
"y": 52
|
||||
},
|
||||
{
|
||||
"src": "window:327926",
|
||||
"id": 327926,
|
||||
"title": "Administrator: Install",
|
||||
"app": "Windows",
|
||||
"w": 979,
|
||||
"h": 512,
|
||||
"x": 85,
|
||||
"y": 78
|
||||
},
|
||||
{
|
||||
"src": "window:65856",
|
||||
"id": 65856,
|
||||
"title": "Program Manager",
|
||||
"app": "Windows",
|
||||
"w": 1280,
|
||||
"h": 800,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
}
|
||||
],
|
||||
"displays": [
|
||||
{
|
||||
"src": "display:65537",
|
||||
"name": "Display 1",
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 1280,
|
||||
"h": 800
|
||||
}
|
||||
],
|
||||
"runs": [
|
||||
{
|
||||
"source": "test",
|
||||
"codec": "h264",
|
||||
"frames": 5,
|
||||
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"last_bytes": 154
|
||||
},
|
||||
{
|
||||
"source": "test",
|
||||
"codec": "jpeg",
|
||||
"frames": 5,
|
||||
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"last_bytes": 7336
|
||||
},
|
||||
{
|
||||
"source": "display:65537",
|
||||
"codec": "h264",
|
||||
"frames": 0,
|
||||
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
|
||||
},
|
||||
{
|
||||
"source": "display:65537",
|
||||
"codec": "jpeg",
|
||||
"frames": 0,
|
||||
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
|
||||
},
|
||||
{
|
||||
"source": "window:131836",
|
||||
"codec": "h264",
|
||||
"frames": 0,
|
||||
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true window-handle=131836 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
|
||||
},
|
||||
{
|
||||
"source": "window:131836",
|
||||
"codec": "jpeg",
|
||||
"frames": 0,
|
||||
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true window-handle=131836 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
|
||||
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
Binary file not shown.
|
After Width: | Height: | Size: 429 KiB |
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"gstreamer": "1.28.8",
|
||||
"plugins": [
|
||||
"coreelements",
|
||||
"videotestsrc",
|
||||
"videoconvertscale",
|
||||
"app",
|
||||
"jpeg",
|
||||
"videoparsersbad",
|
||||
"d3d11",
|
||||
"mediafoundation"
|
||||
],
|
||||
"packages": [
|
||||
"mingw-w64-ucrt-x86_64-gst-plugins-bad 1.28.8-2",
|
||||
"mingw-w64-ucrt-x86_64-gst-plugins-bad-libs 1.28.8-2",
|
||||
"mingw-w64-ucrt-x86_64-gstreamer 1.28.8-1"
|
||||
],
|
||||
"elements": {
|
||||
"d3d11screencapturesrc": {
|
||||
"name": "The name of the object",
|
||||
"parent": "The parent of the object",
|
||||
"blocksize": "Size in bytes to read per buffer (-1 = default)",
|
||||
"num-buffers": "Number of buffers to output before sending EOS (-1 = unlimited)",
|
||||
"typefind": "Run typefind before negotiating (deprecated, non-functional)",
|
||||
"do-timestamp": "Apply current stream time to buffers",
|
||||
"automatic-eos": "Automatically EOS when the segment is done",
|
||||
"monitor-index": "Zero-based index for monitor to capture (-1 = primary monitor)",
|
||||
"monitor-handle": "A HMONITOR handle of monitor to capture",
|
||||
"show-cursor": "Whether to show mouse cursor",
|
||||
"crop-x": "Horizontal coordinate of top left corner for the screen capture area",
|
||||
"crop-y": "Vertical coordinate of top left corner for the screen capture area",
|
||||
"crop-width": "Width of screen capture area (0 = maximum)",
|
||||
"crop-height": "Height of screen capture area (0 = maximum)"
|
||||
},
|
||||
"d3d12screencapturesrc": null,
|
||||
"mfh264enc": {
|
||||
"name": "The name of the object",
|
||||
"parent": "The parent of the object",
|
||||
"qos": "Handle Quality-of-Service events from downstream",
|
||||
"min-force-key-unit-interval": "Minimum interval between force-keyunit requests in nanoseconds",
|
||||
"bitrate": "Bitrate in kbit/sec",
|
||||
"rc-mode": "Rate Control Mode",
|
||||
"adaptive-mode": "Adaptive Mode",
|
||||
"vbv-buffer-size": "VBV(HRD) Buffer Size in bytes (0 = MFT default)",
|
||||
"max-bitrate": "The maximum bitrate applied when rc-mode is \"pcvbr\" in kbit/sec",
|
||||
"quality-vs-speed": "Quality and speed tradeoff, [0, 33]: Low complexity, [34, 66]: Medium complexity, [67, 100]: High complexity",
|
||||
"cabac": "Enable CABAC entropy coding",
|
||||
"sps-id": "The SPS id to use",
|
||||
"pps-id": "The PPS id to use",
|
||||
"bframes": "The maximum number of consecutive B frames",
|
||||
"gop-size": "The number of pictures from one GOP header to the next. Depending on GPU vendor implementation, zero gop-size might produce only one keyframe at the beginning (-1 for automatic)",
|
||||
"threads": "The number of worker threads used by a encoder, (0 = MFT default)",
|
||||
"content-type": "Indicates the type of video content",
|
||||
"qp": "QP applied when rc-mode is \"qvbr\"",
|
||||
"low-latency": "Enable low latency encoding",
|
||||
"min-qp": "The minimum allowed QP applied to all rc-mode",
|
||||
"max-qp": "The maximum allowed QP applied to all rc-mode",
|
||||
"qp-i": "QP applied to I frames",
|
||||
"qp-p": "QP applied to P frames",
|
||||
"qp-b": "QP applied to B frames",
|
||||
"ref": "The number of reference frames",
|
||||
"d3d11-aware": "Whether device can support Direct3D11 interop"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
|
||||
WindowsProductName WindowsVersion OsBuildNumber
|
||||
------------------ -------------- -------------
|
||||
Windows 10 Pro 2009 26200
|
||||
|
||||
|
||||
|
||||
Name DriverVersion Status
|
||||
---- ------------- ------
|
||||
Red Hat VirtIO GPU DOD controller 100.103.104.30200 OK
|
||||
|
||||
|
||||
|
||||
Used Free
|
||||
---- ----
|
||||
23951253504 9314816000
|
||||
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
python.exe : test_original_mac_congestion_and_recovery
|
||||
(test_pc_controller.ControllerTrace.test_original_mac_congestion_and_recovery) ... skipped 'C
|
||||
compiler not installed; native build jobs cover this trace'
|
||||
At line:12 char:1
|
||||
+ & $python -B -m unittest discover -s tests -p 'test_pc*.py' -v 2>&1 | ...
|
||||
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
+ CategoryInfo : NotSpecified: (test_original_m...ver this trace':String) [], Remote
|
||||
Exception
|
||||
+ FullyQualifiedErrorId : NativeCommandError
|
||||
|
||||
test_controller_is_inert_after_close
|
||||
(test_pcview.AdapterTests.test_controller_is_inert_after_close) ... ok
|
||||
test_dxgi_only_bundle_captures_displays_without_optional_wgc_properties (test_pcview.AdapterTests.
|
||||
test_dxgi_only_bundle_captures_displays_without_optional_wgc_properties) ... ok
|
||||
test_idle_source_is_not_a_stall (test_pcview.AdapterTests.test_idle_source_is_not_a_stall) ... ok
|
||||
test_library_path_prepend_adds_no_empty_entry
|
||||
(test_pcview.AdapterTests.test_library_path_prepend_adds_no_empty_entry) ... ok
|
||||
test_linux_uses_only_portal_fd_and_node
|
||||
(test_pcview.AdapterTests.test_linux_uses_only_portal_fd_and_node) ... ok
|
||||
test_malformed_viewer_input_is_dropped_not_fatal
|
||||
(test_pcview.AdapterTests.test_malformed_viewer_input_is_dropped_not_fatal) ... ok
|
||||
test_pc_card_receives_window_capture_limitation
|
||||
(test_pcview.AdapterTests.test_pc_card_receives_window_capture_limitation) ... ok
|
||||
test_pc_reuses_launcher_and_namespaces_panel_ids
|
||||
(test_pcview.AdapterTests.test_pc_reuses_launcher_and_namespaces_panel_ids) ... ok
|
||||
test_portal_input_clamps_and_releases_held_state
|
||||
(test_pcview.AdapterTests.test_portal_input_clamps_and_releases_held_state) ... ok
|
||||
test_refused_windows_focus_never_clicks_the_covering_app
|
||||
(test_pcview.AdapterTests.test_refused_windows_focus_never_clicks_the_covering_app) ... ok
|
||||
test_session_end_releases_native_input_only_once
|
||||
(test_pcview.AdapterTests.test_session_end_releases_native_input_only_once) ... ok
|
||||
test_stats_keep_capture_time_and_bound_records
|
||||
(test_pcview.AdapterTests.test_stats_keep_capture_time_and_bound_records) ... ok
|
||||
test_windows_release_and_negative_monitor_origin
|
||||
(test_pcview.AdapterTests.test_windows_release_and_negative_monitor_origin) ... ok
|
||||
test_windows_sources_follow_actual_wgc_capabilities
|
||||
(test_pcview.AdapterTests.test_windows_sources_follow_actual_wgc_capabilities) ... ok
|
||||
test_windows_uses_wgc_and_media_foundation
|
||||
(test_pcview.AdapterTests.test_windows_uses_wgc_and_media_foundation) ... ok
|
||||
test_never_offered_off_windows_even_when_turned_on
|
||||
(test_pcview.GatingTests.test_never_offered_off_windows_even_when_turned_on) ... ok
|
||||
test_off_by_default_starts_nothing (test_pcview.GatingTests.test_off_by_default_starts_nothing)
|
||||
... ok
|
||||
test_only_windows_gets_the_pc_host (test_pcview.GatingTests.test_only_windows_gets_the_pc_host)
|
||||
... ok
|
||||
test_server_enable_action (test_pcview.GatingTests.test_server_enable_action) ... ok
|
||||
test_turning_off_stops_streams_and_the_helper
|
||||
(test_pcview.GatingTests.test_turning_off_stops_streams_and_the_helper) ... ok
|
||||
test_turning_on_checks_for_the_bundled_libraries
|
||||
(test_pcview.GatingTests.test_turning_on_checks_for_the_bundled_libraries) ... ok
|
||||
test_expiry_and_stop_before_redeem (test_pcview.GrantsTests.test_expiry_and_stop_before_redeem)
|
||||
... ok
|
||||
test_source_bound_retry_ack_and_stop
|
||||
(test_pcview.GrantsTests.test_source_bound_retry_ack_and_stop) ... ok
|
||||
test_agent_reports_its_host_and_encoder
|
||||
(test_pcview.NativeTests.test_agent_reports_its_host_and_encoder) ... ok
|
||||
test_auth_and_real_h264_stats (test_pcview.NativeTests.test_auth_and_real_h264_stats) ... skipped
|
||||
'x264 is only in CI test bundles, not the installer'
|
||||
test_bundled_pipewire_context_needs_no_system_modules
|
||||
(test_pcview.NativeTests.test_bundled_pipewire_context_needs_no_system_modules) ... skipped
|
||||
'Linux PipeWire client libraries'
|
||||
test_installer_bundle_leaves_out_gpl_x264
|
||||
(test_pcview.NativeTests.test_installer_bundle_leaves_out_gpl_x264) ... ok
|
||||
test_last_raw_picture_survives_a_closed_gate
|
||||
(test_pcview.NativeTests.test_last_raw_picture_survives_a_closed_gate) ... skipped 'x264 is only
|
||||
in CI test bundles, not the installer'
|
||||
test_native_property_probe_checks_the_instantiated_element
|
||||
(test_pcview.NativeTests.test_native_property_probe_checks_the_instantiated_element) ... ok
|
||||
test_card_says_experimental_and_what_is_unverified
|
||||
(test_pcview.PageTests.test_card_says_experimental_and_what_is_unverified) ... ok
|
||||
test_opt_in_is_remembered_with_the_client_preferences
|
||||
(test_pcview.PageTests.test_opt_in_is_remembered_with_the_client_preferences) ... ok
|
||||
test_tool_link_is_windows_only_and_marked_experimental
|
||||
(test_pcview.PageTests.test_tool_link_is_windows_only_and_marked_experimental) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 33 tests in 1.684s
|
||||
|
||||
OK (skipped=4)
|
||||
+31
-13
@@ -1,15 +1,19 @@
|
||||
# PC in the headset (Experimental)
|
||||
|
||||
**Tools → PC in the headset**, on Windows only. It's the Windows version of
|
||||
[Mac in the headset](mac-in-headset.md): a window or a whole screen on your PC
|
||||
becomes its own SteamVR panel in the Frame. Place it with the dashboard's Float
|
||||
[Mac in the headset](mac-in-headset.md): a whole screen on your PC
|
||||
becomes its own SteamVR panel in the Frame. Individual windows are available only
|
||||
with streaming libraries built with Windows Graphics Capture (WGC); the shipped
|
||||
MSYS2 libraries disable WGC, so this build explains that limitation and offers
|
||||
whole screens. Place the panel with the dashboard's Float
|
||||
in World, Move and Size, point and click with the laser, scroll with the
|
||||
thumbstick, and type on the PC's keyboard.
|
||||
|
||||
**It's experimental and not yet tried on a real Windows desktop or with a
|
||||
headset.** It builds and passes its protocol tests on Windows in CI, but desktop
|
||||
capture, Media Foundation encoding and laser input have never run on a real
|
||||
PC. Expect it not to work yet. Reports are welcome through the
|
||||
**It's experimental.** A Windows 11 VM has captured its real desktop with
|
||||
Desktop Duplication and encoded it with Media Foundation and JPEG. Physical PCs,
|
||||
hardware encoding and physical laser input remain unverified. See
|
||||
[the dated evidence](evidence/windows-vm-2026-10-24.md) for results and limits.
|
||||
Reports are welcome through the
|
||||
[feedback form](https://frame-control.pages.dev/feedback/).
|
||||
|
||||
## Turning it on
|
||||
@@ -31,17 +35,18 @@ streaming app. GStreamer is bundled as ordinary libraries.
|
||||
|
||||
| | Windows |
|
||||
|---|---|
|
||||
| Capture | Windows.Graphics.Capture, through GStreamer's `d3d11screencapturesrc capture-api=wgc`, by window (HWND) or screen (HMONITOR) |
|
||||
| Capture | GStreamer's `d3d11screencapturesrc`: Desktop Duplication for whole screens in the shipped build; Windows.Graphics.Capture by window (HWND) or screen (HMONITOR) when that implementation is present |
|
||||
| Encoding | Media Foundation's H.264 encoder (`mfh264enc`), low latency, no B-frames. Whether it uses the GPU depends on the PC; **not verified** |
|
||||
| Input | `SendInput`, mapped through the source's bounds, with per-monitor DPI awareness |
|
||||
|
||||
API choices are documented, not device-verified:
|
||||
API references:
|
||||
[Windows capture](https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture),
|
||||
[GStreamer WGC](https://gstreamer.freedesktop.org/documentation/d3d11/d3d11screencapturesrc.html),
|
||||
[Media Foundation encoder](https://gstreamer.freedesktop.org/documentation/mediafoundation/mfh264enc.html).
|
||||
|
||||
It needs Windows 10 or 11 with Windows.Graphics.Capture and a Media Foundation
|
||||
H.264 encoder (Windows "N" editions need the Media Feature Pack). Clicks and
|
||||
It needs Windows 10 or 11 with Desktop Duplication and a Media Foundation
|
||||
H.264 encoder (Windows "N" editions need the Media Feature Pack). WGC-enabled
|
||||
builds can also capture individual windows. Clicks and
|
||||
keys can't reach administrator apps or the sign-in screen, which an ordinary
|
||||
process can't drive. If Windows won't bring the chosen window to the front,
|
||||
input is refused rather than sent to the app covering it: bring the window
|
||||
@@ -77,6 +82,13 @@ UCRT64 packages. `libraries.json` records the GStreamer version, the plugins,
|
||||
each copied DLL and the exact package versions; licence texts are in
|
||||
`licenses/`.
|
||||
|
||||
MSYS2's `gst-plugins-bad` build disables `d3d11-wgc`. The presence of
|
||||
`d3d11screencapturesrc` alone does not prove WGC support: its `capture-api`,
|
||||
`window-handle` and `show-border` properties are conditional. The helper queries
|
||||
those properties on the instantiated element, advertises only supported
|
||||
sources, and omits all three properties for the DXGI whole-screen pipeline.
|
||||
It never substitutes a screen crop for an unavailable window capture.
|
||||
|
||||
The x264 encoder is **not** in the installer. It's GPL, and Windows encodes
|
||||
with Media Foundation, so only CI's test builds include it
|
||||
(`desktop/build.py --with-x264`). The macOS and Linux installers are unchanged.
|
||||
@@ -127,11 +139,17 @@ python scripts/macview-bench.py run --pc --scenario capture --source window:1234
|
||||
[earlier probe](../bench/results/2026-09-28-c5fc552-dirty-pc-agent-frame-arm64-hairpin.json).
|
||||
- **Verified, #38, on a Mac:** the C controller matched all 600 states of the
|
||||
Swift controller's congestion and recovery trace.
|
||||
- **Not verified:** anything on a real Windows desktop. Windows.Graphics.Capture,
|
||||
Media Foundation (hardware or software), encoder restarts when the bitrate
|
||||
- **Verified, 2026-10-24:** Windows 11 VM build 26200.8037 on lxso2, using
|
||||
the v0.4.10 installer with this fix, captured its actual desktop and streamed
|
||||
H.264 to Frame BUILD_ID `20261015.6141207`: 219 sent frame records, 215 decoded
|
||||
and drawn before the snapshot. The own-panel screenshot shows the Windows
|
||||
desktop. This short, mostly idle run does not establish performance targets.
|
||||
[Evidence, raw traces and cleanup](evidence/windows-vm-2026-10-24.md).
|
||||
- **Not verified:** Windows.Graphics.Capture, physical-PC capture, hardware
|
||||
Media Foundation encoding, encoder restarts when the bitrate
|
||||
changes, mixed-DPI and multi-monitor input alignment, window resize and
|
||||
minimise, non-US keyboard layouts, physical laser input, and how it performs
|
||||
in a worn headset. No numbers are claimed for these. Not tested with a
|
||||
headset from this branch.
|
||||
worn headset. See the dated VM evidence for the separate automated checks.
|
||||
- **Not reviewed independently.** #38 tried an independent review that never
|
||||
completed.
|
||||
+33
-7
@@ -255,15 +255,19 @@ tests the installer's own bundle with its bundled Python and checks x264 isn't
|
||||
in it. `FRAME_PC_REQUIRE_NATIVE=1` and `FRAME_PC_REQUIRE_X264=1` make a missing
|
||||
bundle or encoder fail CI instead of skipping.
|
||||
|
||||
None of this captures a desktop, exercises Media Foundation or a GPU, or
|
||||
checks laser alignment. The [Windows test VM](#windows-test-vm) is the next
|
||||
step for that. See [what's verified](pc-in-headset.md#whats-verified).
|
||||
These automated CI checks do not capture a desktop, exercise Media Foundation
|
||||
or a GPU, or check laser alignment. A [Windows test VM](#windows-test-vm) run
|
||||
on 2026-10-24 captured its actual desktop with Desktop Duplication, encoded
|
||||
with Media Foundation and streamed to a real Frame. It found MSYS2 disables
|
||||
WGC: an element name alone does not prove its optional capture properties
|
||||
exist. Native property-probe tests and source-selection regressions now cover
|
||||
that boundary. [Dated evidence and limits](evidence/windows-vm-2026-10-24.md);
|
||||
[what's verified](pc-in-headset.md#whats-verified).
|
||||
|
||||
## Windows test VM
|
||||
|
||||
The unit tests run on Windows in CI, but the app itself doesn't. Anything that
|
||||
depends on the Windows desktop (Remote Desktop, the installer, the bundled
|
||||
Python, file dialogs) needs a real Windows machine. This is a Windows 11 VM in a
|
||||
Windows CI runs the unit tests and checks installer contents. Desktop capture,
|
||||
installer launch and interactive behavior need a Windows machine. This is a Windows 11 VM in a
|
||||
[dockur/windows](https://github.com/dockur/windows) container on a Linux machine
|
||||
with KVM, driven from the Mac with `scripts/windows-vm.sh`.
|
||||
|
||||
@@ -283,7 +287,14 @@ Setting it up, once, on the Linux machine:
|
||||
`~/.ssh/id_ed25519_winvm` on the Mac) in
|
||||
`C:\ProgramData\ssh\administrators_authorized_keys`.
|
||||
- Give it 4 cores, 8 GB of memory and a 32 GB disk. That's enough for the app
|
||||
and the tests.
|
||||
and the tests. Inspect free memory/disk, KVM access and existing VMs before
|
||||
starting; leave other VMs alone. The first install can take tens of minutes
|
||||
(the 2026-10-24 run took about 16 minutes to reach the desktop), rather than
|
||||
the 1–2 minute startup of an already installed guest.
|
||||
- For PC streaming, check the actual bundled capture properties. MSYS2's
|
||||
GStreamer build disables WGC: this app falls back to Desktop Duplication for
|
||||
whole screens and explains why individual windows are unavailable. A VM
|
||||
capture test does not prove physical GPU encoding or laser alignment.
|
||||
|
||||
Using it, with `WINVM_HOST` set to the Linux machine's ssh alias:
|
||||
|
||||
@@ -426,3 +437,18 @@ files, source preference, missing/invalid sensors and permission errors.
|
||||
`tests/control_center.cjs` covers charging states and optional health fields,
|
||||
alongside existing stale/offline handling. No headset is used. See
|
||||
[battery monitoring](battery-health.md) for sources, units and limitations.
|
||||
|
||||
## Host-key recovery presentation
|
||||
|
||||
`python3 -m unittest discover -s tests -p test_link_ui.py -v` runs the page's
|
||||
connection/banner/selection functions in Node and checks the recovery CSS guard.
|
||||
`test_link.py` covers connection failures and short probe reasons with fake SSH
|
||||
and local sockets. Neither suite uses a real headset.
|
||||
|
||||
Browser verification on 2026-10-24 used the current page's markup, styles and
|
||||
recovery renderers with synthetic changed-key state and two headsets at
|
||||
1280×800. It caught a later `.item .s` rule overriding the earlier wrapping
|
||||
rule, which the original source-only assertion missed. Check computed wrapping,
|
||||
the complete explanation, tooltip punctuation, Refresh bounds and selection
|
||||
after **Use this headset** when changing these styles. This fixture does not
|
||||
exercise Electron integration or a real SSH identity change.
|
||||
@@ -64,6 +64,7 @@ class LinkUI(unittest.TestCase):
|
||||
renderPill();
|
||||
assert.match($('conn').innerHTML, /<b>Office Frame<\/b><span>Offline<span class="via"> · retrying in 12 s<\/span>/);
|
||||
assert.ok($('conn').title.includes(changed.error.message));
|
||||
assert.ok(!$('conn').title.includes('..'), $('conn').title);
|
||||
// Without a short reason: the stage that failed.
|
||||
link.s = {...changed, retry_at: null, error: {stage: 'identity', message: 'Something.'}};
|
||||
renderPill();
|
||||
@@ -120,7 +121,9 @@ class LinkUI(unittest.TestCase):
|
||||
class HeaderLayout(unittest.TestCase):
|
||||
def test_the_devices_tab_explanation_wraps(self):
|
||||
# It was cut to one line with no way to read the rest.
|
||||
self.assertIn(".dev-problem .s { white-space: normal; }", PAGE)
|
||||
# More specific than the later .item .s rule: the old equal-specificity
|
||||
# rule passed this test while the browser still clipped the explanation.
|
||||
self.assertIn(".item.dev-problem .s { white-space: normal; overflow-wrap: anywhere; }", PAGE)
|
||||
self.assertEqual(len(re.findall(r'<div class="item dev-problem" role="alert">', PAGE)), 2)
|
||||
|
||||
def test_the_header_buttons_dont_give_way(self):
|
||||
|
||||
+63
-1
@@ -141,7 +141,7 @@ class PageTests(unittest.TestCase):
|
||||
def test_card_says_experimental_and_what_is_unverified(self):
|
||||
self.assertIn('<span class="tag" id="mvTag" hidden>Experimental</span>', self.page)
|
||||
self.assertIn('<input type="checkbox" id="mvPcOn">', self.page)
|
||||
self.assertIn("hasn't been tried on a real\n Windows desktop or with a headset yet", self.page)
|
||||
self.assertIn('Tested in a Windows VM; physical PCs, hardware encoding and laser clicks remain unverified.', self.page)
|
||||
self.assertIn('docs/pc-in-headset.md', self.page)
|
||||
|
||||
def test_opt_in_is_remembered_with_the_client_preferences(self):
|
||||
@@ -151,6 +151,55 @@ class PageTests(unittest.TestCase):
|
||||
|
||||
|
||||
class AdapterTests(unittest.TestCase):
|
||||
def test_dxgi_only_bundle_captures_displays_without_optional_wgc_properties(self):
|
||||
p = capture.pipeline(dict(src='display:65537', capture_api='dxgi'), 'win32', 'mfh264enc', 640, 360, 30, 1000000)
|
||||
self.assertIn('monitor-handle=65537', p)
|
||||
self.assertNotIn('capture-api=', p)
|
||||
self.assertNotIn('show-border=', p)
|
||||
self.assertIn('mfh264enc', p)
|
||||
for source in (dict(src='window:42', capture_api='dxgi'),
|
||||
dict(src='display:42', capture_api='dxgi ! fakesink')):
|
||||
with self.assertRaises(ValueError):
|
||||
capture.pipeline(source, 'win32', 'mfh264enc', 640, 360, 30, 1000000)
|
||||
|
||||
def test_windows_sources_follow_actual_wgc_capabilities(self):
|
||||
for wgc in (False, True):
|
||||
with self.subTest(wgc=wgc):
|
||||
native, windows = mock.Mock(), mock.Mock()
|
||||
native.has_property.return_value = wgc
|
||||
windows.sources.side_effect = lambda: ([dict(src='window:12', w=800, h=600)],
|
||||
[dict(src='display:4', w=1920, h=1080)])
|
||||
with mock.patch.object(agent.sys, 'platform', 'win32'), \
|
||||
mock.patch.object(agent, 'Windows', return_value=windows):
|
||||
host = agent.Agent(native, 'test', ROOT/'ui/mac-view.html')
|
||||
wins, displays = host.lists()
|
||||
self.assertEqual(len(wins), int(wgc))
|
||||
self.assertEqual(displays[0]['capture_api'], 'wgc' if wgc else 'dxgi')
|
||||
self.assertEqual(host.source('display:4')['capture_api'], displays[0]['capture_api'])
|
||||
if wgc:
|
||||
self.assertEqual(host.source('window:12')['capture_api'], 'wgc')
|
||||
self.assertEqual(host.window_capture_reason, '')
|
||||
else:
|
||||
with self.assertRaisesRegex(ValueError, 'Individual windows are unavailable'):
|
||||
host.source('window:12')
|
||||
self.assertIn('whole screen', host.window_capture_reason)
|
||||
# Synthetic encoder/protocol checks remain available without WGC.
|
||||
self.assertEqual(host.source('test')['src'], 'test')
|
||||
|
||||
def test_pc_card_receives_window_capture_limitation(self):
|
||||
view = frame_pcview.PCView([], lambda *a: None, 'frame')
|
||||
view.enabled = True
|
||||
view.call = mock.Mock(side_effect=lambda path: {
|
||||
'/status': dict(screen=True, accessibility=True, windowCaptureReason=agent.WINDOW_CAPTURE_UNAVAILABLE),
|
||||
'/windows': dict(windows=[]), '/displays': dict(displays=[dict(src='display:4')])}[path])
|
||||
with mock.patch.object(frame_pcview.sys, 'platform', 'win32'), \
|
||||
mock.patch.object(frame_pcview, 'LIBRARY', mock.Mock(is_file=lambda: True)):
|
||||
state = view.state()
|
||||
self.assertTrue(state['available'])
|
||||
self.assertEqual(state['windowCaptureReason'], agent.WINDOW_CAPTURE_UNAVAILABLE)
|
||||
self.assertEqual(state['windows'], [])
|
||||
self.assertEqual(state['displays'], [dict(src='display:4')])
|
||||
|
||||
def test_windows_uses_wgc_and_media_foundation(self):
|
||||
for kind in ('window', 'display'):
|
||||
pipeline = capture.pipeline(dict(src=kind+':42'), 'win32', 'mfh264enc', 640, 360, 30, 1000000)
|
||||
@@ -366,6 +415,19 @@ class NativeTests(unittest.TestCase):
|
||||
def tearDown(self):
|
||||
self.request('/close?k='+self.token, 'POST')
|
||||
|
||||
def test_native_property_probe_checks_the_instantiated_element(self):
|
||||
code = """
|
||||
from frame_pc_capture import Native
|
||||
n = Native()
|
||||
assert n.has_property('fakesink', 'sync')
|
||||
assert not n.has_property('fakesink', 'not-a-gstreamer-property')
|
||||
assert not n.has_property('not-a-gstreamer-element', 'sync')
|
||||
"""
|
||||
env = frame_pcview.PCView([], lambda *a: None, 'frame').agent_environment()
|
||||
env['PYTHONPATH'] = str(ROOT/'ui')
|
||||
result = subprocess.run([sys.executable, '-c', code], env=env, capture_output=True, text=True, timeout=15)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
|
||||
@unittest.skipUnless(sys.platform.startswith('linux'), 'Linux PipeWire client libraries')
|
||||
def test_bundled_pipewire_context_needs_no_system_modules(self):
|
||||
# No portal request, remote connection or desktop capture: create and
|
||||
|
||||
@@ -460,6 +460,7 @@ class MacView:
|
||||
windows = self.call("/windows").get("windows", []) if status.get("screen") else []
|
||||
displays = self.call("/displays").get("displays", [])
|
||||
return {"available": True, "host": self.host, "selecting": status.get("selecting", False),
|
||||
**({"windowCaptureReason": status["windowCaptureReason"]} if status.get("windowCaptureReason") else {}),
|
||||
"selectionError": status.get("selectionError", ""), "screen": status.get("screen", False),
|
||||
"accessibility": status.get("accessibility", False), "streams": status.get("streams", []),
|
||||
"windows": windows, "displays": displays, "tunnel": self.tunnel_up(),
|
||||
|
||||
@@ -29,6 +29,7 @@ from frame_stream_stats import Stats
|
||||
|
||||
# Viewer fields are untrusted JSON: float('x'), int(None), m['t'] missing.
|
||||
MALFORMED = (ValueError, TypeError, KeyError, OverflowError)
|
||||
WINDOW_CAPTURE_UNAVAILABLE = 'Individual windows are unavailable in this build. Choose a whole screen instead.'
|
||||
|
||||
|
||||
class Grants:
|
||||
@@ -386,6 +387,9 @@ class Agent:
|
||||
self.sessions, self.sources = {}, {}
|
||||
self.next_id, self.selecting, self.selection_error = 1, False, ''
|
||||
self.windows = Windows() if sys.platform == 'win32' else None
|
||||
self.wgc = bool(self.windows) and all(native.has_property('d3d11screencapturesrc', p)
|
||||
for p in ('capture-api', 'window-handle', 'show-border'))
|
||||
self.window_capture_reason = WINDOW_CAPTURE_UNAVAILABLE if self.windows and not self.wgc else ''
|
||||
self.encoder = 'mfh264enc' if self.windows else 'vah264enc' if native.has('vah264enc') else 'x264enc'
|
||||
self.shutting_down = False
|
||||
self.selection_generation = 0
|
||||
@@ -393,6 +397,10 @@ class Agent:
|
||||
def lists(self):
|
||||
if self.windows:
|
||||
windows, displays = self.windows.sources()
|
||||
if not self.wgc:
|
||||
windows = []
|
||||
for source in windows + displays:
|
||||
source['capture_api'] = 'wgc' if self.wgc else 'dxgi'
|
||||
self.sources = {s['src']: s for s in windows + displays}
|
||||
return windows, displays
|
||||
return [{k: v for k, v in s.items() if k not in ('portal', 'fd', 'node')} for s in self.sources.values()], []
|
||||
@@ -400,6 +408,8 @@ class Agent:
|
||||
def source(self, src):
|
||||
if src == 'test':
|
||||
return dict(src='test', title='Test pattern', w=1280, h=720)
|
||||
if self.window_capture_reason and isinstance(src, str) and src.startswith('window:'):
|
||||
raise ValueError(self.window_capture_reason)
|
||||
self.lists()
|
||||
if src not in self.sources:
|
||||
raise ValueError('Choose a window or screen on this computer first')
|
||||
@@ -498,6 +508,7 @@ class Handler(BaseHTTPRequestHandler):
|
||||
windows, displays = agent.lists()
|
||||
if method == 'GET' and url.path == '/status':
|
||||
data = dict(version=1, host='windows' if agent.windows else 'linux', screen=True,
|
||||
windowCaptureReason=agent.window_capture_reason,
|
||||
accessibility=True, selecting=agent.selecting, selectionError=agent.selection_error,
|
||||
encoder=agent.encoder, finished=[], streams=[dict(id=i, src=s.src,
|
||||
title=s.source.get('title', ''), stats=s.stats.summary(), controller=s.controller.state())
|
||||
|
||||
+11
-1
@@ -38,6 +38,7 @@ class Native:
|
||||
signatures = {
|
||||
'fc_now': (C.c_int64, []), 'fc_gst_init': (None, []),
|
||||
'fc_has_element': (C.c_int, [C.c_char_p]),
|
||||
'fc_has_property': (C.c_int, [C.c_char_p, C.c_char_p]),
|
||||
'fc_new': (C.c_void_p, [C.c_int, C.c_int]), 'fc_free': (None, [C.c_void_p]),
|
||||
'fc_ceiling': (None, [C.c_void_p, C.c_int]),
|
||||
'fc_gate': (C.c_int, [C.c_void_p, C.c_int64, C.c_int]),
|
||||
@@ -69,6 +70,10 @@ class Native:
|
||||
def has(self, name):
|
||||
return bool(self.lib.fc_has_element(name.encode()))
|
||||
|
||||
def has_property(self, element, name):
|
||||
"""Check optional element properties in the initialized capture DLL."""
|
||||
return bool(self.lib.fc_has_property(element.encode(), name.encode()))
|
||||
|
||||
def now(self):
|
||||
return self.lib.fc_now()
|
||||
|
||||
@@ -133,8 +138,13 @@ def pipeline(source, platform, encoder, w, h, fps, bitrate, codec='h264'):
|
||||
kind, ident = source['src'].split(':')
|
||||
if kind not in ('window', 'display') or not re.fullmatch(r'[0-9]+', ident):
|
||||
raise ValueError('Invalid Windows capture source')
|
||||
api = source.get('capture_api', 'wgc')
|
||||
if api not in ('wgc', 'dxgi') or (api == 'dxgi' and kind != 'display'):
|
||||
raise ValueError('This build can only capture whole screens without Windows Graphics Capture')
|
||||
prop = 'window-handle' if kind == 'window' else 'monitor-handle'
|
||||
capture = 'd3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true %s=%d' % (prop, int(ident))
|
||||
# DXGI-only builds don't expose capture-api or show-border at all.
|
||||
mode = ' capture-api=wgc show-border=true' if api == 'wgc' else ''
|
||||
capture = 'd3d11screencapturesrc%s show-cursor=true %s=%d' % (mode, prop, int(ident))
|
||||
else:
|
||||
capture = 'pipewiresrc fd=%d path=%d do-timestamp=true' % (source['fd'], source['node'])
|
||||
# The source gate runs before conversion/encoding. There is no leaky queue
|
||||
|
||||
+8
-7
@@ -175,7 +175,8 @@
|
||||
.dev-panel select, .dlg select { background: rgba(0,0,0,.28); color: var(--text); border: 1px solid transparent; border-radius: 3px;
|
||||
padding: 8px 8px; font: inherit; font-size: 13px; }
|
||||
.addr .t { overflow-wrap: anywhere; }
|
||||
.addr .s, .dev-problem .s { white-space: normal; }
|
||||
.addr .s { white-space: normal; }
|
||||
.item.dev-problem .s { white-space: normal; overflow-wrap: anywhere; }
|
||||
.addr-edit { display: grid; grid-template-columns: 2fr 1fr 1.4fr auto auto; gap: 8px; align-items: center; width: 100%; }
|
||||
.addr-add { display: grid; grid-template-columns: 2fr 1.4fr auto; gap: 8px; margin-top: 12px; }
|
||||
@media (max-width: 700px) { .dev-form, .addr-edit, .addr-add, .adv .dev-form, .conn-add, .conn-edit { grid-template-columns: 1fr; } }
|
||||
@@ -1426,7 +1427,7 @@
|
||||
|
||||
<div class="page" data-page="tools">
|
||||
<div class="workspace-heading"><div><a href="#tools" id="toolBack" class="workspace-link" hidden>← All tools</a><h1 id="toolTitle">Tools</h1><p id="toolSubtitle">Choose a task for your Frame.</p></div></div>
|
||||
<div id="toolDirectory"><section class="tool-group"><h2>Content & files</h2><div class="quick-routes"><a href="#transfer"><span>Send to Frame</span><small>Add files, APKs or your own games</small></a><a href="#media"><span>Media player</span><small>Play your own videos and media</small></a><a href="#linuxApps"><span>Linux apps</span><small>Install and manage desktop apps</small></a></div></section><section class="tool-group"><h2>Play & control</h2><div class="quick-routes"><a href="#capture"><span>Capture & Live</span><small>See and control the headset view</small></a><a href="#pad" title="Type on the headset (T)"><span>Type on the headset</span><small>Keyboard, trackpad and clipboard</small></a><a href="#panels"><span>Panel switcher</span><small>Bring an open panel into view</small></a><a href="#vrTools"><span>VR tools</span><small>Performance HUD and VR utilities</small></a><a href="#comfort"><span>Family & comfort</span><small>Session reminders and breaks</small></a><a href="#macview" id="macToolLink" hidden><span>Mac in the headset</span><small>Share Mac windows and displays</small></a><a href="#macview" id="pcToolLink" hidden><span>PC in the headset</span><small>Experimental: share this PC's windows and screens</small></a></div></section><section class="tool-group"><h2>Your Frame</h2><div class="quick-routes"><a href="#storage"><span>Game storage</span><small>Free space and Steam install sizes</small></a><a href="#backup" id="backupToolLink"><span>Backup & recovery</span><small>Save games and settings, restore after a reset</small></a></div><a class="workspace-link" href="#settings">Connection, display & preferences →</a></section><section class="tool-group"><h2>Assistant</h2><a class="workspace-link" href="/assistant">Open assistant & AI agents →</a><p class="sub">Use your own model endpoint or review a proposed MCP action.</p></section></div>
|
||||
<div id="toolDirectory"><section class="tool-group"><h2>Content & files</h2><div class="quick-routes"><a href="#transfer"><span>Send to Frame</span><small>Add files, APKs or your own games</small></a><a href="#media"><span>Media player</span><small>Play your own videos and media</small></a><a href="#linuxApps"><span>Linux apps</span><small>Install and manage desktop apps</small></a></div></section><section class="tool-group"><h2>Play & control</h2><div class="quick-routes"><a href="#capture"><span>Capture & Live</span><small>See and control the headset view</small></a><a href="#pad" title="Type on the headset (T)"><span>Type on the headset</span><small>Keyboard, trackpad and clipboard</small></a><a href="#panels"><span>Panel switcher</span><small>Bring an open panel into view</small></a><a href="#vrTools"><span>VR tools</span><small>Performance HUD and VR utilities</small></a><a href="#comfort"><span>Family & comfort</span><small>Session reminders and breaks</small></a><a href="#macview" id="macToolLink" hidden><span>Mac in the headset</span><small>Share Mac windows and displays</small></a><a href="#macview" id="pcToolLink" hidden><span>PC in the headset</span><small>Experimental: share this PC's screen</small></a></div></section><section class="tool-group"><h2>Your Frame</h2><div class="quick-routes"><a href="#storage"><span>Game storage</span><small>Free space and Steam install sizes</small></a><a href="#backup" id="backupToolLink"><span>Backup & recovery</span><small>Save games and settings, restore after a reset</small></a></div><a class="workspace-link" href="#settings">Connection, display & preferences →</a></section><section class="tool-group"><h2>Assistant</h2><a class="workspace-link" href="/assistant">Open assistant & AI agents →</a><p class="sub">Use your own model endpoint or review a proposed MCP action.</p></section></div>
|
||||
<div class="grid-3" id="toolWorkspace">
|
||||
<section class="panel" id="vrTools">
|
||||
<div class="shelf-head"><h2>VR comfort and performance</h2><span class="spacer"></span><span class="sub" id="vrUpdated">Waiting for the Frame</span></div>
|
||||
@@ -1723,9 +1724,9 @@
|
||||
<div class="shelf-head"><h2 id="mvTitle">Mac in the headset</h2><span class="tag" id="mvTag" hidden>Experimental</span><span class="count" id="mvCount"></span><span class="spacer"></span>
|
||||
<button class="small" id="mvRefresh">Refresh</button></div>
|
||||
<div id="mvPc" hidden>
|
||||
<p class="hint" style="color:var(--warn)">Experimental. Shows this PC's windows and screens as panels in the
|
||||
headset, with laser and keyboard input. It builds and passes its tests on Windows in CI, but hasn't been tried on a real
|
||||
Windows desktop or with a headset yet: capture, hardware encoding and laser clicks may not work. Clicks and keys
|
||||
<p class="hint" style="color:var(--warn)">Experimental. Shows a whole PC screen as a panel in the
|
||||
headset, with laser and keyboard input. Individual windows are offered only when this build supports them.
|
||||
Tested in a Windows VM; physical PCs, hardware encoding and laser clicks remain unverified. Clicks and keys
|
||||
can't reach administrator apps or the sign-in screen.
|
||||
<a href="https://github.com/saphid/frame-control/blob/main/docs/pc-in-headset.md" target="_blank" rel="noopener noreferrer">What's verified</a></p>
|
||||
<label class="mv-opt"><input type="checkbox" id="mvPcOn"> Turn on PC in the headset
|
||||
@@ -5674,7 +5675,7 @@ async function loadMacView(restart) {
|
||||
$("mvWinCount").textContent = wins.length ? `${wins.length}` : "";
|
||||
$("mvWindows").innerHTML = !s.screen ? `<div class="sub">Windows appear here once Screen Recording is allowed.</div>`
|
||||
: wins.length ? wins.map(w => mvRow(w.src, w.title || w.app, `${w.title ? w.app + " · " : ""}${w.w}×${w.h}`, w.w, w.h, liveAs(w.src), byStream(liveAs(w.src)))).join("")
|
||||
: `<div class="sub">No windows open on this ${pc ? "PC" : "Mac's current desktop"}.</div>`;
|
||||
: `<div class="sub">${pc && s.windowCaptureReason ? esc(s.windowCaptureReason) : `No windows open on this ${pc ? "PC" : "Mac's current desktop"}.`}</div>`;
|
||||
// Keep the numbers fresh while something is in the headset and the card is on screen.
|
||||
clearTimeout(mvTimer);
|
||||
const tick = () => { mvTimer = document.hidden || !box.offsetParent ? setTimeout(tick, 3000) : (loadMacView(), 0); };
|
||||
@@ -6618,7 +6619,7 @@ function renderPill() {
|
||||
}
|
||||
$("conn").innerHTML = `<span class="dot ${dot}"></span><span class="pt"><b>${esc(dev)}</b><span>${esc(word)}` +
|
||||
`${via ? `<span class="via"> · ${esc(via)}</span>` : ""}</span></span>`;
|
||||
$("conn").title = `${dev}: ${detail}. Click to see or change its addresses.`;
|
||||
$("conn").title = `${dev}: ${detail.replace(/[.\s]+$/, "")}. Click to see or change its addresses.`;
|
||||
return true;
|
||||
}
|
||||
window.connPill = renderPill;
|
||||
|
||||
Reference in new issue
Block a user