Compare commits

..
Author SHA1 Message Date
Alex Southwell eb125b59c7 Merge pull request #131 from saphid/fix/issue-87-recovery-polish
Fix remaining host-key recovery UI clipping and punctuation
2026-10-24 03:59:21 +00:00
SAPHID 08cb267ce4 Fix remaining host-key recovery clipping and tooltip punctuation 2026-10-24 03:52:38 +00:00
Alex Southwell 0119544c3e Merge pull request #130 from saphid/release/0.4.11
Frame Control 0.4.11
2026-10-24 03:42:24 +00:00
SAPHID 2a9e7dcabf Bump Frame Control to 0.4.11 2026-10-24 03:34:25 +00:00
Alex Southwell 5a9099dddb Merge pull request #129 from saphid/t3/test-windows-vm-headset
Fix Windows PC screen capture with the bundled GStreamer libraries
2026-10-24 03:33:31 +00:00
SAPHID ec4086db69 Probe capture properties in initialized native adapter 2026-10-24 03:25:28 +00:00
SAPHID 5a7c3541f8 Merge current main into Windows capture fix 2026-10-24 03:20:54 +00:00
SAPHID 525040162e Fix Windows PC screen capture with DXGI-only GStreamer bundles 2026-10-24 03:17:11 +00:00
Alex Southwell 21e771d16e Merge pull request #128 from saphid/t3/headset-game-storage
Show headset space and read-only Steam game storage
2026-10-24 03:15:37 +00:00
22 changed files with 5234 additions and 34 deletions

No files matched your search

+14
View File
@@ -2,6 +2,20 @@
## Unreleased
- Keep the full host-key recovery explanation readable in Devices, including
long addresses, and remove doubled punctuation from the connection tooltip
(issue #87). Refresh stays within the header and switching headsets selects
the new headset's details.
- Fix experimental PC screen capture on Windows builds whose streaming libraries
lack Windows Graphics Capture. Whole screens use Desktop Duplication; individual
windows are hidden with an explanation instead of failing when shown. A Windows
VM streamed a real desktop to the Frame; physical input remains unverified.
- Add read-only battery health in Tools: charging state, temperature, cycle count,
and available capacity with explicit unknown states. The parser was checked
on a real Frame; the connected desktop UI was tested with synthetic readings.
- Add Tools → Game storage: read-only headset filesystem capacity and Steam
library/game install-size breakdown, with unknown sizes, partial inventories
and permission/missing-drive states. Includes instructions for Steam's own
+1 -1
View File
@@ -226,7 +226,7 @@ Frame's software fits together, all checked against a real headset and labelled
| [Sideloading Linux and Windows games](docs/sideloading.md) | A .zip, folder or .exe as a Steam Devkit Game, runtime detection |
| [Install links for websites](docs/web-install.md) | `frame-control://install` links and manifests, the rules, a button to paste |
| [VR comfort and HUD](docs/vr-utilities.md) · [Steam games](docs/steam-games.md) · [VR video](docs/vr-video.md) · [WebXR in Chromium](docs/webxr-chromium.md) | Installing and buying, watching VR180/360, the Chromium build |
| [PC in the headset](docs/pc-in-headset.md) | Experimental, Windows only, off until you turn it on: your PC's windows and screens as panels in the Frame. Not yet tried on a real Windows desktop or headset |
| [PC in the headset](docs/pc-in-headset.md) | Experimental, Windows only, off until you turn it on: a whole PC screen as a panel in the Frame. Tested in a Windows VM; individual windows need a WGC-enabled build |
| [Mac in the headset](docs/mac-in-headset.md) | Mac windows and screens as panels in the Frame, with laser and keyboard input |
| [VR mods and custom songs](docs/mods.md) | UEVR for Unreal games from Frame Control (install, play in VR, remove), the per-game table, Beat Saber blockers |
| [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes |
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "frame-control",
"version": "0.4.10",
"version": "0.4.11",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "frame-control",
"version": "0.4.10",
"version": "0.4.11",
"license": "MIT",
"devDependencies": {
"electron": "^44.4.5",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "frame-control",
"productName": "Frame Control",
"version": "0.4.10",
"version": "0.4.11",
"description": "Desktop app for managing a Valve Steam Frame over SSH",
"private": true,
"main": "main.js",
+9
View File
@@ -22,6 +22,15 @@ FC_API int fc_has_element(const char *name) {
GstElementFactory *f=gst_element_factory_find(name);
if(!f)return 0;gst_object_unref(f);return 1;
}
FC_API int fc_has_property(const char *element, const char *property) {
/* Probe the same initialized GStreamer instance that capture uses. The
* bundled MSYS2 plugin can omit optional WGC properties entirely. */
GstElement *source=gst_element_factory_make(element,NULL);
if(!source)return 0;
int found=g_object_class_find_property(G_OBJECT_GET_CLASS(source),property)!=NULL;
gst_object_unref(source);
return found;
}
static GstPadProbeReturn probe(GstPad *pad,GstPadProbeInfo *info,gpointer data) {
Capture *c=data; GstBuffer *b=GST_PAD_PROBE_INFO_BUFFER(info);
if(!b)return GST_PAD_PROBE_OK;
+6
View File
@@ -97,6 +97,12 @@ The app rereads the gateway every 5 seconds and Tailscale's state every
## The connection, stage by stage
When a connection fails, the header pill keeps its compact offline state and
retry countdown; its tooltip and connection dialog contain the full reason.
Devices shows a short reason in the headset row (with the full reason in its
tooltip) and a wrapping recovery explanation in the selected headset's details.
**Use this headset** also selects that headset's details.
The connector runs in the server (`frame_link.Link`) and moves through:
1. **Checking this computer's network**: gateway, Wi-Fi, this computer's IP, Tailscale.
+151
View File
@@ -0,0 +1,151 @@
# Windows VM and real Frame test, 2026-10-24
The v0.4.10 installer could enumerate windows/screens and encode a synthetic
Media Foundation stream, but every desktop capture failed at pipeline parsing:
`no property "capture-api" in element "d3d11screencapturesrc"`.
MSYS2's GStreamer 1.28.8 package disables WGC. After the fix, the same installed
bundle captured the actual Windows desktop with Desktop Duplication and sent
H.264 to a real Frame: 219 sent frame records, 215 decoded and drawn before the
snapshot. Individual windows are explicitly unavailable with this bundle.
The base was `e9cbb8fd5555f5cfceeba1babadf2d7d35dcd3be` (`v0.4.10`, origin/main,
including merged #121), in isolated branch `t3/test-windows-vm-headset`.
## Evidence
- [Capture before the fix](windows-vm-2026-10-24/capture-before.json): five
synthetic H.264 frames and five JPEG frames; zero desktop/window frames,
with the missing-property error for both codecs.
- [Actual bundled properties](windows-vm-2026-10-24/plugin-properties.json):
`monitor-handle` exists; `capture-api`, `window-handle`, `show-border` do not.
[MSYS2 package build](https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-gst-plugins-bad/PKGBUILD)
sets `-Dd3d11-wgc=disabled`. Upstream
[installs those properties conditionally](https://github.com/GStreamer/gstreamer/blob/main/subprojects/gst-plugins-bad/sys/d3d11/gstd3d11screencapturesrc.cpp).
- [Capture after the fix](windows-vm-2026-10-24/capture-after.json): the real
display produces five H.264 and five JPEG frames. Window sources are hidden
with an explicit reason, and the display selects `capture_api=dxgi`.
- [Raw locked headset session](windows-vm-2026-10-24/headset-session.json):
Windows helper, Media Foundation encoder, reverse SSH tunnel, production
panel launcher and shared viewer. The image below was captured from that
viewer's own X window on the Frame, rather than the VM's console.
- [Windows tests](windows-vm-2026-10-24/windows-tests.txt) and
[guest system details](windows-vm-2026-10-24/windows-system.txt).
![Windows VM desktop decoded in the Frame's own viewer](windows-vm-2026-10-24/headset-view.png)
## Host and VM prerequisites
lxso2: Intel Core i7-8700, 12 logical CPUs, VT-x/KVM; Docker 29.1.3 and Compose
2.40.3. Before starting: 31 GiB RAM, 15 GiB available; 147 GiB free disk.
Swap was already full. No other VM or service workload was started or stopped.
Docker listed no Windows VM, and ports 2222/8006/13389 were unused.
The execution sandbox hid KVM and denied Docker/network access. Checks outside
it found `/dev/kvm`, `/dev/net/tun` and the Docker daemon. Direct user access to
KVM was denied, but a disposable Docker container with `--device /dev/kvm`
opened it successfully; no host permissions changed.
Created only `frame-winvm-priority3`, labelled
`frame-control.task=priority3-windows-test`: 4 vCPUs, 8 GiB guest RAM, 32 GiB
growable disk, no restart, 150-second stop timeout. SSH 2222 and console 8006
bind only 127.0.0.1; RDP is unpublished. No GPU passthrough. Image:
`dockurr/windows@sha256:16fd7c39a3e48bb946d29c59dc6015942e5b900a1fa5d7e5552053bfc48ee2ea`
(Windows for Docker v6.06, QEMU v11.1.1).
The container downloaded Windows 11 from Microsoft, with no activation changes.
It took about 16 minutes to reach the first desktop. Guest build 26200.8037,
Red Hat VirtIO GPU DOD driver 100.103.104.30200. The old WindowsProductName
registry field still says Windows 10 Pro; the actual desktop/setup is Windows
11. After installing the app, the 32 GiB disk had about 9.3 GB free.
Official `Frame-Control-Setup-x64.exe` v0.4.10 SHA-256:
`30165bc5aa39f941cacc697d195dc5c30b01ad0b9ffb472da8a18a5220549202`.
Its installer exited 0. The VM stream used an intermediate Python/UI-only
fix with the released DLL bundle and bundled Python 3.12.14. A later CI failure
showed that probing GStreamer through a separately loaded DLL could see a
different, uninitialized instance. The final fix probes optional properties
through the capture DLL itself. Windows CI checks that native change; the VM
stream was not repeated with the final DLL.
OEM setup added OpenSSH Server, PowerShell and a dedicated public login key;
password authentication is disabled. Its config needed an explicit
`AuthorizedKeysFile C:/ProgramData/ssh/administrators_authorized_keys` before
key login worked. The default informational share was replaced with a private
writable staging share inside this container. These were test-VM setup steps,
not app defects. Host setup files and the stopped VM disk are retained under
`/tmp/frame-winvm-priority3/`; no private keys are committed or printed.
The Frame test key was generated inside Windows and its private key stayed
there. Only its public key was temporarily authorized on the Frame. After
testing, the guest key and scheduled task were removed; `docker stop -t 150`
shut down the dedicated VM. Inspection confirmed `exited`, restart `no`
(container exit 143; logs show the ACPI shutdown signal, waiting for Windows,
and `Shutdown completed!`). The disk is retained
for reuse; the VM is not running.
## Checks
- Baseline PC-focused suite: 24 tests passed, one native class skipped.
- After fix: full Linux unittest suite **1260 tests, OK, 15 skipped**, 144.366 s;
focused final PC suite **27 tests, OK, one native class skipped**. The C
controller's frozen 600-state trace ran on Linux.
- Actual Windows installer bundle with the intermediate Python/UI fix:
**33 tests, OK, four skipped** (no C compiler, Linux PipeWire test, two
CI-only x264 tests). Its GObject property probe, host/encoder check and
x264-absence check ran. The final native probe is checked in the PR's
Windows CI build, not by this VM run.
- Browser check with the real production HTML/JavaScript and mocked Windows API
responses: DXGI whole-screen Show remains available, the limitation appears
once, WGC window sources retain Show, off hides controls, and the limitation
text is escaped. This used a static asset server, without headset API access.
- `node --test app/test/*.test.js`: passed, three test files.
- `node --test site/test/*.test.mjs`: passed, one test file.
- The first full Linux run inside the sandbox failed (11 failures, 139 errors),
including loopback socket `PermissionError`. The successful full runs were
outside that socket restriction.
- An initial guest test archive omitted `apk-catalog`, causing one import
error; the complete archive produced the 33-test passing Windows run.
Raw local logs: `/tmp/frame-windows-{pc,app,site}-tests.log`,
`/tmp/frame-windows-full-tests-unrestricted.log`,
`/tmp/frame-windows-full-fixed-tests.log`, and
`/tmp/frame-winvm-priority3/`. The committed artifacts above preserve the
capture, Windows test and headset evidence independently of those local files.
## Shared Frame session and cleanup
Read-only checks first: SteamOS 0.5.6, BUILD_ID `20261015.6141207`, battery 99%,
Not charging (another supply reports Full). The lock was free.
Both attempts acquired `/tmp/frame-test.lock` atomically before any key or
viewer changes, and removed only the lock acquired by this task. The first,
35-second attempt failed because the test wrapper sent CRLF script input and
left SSH stdin open. Production already sends script bytes and closes stdin;
correcting the wrapper resolved both. Its key was removed and lock released.
The successful session held the lock **03:09:36–03:10:05 UTC (29 seconds)**.
It used a dedicated `pc-view-priority3` profile and port range 47960–47964,
leaving existing viewers/profiles alone. The app's Windows `PCView` captured
`display:65537`, 1280×800, Light preset (30 fps ceiling), and opened panel
`valve.steam.desktopgame.2001396898`. The viewer reported H.264 software decode,
218 received frames, 215 decoded/drawn before the snapshot (219 sent records).
This mostly idle, roughly ten-second sample reported 19 fps and content delay
p50/p95 51.6/75.2 ms. These are observations, not a performance-target pass.
Cleanup stopped the helper and tunnel, closed the own viewer, removed its two
possible profile paths and log, then removed the exact public-key line. The
`authorized_keys` SHA-256 was identical before/after. A follow-up login from
Windows with that key failed with exit 255. Separate checks found no task-owned
viewer processes, the profile absent, and the shared lock released. Battery
remained 99%; the sampled gamescope focus properties matched before/after.
No reboot, global settings changes, game launch or Steam/SteamVR stop occurred.
## Limits
This verifies a Windows VM's desktop capture, Media Foundation encoding and
real Frame viewer decode/rendering. It does not verify physical-PC hardware
encoding, WGC or individual-window capture, mixed DPI, multiple displays,
physical laser alignment/clicks, keyboard layouts, resize/minimise, busy-scene
performance, bitrate/encoder restarts, or worn-headset comfort. No input was
injected during the real Frame session. The installed app’s Tools navigation was not exercised; its streaming
controller, tunnel and viewer were exercised directly. No independent review was performed.
@@ -0,0 +1,53 @@
{
"python": "3.12.14 (main, Sep 24 2026, 17:57:31) [MSC v.1944 64 bit (AMD64)]",
"resources": "C:\\Users\\frame\\AppData\\Local\\Programs\\Frame Control\\resources",
"elements": {
"d3d11screencapturesrc": true,
"mfh264enc": true,
"jpegenc": true,
"x264enc": false
},
"windowCaptureReason": "Individual windows are unavailable in this build. Choose a whole screen instead.",
"windows": [],
"displays": [
{
"src": "display:65537",
"name": "Display 1",
"x": 0,
"y": 0,
"w": 1280,
"h": 800,
"capture_api": "dxgi"
}
],
"runs": [
{
"source": "test",
"codec": "h264",
"frames": 5,
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
"last_bytes": 154
},
{
"source": "test",
"codec": "jpeg",
"frames": 5,
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
"last_bytes": 7336
},
{
"source": "display:65537",
"codec": "h264",
"frames": 5,
"pipeline": "d3d11screencapturesrc show-cursor=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
"last_bytes": 65
},
{
"source": "display:65537",
"codec": "jpeg",
"frames": 5,
"pipeline": "d3d11screencapturesrc show-cursor=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
"last_bytes": 113339
}
]
}
@@ -0,0 +1,96 @@
{
"python": "3.12.14 (main, Sep 24 2026, 17:57:31) [MSC v.1944 64 bit (AMD64)]",
"resources": "C:\\Users\\frame\\AppData\\Local\\Programs\\Frame Control\\resources",
"elements": {
"d3d11screencapturesrc": true,
"mfh264enc": true,
"jpegenc": true,
"x264enc": false
},
"windows": [
{
"src": "window:131836",
"id": 131836,
"title": "Administrator: C:\\WINDOWS\\system32\\cmd.exe - powershell -noprofile -executionpolicy bypass -file \\\\host.lan\\data\\bootstrap.ps1",
"app": "Windows",
"w": 1115,
"h": 628,
"x": 51,
"y": 52
},
{
"src": "window:327926",
"id": 327926,
"title": "Administrator: Install",
"app": "Windows",
"w": 979,
"h": 512,
"x": 85,
"y": 78
},
{
"src": "window:65856",
"id": 65856,
"title": "Program Manager",
"app": "Windows",
"w": 1280,
"h": 800,
"x": 0,
"y": 0
}
],
"displays": [
{
"src": "display:65537",
"name": "Display 1",
"x": 0,
"y": 0,
"w": 1280,
"h": 800
}
],
"runs": [
{
"source": "test",
"codec": "h264",
"frames": 5,
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
"last_bytes": 154
},
{
"source": "test",
"codec": "jpeg",
"frames": 5,
"pipeline": "videotestsrc name=source is-live=true pattern=ball ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
"last_bytes": 7336
},
{
"source": "display:65537",
"codec": "h264",
"frames": 0,
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
},
{
"source": "display:65537",
"codec": "jpeg",
"frames": 0,
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true monitor-handle=65537 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
},
{
"source": "window:131836",
"codec": "h264",
"frames": 0,
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true window-handle=131836 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360,format=NV12 ! mfh264enc name=enc low-latency=true bframes=0 gop-size=60 bitrate=1000 ! h264parse config-interval=-1 ! video/x-h264,stream-format=byte-stream,alignment=au ! appsink name=out sync=false max-buffers=2 drop=false",
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
},
{
"source": "window:131836",
"codec": "jpeg",
"frames": 0,
"pipeline": "d3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true window-handle=131836 ! video/x-raw,framerate=30/1 ! queue name=raw_queue leaky=downstream max-size-buffers=1 max-size-bytes=0 max-size-time=0 ! identity name=gate ! videoconvert ! videoscale add-borders=false ! video/x-raw,width=640,height=360 ! jpegenc name=enc quality=80 ! appsink name=out sync=false max-buffers=2 drop=false",
"error": "no property \"capture-api\" in element \"d3d11screencapturesrc\""
}
]
}
File diff suppressed because it is too large. Load diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 429 KiB

@@ -0,0 +1,65 @@
{
"gstreamer": "1.28.8",
"plugins": [
"coreelements",
"videotestsrc",
"videoconvertscale",
"app",
"jpeg",
"videoparsersbad",
"d3d11",
"mediafoundation"
],
"packages": [
"mingw-w64-ucrt-x86_64-gst-plugins-bad 1.28.8-2",
"mingw-w64-ucrt-x86_64-gst-plugins-bad-libs 1.28.8-2",
"mingw-w64-ucrt-x86_64-gstreamer 1.28.8-1"
],
"elements": {
"d3d11screencapturesrc": {
"name": "The name of the object",
"parent": "The parent of the object",
"blocksize": "Size in bytes to read per buffer (-1 = default)",
"num-buffers": "Number of buffers to output before sending EOS (-1 = unlimited)",
"typefind": "Run typefind before negotiating (deprecated, non-functional)",
"do-timestamp": "Apply current stream time to buffers",
"automatic-eos": "Automatically EOS when the segment is done",
"monitor-index": "Zero-based index for monitor to capture (-1 = primary monitor)",
"monitor-handle": "A HMONITOR handle of monitor to capture",
"show-cursor": "Whether to show mouse cursor",
"crop-x": "Horizontal coordinate of top left corner for the screen capture area",
"crop-y": "Vertical coordinate of top left corner for the screen capture area",
"crop-width": "Width of screen capture area (0 = maximum)",
"crop-height": "Height of screen capture area (0 = maximum)"
},
"d3d12screencapturesrc": null,
"mfh264enc": {
"name": "The name of the object",
"parent": "The parent of the object",
"qos": "Handle Quality-of-Service events from downstream",
"min-force-key-unit-interval": "Minimum interval between force-keyunit requests in nanoseconds",
"bitrate": "Bitrate in kbit/sec",
"rc-mode": "Rate Control Mode",
"adaptive-mode": "Adaptive Mode",
"vbv-buffer-size": "VBV(HRD) Buffer Size in bytes (0 = MFT default)",
"max-bitrate": "The maximum bitrate applied when rc-mode is \"pcvbr\" in kbit/sec",
"quality-vs-speed": "Quality and speed tradeoff, [0, 33]: Low complexity, [34, 66]: Medium complexity, [67, 100]: High complexity",
"cabac": "Enable CABAC entropy coding",
"sps-id": "The SPS id to use",
"pps-id": "The PPS id to use",
"bframes": "The maximum number of consecutive B frames",
"gop-size": "The number of pictures from one GOP header to the next. Depending on GPU vendor implementation, zero gop-size might produce only one keyframe at the beginning (-1 for automatic)",
"threads": "The number of worker threads used by a encoder, (0 = MFT default)",
"content-type": "Indicates the type of video content",
"qp": "QP applied when rc-mode is \"qvbr\"",
"low-latency": "Enable low latency encoding",
"min-qp": "The minimum allowed QP applied to all rc-mode",
"max-qp": "The maximum allowed QP applied to all rc-mode",
"qp-i": "QP applied to I frames",
"qp-p": "QP applied to P frames",
"qp-b": "QP applied to B frames",
"ref": "The number of reference frames",
"d3d11-aware": "Whether device can support Direct3D11 interop"
}
}
}
@@ -0,0 +1,18 @@
WindowsProductName WindowsVersion OsBuildNumber
------------------ -------------- -------------
Windows 10 Pro 2009 26200
Name DriverVersion Status
---- ------------- ------
Red Hat VirtIO GPU DOD controller 100.103.104.30200 OK
Used Free
---- ----
23951253504 9314816000
@@ -0,0 +1,79 @@
python.exe : test_original_mac_congestion_and_recovery
(test_pc_controller.ControllerTrace.test_original_mac_congestion_and_recovery) ... skipped 'C
compiler not installed; native build jobs cover this trace'
At line:12 char:1
+ & $python -B -m unittest discover -s tests -p 'test_pc*.py' -v 2>&1 | ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (test_original_m...ver this trace':String) [], Remote
Exception
+ FullyQualifiedErrorId : NativeCommandError
test_controller_is_inert_after_close
(test_pcview.AdapterTests.test_controller_is_inert_after_close) ... ok
test_dxgi_only_bundle_captures_displays_without_optional_wgc_properties (test_pcview.AdapterTests.
test_dxgi_only_bundle_captures_displays_without_optional_wgc_properties) ... ok
test_idle_source_is_not_a_stall (test_pcview.AdapterTests.test_idle_source_is_not_a_stall) ... ok
test_library_path_prepend_adds_no_empty_entry
(test_pcview.AdapterTests.test_library_path_prepend_adds_no_empty_entry) ... ok
test_linux_uses_only_portal_fd_and_node
(test_pcview.AdapterTests.test_linux_uses_only_portal_fd_and_node) ... ok
test_malformed_viewer_input_is_dropped_not_fatal
(test_pcview.AdapterTests.test_malformed_viewer_input_is_dropped_not_fatal) ... ok
test_pc_card_receives_window_capture_limitation
(test_pcview.AdapterTests.test_pc_card_receives_window_capture_limitation) ... ok
test_pc_reuses_launcher_and_namespaces_panel_ids
(test_pcview.AdapterTests.test_pc_reuses_launcher_and_namespaces_panel_ids) ... ok
test_portal_input_clamps_and_releases_held_state
(test_pcview.AdapterTests.test_portal_input_clamps_and_releases_held_state) ... ok
test_refused_windows_focus_never_clicks_the_covering_app
(test_pcview.AdapterTests.test_refused_windows_focus_never_clicks_the_covering_app) ... ok
test_session_end_releases_native_input_only_once
(test_pcview.AdapterTests.test_session_end_releases_native_input_only_once) ... ok
test_stats_keep_capture_time_and_bound_records
(test_pcview.AdapterTests.test_stats_keep_capture_time_and_bound_records) ... ok
test_windows_release_and_negative_monitor_origin
(test_pcview.AdapterTests.test_windows_release_and_negative_monitor_origin) ... ok
test_windows_sources_follow_actual_wgc_capabilities
(test_pcview.AdapterTests.test_windows_sources_follow_actual_wgc_capabilities) ... ok
test_windows_uses_wgc_and_media_foundation
(test_pcview.AdapterTests.test_windows_uses_wgc_and_media_foundation) ... ok
test_never_offered_off_windows_even_when_turned_on
(test_pcview.GatingTests.test_never_offered_off_windows_even_when_turned_on) ... ok
test_off_by_default_starts_nothing (test_pcview.GatingTests.test_off_by_default_starts_nothing)
... ok
test_only_windows_gets_the_pc_host (test_pcview.GatingTests.test_only_windows_gets_the_pc_host)
... ok
test_server_enable_action (test_pcview.GatingTests.test_server_enable_action) ... ok
test_turning_off_stops_streams_and_the_helper
(test_pcview.GatingTests.test_turning_off_stops_streams_and_the_helper) ... ok
test_turning_on_checks_for_the_bundled_libraries
(test_pcview.GatingTests.test_turning_on_checks_for_the_bundled_libraries) ... ok
test_expiry_and_stop_before_redeem (test_pcview.GrantsTests.test_expiry_and_stop_before_redeem)
... ok
test_source_bound_retry_ack_and_stop
(test_pcview.GrantsTests.test_source_bound_retry_ack_and_stop) ... ok
test_agent_reports_its_host_and_encoder
(test_pcview.NativeTests.test_agent_reports_its_host_and_encoder) ... ok
test_auth_and_real_h264_stats (test_pcview.NativeTests.test_auth_and_real_h264_stats) ... skipped
'x264 is only in CI test bundles, not the installer'
test_bundled_pipewire_context_needs_no_system_modules
(test_pcview.NativeTests.test_bundled_pipewire_context_needs_no_system_modules) ... skipped
'Linux PipeWire client libraries'
test_installer_bundle_leaves_out_gpl_x264
(test_pcview.NativeTests.test_installer_bundle_leaves_out_gpl_x264) ... ok
test_last_raw_picture_survives_a_closed_gate
(test_pcview.NativeTests.test_last_raw_picture_survives_a_closed_gate) ... skipped 'x264 is only
in CI test bundles, not the installer'
test_native_property_probe_checks_the_instantiated_element
(test_pcview.NativeTests.test_native_property_probe_checks_the_instantiated_element) ... ok
test_card_says_experimental_and_what_is_unverified
(test_pcview.PageTests.test_card_says_experimental_and_what_is_unverified) ... ok
test_opt_in_is_remembered_with_the_client_preferences
(test_pcview.PageTests.test_opt_in_is_remembered_with_the_client_preferences) ... ok
test_tool_link_is_windows_only_and_marked_experimental
(test_pcview.PageTests.test_tool_link_is_windows_only_and_marked_experimental) ... ok
----------------------------------------------------------------------
Ran 33 tests in 1.684s
OK (skipped=4)
+31 -13
View File
@@ -1,15 +1,19 @@
# PC in the headset (Experimental)
**Tools → PC in the headset**, on Windows only. It's the Windows version of
[Mac in the headset](mac-in-headset.md): a window or a whole screen on your PC
becomes its own SteamVR panel in the Frame. Place it with the dashboard's Float
[Mac in the headset](mac-in-headset.md): a whole screen on your PC
becomes its own SteamVR panel in the Frame. Individual windows are available only
with streaming libraries built with Windows Graphics Capture (WGC); the shipped
MSYS2 libraries disable WGC, so this build explains that limitation and offers
whole screens. Place the panel with the dashboard's Float
in World, Move and Size, point and click with the laser, scroll with the
thumbstick, and type on the PC's keyboard.
**It's experimental and not yet tried on a real Windows desktop or with a
headset.** It builds and passes its protocol tests on Windows in CI, but desktop
capture, Media Foundation encoding and laser input have never run on a real
PC. Expect it not to work yet. Reports are welcome through the
**It's experimental.** A Windows 11 VM has captured its real desktop with
Desktop Duplication and encoded it with Media Foundation and JPEG. Physical PCs,
hardware encoding and physical laser input remain unverified. See
[the dated evidence](evidence/windows-vm-2026-10-24.md) for results and limits.
Reports are welcome through the
[feedback form](https://frame-control.pages.dev/feedback/).
## Turning it on
@@ -31,17 +35,18 @@ streaming app. GStreamer is bundled as ordinary libraries.
| | Windows |
|---|---|
| Capture | Windows.Graphics.Capture, through GStreamer's `d3d11screencapturesrc capture-api=wgc`, by window (HWND) or screen (HMONITOR) |
| Capture | GStreamer's `d3d11screencapturesrc`: Desktop Duplication for whole screens in the shipped build; Windows.Graphics.Capture by window (HWND) or screen (HMONITOR) when that implementation is present |
| Encoding | Media Foundation's H.264 encoder (`mfh264enc`), low latency, no B-frames. Whether it uses the GPU depends on the PC; **not verified** |
| Input | `SendInput`, mapped through the source's bounds, with per-monitor DPI awareness |
API choices are documented, not device-verified:
API references:
[Windows capture](https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture),
[GStreamer WGC](https://gstreamer.freedesktop.org/documentation/d3d11/d3d11screencapturesrc.html),
[Media Foundation encoder](https://gstreamer.freedesktop.org/documentation/mediafoundation/mfh264enc.html).
It needs Windows 10 or 11 with Windows.Graphics.Capture and a Media Foundation
H.264 encoder (Windows "N" editions need the Media Feature Pack). Clicks and
It needs Windows 10 or 11 with Desktop Duplication and a Media Foundation
H.264 encoder (Windows "N" editions need the Media Feature Pack). WGC-enabled
builds can also capture individual windows. Clicks and
keys can't reach administrator apps or the sign-in screen, which an ordinary
process can't drive. If Windows won't bring the chosen window to the front,
input is refused rather than sent to the app covering it: bring the window
@@ -77,6 +82,13 @@ UCRT64 packages. `libraries.json` records the GStreamer version, the plugins,
each copied DLL and the exact package versions; licence texts are in
`licenses/`.
MSYS2's `gst-plugins-bad` build disables `d3d11-wgc`. The presence of
`d3d11screencapturesrc` alone does not prove WGC support: its `capture-api`,
`window-handle` and `show-border` properties are conditional. The helper queries
those properties on the instantiated element, advertises only supported
sources, and omits all three properties for the DXGI whole-screen pipeline.
It never substitutes a screen crop for an unavailable window capture.
The x264 encoder is **not** in the installer. It's GPL, and Windows encodes
with Media Foundation, so only CI's test builds include it
(`desktop/build.py --with-x264`). The macOS and Linux installers are unchanged.
@@ -127,11 +139,17 @@ python scripts/macview-bench.py run --pc --scenario capture --source window:1234
[earlier probe](../bench/results/2026-09-28-c5fc552-dirty-pc-agent-frame-arm64-hairpin.json).
- **Verified, #38, on a Mac:** the C controller matched all 600 states of the
Swift controller's congestion and recovery trace.
- **Not verified:** anything on a real Windows desktop. Windows.Graphics.Capture,
Media Foundation (hardware or software), encoder restarts when the bitrate
- **Verified, 2026-10-24:** Windows 11 VM build 26200.8037 on lxso2, using
the v0.4.10 installer with this fix, captured its actual desktop and streamed
H.264 to Frame BUILD_ID `20261015.6141207`: 219 sent frame records, 215 decoded
and drawn before the snapshot. The own-panel screenshot shows the Windows
desktop. This short, mostly idle run does not establish performance targets.
[Evidence, raw traces and cleanup](evidence/windows-vm-2026-10-24.md).
- **Not verified:** Windows.Graphics.Capture, physical-PC capture, hardware
Media Foundation encoding, encoder restarts when the bitrate
changes, mixed-DPI and multi-monitor input alignment, window resize and
minimise, non-US keyboard layouts, physical laser input, and how it performs
in a worn headset. No numbers are claimed for these. Not tested with a
headset from this branch.
worn headset. See the dated VM evidence for the separate automated checks.
- **Not reviewed independently.** #38 tried an independent review that never
completed.
+33 -7
View File
@@ -255,15 +255,19 @@ tests the installer's own bundle with its bundled Python and checks x264 isn't
in it. `FRAME_PC_REQUIRE_NATIVE=1` and `FRAME_PC_REQUIRE_X264=1` make a missing
bundle or encoder fail CI instead of skipping.
None of this captures a desktop, exercises Media Foundation or a GPU, or
checks laser alignment. The [Windows test VM](#windows-test-vm) is the next
step for that. See [what's verified](pc-in-headset.md#whats-verified).
These automated CI checks do not capture a desktop, exercise Media Foundation
or a GPU, or check laser alignment. A [Windows test VM](#windows-test-vm) run
on 2026-10-24 captured its actual desktop with Desktop Duplication, encoded
with Media Foundation and streamed to a real Frame. It found MSYS2 disables
WGC: an element name alone does not prove its optional capture properties
exist. Native property-probe tests and source-selection regressions now cover
that boundary. [Dated evidence and limits](evidence/windows-vm-2026-10-24.md);
[what's verified](pc-in-headset.md#whats-verified).
## Windows test VM
The unit tests run on Windows in CI, but the app itself doesn't. Anything that
depends on the Windows desktop (Remote Desktop, the installer, the bundled
Python, file dialogs) needs a real Windows machine. This is a Windows 11 VM in a
Windows CI runs the unit tests and checks installer contents. Desktop capture,
installer launch and interactive behavior need a Windows machine. This is a Windows 11 VM in a
[dockur/windows](https://github.com/dockur/windows) container on a Linux machine
with KVM, driven from the Mac with `scripts/windows-vm.sh`.
@@ -283,7 +287,14 @@ Setting it up, once, on the Linux machine:
`~/.ssh/id_ed25519_winvm` on the Mac) in
`C:\ProgramData\ssh\administrators_authorized_keys`.
- Give it 4 cores, 8 GB of memory and a 32 GB disk. That's enough for the app
and the tests.
and the tests. Inspect free memory/disk, KVM access and existing VMs before
starting; leave other VMs alone. The first install can take tens of minutes
(the 2026-10-24 run took about 16 minutes to reach the desktop), rather than
the 1–2 minute startup of an already installed guest.
- For PC streaming, check the actual bundled capture properties. MSYS2's
GStreamer build disables WGC: this app falls back to Desktop Duplication for
whole screens and explains why individual windows are unavailable. A VM
capture test does not prove physical GPU encoding or laser alignment.
Using it, with `WINVM_HOST` set to the Linux machine's ssh alias:
@@ -426,3 +437,18 @@ files, source preference, missing/invalid sensors and permission errors.
`tests/control_center.cjs` covers charging states and optional health fields,
alongside existing stale/offline handling. No headset is used. See
[battery monitoring](battery-health.md) for sources, units and limitations.
## Host-key recovery presentation
`python3 -m unittest discover -s tests -p test_link_ui.py -v` runs the page's
connection/banner/selection functions in Node and checks the recovery CSS guard.
`test_link.py` covers connection failures and short probe reasons with fake SSH
and local sockets. Neither suite uses a real headset.
Browser verification on 2026-10-24 used the current page's markup, styles and
recovery renderers with synthetic changed-key state and two headsets at
1280×800. It caught a later `.item .s` rule overriding the earlier wrapping
rule, which the original source-only assertion missed. Check computed wrapping,
the complete explanation, tooltip punctuation, Refresh bounds and selection
after **Use this headset** when changing these styles. This fixture does not
exercise Electron integration or a real SSH identity change.
+4 -1
View File
@@ -64,6 +64,7 @@ class LinkUI(unittest.TestCase):
renderPill();
assert.match($('conn').innerHTML, /<b>Office Frame<\/b><span>Offline<span class="via"> · retrying in 12 s<\/span>/);
assert.ok($('conn').title.includes(changed.error.message));
assert.ok(!$('conn').title.includes('..'), $('conn').title);
// Without a short reason: the stage that failed.
link.s = {...changed, retry_at: null, error: {stage: 'identity', message: 'Something.'}};
renderPill();
@@ -120,7 +121,9 @@ class LinkUI(unittest.TestCase):
class HeaderLayout(unittest.TestCase):
def test_the_devices_tab_explanation_wraps(self):
# It was cut to one line with no way to read the rest.
self.assertIn(".dev-problem .s { white-space: normal; }", PAGE)
# More specific than the later .item .s rule: the old equal-specificity
# rule passed this test while the browser still clipped the explanation.
self.assertIn(".item.dev-problem .s { white-space: normal; overflow-wrap: anywhere; }", PAGE)
self.assertEqual(len(re.findall(r'<div class="item dev-problem" role="alert">', PAGE)), 2)
def test_the_header_buttons_dont_give_way(self):
+63 -1
View File
@@ -141,7 +141,7 @@ class PageTests(unittest.TestCase):
def test_card_says_experimental_and_what_is_unverified(self):
self.assertIn('<span class="tag" id="mvTag" hidden>Experimental</span>', self.page)
self.assertIn('<input type="checkbox" id="mvPcOn">', self.page)
self.assertIn("hasn't been tried on a real\n Windows desktop or with a headset yet", self.page)
self.assertIn('Tested in a Windows VM; physical PCs, hardware encoding and laser clicks remain unverified.', self.page)
self.assertIn('docs/pc-in-headset.md', self.page)
def test_opt_in_is_remembered_with_the_client_preferences(self):
@@ -151,6 +151,55 @@ class PageTests(unittest.TestCase):
class AdapterTests(unittest.TestCase):
def test_dxgi_only_bundle_captures_displays_without_optional_wgc_properties(self):
p = capture.pipeline(dict(src='display:65537', capture_api='dxgi'), 'win32', 'mfh264enc', 640, 360, 30, 1000000)
self.assertIn('monitor-handle=65537', p)
self.assertNotIn('capture-api=', p)
self.assertNotIn('show-border=', p)
self.assertIn('mfh264enc', p)
for source in (dict(src='window:42', capture_api='dxgi'),
dict(src='display:42', capture_api='dxgi ! fakesink')):
with self.assertRaises(ValueError):
capture.pipeline(source, 'win32', 'mfh264enc', 640, 360, 30, 1000000)
def test_windows_sources_follow_actual_wgc_capabilities(self):
for wgc in (False, True):
with self.subTest(wgc=wgc):
native, windows = mock.Mock(), mock.Mock()
native.has_property.return_value = wgc
windows.sources.side_effect = lambda: ([dict(src='window:12', w=800, h=600)],
[dict(src='display:4', w=1920, h=1080)])
with mock.patch.object(agent.sys, 'platform', 'win32'), \
mock.patch.object(agent, 'Windows', return_value=windows):
host = agent.Agent(native, 'test', ROOT/'ui/mac-view.html')
wins, displays = host.lists()
self.assertEqual(len(wins), int(wgc))
self.assertEqual(displays[0]['capture_api'], 'wgc' if wgc else 'dxgi')
self.assertEqual(host.source('display:4')['capture_api'], displays[0]['capture_api'])
if wgc:
self.assertEqual(host.source('window:12')['capture_api'], 'wgc')
self.assertEqual(host.window_capture_reason, '')
else:
with self.assertRaisesRegex(ValueError, 'Individual windows are unavailable'):
host.source('window:12')
self.assertIn('whole screen', host.window_capture_reason)
# Synthetic encoder/protocol checks remain available without WGC.
self.assertEqual(host.source('test')['src'], 'test')
def test_pc_card_receives_window_capture_limitation(self):
view = frame_pcview.PCView([], lambda *a: None, 'frame')
view.enabled = True
view.call = mock.Mock(side_effect=lambda path: {
'/status': dict(screen=True, accessibility=True, windowCaptureReason=agent.WINDOW_CAPTURE_UNAVAILABLE),
'/windows': dict(windows=[]), '/displays': dict(displays=[dict(src='display:4')])}[path])
with mock.patch.object(frame_pcview.sys, 'platform', 'win32'), \
mock.patch.object(frame_pcview, 'LIBRARY', mock.Mock(is_file=lambda: True)):
state = view.state()
self.assertTrue(state['available'])
self.assertEqual(state['windowCaptureReason'], agent.WINDOW_CAPTURE_UNAVAILABLE)
self.assertEqual(state['windows'], [])
self.assertEqual(state['displays'], [dict(src='display:4')])
def test_windows_uses_wgc_and_media_foundation(self):
for kind in ('window', 'display'):
pipeline = capture.pipeline(dict(src=kind+':42'), 'win32', 'mfh264enc', 640, 360, 30, 1000000)
@@ -366,6 +415,19 @@ class NativeTests(unittest.TestCase):
def tearDown(self):
self.request('/close?k='+self.token, 'POST')
def test_native_property_probe_checks_the_instantiated_element(self):
code = """
from frame_pc_capture import Native
n = Native()
assert n.has_property('fakesink', 'sync')
assert not n.has_property('fakesink', 'not-a-gstreamer-property')
assert not n.has_property('not-a-gstreamer-element', 'sync')
"""
env = frame_pcview.PCView([], lambda *a: None, 'frame').agent_environment()
env['PYTHONPATH'] = str(ROOT/'ui')
result = subprocess.run([sys.executable, '-c', code], env=env, capture_output=True, text=True, timeout=15)
self.assertEqual(result.returncode, 0, result.stderr)
@unittest.skipUnless(sys.platform.startswith('linux'), 'Linux PipeWire client libraries')
def test_bundled_pipewire_context_needs_no_system_modules(self):
# No portal request, remote connection or desktop capture: create and
+1
View File
@@ -460,6 +460,7 @@ class MacView:
windows = self.call("/windows").get("windows", []) if status.get("screen") else []
displays = self.call("/displays").get("displays", [])
return {"available": True, "host": self.host, "selecting": status.get("selecting", False),
**({"windowCaptureReason": status["windowCaptureReason"]} if status.get("windowCaptureReason") else {}),
"selectionError": status.get("selectionError", ""), "screen": status.get("screen", False),
"accessibility": status.get("accessibility", False), "streams": status.get("streams", []),
"windows": windows, "displays": displays, "tunnel": self.tunnel_up(),
+11
View File
@@ -29,6 +29,7 @@ from frame_stream_stats import Stats
# Viewer fields are untrusted JSON: float('x'), int(None), m['t'] missing.
MALFORMED = (ValueError, TypeError, KeyError, OverflowError)
WINDOW_CAPTURE_UNAVAILABLE = 'Individual windows are unavailable in this build. Choose a whole screen instead.'
class Grants:
@@ -386,6 +387,9 @@ class Agent:
self.sessions, self.sources = {}, {}
self.next_id, self.selecting, self.selection_error = 1, False, ''
self.windows = Windows() if sys.platform == 'win32' else None
self.wgc = bool(self.windows) and all(native.has_property('d3d11screencapturesrc', p)
for p in ('capture-api', 'window-handle', 'show-border'))
self.window_capture_reason = WINDOW_CAPTURE_UNAVAILABLE if self.windows and not self.wgc else ''
self.encoder = 'mfh264enc' if self.windows else 'vah264enc' if native.has('vah264enc') else 'x264enc'
self.shutting_down = False
self.selection_generation = 0
@@ -393,6 +397,10 @@ class Agent:
def lists(self):
if self.windows:
windows, displays = self.windows.sources()
if not self.wgc:
windows = []
for source in windows + displays:
source['capture_api'] = 'wgc' if self.wgc else 'dxgi'
self.sources = {s['src']: s for s in windows + displays}
return windows, displays
return [{k: v for k, v in s.items() if k not in ('portal', 'fd', 'node')} for s in self.sources.values()], []
@@ -400,6 +408,8 @@ class Agent:
def source(self, src):
if src == 'test':
return dict(src='test', title='Test pattern', w=1280, h=720)
if self.window_capture_reason and isinstance(src, str) and src.startswith('window:'):
raise ValueError(self.window_capture_reason)
self.lists()
if src not in self.sources:
raise ValueError('Choose a window or screen on this computer first')
@@ -498,6 +508,7 @@ class Handler(BaseHTTPRequestHandler):
windows, displays = agent.lists()
if method == 'GET' and url.path == '/status':
data = dict(version=1, host='windows' if agent.windows else 'linux', screen=True,
windowCaptureReason=agent.window_capture_reason,
accessibility=True, selecting=agent.selecting, selectionError=agent.selection_error,
encoder=agent.encoder, finished=[], streams=[dict(id=i, src=s.src,
title=s.source.get('title', ''), stats=s.stats.summary(), controller=s.controller.state())
+11 -1
View File
@@ -38,6 +38,7 @@ class Native:
signatures = {
'fc_now': (C.c_int64, []), 'fc_gst_init': (None, []),
'fc_has_element': (C.c_int, [C.c_char_p]),
'fc_has_property': (C.c_int, [C.c_char_p, C.c_char_p]),
'fc_new': (C.c_void_p, [C.c_int, C.c_int]), 'fc_free': (None, [C.c_void_p]),
'fc_ceiling': (None, [C.c_void_p, C.c_int]),
'fc_gate': (C.c_int, [C.c_void_p, C.c_int64, C.c_int]),
@@ -69,6 +70,10 @@ class Native:
def has(self, name):
return bool(self.lib.fc_has_element(name.encode()))
def has_property(self, element, name):
"""Check optional element properties in the initialized capture DLL."""
return bool(self.lib.fc_has_property(element.encode(), name.encode()))
def now(self):
return self.lib.fc_now()
@@ -133,8 +138,13 @@ def pipeline(source, platform, encoder, w, h, fps, bitrate, codec='h264'):
kind, ident = source['src'].split(':')
if kind not in ('window', 'display') or not re.fullmatch(r'[0-9]+', ident):
raise ValueError('Invalid Windows capture source')
api = source.get('capture_api', 'wgc')
if api not in ('wgc', 'dxgi') or (api == 'dxgi' and kind != 'display'):
raise ValueError('This build can only capture whole screens without Windows Graphics Capture')
prop = 'window-handle' if kind == 'window' else 'monitor-handle'
capture = 'd3d11screencapturesrc capture-api=wgc show-cursor=true show-border=true %s=%d' % (prop, int(ident))
# DXGI-only builds don't expose capture-api or show-border at all.
mode = ' capture-api=wgc show-border=true' if api == 'wgc' else ''
capture = 'd3d11screencapturesrc%s show-cursor=true %s=%d' % (mode, prop, int(ident))
else:
capture = 'pipewiresrc fd=%d path=%d do-timestamp=true' % (source['fd'], source['node'])
# The source gate runs before conversion/encoding. There is no leaky queue
+8 -7
View File
@@ -175,7 +175,8 @@
.dev-panel select, .dlg select { background: rgba(0,0,0,.28); color: var(--text); border: 1px solid transparent; border-radius: 3px;
padding: 8px 8px; font: inherit; font-size: 13px; }
.addr .t { overflow-wrap: anywhere; }
.addr .s, .dev-problem .s { white-space: normal; }
.addr .s { white-space: normal; }
.item.dev-problem .s { white-space: normal; overflow-wrap: anywhere; }
.addr-edit { display: grid; grid-template-columns: 2fr 1fr 1.4fr auto auto; gap: 8px; align-items: center; width: 100%; }
.addr-add { display: grid; grid-template-columns: 2fr 1.4fr auto; gap: 8px; margin-top: 12px; }
@media (max-width: 700px) { .dev-form, .addr-edit, .addr-add, .adv .dev-form, .conn-add, .conn-edit { grid-template-columns: 1fr; } }
@@ -1426,7 +1427,7 @@
<div class="page" data-page="tools">
<div class="workspace-heading"><div><a href="#tools" id="toolBack" class="workspace-link" hidden>← All tools</a><h1 id="toolTitle">Tools</h1><p id="toolSubtitle">Choose a task for your Frame.</p></div></div>
<div id="toolDirectory"><section class="tool-group"><h2>Content & files</h2><div class="quick-routes"><a href="#transfer"><span>Send to Frame</span><small>Add files, APKs or your own games</small></a><a href="#media"><span>Media player</span><small>Play your own videos and media</small></a><a href="#linuxApps"><span>Linux apps</span><small>Install and manage desktop apps</small></a></div></section><section class="tool-group"><h2>Play & control</h2><div class="quick-routes"><a href="#capture"><span>Capture & Live</span><small>See and control the headset view</small></a><a href="#pad" title="Type on the headset (T)"><span>Type on the headset</span><small>Keyboard, trackpad and clipboard</small></a><a href="#panels"><span>Panel switcher</span><small>Bring an open panel into view</small></a><a href="#vrTools"><span>VR tools</span><small>Performance HUD and VR utilities</small></a><a href="#comfort"><span>Family & comfort</span><small>Session reminders and breaks</small></a><a href="#macview" id="macToolLink" hidden><span>Mac in the headset</span><small>Share Mac windows and displays</small></a><a href="#macview" id="pcToolLink" hidden><span>PC in the headset</span><small>Experimental: share this PC's windows and screens</small></a></div></section><section class="tool-group"><h2>Your Frame</h2><div class="quick-routes"><a href="#storage"><span>Game storage</span><small>Free space and Steam install sizes</small></a><a href="#backup" id="backupToolLink"><span>Backup &amp; recovery</span><small>Save games and settings, restore after a reset</small></a></div><a class="workspace-link" href="#settings">Connection, display &amp; preferences →</a></section><section class="tool-group"><h2>Assistant</h2><a class="workspace-link" href="/assistant">Open assistant &amp; AI agents →</a><p class="sub">Use your own model endpoint or review a proposed MCP action.</p></section></div>
<div id="toolDirectory"><section class="tool-group"><h2>Content & files</h2><div class="quick-routes"><a href="#transfer"><span>Send to Frame</span><small>Add files, APKs or your own games</small></a><a href="#media"><span>Media player</span><small>Play your own videos and media</small></a><a href="#linuxApps"><span>Linux apps</span><small>Install and manage desktop apps</small></a></div></section><section class="tool-group"><h2>Play & control</h2><div class="quick-routes"><a href="#capture"><span>Capture & Live</span><small>See and control the headset view</small></a><a href="#pad" title="Type on the headset (T)"><span>Type on the headset</span><small>Keyboard, trackpad and clipboard</small></a><a href="#panels"><span>Panel switcher</span><small>Bring an open panel into view</small></a><a href="#vrTools"><span>VR tools</span><small>Performance HUD and VR utilities</small></a><a href="#comfort"><span>Family & comfort</span><small>Session reminders and breaks</small></a><a href="#macview" id="macToolLink" hidden><span>Mac in the headset</span><small>Share Mac windows and displays</small></a><a href="#macview" id="pcToolLink" hidden><span>PC in the headset</span><small>Experimental: share this PC's screen</small></a></div></section><section class="tool-group"><h2>Your Frame</h2><div class="quick-routes"><a href="#storage"><span>Game storage</span><small>Free space and Steam install sizes</small></a><a href="#backup" id="backupToolLink"><span>Backup &amp; recovery</span><small>Save games and settings, restore after a reset</small></a></div><a class="workspace-link" href="#settings">Connection, display &amp; preferences →</a></section><section class="tool-group"><h2>Assistant</h2><a class="workspace-link" href="/assistant">Open assistant &amp; AI agents →</a><p class="sub">Use your own model endpoint or review a proposed MCP action.</p></section></div>
<div class="grid-3" id="toolWorkspace">
<section class="panel" id="vrTools">
<div class="shelf-head"><h2>VR comfort and performance</h2><span class="spacer"></span><span class="sub" id="vrUpdated">Waiting for the Frame</span></div>
@@ -1723,9 +1724,9 @@
<div class="shelf-head"><h2 id="mvTitle">Mac in the headset</h2><span class="tag" id="mvTag" hidden>Experimental</span><span class="count" id="mvCount"></span><span class="spacer"></span>
<button class="small" id="mvRefresh">Refresh</button></div>
<div id="mvPc" hidden>
<p class="hint" style="color:var(--warn)">Experimental. Shows this PC's windows and screens as panels in the
headset, with laser and keyboard input. It builds and passes its tests on Windows in CI, but hasn't been tried on a real
Windows desktop or with a headset yet: capture, hardware encoding and laser clicks may not work. Clicks and keys
<p class="hint" style="color:var(--warn)">Experimental. Shows a whole PC screen as a panel in the
headset, with laser and keyboard input. Individual windows are offered only when this build supports them.
Tested in a Windows VM; physical PCs, hardware encoding and laser clicks remain unverified. Clicks and keys
can't reach administrator apps or the sign-in screen.
<a href="https://github.com/saphid/frame-control/blob/main/docs/pc-in-headset.md" target="_blank" rel="noopener noreferrer">What's verified</a></p>
<label class="mv-opt"><input type="checkbox" id="mvPcOn"> Turn on PC in the headset
@@ -5674,7 +5675,7 @@ async function loadMacView(restart) {
$("mvWinCount").textContent = wins.length ? `${wins.length}` : "";
$("mvWindows").innerHTML = !s.screen ? `<div class="sub">Windows appear here once Screen Recording is allowed.</div>`
: wins.length ? wins.map(w => mvRow(w.src, w.title || w.app, `${w.title ? w.app + " · " : ""}${w.w}×${w.h}`, w.w, w.h, liveAs(w.src), byStream(liveAs(w.src)))).join("")
: `<div class="sub">No windows open on this ${pc ? "PC" : "Mac's current desktop"}.</div>`;
: `<div class="sub">${pc && s.windowCaptureReason ? esc(s.windowCaptureReason) : `No windows open on this ${pc ? "PC" : "Mac's current desktop"}.`}</div>`;
// Keep the numbers fresh while something is in the headset and the card is on screen.
clearTimeout(mvTimer);
const tick = () => { mvTimer = document.hidden || !box.offsetParent ? setTimeout(tick, 3000) : (loadMacView(), 0); };
@@ -6618,7 +6619,7 @@ function renderPill() {
}
$("conn").innerHTML = `<span class="dot ${dot}"></span><span class="pt"><b>${esc(dev)}</b><span>${esc(word)}` +
`${via ? `<span class="via"> · ${esc(via)}</span>` : ""}</span></span>`;
$("conn").title = `${dev}: ${detail}. Click to see or change its addresses.`;
$("conn").title = `${dev}: ${detail.replace(/[.\s]+$/, "")}. Click to see or change its addresses.`;
return true;
}
window.connPill = renderPill;