mirror of
https://github.com/saphid/frame-control.git
synced 2026-10-26 14:02:09 +01:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 SAPHID
|
a68c5584a9 |
No files matched your search
@@ -1,11 +1,11 @@
|
||||
# What we have tested
|
||||
|
||||
Release baseline: Frame Control **v0.4.40**, 26 October 2026. Rows also identify verified post-release main changes explicitly. “Real Frame + physical PC” means an actual headset and a non-virtual computer. “VM only” means a Windows virtual machine, sometimes connected to the real Frame. A CI test or a simulated device is **not** a physical hardware test. The date/version column is the last relevant check, not a promise that every new release was retested on that platform. See each link for the exact procedure and limits.
|
||||
Release baseline: Frame Control **v0.4.41**, 26 October 2026. Rows also identify verified post-release main changes explicitly. “Real Frame + physical PC” means an actual headset and a non-virtual computer. “VM only” means a Windows virtual machine, sometimes connected to the real Frame. A CI test or a simulated device is **not** a physical hardware test. The date/version column is the last relevant check, not a promise that every new release was retested on that platform. See each link for the exact procedure and limits.
|
||||
|
||||
| Feature | Real Frame + physical PC tested | VM only | Untested or limited | Last check and proof |
|
||||
|---|---|---|---|---|
|
||||
| Mac desktop release — tested on a real Mac | Physical MacBook Air (MacBookAir9,1), Intel Core i3, macOS 15.7.7 (24G720): official v0.4.39 x64 DMG verified, installed and launched. Native UI, preference persistence across restart (original file restored), bundled Python logs, update check, offline OBS Start/Stop and 116-entry Decky store checked. A 40-byte relay upload/readback/delete passed under coordinator/headset locks | No | No quarantined Safari/Chrome download or Gatekeeper override test; native Open at Login/menu automation unavailable. Direct frame.local had no route and the app refused a stale SSH identity. Relay transfer is not native Mac transfer acceptance. Real OBS/Framey VR unverified. Quit exposed a server cleanup race; fix #284 shipped in v0.4.40 with physical-Mac source regression coverage, but the tagged Intel installer was not launched on physical Intel hardware | 26 Oct, official v0.4.39 tag `fd0b0a9`: [Intel release evidence](evidence/mac-v0439-2026-10-26.md); [earlier ARM64 failure](evidence/mac-desktop-2026-10-26.md) |
|
||||
| Core title smoke test | Linux lxso2 ran the exact v0.4.40 tagged source against the real Frame; all 13 reserved native/Proton title steps passed and cleanup completed | No | The headset was unworn; visible rendering and controller input were not checked. Other feature paths retain their separate limits below | 26 Oct, exact v0.4.40 tag `a4d8073`, Frame OS 0.5.6 build `20261022.6153909`: [tagged release evidence](evidence/v0440-release-2026-10-26.md); [earlier candidate smoke](evidence/v0440-candidate-frame-2026-10-26.md) |
|
||||
| Core title smoke test | Linux lxso2 ran the exact v0.4.41 tagged source against the real Frame; 13/13 reserved title steps passed and cleanup completed | No | The headset was unworn; visible rendering and controller input were not checked. Optional Steam Linux Runtime 4 was absent, so the x86_64 test executable did not actually launch. Other feature paths retain their separate limits below | 26 Oct, exact v0.4.41 tag `b5d50a8`, Frame OS 0.5.6 build `20261022.6153909`: [tagged release evidence](evidence/v0441-release-2026-10-26.md); [earlier candidate smoke](evidence/v0441-candidate-frame-2026-10-26.md) |
|
||||
| Windows desktop in the headset | No physical Windows PC test | Windows 11 VM sent a whole desktop to a real Frame; this used an intermediate fix, not the final native DLL | Physical GPU, capture and controller input; individual windows unavailable in bundled DXGI build | 24 Oct, v0.4.10 installer + candidate fix: [VM log](evidence/windows-vm-2026-10-24.md), [PC guide](pc-in-headset.md) |
|
||||
| Linux desktop in the headset | No | No | Native Linux PC capture is disabled; lxso2 lacked a desktop portal and PipeWire session. The separate Steam `-pipewire` helper is not Frame Control's PC capture | 24 Oct, v0.4.17: [host audit](evidence/linux-pc-host-lxso2-2026-10-24.md), [Steam helper PR #163](https://github.com/saphid/frame-control/pull/163) |
|
||||
| Mac desktop in the headset | A physical Mac sent its screen/window to a real Frame | No | Current v0.4.26 Mac installer and worn-headset controller input have not been repeated | 28 Sep, pre-v0.4 / Frame OS 0.4.1: [Mac guide and measurement](streaming.md#b-show-the-macs-desktop-inside-the-frame) |
|
||||
@@ -24,7 +24,7 @@ Release baseline: Frame Control **v0.4.40**, 26 October 2026. Rows also identify
|
||||
| Open at login | No Frame needed; Linux autostart file toggled on a physical Linux host with an isolated config directory | No | A full login session and physical Windows/macOS host login have not been checked; OS API calls were unit tested | 25 Oct, v0.4.26: [PR #189](https://github.com/saphid/frame-control/pull/189) |
|
||||
| Local Frame Control extensions | Example panel displayed live Frame status in a browser preview; the real Frame served a read-only example in the experimental headset panel | No | No wearer-visible extension view; third-party extensions and physical Windows/macOS extension folders untested | 25 Oct, v0.4.29: [PR #194](https://github.com/saphid/frame-control/pull/194), [PR #201](https://github.com/saphid/frame-control/pull/201), [extension guide](extensions.md) |
|
||||
| Experimental Frame Control SteamVR panel | Frame OS 0.5.6 registered and served the opt-in overlay. A later unattended check safely woke SteamVR and found the panel in stereo compositor and mirror captures; it removed its temporary service and restored power settings | No | No person wore the headset for that check. Physical controller input, scrolling, VR keyboard, comfort and legibility remain unverified. The large status view has browser and packaging checks, not a wearer review | 26 Oct, v0.4.29–0.4.40: [panel PR #201](https://github.com/saphid/frame-control/pull/201), [unattended evidence](evidence/framey-vr-2026-10-25/README.md), [post-#292 unworn sample exercise](evidence/v0440-release-2026-10-26.md#post-tag-292-panel-regression), [earlier standby-gated exercise](evidence/framey-vr-standby-gate-2026-10-26.md), [worn checklist](framey-vr-check.md) |
|
||||
| Experimental Decky Loader manager | Frame OS 0.5.6 ARM64 user-service install/status/update/remove and managed ProtonDB install/update/remove passed earlier. A private Decky 3.3.0 service returned read-only ProtonDB and Steam Status setting RPCs; a later locked check verified the official Notebook default drawing/save/reopen flow and Simple Timer backend events in local Chromium | No | The loader remains opt-in. Installs/updates reject an archive whose Decky name collides with another installed plugin before download and under the headset transaction lock; this safety path has host/fake Frame tests, not a new real-headset collision test. v0.4.40 classifies the 116 exact-version plugins as 1 works (Notebook default flow only), 49 partial, 55 broken and 11 not applicable. Post-tag PRs #289 and #293 give 3 works / 48 partial / 54 broken / 11 N/A on current main: TabMaster is partial, and Volume Mixer and Mixer work for tested per-app silent-stream volume control only. Deck Shelves moved to partial after its exact chunked frontend rendered settings in local Chromium; its backend and Steam Home shelves remain unverified. Steamback, PlayTime and Deck Settings became partial after detached read-only library/user snapshots and isolated amd64 backend checks; their official ARM64 backends, automatic behavior and VR panels remain unverified. A read-only Steam bridge returned the Frame country, install folders and system information; IsThereAnyDeal selected that country in local Chromium, but its deployed HTTP endpoint and deal lookup remain unverified. An isolated ARM64 LocalSend core answered its info endpoint, but the official x86-only plugin archive and file transfers remain blocked. Volume Mixer and Mixer controlled a named silent stream through real ARM64 Decky with independent Pulse readbacks; Volume Boost default-sink reads matched, but real writes were denied. Audible output, Bluetooth playback, internal-node controls and worn VR remain unverified. Most official backends and VR controls remain unverified. Physical controller use and wearer-visible official panels remain unverified | 26 Oct, v0.4.31–0.4.40: [managed store PR #208](https://github.com/saphid/frame-control/pull/208), [name-collision PR #269](https://github.com/saphid/frame-control/pull/269), [catalog PR #266](https://github.com/saphid/frame-control/pull/266), [backend event evidence](evidence/framey-events-backend-2026-10-26.json), [Steam read PR #274](https://github.com/saphid/frame-control/pull/274), [LocalSend evidence PR #275](https://github.com/saphid/frame-control/pull/275), [local audio evidence](evidence/framey-audio-backends-2026-10-26.md), [ARM64 audio evidence](evidence/framey-audio-arm64-2026-10-26.md), [post-release library evidence](evidence/framey-library-context-2026-10-26.md), [compatibility table](framey-compatibility.md) |
|
||||
| Experimental Decky Loader manager | Frame OS 0.5.6 ARM64 user-service install/status/update/remove and managed ProtonDB install/update/remove passed earlier. A private Decky 3.3.0 service returned read-only ProtonDB and Steam Status setting RPCs; a later locked check verified the official Notebook default drawing/save/reopen flow and Simple Timer backend events in local Chromium | No | The loader remains opt-in. Installs/updates reject an archive whose Decky name collides with another installed plugin before download and under the headset transaction lock; this safety path has host/fake Frame tests, not a new real-headset collision test. v0.4.41 classifies the 116 exact-version plugins as 3 works, 48 partial, 54 broken and 11 not applicable. Notebook works for its default flow; Volume Mixer and Mixer work for tested per-app silent-stream volume control only. TabMaster is partial. Deck Shelves moved to partial after its exact chunked frontend rendered settings in local Chromium; its backend and Steam Home shelves remain unverified. Steamback, PlayTime and Deck Settings became partial after detached read-only library/user snapshots and isolated amd64 backend checks; their official ARM64 backends, automatic behavior and VR panels remain unverified. A read-only Steam bridge returned the Frame country, install folders and system information; IsThereAnyDeal selected that country in local Chromium, but its deployed HTTP endpoint and deal lookup remain unverified. An isolated ARM64 LocalSend core answered its info endpoint, but the official x86-only plugin archive and file transfers remain blocked. Volume Mixer and Mixer controlled a named silent stream through real ARM64 Decky with independent Pulse readbacks; Volume Boost default-sink reads matched, but real writes were denied. Audible output, Bluetooth playback, internal-node controls and worn VR remain unverified. Most official backends and VR controls remain unverified. Physical controller use and wearer-visible official panels remain unverified | 26 Oct, v0.4.31–0.4.41: [managed store PR #208](https://github.com/saphid/frame-control/pull/208), [name-collision PR #269](https://github.com/saphid/frame-control/pull/269), [catalog PR #266](https://github.com/saphid/frame-control/pull/266), [backend event evidence](evidence/framey-events-backend-2026-10-26.json), [Steam read PR #274](https://github.com/saphid/frame-control/pull/274), [LocalSend evidence PR #275](https://github.com/saphid/frame-control/pull/275), [local audio evidence](evidence/framey-audio-backends-2026-10-26.md), [ARM64 audio evidence](evidence/framey-audio-arm64-2026-10-26.md), [post-release library evidence](evidence/framey-library-context-2026-10-26.md), [compatibility table](framey-compatibility.md) |
|
||||
| Experimental Framey VR plugin panels and store | Unattended Frame OS 0.5.6 runs showed the Framey sample panel in stereo compositor and mirror captures. The latest standby-gated exercise restored the original power settings and released its shared lock. Synthetic OpenVR clicks changed a live sample setting, read its Decky backend file and confirmed it after a fresh iframe reopen | No | This was the repository sample, not an official store plugin. Physical controllers, native scrolling, VR keyboard, wearer comfort and legibility remain unverified. CSS Loader and SteamGridDB exact archives passed fake Frame install/remove and panel packing; this does not establish theme or artwork controls in VR. Panel-local backend events and React toasts have local/real-backend checks, but no wearer review. Pause Games and Tailscale Control settings render in local Chromium; their live actions are unverified. The 116-plugin catalog had 60 fake installs/removals and 56 expected blocks; v0.4.40 recorded 101 local Chromium initializers ready and 15 failed, up from 98/18 in v0.4.39. Notebook is marked works only for its default drawing/save/reopen flow | 26 Oct, v0.4.38–0.4.40: [post-#294 unworn sample and scroll evidence](evidence/framey-scroll-probe-2026-10-26.md#post-294-exact-main-sample-check), [backend event PR #270](https://github.com/saphid/frame-control/pull/270), [settings/cache PR #271](https://github.com/saphid/frame-control/pull/271), [post-release sample exercise](evidence/v0439-release-2026-10-26.md#post-release-framey-library-change-282), [worn checklist](framey-vr-check.md) |
|
||||
|
||||
Release notes must preserve these limits until a newer recorded check changes them. When a feature is retested, add its date, exact app/Frame version, host type and proof here. The older [v0.3.1 matrix](support-matrix.html) is a historical snapshot and does not describe current coverage.
|
||||
@@ -1,9 +0,0 @@
|
||||
# Headset smoke: unavailable x86 runtime is a skip
|
||||
|
||||
The v0.4.41 tagged smoke result recorded 13/13 passed even though its x86_64 launch outcome was `runtime missing`: Steam had acknowledged the title but had not started it. This follow-up separates an unavailable optional runtime from a verified launch. It keeps the smoke usable on a Frame without Steam Linux Runtime 4 while making the missing coverage explicit in its console summary and saved JSON.
|
||||
|
||||
Focused local `test_smoke_cleanup.py` ran **22 tests successfully**. The new regression starts an x86_64 title cycle with a fresh missing-runtime Steam log, verifies `skipped=true` and `outcome=runtime missing`, and verifies that the title is still removed. A result test checks separate passed/skipped/failed counts and a zero exit code when an optional runtime is absent but nothing fails.
|
||||
|
||||
The revised `scripts/frame-smoke.sh` then ran on the real Frame from this branch. It acquired its unique owner in `/tmp/frame-test.lock` before any headset read. Locked preflight found SteamVR activity 3 (standby), proximity 6.25 and battery 99%. On Frame OS 0.5.6 build `20261022.6153909`, the result was **12 passed / 1 skipped / 0 failed**. ARM64 and Proton test executables ran; x86_64 was explicitly skipped because Steam Linux Runtime 4 was not installed. Both title cleanup steps passed. The command exited 0, including release of its owned shared lock. The detailed result remains private at `tests/smoke/results/2026-10-26T050711Z-20261022.6153909.json` in the test checkout.
|
||||
|
||||
The headset was in standby and nobody wore it for this test. Title smoke does not prove wearer-visible pixels, physical controller input, official Decky panels, audio output or comfort. It also does not prove x86_64 runtime execution on this Frame; that path needs the optional runtime present. The hourly worn check remains enabled.
|
||||
@@ -0,0 +1,23 @@
|
||||
# v0.4.41 tagged release, 26 October 2026
|
||||
|
||||
The annotated `v0.4.41` tag dereferences to main merge commit `b5d50a8d896f790a94baf4e937db914eab475ed9` from PR #297. The [tagged release workflow](https://github.com/saphid/frame-control/actions/runs/40271174791) passed its draft, Linux, Windows, Apple Silicon Mac, Intel Mac and checksum jobs. The platform jobs launched the built Linux AppImage, checked the Windows installation and bundled PC Python libraries, and checked Mac bundle architectures. These CI results do not establish a physical Windows or Mac launch.
|
||||
|
||||
## Real Frame, exact tag
|
||||
|
||||
From an exact tag checkout, `scripts/frame-smoke.sh` acquired its unique owner in `/tmp/frame-test.lock` before reading the headset. Preflight found SteamVR activity 3 (standby) and battery 99%. On Frame OS 0.5.6 build `20261022.6153909`, all **13/13 reserved title steps passed**; the command exited 0, including test title and process cleanup and release of its owned lock. ARM64 and Proton test executables launched. The x86_64 launch step recorded that optional Steam Linux Runtime 4 was **not installed**, so that executable did not actually launch. The private detailed result is `tests/smoke/results/2026-10-26T045140Z-20261022.6153909.json` in the exact-tag checkout.
|
||||
|
||||
No person wore the headset. This title smoke does not verify active passthrough pixels, wearer-visible panels, physical controller input, official Decky plugin panels, audio output, comfort or VR keyboard use. The separate [unworn Framey sample exercise](framey-scroll-probe-2026-10-26.md) established live sample setting persistence, stereo compositor changes and trusted synthetic OpenVR input; it did not establish a worn result. The hourly worn check remains enabled.
|
||||
|
||||
## Downloaded installer and checksum checks
|
||||
|
||||
The draft held ten installers plus `SHA256SUMS`, each with a GitHub SHA-256 digest. `scripts/publish-release.sh --dry-run v0.4.41` checked the exact tag and all ten installer digests, regenerated `SHA256SUMS` from those digests and matched its own GitHub digest. A fresh downloaded checksum list independently matched `e7555956aeb9ec3a464081aaafe1ed6762f65a17c04edaa6ea45286f10927109`.
|
||||
|
||||
A fresh Linux x86_64 AppImage download matched `5d1f50dc8527faf299f2058c7b5ed19057ec7c81c41449f6ccd44d68e6b4d212`. `scripts/check-appimage-launch.py` launched that exact file under isolated Xvfb: the bundled server answered HTTP 200 with `FrameControl/1`, then stopped. The first local attempt failed at Chromium sandbox startup inside the command sandbox; rerunning the same downloaded file with normal host permissions passed. The release workflow's independent Linux launch also passed.
|
||||
|
||||
Fresh Mac ZIP downloads matched GitHub digests: arm64 `84615ec8a0c62481a9f33bae1815d55ec902db9aa9d4bd767fcf9380c3d44d7a`, x64 `2800b749cda98d15207950eca4a6bd205d35f30551f16efd7720b41347edc7ca`. Both contain bundle version 0.4.41; the app executable Mach-O CPU types are arm64 and x86_64 respectively. The physical Mac was unreachable over SSH, so neither tagged ZIP was launched there. The physical Intel Air was also not tested with this tag.
|
||||
|
||||
## Publication and updater readback
|
||||
|
||||
The final dry run passed with the release notes and hardware limits in the draft. `scripts/publish-release.sh v0.4.41` published [v0.4.41](https://github.com/saphid/frame-control/releases/tag/v0.4.41) at 2026-10-26 05:01:47 UTC. GitHub's latest-release API read back `draft=false`, tag `v0.4.41`, ten installers, `SHA256SUMS` and `update.json`. A fresh public latest-download of `update.json` parsed as version 0.4.41 with the stable release page, ten SHA-256 digests and the unworn-test limits in its notes. The release contains the tagged `b5d50a8` commit; no post-tag changes are implied.
|
||||
|
||||
The exact-version official Decky catalog in this release is **3 works / 48 partial / 54 broken / 11 N/A**. Volume Mixer and Mixer are works only for the tested per-app silent-stream volume scope on real ARM64 Decky; [audio evidence](framey-audio-arm64-2026-10-26.md) describes the independent Pulse readbacks. Volume Boost remains partial. Audible playback, Bluetooth, other mixer controls and official-plugin worn VR remain unverified.
|
||||
+12
-16
@@ -176,21 +176,19 @@ switch while SSH is down:
|
||||
|
||||
## Headset smoke test
|
||||
|
||||
**Documented shared-device procedure:** before any headset read or a test that
|
||||
installs, launches or stops an application, atomically create
|
||||
`/tmp/frame-test.lock` on the Frame and write a unique token to its `owner`
|
||||
file. If creation fails, leave that lock alone and continue offline work. Only
|
||||
the owner that verifies its token may remove its `owner` file and lock directory
|
||||
after cleanup. Keep each device session to a few minutes.
|
||||
**Documented shared-device procedure:** before a test installs, launches or
|
||||
stops an application, acquire `ssh frame 'mkdir /tmp/frame-test.lock'`. If it
|
||||
fails, leave that lock alone and continue offline work. Only the thread that
|
||||
acquired it releases it with `ssh frame 'rmdir /tmp/frame-test.lock'`, after
|
||||
cleanup. Keep each device session to a few minutes.
|
||||
|
||||
Check battery capacity and charging state under `/sys/class/power_supply`
|
||||
before and after; keep capacity above 20%. Stop only processes started by the
|
||||
test, remove temporary installs and profiles, and restore the prior dashboard
|
||||
state. Leave Steam and SteamVR running. Do not reboot or change global settings.
|
||||
Record the build, actual interaction results, cleanup and any unworn-headset
|
||||
limits alongside screenshots or logs. The smoke script below acquires and
|
||||
releases its own tokenized shared lock; other device tests follow this
|
||||
procedure themselves.
|
||||
limits alongside screenshots or logs. These are caller responsibilities; the
|
||||
smoke script below does not acquire this shared lock itself.
|
||||
|
||||
[Packaged Desktop Capture evidence, 2026-10-24](evidence/desktop-capture-packaged-2026-10-24.md)
|
||||
records v0.4.17 GUI button captures of a real mapped nested desktop on SteamOS
|
||||
@@ -222,19 +220,17 @@ removes three tiny titles built from bytes by `tests/smoke/tiny_programs.py`
|
||||
(an ARM64 and an x86-64 static Linux program that sleep for ten seconds, and
|
||||
an x86-64 `.exe` that exits at once). A launch passes only with fresh evidence:
|
||||
the ARM64 program running, the `.exe` started (its process or Steam's log),
|
||||
and the x86-64 program running. If Steam logs that the optional x86-64 runtime
|
||||
is not installed, that launch is **skipped**, with the reason and log saved in
|
||||
the result. It does not prove that the x86-64 title ran. Steam's log lines about
|
||||
each title are kept.
|
||||
and the x86-64 program running or Steam logging that its runtime isn't
|
||||
installed, which is what the Frame does today. Steam's log lines about each
|
||||
title are kept.
|
||||
|
||||
Everything it installs is removed again, also after a failure: the titles and
|
||||
their Steam shortcuts, a paired key, and `~/devkit-utils` if it wasn't there
|
||||
before (if it was, it stays, synced to this checkout as Frame Control always
|
||||
does). A cleanup that fails counts as a failed step. Results go to
|
||||
`tests/smoke/results/<time>-<BUILD_ID>.json` (not committed) with a summary on
|
||||
screen. The summary counts passed, skipped and failed steps separately. It
|
||||
exits 0 when no step failed (including a skipped optional runtime), 1 if a step
|
||||
failed, or 2 if the headset isn't reachable.
|
||||
screen; it exits 0 when every step passed, 1 if one failed, 2 if the headset
|
||||
isn't reachable.
|
||||
|
||||
Smoke process cleanup runs before and after title removal and during the
|
||||
initial/final leftover sweeps. It targets only the reserved `fc_smoke_arm64`,
|
||||
|
||||
+10
-26
@@ -27,8 +27,7 @@ rechecked around deletion; shared Proton services are never stopped by name.
|
||||
|
||||
Usage: python3 tests/smoke/frame_smoke.py [--pair] (or scripts/frame-smoke.sh)
|
||||
Env: FRAME_ALIAS (default frame), FRAME_SMOKE_RESULTS (default tests/smoke/results)
|
||||
Exit status: 0 no step failed (optional runtime may skip a launch), 1 a step
|
||||
failed, 2 the headset isn't reachable.
|
||||
Exit status: 0 all passed, 1 a step failed, 2 the headset isn't reachable.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
@@ -126,32 +125,21 @@ class Smoke:
|
||||
self.installed = set()
|
||||
|
||||
def step(self, name, fn, *args):
|
||||
"""Run one step; record passed/skipped/failed and how long it took."""
|
||||
"""Run one step; record ok/failed, its detail and how long it took."""
|
||||
t0 = time.monotonic()
|
||||
rec = {'step': name}
|
||||
try:
|
||||
detail = fn(*args)
|
||||
rec.update(ok=True, detail=detail)
|
||||
except Skipped as e:
|
||||
rec.update(ok=False, skipped=True, reason=str(e), detail=e.detail)
|
||||
except Exception as e: # a failed step is a result, not the end of the run
|
||||
rec.update(ok=False, error=f'{type(e).__name__}: {e}', trace=traceback.format_exc(limit=3))
|
||||
rec['seconds'] = round(time.monotonic() - t0, 1)
|
||||
self.steps.append(rec)
|
||||
label = 'ok ' if rec['ok'] else 'SKIP' if rec.get('skipped') else 'FAIL'
|
||||
message = '' if rec['ok'] else f": {rec.get('reason') or rec['error']}"
|
||||
print(f" {label} {name} ({rec['seconds']} s){message}", flush=True)
|
||||
print(f" {'ok ' if rec['ok'] else 'FAIL'} {name} ({rec['seconds']} s)"
|
||||
+ ('' if rec['ok'] else f": {rec['error']}"), flush=True)
|
||||
return rec
|
||||
|
||||
|
||||
class Skipped(Exception):
|
||||
"""A prerequisite was unavailable; preserve its evidence without claiming a launch."""
|
||||
|
||||
def __init__(self, reason, detail):
|
||||
super().__init__(reason)
|
||||
self.detail = detail
|
||||
|
||||
|
||||
def ssh(cmd, timeout=60):
|
||||
return frame_android.ssh(cmd, timeout=timeout)
|
||||
|
||||
@@ -206,8 +194,8 @@ def steam_log_lines(*patterns):
|
||||
|
||||
# What a launch must show, per kind. The x86-64 runtime isn't installed on the
|
||||
# Frame, so Steam acknowledges that launch and logs "... is not installed"
|
||||
# instead (docs/sideloading.md, 2026-09-26): record a skip, not a launch pass.
|
||||
EXPECTED = {'arm64': ('running',), 'x86_64': ('running',), 'exe': ('running', 'started')}
|
||||
# instead (docs/sideloading.md, 2026-09-26): recorded, not a failure.
|
||||
EXPECTED = {'arm64': ('running',), 'x86_64': ('running', 'runtime missing'), 'exe': ('running', 'started')}
|
||||
|
||||
|
||||
def title_cycle(smoke, kind, folder):
|
||||
@@ -241,12 +229,10 @@ def title_cycle(smoke, kind, folder):
|
||||
outcome, new = 'runtime missing', new + missing
|
||||
elif any('started devkit game' in line or 'chdir' in line for line in new):
|
||||
outcome = 'started'
|
||||
if outcome in EXPECTED[kind] or (kind == 'x86_64' and outcome == 'runtime missing'):
|
||||
if outcome in EXPECTED[kind]: # else keep looking: the log can come before the process
|
||||
break
|
||||
time.sleep(0.5)
|
||||
detail = {'outcome': outcome, 'processes': procs, 'steam_log': new[-12:]}
|
||||
if kind == 'x86_64' and outcome == 'runtime missing':
|
||||
raise Skipped('Steam Linux Runtime 4 is not installed; x86_64 title did not launch', detail)
|
||||
if outcome not in EXPECTED[kind]:
|
||||
raise AssertionError(f"Steam acknowledged the launch, but {outcome} (expected "
|
||||
f"{' or '.join(EXPECTED[kind])}): {json.dumps(detail)[:400]}")
|
||||
@@ -410,16 +396,14 @@ def run_smoke(args, before):
|
||||
'rm -rf ~/devkit-utils ~/.devkit-utils.frame-control && echo removed')
|
||||
subprocess.run(['rm', '-rf', folder])
|
||||
passed = sum(s['ok'] for s in smoke.steps)
|
||||
skipped = sum(s.get('skipped', False) for s in smoke.steps)
|
||||
failed = len(smoke.steps) - passed - skipped
|
||||
RESULTS.mkdir(exist_ok=True)
|
||||
out = RESULTS / f"{started.replace(':', '')}-{build}.json"
|
||||
out.write_text(json.dumps({'started': started, 'alias': ALIAS, 'preflight': before,
|
||||
'os_release': release, 'paired': args.pair,
|
||||
'passed': passed, 'skipped': skipped, 'failed': failed,
|
||||
'passed': passed, 'failed': len(smoke.steps) - passed,
|
||||
'devkit_utils_there_before': had_utils, 'steps': smoke.steps}, indent=1))
|
||||
print(f'==> {passed} passed, {skipped} skipped, {failed} failed on build {build}; results in {out}')
|
||||
return 0 if failed == 0 else 1
|
||||
print(f'==> {passed}/{len(smoke.steps)} steps passed on build {build}; results in {out}')
|
||||
return 0 if passed == len(smoke.steps) else 1
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
@@ -95,60 +95,6 @@ class RealProcesses(unittest.TestCase):
|
||||
|
||||
|
||||
class Safety(unittest.TestCase):
|
||||
def test_missing_x86_runtime_skips_launch_but_still_removes_title(self):
|
||||
smoke = frame_smoke.Smoke()
|
||||
gid = 'fc_smoke_x86_64'
|
||||
with mock.patch.object(frame_smoke.tiny_programs, 'write', return_value=Path('/tmp/tiny')), \
|
||||
mock.patch.object(frame_smoke.frame_titles, 'install', return_value={
|
||||
'runtime': 'SteamLinuxRuntime_4'}), \
|
||||
mock.patch.object(frame_smoke.frame_titles, 'list_titles', side_effect=[
|
||||
[{'id': gid}], []]), \
|
||||
mock.patch.object(frame_smoke.frame_titles, 'launch') as launch, \
|
||||
mock.patch.object(frame_smoke.frame_titles, 'remove') as remove, \
|
||||
mock.patch.object(frame_smoke, 'steam_log_lines', side_effect=[
|
||||
[], [], [], ['Tool 4183110 Steam Linux Runtime 4.0 is found, but is not installed']]), \
|
||||
mock.patch.object(frame_smoke, 'ssh', return_value=''), \
|
||||
mock.patch.object(frame_smoke, 'stop_titles', return_value={'stopped_pids': []}):
|
||||
frame_smoke.title_cycle(smoke, 'x86_64', '/tmp')
|
||||
self.assertEqual([step['step'] for step in smoke.steps],
|
||||
['x86_64: install', 'x86_64: launch', 'x86_64: remove'])
|
||||
self.assertTrue(smoke.steps[0]['ok'])
|
||||
self.assertFalse(smoke.steps[1]['ok'])
|
||||
self.assertTrue(smoke.steps[1]['skipped'])
|
||||
self.assertEqual(smoke.steps[1]['detail']['outcome'], 'runtime missing')
|
||||
self.assertTrue(smoke.steps[2]['ok'])
|
||||
self.assertNotIn(gid, smoke.installed)
|
||||
launch.assert_called_once_with(gid)
|
||||
remove.assert_called_once_with(gid)
|
||||
|
||||
def test_optional_runtime_skip_is_counted_separately_in_saved_result(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
output = Path(root) / 'results'
|
||||
folder = Path(root) / 'titles'
|
||||
folder.mkdir()
|
||||
def cycle(smoke, kind, _folder):
|
||||
if kind == 'x86_64':
|
||||
def missing():
|
||||
raise frame_smoke.Skipped('optional runtime absent', {'outcome': 'runtime missing'})
|
||||
smoke.step(f'{kind}: launch', missing)
|
||||
else:
|
||||
smoke.step(f'{kind}: launch', lambda: {'outcome': 'running'})
|
||||
with mock.patch.object(frame_smoke, 'RESULTS', output), \
|
||||
mock.patch.object(frame_smoke, 'os_release', return_value={
|
||||
'BUILD_ID': 'fake', 'VERSION_ID': 'fake', 'VARIANT_ID': 'fake'}), \
|
||||
mock.patch.object(frame_smoke, 'ssh', return_value='yes'), \
|
||||
mock.patch.object(frame_smoke, 'cleanup', return_value={}), \
|
||||
mock.patch.object(frame_smoke, 'properties', return_value={}), \
|
||||
mock.patch.object(frame_smoke, 'status', return_value={}), \
|
||||
mock.patch.object(frame_smoke, 'title_cycle', side_effect=cycle), \
|
||||
mock.patch.object(frame_smoke.tempfile, 'mkdtemp', return_value=str(folder)):
|
||||
self.assertEqual(frame_smoke.run_smoke(mock.Mock(pair=False),
|
||||
{'activity': 3, 'proximity': [], 'battery': [100]}), 0)
|
||||
result = json.loads(next(output.glob('*-fake.json')).read_text())
|
||||
self.assertEqual((result['passed'], result['skipped'], result['failed']), (6, 1, 0))
|
||||
self.assertEqual(next(s for s in result['steps'] if s['step'] == 'x86_64: launch')
|
||||
['detail']['outcome'], 'runtime missing')
|
||||
|
||||
def test_non_test_ids_rejected_before_scanning(self):
|
||||
for ids in ([], ['portal2'], ['fc_smoke_exe/../portal2'], ['fc_smoke_custom']):
|
||||
with self.subTest(ids=ids), self.assertRaises(ValueError):
|
||||
|
||||
Reference in new issue
Block a user