Report unavailable x86 runtime as skipped in Frame smoke

docs/evidence/frame-smoke-runtime-skip-2026-10-26.md

@@ -0,0 +1,9 @@
+# Headset smoke: unavailable x86 runtime is a skip
+
+The v0.4.41 tagged smoke result recorded 13/13 passed even though its x86_64 launch outcome was `runtime missing`: Steam had acknowledged the title but had not started it. This follow-up separates an unavailable optional runtime from a verified launch. It keeps the smoke usable on a Frame without Steam Linux Runtime 4 while making the missing coverage explicit in its console summary and saved JSON.
+
+Focused local `test_smoke_cleanup.py` ran **22 tests successfully**. The new regression starts an x86_64 title cycle with a fresh missing-runtime Steam log, verifies `skipped=true` and `outcome=runtime missing`, and verifies that the title is still removed. A result test checks separate passed/skipped/failed counts and a zero exit code when an optional runtime is absent but nothing fails.
+
+The revised `scripts/frame-smoke.sh` then ran on the real Frame from this branch. It acquired its unique owner in `/tmp/frame-test.lock` before any headset read. Locked preflight found SteamVR activity 3 (standby), proximity 6.25 and battery 99%. On Frame OS 0.5.6 build `20261022.6153909`, the result was **12 passed / 1 skipped / 0 failed**. ARM64 and Proton test executables ran; x86_64 was explicitly skipped because Steam Linux Runtime 4 was not installed. Both title cleanup steps passed. The command exited 0, including release of its owned shared lock. The detailed result remains private at `tests/smoke/results/2026-10-26T050711Z-20261022.6153909.json` in the test checkout.
+
+The headset was in standby and nobody wore it for this test. Title smoke does not prove wearer-visible pixels, physical controller input, official Decky panels, audio output or comfort. It also does not prove x86_64 runtime execution on this Frame; that path needs the optional runtime present. The hourly worn check remains enabled.

docs/testing.md

@@ -176,19 +176,21 @@ switch while SSH is down:
 
 ## Headset smoke test
 
-**Documented shared-device procedure:** before a test installs, launches or
-stops an application, acquire `ssh frame 'mkdir /tmp/frame-test.lock'`. If it
-fails, leave that lock alone and continue offline work. Only the thread that
-acquired it releases it with `ssh frame 'rmdir /tmp/frame-test.lock'`, after
-cleanup. Keep each device session to a few minutes.
+**Documented shared-device procedure:** before any headset read or a test that
+installs, launches or stops an application, atomically create
+`/tmp/frame-test.lock` on the Frame and write a unique token to its `owner`
+file. If creation fails, leave that lock alone and continue offline work. Only
+the owner that verifies its token may remove its `owner` file and lock directory
+after cleanup. Keep each device session to a few minutes.
 
 Check battery capacity and charging state under `/sys/class/power_supply`
 before and after; keep capacity above 20%. Stop only processes started by the
 test, remove temporary installs and profiles, and restore the prior dashboard
 state. Leave Steam and SteamVR running. Do not reboot or change global settings.
 Record the build, actual interaction results, cleanup and any unworn-headset
-limits alongside screenshots or logs. These are caller responsibilities; the
-smoke script below does not acquire this shared lock itself.
+limits alongside screenshots or logs. The smoke script below acquires and
+releases its own tokenized shared lock; other device tests follow this
+procedure themselves.
 
 [Packaged Desktop Capture evidence, 2026-10-24](evidence/desktop-capture-packaged-2026-10-24.md)
 records v0.4.17 GUI button captures of a real mapped nested desktop on SteamOS
@@ -220,17 +222,19 @@ removes three tiny titles built from bytes by `tests/smoke/tiny_programs.py`
 (an ARM64 and an x86-64 static Linux program that sleep for ten seconds, and
 an x86-64 `.exe` that exits at once). A launch passes only with fresh evidence:
 the ARM64 program running, the `.exe` started (its process or Steam's log),
-and the x86-64 program running or Steam logging that its runtime isn't
-installed, which is what the Frame does today. Steam's log lines about each
-title are kept.
+and the x86-64 program running. If Steam logs that the optional x86-64 runtime
+is not installed, that launch is **skipped**, with the reason and log saved in
+the result. It does not prove that the x86-64 title ran. Steam's log lines about
+each title are kept.
 
 Everything it installs is removed again, also after a failure: the titles and
 their Steam shortcuts, a paired key, and `~/devkit-utils` if it wasn't there
 before (if it was, it stays, synced to this checkout as Frame Control always
 does). A cleanup that fails counts as a failed step. Results go to
 `tests/smoke/results/<time>-<BUILD_ID>.json` (not committed) with a summary on
-screen; it exits 0 when every step passed, 1 if one failed, 2 if the headset
-isn't reachable.
+screen. The summary counts passed, skipped and failed steps separately. It
+exits 0 when no step failed (including a skipped optional runtime), 1 if a step
+failed, or 2 if the headset isn't reachable.
 
 Smoke process cleanup runs before and after title removal and during the
 initial/final leftover sweeps. It targets only the reserved `fc_smoke_arm64`,

tests/smoke/frame_smoke.py

@@ -27,7 +27,8 @@ rechecked around deletion; shared Proton services are never stopped by name.
 
 Usage: python3 tests/smoke/frame_smoke.py [--pair]     (or scripts/frame-smoke.sh)
 Env:   FRAME_ALIAS (default frame), FRAME_SMOKE_RESULTS (default tests/smoke/results)
-Exit status: 0 all passed, 1 a step failed, 2 the headset isn't reachable.
+Exit status: 0 no step failed (optional runtime may skip a launch), 1 a step
+failed, 2 the headset isn't reachable.
 """
 import argparse
 import json
@@ -125,21 +126,32 @@ class Smoke:
         self.installed = set()
 
     def step(self, name, fn, *args):
-        """Run one step; record ok/failed, its detail and how long it took."""
+        """Run one step; record passed/skipped/failed and how long it took."""
         t0 = time.monotonic()
         rec = {'step': name}
         try:
             detail = fn(*args)
             rec.update(ok=True, detail=detail)
+        except Skipped as e:
+            rec.update(ok=False, skipped=True, reason=str(e), detail=e.detail)
         except Exception as e:  # a failed step is a result, not the end of the run
             rec.update(ok=False, error=f'{type(e).__name__}: {e}', trace=traceback.format_exc(limit=3))
         rec['seconds'] = round(time.monotonic() - t0, 1)
         self.steps.append(rec)
-        print(f"  {'ok  ' if rec['ok'] else 'FAIL'} {name} ({rec['seconds']} s)"
-              + ('' if rec['ok'] else f": {rec['error']}"), flush=True)
+        label = 'ok  ' if rec['ok'] else 'SKIP' if rec.get('skipped') else 'FAIL'
+        message = '' if rec['ok'] else f": {rec.get('reason') or rec['error']}"
+        print(f"  {label} {name} ({rec['seconds']} s){message}", flush=True)
         return rec
 
 
+class Skipped(Exception):
+    """A prerequisite was unavailable; preserve its evidence without claiming a launch."""
+
+    def __init__(self, reason, detail):
+        super().__init__(reason)
+        self.detail = detail
+
+
 def ssh(cmd, timeout=60):
     return frame_android.ssh(cmd, timeout=timeout)
 
@@ -194,8 +206,8 @@ def steam_log_lines(*patterns):
 
 # What a launch must show, per kind. The x86-64 runtime isn't installed on the
 # Frame, so Steam acknowledges that launch and logs "... is not installed"
-# instead (docs/sideloading.md, 2026-09-26): recorded, not a failure.
-EXPECTED = {'arm64': ('running',), 'x86_64': ('running', 'runtime missing'), 'exe': ('running', 'started')}
+# instead (docs/sideloading.md, 2026-09-26): record a skip, not a launch pass.
+EXPECTED = {'arm64': ('running',), 'x86_64': ('running',), 'exe': ('running', 'started')}
 
 
 def title_cycle(smoke, kind, folder):
@@ -229,10 +241,12 @@ def title_cycle(smoke, kind, folder):
                 outcome, new = 'runtime missing', new + missing
             elif any('started devkit game' in line or 'chdir' in line for line in new):
                 outcome = 'started'
-            if outcome in EXPECTED[kind]:   # else keep looking: the log can come before the process
+            if outcome in EXPECTED[kind] or (kind == 'x86_64' and outcome == 'runtime missing'):
                 break
             time.sleep(0.5)
         detail = {'outcome': outcome, 'processes': procs, 'steam_log': new[-12:]}
+        if kind == 'x86_64' and outcome == 'runtime missing':
+            raise Skipped('Steam Linux Runtime 4 is not installed; x86_64 title did not launch', detail)
         if outcome not in EXPECTED[kind]:
             raise AssertionError(f"Steam acknowledged the launch, but {outcome} (expected "
                                  f"{' or '.join(EXPECTED[kind])}): {json.dumps(detail)[:400]}")
@@ -396,14 +410,16 @@ def run_smoke(args, before):
                        'rm -rf ~/devkit-utils ~/.devkit-utils.frame-control && echo removed')
         subprocess.run(['rm', '-rf', folder])
     passed = sum(s['ok'] for s in smoke.steps)
+    skipped = sum(s.get('skipped', False) for s in smoke.steps)
+    failed = len(smoke.steps) - passed - skipped
     RESULTS.mkdir(exist_ok=True)
     out = RESULTS / f"{started.replace(':', '')}-{build}.json"
     out.write_text(json.dumps({'started': started, 'alias': ALIAS, 'preflight': before,
                                'os_release': release, 'paired': args.pair,
-                               'passed': passed, 'failed': len(smoke.steps) - passed,
+                               'passed': passed, 'skipped': skipped, 'failed': failed,
                                'devkit_utils_there_before': had_utils, 'steps': smoke.steps}, indent=1))
-    print(f'==> {passed}/{len(smoke.steps)} steps passed on build {build}; results in {out}')
-    return 0 if passed == len(smoke.steps) else 1
+    print(f'==> {passed} passed, {skipped} skipped, {failed} failed on build {build}; results in {out}')
+    return 0 if failed == 0 else 1
 
 
 def main():

tests/test_smoke_cleanup.py

@@ -95,6 +95,60 @@ class RealProcesses(unittest.TestCase):
 
 
 class Safety(unittest.TestCase):
+    def test_missing_x86_runtime_skips_launch_but_still_removes_title(self):
+        smoke = frame_smoke.Smoke()
+        gid = 'fc_smoke_x86_64'
+        with mock.patch.object(frame_smoke.tiny_programs, 'write', return_value=Path('/tmp/tiny')), \
+             mock.patch.object(frame_smoke.frame_titles, 'install', return_value={
+                 'runtime': 'SteamLinuxRuntime_4'}), \
+             mock.patch.object(frame_smoke.frame_titles, 'list_titles', side_effect=[
+                 [{'id': gid}], []]), \
+             mock.patch.object(frame_smoke.frame_titles, 'launch') as launch, \
+             mock.patch.object(frame_smoke.frame_titles, 'remove') as remove, \
+             mock.patch.object(frame_smoke, 'steam_log_lines', side_effect=[
+                 [], [], [], ['Tool 4183110 Steam Linux Runtime 4.0 is found, but is not installed']]), \
+             mock.patch.object(frame_smoke, 'ssh', return_value=''), \
+             mock.patch.object(frame_smoke, 'stop_titles', return_value={'stopped_pids': []}):
+            frame_smoke.title_cycle(smoke, 'x86_64', '/tmp')
+        self.assertEqual([step['step'] for step in smoke.steps],
+                         ['x86_64: install', 'x86_64: launch', 'x86_64: remove'])
+        self.assertTrue(smoke.steps[0]['ok'])
+        self.assertFalse(smoke.steps[1]['ok'])
+        self.assertTrue(smoke.steps[1]['skipped'])
+        self.assertEqual(smoke.steps[1]['detail']['outcome'], 'runtime missing')
+        self.assertTrue(smoke.steps[2]['ok'])
+        self.assertNotIn(gid, smoke.installed)
+        launch.assert_called_once_with(gid)
+        remove.assert_called_once_with(gid)
+
+    def test_optional_runtime_skip_is_counted_separately_in_saved_result(self):
+        with tempfile.TemporaryDirectory() as root:
+            output = Path(root) / 'results'
+            folder = Path(root) / 'titles'
+            folder.mkdir()
+            def cycle(smoke, kind, _folder):
+                if kind == 'x86_64':
+                    def missing():
+                        raise frame_smoke.Skipped('optional runtime absent', {'outcome': 'runtime missing'})
+                    smoke.step(f'{kind}: launch', missing)
+                else:
+                    smoke.step(f'{kind}: launch', lambda: {'outcome': 'running'})
+            with mock.patch.object(frame_smoke, 'RESULTS', output), \
+                 mock.patch.object(frame_smoke, 'os_release', return_value={
+                     'BUILD_ID': 'fake', 'VERSION_ID': 'fake', 'VARIANT_ID': 'fake'}), \
+                 mock.patch.object(frame_smoke, 'ssh', return_value='yes'), \
+                 mock.patch.object(frame_smoke, 'cleanup', return_value={}), \
+                 mock.patch.object(frame_smoke, 'properties', return_value={}), \
+                 mock.patch.object(frame_smoke, 'status', return_value={}), \
+                 mock.patch.object(frame_smoke, 'title_cycle', side_effect=cycle), \
+                 mock.patch.object(frame_smoke.tempfile, 'mkdtemp', return_value=str(folder)):
+                self.assertEqual(frame_smoke.run_smoke(mock.Mock(pair=False),
+                                 {'activity': 3, 'proximity': [], 'battery': [100]}), 0)
+            result = json.loads(next(output.glob('*-fake.json')).read_text())
+            self.assertEqual((result['passed'], result['skipped'], result['failed']), (6, 1, 0))
+            self.assertEqual(next(s for s in result['steps'] if s['step'] == 'x86_64: launch')
+                             ['detail']['outcome'], 'runtime missing')
+
     def test_non_test_ids_rejected_before_scanning(self):
         for ids in ([], ['portal2'], ['fc_smoke_exe/../portal2'], ['fc_smoke_custom']):
             with self.subTest(ids=ids), self.assertRaises(ValueError):