Compare commits

...
Author SHA1 Message Date
saphid b726817ac3 Fix Android screenshot sync lifecycle, storage, and account baselines 2026-10-26 17:56:15 +11:00
saphid ae9df35442 Add Android phone companion with automatic screenshot downloads 2026-10-24 21:22:57 +11:00
saphid ce27c0e8f1 Add durable foreground iOS screenshot downloads and Photos preference 2026-10-24 16:43:44 +11:00
saphid 5277232367 Merge remote-tracking branch 'origin/main' into feat/phone-screenshot-autosync-108
# Conflicts:
#	README.md
#	ui/frame_macview.py
#	ui/index.html
#	ui/mac-view.html
2026-10-24 16:31:05 +11:00
saphid 3fdc6c46c3 Document Mac capture and live-video verification 2026-10-17 13:01:09 +11:00
saphid 9a6b48dea5 iPhone: record physical feature tests and remaining checks 2026-10-16 20:36:44 +11:00
saphid a3e2b70555 iPhone: record approved beta and phone test blocker 2026-10-16 20:11:41 +11:00
saphid 092aae24bd iPhone: document cloud signing access and the public beta link 2026-10-16 19:33:28 +11:00
saphid 7adf0805d7 iPhone: declare encryption compliance and document TestFlight releases
Declare no non-exempt encryption in the generated app, document upload and tester setup, and record the physical iPhone connection check and remaining tests. Refs #100.
2026-10-16 19:29:45 +11:00
saphidandClaude Opus 5.5 7f23acb03f iPhone: show/hide toggle on the Developer Mode password; TestFlight upload script
The setup screen's password field gets an eye button (44 pt target) that
swaps between hidden and visible text and keeps focus.

ios/scripts/testflight.sh archives with automatic signing and uploads to
App Store Connect using an API key (ASC_KEY_ID, ASC_ISSUER_ID,
ASC_KEY_PATH); the build number is the UTC time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:24:37 +11:00
saphid d9454ef099 Improve Frame Control website onboarding and audit fixes 2026-09-30 19:56:02 +10:00
saphidandClaude Opus 5.5 71a3a96dd2 Mac space: fix separated windows landing on the wrong display; steadier laser; room-matched displays; text size
From Alex's first try in the headset:
- Separated windows could end up on another window's display, so panels
  showed the wrong window or froze. New virtual displays were slotted in
  next to the main one, pushing earlier ones along while a window was being
  placed; and displays under 800x600 points never got their 2x mode.
  Now each display goes left of the leftmost one, the window is placed
  only once the display is in its 2x mode and checked afterwards, and
  displays are at least 800x600 (probed: smaller ones have no usable 2x
  mode on macOS 26). Verified: detach A, detach B, dock A, detach A again,
  each panel streams its own window at 2x.
- Laser: One Euro smoothing on the ray; a click uses where it pointed
  80 ms before the trigger pull, and the Mac pointer holds still until the
  ray moves 1.2 degrees, so clicks don't become drags.
- The Mac lays the separated windows' displays out left/right of the main
  one as their panels sit in the room, so the Mac's own pointer crosses
  into the panel beside it. Verified in the preview.
- Text size: A-/A+ on window panels send the window's own Cmd+-/Cmd+=;
  automatic by default, following distance after a move. WebGL 2 with
  mipmaps and anisotropic filtering, so distant panels don't shimmer.
- The agent logs each stream's start/end and Separate mode's steps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 23:09:15 +10:00
saphidandClaude Opus 5.5 952b660bf7 Mac space: launch Chromium XR with OpenXR on and the seccomp filter off (as its own launcher does)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 22:45:26 +10:00
saphidandClaude Opus 5.5 a4d840a148 Mac space (first version): the Mac's screen in the room, windows pulled out as their own panels
- docs/mac-space.md: what Vision Pro's Mac Virtual Display does, what the
  Frame allows (another process can't move SteamVR's panels: verified,
  PermissionDenied), and why a WebXR page comes first.
- ui/mac-stream.js: the stream protocol shared by the viewers (connect,
  clock sync, decode, acks/reports, reconnect, input); mac-view.html uses it.
- ui/mac-space.html: WebXR (passthrough behind) or an in-page preview.
  Desktop panel with a Vision Pro-style bar and close button; laser clicks,
  thumbstick scrolls, squeeze or the bar grabs; dragging a window's title
  bar off the desktop opens it on its own display as a new panel that
  follows the pointer; dropping it back sends it home.
- Agent: space tickets -> a space key (/space/key) that lists windows and
  displays (/space/windows, now with positions) and opens any stream;
  Stop All revokes it. Serves /space and /mac-stream.js.
- Frame Control: "Open Mac space" in the Tools card.

Verified on the Mac (preview in headless Chrome, lab agent): the main
display streams into the panel; a test window detached onto its own
display and streamed; dropped back, it returned to its exact frame and the
display was removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 22:24:36 +10:00
93 changed files with 8334 additions and 387 deletions

No files matched your search

+1 -1
View File
@@ -7,7 +7,7 @@ body:
value: |
**Before you start:** read [CONTRIBUTING.md](https://github.com/saphid/frame-control/blob/main/CONTRIBUTING.md).
Issues from new contributors are auto-closed by default. A maintainer reviews them and reopens worthwhile ones. The [website feedback form](https://frame-control.pages.dev/feedback/) skips that queue.
Issues from new contributors are auto-closed by default. A maintainer reviews them and reopens worthwhile ones. The [website feedback form](https://framecontrol.fun/feedback/) skips that queue.
Keep this short. If it doesn't fit on one screen, it's too long. Write in your own voice.
+1 -1
View File
@@ -1,5 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Feedback form (no GitHub account needed, skips the queue)
url: https://frame-control.pages.dev/feedback/
url: https://framecontrol.fun/feedback/
about: Bugs, ideas and questions from the website become issues here without being auto-closed.
+22
View File
@@ -0,0 +1,22 @@
name: Android phone
on:
pull_request:
paths: ['android/**', 'ui/**', 'scripts/**', 'frame/**', 'apk-catalog/**', 'ios/scripts/make_frame_bundle.py', '.github/workflows/android.yml']
push:
branches: [main]
paths: ['android/**', 'ui/**', 'scripts/**', 'frame/**', 'apk-catalog/**', 'ios/scripts/make_frame_bundle.py', '.github/workflows/android.yml']
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '17'
- uses: android-actions/setup-android@v3
- uses: gradle/actions/setup-gradle@v4
- run: sdkmanager 'platforms;android-36' 'build-tools;36.0.0'
- run: ./android/gradlew -p android assembleDebug assembleDebugAndroidTest testDebugUnitTest lintDebug --no-daemon
+1 -1
View File
@@ -104,7 +104,7 @@ jobs:
'',
`Maintainers review auto-closed issues regularly and reopen worthwhile ones. Issues that do not meet the quality bar in [CONTRIBUTING.md](https://github.com/${context.repo.owner}/${context.repo.repo}/blob/${defaultBranch}/CONTRIBUTING.md) will not be reopened or receive a reply.`,
'',
'Just want to report a bug or share an idea? The [website feedback form](https://frame-control.pages.dev/feedback/) skips this queue.',
'Just want to report a bug or share an idea? The [website feedback form](https://framecontrol.fun/feedback/) skips this queue.',
'',
'If a maintainer replies `lgtmi` on one of your issues, your future issues will stay open. If a maintainer replies `lgtm`, your future issues and PRs will stay open. The command must be at the start of the reply (optionally after one or more `@username` mentions) or at the end.',
'',
+17 -1
View File
@@ -12,7 +12,7 @@ See what the headset sees, install games and Android apps, move files and text a
[![Checks](https://img.shields.io/github/actions/workflow/status/saphid/steam-frame/checks.yml?branch=main&label=checks)](https://github.com/saphid/steam-frame/actions/workflows/checks.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-66c0f4)](LICENSE)
[**Website**](https://frame-control.pages.dev) · [**Download**](#install) · [Trailer](#trailer) · [Features](#features) · [Set up the headset](#set-up-the-headset) · [Feedback](#feedback) · [Docs](#going-further)
[**Get started**](https://framecontrol.fun) · [**Download**](https://framecontrol.fun/#download) · [Trailer](#trailer) · [Features](#features) · [Set up the headset](#set-up-the-headset) · [Feedback](#feedback) · [Docs](#going-further)
<br>
@@ -116,6 +116,17 @@ already ships (sideloading a game copies Valve's own devkit scripts to
## Install
**New here? [Get Frame Control and follow the setup guide on the website](https://framecontrol.fun/#download).**
Install it on your **computer, not the headset**. You don't need to clone this
repository or install Python. Download for your computer, install and open the
app, then choose **Set Up Connection** with the Frame awake on the same network.
- **macOS:** open the `.dmg`, drag Frame Control to Applications, then open it there.
- **Windows:** run the installer and launch Frame Control from Start, or extract the portable `.zip` and run `Frame Control.exe`.
- **Linux:** make the AppImage executable and open it, or install the `.deb` and launch Frame Control.
If the app is blocked, see the first-launch notes below before proceeding.
| | Download | Needs |
|---|---|---|
| **macOS** (Apple Silicon) | [Frame-Control-mac-arm64.dmg](https://github.com/saphid/steam-frame/releases/latest/download/Frame-Control-mac-arm64.dmg) | Nothing extra |
@@ -126,6 +137,10 @@ already ships (sideloading a game copies Valve's own devkit scripts to
**iPhone and iPad:** the same features from your phone, with nothing to install on
a computer. Build it from [`ios/`](ios) in Xcode; see [docs/iphone.md](docs/iphone.md).
**Android phones (development):** a native companion is in [`android/`](android).
It can download new screenshots during a user-started sync session, with Gallery
saving enabled by default. See [build instructions and current testing limits](docs/android-phone.md).
The app brings its own Python and `adb`; SSH is built into macOS and Windows.
From 0.4 it updates itself: when a new version is published, a banner offers
**Update and restart**. It sends anonymous usage statistics, which you can turn
@@ -234,6 +249,7 @@ Frame's software fits together, all checked against a real headset and labelled
| [VR mods and custom songs](docs/mods.md) | UEVR for Unreal games from Frame Control (install, play in VR, remove), the per-game table, Beat Saber blockers |
| [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes |
| [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing |
| [Frame Control for Android phones](docs/android-phone.md) | Development build, pairing, screenshot downloads and foreground sync limits |
| [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset |
| [AI agents and assistant](docs/agents.md) | Key-free MCP tools, human approvals, and an opt-in assistant panel |
| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, the headset smoke test and a Windows test VM |
+5
View File
@@ -0,0 +1,5 @@
.gradle/
**/build/
local.properties
*.keystore
*.jks
+35
View File
@@ -0,0 +1,35 @@
plugins { id 'com.android.application' }
android {
namespace 'com.saphid.framecontrol'
compileSdk 36
defaultConfig {
applicationId 'com.saphid.framecontrol'
minSdk 29
targetSdk 36
versionCode 1
versionName '0.1.0'
testInstrumentationRunner 'androidx.test.runner.AndroidJUnitRunner'
}
compileOptions { sourceCompatibility JavaVersion.VERSION_17; targetCompatibility JavaVersion.VERSION_17 }
sourceSets.main.assets.srcDir layout.buildDirectory.dir('generated/frameAssets').get().asFile
}
def frameAssets = tasks.register('frameAssets', Exec) {
workingDir rootProject.projectDir.parentFile
commandLine 'python3', 'android/scripts/make_assets.py', layout.buildDirectory.dir('generated/frameAssets').get().asFile
inputs.files(fileTree(rootProject.file('../ui')), fileTree(rootProject.file('../scripts')), fileTree(rootProject.file('../frame')), fileTree(rootProject.file('../apk-catalog')) { exclude 'data/**' })
inputs.file(rootProject.file('../ios/scripts/make_frame_bundle.py'))
inputs.file(rootProject.file('scripts/make_assets.py'))
outputs.dir layout.buildDirectory.dir('generated/frameAssets')
}
tasks.named('preBuild').configure { dependsOn frameAssets }
dependencies {
implementation 'com.github.mwiede:jsch:2.28.8'
implementation 'com.google.code.gson:gson:2.13.2'
implementation 'androidx.core:core:1.17.0'
testImplementation 'junit:junit:4.13.2'
androidTestImplementation 'androidx.test:runner:1.7.0'
androidTestImplementation 'androidx.test.ext:junit:1.3.0'
}
@@ -0,0 +1,686 @@
package com.saphid.framecontrol;
import static org.junit.Assert.*;
import android.app.Activity;
import android.app.Instrumentation;
import android.content.*;
import android.database.Cursor;
import android.graphics.*;
import android.net.Uri;
import android.provider.MediaStore;
import android.system.ErrnoException;
import android.system.Os;
import android.system.OsConstants;
import android.view.View;
import android.view.ViewGroup;
import android.widget.Button;
import androidx.test.ext.junit.runners.AndroidJUnit4;
import androidx.test.platform.app.InstrumentationRegistry;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import org.junit.*;
import org.junit.runner.RunWith;
@RunWith(AndroidJUnit4.class)
public class ScreenshotDeviceTest {
private Context context;
private final Set<Uri> cleanup = new LinkedHashSet<>();
@Before
public void setup() {
context = InstrumentationRegistry.getInstrumentation().getTargetContext();
}
@After
public void cleanup() {
for (Uri uri : cleanup) context.getContentResolver().delete(uri, null, null);
}
/** Fixture Steam accounts exist on the simulated headset from the first poll. */
private static SyncEngine.Listing listing(SyncEngine.Shot... shots) {
return new SyncEngine.Listing(Set.of("123", "999999999991"), List.of(shots));
}
private String name(String headset, SyncEngine.Shot shot) throws Exception {
String file = shot.id.substring(shot.id.lastIndexOf('/') + 1);
int dot = file.lastIndexOf('.');
return file.substring(0, dot)
+ "_"
+ FrameConnection.hash(headset + "/" + shot.id).substring(0, 20)
+ file.substring(dot);
}
private Uri row(String headset, SyncEngine.Shot shot) throws Exception {
Uri collection = MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY);
@SuppressWarnings("deprecation")
Uri includingPending = MediaStore.setIncludePending(collection);
try (Cursor c =
context
.getContentResolver()
.query(
includingPending,
new String[] {"_id"},
MediaStore.Images.Media.DISPLAY_NAME
+ "=? AND "
+ MediaStore.Images.Media.RELATIVE_PATH
+ "=?",
new String[] {name(headset, shot), "Pictures/Frame Control/"},
null)) {
if (c != null && c.moveToFirst()) {
Uri uri = ContentUris.withAppendedId(collection, c.getLong(0));
cleanup.add(uri);
assertFalse("Duplicate Gallery row", c.moveToNext());
return uri;
}
return null;
}
}
@Test
public void actualGalleryOriginalAndAppOnlySurviveRelaunch() throws Exception {
String headset = "instrumented-" + UUID.randomUUID();
File root = new File(context.getFilesDir(), "instrumented/" + headset);
Object gate = new Object();
Bitmap bitmap = Bitmap.createBitmap(48, 32, Bitmap.Config.ARGB_8888);
bitmap.eraseColor(Color.rgb(25, 184, 143));
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, bytes);
bitmap.recycle();
byte[] png = bytes.toByteArray();
SyncEngine.Transport transport =
new SyncEngine.Transport() {
public void download(SyncEngine.Shot shot, File target) throws Exception {
Files.write(target.toPath(), png);
}
public boolean validImage(File f) {
Bitmap b = BitmapFactory.decodeFile(f.getPath());
if (b == null) return false;
b.recycle();
return true;
}
};
GalleryExporter exporter = new GalleryExporter(context, headset, gate);
SyncEngine engine = new SyncEngine(root, true);
engine.reconcile(listing(), transport, exporter);
SyncEngine.Shot shot =
new SyncEngine.Shot("999999999991/108/20261024000000_1.png", png.length, 1792800000);
engine.reconcile(listing(shot), transport, exporter);
assertNull(row(headset, shot));
engine.reconcile(listing(shot), transport, exporter);
Uri saved = row(headset, shot);
assertNotNull(saved);
try (InputStream in = context.getContentResolver().openInputStream(saved)) {
assertArrayEquals(png, read(in));
}
assertEquals("saved", engine.items().get(0).destination);
engine = new SyncEngine(root, true);
engine.reconcile(listing(shot), transport, exporter);
assertEquals(saved, row(headset, shot));
engine.setGallery(false);
SyncEngine.Shot second =
new SyncEngine.Shot("999999999991/108/20261024000001_1.png", png.length, 1792800001);
engine.reconcile(listing(shot, second), transport, exporter);
engine.reconcile(listing(shot, second), transport, exporter);
engine.setGallery(true);
engine = new SyncEngine(root, true);
engine.reconcile(listing(shot, second), transport, exporter);
assertNull(row(headset, second));
assertTrue(
engine.items().stream()
.anyMatch(
i ->
i.shot.id.equals(second.id)
&& i.destination.equals("appOnly")
&& i.file.exists()));
}
@Test
public void interruptedMediaStoreRowIsRewrittenWithoutDuplicate() throws Exception {
String headset = "pending-" + UUID.randomUUID();
SyncEngine.Shot shot =
new SyncEngine.Shot("999999999992/108/20261024000002_1.png", 4, 1792800002);
ContentValues values = new ContentValues();
values.put(MediaStore.Images.Media.DISPLAY_NAME, name(headset, shot));
values.put(MediaStore.Images.Media.RELATIVE_PATH, "Pictures/Frame Control/");
values.put(MediaStore.Images.Media.MIME_TYPE, "image/png");
values.put(MediaStore.Images.Media.IS_PENDING, 1);
Uri pending =
context
.getContentResolver()
.insert(
MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY), values);
assertNotNull(pending);
cleanup.add(pending);
File file = File.createTempFile("gallery-proof", ".png", context.getCacheDir());
byte[] original = {1, 2, 3, 4};
Files.write(file.toPath(), original);
GalleryExporter exporter = new GalleryExporter(context, headset, new Object());
assertTrue(exporter.export(shot, file, () -> true));
assertEquals(pending, row(headset, shot));
assertTrue(exporter.export(shot, file, () -> true));
assertEquals(pending, row(headset, shot));
try (InputStream in = context.getContentResolver().openInputStream(pending)) {
assertArrayEquals(original, read(in));
}
file.delete();
}
/**
* Stop/restart overlap: a stopped session's exporter is still writing its pending row when the
* next session exports the same screenshot. The old writer reads its original from a FIFO, so
* it stays mid-write, with the row open, until the test supplies the bytes. Without the
* process-wide export lock the next session publishes that row first and the old session's
* cancellation then deletes it, leaving a "saved" receipt with no Gallery image.
*/
@Test
public void stoppedSessionExportCannotDeleteNextSessionsGalleryImage() throws Exception {
String headset = "handoff-" + UUID.randomUUID();
Bitmap bitmap = Bitmap.createBitmap(16, 16, Bitmap.Config.ARGB_8888);
bitmap.eraseColor(Color.rgb(184, 25, 143));
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, bytes);
bitmap.recycle();
byte[] png = bytes.toByteArray();
SyncEngine.Shot shot =
new SyncEngine.Shot("999999999993/108/20261026000000_1.png", png.length, 1792900000);
File original = File.createTempFile("handoff", ".png", context.getCacheDir());
Files.write(original.toPath(), png);
File fifo = new File(context.getCacheDir(), "handoff-" + UUID.randomUUID() + ".fifo");
Os.mkfifo(fifo.getPath(), 0600);
// Both sessions share one settings lock, as both use SyncService.SETTINGS_LOCK.
Object settings = new Object();
GalleryExporter stopped = new GalleryExporter(context, headset, settings);
GalleryExporter next = new GalleryExporter(context, headset, settings);
AtomicBoolean stoppedSessionCurrent = new AtomicBoolean(true);
AtomicReference<Object> stoppedResult = new AtomicReference<>();
AtomicReference<Object> nextResult = new AtomicReference<>();
Thread old =
exporting(stoppedResult, () -> stopped.export(shot, fifo, stoppedSessionCurrent::get));
Thread successor = null;
FileDescriptor writer = null;
try {
old.start();
writer = openWriter(fifo, old);
// The old export has inserted its pending row and waits for the original's bytes.
stoppedSessionCurrent.set(false); // Stop closed the old engine.
successor = exporting(nextResult, () -> next.export(shot, original, () -> true));
successor.start();
Thread waiting = successor;
// Fixed: the successor queues behind the old export. Former code: it publishes and ends.
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(20);
while (waiting.isAlive() && !GalleryExporter.EXPORTS.hasQueuedThread(waiting)) {
assertTrue("The next session neither waited nor finished", System.nanoTime() < deadline);
Thread.sleep(10);
}
int written = Os.write(writer, png, 0, png.length);
assertEquals(png.length, written);
} finally {
// An exporter blocked opening the FIFO is not interruptible; a writer's EOF releases it.
if (writer == null) {
try {
writer = Os.open(fifo.getPath(), OsConstants.O_WRONLY | OsConstants.O_NONBLOCK, 0);
} catch (ErrnoException ignored) {
// No reader is waiting.
}
}
if (writer != null) Os.close(writer);
join(old);
if (successor != null) join(successor);
fifo.delete();
original.delete();
try {
row(headset, shot); // Registers whatever row a failed run left for cleanup.
} catch (Throwable ignored) {
// Cleanup only; the assertions below report the outcome.
}
}
assertEquals("The stopped session must not save", Boolean.FALSE, result(stoppedResult));
assertEquals("The next session must save", Boolean.TRUE, result(nextResult));
Uri saved = row(headset, shot);
assertNotNull("The next session's Gallery image must survive the old callback", saved);
@SuppressWarnings("deprecation")
Uri includingPending = MediaStore.setIncludePending(saved);
try (Cursor c =
context
.getContentResolver()
.query(
includingPending,
new String[] {MediaStore.Images.Media.IS_PENDING},
null,
null,
null)) {
assertTrue(c != null && c.moveToFirst());
assertEquals("The Gallery image must be published", 0, c.getInt(0));
}
try (InputStream in = context.getContentResolver().openInputStream(saved)) {
assertArrayEquals(png, read(in));
}
}
private interface Export {
boolean run() throws Exception;
}
private static Thread exporting(AtomicReference<Object> result, Export export) {
return new Thread(
() -> {
try {
result.set(export.run());
} catch (Throwable e) {
result.set(e);
}
},
"handoff-export");
}
/** Opens the FIFO's write end once the exporter has opened its read end; bounded. */
private static FileDescriptor openWriter(File fifo, Thread reader) throws Exception {
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(20);
while (true) {
try {
return Os.open(fifo.getPath(), OsConstants.O_WRONLY | OsConstants.O_NONBLOCK, 0);
} catch (ErrnoException e) {
if (e.errno != OsConstants.ENXIO) throw e; // ENXIO: no reader yet.
}
assertTrue("The old export ended before reading its original", reader.isAlive());
assertTrue("The old export never started reading", System.nanoTime() < deadline);
Thread.sleep(10);
}
}
private static void join(Thread thread) throws InterruptedException {
thread.join(TimeUnit.SECONDS.toMillis(60));
if (thread.isAlive()) thread.interrupt();
assertFalse("Export thread did not finish", thread.isAlive());
}
private static Object result(AtomicReference<Object> result) {
Object value = result.get();
if (value instanceof Throwable) throw new AssertionError("Export failed", (Throwable) value);
return value;
}
@Test
public void galleryOptOutLatchesPreviouslyPairedLibraries() throws Exception {
assertFalse("Instrumentation must own a stopped product", SyncService.running());
File files = Files.createTempDirectory(context.getCacheDir().toPath(), "libraries-").toFile();
String preferences = "library-test-" + UUID.randomUUID();
Context isolated =
new ContextWrapper(context) {
@Override
public File getFilesDir() {
return files;
}
@Override
public SharedPreferences getSharedPreferences(String name, int mode) {
return context.getSharedPreferences(preferences, mode);
}
};
try {
SyncEngine.Transport transfer =
new SyncEngine.Transport() {
public void download(SyncEngine.Shot shot, File target) throws Exception {
Files.write(target.toPath(), new byte[] {1, 2, 3, 4});
}
public boolean validImage(File file) {
return true;
}
};
SyncEngine.Exporter unavailable = (shot, file, eligible) -> false;
SyncEngine.Shot shot = new SyncEngine.Shot("123/108/20261024120000_1.png", 4, 100);
File a = SyncService.library(isolated, "headset-a");
File b = SyncService.library(isolated, "headset-b");
// A retained library still in the version 1 format (no account set), never polled since.
File legacy = SyncService.library(isolated, "headset-legacy");
assertTrue(legacy.mkdirs());
Files.write(
new File(legacy, "receipts.json").toPath(),
("{\"version\":1,\"initialized\":true,\"gallery\":true,\"records\":{\""
+ shot.id
+ "\":{\"shot\":{\"id\":\""
+ shot.id
+ "\",\"size\":4,\"time\":100},\"state\":\"queued\",\"observations\":2,"
+ "\"destination\":\"pending\"}}}")
.getBytes(StandardCharsets.UTF_8));
for (File root : List.of(a, b)) {
SyncEngine engine = new SyncEngine(root, true);
engine.reconcile(listing(), transfer, unavailable);
engine.reconcile(listing(shot), transfer, unavailable);
engine.reconcile(listing(shot), transfer, unavailable);
assertEquals("pending", engine.items().get(0).destination);
}
assertTrue(
SyncService.prefs(isolated)
.edit()
.putString("pin", "headset-b")
.putBoolean("gallery", true)
.commit());
SyncService.gallery(isolated, false);
// The latched legacy journal must stay readable immediately, by the screen and an engine.
assertEquals("appOnly", SyncEngine.snapshot(legacy).get(0).destination);
assertEquals("appOnly", new SyncEngine(legacy, false).items().get(0).destination);
SyncService.gallery(isolated, true);
for (File root : List.of(a, b, legacy)) {
SyncEngine engine = new SyncEngine(root, true);
assertEquals("appOnly", engine.items().get(0).destination);
engine.reconcile(
listing(shot),
transfer,
(s, f, eligible) -> {
fail("Opted-out items must never be exported after re-pairing");
return false;
});
}
} finally {
SyncService.prefs(isolated).edit().clear().commit();
try (java.util.stream.Stream<java.nio.file.Path> paths = Files.walk(files.toPath())) {
for (java.nio.file.Path path :
paths.sorted(Comparator.reverseOrder()).collect(java.util.stream.Collectors.toList()))
Files.delete(path);
}
}
}
@Test
public void changingHeadsetsRebindsShareToTheSelectedOriginal() throws Exception {
assertFalse(SyncService.running());
String aPin = "switch-a-" + UUID.randomUUID(), bPin = "switch-b-" + UUID.randomUUID();
File a = SyncService.library(context, aPin), b = SyncService.library(context, bPin);
SharedPreferences prefs = SyncService.prefs(context);
String previousPin = prefs.getString("pin", null);
boolean previousGallery = prefs.getBoolean("gallery", true);
Instrumentation instrumentation = InstrumentationRegistry.getInstrumentation();
AtomicReference<Intent> shared = new AtomicReference<>();
Instrumentation.ActivityMonitor monitor =
new Instrumentation.ActivityMonitor() {
@Override
public Instrumentation.ActivityResult onStartActivity(Intent intent) {
if (!Intent.ACTION_CHOOSER.equals(intent.getAction())) return null;
shared.set(intent.getParcelableExtra(Intent.EXTRA_INTENT));
return new Instrumentation.ActivityResult(Activity.RESULT_CANCELED, null);
}
};
Activity activity = null;
try {
SyncEngine.Shot shot = new SyncEngine.Shot("123/108/20261024120000_1.png", 4, 100);
for (File root : List.of(a, b)) {
byte[] original = root.equals(a) ? new byte[] {1, 2, 3, 4} : new byte[] {5, 6, 7, 8};
SyncEngine.Transport transfer =
new SyncEngine.Transport() {
public void download(SyncEngine.Shot s, File file) throws Exception {
Files.write(file.toPath(), original);
}
public boolean validImage(File file) {
return true;
}
};
SyncEngine engine = new SyncEngine(root, false);
engine.reconcile(listing(), transfer, (s, f, e) -> false);
engine.reconcile(listing(shot), transfer, (s, f, e) -> false);
engine.reconcile(listing(shot), transfer, (s, f, e) -> false);
}
assertTrue(prefs.edit().putString("pin", aPin).putBoolean("gallery", false).commit());
activity =
instrumentation.startActivitySync(
new Intent(context, MainActivity.class)
.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK | Intent.FLAG_ACTIVITY_CLEAR_TASK));
instrumentation.addMonitor(monitor);
Activity screen = activity;
// The library loads on a background executor, so wait for it to render, not for idle.
Uri first = shareWhenRendered(instrumentation, screen, shared, null);
try (InputStream in = context.getContentResolver().openInputStream(first)) {
assertArrayEquals(new byte[] {1, 2, 3, 4}, read(in));
}
assertTrue(prefs.edit().putString("pin", bPin).commit());
// Until the screen notices the new pin it may still share A; it then clears A's list
// (no button while B loads) and shows only B's original.
Uri selected = shareWhenRendered(instrumentation, screen, shared, first);
try (InputStream in = context.getContentResolver().openInputStream(selected)) {
assertArrayEquals(new byte[] {5, 6, 7, 8}, read(in));
}
// Once B is shown, A's list is gone: one button, and sharing again still targets B.
AtomicReference<Integer> buttons = new AtomicReference<>();
shared.set(null);
instrumentation.runOnMainSync(
() -> {
View content = screen.findViewById(android.R.id.content);
buttons.set(shareButtons(content));
Button button = shareButton(content);
if (button != null) assertTrue(button.performClick());
});
assertEquals("The previous headset's original must be gone", Integer.valueOf(1), buttons.get());
assertNotNull(shared.get());
assertEquals(selected, shared.get().getParcelableExtra(Intent.EXTRA_STREAM));
} finally {
instrumentation.removeMonitor(monitor);
Activity screen = activity;
if (screen != null) instrumentation.runOnMainSync(screen::finish);
SharedPreferences.Editor restore = prefs.edit().putBoolean("gallery", previousGallery);
if (previousPin == null) restore.remove("pin");
else restore.putString("pin", previousPin);
assertTrue(restore.commit());
for (File root : List.of(a, b)) {
if (!root.exists()) continue;
try (java.util.stream.Stream<java.nio.file.Path> paths = Files.walk(root.toPath())) {
for (java.nio.file.Path path :
paths.sorted(Comparator.reverseOrder()).collect(java.util.stream.Collectors.toList()))
Files.delete(path);
}
}
}
}
/**
* Clicks the share button once the asynchronously loaded library shows one and the chosen
* original differs from {@code previous}. A missing button is the expected loading state.
*/
private static Uri shareWhenRendered(
Instrumentation instrumentation,
Activity screen,
AtomicReference<Intent> shared,
Uri previous)
throws InterruptedException {
long deadline = System.nanoTime() + java.util.concurrent.TimeUnit.SECONDS.toNanos(15);
do {
shared.set(null);
instrumentation.runOnMainSync(
() -> {
Button button = shareButton(screen.findViewById(android.R.id.content));
if (button != null) assertTrue(button.performClick());
});
Intent sent = shared.get();
if (sent != null) {
Uri uri = sent.getParcelableExtra(Intent.EXTRA_STREAM);
assertNotNull(uri);
if (!uri.equals(previous)) return uri;
}
// Re-check after the screen's next refresh or library load; bounded by the deadline.
Thread.sleep(50);
} while (System.nanoTime() < deadline);
fail("The selected headset's original never became shareable");
return null;
}
private static int shareButtons(View view) {
if (view instanceof Button && "Open / share original".contentEquals(((Button) view).getText()))
return 1;
int count = 0;
if (view instanceof ViewGroup) {
ViewGroup group = (ViewGroup) view;
for (int i = 0; i < group.getChildCount(); i++) count += shareButtons(group.getChildAt(i));
}
return count;
}
private static Button shareButton(View view) {
if (view instanceof Button && "Open / share original".contentEquals(((Button) view).getText()))
return (Button) view;
if (view instanceof ViewGroup) {
ViewGroup group = (ViewGroup) view;
for (int i = 0; i < group.getChildCount(); i++) {
Button found = shareButton(group.getChildAt(i));
if (found != null) return found;
}
}
return null;
}
private Uri insert(String name, boolean pending, byte[] bytes) throws Exception {
ContentResolver resolver = context.getContentResolver();
ContentValues values = new ContentValues();
values.put(MediaStore.Images.Media.DISPLAY_NAME, name);
values.put(MediaStore.Images.Media.RELATIVE_PATH, GalleryExporter.FOLDER);
values.put(MediaStore.Images.Media.MIME_TYPE, "image/png");
values.put(MediaStore.Images.Media.IS_PENDING, 1);
Uri uri =
resolver.insert(
MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY), values);
assertNotNull(uri);
cleanup.add(uri);
try (OutputStream out = resolver.openOutputStream(uri, "wt")) {
out.write(bytes);
}
if (!pending) {
ContentValues done = new ContentValues();
done.put(MediaStore.Images.Media.IS_PENDING, 0);
assertEquals(1, resolver.update(uri, done, null, null));
}
return uri;
}
private boolean exists(Uri uri) {
@SuppressWarnings("deprecation")
Uri includingPending = MediaStore.setIncludePending(uri);
try (Cursor c =
context.getContentResolver().query(includingPending, new String[] {"_id"}, null, null, null)) {
return c != null && c.moveToFirst();
}
}
@Test
public void optOutDiscardsInterruptedPendingRowsButKeepsPublishedImages() throws Exception {
assertFalse(SyncService.running());
File files = Files.createTempDirectory(context.getCacheDir().toPath(), "pending-").toFile();
String preferences = "pending-test-" + UUID.randomUUID();
Context isolated =
new ContextWrapper(context) {
@Override
public File getFilesDir() {
return files;
}
@Override
public SharedPreferences getSharedPreferences(String name, int mode) {
return context.getSharedPreferences(preferences, mode);
}
};
try {
Bitmap bitmap = Bitmap.createBitmap(8, 8, Bitmap.Config.ARGB_8888);
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, bytes);
bitmap.recycle();
// A pending row with app-only receipts, from a retained (not currently paired) library.
SyncEngine.Shot shot = new SyncEngine.Shot("123/108/20261024120003_1.png", 4, 100);
SyncEngine engine = new SyncEngine(SyncService.library(isolated, "retained"), true);
engine.reconcile(listing(), null, null);
Uri interrupted = insert(name("retained", shot), true, bytes.toByteArray());
Uri published = insert("published-" + UUID.randomUUID() + ".png", false, bytes.toByteArray());
assertTrue(SyncService.prefs(isolated).edit().putBoolean("gallery", true).commit());
SyncService.gallery(isolated, false);
assertFalse("Hidden pending row must not outlive the opt-out", exists(interrupted));
// Proven deleted by the product; deleting it again throws SecurityException on API 30+.
cleanup.remove(interrupted);
assertTrue("A published Gallery image must stay", exists(published));
SyncService.gallery(isolated, true);
assertTrue(exists(published));
assertFalse(exists(interrupted));
} finally {
SyncService.prefs(isolated).edit().clear().commit();
try (java.util.stream.Stream<java.nio.file.Path> paths = Files.walk(files.toPath())) {
for (java.nio.file.Path path :
paths.sorted(Comparator.reverseOrder()).collect(java.util.stream.Collectors.toList()))
Files.delete(path);
}
}
}
/** Optional real-headset provisioning. Only this test APK exposes the hook. */
@Test
public void provisionRealHeadsetIfRequested() throws Exception {
android.os.Bundle args = InstrumentationRegistry.getArguments();
Assume.assumeTrue("Optional real-headset provisioning", "true".equals(args.getString("live")));
String host = args.getString("host"), pin = args.getString("pin");
assertNotNull(host);
assertNotNull(pin);
assertEquals(
"Probe must return the expected host key before authentication",
pin,
FrameConnection.probe(host, 22, "steamos"));
String publicKey = new PhoneIdentity().authorizedKey(SyncService.install(context));
Files.write(
new File(context.getFilesDir(), "live-test-public.pub").toPath(),
(publicKey + "\n").getBytes(StandardCharsets.US_ASCII));
assertTrue(
SyncService.prefs(context)
.edit()
.putString("host", host)
.putString("user", "steamos")
.putInt("port", 22)
.putString("pin", pin)
.commit());
}
@Test
public void verifyRealHeadsetIfRequested() throws Exception {
android.os.Bundle args = InstrumentationRegistry.getArguments();
String id = args.getString("verifyId");
Assume.assumeNotNull(id);
String pin = SyncService.prefs(context).getString("pin", null);
assertNotNull(pin);
// Read-only: verification must not rewrite the product's journal.
SyncEngine.Item item =
SyncEngine.snapshot(SyncService.library(context, pin)).stream().filter(i -> i.shot.id.equals(id)).findFirst().orElseThrow();
assertTrue(item.file.exists());
String expected = args.getString("sha256");
assertNotNull(expected);
assertEquals(expected, sha(Files.readAllBytes(item.file.toPath())));
boolean gallery = "true".equals(args.getString("gallery"));
Uri saved = row(pin, item.shot);
cleanup.clear();
if (gallery) {
assertEquals("saved", item.destination);
assertNotNull(saved);
try (InputStream in = context.getContentResolver().openInputStream(saved)) {
assertEquals(expected, sha(read(in)));
}
} else {
assertEquals("appOnly", item.destination);
assertNull(saved);
}
}
private static String sha(byte[] bytes) throws Exception {
StringBuilder out = new StringBuilder();
for (byte b : java.security.MessageDigest.getInstance("SHA-256").digest(bytes))
out.append(String.format(java.util.Locale.ROOT, "%02x", b & 255));
return out.toString();
}
private static byte[] read(InputStream input) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] b = new byte[4096];
int n;
while ((n = input.read(b)) != -1) out.write(b, 0, n);
return out.toByteArray();
}
}
+19
View File
@@ -0,0 +1,19 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.CHANGE_NETWORK_STATE" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_CONNECTED_DEVICE" />
<application android:label="Frame Control" android:theme="@style/AppTheme"
android:icon="@drawable/ic_frame" android:allowBackup="false" android:dataExtractionRules="@xml/data_extraction"
android:networkSecurityConfig="@xml/network_security" android:supportsRtl="true">
<activity android:name=".MainActivity" android:exported="true" android:configChanges="orientation|screenSize|keyboardHidden">
<intent-filter><action android:name="android.intent.action.MAIN" /><category android:name="android.intent.category.LAUNCHER" /></intent-filter>
</activity>
<service android:name=".SyncService" android:exported="false" android:foregroundServiceType="connectedDevice" />
<provider android:name="androidx.core.content.FileProvider" android:authorities="${applicationId}.files" android:exported="false" android:grantUriPermissions="true">
<meta-data android:name="android.support.FILE_PROVIDER_PATHS" android:resource="@xml/file_paths" />
</provider>
</application>
</manifest>
@@ -0,0 +1,461 @@
package com.saphid.framecontrol;
import android.content.Context;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonParser;
import com.jcraft.jsch.*;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.*;
import java.util.concurrent.*;
import java.util.regex.*;
/** All headset traffic goes through one pinned, authenticated SSH connection. */
final class FrameConnection implements AutoCloseable, SyncEngine.Transport {
private final Context context;
private Session session;
private ChannelExec server;
private PipedOutputStream serverInput;
private int localPort;
private String apiKey;
private static final String HOST_ALGORITHMS = "ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256";
FrameConnection(Context context) {
this.context = context.getApplicationContext();
}
static String fingerprint(String pin) throws Exception {
return "SHA256:"
+ Base64.getEncoder()
.withoutPadding()
.encodeToString(
MessageDigest.getInstance("SHA-256").digest(Base64.getDecoder().decode(pin)));
}
static String hash(String value) throws Exception {
byte[] bytes =
MessageDigest.getInstance("SHA-256").digest(value.getBytes(StandardCharsets.UTF_8));
StringBuilder result = new StringBuilder();
for (byte b : bytes) result.append(String.format(Locale.ROOT, "%02x", b & 255));
return result.toString();
}
static String quote(String s) {
return "'" + s.replace("'", "'\\''") + "'";
}
private static Session session(JSch jsch, String host, int port, String user) throws Exception {
if (host.trim().isEmpty()
|| host.length() > 253
|| host.contains("/")
|| host.chars().anyMatch(Character::isWhitespace)
|| port < 1
|| port > 65535
|| !user.matches("[a-zA-Z0-9_-]{1,32}"))
throw new IOException("Check the headset address, port and username.");
Session s = jsch.getSession(user, host, port);
s.setConfig("StrictHostKeyChecking", "yes");
s.setConfig("server_host_key", HOST_ALGORITHMS);
s.setConfig("PreferredAuthentications", "publickey");
s.setTimeout(20000);
s.setServerAliveInterval(15000);
s.setServerAliveCountMax(2);
return s;
}
private static final class PinRepository implements HostKeyRepository {
String observed;
final String expected;
PinRepository(String expected) {
this.expected = expected;
}
@Override
public int check(String host, byte[] key) {
observed = Base64.getEncoder().encodeToString(key);
return observed.equals(expected) ? OK : CHANGED;
}
@Override
public void add(HostKey k, UserInfo u) {}
@Override
public void remove(String h, String t) {}
@Override
public void remove(String h, String t, byte[] k) {}
@Override
public String getKnownHostsRepositoryID() {
return "Frame Control pinned headset";
}
@Override
public HostKey[] getHostKey() {
return new HostKey[0];
}
@Override
public HostKey[] getHostKey(String h, String t) {
return new HostKey[0];
}
}
/** Intentionally refuses the key exchange before any authentication is sent. */
static String probe(String host, int port, String user) throws Exception {
JSch jsch = new JSch();
PinRepository pin = new PinRepository(null);
jsch.setHostKeyRepository(pin);
Session s = session(jsch, host, port, user);
try {
s.connect(10000);
} catch (JSchException e) {
if (pin.observed == null) throw e;
} finally {
s.disconnect();
}
return pin.observed;
}
static void pair(String host, int port, String user, String pin, String password, String install)
throws Exception {
JSch jsch = new JSch();
jsch.setHostKeyRepository(new PinRepository(pin));
Session s = session(jsch, host, port, user);
s.setConfig("PreferredAuthentications", "password,keyboard-interactive");
s.setPassword(password);
s.setUserInfo(
new UserInfo() {
public String getPassphrase() {
return null;
}
public String getPassword() {
return password;
}
public boolean promptPassword(String x) {
return true;
}
public boolean promptPassphrase(String x) {
return false;
}
public boolean promptYesNo(String x) {
return false;
}
public void showMessage(String x) {}
});
try {
s.connect(15000);
run(s, authorizeCommand(new PhoneIdentity().authorizedKey(install)), 10000);
} finally {
s.disconnect();
}
// Verify the new key before storing any pairing settings.
JSch verified = new JSch();
verified.setHostKeyRepository(new PinRepository(pin));
verified.addIdentity(new PhoneIdentity(), null);
Session check = session(verified, host, port, user);
try {
check.connect(15000);
} finally {
check.disconnect();
}
}
/**
* Idempotently appends {@code key}. A last line without a newline is terminated first, so the
* existing key is never merged with the new one (the same guard as the iPhone app).
*/
static String authorizeCommand(String key) {
String file = "~/.ssh/authorized_keys";
return "umask 077; mkdir -p ~/.ssh && chmod 700 ~/.ssh && touch "
+ file
+ " && chmod 600 "
+ file
+ " && { grep -qxF "
+ quote(key)
+ " "
+ file
+ " || { if [ -s "
+ file
+ " ] && [ -n \"$(tail -c 1 "
+ file
+ ")\" ]; then printf '\\n' >> "
+ file
+ "; fi; printf '%s\\n' "
+ quote(key)
+ " >> "
+ file
+ "; }; }";
}
/**
* Lists Steam account directories, including ones with no screenshots. Prints "absent" when
* Steam has not created userdata yet; any other failure exits non-zero.
*/
static final String ACCOUNTS_COMMAND =
"d=\"$HOME/.local/share/Steam/userdata\"; if [ ! -e \"$d\" ]; then echo absent; else"
+ " cd \"$d\" || exit 1; echo present; for a in *; do case \"$a\" in ''|*[!0-9]*) ;;"
+ " *) if [ -d \"$a\" ]; then printf '%s\\n' \"$a\"; fi ;; esac; done; fi";
static Set<String> parseAccounts(String output) throws IOException {
String[] lines = output.split("\n", -1);
Set<String> accounts = new TreeSet<>();
if (lines.length == 2 && lines[0].equals("absent") && lines[1].isEmpty()) return accounts;
if (lines.length < 2 || !lines[0].equals("present") || !lines[lines.length - 1].isEmpty())
throw new IOException("Could not read Steam accounts on the headset.");
for (int i = 1; i < lines.length - 1; i++) {
if (!lines[i].matches("[0-9]+")) throw new IOException("Unexpected Steam account entry.");
// The headset server ignores longer ids; so does the screenshot listing.
if (lines[i].length() <= 12) accounts.add(lines[i]);
}
return accounts;
}
static String run(Session s, String command, int timeout) throws Exception {
ChannelExec channel = (ChannelExec) s.openChannel("exec");
channel.setCommand(command);
channel.setInputStream(null);
channel.setErrStream(
new OutputStream() {
public void write(int b) {}
public void write(byte[] b, int offset, int length) {}
});
InputStream input = channel.getInputStream();
long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout);
ByteArrayOutputStream output = new ByteArrayOutputStream();
try {
channel.connect(timeout);
while (true) {
while (input.available() > 0) {
byte[] b = new byte[Math.min(input.available(), 8192)];
int n = input.read(b);
if (n > 0) output.write(b, 0, n);
if (output.size() > 2 * 1024 * 1024)
throw new IOException("Headset command returned too much data.");
}
if (channel.isClosed() && input.available() == 0) break;
if (System.nanoTime() > deadline) throw new IOException("The headset command timed out.");
Thread.sleep(25);
}
if (channel.getExitStatus() != 0)
throw new IOException("The headset command failed (" + channel.getExitStatus() + ").");
return output.toString(StandardCharsets.UTF_8.name());
} finally {
channel.disconnect();
}
}
void connect(String host, int port, String user, String pin, String install) throws Exception {
JSch jsch = new JSch();
jsch.setHostKeyRepository(new PinRepository(pin));
jsch.addIdentity(new PhoneIdentity(), null);
session = session(jsch, host, port, user);
session.connect(15000);
String version;
try (InputStream in = context.getAssets().open("frame-bundle.version")) {
version = new String(readBounded(in, 128), StandardCharsets.US_ASCII).trim();
}
if (!version.matches("[0-9a-f]{16}")) throw new IOException("Invalid bundled server version.");
String dir = ".cache/frame-control/" + version;
String present =
run(session, "test -f " + dir + "/ui/server.py && printf ready || true", 10000);
if (!present.equals("ready")) {
run(session, "mkdir -p .cache/frame-control", 10000);
String temp = dir + ".android-" + UUID.randomUUID();
ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");
try (InputStream in = context.getAssets().open("frame-bundle.tar.gz")) {
sftp.connect(10000);
sftp.put(in, temp + ".tar.gz");
} finally {
sftp.disconnect();
}
run(
session,
"mkdir "
+ temp
+ " && tar xzf "
+ temp
+ ".tar.gz -C "
+ temp
+ " && "
+ "(test -f "
+ dir
+ "/ui/server.py && rm -rf "
+ temp
+ " || mv "
+ temp
+ " "
+ dir
+ ") && rm -f "
+ temp
+ ".tar.gz",
60000);
}
apiKey = UUID.randomUUID().toString().replace("-", "");
server = (ChannelExec) session.openChannel("exec");
serverInput = new PipedOutputStream();
server.setInputStream(new PipedInputStream(serverInput));
server.setCommand(
"cd "
+ dir
+ " && FRAME_LOCAL=1 FRAME_UI_KEY="
+ apiKey
+ " FRAME_DEVICE="
+ quote(host)
+ " FRAME_CLIENT="
+ quote(install)
+ " exec python3 -I -u -B \"$PWD/ui/server.py\" --port 0 --exit-on-eof 2>&1");
InputStream input = server.getInputStream();
server.connect(10000);
ByteArrayOutputStream startup = new ByteArrayOutputStream();
Pattern portPattern = Pattern.compile("Frame Control on http://127\\.0\\.0\\.1:([0-9]+)\\s");
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(40);
int portFound = 0;
while (System.nanoTime() < deadline) {
while (input.available() > 0) {
startup.write(input.read());
if (startup.size() > 65536) throw new IOException("The headset server did not start.");
}
Matcher m = portPattern.matcher(startup.toString(StandardCharsets.UTF_8.name()));
if (m.find()) {
portFound = Integer.parseInt(m.group(1));
break;
}
if (server.isClosed())
throw new IOException("The headset server stopped before it was ready.");
Thread.sleep(50);
}
if (portFound < 1 || portFound > 65535)
throw new IOException("The headset server did not answer in time.");
localPort = session.setPortForwardingL("127.0.0.1", 0, "127.0.0.1", portFound);
// Drain the session-bound server output; never log session keys or user data.
Thread drain =
new Thread(
() -> {
try {
byte[] b = new byte[4096];
while (input.read(b) != -1) {}
} catch (IOException ignored) {
}
},
"frame-server-output");
drain.setDaemon(true);
drain.start();
}
String page() {
return "http://127.0.0.1:" + localPort + "/?key=" + apiKey;
}
boolean connected() {
return session != null && session.isConnected() && server != null && !server.isClosed();
}
private HttpURLConnection request(String path) throws Exception {
HttpURLConnection c =
(HttpURLConnection) new URL("http://127.0.0.1:" + localPort + path).openConnection();
c.setConnectTimeout(10000);
c.setReadTimeout(20000);
c.setInstanceFollowRedirects(false);
c.setRequestProperty("X-Frame-UI", apiKey);
return c;
}
/** Accounts are read first; one that appears before the screenshot query is still new. */
SyncEngine.Listing listing() throws Exception {
Set<String> accounts = parseAccounts(run(session, ACCOUNTS_COMMAND, 10000));
return new SyncEngine.Listing(accounts, shots());
}
List<SyncEngine.Shot> shots() throws Exception {
HttpURLConnection c = request("/api/shots");
try (InputStream in = c.getInputStream()) {
byte[] body = readBounded(in, 2 * 1024 * 1024);
JsonArray array =
JsonParser.parseString(new String(body, StandardCharsets.UTF_8))
.getAsJsonObject()
.getAsJsonArray("shots");
List<SyncEngine.Shot> shots = new ArrayList<>();
for (JsonElement e : array) {
var s = e.getAsJsonObject();
shots.add(
new SyncEngine.Shot(
s.get("id").getAsString(), s.get("size").getAsLong(), s.get("time").getAsDouble()));
}
return shots;
} finally {
c.disconnect();
}
}
private static byte[] readBounded(InputStream input, int limit) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] b = new byte[8192];
int n;
while ((n = input.read(b)) != -1) {
if (out.size() + n > limit) throw new IOException("The headset response is too large.");
out.write(b, 0, n);
}
return out.toByteArray();
}
@Override
public void download(SyncEngine.Shot shot, File destination) throws Exception {
HttpURLConnection c =
request("/api/shots/image?id=" + URLEncoder.encode(shot.id, StandardCharsets.UTF_8.name()));
try (InputStream in = c.getInputStream();
FileOutputStream out = new FileOutputStream(destination)) {
byte[] b = new byte[65536];
long count = 0;
int n;
while ((n = in.read(b)) != -1) {
count += n;
if (count > shot.size) throw new IOException("Screenshot changed while downloading.");
out.write(b, 0, n);
}
out.getFD().sync();
} finally {
c.disconnect();
}
}
@Override
public boolean validImage(File file) {
android.graphics.BitmapFactory.Options options = new android.graphics.BitmapFactory.Options();
options.inJustDecodeBounds = true;
android.graphics.BitmapFactory.decodeFile(file.getPath(), options);
if (options.outWidth <= 0
|| options.outHeight <= 0
|| options.outWidth > 16384
|| options.outHeight > 16384
|| (long) options.outWidth * options.outHeight > 60_000_000) return false;
options.inJustDecodeBounds = false;
options.inSampleSize = Math.max(1, Math.max(options.outWidth, options.outHeight) / 2048);
android.graphics.Bitmap b = android.graphics.BitmapFactory.decodeFile(file.getPath(), options);
if (b == null) return false;
b.recycle();
return true;
}
@Override
public void close() {
try {
if (serverInput != null) serverInput.close();
} catch (IOException ignored) {
}
if (server != null) server.disconnect();
if (session != null) session.disconnect();
}
}
@@ -0,0 +1,161 @@
package com.saphid.framecontrol;
import android.content.*;
import android.database.Cursor;
import android.net.Uri;
import android.provider.MediaStore;
import java.io.*;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.BooleanSupplier;
/** Deterministic, app-owned MediaStore rows make interrupted exports retryable. */
final class GalleryExporter implements SyncEngine.Exporter {
static final String FOLDER = "Pictures/Frame Control/";
/**
* Serializes every export in the process, across exporter instances and service sessions. A
* stopped session's exporter may still be writing when a new session starts; without this, the
* new one could publish the same deterministic row and the old one's cancellation would then
* delete it. Lock order is EXPORTS, then the settings lock. Never take EXPORTS while holding
* the settings lock: {@link #discardPending} runs under it and must not wait for exports.
*/
static final ReentrantLock EXPORTS = new ReentrantLock();
private static final long EXPORT_WAIT_SECONDS = 30;
private final Context context;
private final String headset;
private final Object settingsLock;
GalleryExporter(Context context, String headset, Object settingsLock) {
this.context = context.getApplicationContext();
this.headset = headset;
this.settingsLock = settingsLock;
}
/**
* Best-effort removal of this app's hidden pending rows in the Frame Control folder. Only an
* interrupted export leaves one; published (non-pending) images are never touched. Callers
* hold the settings lock, so no export can be publishing a row at the same time.
*/
static int discardPending(Context context) {
ContentResolver resolver = context.getContentResolver();
Uri collection = MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY);
@SuppressWarnings("deprecation")
Uri includingPending = MediaStore.setIncludePending(collection);
List<Uri> rows = new ArrayList<>();
int removed = 0;
try {
try (Cursor cursor =
resolver.query(
includingPending,
new String[] {"_id"},
MediaStore.Images.Media.IS_PENDING
+ "=1 AND "
+ MediaStore.Images.Media.RELATIVE_PATH
+ "=? AND "
+ MediaStore.Images.Media.OWNER_PACKAGE_NAME
+ "=?",
new String[] {FOLDER, context.getPackageName()},
null)) {
while (cursor != null && cursor.moveToNext())
rows.add(ContentUris.withAppendedId(collection, cursor.getLong(0)));
}
for (Uri row : rows) removed += resolver.delete(row, null, null);
} catch (RuntimeException ignored) {
// Android 11+ also expires abandoned pending rows; never fail the setting change here.
}
return removed;
}
@Override
public boolean export(SyncEngine.Shot shot, File file, BooleanSupplier eligible)
throws Exception {
// Interruptible and bounded: Stop still ends a waiting session, and a stuck earlier write
// only delays this item to a later poll.
if (!EXPORTS.tryLock(EXPORT_WAIT_SECONDS, TimeUnit.SECONDS))
throw new IOException("Gallery is still finishing an earlier save.");
try {
return exportLocked(shot, file, eligible);
} finally {
EXPORTS.unlock();
}
}
private boolean exportLocked(SyncEngine.Shot shot, File file, BooleanSupplier eligible)
throws Exception {
String name = shot.id.substring(shot.id.lastIndexOf('/') + 1);
int dot = name.lastIndexOf('.');
name =
name.substring(0, dot)
+ "_"
+ FrameConnection.hash(headset + "/" + shot.id).substring(0, 20)
+ name.substring(dot);
String folder = FOLDER;
ContentResolver resolver = context.getContentResolver();
Uri collection = MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY);
// API 29 excludes even app-owned pending rows unless explicitly included.
@SuppressWarnings("deprecation")
Uri recoveryCollection = MediaStore.setIncludePending(collection);
Uri row = null;
boolean pending = true;
try (Cursor cursor =
resolver.query(
recoveryCollection,
new String[] {"_id", MediaStore.Images.Media.IS_PENDING},
MediaStore.Images.Media.DISPLAY_NAME
+ "=? AND "
+ MediaStore.Images.Media.RELATIVE_PATH
+ "=? AND "
+ MediaStore.Images.Media.OWNER_PACKAGE_NAME
+ "=?",
new String[] {name, folder, context.getPackageName()},
null)) {
if (cursor != null && cursor.moveToFirst()) {
row = ContentUris.withAppendedId(collection, cursor.getLong(0));
pending = cursor.getInt(1) != 0;
}
}
// A completed row is the commit receipt, including a crash before JSON save.
if (row != null && !pending) return true;
if (!eligible.getAsBoolean()) return false;
if (row == null) {
ContentValues values = new ContentValues();
values.put(MediaStore.Images.Media.DISPLAY_NAME, name);
values.put(MediaStore.Images.Media.RELATIVE_PATH, folder);
values.put(
MediaStore.Images.Media.MIME_TYPE, shot.id.endsWith(".png") ? "image/png" : "image/jpeg");
values.put(MediaStore.Images.Media.IS_PENDING, 1);
values.put(MediaStore.Images.Media.DATE_TAKEN, (long) (shot.time * 1000));
row = resolver.insert(collection, values);
if (row == null) throw new IOException("Gallery could not create the screenshot.");
}
try {
try (InputStream in = new FileInputStream(file);
OutputStream out = resolver.openOutputStream(row, "wt")) {
if (out == null) throw new IOException("Gallery could not open the screenshot.");
byte[] b = new byte[65536];
int n;
while ((n = in.read(b)) != -1) out.write(b, 0, n);
}
synchronized (settingsLock) {
if (!eligible.getAsBoolean()) {
// Holding EXPORTS, no other exporter can have published this row since we found it.
resolver.delete(row, null, null);
return false;
}
ContentValues done = new ContentValues();
done.put(MediaStore.Images.Media.IS_PENDING, 0);
if (resolver.update(row, done, null, null) != 1)
throw new IOException("Gallery could not finish saving the screenshot.");
}
return true;
} catch (Exception e) {
// A pending row is retained and rewritten on retry. A completed row is
// discovered before any future insert, even after process death.
throw e;
}
}
}
@@ -0,0 +1,553 @@
package com.saphid.framecontrol;
import android.Manifest;
import android.app.*;
import android.content.*;
import android.content.pm.PackageManager;
import android.graphics.Color;
import android.net.Uri;
import android.os.*;
import android.text.InputType;
import android.view.*;
import android.webkit.*;
import android.widget.*;
import androidx.core.content.FileProvider;
import java.util.*;
import java.util.concurrent.*;
public final class MainActivity extends Activity {
private final ExecutorService network = Executors.newSingleThreadExecutor();
// Journal parsing and file checks stay off the main thread.
private final ExecutorService library = Executors.newSingleThreadExecutor();
private boolean loading;
private String renderedPin;
private final Handler handler = new Handler(Looper.getMainLooper());
private LinearLayout root, downloads;
private FrameLayout content;
private ScrollView companion;
private EditText host, port, user;
private TextView status, empty;
private Switch gallery;
private Button pair, connect, stop;
private WebView web;
private String loadedPage, librarySignature = "";
private int downloadLimit = 100;
private boolean busy, visible, controls;
private ValueCallback<Uri[]> fileSelection;
private static final int INK = Color.rgb(232, 240, 238),
MUTED = Color.rgb(164, 183, 179),
ACCENT = Color.rgb(124, 216, 195);
private static final String EMPTY =
"Pair and start syncing. Existing headset screenshots stay on the headset; new captures"
+ " will appear here.";
private final Runnable refresh =
new Runnable() {
public void run() {
if (!visible) return;
update();
handler.postDelayed(this, 1500);
}
};
private int dp(int n) {
return Math.round(getResources().getDisplayMetrics().density * n);
}
private TextView text(String value, int size, int color) {
TextView v = new TextView(this);
v.setText(value);
v.setTextSize(size);
v.setTextColor(color);
v.setPadding(0, dp(6), 0, dp(6));
return v;
}
private Button button(String label, Runnable action) {
Button b = new Button(this);
b.setText(label);
b.setAllCaps(false);
b.setOnClickListener(v -> action.run());
return b;
}
private EditText field(String label, String value, int type) {
companionBody.addView(text(label, 13, MUTED));
EditText v = new EditText(this);
v.setSingleLine(true);
v.setText(value);
v.setHint(label);
v.setContentDescription(label);
v.setInputType(type);
v.setTextColor(INK);
v.setSaveEnabled(false);
companionBody.addView(v);
return v;
}
private LinearLayout companionBody;
@Override
public void onCreate(Bundle state) {
super.onCreate(state);
root = new LinearLayout(this);
root.setOrientation(LinearLayout.VERTICAL);
root.setBackgroundColor(Color.rgb(17, 24, 25));
root.setPadding(dp(20), dp(12), dp(20), dp(12));
root.setOnApplyWindowInsetsListener(
(v, insets) -> {
if (Build.VERSION.SDK_INT >= 30) {
android.graphics.Insets i =
insets.getInsets(
WindowInsets.Type.systemBars() | WindowInsets.Type.displayCutout());
v.setPadding(dp(20) + i.left, dp(12) + i.top, dp(20) + i.right, dp(12) + i.bottom);
} else
v.setPadding(
dp(20) + insets.getSystemWindowInsetLeft(),
dp(12) + insets.getSystemWindowInsetTop(),
dp(20) + insets.getSystemWindowInsetRight(),
dp(12) + insets.getSystemWindowInsetBottom());
return insets;
});
root.addView(text("FRAME CONTROL", 13, ACCENT));
root.addView(text("Your headset, closer.", 27, INK));
status = text(SyncService.state(), 14, MUTED);
status.setAccessibilityLiveRegion(View.ACCESSIBILITY_LIVE_REGION_POLITE);
root.addView(status);
LinearLayout tabs = new LinearLayout(this);
tabs.addView(
button("Screenshots", () -> show(false)), new LinearLayout.LayoutParams(0, dp(48), 1));
tabs.addView(
button("Headset controls", () -> show(true)), new LinearLayout.LayoutParams(0, dp(48), 1));
root.addView(tabs);
content = new FrameLayout(this);
root.addView(content, new LinearLayout.LayoutParams(-1, 0, 1));
companion = new ScrollView(this);
companion.setFillViewport(true);
companionBody = new LinearLayout(this);
companionBody.setOrientation(LinearLayout.VERTICAL);
companion.addView(companionBody);
SharedPreferences p = SyncService.prefs(this);
companionBody.addView(text("Connect your Steam Frame", 20, INK));
companionBody.addView(
text(
"Enable Developer Mode and connect to the same network, or use your headset’s Tailscale"
+ " address.",
14,
MUTED));
host =
field(
"Headset address",
p.getString("host", "frame.local"),
InputType.TYPE_CLASS_TEXT | InputType.TYPE_TEXT_VARIATION_URI);
port = field("SSH port", Integer.toString(p.getInt("port", 22)), InputType.TYPE_CLASS_NUMBER);
user =
field(
"SSH username",
p.getString("user", "steamos"),
InputType.TYPE_CLASS_TEXT | InputType.TYPE_TEXT_VARIATION_VISIBLE_PASSWORD);
pair = button("Pair headset", this::probe);
companionBody.addView(pair);
LinearLayout actions = new LinearLayout(this);
connect = button("Start syncing", this::startSync);
stop = button("Stop", () -> stopService(new Intent(this, SyncService.class)));
actions.addView(connect, new LinearLayout.LayoutParams(0, dp(52), 1));
actions.addView(stop, new LinearLayout.LayoutParams(0, dp(52), 1));
companionBody.addView(actions);
companionBody.addView(text("Screenshot downloads", 20, INK));
gallery = new Switch(this);
gallery.setText("Save screenshots to Gallery");
gallery.setTextColor(INK);
gallery.setTextSize(16);
gallery.setMinHeight(dp(56));
gallery.setChecked(p.getBoolean("gallery", true));
gallery.setOnCheckedChangeListener(
(b, value) -> {
try {
SyncService.gallery(this, value);
librarySignature = "";
} catch (Exception e) {
gallery.setOnCheckedChangeListener(null);
gallery.setChecked(!value);
error(SyncService.message(e));
recreate();
}
});
companionBody.addView(gallery);
companionBody.addView(
text(
"New captures download automatically while sync is running. A notification keeps the"
+ " connection visible. Stop syncing to disconnect. Android may delay transfers"
+ " while the phone is locked or in battery-saving modes.",
14,
MUTED));
companionBody.addView(
text(
"Turning Gallery off keeps new downloads in this app. Turning it on again won’t export"
+ " earlier app-only downloads.",
14,
MUTED));
empty = text(EMPTY, 14, MUTED);
companionBody.addView(empty);
downloads = new LinearLayout(this);
downloads.setOrientation(LinearLayout.VERTICAL);
companionBody.addView(downloads);
show(false);
setContentView(root);
}
private void show(boolean useControls) {
controls = useControls;
content.removeAllViews();
if (!controls) {
content.addView(companion, new FrameLayout.LayoutParams(-1, -1));
return;
}
String page = SyncService.page();
if (page == null) {
loadedPage = null;
if (web != null) web.loadUrl("about:blank");
content.addView(text("Start syncing to open your headset’s controls.", 17, MUTED));
return;
}
if (web == null) {
web = new WebView(this);
web.setBackgroundColor(Color.rgb(17, 24, 25));
WebSettings settings = web.getSettings();
settings.setJavaScriptEnabled(true);
settings.setDomStorageEnabled(true);
settings.setAllowFileAccess(false);
settings.setAllowContentAccess(false);
settings.setMixedContentMode(WebSettings.MIXED_CONTENT_NEVER_ALLOW);
settings.setSaveFormData(false);
web.setWebViewClient(
new WebViewClient() {
@Override
public boolean shouldOverrideUrlLoading(WebView view, WebResourceRequest request) {
Uri url = request.getUrl();
Uri origin = Uri.parse(loadedPage == null ? "" : loadedPage);
if ("http".equals(url.getScheme())
&& "127.0.0.1".equals(url.getHost())
&& url.getPort() == origin.getPort()) return false;
if (request.hasGesture()
&& ("https".equals(url.getScheme()) || "http".equals(url.getScheme()))) {
try {
startActivity(new Intent(Intent.ACTION_VIEW, url));
} catch (ActivityNotFoundException ignored) {
}
}
return true;
}
});
web.setWebChromeClient(
new WebChromeClient() {
@Override
public boolean onShowFileChooser(
WebView w, ValueCallback<Uri[]> callback, FileChooserParams params) {
if (fileSelection != null) fileSelection.onReceiveValue(null);
fileSelection = callback;
Intent choose =
new Intent(Intent.ACTION_OPEN_DOCUMENT)
.setType("*/*")
.addCategory(Intent.CATEGORY_OPENABLE);
choose.putExtra(
Intent.EXTRA_ALLOW_MULTIPLE,
params.getMode() == FileChooserParams.MODE_OPEN_MULTIPLE);
try {
startActivityForResult(choose, 110);
} catch (ActivityNotFoundException e) {
fileSelection.onReceiveValue(null);
fileSelection = null;
return false;
}
return true;
}
});
web.setDownloadListener(
(url, ua, disposition, mime, length) -> {
show(false);
Toast.makeText(
this,
"New screenshot originals are in Screenshot downloads.",
Toast.LENGTH_LONG)
.show();
});
}
content.addView(web, new FrameLayout.LayoutParams(-1, -1));
if (!page.equals(loadedPage)) {
loadedPage = page;
web.loadUrl(page);
}
}
@Override
protected void onActivityResult(int request, int result, Intent data) {
super.onActivityResult(request, result, data);
if (request == 110 && fileSelection != null) {
fileSelection.onReceiveValue(WebChromeClient.FileChooserParams.parseResult(result, data));
fileSelection = null;
}
}
private void update() {
if (!busy) status.setText(SyncService.state());
boolean paired = SyncService.prefs(this).contains("pin");
boolean running = SyncService.running();
connect.setEnabled(paired && !busy && !running);
pair.setEnabled(!busy && !running);
stop.setEnabled(!busy && running);
host.setEnabled(!running && !busy);
port.setEnabled(!running && !busy);
user.setEnabled(!running && !busy);
if (controls && !Objects.equals(loadedPage, SyncService.page())) show(true);
String pin = SyncService.prefs(this).getString("pin", null);
if (!Objects.equals(pin, renderedPin)) {
// Never leave another headset's originals shareable while the new library loads.
renderedPin = pin;
librarySignature = "";
downloads.removeAllViews();
empty.setText(EMPTY);
empty.setVisibility(View.VISIBLE);
}
if (loading) return;
loading = true;
library.execute(
() -> {
List<SyncEngine.Item> items = null;
Set<SyncEngine.Item> present = new HashSet<>();
Exception failure = null;
try {
items = SyncService.items(this, pin);
for (SyncEngine.Item item : items) if (item.file.exists()) present.add(item);
} catch (Exception e) {
failure = e;
}
List<SyncEngine.Item> loaded = items;
Exception error = failure;
runOnUiThread(
() -> {
loading = false;
if (isFinishing() || isDestroyed()) return;
// A result for a previously paired headset is discarded; the next tick reloads.
if (!Objects.equals(pin, SyncService.prefs(this).getString("pin", null))) return;
if (error != null) {
librarySignature = "";
empty.setVisibility(View.VISIBLE);
empty.setText("Could not read screenshot downloads: " + SyncService.message(error));
return;
}
render(loaded, present);
});
});
}
private void render(List<SyncEngine.Item> items, Set<SyncEngine.Item> present) {
StringBuilder signature = new StringBuilder();
for (var item : items)
signature
.append(item.file.getAbsolutePath())
.append('|')
.append(item.destination)
.append('|')
.append(present.contains(item))
.append('\n');
if (signature.toString().equals(librarySignature)) return;
librarySignature = signature.toString();
downloads.removeAllViews();
empty.setText(EMPTY);
empty.setVisibility(items.isEmpty() ? View.VISIBLE : View.GONE);
int shown = 0;
for (SyncEngine.Item item : items) {
if (shown++ >= downloadLimit) {
downloads.addView(
button(
"Show 100 more downloads",
() -> {
downloadLimit += 100;
librarySignature = "";
update();
}));
break;
}
String label = item.shot.id.substring(item.shot.id.lastIndexOf('/') + 1);
LinearLayout card = new LinearLayout(this);
card.setOrientation(LinearLayout.VERTICAL);
card.setPadding(0, dp(12), 0, dp(12));
card.addView(text(label, 16, INK));
String where =
item.destination.equals("saved")
? "Saved to Gallery"
: item.destination.equals("appOnly") ? "In Frame Control" : "Gallery save pending";
boolean exists = present.contains(item);
if (!exists) where = "Download pending";
card.addView(text(where + " · Game " + item.shot.id.split("/")[1], 13, MUTED));
if (exists) card.addView(button("Open / share original", () -> share(item)));
downloads.addView(card);
}
}
private void share(SyncEngine.Item item) {
Uri uri = FileProvider.getUriForFile(this, getPackageName() + ".files", item.file);
Intent send =
new Intent(Intent.ACTION_SEND)
.setType(item.shot.id.endsWith(".png") ? "image/png" : "image/jpeg")
.putExtra(Intent.EXTRA_STREAM, uri)
.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION);
startActivity(Intent.createChooser(send, "Screenshot original"));
}
private void startSync() {
if (Build.VERSION.SDK_INT >= 33
&& checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS)
!= PackageManager.PERMISSION_GRANTED) {
requestPermissions(new String[] {Manifest.permission.POST_NOTIFICATIONS}, 108);
return;
}
startForegroundService(new Intent(this, SyncService.class));
}
@Override
public void onRequestPermissionsResult(int request, String[] permissions, int[] grants) {
super.onRequestPermissionsResult(request, permissions, grants);
if (request == 108) {
if (grants.length > 0 && grants[0] == PackageManager.PERMISSION_GRANTED)
startForegroundService(new Intent(this, SyncService.class));
else
error(
"Allow notifications in Android Settings to start a visible screenshot sync session.");
}
}
private void probe() {
final String address = host.getText().toString().trim(),
username = user.getText().toString().trim();
final int sshPort;
try {
sshPort = Integer.parseInt(port.getText().toString());
} catch (NumberFormatException e) {
error("Enter an SSH port from 1 to 65535.");
return;
}
stopService(new Intent(this, SyncService.class));
busy = true;
pair.setEnabled(false);
status.setText("Checking headset identity…");
network.execute(
() -> {
try {
String pin = FrameConnection.probe(address, sshPort, username);
String fingerprint = FrameConnection.fingerprint(pin);
runOnUiThread(
() -> {
if (isFinishing() || isDestroyed()) return;
busy = false;
new AlertDialog.Builder(this)
.setTitle("Trust this headset?")
.setMessage(
address
+ ":"
+ sshPort
+ "\n\n"
+ fingerprint
+ "\n\n"
+ "Compare this SSH fingerprint with your headset before entering its"
+ " Developer Mode password.")
.setNegativeButton("Cancel", null)
.setPositiveButton(
"Trust and pair", (d, w) -> password(address, sshPort, username, pin))
.show();
});
} catch (Exception e) {
runOnUiThread(
() -> {
busy = false;
error(SyncService.message(e));
});
}
});
}
private void password(String address, int sshPort, String username, String pin) {
EditText password = new EditText(this);
password.setInputType(InputType.TYPE_CLASS_TEXT | InputType.TYPE_TEXT_VARIATION_PASSWORD);
password.setHint("Developer Mode password");
password.setSaveEnabled(false);
new AlertDialog.Builder(this)
.setTitle("Pair with " + address)
.setMessage(
"Used once to authorize this phone’s protected SSH key. The password is not saved.")
.setView(password)
.setNegativeButton("Cancel", (d, w) -> password.setText(""))
.setPositiveButton(
"Pair",
(d, w) -> {
String secret = password.getText().toString();
password.setText("");
busy = true;
status.setText("Pairing this phone…");
network.execute(
() -> {
try {
FrameConnection.pair(
address, sshPort, username, pin, secret, SyncService.install(this));
if (!SyncService.prefs(this)
.edit()
.putString("host", address)
.putInt("port", sshPort)
.putString("user", username)
.putString("pin", pin)
.commit()) throw new java.io.IOException("Could not save pairing.");
runOnUiThread(
() -> {
busy = false;
status.setText("Paired. Start syncing when you’re ready.");
update();
});
} catch (Exception e) {
runOnUiThread(
() -> {
busy = false;
error(SyncService.message(e));
});
}
});
})
.show();
}
private void error(String message) {
if (isFinishing() || isDestroyed()) return;
new AlertDialog.Builder(this)
.setTitle("Frame Control")
.setMessage(message)
.setPositiveButton("OK", null)
.show();
}
@Override
protected void onResume() {
super.onResume();
visible = true;
if (web != null) web.onResume();
handler.post(refresh);
}
@Override
protected void onPause() {
visible = false;
handler.removeCallbacks(refresh);
if (web != null) web.onPause();
super.onPause();
}
@Override
protected void onDestroy() {
network.shutdownNow();
library.shutdownNow();
if (fileSelection != null) fileSelection.onReceiveValue(null);
if (web != null) {
web.destroy();
web = null;
}
super.onDestroy();
}
}
@@ -0,0 +1,115 @@
package com.saphid.framecontrol;
import android.security.keystore.KeyGenParameterSpec;
import android.security.keystore.KeyProperties;
import com.jcraft.jsch.Identity;
import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.math.BigInteger;
import java.nio.charset.StandardCharsets;
import java.security.KeyPairGenerator;
import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.Signature;
import java.security.interfaces.RSAPublicKey;
import java.util.Base64;
/** The SSH private key is a non-exportable Android Keystore signing key. */
final class PhoneIdentity implements Identity {
private static final String ALIAS = "frame-control-ssh-v1";
private final PrivateKey key;
private final byte[] publicBlob;
PhoneIdentity() throws Exception {
KeyStore store = KeyStore.getInstance("AndroidKeyStore");
store.load(null);
if (!store.containsAlias(ALIAS)) {
KeyPairGenerator generator = KeyPairGenerator.getInstance("RSA", "AndroidKeyStore");
generator.initialize(
new KeyGenParameterSpec.Builder(ALIAS, KeyProperties.PURPOSE_SIGN)
.setKeySize(3072)
.setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA512)
.setSignaturePaddings(KeyProperties.SIGNATURE_PADDING_RSA_PKCS1)
.build());
generator.generateKeyPair();
}
key = (PrivateKey) store.getKey(ALIAS, null);
RSAPublicKey publicKey = (RSAPublicKey) store.getCertificate(ALIAS).getPublicKey();
publicBlob =
blob(
"ssh-rsa".getBytes(StandardCharsets.US_ASCII),
integer(publicKey.getPublicExponent()),
integer(publicKey.getModulus()));
}
private static byte[] integer(BigInteger value) {
return value.toByteArray();
}
private static byte[] blob(byte[]... strings) throws Exception {
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(bytes);
for (byte[] s : strings) {
out.writeInt(s.length);
out.write(s);
}
return bytes.toByteArray();
}
String authorizedKey(String install) {
return "ssh-rsa "
+ Base64.getEncoder().encodeToString(publicBlob)
+ " frame-control-android-"
+ install;
}
@Override
public byte[] getPublicKeyBlob() {
return publicBlob.clone();
}
@Override
public byte[] getSignature(byte[] data) {
return getSignature(data, "rsa-sha2-512");
}
@Override
public byte[] getSignature(byte[] data, String algorithm) {
try {
if (!algorithm.equals("rsa-sha2-256") && !algorithm.equals("rsa-sha2-512")) return null;
Signature signer =
Signature.getInstance(
algorithm.equals("rsa-sha2-512") ? "SHA512withRSA" : "SHA256withRSA");
signer.initSign(key);
signer.update(data);
return blob(algorithm.getBytes(StandardCharsets.US_ASCII), signer.sign());
} catch (Exception e) {
return null;
}
}
@Override
public boolean setPassphrase(byte[] ignored) {
return true;
}
@Override
public String getAlgName() {
return "ssh-rsa";
}
@Override
public String getName() {
return ALIAS;
}
@Override
public boolean isEncrypted() {
return false;
}
@Override
public void clear() {
/* Keystore owns the key; disconnect never deletes it. */
}
}
@@ -0,0 +1,71 @@
package com.saphid.framecontrol;
import java.util.function.Consumer;
/**
* Owns the visible sync status. Each service instance gets one session token; once a session
* ends, its late worker can no longer change the status, page or notification, even after a
* newer session has started.
*/
final class SessionGate {
static final String DISCONNECTED = "Disconnected";
private long issued;
private long current;
private String state = DISCONNECTED;
private String page;
synchronized long begin() {
current = ++issued;
page = null;
return current;
}
synchronized boolean active(long session) {
return session != 0 && session == current;
}
synchronized boolean running() {
return current != 0;
}
synchronized String state() {
return state;
}
synchronized String page() {
return page;
}
/** Publishes a status and runs {@code show} (the notification update) only while current. */
synchronized boolean status(long session, String text, Consumer<String> show) {
if (!active(session)) return false;
state = text;
if (show != null) show.accept(text);
return true;
}
synchronized boolean page(long session, String value) {
if (!active(session)) return false;
page = value;
return true;
}
/** Runs {@code action} only while the session is current, serialized against {@link #end}. */
synchronized boolean ifActive(long session, Runnable action) {
if (!active(session)) return false;
action.run();
return true;
}
/** Ends only the named session. Progress text becomes Disconnected; a final error stays. */
synchronized boolean end(long session) {
if (!active(session)) return false;
current = 0;
page = null;
if (state.startsWith("Syncing")
|| state.startsWith("Waiting")
|| state.startsWith("Connecting")
|| state.startsWith("Screenshot pending")) state = DISCONNECTED;
return true;
}
}
@@ -0,0 +1,724 @@
package com.saphid.framecontrol;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.Strictness;
import com.google.gson.TypeAdapter;
import com.google.gson.stream.JsonReader;
import com.google.gson.stream.JsonToken;
import com.google.gson.stream.JsonWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.NoSuchFileException;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import java.util.function.BooleanSupplier;
import java.util.regex.Pattern;
/** One reusable engine per root. The caller supplies an account/headset-specific root. */
public final class SyncEngine {
private static final long MAX_SIZE = 30L * 1024 * 1024;
private static final Pattern ID = Pattern.compile(
"^[0-9]{1,12}/[0-9]{1,20}/[0-9]{14}_[0-9]{1,4}\\.(jpg|png)$");
private static final Pattern ACCOUNT = Pattern.compile("^[0-9]{1,12}$");
private static final Gson GSON = new GsonBuilder().registerTypeAdapter(Shot.class,
new TypeAdapter<Shot>() {
@Override public void write(JsonWriter writer, Shot shot) throws IOException {
writer.beginObject();
writer.name("id").value(shot.id);
writer.name("size").value(shot.size);
writer.name("time").value(shot.time);
writer.endObject();
}
@Override public Shot read(JsonReader reader) throws IOException {
String id = null;
Long size = null;
Double time = null;
Set<String> fields = new HashSet<>();
reader.beginObject();
while (reader.hasNext()) {
String field = reader.nextName();
if (!fields.add(field)) throw new IOException("Duplicate screenshot field");
switch (field) {
case "id":
if (reader.peek() != JsonToken.STRING) throw new IOException("Invalid id");
id = reader.nextString();
break;
case "size":
if (reader.peek() != JsonToken.NUMBER) throw new IOException("Invalid size");
size = reader.nextLong();
break;
case "time":
if (reader.peek() != JsonToken.NUMBER) throw new IOException("Invalid time");
time = reader.nextDouble();
break;
default: throw new IOException("Unknown screenshot field");
}
}
reader.endObject();
if (id == null || size == null || time == null) throw new IOException("Incomplete shot");
return new Shot(id, size, time);
}
}).create();
public static final class Shot {
public final String id;
public final long size;
public final double time;
public Shot(String id, long size, double time) {
this.id = id;
this.size = size;
this.time = time;
}
}
/**
* One headset poll. {@code accounts} names every Steam account directory present on the
* headset, including accounts with no screenshots yet. An absent userdata root is an empty set.
*/
public static final class Listing {
public final Set<String> accounts;
public final List<Shot> shots;
public Listing(Set<String> accounts, List<Shot> shots) {
this.accounts = accounts;
this.shots = shots;
}
}
public interface Transport {
void download(Shot shot, File destination) throws Exception;
boolean validImage(File f) throws Exception;
}
public interface Exporter {
boolean export(Shot shot, File original, BooleanSupplier eligible) throws Exception;
}
public static final class Item {
public final Shot shot;
/** Future original path; may not exist while the shot is observing or queued. */
public final File file;
public final String destination;
private Item(Shot shot, File file, String destination) {
this.shot = shot;
this.file = file;
this.destination = destination;
}
}
// Boxed fields make missing journal fields distinguishable from legitimate zero values.
private static final class Ledger {
Integer version;
Boolean initialized;
Boolean gallery;
/** Accounts already baselined. Absent only in version 1 journals. */
Set<String> accounts;
Map<String, Receipt> records;
}
private static final class Poll {
final Map<String, Shot> eligible = new LinkedHashMap<>();
final Set<String> accounts = new TreeSet<>();
int skipped;
}
private static final class Receipt {
Shot shot;
String state;
Integer observations;
String destination;
Receipt(Shot shot, String state, int observations, String destination) {
this.shot = shot;
this.state = state;
this.observations = observations;
this.destination = destination;
}
}
/** Only external callbacks and rejected transfer contents are safe to retry per item. */
private static final class ItemFailure extends IOException {
private static final long serialVersionUID = 1L;
ItemFailure(String message, Exception cause) {
super(message, cause);
}
}
private final Object lock = new Object();
private final Path root;
private final Path originals;
private final Path journal;
private Ledger ledger;
private long generation;
private long active;
private boolean closed;
private volatile int skipped;
public SyncEngine(File root, boolean gallery) throws Exception {
this(root, Boolean.valueOf(gallery));
}
/** A null gallery opens a read-only view that never creates, upgrades or rewrites anything. */
private SyncEngine(File root, Boolean gallery) throws Exception {
if (root == null || Files.isSymbolicLink(root.toPath())) {
throw new IOException("Unsafe sync root");
}
this.root = root.getCanonicalFile().toPath();
if (gallery == null) {
closed = true;
} else {
List<Path> missingDirectories = new ArrayList<>();
for (Path path = this.root; path != null && !Files.exists(path); path = path.getParent()) {
missingDirectories.add(path);
}
Files.createDirectories(this.root);
for (Path directory : missingDirectories) syncDirectory(directory.getParent());
}
originals = this.root.resolve("originals");
journal = this.root.resolve("receipts.json");
checkPath(originals);
checkPath(journal);
if (Files.exists(originals, LinkOption.NOFOLLOW_LINKS)
&& !Files.isDirectory(originals, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("Invalid originals directory");
}
if (Files.exists(journal, LinkOption.NOFOLLOW_LINKS)) {
if (!Files.isRegularFile(journal, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("Invalid sync journal");
}
// Never recover a damaged or incomplete journal by establishing another baseline.
try (JsonReader reader = new JsonReader(new InputStreamReader(
Files.newInputStream(journal), StandardCharsets.UTF_8))) {
reader.setStrictness(Strictness.STRICT);
ledger = GSON.fromJson(reader, Ledger.class);
if (reader.peek() != JsonToken.END_DOCUMENT) {
throw new IOException("Trailing sync journal data");
}
} catch (RuntimeException e) {
throw new IOException("Corrupt sync journal", e);
}
validateLedger();
if (gallery != null) setGallery(gallery);
} else {
if (Files.exists(originals, LinkOption.NOFOLLOW_LINKS)) {
try (java.util.stream.Stream<Path> paths = Files.walk(originals)) {
if (paths.anyMatch(p -> !Files.isDirectory(p, LinkOption.NOFOLLOW_LINKS))) {
throw new IOException("Originals exist without a sync journal");
}
}
}
ledger = emptyLedger();
if (gallery == null) {
ledger.gallery = false;
} else {
ledger.gallery = gallery;
persist(ledger);
}
}
}
/** Post-baseline receipts read without locks, directory creation or journal writes. */
public static List<Item> snapshot(File root) throws Exception {
return new SyncEngine(root, (Boolean) null).items();
}
/** Returns false, without writing, once this engine is closed. */
public boolean setGallery(boolean enabled) throws Exception {
synchronized (lock) {
if (closed) return false;
if (ledger.gallery == enabled) return true;
Ledger next = copy();
next.gallery = enabled;
if (!enabled) {
for (Receipt receipt : next.records.values()) {
if ("pending".equals(receipt.destination)) receipt.destination = "appOnly";
}
}
commit(next);
return true;
}
}
/**
* A second live reconcile is rejected. stop() cancels its generation and permits a new run;
* close() cancels it permanently. The journal is rewritten only when a receipt changes.
*/
public void reconcile(Listing listing, Transport transport, Exporter exporter) throws Exception {
final long run;
synchronized (lock) {
if (closed) throw new IllegalStateException("Sync stopped");
if (active != 0) throw new IllegalStateException("Sync already in progress");
run = ++generation;
active = run;
}
try {
Poll poll = poll(listing);
synchronized (lock) {
if (active != run) return;
skipped = poll.skipped;
Ledger next = copy();
if (!next.initialized) {
for (Shot shot : poll.eligible.values()) {
next.records.put(shot.id, new Receipt(shot, "baseline", 0, null));
}
next.initialized = true;
next.version = 2;
next.accounts = poll.accounts;
commit(next);
return;
}
boolean changed = false;
Set<String> known = next.accounts;
if (known == null) {
// Version 1 journals did not record accounts. Trust only accounts that already
// have receipts; any other account present now is history and is baselined.
known = new TreeSet<>();
for (String id : next.records.keySet()) known.add(account(id));
// The account comparison below always records the set with this version.
next.version = 2;
changed = true;
}
// Missing observations break consecutiveness, including across restarts.
for (Receipt receipt : next.records.values()) {
if (!poll.eligible.containsKey(receipt.shot.id) && receipt.observations != 0
&& ("observing".equals(receipt.state) || "queued".equals(receipt.state))) {
receipt.observations = 0;
changed = true;
}
}
for (Shot shot : poll.eligible.values()) {
Receipt receipt = next.records.get(shot.id);
if (receipt == null && !known.contains(account(shot.id))) {
// A newly appearing account (or a returning one) starts with its own
// baseline, so its existing screenshots are never imported.
next.records.put(shot.id, new Receipt(shot, "baseline", 0, null));
changed = true;
} else if (receipt == null) {
// Latch eligibility at discovery, before waiting for a stable second sample.
next.records.put(shot.id, new Receipt(shot, "observing", 1,
next.gallery ? "pending" : "appOnly"));
changed = true;
} else if ("observing".equals(receipt.state) || "queued".equals(receipt.state)) {
boolean stable = same(receipt.shot, shot);
int observations = stable ? Math.min(2, receipt.observations + 1) : 1;
if (!stable || observations != receipt.observations) changed = true;
receipt.observations = observations;
receipt.shot = shot;
if (receipt.observations == 2 && "observing".equals(receipt.state)) {
receipt.state = "queued";
}
}
}
// Accounts absent now are forgotten; if one returns it is baselined again.
if (!poll.accounts.equals(next.accounts)) {
next.accounts = poll.accounts;
changed = true;
}
// Queue and its disposition reach disk before any download.
if (changed) commit(next);
}
List<String> ids;
synchronized (lock) {
if (active != run) return;
ids = new ArrayList<>(ledger.records.keySet());
}
ItemFailure failure = null;
for (String id : ids) {
Shot download = null;
synchronized (lock) {
if (active != run) return;
Receipt receipt = ledger.records.get(id);
if ("queued".equals(receipt.state) && receipt.observations == 2) {
download = receipt.shot;
}
}
try {
if (download != null) download(run, download, transport);
export(run, id, exporter);
} catch (ItemFailure e) {
if (failure == null) failure = e;
else failure.addSuppressed(e);
}
}
// Surface retryable errors to the service after independent items have progressed.
// Untagged local I/O/journal failures escape immediately and halt the run.
if (failure != null) throw failure;
} catch (Exception e) {
synchronized (lock) {
if (active != run) return; // A late failed callback belongs to a cancelled run.
}
throw e;
} finally {
synchronized (lock) {
if (active == run) active = 0;
}
}
}
/** Snapshot of all post-baseline receipts, including downloads still pending. */
public List<Item> items() {
synchronized (lock) {
List<Item> result = new ArrayList<>();
for (Receipt receipt : ledger.records.values()) {
if (!"baseline".equals(receipt.state)) {
result.add(new Item(receipt.shot, originals.resolve(receipt.shot.id).toFile(),
receipt.destination));
}
}
result.sort(Comparator.comparingDouble((Item item) -> item.shot.time).reversed()
.thenComparing(item -> item.shot.id));
return Collections.unmodifiableList(result);
}
}
/** Listed files left out of the last poll because they were empty or over 30 MiB. */
public int skipped() {
return skipped;
}
public void stop() {
synchronized (lock) {
++generation;
active = 0;
}
}
/**
* Permanently cancels this engine. Once it returns, no in-flight or later call can write the
* journal, so another engine may safely take over the same root.
*/
public void close() {
synchronized (lock) {
closed = true;
++generation;
active = 0;
}
}
public boolean isClosed() {
synchronized (lock) {
return closed;
}
}
private void download(long run, Shot shot, Transport transport) throws Exception {
Path temporary;
synchronized (lock) {
if (active != run) return;
temporary = Files.createTempFile(root, ".download-", ".part");
}
try {
try {
transport.download(shot, temporary.toFile());
} catch (InterruptedException e) {
throw e;
} catch (Exception e) {
throw new ItemFailure("Download failed: " + shot.id + ": " + e.getMessage(), e);
}
synchronized (lock) {
if (active != run) return;
}
checkTransfer(temporary, shot);
boolean valid;
try {
valid = transport.validImage(temporary.toFile());
} catch (InterruptedException e) {
throw e;
} catch (Exception e) {
throw new ItemFailure("Image validation failed: " + shot.id + ": " + e.getMessage(), e);
}
if (!valid) throw new ItemFailure("Invalid image: " + shot.id, null);
checkTransfer(temporary, shot);
try (FileChannel channel = FileChannel.open(temporary, StandardOpenOption.WRITE)) {
channel.force(true);
}
synchronized (lock) {
if (active != run) return;
Path target = originals.resolve(shot.id);
createParents(target.getParent());
checkPath(target);
Files.move(temporary, target, StandardCopyOption.ATOMIC_MOVE,
StandardCopyOption.REPLACE_EXISTING);
syncDirectory(target.getParent());
Ledger next = copy();
// Read the current destination: setGallery may have changed it during the transfer.
next.records.get(shot.id).state = "downloaded";
commit(next);
}
} finally {
Files.deleteIfExists(temporary);
}
}
private void export(long run, String id, Exporter exporter) throws Exception {
Shot shot;
File original;
synchronized (lock) {
if (!eligible(run, id)) return;
Receipt receipt = ledger.records.get(id);
shot = receipt.shot;
Path path = originals.resolve(id);
checkDownload(path, shot);
original = path.toFile();
}
boolean saved;
try {
saved = exporter.export(shot, original, () -> {
synchronized (lock) {
return eligible(run, id);
}
});
} catch (InterruptedException e) {
throw e;
} catch (Exception e) {
throw new ItemFailure("Gallery export failed: " + shot.id + ": " + e.getMessage(), e);
}
synchronized (lock) {
if (active != run || !saved) return;
Ledger next = copy();
// A true result means the exporter actually saved (or recovered) the MediaStore item.
next.records.get(id).destination = "saved";
commit(next);
}
}
private boolean eligible(long run, String id) {
Receipt receipt = ledger.records.get(id);
return active == run && ledger.gallery && receipt != null
&& "downloaded".equals(receipt.state) && "pending".equals(receipt.destination);
}
private static Poll poll(Listing listing) throws IOException {
if (listing == null || listing.shots == null || listing.accounts == null) {
throw new IOException("Missing screenshot listing");
}
Poll result = new Poll();
for (String account : listing.accounts) {
if (account == null || !ACCOUNT.matcher(account).matches()) {
throw new IOException("Invalid Steam account");
}
result.accounts.add(account);
}
Map<String, Shot> seen = new LinkedHashMap<>();
for (Shot shot : listing.shots) {
// Unsafe ids, non-finite times and conflicting duplicates reject the whole listing.
validateIdentity(shot);
Shot previous = seen.put(shot.id, shot);
if (previous != null && !same(previous, shot)) {
throw new IOException("Conflicting screenshot identity: " + shot.id);
}
// The account directory may appear between the account and screenshot queries.
result.accounts.add(account(shot.id));
}
for (Shot shot : seen.values()) {
// An empty, still-writing or oversized file is only this item's problem: treat it
// as absent from this poll so it never blocks other screenshots or the baseline.
if (eligibleSize(shot)) result.eligible.put(shot.id, shot);
else result.skipped++;
}
return result;
}
private static String account(String id) {
return id.substring(0, id.indexOf('/'));
}
private static void validateIdentity(Shot shot) throws IOException {
if (shot == null || shot.id == null || !ID.matcher(shot.id).matches()
|| !Double.isFinite(shot.time)) {
throw new IOException("Invalid screenshot metadata");
}
}
private static boolean eligibleSize(Shot shot) {
return shot.size > 0 && shot.size <= MAX_SIZE;
}
private static void validateShot(Shot shot) throws IOException {
validateIdentity(shot);
if (!eligibleSize(shot)) throw new IOException("Invalid screenshot metadata");
}
private static boolean same(Shot a, Shot b) {
return a.size == b.size && Double.compare(a.time, b.time) == 0;
}
private void validateLedger() throws IOException {
if (ledger == null || ledger.version == null || ledger.initialized == null
|| ledger.gallery == null || ledger.records == null
|| (ledger.version == 1 && ledger.accounts != null)
|| (ledger.version == 2 && ledger.accounts == null)
|| (ledger.version != 1 && ledger.version != 2)
|| (!ledger.initialized && !ledger.records.isEmpty())
|| (!ledger.initialized && ledger.accounts != null && !ledger.accounts.isEmpty())) {
throw new IOException("Invalid sync journal header");
}
if (ledger.accounts != null) {
for (String account : ledger.accounts) {
if (account == null || !ACCOUNT.matcher(account).matches()) {
throw new IOException("Invalid sync journal header");
}
}
ledger.accounts = new TreeSet<>(ledger.accounts);
}
for (Map.Entry<String, Receipt> entry : ledger.records.entrySet()) {
Receipt receipt = entry.getValue();
if (receipt == null) throw new IOException("Invalid sync receipt");
validateShot(receipt.shot);
if (!entry.getKey().equals(receipt.shot.id) || receipt.observations == null
|| receipt.observations < 0 || receipt.observations > 2) {
throw new IOException("Invalid sync receipt identity");
}
boolean baseline = "baseline".equals(receipt.state);
boolean observing = "observing".equals(receipt.state);
boolean queued = "queued".equals(receipt.state);
boolean downloaded = "downloaded".equals(receipt.state);
if ((!baseline && !observing && !queued && !downloaded)
|| (baseline && receipt.destination != null)
|| (baseline && receipt.observations != 0)
|| (observing && receipt.observations > 1)
|| (downloaded && receipt.observations != 2)
|| ((observing || queued || downloaded) && !"pending".equals(receipt.destination)
&& !"appOnly".equals(receipt.destination) && !"saved".equals(receipt.destination))
|| ((observing || queued) && "saved".equals(receipt.destination))
|| (!ledger.gallery && "pending".equals(receipt.destination))) {
throw new IOException("Invalid sync receipt state");
}
Path path = originals.resolve(receipt.shot.id);
checkPath(path);
if (downloaded) checkDownload(path, receipt.shot);
}
}
private void checkDownload(Path path, Shot shot) throws IOException {
checkPath(path);
if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS) || Files.size(path) != shot.size) {
throw new IOException("Incomplete original: " + shot.id);
}
}
private void checkTransfer(Path path, Shot shot) throws IOException {
checkPath(path); // Unsafe paths and local filesystem errors remain fatal.
BasicFileAttributes attributes;
try {
attributes = Files.readAttributes(path, BasicFileAttributes.class, LinkOption.NOFOLLOW_LINKS);
} catch (NoSuchFileException e) {
throw new ItemFailure("Missing downloaded image: " + shot.id, e);
}
if (!attributes.isRegularFile() || attributes.size() != shot.size) {
throw new ItemFailure("Incomplete downloaded image: " + shot.id, null);
}
}
private void checkPath(Path path) throws IOException {
if (!path.normalize().startsWith(root)) throw new IOException("Path escaped sync root");
Path current = root;
if (Files.isSymbolicLink(current)) throw new IOException("Symlink in sync root");
for (Path component : root.relativize(path)) {
current = current.resolve(component);
if (Files.isSymbolicLink(current)) throw new IOException("Symlink in sync path");
}
}
private void createParents(Path directory) throws IOException {
checkPath(directory);
Path current = root;
for (Path component : root.relativize(directory)) {
Path parent = current;
current = current.resolve(component);
if (!Files.exists(current, LinkOption.NOFOLLOW_LINKS)) {
Files.createDirectory(current);
syncDirectory(parent);
} else if (!Files.isDirectory(current, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("Invalid originals directory");
}
}
}
/**
* Keeps the version: a version 1 journal (no account set) stays version 1 through setting
* changes, and only reconcile upgrades it, recording the version and accounts together.
*/
private Ledger copy() {
Ledger next = emptyLedger();
next.version = ledger.version;
next.initialized = ledger.initialized;
next.gallery = ledger.gallery;
next.accounts = ledger.accounts == null ? null : new TreeSet<>(ledger.accounts);
for (Map.Entry<String, Receipt> entry : ledger.records.entrySet()) {
Receipt receipt = entry.getValue();
next.records.put(entry.getKey(), new Receipt(receipt.shot, receipt.state,
receipt.observations, receipt.destination));
}
return next;
}
private static Ledger emptyLedger() {
Ledger result = new Ledger();
result.version = 2;
result.initialized = false;
result.accounts = new TreeSet<>();
result.records = new LinkedHashMap<>();
return result;
}
private void commit(Ledger next) throws IOException {
if (closed) throw new IOException("Sync journal is read-only");
persist(next);
ledger = next;
}
private void persist(Ledger next) throws IOException {
checkPath(journal);
Path temporary = Files.createTempFile(root, ".receipts-", ".tmp");
try {
try (FileOutputStream output = new FileOutputStream(temporary.toFile());
Writer writer = new OutputStreamWriter(output, StandardCharsets.UTF_8)) {
GSON.toJson(next, writer);
writer.flush();
output.getFD().sync();
}
Files.move(temporary, journal, StandardCopyOption.ATOMIC_MOVE,
StandardCopyOption.REPLACE_EXISTING);
syncDirectory(root);
} finally {
Files.deleteIfExists(temporary);
}
}
private static void syncDirectory(Path directory) {
// Some Android filesystems reject directory channels. File contents were already fsynced;
// the rename is atomic either way. Do not report rollback after a committed rename.
try (FileChannel channel = FileChannel.open(directory, StandardOpenOption.READ)) {
channel.force(true);
} catch (IOException | UnsupportedOperationException ignored) {
// Best available portable Java directory durability.
}
}
}
@@ -0,0 +1,266 @@
package com.saphid.framecontrol;
import android.app.*;
import android.content.*;
import android.content.pm.ServiceInfo;
import android.os.*;
import java.io.File;
import java.util.*;
import java.util.concurrent.*;
/** Explicit user-started connected-device session, with a visible Stop action. */
public final class SyncService extends Service {
static final String CHANNEL = "headset-sync";
static final Object SETTINGS_LOCK = new Object();
private static final int NOTIFICATION = 108;
private static final SessionGate GATE = new SessionGate();
private static volatile SyncService instance;
private final ExecutorService worker = Executors.newSingleThreadExecutor();
private long session;
private volatile boolean stopping;
private volatile FrameConnection connection;
private volatile SyncEngine engine;
private volatile String pin;
static SharedPreferences prefs(Context c) {
return c.getSharedPreferences("frame", MODE_PRIVATE);
}
static String state() {
return GATE.state();
}
static String page() {
return GATE.page();
}
static boolean running() {
SyncService service = instance;
return service != null && !service.stopping && GATE.active(service.session);
}
static String install(Context c) {
SharedPreferences p = prefs(c);
String id = p.getString("install", null);
if (id == null) {
id = UUID.randomUUID().toString();
if (!p.edit().putString("install", id).commit())
throw new IllegalStateException("Could not save this phone's identity.");
}
return id;
}
static File library(Context c, String pin) throws Exception {
return new File(c.getFilesDir(), "screenshots/" + FrameConnection.hash(pin));
}
static void gallery(Context c, boolean value) throws Exception {
synchronized (SETTINGS_LOCK) {
SyncService service = instance;
// Journal first: a failed disk write must not appear as a saved setting.
String pin = prefs(c).getString("pin", null);
SyncEngine live = service == null ? null : service.engine;
// A closed engine belongs to a stopped session; update its library from disk instead.
if (live != null && !live.setGallery(value)) live = null;
if (!value) {
// The setting is global. Latch the opt-out in every retained library,
// including headsets that are not currently paired or connected.
File libraries = new File(c.getFilesDir(), "screenshots");
if (libraries.exists()) {
File[] roots = libraries.listFiles();
if (roots == null) throw new java.io.IOException("Could not read screenshot libraries.");
File activeRoot = live != null && pin != null ? library(c, pin) : null;
for (File root : roots) {
if (!root.getName().matches("[0-9a-f]{64}")) continue;
if (activeRoot != null && root.equals(activeRoot)) continue;
new SyncEngine(root, false).setGallery(false);
}
}
} else if (live == null && pin != null) {
new SyncEngine(library(c, pin), true).setGallery(true);
}
if (!prefs(c).edit().putBoolean("gallery", value).commit())
throw new java.io.IOException("Could not save the Gallery setting.");
// Interrupted exports can leave hidden pending rows; app-only receipts never retry them.
if (!value) GalleryExporter.discardPending(c);
}
}
/** Read-only, for a background thread. Never creates or rewrites a library. */
static List<SyncEngine.Item> items(Context c, String pin) throws Exception {
if (pin == null) return Collections.emptyList();
SyncService service = instance;
SyncEngine live = service == null ? null : service.engine;
if (live != null && !live.isClosed() && pin.equals(service.pin)) return live.items();
return SyncEngine.snapshot(library(c, pin));
}
@Override
public void onCreate() {
super.onCreate();
session = GATE.begin();
instance = this;
stopping = false;
NotificationManager manager = getSystemService(NotificationManager.class);
manager.createNotificationChannel(
new NotificationChannel(
CHANNEL, "Headset screenshot sync", NotificationManager.IMPORTANCE_LOW));
}
private Notification notification(String message) {
PendingIntent open =
PendingIntent.getActivity(
this,
0,
new Intent(this, MainActivity.class),
PendingIntent.FLAG_IMMUTABLE | PendingIntent.FLAG_UPDATE_CURRENT);
PendingIntent stop =
PendingIntent.getService(
this,
1,
new Intent(this, SyncService.class).setAction("STOP"),
PendingIntent.FLAG_IMMUTABLE | PendingIntent.FLAG_UPDATE_CURRENT);
return new Notification.Builder(this, CHANNEL)
.setSmallIcon(com.saphid.framecontrol.R.drawable.ic_frame)
.setContentTitle("Frame Control")
.setContentText(message)
.setContentIntent(open)
.setOngoing(true)
.addAction(new Notification.Action.Builder(null, "Stop syncing", stop).build())
.build();
}
/** A stale worker cannot reach the notification once its session has ended. */
private void status(String text) {
GATE.status(
session,
text,
t -> getSystemService(NotificationManager.class).notify(NOTIFICATION, notification(t)));
}
@Override
public int onStartCommand(Intent intent, int flags, int startId) {
if (intent != null && "STOP".equals(intent.getAction())) {
stopSelf();
return START_NOT_STICKY;
}
if (connection != null || stopping) return START_NOT_STICKY;
String text = "Connecting to headset";
GATE.status(session, text, null);
if (Build.VERSION.SDK_INT >= 34)
startForeground(
NOTIFICATION, notification(text), ServiceInfo.FOREGROUND_SERVICE_TYPE_CONNECTED_DEVICE);
else startForeground(NOTIFICATION, notification(text));
connection = new FrameConnection(this);
worker.execute(this::runSession);
return START_NOT_STICKY;
}
private void runSession() {
try {
SharedPreferences p = prefs(this);
String pin = p.getString("pin", null);
if (pin == null) throw new java.io.IOException("Pair with your headset first.");
SyncEngine created;
synchronized (SETTINGS_LOCK) {
boolean gallery = p.getBoolean("gallery", true);
created = new SyncEngine(library(this, pin), gallery);
// Retry cleanup of hidden rows if an earlier opt-out's cleanup was interrupted.
if (!gallery) GalleryExporter.discardPending(this);
this.pin = pin;
engine = created;
}
// Pairs with onDestroy: it sets stopping before reading engine.
if (stopping) created.close();
while (!stopping) {
try {
FrameConnection active = new FrameConnection(this);
connection = active;
if (stopping) break;
status("Connecting to headset");
active.connect(
p.getString("host", ""),
p.getInt("port", 22),
p.getString("user", "steamos"),
pin,
install(this));
if (stopping) {
active.close();
break;
}
GATE.page(session, active.page());
status("Syncing new screenshots");
GalleryExporter exporter = new GalleryExporter(this, pin, SETTINGS_LOCK);
while (!stopping && active.connected()) {
try {
SyncEngine.Listing listing = active.listing();
created.reconcile(listing, active, exporter);
status(progress(listing, created.skipped()));
} catch (Exception e) {
status("Screenshot pending · " + message(e));
}
Thread.sleep(4000);
}
if (!stopping) throw new java.io.IOException("Connection lost. Reconnecting…");
} catch (Exception e) {
GATE.page(session, null);
FrameConnection c = connection;
if (c != null) c.close();
if (stopping) break;
status("Waiting for headset · " + message(e));
Thread.sleep(10000);
}
}
} catch (InterruptedException ignored) {
Thread.currentThread().interrupt();
} catch (Exception e) {
GATE.status(session, message(e), null);
} finally {
GATE.page(session, null);
FrameConnection c = connection;
if (c != null) c.close();
// Only a still-current session may stop the service; Stop already ended a stale one.
GATE.ifActive(session, this::stopSelf);
}
}
static String progress(SyncEngine.Listing listing, int skipped) {
if (listing.accounts.isEmpty()) return "Waiting for a Steam account on the headset";
String text = "Syncing new screenshots";
if (skipped == 1) text += " · 1 file skipped (empty or over 30 MB)";
else if (skipped > 1) text += " · " + skipped + " files skipped (empty or over 30 MB)";
return text;
}
static String message(Exception e) {
String text = e.getMessage();
if (text == null || text.trim().isEmpty()) return "Try reconnecting.";
if (text.contains("HostKey")
|| text.contains("HostKey has been changed")
|| text.contains("reject HostKey")) return "Headset identity changed. Stop and pair again.";
if (text.contains("Auth fail")) return "Headset login refused. Pair again.";
return text.length() > 180 ? text.substring(0, 180) : text;
}
@Override
public void onDestroy() {
stopping = true;
// After this, the old worker cannot post status, page or notification updates.
GATE.end(session);
SyncEngine live = engine;
if (live != null) live.close();
FrameConnection c = connection;
if (c != null) c.close();
worker.shutdownNow();
if (instance == this) instance = null;
stopForeground(STOP_FOREGROUND_REMOVE);
// Also clear a notification posted just before the gate closed.
getSystemService(NotificationManager.class).cancel(NOTIFICATION);
super.onDestroy();
}
@Override
public IBinder onBind(Intent intent) {
return null;
}
}
@@ -0,0 +1,3 @@
<vector xmlns:android="http://schemas.android.com/apk/res/android" android:width="48dp" android:height="48dp" android:viewportWidth="48" android:viewportHeight="48">
<path android:fillColor="#7CD8C3" android:pathData="M8,12L40,12L44,30L34,36L26,28L22,28L14,36L4,30ZM11,17L9,27L14,29L20,23L28,23L34,29L39,27L37,17Z" />
</vector>
@@ -0,0 +1,9 @@
<resources>
<style name="AppTheme" parent="android:style/Theme.Material.NoActionBar">
<item name="android:fontFamily">sans</item>
<item name="android:colorAccent">#7CD8C3</item>
<item name="android:windowLightStatusBar">false</item>
<item name="android:windowBackground">#111819</item>
<item name="android:navigationBarColor">#111819</item>
</style>
</resources>
@@ -0,0 +1,4 @@
<data-extraction-rules>
<cloud-backup><exclude domain="root" path="." /><exclude domain="file" path="." /><exclude domain="database" path="." /><exclude domain="sharedpref" path="." /><exclude domain="external" path="." /></cloud-backup>
<device-transfer><exclude domain="root" path="." /><exclude domain="file" path="." /><exclude domain="database" path="." /><exclude domain="sharedpref" path="." /><exclude domain="external" path="." /></device-transfer>
</data-extraction-rules>
@@ -0,0 +1 @@
<paths><files-path name="screenshots" path="screenshots/" /></paths>
@@ -0,0 +1,4 @@
<network-security-config>
<base-config cleartextTrafficPermitted="false" />
<domain-config cleartextTrafficPermitted="true"><domain includeSubdomains="false">127.0.0.1</domain><domain includeSubdomains="false">localhost</domain></domain-config>
</network-security-config>
@@ -0,0 +1,106 @@
package com.saphid.framecontrol;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.attribute.PosixFilePermissions;
import java.util.Arrays;
import java.util.Collections;
import java.util.TreeSet;
import java.util.concurrent.TimeUnit;
import static org.junit.Assert.*;
/** Runs the generated headset shell against a temporary HOME. */
public class FrameConnectionTest {
@Rule public TemporaryFolder temporary = new TemporaryFolder();
private static final String KEY = "rsa-sha2-256 AAAAB3NzaC1yc2EAAAADAQAB frame-control 'phone'";
private static final String MAC = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMac mac";
private static final class Result {
final int exit;
final String output;
Result(int exit, String output) { this.exit = exit; this.output = output; }
}
private static Result sh(File home, String command) throws Exception {
ProcessBuilder builder = new ProcessBuilder("/bin/sh", "-c", command);
builder.environment().put("HOME", home.getPath());
builder.redirectError(ProcessBuilder.Redirect.to(new File("/dev/null")));
Process process = builder.start();
java.io.ByteArrayOutputStream bytes = new java.io.ByteArrayOutputStream();
try (java.io.InputStream in = process.getInputStream()) {
byte[] buffer = new byte[4096];
for (int n; (n = in.read(buffer)) != -1; ) bytes.write(buffer, 0, n);
}
String output = new String(bytes.toByteArray(), StandardCharsets.UTF_8);
assertTrue("Shell timed out", process.waitFor(10, TimeUnit.SECONDS));
return new Result(process.exitValue(), output);
}
private String authorize(String existing) throws Exception {
File home = temporary.newFolder();
File keys = new File(home, ".ssh/authorized_keys");
if (existing != null) {
assertTrue(keys.getParentFile().mkdirs());
Files.write(keys.toPath(), existing.getBytes(StandardCharsets.UTF_8));
}
for (int i = 0; i < 2; i++) assertEquals(0, sh(home, FrameConnection.authorizeCommand(KEY)).exit);
assertEquals("rw-------", PosixFilePermissions.toString(Files.getPosixFilePermissions(keys.toPath())));
return new String(Files.readAllBytes(keys.toPath()), StandardCharsets.UTF_8);
}
@Test public void authorizingTerminatesALastLineWithoutNewlineAndIsIdempotent() throws Exception {
assertEquals(KEY + "\n", authorize(null));
assertEquals(KEY + "\n", authorize(""));
assertEquals(MAC + "\n" + KEY + "\n", authorize(MAC));
assertEquals(MAC + "\n" + KEY + "\n", authorize(MAC + "\n"));
assertEquals(MAC + "\nother\n" + KEY + "\n", authorize(MAC + "\nother"));
// Already present as an unterminated last line: left exactly as it was.
assertEquals(MAC + "\n" + KEY, authorize(MAC + "\n" + KEY));
}
@Test public void accountListingSeparatesAbsentUserdataFromEmptyAccounts() throws Exception {
File home = temporary.newFolder();
Result absent = sh(home, FrameConnection.ACCOUNTS_COMMAND);
assertEquals(0, absent.exit);
assertEquals("absent\n", absent.output);
assertTrue(FrameConnection.parseAccounts(absent.output).isEmpty());
File userdata = new File(home, ".local/share/Steam/userdata");
assertTrue(userdata.mkdirs());
assertEquals("present\n", sh(home, FrameConnection.ACCOUNTS_COMMAND).output);
for (String directory : Arrays.asList("123", "4567890", "anonymous", "0123456789012", "12a"))
assertTrue(new File(userdata, directory).mkdir());
Files.write(new File(userdata, "999").toPath(), new byte[]{1});
Result present = sh(home, FrameConnection.ACCOUNTS_COMMAND);
assertEquals(0, present.exit);
assertEquals(new TreeSet<>(Arrays.asList("123", "4567890")),
FrameConnection.parseAccounts(present.output));
// An unreadable root is an error, never an empty account list.
File broken = temporary.newFolder();
File file = new File(broken, ".local/share/Steam/userdata");
assertTrue(file.getParentFile().mkdirs());
Files.write(file.toPath(), new byte[]{1});
assertNotEquals(0, sh(broken, FrameConnection.ACCOUNTS_COMMAND).exit);
}
@Test public void unexpectedAccountOutputIsRejected() throws Exception {
for (String output : Arrays.asList("", "absent", "present", "present\n123", "absent\n123\n",
"present\nabc\n", "present\n\n", "ready\n", "present\n12 3\n")) {
try {
FrameConnection.parseAccounts(output);
fail("Accepted " + output);
} catch (IOException expected) {
// Fail closed: the poll is retried and no baseline changes.
}
}
assertEquals(Collections.emptySet(), FrameConnection.parseAccounts("present\n"));
}
}
@@ -0,0 +1,99 @@
package com.saphid.framecontrol;
import org.junit.Test;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import static org.junit.Assert.*;
public class SessionGateTest {
@Test public void endedSessionCannotPublishPageOrStop() {
SessionGate gate = new SessionGate();
List<String> shown = new ArrayList<>();
long session = gate.begin();
assertTrue(gate.status(session, "Syncing new screenshots", shown::add));
assertTrue(gate.page(session, "http://127.0.0.1:1/?key=a"));
assertTrue(gate.end(session));
assertFalse(gate.running());
assertNull(gate.page());
// The worker finishes its download after Stop and reports the closed tunnel.
assertFalse(gate.status(session, "Screenshot pending · Connection reset", shown::add));
assertFalse(gate.status(session, "Syncing new screenshots", shown::add));
assertFalse(gate.page(session, "http://127.0.0.1:1/?key=a"));
AtomicBoolean stopped = new AtomicBoolean();
assertFalse(gate.ifActive(session, () -> stopped.set(true)));
assertFalse(stopped.get());
assertEquals(Collections.singletonList("Syncing new screenshots"), shown);
assertEquals(SessionGate.DISCONNECTED, gate.state());
}
@Test public void lateWorkerCannotChangeANewerSession() {
SessionGate gate = new SessionGate();
List<String> shown = new ArrayList<>();
long first = gate.begin();
gate.end(first);
long second = gate.begin();
assertTrue(gate.status(second, "Connecting to headset", shown::add));
assertTrue(gate.page(second, "page-2"));
assertFalse(gate.status(first, "Syncing new screenshots", shown::add));
assertFalse(gate.page(first, null));
assertFalse(gate.end(first));
assertFalse(gate.ifActive(first, () -> fail("Stale session stopped the new one")));
assertEquals("Connecting to headset", gate.state());
assertEquals("page-2", gate.page());
assertTrue(gate.running());
assertEquals(Collections.singletonList("Connecting to headset"), shown);
long third = gate.begin(); // Even without an end, a newer session supersedes.
assertFalse(gate.status(second, "Syncing new screenshots", null));
assertTrue(gate.active(third));
assertNull(gate.page());
}
@Test public void endClearsProgressButKeepsFinalError() {
for (String progress : Arrays.asList("Syncing new screenshots",
"Syncing new screenshots · 1 file skipped (empty or over 30 MB)",
"Waiting for headset · Connection refused", "Waiting for a Steam account on the headset",
"Connecting to headset", "Screenshot pending · Download failed")) {
SessionGate gate = new SessionGate();
long session = gate.begin();
gate.status(session, progress, null);
gate.end(session);
assertEquals(progress, SessionGate.DISCONNECTED, gate.state());
}
SessionGate gate = new SessionGate();
long session = gate.begin();
gate.status(session, "Pair with your headset first.", null);
gate.end(session);
assertEquals("Pair with your headset first.", gate.state());
}
@Test public void endWaitsForAnInFlightNotificationAndNothingPostsAfterIt() throws Exception {
SessionGate gate = new SessionGate();
long session = gate.begin();
List<String> events = Collections.synchronizedList(new ArrayList<>());
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
Thread worker = new Thread(() -> gate.status(session, "Syncing new screenshots", text -> {
events.add("posting");
entered.countDown();
try { release.await(5, TimeUnit.SECONDS); } catch (InterruptedException e) { return; }
events.add("posted");
}));
worker.start();
assertTrue(entered.await(5, TimeUnit.SECONDS));
Thread stop = new Thread(() -> { gate.end(session); events.add("ended"); });
stop.start();
stop.join(200);
assertTrue("Stop must wait for the notification in flight", stop.isAlive());
release.countDown();
worker.join(5000);
stop.join(5000);
assertFalse(gate.status(session, "Syncing new screenshots", text -> events.add("late")));
assertEquals(Arrays.asList("posting", "posted", "ended"), events);
}
}
File diff suppressed because it is too large. Load diff
+1
View File
@@ -0,0 +1 @@
plugins { id 'com.android.application' version '9.1.0' apply false }
+2
View File
@@ -0,0 +1,2 @@
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
android.useAndroidX=true
Binary file not shown.
+8
View File
@@ -0,0 +1,8 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-9.4.1-bin.zip
networkTimeout=10000
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionSha256Sum=2ab2958f2a1e51120c326cad6f385153bb11ee93b3c216c5fccebfdfbb7ec6cb
Vendored Executable
+248
View File
@@ -0,0 +1,248 @@
#!/bin/sh
#
# Copyright © 2015 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
#
##############################################################################
#
# Gradle start up script for POSIX generated by Gradle.
#
# Important for running:
#
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
# noncompliant, but you have some other compliant shell such as ksh or
# bash, then to run this script, type that shell name before the whole
# command line, like:
#
# ksh Gradle
#
# Busybox and similar reduced shells will NOT work, because this script
# requires all of these POSIX shell features:
# * functions;
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
# * compound commands having a testable exit status, especially «case»;
# * various built-in commands including «command», «set», and «ulimit».
#
# Important for patching:
#
# (2) This script targets any POSIX shell, so it avoids extensions provided
# by Bash, Ksh, etc; in particular arrays are avoided.
#
# The "traditional" practice of packing multiple parameters into a
# space-separated string is a well documented source of bugs and security
# problems, so this is (mostly) avoided, by progressively accumulating
# options in "$@", and eventually passing that to Java.
#
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
# see the in-line comments for details.
#
# There are tweaks for specific operating systems such as AIX, CygWin,
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/2d6327017519d23b96af35865dc997fcb544fb40/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
#
##############################################################################
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
app_path=$0
# Need this for daisy-chained symlinks.
while
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
[ -h "$app_path" ]
do
ls=$( ls -ld "$app_path" )
link=${ls#*' -> '}
case $link in #(
/*) app_path=$link ;; #(
*) app_path=$APP_HOME$link ;;
esac
done
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
warn () {
echo "$*"
} >&2
die () {
echo
echo "$*"
echo
exit 1
} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD=java
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi
# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
fi
# Collect all arguments for the java command, stacking in reverse order:
# * args from the command line
# * the main class name
# * -classpath
# * -D...appname settings
# * --module-path (only if needed)
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
if "$cygwin" || "$msys" ; then
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
JAVACMD=$( cygpath --unix "$JAVACMD" )
# Now convert the arguments - kludge to limit ourselves to /bin/sh
for arg do
if
case $arg in #(
-*) false ;; # don't mess with options #(
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
[ -e "$t" ] ;; #(
*) false ;;
esac
then
arg=$( cygpath --path --ignore --mixed "$arg" )
fi
# Roll the args list around exactly as many times as the number of
# args, so each arg winds up back in the position where it started, but
# possibly modified.
#
# NB: a `for` loop captures its iteration list before it begins, so
# changing the positional parameters here affects neither the number of
# iterations, nor the values presented in `arg`.
shift # remove old arg
set -- "$@" "$arg" # push replacement arg
done
fi
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Collect all arguments for the java command:
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
# and any embedded shellness will be escaped.
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
# treated as '${Hostname}' itself on the command line.
set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
-jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
#
# In Bash we could simply go:
#
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
# set -- "${ARGS[@]}" "$@"
#
# but POSIX shell has neither arrays nor command substitution, so instead we
# post-process each arg (as a line of input to sed) to backslash-escape any
# character that might be a shell metacharacter, then use eval to reverse
# that process (while maintaining the separation between arguments), and wrap
# the whole thing up as a single "set" statement.
#
# This will of course break if any of these variables contains a newline or
# an unmatched quote.
#
eval "set -- $(
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
xargs -n1 |
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
tr '\n' ' '
)" '"$@"'
exec "$JAVACMD" "$@"
+93
View File
@@ -0,0 +1,93 @@
@rem
@rem Copyright 2015 the original author or authors.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem https://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@rem SPDX-License-Identifier: Apache-2.0
@rem
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@rem
@rem ##########################################################################
@rem Set local scope for the variables with windows NT shell
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
@rem Find java.exe
if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:findJavaFromJavaHome
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
if exist "%JAVA_EXE%" goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:execute
@rem Setup the command line
@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -jar "%APP_HOME%\gradle\wrapper\gradle-wrapper.jar" %*
:end
@rem End local scope for the variables with windows NT shell
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal
:omega
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env python3
"""Use the same reproducible headset bundle as the iPhone app."""
import pathlib
import subprocess
import sys
root = pathlib.Path(__file__).resolve().parents[2]
out = pathlib.Path(sys.argv[1]).resolve()
out.mkdir(parents=True, exist_ok=True)
version = subprocess.check_output(
[sys.executable, str(root / 'ios/scripts/make_frame_bundle.py'),
str(out / 'frame-bundle.tar.gz')], cwd=root, text=True).strip().splitlines()[-1]
(out / 'frame-bundle.version').write_text(version + '\n')
+4
View File
@@ -0,0 +1,4 @@
pluginManagement { repositories { google(); mavenCentral(); gradlePluginPortal() } }
dependencyResolutionManagement { repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS); repositories { google(); mavenCentral() } }
rootProject.name = 'FrameControl'
include ':app'
+107
View File
@@ -0,0 +1,107 @@
# Frame Control for Android phones
The native Android companion lives in `android/`. It requires Android 10 or
later and a Steam Frame with Developer Mode and SSH enabled. This is a new
implementation, separate from the headset's `frame/android` Lepton tools.
## Pair and sync
1. Enter the headset address, SSH port and username (normally `steamos`).
2. Choose **Pair headset**, compare the displayed SSH fingerprint with the
headset's identity, and accept it. Enter the Developer Mode password once.
The app installs this phone's SSH public key and verifies key login before
storing the pairing. The private key stays in Android Keystore; the
password is not saved. A changed headset host key is refused.
3. Choose **Start syncing** and allow notifications. The persistent notification
shows the active connected-device session and offers **Stop syncing**.
4. Take a new screenshot. Its original downloads into Frame Control. **Save
screenshots to Gallery** is enabled by default and adds it to
`Pictures/Frame Control`. Android 10+ lets an app create its own Gallery
items without broad photo-reading permission.
The first successful listing establishes a history baseline. Existing screenshots
are not imported automatically. The baseline is kept per Steam account: the app
also lists the headset's account folders, so an account that is present (even
with no screenshots yet) is established, and its later captures download. If
Steam's userdata folder is missing, or an account folder appears later (a new
sign-in, or one removed and restored), that account's screenshots at that moment
become its baseline instead of being imported. The status reads "Waiting for a
Steam account on the headset" while no account exists.
The app waits for two equal size/time listings, checks the original's byte count
and image decode, and saves it atomically. Identity includes the paired headset,
Steam account, game and filename. An empty (still being written) or over-30 MB
file is skipped for that poll only and does not hold up other screenshots; the
status shows how many were skipped. A capture still being written when the
baseline is taken downloads once it is complete. Unsafe identifiers, non-finite
times and conflicting duplicate rows still reject the whole listing. The receipt
journal is rewritten only when a receipt changes, not on every poll.
Turning Gallery off keeps automatic downloads inside Frame Control. Pending
exports become app-only; enabling it again does not export those earlier items.
Turning it off also removes this app's hidden, unfinished Gallery rows left by an
interrupted export, for every retained headset library (best effort; retried
when sync starts with Gallery off). Published Gallery images are never removed.
Previously saved Gallery items stay there. Original files remain available from
**Open / share original** when disconnected. The app keeps each headset's
library separately.
Gallery export uses a deterministic app-owned MediaStore row. A pending row is
rewritten after interruption, and a completed row is recognized before inserting
another. A failed export leaves the app original available for retry. Preferences
and the final Gallery commit are serialized so a disabled setting is respected.
Gallery exports are also serialized across sync sessions, so an export still
finishing after **Stop syncing** cannot remove or overwrite an image that a newer
session saves.
**Headset controls** opens the shared Frame Control web interface through a
loopback-only SSH tunnel. The server has a fresh session key and listens only on
the headset's loopback interface. The tunnel and server end with the phone's
SSH session. This initial Android shell does not implement all iPhone-specific
web bridges; native screenshot originals use the downloads screen.
## Background behavior and release boundary
Sync is an explicit, user-started Android `connectedDevice` foreground service.
It can continue after leaving the app while its notification is present and
reconnects when the headset returns. It is not a Firebase push receiver.
Android battery-saving/Doze, a locked phone with its screen off, loss of
network, force-stop or process termination can delay or end the session. The app
holds no wake lock; downloads may wait until the phone wakes. The app does not request battery-optimization
exemption, use unrelated background modes, or restart itself after force-stop.
Restart sync from the app if the session ends.
The full cross-platform request still requires actual locked-phone background
proof. iOS APNs delivery is a separate unimplemented dependency. An emulator
test is not evidence that either physical platform meets that requirement.
Stopping sync ends that session's status and notification; a transfer still
finishing in the background cannot bring them back or change a newer session.
Do not publish this as a verified always-on screenshot feature.
## Build and verify
Use JDK 17 and an Android SDK containing platform 36/build-tools 36.0.0:
```sh
export JAVA_HOME=/path/to/jdk17
export ANDROID_HOME=/path/to/android-sdk
./android/gradlew -p android assembleDebug testDebugUnitTest assembleDebugAndroidTest lintDebug
```
The Gradle wrapper pins version 9.4.1 and its distribution checksum. The build
generates the same reproducible headset bundle as iOS, including the existing
SHA-256-checked KDE Connect package fetch. Generated assets and build outputs
are ignored. No signing key, headset password or machine-specific SDK path is
committed. `assembleDebug` produces `android/app/build/outputs/apk/debug/app-debug.apk`.
It uses a development signing identity, not a configured Play release key.
`assembleDebugAndroidTest` builds instrumentation for real MediaStore round-trip,
retry recovery and app-only behavior. Optional real-headset hooks exist only in
the separate test APK (`ScreenshotDeviceTest`), not in the product app. They
provision a public test key and permit original/Gallery hash readback without
adding a product authentication bypass. Remove the temporary headset key and
fixtures after live tests.
No Play app creation, store upload, invitation or public release is part of a
debug build. Preserve a dedicated release signing identity and verify physical
devices before a production release.
@@ -0,0 +1,97 @@
# Android phone companion: review fixes (2026-10-26)
Base: `ae9df354`. This evidence covers fixes for seven findings from an
independent review of the Android companion (issue #108).
## Status of earlier evidence
The APK, emulator and physical-device evidence in
`android-phone-screenshot-sync-2026-10-24.md` was captured from `ae9df354`.
It is **historical**. It does not show that the fixes below work on a device.
No emulator, phone or headset was used for the initial implementation round.
Later parent verification is recorded below.
## What changed
| Finding | Fix | Covered by |
| --- | --- | --- |
| F1 Stop could leave an ongoing notification or wrong status | `SessionGate` gives each service instance one session token, so an ended session cannot post status, page or notification updates. `SyncEngine.close()` stops all writes before a new session opens the library. `onDestroy` also cancels notification 108. | `SessionGateTest` (4), `closedEngineCannotWriteAfterAnotherEngineTakesOver` |
| F2 Pairing could merge a key into an unterminated last line | `FrameConnection.authorizeCommand` uses the same newline guard as the iPhone app. | `FrameConnectionTest` runs the generated shell against a temporary HOME |
| F3 Every 4-second poll rewrote and fsynced the journal | `reconcile` commits only when a receipt or the account set changes. | 3 tests that turn any journal write into a failure |
| F4 One empty or over-30 MiB file blocked every listing | Out-of-range sizes are skipped for that poll only. Unsafe ids, non-finite times and conflicting duplicates still reject the whole listing. | `emptyOrOversizedFiles…` |
| F5 The screen parsed, and could write, the journal on the UI thread | `SyncEngine.snapshot` is a read-only view. The screen loads it on a background thread, drops results for a headset that is no longer paired, and clears the list when the pairing changes. | `readOnlySnapshotNeverCreatesUpgradesOrRewrites`; existing native `changingHeadsetsRebindsShareToTheSelectedOriginal` compiles |
| F6 An opt-out after an interrupted export left a hidden pending row | `GalleryExporter.discardPending` removes this app's pending rows in `Pictures/Frame Control/` when Gallery is turned off, and again when sync starts with Gallery off. Published rows are kept. | Native `optOutDiscardsInterruptedPendingRowsButKeepsPublishedImages` (compiled, not run) |
| F7 Missing userdata, or a new account, could import its history | Each Steam account gets its own baseline. The app lists the account folders over SSH, so an account with no screenshots yet still counts as established. Version 1 journals upgrade to version 2 and trust only the accounts they already have receipts for. `ui/server.py` is unchanged. | 6 JVM tests plus `accountListingSeparatesAbsentUserdataFromEmptyAccounts` |
Nonblocking items: the workflow's `push` trigger is now `main` only, and the
docs and app text now say that a locked phone can delay transfers.
## Verification
Command (JDK 17, Android SDK platform 36):
```sh
./android/gradlew -p android clean assembleDebug testDebugUnitTest assembleDebugAndroidTest lintDebug --no-daemon
```
- Exit 0. There are 53 JVM tests (34 before), all passing, none skipped.
- Lint reports 0 errors and 12 warnings. All 12 are pre-existing kinds (version hints, i18n, WebView JavaScript).
- Mutation check: restoring the old unconditional commit and the old key append made 4 tests fail.
- `app-debug.apk` SHA-256: `e4a1f498fef6fbc9919ccb4077b8e15b609caea6a2192d3f1bd2b13ea78bcb15`
- `app-debug-androidTest.apk` SHA-256: `f9acb6bc002665cff6a31d0aaf2d528f4b1eaaaa82ead9fb383867e1154f6053`
## Round 2: fixes for the independent Sol review
An independent review (GPT-6.1 Sol, high reasoning) of the fixes above found two
product defects and one test timing defect. All three are addressed below.
| Finding | Fix | Covered by |
| --- | --- | --- |
| R1 A Gallery change before the first poll on a version 1 journal wrote an unreadable journal | A setting-only write now keeps the journal's version, so a version 1 journal stays a valid version 1 journal. Only the first poll upgrades it, and it writes version 2 and the account set together. The baseline policy is unchanged. | JVM `versionOneJournalStaysValidWhenGalleryChangesBeforeFirstPoll` (constructor and toggle paths, immediate reopen and snapshot, then the upgrade). Native `galleryOptOutLatchesPreviouslyPairedLibraries` now includes a retained version 1 library. |
| R2 After Stop, a late export from the old session could delete the Gallery image that the next session had just saved | `GalleryExporter.EXPORTS` is one process-wide lock. Each export holds it from the row lookup through publish or cancellation, across exporter instances and sessions. It is taken before the settings lock and never while holding it. The wait is interruptible and limited to 30 seconds. After that, the item fails for this poll and is retried. | Native `stoppedSessionExportCannotDeleteNextSessionsGalleryImage`. A JVM harness outside the repository runs the real exporter and engine with an in-memory resolver: with the lock, 10 of 10 runs keep the image; with the lock removed, 10 of 10 runs lose it. |
| R3 The headset-switch share test assumed the background library load had finished | The test waits, up to a time limit, until the expected original is shareable. It accepts no button while a library loads. It then checks that only one share button remains and that it shares the new headset's original. | `changingHeadsetsRebindsShareToTheSelectedOriginal` |
The parent's first native run (API 36, APK `e4a1f498…`) ran 5 tests: 4 passed.
`optOutDiscardsInterruptedPendingRowsButKeepsPublishedImages` reached and
passed its assertions, then its cleanup threw `SecurityException` deleting a
row the product had already removed. The test now drops that row from cleanup
after it proves the row is gone.
The parent also ran the account-listing command read-only against the real
headset. It exited 0 and listed one account. That is not proof of the phone's
download pipeline.
Round 2 verification (same command, JDK 17, platform 36):
- Exit 0. There are 54 JVM tests, all passing: `SyncEngineTest` 47, `SessionGateTest` 4, `FrameConnectionTest` 3.
- Lint reports 0 errors and 12 warnings, the same pre-existing kinds as round 1.
- Mutation check for R1: removing the version-preserving line made only the new R1 test fail, with the invalid version 2 header the review described.
- Two clean builds gave the same hashes. They replace the round-1 APKs:
- `app-debug.apk` SHA-256: `139443c8f3246527ed629731b4f2a099939a2ba7590b628cde415f3f0d7ab996`
- `app-debug-androidTest.apk` SHA-256: `3a84246a7b40436fd5fc521154541603050805270e21e052e26b31a5fa1e0cc2`
## Parent verification of the round-2 APKs
The parent installed both round-2 APKs above on the isolated API 36 emulator
and ran the six non-live instrumentation methods explicitly. All six passed,
none skipped; the supervisor exited 0. This includes actual MediaStore
round-trip and retry, retained version 1 library opt-out, asynchronous
headset-switch sharing, hidden pending-row removal, and the FIFO-controlled
Stop/restart export overlap. The chooser was intercepted; no original was
sent to another app.
The native overlap test exercised real MediaProvider rows: after both old and
new exporters finished, the successor's image remained published with matching
original bytes. This verifies the overlap mechanism, rather than a complete
service Stop/restart against a live headset.
The parent also checked the 54 passing JVM XML results, the build's recorded
exit 0, whitespace, and both APK hashes. Native output and sanitized metadata
are in `android-phone-review-fixes-2026-10-26/` beside this document.
## Not verified
- API 29 runtime behavior and physical Android locked/Doze behavior remain unverified.
- The account-listing command ran against the real headset; the final phone download pipeline and password-pairing UI have not been rerun against it.
- A complete service Stop/restart during a live headset download is not proven by the exporter overlap test.
- iOS APNs delivery, physical locked-phone Photos export, production signing, and release remain open.
@@ -0,0 +1,101 @@
1
Performing Streamed Install
Success
Performing Streamed Install
Success
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=
com.saphid.framecontrol.ScreenshotDeviceTest:
INSTRUMENTATION_STATUS: test=changingHeadsetsRebindsShareToTheSelectedOriginal
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=1
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=changingHeadsetsRebindsShareToTheSelectedOriginal
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=2
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=
INSTRUMENTATION_STATUS: test=stoppedSessionExportCannotDeleteNextSessionsGalleryImage
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=2
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=stoppedSessionExportCannotDeleteNextSessionsGalleryImage
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=3
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=
INSTRUMENTATION_STATUS: test=optOutDiscardsInterruptedPendingRowsButKeepsPublishedImages
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=3
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=optOutDiscardsInterruptedPendingRowsButKeepsPublishedImages
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=4
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=
INSTRUMENTATION_STATUS: test=galleryOptOutLatchesPreviouslyPairedLibraries
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=4
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=galleryOptOutLatchesPreviouslyPairedLibraries
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=5
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=
INSTRUMENTATION_STATUS: test=actualGalleryOriginalAndAppOnlySurviveRelaunch
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=5
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=actualGalleryOriginalAndAppOnlySurviveRelaunch
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=6
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=
INSTRUMENTATION_STATUS: test=interruptedMediaStoreRowIsRewrittenWithoutDuplicate
INSTRUMENTATION_STATUS_CODE: 1
INSTRUMENTATION_STATUS: class=com.saphid.framecontrol.ScreenshotDeviceTest
INSTRUMENTATION_STATUS: current=6
INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
INSTRUMENTATION_STATUS: numtests=6
INSTRUMENTATION_STATUS: stream=.
INSTRUMENTATION_STATUS: test=interruptedMediaStoreRowIsRewrittenWithoutDuplicate
INSTRUMENTATION_STATUS_CODE: 0
INSTRUMENTATION_RESULT: stream=
Time: 3.171
OK (6 tests)
INSTRUMENTATION_CODE: -1
@@ -0,0 +1,51 @@
# Independent Android fixes review — 2026-10-26, round 2
**Verdict: no actionable findings in the frozen Android hardening delta.** The previous Sol findings R1, R2 and R3 are addressed. This is a source-slice review, not a release-readiness verdict.
Reviewer: current T3 Codex runtime, `gpt-6.1-sol`, high reasoning. I performed the review myself, without delegation.
## Exact target and boundaries
- Worktree: `/Users/saphid/.t3/worktrees/steam-frame/feat-phone-screenshot-autosync-108`.
- Branch: `feat/phone-screenshot-autosync-108`.
- HEAD: `ae9df3544239afc1f2cab17ec10fe07f6fe9e657`, plus the frozen uncommitted delta.
- All **15 paths** in `/tmp/frame108-android-fixes-review-20261026-r2.json` matched their SHA-256 values before and after review. Manifest SHA-256: `e3417c56926d08faecb6e5beacaf081c9e4fc395ef494c4de999a4b2282900ad`.
- I inspected the complete tracked delta, the untracked SessionGate and test files, the evidence files, and surrounding engine, exporter, service, activity and connection logic. I checked callers and lock order as well as the changed lines.
- I read the original Opus review, first Opus fixes report, previous Sol review and reproductions, latest Opus fixes report, and `docs/evidence/android-phone-review-fixes-2026-10-26.md`.
- No `CODING_STANDARDS.md` exists in the worktree, its ancestor directories, or the primary checkout root. I read `CONTRIBUTING.md`; no relevant violated rule was found.
I made no repository edits, did not read `.frame_pwd` or unrelated dirty primary-checkout files, and used no device, headset or browser. I made no external writes, commits, pushes, PR changes or releases. Review outputs and local test classes are under `/tmp`.
## Previous Sol findings
- **R1 — addressed.** `SyncEngine.java:668-673` now preserves both the existing journal version and its account-set representation in `copy()`. Setting-only writes on v1 stay valid v1. Reconcile upgrades to v2 alongside its account set at `:285-304` and `:337-343`. The new JVM regression covers constructor preference reconciliation and an explicit toggle before any poll, immediate snapshot/reopen, repeated toggles, and the subsequent migration/baseline behavior. It passed in my independent run. The retained-v1 instrumentation path is present and passed in the parent's supplied API 36 log.
- **R2 — addressed.** `GalleryExporter.java:74-84` acquires the process-wide `EXPORTS` lock interruptibly with a 30-second timeout, and releases it in `finally`. It covers deterministic row lookup, byte copying, publication and cancellation deletion across exporter instances. The order is EXPORTS → settings monitor → engine monitor. Neither Gallery toggle nor startup cleanup acquires EXPORTS while holding the settings monitor; I found no inverse acquisition path. The old exporter must finish its row work before its successor recovers/inserts a row. I independently recompiled the frozen exporter and engine and ran the existing bounded HandoffRace reproduction with an in-memory resolver: the successor waited, the old cancellation deleted only its pending row, and the successor's published row survived with a saved receipt. The native FIFO test separately checks actual published-row existence and original bytes; its passing result is parent-supplied evidence, not my own device run.
- **R3 — addressed.** `ScreenshotDeviceTest.java:487-517` tolerates a missing button while the background library loads and waits up to 15 seconds for a share URI different from the previous one. The test verifies A and B bytes, then checks exactly one final share button targeting B. It no longer treats main-queue idleness as library completion. The cleanup adjustment removes only the fixture URI already proven deleted; other cleanup errors remain visible. I inspected this code and the parent's passing instrumentation result; I did not independently execute instrumentation.
## Fresh pass over the earlier fixes
- **F1:** SessionGate serializes visible status/page/notification effects against session end. Permanent engine close rejects subsequent reconciles and setting writes, and invalidates in-flight journal commits. Late engine assignment and stale-worker finalization are guarded. Gate and takeover tests passed; complete live-service lifecycle behavior remains outside this review.
- **F2:** the generated authorized_keys command terminates an unterminated existing line before appending and remains idempotent. The actual shell fixture cases passed locally.
- **F3:** unchanged listings no longer persist the journal, while retryable downloads/exports continue. Observation resets and account-set changes still commit. The write-blocking regression tests passed.
- **F4:** invalid sizes are excluded per item; unsafe identities, non-finite times and conflicting duplicates still reject a poll. Stability resets and completion after a skipped observation are covered by passing JVM tests.
- **F5:** library snapshots do not create or rewrite storage. Activity parsing/file checks run on its library executor, live-engine selection matches the requested pin, pin changes clear the old list, and results for a no-longer-selected pin are discarded. I found no further actionable sharing regression.
- **F6:** cleanup selects app-owned pending rows in the expected folder, serialized against final publication, and keeps published images. It remains best effort. Provider failure can leave hidden appOnly bytes, particularly on API 29 if Gallery is re-enabled before another cleanup. This documented residual limit is not a newly discovered blocker.
- **F7:** per-account baselines handle absent userdata, established empty accounts, newly appearing accounts, and accounts observed between the SSH and screenshot queries. The migration preserves the intended baseline policy. The shell/parser and account-baseline tests passed.
The workflow trigger and documentation changes are consistent with this scope. I found no new concrete regression requiring a fix.
## Actual verification
All review-generated artifacts are in `/tmp/frame108-android-sol-review-fixes-20261026-r2-evidence/`.
1. **Independent JVM execution:** compiled current SyncEngine, SessionGate, FrameConnection, PhoneIdentity and all three JVM test classes with JDK 17, cached declared dependencies and SDK 36 into a fresh `/tmp` classes directory. Direct JUnit run: **54 tests passed**, exit 0. No Gradle task or repository build output was changed. Commands, compile log/exit and JUnit log/exit are saved there.
2. **Independent local handoff execution:** compiled the current GalleryExporter against those classes and the existing HandoffRace Android substitutes. Compile and execution both exited 0. Output ended with `exists=true pending=false receipt=saved` and `RESULT: published row survived`. This reuses the inspected existing harness and proves the application-level interleaving; it is not MediaProvider instrumentation. See `race.log` and `race.rc`.
3. **Parent evidence inspected:** final clean build log and exit file record success; existing XML totals are 47 engine + 4 gate + 3 connection tests, all zero failures/errors/skips. The frozen API 36 instrumentation log records **six passing tests, no skips**, and supervisor exit 0. I did not rerun assemble, lint or instrumentation. The parent's lint result is 0 errors and 12 existing warnings.
4. **APK hashes independently recomputed:** app `139443c8f3246527ed629731b4f2a099939a2ba7590b628cde415f3f0d7ab996`; test `3a84246a7b40436fd5fc521154541603050805270e21e052e26b31a5fa1e0cc2`. Both match the supplied round-2 evidence.
5. **Freeze/whitespace:** all 15 source/evidence hashes matched again after checks; branch and HEAD matched; `git diff --check` exited 0. Final hashes and whitespace result are saved in the evidence directory.
## Verification limits
Physical locked/Doze behavior, API 29 runtime behavior, complete service Stop/restart during a live headset download, the final phone-to-headset pipeline, iOS APNs/locked Photos behavior, production signing and release are not established here. A stuck old provider write can delay successor exports; the timed lock makes those attempts retryable but does not cancel the stuck I/O. Gallery-toggle I/O on the main thread is an existing disclosed limitation.
The successful read-only headset account listing supports that command only. Historical device evidence from the committed ae9df354 implementation does not establish the frozen fixes. These limits do not create an actionable defect in this reviewed slice or justify a release claim.
@@ -0,0 +1,32 @@
{
"taskId": "node:delegated-task:command%3Amcp%3A3ef9db23-13cf-4959-afe1-b003a3f2d950%3Adelegate-task%3Aframe108-android-sol-review-fixes-20261026-r2",
"providerInstanceId": "codex_secondary",
"model": "gpt-6.1-sol",
"reasoningEffort": "high",
"status": "completed",
"findings": "No actionable findings in the frozen Android hardening delta; R1-R3 addressed.",
"frozenFiles": {
".github/workflows/android.yml": "aaea48149abd1030f18a844d9f245d9a0ad249152e5df838cd53c8863f7f2220",
"android/app/src/androidTest/java/com/saphid/framecontrol/ScreenshotDeviceTest.java": "87f9629858a75a2ba4ced1a0f4ccc792d03a40bd39fed7e25e211cd6c1d15cbf",
"android/app/src/main/java/com/saphid/framecontrol/FrameConnection.java": "edb7b62b4d9a01e8e6802e8758504044c965f6d1b2939536617a3f57832b13a9",
"android/app/src/main/java/com/saphid/framecontrol/GalleryExporter.java": "b0c0c5331ea1184baf780f67329908d481491e7383bfb98c09733ada84582b5d",
"android/app/src/main/java/com/saphid/framecontrol/MainActivity.java": "8069a9c6ac3ab0e5e6bb315055dd2bef2a28b6f1c0084f8355db237857b736c1",
"android/app/src/main/java/com/saphid/framecontrol/SyncEngine.java": "0096ae5002c7ba5d1c7b8128aeee920eb4f29d8d94ad71d885ce86c5be70c854",
"android/app/src/main/java/com/saphid/framecontrol/SyncService.java": "dd852b8ef7f45fc472a8d790ba7e335040355889ef8d082ba8f1e80a1d4fd690",
"android/app/src/test/java/com/saphid/framecontrol/SyncEngineTest.java": "df1fa6ce8873098ea78fca3f5069e31712eae5eedde54824dd83216b6ac67f09",
"docs/android-phone.md": "a274daed1118be64b83313eaad03ddbf080fbbc7dad6e39ce601843bc227c1a3",
"android/app/src/main/java/com/saphid/framecontrol/SessionGate.java": "470a164820ececcbe897f55b3f4c90498c215d9dd1b93554e08d15fcc22a9fa2",
"android/app/src/test/java/com/saphid/framecontrol/FrameConnectionTest.java": "05010052ef11e9d9e019ae4ccb2959a8931a03b2d23528832d4821d184623727",
"android/app/src/test/java/com/saphid/framecontrol/SessionGateTest.java": "5e1e88bee54c82188ce0a1ac8c37d081565fc41c827c67b7d5ef0070b00ab15e",
"docs/evidence/android-phone-review-fixes-2026-10-26.md": "eb2a961abe935541c04e65d43ad8d61036b73afe8f63696891771f5e8bc6b7b1",
"docs/evidence/android-phone-review-fixes-2026-10-26/api36-instrumentation.txt": "d31e729b455ddd8ff3122e5a68bd2221361e27ef3553ba737447598a39d89bd9",
"docs/evidence/android-phone-review-fixes-2026-10-26/verification.json": "2bd1644b0c77c83f3ce85ae1713baba1ea7c0cedce0161bc35e3eff587bce7a7"
},
"manifestSha256": "e3417c56926d08faecb6e5beacaf081c9e4fc395ef494c4de999a4b2282900ad",
"independentVerification": {
"jvmTests": 54,
"jvmFailures": 0,
"localExportHandoff": "passed"
},
"productionReleaseReady": false
}
@@ -0,0 +1,23 @@
{
"platform": "Android API 36 emulator",
"nativeTests": 6,
"nativeFailures": 0,
"nativeSkipped": 0,
"supervisorExit": 0,
"jvmTests": 54,
"jvmFailures": 0,
"jvmErrors": 0,
"jvmSkipped": 0,
"buildExit": 0,
"headsetAccountListing": {
"sshExit": 0,
"state": "present",
"accountCount": 1,
"numericEntriesOnly": true
},
"apkSha256": {
"android/app/build/outputs/apk/debug/app-debug.apk": "139443c8f3246527ed629731b4f2a099939a2ba7590b628cde415f3f0d7ab996",
"android/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk": "3a84246a7b40436fd5fc521154541603050805270e21e052e26b31a5fa1e0cc2"
},
"productionReleased": false
}
@@ -0,0 +1,168 @@
# Android phone screenshot sync evidence — 2026-10-24
Work: [#108](https://github.com/saphid/frame-control/issues/108). The user
authorized a new Android companion after the earlier APK could not be located.
This is a development build, not a production or Play release.
## Implemented
Native Android 10+ companion in `android/`, with pinned SSH transport and a
non-exportable Android Keystore phone key. Pairing checks the host fingerprint
before accepting a password and verifies key login before saving the pairing.
The shared headset server runs through a session-bound loopback tunnel.
All retained libraries receive Gallery opt-out, including previously paired
headsets. Paging keeps older offline originals reachable beyond the first 100.
A failed image or Gallery export retries independently without starving later
downloads; local persistence errors still halt immediately.
The user starts a connected-device foreground service with an ongoing Stop
notification. New screenshot originals download automatically. Gallery is ON
by default; turning it off latches pending and newly discovered items to
app-only, including across relaunch and later re-enabling. Historical captures
are baselined instead of imported. Receipts, original files, and MediaStore
recovery support interrupted downloads/exports without duplicate Gallery rows.
See [usage and background limits](../android-phone.md).
## Build and automated checks
- JDK 17, Gradle 9.4.1, AGP 9.1.0; compile/target API 36, minimum API 29.
- `assembleDebug testDebugUnitTest assembleDebugAndroidTest lintDebug` exited
zero after review fixes: `/tmp/frame-108-android-build-final-r5.log`.
- 34 JVM sync-engine tests, zero failures/errors/skips. Tests cover baseline,
stability, identity, metadata limits, destination latching, retries, atomic
receipts, corrupt state, unsafe paths, late cancellation callbacks, per-item
failure isolation and fatal persistence errors.
- API 36 arm64 emulator MediaStore tests passed: original byte round-trip,
app-only across relaunch, interrupted pending row recovery and retry dedup.
The final rebuilt APK passed **five selected native checks, no skips**:
three MediaStore/multiple-library cases plus the actual headset fixture
Gallery hash (`/tmp/frame-108-android-final-device-r4.log`, four tests), then
app-only original/no-Gallery readback
(`/tmp/frame-108-android-final-apponly-device-r4.log`, one test).
- Existing server suite: 1,333 tests, 11 skips, exit zero,
`/tmp/frame-108-android-server-tests.log`. Website: 29 passed, zero skips,
`/tmp/frame-108-android-site-tests.log`.
- Lint has no errors. Warnings include dependency/target freshness, English UI
literals, backup-version compatibility, and the shared controls' JavaScript.
## Actual headset → Android emulator → Gallery
Target: task-owned headless API 36 emulator `emulator-5588`, with the actual
Steam Frame reached over SSH. Initial baseline contained nine historical
captures and imported none. An emulator-generated public test key was installed
temporarily through the existing trusted SSH alias; no password was exposed.
The unauthenticated native host-key probe matched the trusted headset key.
Synthetic fixture IDs use a separate test account/game namespace; this verifies
new-file delivery from the real headset, not the hardware capture shortcut.
| Check | Result |
| --- | --- |
| Gallery default ON | `999999999990/108/20261024100101_1.png` automatically downloaded and receipt became `saved` |
| Source / app / Gallery SHA-256 | `c9411d718c8a3477b7c7a6eed5ab827570605fce977bdeef5aaf11d29ba88316` |
| MediaStore dedup | Exactly one matching Gallery row; native test reads its original bytes |
| Shared headset controls | Actual authenticated WebView loaded the headset app library |
| Gallery OFF, app on Home | Second fixture automatically downloaded in 4.293 seconds (one synthetic trial) |
| App-only source / app SHA-256 | `6a1513b2dc3459b457f7dd86adcde7298d99dccc31c73904d7c6698703edb961` |
| App-only independent readback | One selected native test passed: correct app original and no matching Gallery row |
| Re-enable and restart | App-only receipt and absent Gallery row preserved; first image remains saved, no duplicate |
| Open/share original | Native chooser opened for the app-only original and was canceled; nothing sent |
| Final r5 APK, Gallery ON while Home | Third new fixture `999999999990/108/20261024101413_1.png` reached Gallery in 5.288 seconds (one synthetic trial); same source/app/Gallery hash as the first fixture, one matching row |
Background readback: `/tmp/frame-108-android-background-device-r1.log`.
Home was foreground with the app service active; the screen stayed on. This
is not a locked/Doze or long-duration physical-phone test.
The final r5 background Gallery readback passed separately (one selected native
test, no skip): `/tmp/frame-108-android-final-background-gallery-r5.log`.
Sanitized [machine readback](android-phone-screenshot-sync-2026-10-24/live-proof.json)
is committed; local full readback: `/tmp/frame-108-android-live-proof.json`. Fixture source
metadata: `/tmp/frame-108-android-fixtures.json`. The optional provisioning test
passed separately. Earlier full-class instrumentation runs included two
optional skipped hooks; those are not counted as four executed scenarios.
![Native saved receipt with Gallery enabled](android-phone-screenshot-sync-2026-10-24/gallery-on.png)
![Gallery re-enabled without backfilling app-only downloads](android-phone-screenshot-sync-2026-10-24/reenabled.png)
![Native connection and Gallery default](android-phone-screenshot-sync-2026-10-24/setup.png)
![Shared controls loaded from the real headset](android-phone-screenshot-sync-2026-10-24/controls.png)
## Verification boundaries
No physical Android phone, Doze reliability, hardware screenshot latency,
password-pairing UI, Play signing identity or production upload has been
verified. Android sync uses an explicit foreground session, not Firebase.
The iOS foreground implementation has separate
[evidence](phone-screenshot-sync-2026-10-24.md); APNs, its delivery service,
notification extension and locked-phone Photos export remain unimplemented or
unverified. The user's full-feature tested-and-working release condition is
not satisfied.
T3's device tools incorrectly inferred the Android SDK from the platform-tools
Cask and could not open this emulator. The Astra worker used the app-owned
agent-device launcher with a task-local session/config and the correct SDK.
No global SDK/T3 settings or physical devices were changed.
UI verification: provider `codex`, model `gpt-6-astra`, completed task
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-ui-astra-20261024-r1`.
Report: `/tmp/frame-108-android-ui-report.md`.
## Review and final artifacts
The APK is `android/app/build/outputs/apk/debug/app-debug.apk`, signed with the
development identity. No store upload or release has been performed.
Final r6 APK SHA-256: `e6a99a35154c21293975ea165eb967d8cc7ad84565f063255c3848ada1169f6e`.
Preferred different-model review (`claudeAgent` / `claude-opus-5-5`) failed
before work with `Claude API rate limit reached. Try again later.` Task:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-opus-20261024-r1`.
Implementation and independent fresh fallback reviews used Sol; they are not
different-model evidence.
Provider `codex_secondary`, model `gpt-6.1-sol`, high, completed round 1:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-sol-20261024-r1`.
All four findings were accepted and fixed: failed-item starvation, API-29
pending-row recovery query, opt-out for other retained headset libraries, and
inaccessible originals beyond the latest 100. Report:
`/tmp/frame-108-android-review-sol-r1.md`. No CODING_STANDARDS.md was found in
the task or primary worktree; CONTRIBUTING.md was included in review context.
Provider `codex_secondary`, model `gpt-6.1-sol`, high, completed round 2:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-sol-20261024-r2`.
It cleared all four prior fixes and independently reran 34 JVM tests, then found
a library-switch cache identity issue: matching receipt fields could retain a
Share action bound to the previous headset. The refresh signature now includes
the absolute original path, including the headset library root.
The final r6 build (`assembleDebug assembleDebugAndroidTest lintDebug`) exited
zero: `/tmp/frame-108-android-build-r6.log`. A new native regression launches
the real activity, switches between two libraries with matching receipt fields
but different original bytes, and verifies that Share now selects B’s bytes.
The chooser is intercepted; nothing is sent externally. This regression and
the three existing MediaStore/multiple-library cases passed together: **four
selected tests, no skips**, `/tmp/frame-108-android-final-device-r6.log`. Core
code is unchanged from the 34 independently passing JVM tests. The r6 change
does not alter the r5 transfer/storage path exercised against the real headset.
Provider `codex_secondary`, model `gpt-6.1-sol`, high, completed round 3:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-sol-20261024-r3`.
**No actionable findings** in the frozen 25-file executable target (`android/`
and `.github/workflows/android.yml`), with the remaining share-binding finding
resolved. All executable fingerprints stayed unchanged during review. Report:
`/tmp/frame-108-android-review-sol-r3.md`. The review inspected the passing r6
native/build logs and unchanged core covered by round 2’s independent 34-test
run; it did not claim new device runs or release validation.
Cleanup removed the one exact temporary public key and all three exact synthetic
headset fixture paths, verified absent. Other authorized keys and real captures
were preserved. Readback: `/tmp/frame-108-android-cleanup.json`. The task-owned
`emulator-5588` was shut down after the live tests, then restarted only for
the r6 share-binding regression and shut down again;
no other device was stopped. Build artifacts, the AVD and local test logs remain
available for follow-up verification.
@@ -0,0 +1 @@
{"removedTestKeys": 1, "removedFixtures": ["999999999990/108/20261024100101_1.png", "999999999990/108/20261024100102_1.png", "999999999990/108/20261024101413_1.png"], "testKeyAbsent": true, "allFixturesAbsent": true}
Binary file not shown.

After

Width:  |  Height:  |  Size: 391 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 226 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 185 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 177 KiB

@@ -0,0 +1,57 @@
{
"gallery": {
"id": "999999999990/108/20261024100101_1.png",
"sha256": "c9411d718c8a3477b7c7a6eed5ab827570605fce977bdeef5aaf11d29ba88316",
"bytes": 3620,
"createdAt": 1792836168.6100879,
"receipt": {
"destination": "saved",
"observations": 2,
"shot": {
"id": "999999999990/108/20261024100101_1.png",
"size": 3620,
"time": 1792836168.6217859
},
"state": "downloaded"
},
"readbackAt": 1792836209.819923
},
"appOnly": {
"id": "999999999990/108/20261024100102_1.png",
"sha256": "6a1513b2dc3459b457f7dd86adcde7298d99dccc31c73904d7c6698703edb961",
"bytes": 3623,
"createdAt": 1792836368.616906,
"background": "Home foreground; connectedDevice service active",
"receipt": {
"destination": "appOnly",
"observations": 2,
"shot": {
"id": "999999999990/108/20261024100102_1.png",
"size": 3623,
"time": 1792836368.6332257
},
"state": "downloaded"
},
"readbackAt": 1792836372.909552,
"elapsedSeconds": 4.293
},
"finalBackgroundGallery": {
"id": "999999999990/108/20261024101413_1.png",
"sha256": "c9411d718c8a3477b7c7a6eed5ab827570605fce977bdeef5aaf11d29ba88316",
"bytes": 3620,
"createdAt": 1792836871.5926661,
"background": "Home foreground; connectedDevice service active; final r5 APK",
"receipt": {
"destination": "saved",
"observations": 2,
"shot": {
"id": "999999999990/108/20261024101413_1.png",
"size": 3620,
"time": 1792836871.6072419
},
"state": "downloaded"
},
"readbackAt": 1792836876.8808918,
"elapsedSeconds": 5.288
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 187 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 238 KiB

@@ -0,0 +1,119 @@
# iOS foreground screenshot downloads — 2026-10-24
This is implementation and proof for the foreground iOS portion of
[#108](https://github.com/saphid/frame-control/issues/108). It is not the full
requested background/cross-platform feature and is not released.
## Source and merge
Dedicated worktree: `/Users/saphid/.t3/worktrees/steam-frame/feat-phone-screenshot-autosync-108`.
Branch: `feat/phone-screenshot-autosync-108`.
First merged fetched main
[1d4c57d6](https://github.com/saphid/frame-control/commit/1d4c57d6)
into the existing Mac-space parent
[3fdc6c46](https://github.com/saphid/frame-control/commit/3fdc6c46).
Merge commit: [52772323](https://github.com/saphid/frame-control/commit/52772323).
The dirty primary checkout was left untouched. Review found and fixed the PC
helper's missing public route for the extracted `mac-stream.js` dependency.
Independent merge re-review ran 45 tests successfully, one skipped, and found
no remaining actionable findings in its seven-file target.
## Implemented behavior
- Native polling every four seconds while active, independent of the visible web tab.
- First successful listing becomes the durable baseline; only new captures queue.
- Two equal metadata observations, exact byte count and complete ImageIO decode
precede an atomic original write outside purgeable caches.
- Receipts and originals distinguish pinned headset identity, Steam account,
app ID and filename. Reconnect/relaunch resumes pending work.
- Photos is enabled by default. Add-only permission is requested in foreground.
Denied/failed exports keep originals; app-only captures never automatically
export after re-enabling the setting.
- Native downloads sheet exposes retained files through ShareLink, including
during connection failure/relaunch. Ambiguous interrupted Photos commits need
explicit recovery rather than risking an automatic duplicate.
- Backgrounding stops polling; no unrelated background mode is used.
## Automated verification
Normal signed Simulator build and full native test suite: **20 tests passed,
zero failures**, including eight screenshot tests. Result bundle:
`/tmp/frame-108-ios-all-r3.xcresult`; log: `/tmp/frame-108-ios-all-r3.log`.
Both command exit status and xcresult test count were checked. Private
DerivedData cleanup completed. Test target was the independently created
Simulator `F250F91E-2E7D-4833-B029-A275F11FDC7D` on iOS 26.5.
Screenshot tests cover baseline/stability, truncated data, restart dedup,
Photos failure/retry, changes during download/consent, cancellation,
account/app identity, corrupt-ledger failure, uncertain export handling,
late success and failure across journal replacement, default preference and
offline app-only originals. Download/export dependencies in these tests are
injected; the live test below exercises actual transport and PhotoKit.
Python iOS packaging/Foundation contracts: 5 passed. Page scripts: 5 passed.
`swiftc -typecheck` for the journal and scoped `git diff --check` passed.
## Actual headset → Simulator → Photos proof
Ran the production signed app against the real headset over its pinned SSH
connection and the actual deployed `/api/shots` and `/api/shots/image` endpoints.
Created two temporary valid PNGs under a unique synthetic Steam account/app
directory after the app had baselined the existing nine screenshots.
| Check | Observed result |
|---|---|
| Default Photos ON | Original downloaded and receipt marked `saved`; Photos contained one byte-identical original |
| Source / app copy / Photos SHA-256 | `a36eb2d497e6ecce17c767189a70b3b7e732a22309d2421cdc715cf13a7668ac` |
| Photos OFF | Second original downloaded with `appOnly`; no matching Photos original |
| Re-enable + relaunch | Second receipt stayed `appOnly`; first still had exactly one Photos original |
Machine readback: `/tmp/frame-108-live-proof.json`. Photos readback hashed files
in this dedicated Simulator's own Media/DCIM directory, independently of the
app receipt. No personal phone Photos database was accessed. This is a
synthetic capture-file test, not proof of the headset screenshot-button path,
capture latency, physical-device behavior, or background delivery.
The OFF preference was applied through Simulator defaults on restart, not a
UI tap. The production setting setter and persisted opt-out were exercised in
the native tests. Astra inspected the real native sheet in both states and
found no major clipping, overlap or actionable visual defect.
![Photos enabled and app-only setting](phone-screenshot-sync-2026-10-24/settings-proof.png)
An initial unsigned test build could not retain a stable Keychain identity;
the real transport test used normal Simulator signing. App launches with a
console attached were kept alive for the live checks.
## Review provenance
Preferred Opus routes failed before work with API rate limits. Implementation
used the current GPT-6.1 Sol session; reviews are disclosed as fresh same-model
checks, not different-model evidence.
Provider `codex`, model `gpt-6.1-sol`, high, completed:
- r1: `node:delegated-task:command%3Amcp%3Ac03980f6-1e71-4cc4-af1a-1d0961b9aaad%3Adelegate-task%3Aframe-108-ios-sync-review-20261024-r1`.
Found stale export callback persistence and missing offline access. Both fixed.
- r2: `node:delegated-task:command%3Amcp%3Ac03980f6-1e71-4cc4-af1a-1d0961b9aaad%3Adelegate-task%3Aframe-108-ios-sync-review-20261024-r2`.
No actionable findings in the complete executable diff against `52772323`
plus the four new Swift files. Report: `/tmp/frame-108-ios-sync-review-r2.md`.
Visual inspection used provider `codex`, `gpt-6-astra`, completed task
`frame-108-downloads-proof-inspection-20261024-r3`; report:
`/tmp/frame-108-downloads-proof-inspection.md`.
## Remaining release requirements
- Actual APNs delivery, notification extension/shared state, persistent headset
event source and authenticated delivery service are still unimplemented.
Main-app push capability is not configured; no usable provider credential
location was found in the bounded checks. Existing App Store Connect signing
access is not an APNs provider key.
- Physical locked/background iPhone download and Photos export remain unproven.
- A new native Android companion was subsequently authorized and implemented.
See [Android phone](../android-phone.md) for its explicit foreground-service
behavior and [Android evidence](android-phone-screenshot-sync-2026-10-24.md)
for the build and emulator checks. This does not supply physical-device proof.
- No TestFlight upload, store submission or release was performed. The user's
release condition requires the full behavior to be tested and working.
Binary file not shown.

After

Width:  |  Height:  |  Size: 403 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 103 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

+171 -2
View File
@@ -57,6 +57,25 @@ makes.
| Sleep, restart, shut down ask in a terminal | The page asks for the Developer Mode password |
| Compatibility reports kept on the computer | Kept on the Frame (`~/.local/share/Frame Control`) |
## Automatic screenshot downloads (in development)
The screenshot-download implementation on this branch copies new Steam screenshots
while Frame Control is open and connected. It starts with the first successful
screenshot listing as its baseline; older captures still use the existing Save
action. Originals remain inside the app, and **Save screenshots to Photos** is
enabled by default, subject to add-only Photos permission.
Open **Downloads & Photos** on the Screenshots page, or **Screenshot downloads**
on the native connection screen. Turn the Photos setting off to keep new downloads
only in Frame Control. Re-enabling it does not export those app-only captures.
The local list and its Share buttons work without a headset connection. Failed
Photos exports retain the original and retry while the app is open; an interrupted
Photos transaction needs a manual check before saving again to avoid duplicates.
Background notification delivery and Android support are not implemented by this
change, and it has not been released. See the [verification record](evidence/phone-screenshot-sync-2026-10-24.md)
and [remaining work](https://github.com/saphid/frame-control/issues/108).
## Building
```sh
@@ -69,6 +88,88 @@ The build packs the Frame bundle from the checkout, so the phone always runs
the page and server from the same commit. Running on a phone needs your own
signing team in Xcode (Signing & Capabilities).
## TestFlight
Join the [Frame Control beta](https://testflight.apple.com/join/9xjW91g6),
install Apple's TestFlight app, then tap **Install** or **Update** next to Frame
Control. Internal account testers can also use their invitation. The first
working phone build was version 0.1.0, build 202610140516, uploaded on
2026-10-14. On 2026-10-16, build **202610160824** was uploaded and approved for
internal and external testing. App Store Connect confirmed both builds were in
the external **Public Beta** group, with its public link enabled and a limit of
50 testers. The new build declares the encryption-compliance answer in the app
and preserves the previous build's exact server bundle.
### Upload a build
You need Xcode, XcodeGen, a paid Apple Developer Program team, an App Store
Connect app record for `com.saphid.framecontrol`, and an App Store Connect team
API key with the signing access your release setup needs. This Mac relies on
Apple's cloud-managed distribution certificates: an **App Manager** key was
rejected with `Cloud signing permission error`, so this setup needs **Admin**
access. A setup with a local distribution signing identity may be able to use
a narrower role. Create the key under **Users and Access →
Integrations → App Store Connect API**. Download it directly to
`~/.appstoreconnect/private_keys/AuthKey_<KEY_ID>.p8`, outside the repository,
and restrict it to its owner (`chmod 600`). Apple allows one download. Keep
the private key out of chat, screenshots, logs and Git.
From the repository root:
```sh
ASC_KEY_ID=YOUR_KEY_ID \
ASC_ISSUER_ID=YOUR_ISSUER_UUID \
ASC_KEY_PATH="$HOME/.appstoreconnect/private_keys/AuthKey_YOUR_KEY_ID.p8" \
TEAM_ID=YOUR_TEAM_ID \
ios/scripts/testflight.sh
```
The script regenerates the Xcode project, archives with automatic signing,
checks the archived build number, and uploads to App Store Connect. `TEAM_ID`
defaults to this project's team, `NJZUMEA4BN`. `BUILD_NUMBER` defaults to UTC
`YYYYMMDDHHMM`; wait for a new minute between uploads, or supply a unique,
higher build number. The archive, export options and export output are under
`ios/build/testflight/<BUILD_NUMBER>/` (ignored by Git).
Build from a reviewed, clean checkout. The build packs the server, web pages
and helpers from the current checkout, including uncommitted changes; it does
not limit the bundle to the last commit.
An upload succeeding means Apple received the build. Wait for it to finish
processing in **App Store Connect → Frame Control → TestFlight** and check
that it is ready for testing. `ios/project.yml` declares
`ITSAppUsesNonExemptEncryption: false`, so each generated app declares that it
uses no non-exempt encryption, matching the previous builds' export-compliance
answers. Reassess that declaration if the app's cryptography changes.
To replace a key, create and securely download the replacement, verify that it
can access this app, then revoke the old key in Users and Access. Update the
upload environment to use the replacement. Revocation does not remove an
already uploaded TestFlight build from testers' phones.
### Add testers
**Internal:** add an existing App Store Connect user with an eligible role to
an internal testing group, and add the build to that group (or enable automatic
distribution). New account users must first accept their account invitation.
Internal testing grants App Store Connect access; use external testing for
people who only need to try the app.
**External:** create an external testing group, complete Beta App Information
and review contact details, add a build and its **What to Test** notes, then
submit it for TestFlight App Review when required. Use
<https://framecontrol.fun/privacy/> for the privacy policy and
<https://framecontrol.fun/feedback/> for feedback. Review notes must explain
that the app requires a Steam Frame with Developer Mode enabled and a user
password; do not provide your own headset credentials to reviewers. Apple may
need a way to evaluate features that require the hardware.
After Apple approves the build for external testing, enable the group's public
link and choose a tester limit. Share that link to let testers join without an
App Store Connect account. Creating a group alone does not make a build public.
Do not describe the link as ready until the approved build is in the group and
the link is enabled.
## Verified
<img src="img/iphone-tabs.jpg" alt="The four tabs in the iPhone app, connected to a Frame" width="900">
@@ -106,8 +207,76 @@ Sending text to the Frame's clipboard, or copying the Frame's to the phone
(**Type on the headset → Copy the headset's clipboard**), needs the desktop
panel open in the headset, as on the desktop app.
Not yet exercised: Android display changes through podman (no Android app was
running), a real sleep/restart/shut down on the Frame, and a physical iPhone.
On a physical iPhone, 2026-10-14: Alex reported build 202610140516 was working
after pairing and trying the password visibility button. A server process
started by the phone and its SSH key on the Frame confirmed the connection.
This was a setup/connection smoke check; it did not verify every app feature.
On a physical iPhone 16 Pro running iOS 27.2, 2026-10-16: the same build launched
and automatically reconnected to the Frame. A device screenshot showed the
green connection indicator, 100% headset battery and the loaded Android
catalog. This confirms connection and rendering, not status refresh or actions
inside the app. A separate signed automation runner was installed and passed
its health check. Initial interaction attempts were blocked by the phone's
lock; testing resumed once Alex unlocked it and kept the screen awake.
<img src="img/iphone-physical-20261016.jpg" alt="Frame Control connected to the Frame on a physical iPhone, showing battery and the Android catalog" width="360">
In that continuation, still on build **202610140516**, the status header
refreshed, the Games tab loaded 14 installed titles with sizes and Play
controls, and a uniquely named 114-byte text file was selected through the
iPhone's Files picker and sent to `~/Downloads` on the Frame. An SSH readback
matched the file's contents exactly; this verifies phone-to-Frame upload,
not a download back to the phone. Both the iCloud source fixture and the
uploaded test file were removed and checked absent.
Capture returned a blank headset frame with an asleep/unworn explanation.
Save opened the native share sheet and reached the Photos permission prompt;
no completed Photos save was verified. A final Settings check showed
**Add Photos Only** enabled, without photo-library read access. A test string
sent through the clipboard control was refused with the expected prerequisite:
`plasmashell` was not running, so the desktop panel must be opened in the
headset. The phone's existing clipboard was not read. Live mode started and
reported **30 fps**, but its visible frames remained blank, so meaningful
video playback is still unverified. Android listed 10 installed apps; a
separate SSH check found zero running containers and zero Android ADB
listeners, so display-setting changes were not exercised.
<img src="img/iphone-games-physical-20261016.jpg" alt="Frame Control's Games tab on the physical iPhone, showing the installed library and Play controls" width="360">
<img src="img/iphone-live-physical-20261016.jpg" alt="Live mode reporting 30 fps on the physical iPhone with blank headset content and the page horizontally clipped" width="360">
Testing also exposed an enlarged, horizontally clipped page after text entry
and navigation. Its cause has not yet been isolated. Restarting Frame Control
restored the normal layout and automatically reconnected to the headset.
### Mac follow-up
On 2026-10-17, the installed Mac app **0.4.6** was tested with the headset
unworn. Connection/status worked, and its connection test reported
`Answered in 90 ms · SSH works`. Still capture completed but both eyes were
blank. Live video failed with `Not a video capture device` when opening
`/dev/video99`, then fell back to blank still captures. No visible scene or
video playback was verified. The desktop was inactive; no headset services
or settings were changed. Live was stopped and the existing app left on Home.
These are Mac results; they do not verify the iPhone build or establish the
cause of its earlier blank output.
<img src="img/mac-live-error-20261017.png" alt="Mac Frame Control activity log reporting Live video failed: Not a video capture device" width="900">
### Physical-phone follow-up checklist
- With the headset awake and worn, capture visible headset content, save it to
Photos, then verify live video.
- Send test text to the Frame's clipboard with its desktop panel open.
- With an Android app running, change and restore its display settings.
- Reproduce and fix the page enlargement and horizontal clipping after text
entry and navigation.
- After saving headset work, deliberately test sleep, restart and shut down.
These remaining checks are pending on a physical phone. Android display
changes through podman and real sleep/restart/shut down also remain unverified
by the Simulator checks above.
Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`,
`FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds
+112
View File
@@ -0,0 +1,112 @@
# Mac space: the Mac's desktop and its windows, placed in the room
Design notes, started 2026-09-28. The goal is Apple Vision Pro's Mac Virtual
Display, plus something Vision Pro can't do: pulling any Mac window out
to float on its own. Labels as in [ssh.md](ssh.md): **verified**,
**documented**, **reported**, **inferred**.
## What to match (Vision Pro's Mac Virtual Display)
Researched 2026-09-28. Sources: Apple Support
[118521](https://support.apple.com/en-us/118521) and the
[Vision Pro guide](https://support.apple.com/guide/apple-vision-pro/use-mac-virtual-display-tan357ede966/visionos);
[9to5Mac](https://9to5mac.com/2024/01/30/heres-what-its-like-using-vision-pro-as-an-external-mac-display/),
[UploadVR](https://www.uploadvr.com/visionos-2-2-ultrawide-mac-virtual-display/)
and [AppleInsider](https://appleinsider.com/inside/apple-vision-pro/tips/how-to-use-mac-virtual-display-for-apple-vision-pro-with-desktop-macs-and-macbooks).
- **Connecting.** Look at the Mac and a **Connect** button floats above it.
There's also a Control Center route. The Mac's own screen goes dark
(**documented**/**reported**).
- **Placement.** The display appears where you were looking, near the Mac,
at a large-monitor size. After that it stays put in the room: moving the
laptop doesn't move it. Holding the Digital Crown recenters everything
(**documented**).
- **Handling.**
- A **window bar** under the display moves it, with pinch and drag, and
also moves it nearer or farther.
- Its bottom **corners** resize it.
- **Close** sits to the left of the bar (**documented**).
- **Modes.** Standard (16:9), Wide (21:9) and Ultrawide (32:9, about 10K
pixels across). All are curved since visionOS 2.2 (**documented**).
- **Input.**
- The Mac's keyboard and trackpad drive it.
- The pointer moves to whichever window you look at (Universal Control).
- Mac audio plays in the headset (**documented**).
- **What it can't do.** You can't pull a single Mac window out of the
virtual display (**reported**, several sources). That's our
differentiator.
## What the Frame allows
- **Moving SteamVR's panels from our own process: no** (**verified**
2026-09-28). `FindOverlay("valve.steam.desktopgame.<id>")` works from a
separate OpenVR app, but `SetOverlayTransformAbsolute` returns 12
(`VROverlayError_PermissionDenied`). The panel reported transform type 5
(dashboard tab). Only SteamVR's dashboard, which runs in `vrwebhelper`,
moves them, through its privileged `VRHTML.VROverlay`, and it has no
DevTools port. See `/tmp/fmv-space/ovprobe.py` for the probe, which stays
outside the repo.
- **WebXR in Chromium XR: yes** (**verified** 2026-09-26/27,
[webxr-chromium.md](webxr-chromium.md)).
- `immersive-vr` runs at 72–75 fps.
- Passthrough shows wherever the page leaves pixels transparent.
- Controllers are `tracked-pointer`s with `xr-standard` gamepads, and
squeeze events arrive.
- Hands are tracked with 25 joints.
- **Our own OpenVR overlays: possible, but most work.** We would own their
transforms and get laser input as mouse events for free. But the Frame
would need native H.264 decoding and texture upload.
## Decision: a WebXR "Mac space" page first
The WebXR page (`ui/mac-space.html`) holds everything, with passthrough
behind it:
- **The desktop.** A large curved panel shows the Mac's main display,
placed above the laptop. The first time, you place it yourself; after
that it's remembered in the room.
- **Windows.** Each separated window gets its own panel.
- **Grabbing.** A bar under each panel moves it (squeeze, or pinch with a
hand). Its corners resize it, and there's a close button to the bar's
left, as on Vision Pro.
- **Pulling a window out.** Grab a window's title bar on the desktop panel
and drag it off the edge. The page starts a `separate:<window id>`
stream, the window moves to its own display on the Mac, and its new
panel follows your hand until you let go. Dropping a panel back onto the
desktop panel stops the stream, and the window goes back.
- **Input.** The laser and trigger click and drag, the thumbstick scrolls,
and the Mac's keyboard types, as today.
Trade-offs:
- It's the headset's active app while you use it. Other VR apps pause, but
SteamVR's own overlays still draw on top.
- Text sharpness depends on how the page renders. WebXR quad layers (the
`layers` feature) would let SteamVR's compositor sample each panel
directly, so they're worth checking first.
## Building blocks
- **One module for the protocol.** `ui/mac-stream.js` holds the stream
protocol (connect, clock sync, decoding, acks and reports, input), shared
by `mac-view.html` (the 2D panel) and `mac-space.html`.
- **A space key from the agent.** Frame Control gets a ticket with
`src=space`, and the page trades it once for a **space key**, kept in
memory. That key lists the windows and displays (with their positions,
for picking the one under the pointer) and opens streams for any of
them. Stop All revokes it.
- **Anchoring.**
- v1: manual placement, remembered in the room (to be checked: whether
`local-floor` stays put between sessions on the Frame).
- v2: the Mac shows a marker on its screen for a moment, and the headset
finds it. That needs camera access, which is **unknown** on the Frame:
the passthrough cameras are monochrome, and `IVRTrackedCamera` hasn't
been tried.
## Open questions
- Frame rate with several software-decoded streams at once.
- Whether `local-floor` placements survive restarts.
- Whether WebXR quad layers work in Chromium XR's OpenXR path.
- Whether Chromium grants `requestSession` without a click in the page
(needed for a one-click "Open Mac space").
+20 -3
View File
@@ -1,11 +1,28 @@
# Tailscale: use the Frame from anywhere
# Tailscale: advanced remote-access setup
With Tailscale on the Frame, the `frame` SSH alias works off the home LAN, and
so does everything built on it: Frame Control, the scripts and the Mac app.
Tailscale is intended to make the `frame` SSH alias available off the home LAN,
including to Frame Control and the scripts. Off-LAN access has not yet been
verified; the checks below cover the direct LAN path only. Read the exposure
section before enabling it: services normally restricted to loopback become
reachable by other devices on your tailnet.
When the Mac and the Frame are on the same network, Tailscale connects them
directly, so there's no relay in the way (`tailscale ping frame` → `via
192.168.1.50:41641`, 20 ms).
These are source-checkout commands for macOS, not packaged-app menu actions
or Windows PowerShell commands. The connection script currently uses macOS
netcat options; this route is not supported on Linux. Install Git, OpenSSH,
zsh, Python 3 and curl on your Mac; the desktop app's bundled Python is not
available to these scripts. Then obtain the scripts:
```sh
git clone https://github.com/saphid/frame-control.git
cd frame-control
```
Enable Developer Mode on the awake headset and establish working SSH access
first using the [connection guide](frame-control.md). Run from the checkout:
```sh
scripts/tailscale-on-frame.sh # install or update, then approve the login URL
scripts/connect.sh frame.<tailnet>.ts.net # point the alias at Tailscale (the script prints this)
+24 -8
View File
@@ -7,13 +7,13 @@
objects = {
/* Begin PBXBuildFile section */
C1234567890ABCDEF0123456 /* NativeSessionTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = D1234567890ABCDEF0123456 /* NativeSessionTests.swift */; };
A1234567890ABCDEF0123456 /* NativeSessionState.swift in Sources */ = {isa = PBXBuildFile; fileRef = B1234567890ABCDEF0123456 /* NativeSessionState.swift */; };
0CF0B365CE5974753E8C3C2B /* ScreenshotJournal.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0D48FCBB02F0F2212C7DE3C1 /* ScreenshotJournal.swift */; };
0DEE50BD563B1D8C328C4C0A /* HeadsetServer.swift in Sources */ = {isa = PBXBuildFile; fileRef = EDC7BA8014DBC302D08FD397 /* HeadsetServer.swift */; };
12B21D3319BAF5AE79948560 /* FrameLink.swift in Sources */ = {isa = PBXBuildFile; fileRef = 16644E7FDA7ADD5B232EB700 /* FrameLink.swift */; };
1A07EC692B0FF723907EA77B /* WebShell.swift in Sources */ = {isa = PBXBuildFile; fileRef = D6C4E6C28315CA8729FCAAEA /* WebShell.swift */; };
41A697B9924018DA48F24A1F /* Keys.swift in Sources */ = {isa = PBXBuildFile; fileRef = 237D9AF04EEA257AB382F60E /* Keys.swift */; };
4622FE0F0D6499CD642C29A2 /* InstallLink.swift in Sources */ = {isa = PBXBuildFile; fileRef = BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */; };
46EF7595BE5B36664A11AEB5 /* NativeSessionTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8F1CAE943C2556A960890079 /* NativeSessionTests.swift */; };
476D8858DC2C9E6616B084BC /* PortForwarder.swift in Sources */ = {isa = PBXBuildFile; fileRef = A7F6ED116569D0ABABF6ED65 /* PortForwarder.swift */; };
765661DBC0E6798A27CC60DB /* RootView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9B24E1BCD4F69A24C7DEF02F /* RootView.swift */; };
78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */; };
@@ -21,8 +21,12 @@
9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */ = {isa = PBXBuildFile; fileRef = DB544223FC60A59CC3E8EF5F /* SetupView.swift */; };
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */; };
A8C7AED25A6280682FCE45DC /* Citadel in Frameworks */ = {isa = PBXBuildFile; productRef = 6BA549B6CC0A0CB847126456 /* Citadel */; };
C046F310BA857BCEC98BE00E /* ScreenshotDownloadsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5EBD10B73DF1E95F67A3B074 /* ScreenshotDownloadsView.swift */; };
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */; };
DFD89170E502260F005EEC73 /* ScreenshotDownloads.swift in Sources */ = {isa = PBXBuildFile; fileRef = 07617FD9B214626526FB1F03 /* ScreenshotDownloads.swift */; };
E2FD56BB5BC1D9AF5CDBF210 /* ScreenshotSyncTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = AB8610920C7D797E5963CF88 /* ScreenshotSyncTests.swift */; };
E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */; };
F0ECE06D5DD29F19CC41FDFB /* NativeSessionState.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9E8BE9A8C421AD40A0F4105B /* NativeSessionState.swift */; };
F94D0252F8CC5854314B84B2 /* AppModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8A11F3431826A26B247C0695 /* AppModel.swift */; };
/* End PBXBuildFile section */
@@ -37,19 +41,23 @@
/* End PBXContainerItemProxy section */
/* Begin PBXFileReference section */
D1234567890ABCDEF0123456 /* NativeSessionTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionTests.swift; sourceTree = "<group>"; };
B1234567890ABCDEF0123456 /* NativeSessionState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionState.swift; sourceTree = "<group>"; };
07617FD9B214626526FB1F03 /* ScreenshotDownloads.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotDownloads.swift; sourceTree = "<group>"; };
0D48FCBB02F0F2212C7DE3C1 /* ScreenshotJournal.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotJournal.swift; sourceTree = "<group>"; };
16644E7FDA7ADD5B232EB700 /* FrameLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameLink.swift; sourceTree = "<group>"; };
1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameControlTests.swift; sourceTree = "<group>"; };
237D9AF04EEA257AB382F60E /* Keys.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Keys.swift; sourceTree = "<group>"; };
2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameFinder.swift; sourceTree = "<group>"; };
5EBD10B73DF1E95F67A3B074 /* ScreenshotDownloadsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotDownloadsView.swift; sourceTree = "<group>"; };
6B5B6718C5EA77FA67F6B14C /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist; path = Info.plist; sourceTree = "<group>"; };
6FBC8D0B5ED7BF1C06F99892 /* FrameControlTests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = FrameControlTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
8A11F3431826A26B247C0695 /* AppModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppModel.swift; sourceTree = "<group>"; };
8F1CAE943C2556A960890079 /* NativeSessionTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionTests.swift; sourceTree = "<group>"; };
8F2CB550FC81C01E6BDD5A71 /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = "<group>"; };
93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameControlApp.swift; sourceTree = "<group>"; };
9B24E1BCD4F69A24C7DEF02F /* RootView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RootView.swift; sourceTree = "<group>"; };
9E8BE9A8C421AD40A0F4105B /* NativeSessionState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionState.swift; sourceTree = "<group>"; };
A7F6ED116569D0ABABF6ED65 /* PortForwarder.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PortForwarder.swift; sourceTree = "<group>"; };
AB8610920C7D797E5963CF88 /* ScreenshotSyncTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotSyncTests.swift; sourceTree = "<group>"; };
BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InstallLink.swift; sourceTree = "<group>"; };
D6C4E6C28315CA8729FCAAEA /* WebShell.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WebShell.swift; sourceTree = "<group>"; };
DB544223FC60A59CC3E8EF5F /* SetupView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SetupView.swift; sourceTree = "<group>"; };
@@ -104,9 +112,11 @@
isa = PBXGroup;
children = (
8A11F3431826A26B247C0695 /* AppModel.swift */,
B1234567890ABCDEF0123456 /* NativeSessionState.swift */,
93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */,
BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */,
9E8BE9A8C421AD40A0F4105B /* NativeSessionState.swift */,
07617FD9B214626526FB1F03 /* ScreenshotDownloads.swift */,
0D48FCBB02F0F2212C7DE3C1 /* ScreenshotJournal.swift */,
);
path = App;
sourceTree = "<group>";
@@ -116,7 +126,8 @@
children = (
E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */,
1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */,
D1234567890ABCDEF0123456 /* NativeSessionTests.swift */,
8F1CAE943C2556A960890079 /* NativeSessionTests.swift */,
AB8610920C7D797E5963CF88 /* ScreenshotSyncTests.swift */,
);
path = FrameControlTests;
sourceTree = "<group>";
@@ -125,6 +136,7 @@
isa = PBXGroup;
children = (
9B24E1BCD4F69A24C7DEF02F /* RootView.swift */,
5EBD10B73DF1E95F67A3B074 /* ScreenshotDownloadsView.swift */,
DB544223FC60A59CC3E8EF5F /* SetupView.swift */,
);
path = Views;
@@ -266,15 +278,18 @@
buildActionMask = 2147483647;
files = (
F94D0252F8CC5854314B84B2 /* AppModel.swift in Sources */,
A1234567890ABCDEF0123456 /* NativeSessionState.swift in Sources */,
78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */,
E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */,
12B21D3319BAF5AE79948560 /* FrameLink.swift in Sources */,
0DEE50BD563B1D8C328C4C0A /* HeadsetServer.swift in Sources */,
4622FE0F0D6499CD642C29A2 /* InstallLink.swift in Sources */,
41A697B9924018DA48F24A1F /* Keys.swift in Sources */,
F0ECE06D5DD29F19CC41FDFB /* NativeSessionState.swift in Sources */,
476D8858DC2C9E6616B084BC /* PortForwarder.swift in Sources */,
765661DBC0E6798A27CC60DB /* RootView.swift in Sources */,
DFD89170E502260F005EEC73 /* ScreenshotDownloads.swift in Sources */,
C046F310BA857BCEC98BE00E /* ScreenshotDownloadsView.swift in Sources */,
0CF0B365CE5974753E8C3C2B /* ScreenshotJournal.swift in Sources */,
9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */,
1A07EC692B0FF723907EA77B /* WebShell.swift in Sources */,
);
@@ -286,7 +301,8 @@
files = (
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */,
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */,
C1234567890ABCDEF0123456 /* NativeSessionTests.swift in Sources */,
46EF7595BE5B36664A11AEB5 /* NativeSessionTests.swift in Sources */,
E2FD56BB5BC1D9AF5CDBF210 /* ScreenshotSyncTests.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
+14
View File
@@ -22,6 +22,8 @@ final class AppModel: ObservableObject {
/// Native recovery remains reachable even when WebKit cannot draw a page.
@Published private(set) var webRecoveryMessage: String?
@Published private(set) var webReloadID = 0
let screenshots = ScreenshotDownloads()
@Published var showScreenshotDownloads = false
func webContentFailed(at url: URL, message: String) {
guard phase == .ready(url) else { return }
@@ -45,6 +47,7 @@ final class AppModel: ObservableObject {
let saved = UserDefaults.standard.data(forKey: Self.settingsKey).flatMap { try? JSONDecoder().decode(FrameSettings.self, from: $0) }
settings = saved
phase = saved == nil ? .setup : .connecting("Connecting")
if let identity = UserDefaults.standard.string(forKey: Self.hostKeyKey) { screenshots.load(identity: identity) }
}
var deviceName: String { UIDevice.current.userInterfaceIdiom == .pad ? "iPad" : "iPhone" }
@@ -146,6 +149,7 @@ final class AppModel: ObservableObject {
/// Every connection attempt has a number; anything that finishes after a newer
/// attempt started (or the user went back to setup) closes what it made and stops.
private func invalidate() {
screenshots.stop()
attempt += 1
retrying = false
webRecoveryMessage = nil
@@ -200,6 +204,7 @@ final class AppModel: ObservableObject {
}
#endif
phase = .ready(URL(string: page)!)
startScreenshots()
// Runs at once if it stopped in the moment since the check above.
server.whenExited { [weak self] tail in
Task { @MainActor in self?.lost(mine, "Frame Control on the headset stopped. \(tail.suffix(200))") }
@@ -254,6 +259,7 @@ final class AppModel: ObservableObject {
/// Called when the app comes back to the foreground: iOS may have dropped the
/// connection, or left it looking open, while it was in the background.
func resume() {
startScreenshots()
switch phase {
case .ready:
let mine = attempt
@@ -284,6 +290,7 @@ final class AppModel: ObservableObject {
}
private func teardown() async {
screenshots.stop()
forwarder?.stop()
forwarder = nil
server = nil
@@ -298,4 +305,11 @@ final class AppModel: ObservableObject {
_ = SecRandomCopyBytes(kSecRandomDefault, bytes.count, &bytes)
return bytes.map { String(format: "%02x", $0) }.joined()
}
func suspend() { screenshots.stop() }
private func startScreenshots() {
guard case .ready(let url) = phase, let hostKey else { return }
screenshots.start(url: url, identity: hostKey)
}
}
@@ -38,6 +38,7 @@ struct FrameControlApp: App {
}
.onChange(of: scenePhase) { _, phase in
if phase == .active { model.resume() }
else if phase == .background { model.suspend() }
}
}
}
@@ -0,0 +1,157 @@
import CryptoKit
import ImageIO
import Photos
import SwiftUI
import UIKit
@MainActor
final class ScreenshotDownloads: ObservableObject {
struct Item: Identifiable {
let id: String
let file: URL
let date: Date
let status: String
}
@Published private(set) var items: [Item] = []
@Published private(set) var message = "New screenshots download while Frame Control is open."
@Published private(set) var saveToPhotos: Bool
private let defaults: UserDefaults
private let storageRoot: URL?
private static let preference = "frame.screenshots.saveToPhotos"
private var engine: ScreenshotSyncEngine?
private var task: Task<Void, Never>?
private var session: URL?
private var identity: String?
init(defaults: UserDefaults = .standard, storageRoot: URL? = nil) {
self.defaults = defaults
self.storageRoot = storageRoot
saveToPhotos = defaults.object(forKey: Self.preference) == nil || defaults.bool(forKey: Self.preference)
}
func setPhotos(_ enabled: Bool) {
do {
try engine?.setPhotos(enabled)
defaults.set(enabled, forKey: Self.preference)
saveToPhotos = enabled
refresh()
if enabled { Task { _ = await Self.photoPermission() } }
} catch { message = "Couldn't save the screenshot setting: \(error.localizedDescription)" }
}
/// Local files and settings remain available without a live SSH session.
func load(identity: String) {
guard self.identity != identity || engine == nil else { return }
do {
let digest = SHA256.hash(data: Data(identity.utf8)).map { String(format: "%02x", $0) }.joined()
let base = try storageRoot ?? FileManager.default.url(for: .applicationSupportDirectory, in: .userDomainMask,
appropriateFor: nil, create: true)
self.engine = try ScreenshotSyncEngine(root: base.appendingPathComponent("Screenshots/\(digest)", isDirectory: true),
photos: saveToPhotos)
self.identity = identity
refresh()
} catch {
engine = nil
self.identity = nil
items = []
message = "Screenshot downloads need attention: \(error.localizedDescription)"
}
}
func start(url: URL, identity: String) {
if session == url, task != nil { return }
stop()
load(identity: identity)
if let engine {
session = url
task = Task { [weak self] in
while !Task.isCancelled {
guard let self, self.session == url else { return }
if UIApplication.shared.applicationState == .active {
await self.poll(url: url, engine: engine)
}
do { try await Task.sleep(nanoseconds: 4_000_000_000) }
catch { return }
}
}
}
}
func stop() {
task?.cancel()
task = nil
engine?.stop()
session = nil
}
private func poll(url: URL, engine: ScreenshotSyncEngine) async {
do {
let data = try await Self.get("/api/shots", from: url)
struct Listing: Decodable { let shots: [HeadsetScreenshot] }
let shots = try JSONDecoder().decode(Listing.self, from: data).shots
guard !Task.isCancelled, session == url else { return }
try await engine.reconcile(shots, download: { shot in
try await Self.get("/api/shots/image", id: shot.id, from: url)
}, validate: { data in
guard let image = CGImageSourceCreateWithData(data as CFData, nil),
CGImageSourceGetStatus(image) == .statusComplete else { return false }
return CGImageSourceCreateImageAtIndex(image, 0, nil) != nil
}, export: { file, allowed in
guard !Task.isCancelled, UIApplication.shared.applicationState == .active else { throw CancellationError() }
guard await Self.photoPermission() else {
throw NSError(domain: "FrameScreenshots", code: 1,
userInfo: [NSLocalizedDescriptionKey: "Originals are kept in Frame Control. Allow Photos access in iOS Settings to save them to Photos."])
}
// Cancellation or a toggle change while the permission prompt was
// visible must not start a Photos transaction.
guard !Task.isCancelled, allowed() else { throw CancellationError() }
try await PHPhotoLibrary.shared().performChanges {
PHAssetCreationRequest.forAsset().addResource(with: .photo, fileURL: file, options: nil)
}
})
guard !Task.isCancelled, session == url else { return }
message = "New screenshots download while Frame Control is open."
} catch {
guard !Task.isCancelled, session == url else { return }
message = "\(error.localizedDescription) Downloads retry while the app is open."
}
refresh()
}
private static func get(_ path: String, id: String? = nil, from page: URL) async throws -> Data {
let original = URLComponents(url: page, resolvingAgainstBaseURL: false)
let key = original?.queryItems?.first(where: { $0.name == "key" })?.value
var parts = original!
parts.path = path
parts.queryItems = id.map { [URLQueryItem(name: "id", value: $0)] }
parts.fragment = nil
var request = URLRequest(url: parts.url!, cachePolicy: .reloadIgnoringLocalCacheData, timeoutInterval: 30)
request.setValue(key, forHTTPHeaderField: "X-Frame-UI")
let (data, response) = try await URLSession.shared.data(for: request)
guard let http = response as? HTTPURLResponse, http.statusCode == 200 else { throw URLError(.badServerResponse) }
return data
}
private static func photoPermission() async -> Bool {
var status = PHPhotoLibrary.authorizationStatus(for: .addOnly)
if status == .notDetermined, UIApplication.shared.applicationState == .active {
status = await PHPhotoLibrary.requestAuthorization(for: .addOnly)
}
return status == .authorized || status == .limited
}
private func refresh() {
guard let engine else { return }
items = engine.journal.receipts.values.filter(\.downloaded).map { receipt in
let status: String
switch receipt.destination {
case .saved: status = "Saved to Photos"
case .appOnly: status = "In Frame Control"
case .pending: status = "In Frame Control · waiting for Photos"
case .committing: status = "In Frame Control · Photos save interrupted; check Photos before saving again"
}
return Item(id: receipt.shot.id, file: engine.file(for: receipt.shot.id),
date: Date(timeIntervalSince1970: receipt.shot.time), status: status)
}.sorted { $0.date > $1.date }
}
}
@@ -0,0 +1,151 @@
import Foundation
struct HeadsetScreenshot: Codable, Equatable {
let id: String
let size: Int
let time: Double
var valid: Bool {
id.range(of: #"^[0-9]{1,12}/[0-9]{1,20}/[0-9]{14}_[0-9]{1,4}\.(jpg|png)$"#, options: .regularExpression) != nil
&& size > 0 && size <= 30 * 1024 * 1024 && time.isFinite
}
}
/// All receipt state lives beside the originals, outside purgeable caches.
/// An interrupted Photos transaction is deliberately not retried automatically:
/// add-only access cannot check whether its asset was committed before a crash.
struct ScreenshotJournal: Codable {
enum Export: String, Codable { case pending, appOnly, committing, saved }
struct Receipt: Codable {
var shot: HeadsetScreenshot
var downloaded = false
var destination: Export
}
var baseline: Set<String>?
var receipts: [String: Receipt] = [:]
mutating func observe(_ shots: [HeadsetScreenshot], photos: Bool) {
let shots = shots.filter(\.valid)
guard let baseline else {
self.baseline = Set(shots.map(\.id))
return
}
for shot in shots where !baseline.contains(shot.id) {
if var receipt = receipts[shot.id] {
if !receipt.downloaded { receipt.shot = shot; receipts[shot.id] = receipt }
} else {
receipts[shot.id] = Receipt(shot: shot, destination: photos ? .pending : .appOnly)
}
}
}
mutating func keepPendingInApp() {
for id in receipts.keys where receipts[id]?.destination == .pending || receipts[id]?.destination == .committing {
receipts[id]?.destination = .appOnly
}
}
}
/// Serialized reconciler shared by foreground polling and future event delivery.
/// Network and Photos operations are injected so failures and late callbacks can
/// be exercised against the actual persistence and state transitions.
@MainActor
final class ScreenshotSyncEngine {
let root: URL
private(set) var journal: ScreenshotJournal
private var previous: [String: HeadsetScreenshot] = [:]
private var generation = UUID()
private var running = false
private(set) var saveToPhotos: Bool
init(root: URL, photos: Bool) throws {
self.root = root
saveToPhotos = photos
let ledger = root.appendingPathComponent("receipts.json")
if FileManager.default.fileExists(atPath: ledger.path) {
// Corrupt state must never silently become a fresh receipt ledger.
journal = try JSONDecoder().decode(ScreenshotJournal.self, from: Data(contentsOf: ledger))
} else { journal = ScreenshotJournal() }
if !photos { journal.keepPendingInApp() }
try persist()
}
func stop() { generation = UUID() }
func setPhotos(_ enabled: Bool) throws {
let oldJournal = journal
let oldSetting = saveToPhotos
saveToPhotos = enabled
if !enabled { journal.keepPendingInApp() }
do { try persist() }
catch { journal = oldJournal; saveToPhotos = oldSetting; throw error }
}
func file(for id: String) -> URL { root.appendingPathComponent(id) }
private func persist() throws {
try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true)
try JSONEncoder().encode(journal).write(to: root.appendingPathComponent("receipts.json"), options: .atomic)
}
func reconcile(_ shots: [HeadsetScreenshot],
download: (HeadsetScreenshot) async throws -> Data,
validate: (Data) -> Bool,
export: (URL, () -> Bool) async throws -> Void) async throws {
guard !running else { return }
running = true
defer { running = false }
let mine = generation
func current() -> Bool { mine == generation && !Task.isCancelled }
guard current() else { return }
let valid = shots.filter(\.valid)
let observed = Dictionary(valid.map { ($0.id, $0) }, uniquingKeysWith: { _, last in last })
let stable = previous
previous = observed
journal.observe(valid, photos: saveToPhotos)
try persist() // Queue before touching an original or the photo library.
var firstError: Error?
for id in journal.receipts.keys.sorted() {
guard current(), var receipt = journal.receipts[id], receipt.shot.valid else { continue }
do {
let file = file(for: id)
if !receipt.downloaded {
// Steam may still be writing. Require two equal observations.
guard observed[id] == receipt.shot, stable[id] == receipt.shot else { continue }
let data = try await download(receipt.shot)
guard current() else { return }
guard data.count == receipt.shot.size, validate(data) else {
throw CocoaError(.fileReadCorruptFile)
}
try FileManager.default.createDirectory(at: file.deletingLastPathComponent(), withIntermediateDirectories: true)
try data.write(to: file, options: .atomic)
// Re-read destination: the user can change it during download.
receipt = journal.receipts[id]!
receipt.downloaded = true
journal.receipts[id] = receipt
try persist()
}
if receipt.destination == .pending, saveToPhotos {
journal.receipts[id]?.destination = .committing
try persist()
do {
try await export(file, { current() && self.saveToPhotos && self.journal.receipts[id]?.destination == .committing })
} catch {
guard current() else { return }
// A reported transaction failure is safe to retry. A crash
// leaves committing, which needs explicit user recovery.
if journal.receipts[id]?.destination == .committing {
journal.receipts[id]?.destination = saveToPhotos ? .pending : .appOnly
}
try persist()
throw error
}
guard current() else { return }
journal.receipts[id]?.destination = .saved
try persist()
}
} catch { if firstError == nil { firstError = error } }
}
if let firstError { throw firstError }
}
}
+3 -1
View File
@@ -31,6 +31,8 @@
</array>
<key>CFBundleVersion</key>
<string>1</string>
<key>ITSAppUsesNonExemptEncryption</key>
<false/>
<key>LSApplicationQueriesSchemes</key>
<array>
<string>ssh</string>
@@ -50,7 +52,7 @@
<key>NSLocalNetworkUsageDescription</key>
<string>Frame Control finds your Steam Frame on your network and connects to it.</string>
<key>NSPhotoLibraryAddUsageDescription</key>
<string>Frame Control saves headset captures and screenshots to your photo library when you ask it to.</string>
<string>Frame Control automatically saves new headset screenshots to Photos when Save screenshots to Photos is enabled.</string>
<key>UILaunchScreen</key>
<dict>
<key>UIColorName</key>
+9
View File
@@ -39,6 +39,15 @@ struct RootView: View {
}
.preferredColorScheme(.dark)
.tint(.frameBlue)
.safeAreaInset(edge: .bottom) {
if case .ready = model.phase {} else {
Button("Screenshot downloads") { model.showScreenshotDownloads = true }
.padding().frame(maxWidth: .infinity).background(Color.frameBackground)
}
}
.sheet(isPresented: $model.showScreenshotDownloads) {
ScreenshotDownloadsView(downloads: model.screenshots)
}
}
}
@@ -0,0 +1,40 @@
import SwiftUI
struct ScreenshotDownloadsView: View {
@ObservedObject var downloads: ScreenshotDownloads
@Environment(\.dismiss) private var dismiss
var body: some View {
NavigationStack {
List {
Section {
Toggle("Save screenshots to Photos", isOn: Binding(get: { downloads.saveToPhotos }, set: downloads.setPhotos))
} footer: {
Text("Enabled by default. Turn off to keep new downloads only in Frame Control. Existing Photos are kept. App-only downloads stay app-only when this is turned on again.")
}
Section {
Text(downloads.message)
Text("Automatic downloads begin with new screenshots after the first connection. Keep the app open and connected; background delivery is not available yet.")
.foregroundStyle(.secondary)
}
Section("Downloaded from this headset") {
if downloads.items.isEmpty { Text("No new screenshots downloaded yet.").foregroundStyle(.secondary) }
ForEach(downloads.items) { item in
HStack {
VStack(alignment: .leading, spacing: 4) {
Text(item.date, format: .dateTime.month().day().hour().minute())
Text(item.status).font(.caption).foregroundStyle(.secondary)
}
Spacer()
ShareLink(item: item.file) { Image(systemName: "square.and.arrow.up") }
.accessibilityLabel("Share screenshot")
}
}
}
}
.navigationTitle("Screenshot downloads")
.navigationBarTitleDisplayMode(.inline)
.toolbar { ToolbarItem(placement: .confirmationAction) { Button("Done") { dismiss() } } }
}
}
}
+26 -3
View File
@@ -7,6 +7,7 @@ struct SetupView: View {
@ObservedObject var model: AppModel
@StateObject private var finder = FrameFinder()
@State private var password = ""
@State private var showPassword = false
@State private var manualHost = ""
@State private var user = "steamos"
@State private var showOther = false
@@ -92,13 +93,35 @@ struct SetupView: View {
// MARK: step 2
@ViewBuilder private var passwordStep: some View {
SecureField("Developer Mode password", text: $password)
HStack(spacing: 8) {
Group {
if showPassword {
TextField("Developer Mode password", text: $password)
} else {
SecureField("Developer Mode password", text: $password)
}
}
.textContentType(.password)
.textInputAutocapitalization(.never)
.autocorrectionDisabled()
.submitLabel(.go)
.focused($passwordFocused)
.onSubmit(connect)
.padding(12)
.background(Color.black.opacity(0.28), in: RoundedRectangle(cornerRadius: 10))
.padding(.vertical, 12)
Button {
showPassword.toggle()
passwordFocused = true // keep typing after the field swaps
} label: {
Image(systemName: showPassword ? "eye.slash" : "eye").foregroundStyle(Color.frameMuted)
.frame(width: 44, height: 44)
.contentShape(Rectangle())
}
.buttonStyle(.plain)
.accessibilityLabel(showPassword ? "Hide password" : "Show password")
}
.padding(.leading, 12)
.padding(.trailing, 2)
.background(Color.black.opacity(0.28), in: RoundedRectangle(cornerRadius: 10))
Text("Haven't set one? On the Frame: Steam Settings → Developer → Set User Password. It's only used now, to let this \(model.deviceName) in; it isn't saved.")
.font(.footnote).foregroundStyle(Color.frameMuted)
Button(action: connect) {
+4
View File
@@ -59,6 +59,7 @@ struct WebShell: UIViewRepresentable {
setUpConnection: () => call("setUpConnection"),
open: (what) => call("open", what),
saveImages: (images) => call("saveImages", images),
screenshotDownloads: () => call("screenshotDownloads"),
onInstallLink: (cb) => { installCb = cb; return call("installLinkReady", window.__frameDocumentID); },
};
window.__frameInstallLink = (req) => { if (!installCb) return false; installCb(req); return true; };
@@ -162,6 +163,9 @@ struct WebShell: UIViewRepresentable {
guard !images.isEmpty else { return replyHandler(nil, "No images to save") }
share(images)
replyHandler(["message": "Choose Save Image to keep \(images.count == 1 ? "it" : "them") in Photos"], nil)
case "screenshotDownloads":
model.showScreenshotDownloads = true
replyHandler(true, nil)
case "installLinkReady":
guard let id = arg as? String, let web,
let data = try? JSONSerialization.data(withJSONObject: id, options: .fragmentsAllowed),
@@ -0,0 +1,198 @@
import Foundation
import CryptoKit
import XCTest
@testable import Frame_Control
@MainActor
final class ScreenshotSyncTests: XCTestCase {
private let old = HeadsetScreenshot(id: "1/10/20261024010000_1.jpg", size: 3, time: 1)
private let new = HeadsetScreenshot(id: "1/10/20261024010001_1.jpg", size: 3, time: 2)
private func temporary() throws -> URL {
let root = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true)
addTeardownBlock { try? FileManager.default.removeItem(at: root) }
return root
}
func testBaselineStabilityAtomicOriginalAndRestartDedup() async throws {
let root = try temporary()
let engine = try ScreenshotSyncEngine(root: root, photos: true)
var downloads = 0, exports = 0
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in downloads += 1; return Data([1, 2, 3]) }
let save: (URL, () -> Bool) async throws -> Void = { file, allowed in
XCTAssertTrue(allowed())
XCTAssertEqual(try Data(contentsOf: file), Data([1, 2, 3]))
exports += 1
}
try await engine.reconcile([old], download: fetch, validate: { _ in true }, export: save)
try await engine.reconcile([old, new], download: fetch, validate: { _ in true }, export: save)
XCTAssertEqual(downloads, 0, "Do not import history or read a file still being written")
try await engine.reconcile([old, new, new], download: fetch, validate: { _ in true }, export: save)
XCTAssertEqual(downloads, 1)
XCTAssertEqual(exports, 1)
let restarted = try ScreenshotSyncEngine(root: root, photos: true)
try await restarted.reconcile([old, new], download: fetch, validate: { _ in true }, export: save)
XCTAssertEqual(downloads, 1)
XCTAssertEqual(exports, 1)
XCTAssertEqual(restarted.journal.receipts[new.id]?.destination, .saved)
}
func testInterruptedDownloadRetriesAndAppOnlyNeverExportsLater() async throws {
let root = try temporary()
let engine = try ScreenshotSyncEngine(root: root, photos: false)
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in Data([1, 2, 3]) }
let noExport: (URL, () -> Bool) async throws -> Void = { _, _ in XCTFail("App-only item exported") }
try await engine.reconcile([], download: fetch, validate: { _ in true }, export: noExport)
try await engine.reconcile([new], download: fetch, validate: { _ in true }, export: noExport)
do {
try await engine.reconcile([new], download: { _ in Data([1]) }, validate: { _ in true }, export: noExport)
XCTFail("Truncated original accepted")
} catch {}
XCTAssertFalse(FileManager.default.fileExists(atPath: engine.file(for: new.id).path))
let restarted = try ScreenshotSyncEngine(root: root, photos: true)
try await restarted.reconcile([new], download: fetch, validate: { _ in true }, export: noExport)
try await restarted.reconcile([new], download: fetch, validate: { _ in true }, export: noExport)
XCTAssertEqual(restarted.journal.receipts[new.id]?.destination, .appOnly)
XCTAssertTrue(restarted.journal.receipts[new.id]!.downloaded)
}
func testDestinationChangeDuringDownloadAndPermissionPrompt() async throws {
let engine = try ScreenshotSyncEngine(root: temporary(), photos: true)
let noExport: (URL, () -> Bool) async throws -> Void = { _, _ in XCTFail("Destination changed while downloading") }
try await engine.reconcile([], download: { _ in Data() }, validate: { _ in true }, export: noExport)
try await engine.reconcile([new], download: { _ in Data() }, validate: { _ in true }, export: noExport)
try await engine.reconcile([new], download: { _ in
try engine.setPhotos(false)
try engine.setPhotos(true)
return Data([1, 2, 3])
}, validate: { _ in true }, export: noExport)
XCTAssertEqual(engine.journal.receipts[new.id]?.destination, .appOnly)
let other = HeadsetScreenshot(id: "2/10/20261024010001_1.jpg", size: 3, time: 2)
try await engine.reconcile([new, other], download: { _ in Data() }, validate: { _ in true }, export: noExport)
do {
try await engine.reconcile([new, other], download: { _ in Data([1, 2, 3]) }, validate: { _ in true }, export: { _, allowed in
try engine.setPhotos(false)
try engine.setPhotos(true)
XCTAssertFalse(allowed(), "Re-enabling must not revive an export cancelled during consent")
throw CancellationError()
})
} catch {}
XCTAssertEqual(engine.journal.receipts[other.id]?.destination, .appOnly)
}
func testCancellationRejectsLateOriginalAndIdentityIncludesAccountAndApp() async throws {
let engine = try ScreenshotSyncEngine(root: temporary(), photos: false)
let variants = [new, HeadsetScreenshot(id: "2/10/20261024010001_1.jpg", size: 3, time: 2),
HeadsetScreenshot(id: "1/20/20261024010001_1.jpg", size: 3, time: 2)]
let save: (URL, () -> Bool) async throws -> Void = { _, _ in XCTFail("Unexpected export") }
try await engine.reconcile([], download: { _ in Data() }, validate: { _ in true }, export: save)
try await engine.reconcile(variants, download: { _ in Data() }, validate: { _ in true }, export: save)
try await engine.reconcile(variants, download: { _ in engine.stop(); return Data([1, 2, 3]) }, validate: { _ in true }, export: save)
XCTAssertEqual(engine.journal.receipts.count, 3)
XCTAssertTrue(engine.journal.receipts.values.allSatisfy { !$0.downloaded })
XCTAssertNotEqual(engine.file(for: variants[0].id), engine.file(for: variants[1].id))
XCTAssertNotEqual(engine.file(for: variants[0].id), engine.file(for: variants[2].id))
}
func testUncertainPhotosCommitIsNotDuplicatedAndCorruptLedgerFailsClosed() async throws {
let root = try temporary()
var journal = ScreenshotJournal()
journal.observe([], photos: true)
journal.observe([new], photos: true)
journal.receipts[new.id]?.downloaded = true
journal.receipts[new.id]?.destination = .committing
try JSONEncoder().encode(journal).write(to: root.appendingPathComponent("receipts.json"))
let engine = try ScreenshotSyncEngine(root: root, photos: true)
try await engine.reconcile([new], download: { _ in XCTFail(); return Data() }, validate: { _ in true }, export: { _, _ in XCTFail("Possible duplicate Photos asset") })
XCTAssertEqual(engine.journal.receipts[new.id]?.destination, .committing)
try Data("broken".utf8).write(to: root.appendingPathComponent("receipts.json"))
XCTAssertThrowsError(try ScreenshotSyncEngine(root: root, photos: true))
XCTAssertFalse(HeadsetScreenshot(id: "../../private.jpg", size: 3, time: 2).valid)
}
func testPhotosFailureRetainsOriginalAndRetriesOnlyExport() async throws {
let root = try temporary()
let engine = try ScreenshotSyncEngine(root: root, photos: true)
var downloads = 0, exports = 0
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in downloads += 1; return Data([1, 2, 3]) }
let denied: (URL, () -> Bool) async throws -> Void = { _, _ in throw CocoaError(.fileWriteNoPermission) }
try await engine.reconcile([], download: fetch, validate: { _ in true }, export: denied)
try await engine.reconcile([new], download: fetch, validate: { _ in true }, export: denied)
do {
try await engine.reconcile([new], download: fetch, validate: { _ in true }, export: denied)
XCTFail("Denied export reported success")
} catch {}
XCTAssertEqual(downloads, 1)
XCTAssertEqual(engine.journal.receipts[new.id]?.destination, .pending)
XCTAssertEqual(try Data(contentsOf: engine.file(for: new.id)), Data([1, 2, 3]))
let restarted = try ScreenshotSyncEngine(root: root, photos: true)
try await restarted.reconcile([], download: fetch, validate: { _ in true }, export: { _, allowed in
XCTAssertTrue(allowed()); exports += 1
})
XCTAssertEqual(downloads, 1)
XCTAssertEqual(exports, 1)
XCTAssertEqual(restarted.journal.receipts[new.id]?.destination, .saved)
}
func testLateExportSuccessAndFailureCannotOverwriteReplacementJournal() async throws {
for succeeds in [true, false] {
let root = try temporary()
let oldEngine = try ScreenshotSyncEngine(root: root, photos: true)
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in Data([1, 2, 3]) }
let noop: (URL, () -> Bool) async throws -> Void = { _, _ in }
try await oldEngine.reconcile([], download: fetch, validate: { _ in true }, export: noop)
try await oldEngine.reconcile([new], download: fetch, validate: { _ in true }, export: noop)
var suspended: CheckedContinuation<Void, Never>?
let task = Task {
try await oldEngine.reconcile([new], download: fetch, validate: { _ in true }, export: { _, allowed in
await withCheckedContinuation { suspended = $0 }
XCTAssertFalse(allowed(), "Stopped export must lose eligibility")
if !succeeds { throw CocoaError(.fileWriteNoPermission) }
})
}
while suspended == nil { await Task.yield() }
oldEngine.stop()
let replacement = try ScreenshotSyncEngine(root: root, photos: false)
let other = HeadsetScreenshot(id: "2/10/20261024010001_1.jpg", size: 3, time: 3)
try await replacement.reconcile([new, other], download: fetch, validate: { _ in true }, export: noop)
try await replacement.reconcile([new, other], download: fetch, validate: { _ in true }, export: noop)
suspended!.resume()
try await task.value
let reloaded = try ScreenshotSyncEngine(root: root, photos: true)
XCTAssertEqual(reloaded.journal.receipts.count, 2)
XCTAssertEqual(reloaded.journal.receipts[other.id]?.destination, .appOnly)
XCTAssertTrue(reloaded.journal.receipts[other.id]!.downloaded)
XCTAssertEqual(reloaded.journal.receipts[new.id]?.destination, .appOnly)
}
}
func testDefaultPreferenceAndOfflineAppOnlyLibraryAfterRelaunch() throws {
let root = try temporary()
let suite = "frame-screenshot-test-\(UUID().uuidString)"
let defaults = try XCTUnwrap(UserDefaults(suiteName: suite))
defer { defaults.removePersistentDomain(forName: suite) }
let downloads = ScreenshotDownloads(defaults: defaults, storageRoot: root)
XCTAssertTrue(downloads.saveToPhotos)
downloads.setPhotos(false)
let identity = "test-headset-pinned-key"
let digest = SHA256.hash(data: Data(identity.utf8)).map { String(format: "%02x", $0) }.joined()
let folder = root.appendingPathComponent("Screenshots/\(digest)")
try FileManager.default.createDirectory(at: folder, withIntermediateDirectories: true)
var ledger = ScreenshotJournal()
ledger.observe([], photos: false)
ledger.observe([new], photos: false)
ledger.receipts[new.id]?.downloaded = true
try JSONEncoder().encode(ledger).write(to: folder.appendingPathComponent("receipts.json"))
let file = folder.appendingPathComponent(new.id)
try FileManager.default.createDirectory(at: file.deletingLastPathComponent(), withIntermediateDirectories: true)
try Data([1, 2, 3]).write(to: file)
let restarted = ScreenshotDownloads(defaults: defaults, storageRoot: root)
XCTAssertFalse(restarted.saveToPhotos)
restarted.load(identity: identity) // No URL, connection or network task.
XCTAssertEqual(restarted.items.count, 1)
XCTAssertEqual(restarted.items[0].status, "In Frame Control")
XCTAssertEqual(try Data(contentsOf: restarted.items[0].file), Data([1, 2, 3]))
}
}
+2 -1
View File
@@ -34,6 +34,7 @@ targets:
path: FrameControl/Info.plist
properties:
CFBundleDisplayName: Frame Control
ITSAppUsesNonExemptEncryption: false
UILaunchScreen:
UIColorName: ""
UISupportedInterfaceOrientations: [UIInterfaceOrientationPortrait, UIInterfaceOrientationLandscapeLeft, UIInterfaceOrientationLandscapeRight]
@@ -41,7 +42,7 @@ targets:
UIUserInterfaceStyle: Dark
NSLocalNetworkUsageDescription: Frame Control finds your Steam Frame on your network and connects to it.
NSBonjourServices: [_steamos-devkit._tcp]
NSPhotoLibraryAddUsageDescription: Frame Control saves headset captures and screenshots to your photo library when you ask it to.
NSPhotoLibraryAddUsageDescription: Frame Control automatically saves new headset screenshots to Photos when Save screenshots to Photos is enabled.
NSAppTransportSecurity:
NSAllowsLocalNetworking: true
LSApplicationQueriesSchemes: [ssh, sftp, steamlink, rdp]
+60
View File
@@ -0,0 +1,60 @@
#!/bin/zsh
# Archive Frame Control and upload it to TestFlight.
#
# Needs a paid Apple Developer Program team and an App Store Connect API key
# (Users and Access → Integrations → App Store Connect API). This Mac needs
# Admin access for Apple's cloud-managed distribution signing; App Manager
# alone was rejected with "Cloud signing permission error". See docs/iphone.md.
#
# ASC_KEY_ID=ABC123 ASC_ISSUER_ID=uuid ASC_KEY_PATH=~/.appstoreconnect/AuthKey_ABC123.p8 \
# ios/scripts/testflight.sh
#
# TEAM_ID defaults to the team in this Mac's signing certificate. The build
# number is the current UTC time, so every upload is higher than the last.
set -euo pipefail
: "${ASC_KEY_ID:?set ASC_KEY_ID}" "${ASC_ISSUER_ID:?set ASC_ISSUER_ID}" "${ASC_KEY_PATH:?set ASC_KEY_PATH}"
TEAM_ID="${TEAM_ID:-NJZUMEA4BN}"
BUILD_NUMBER="${BUILD_NUMBER:-$(date -u +%Y%m%d%H%M)}"
# Resolve the key before changing directory; expand a quoted leading ~/.
key_path="${ASC_KEY_PATH/#\~\//$HOME/}"
key_path="${key_path:A}"
[[ -f "$key_path" ]] || { echo "no API key at $key_path" >&2; exit 1; }
cd "${0:A:h}/.."
out="build/testflight/$BUILD_NUMBER"
mkdir -p "$out"
auth=(-allowProvisioningUpdates
-authenticationKeyPath "$key_path"
-authenticationKeyID "$ASC_KEY_ID"
-authenticationKeyIssuerID "$ASC_ISSUER_ID")
xcodegen generate
xcodebuild archive -project FrameControl.xcodeproj -scheme FrameControl \
-configuration Release -destination generic/platform=iOS \
-archivePath "$out/FrameControl.xcarchive" \
DEVELOPMENT_TEAM="$TEAM_ID" CODE_SIGN_STYLE=Automatic \
CURRENT_PROJECT_VERSION="$BUILD_NUMBER" "${auth[@]}"
# Check the archived build number, not just the argument passed.
archived=$(plutil -extract CFBundleVersion raw \
"$out/FrameControl.xcarchive/Products/Applications/Frame Control.app/Info.plist")
[[ "$archived" == "$BUILD_NUMBER" ]] || { echo "archived build $archived, expected $BUILD_NUMBER" >&2; exit 1; }
cat > "$out/ExportOptions.plist" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
<key>method</key><string>app-store-connect</string>
<key>destination</key><string>upload</string>
<key>teamID</key><string>$TEAM_ID</string>
<key>signingStyle</key><string>automatic</string>
<key>manageAppVersionAndBuildNumber</key><false/>
</dict></plist>
EOF
xcodebuild -exportArchive -archivePath "$out/FrameControl.xcarchive" \
-exportOptionsPlist "$out/ExportOptions.plist" -exportPath "$out/export" "${auth[@]}"
echo "Uploaded build $BUILD_NUMBER. It shows in TestFlight once App Store Connect finishes processing."
+94 -8
View File
@@ -18,11 +18,15 @@ final class VirtualDisplay {
static let refreshRate = Double(ProcessInfo.processInfo.environment["FRAME_MAC_VIEW_VD_HZ"] ?? "") ?? 60
private var display: CGVirtualDisplay?
let id: CGDirectDisplayID
/// The size asked for, in points (the display starts in another mode and
/// switches a moment later).
let size: (width: Int, height: Int)
/// Removal is deferred while the Mac's screen sleeps, and an identity
/// can't be reused until it's gone, so each display gets a fresh serial.
private static var serial = (UInt32(truncatingIfNeeded: ProcessInfo.processInfo.processIdentifier) & 0xFFFF) << 12
init?(name: String, width: Int, height: Int) {
size = (width, height)
var made: CGVirtualDisplay?
for _ in 0..<8 where made == nil {
VirtualDisplay.serial &+= 1
@@ -40,6 +44,9 @@ final class VirtualDisplay {
let s = CGVirtualDisplaySettings()
s.hiDPI = 1
let hz = VirtualDisplay.refreshRate
// Both modes: macOS then starts the display at 2x by itself, as long
// as it's at least 800x600 points; smaller, its 2x mode is never
// usable (probed 2026-09-28 on macOS 26.5.2).
s.modes = [CGVirtualDisplayMode(width: UInt32(2 * width), height: UInt32(2 * height), refreshRate: hz)!,
CGVirtualDisplayMode(width: UInt32(width), height: UInt32(height), refreshRate: hz)!]
if vd.apply(s), vd.displayID != 0 { made = vd }
@@ -47,23 +54,89 @@ final class VirtualDisplay {
guard let made else { return nil }
display = made
id = made.displayID
// Pick the HiDPI mode: `width` points drawn with twice the pixels.
}
/// Where it should sit: left of everything else, touching the leftmost
/// display. Otherwise macOS slots each new display in next to the main
/// one and pushes earlier ones along, and a window placed a moment before
/// ends up on the wrong display (seen 2026-09-28 with two windows).
private var wantOrigin: CGPoint?
private var modesSeen = -1
private var modeAskedAt = Date.distantPast
/// Applies the HiDPI mode and the position once macOS has registered the
/// display (right after creation both fail: no modes yet, and error 1001
/// for the origin). True once both are in place. Call until it is.
func settle() -> Bool {
if wantOrigin == nil {
var ids = [CGDirectDisplayID](repeating: 0, count: 32)
var n: UInt32 = 0
CGGetOnlineDisplayList(32, &ids, &n)
let others = ids.prefix(Int(n)).filter { $0 != id }.map { CGDisplayBounds($0) }
let left = others.map(\.minX).min() ?? 0
let top = others.first(where: { $0.minX == left })?.minY ?? 0
wantOrigin = CGPoint(x: left - CGFloat(size.width), y: top)
}
let placed = bounds.origin == wantOrigin
if isReady && placed { return true }
let (width, height) = size
// The HiDPI mode: `width` points drawn with twice the pixels.
let modes = (CGDisplayCopyAllDisplayModes(id, [kCGDisplayShowDuplicateLowResolutionModes: true] as CFDictionary)
as? [CGDisplayMode]) ?? []
if let hidpi = modes.first(where: {
let hidpi = modes.first(where: {
$0.width == width && $0.height == height && $0.pixelWidth == 2 * width && $0.refreshRate == VirtualDisplay.refreshRate
}) ?? modes.first(where: { $0.width == width && $0.height == height && $0.pixelWidth == 2 * width }) {
}) ?? modes.first(where: { $0.width == width && $0.height == height && $0.pixelWidth == 2 * width })
// Separately: a refused position would take the mode change down with it.
// A mode switch takes a moment; asking again restarts it.
if !isReady, let hidpi, Date().timeIntervalSince(modeAskedAt) > 1.5 {
modeAskedAt = Date()
let r = CGDisplaySetDisplayMode(id, hidpi, nil) // only if macOS didn't start it at 2x
agentLog("display \(id): set \(hidpi.width)x\(hidpi.height)@\(hidpi.pixelWidth) -> \(r.rawValue), "
+ "usable \(hidpi.isUsableForDesktopGUI()), now \(CGDisplayCopyDisplayMode(id).map { "\($0.width)x\($0.height)@\($0.pixelWidth)" } ?? "?")")
}
if !placed, let o = wantOrigin {
var cfg: CGDisplayConfigRef?
CGBeginDisplayConfiguration(&cfg)
CGConfigureDisplayWithDisplayMode(cfg, id, hidpi, nil)
CGConfigureDisplayOrigin(cfg, id, Int32(o.x), Int32(o.y))
CGCompleteDisplayConfiguration(cfg, .forSession)
}
if modesSeen != modes.count {
modesSeen = modes.count
let cur = CGDisplayCopyDisplayMode(id).map { "\($0.width)x\($0.height)@\($0.pixelWidth)" } ?? "?"
agentLog("display \(id): \(modes.count) modes [\(modes.map { "\($0.width)x\($0.height)@\($0.pixelWidth)\($0.isUsableForDesktopGUI() ? "u" : "")" }.joined(separator: " "))], "
+ "hidpi \(hidpi.map { "\($0.width)x\($0.height)@\($0.pixelWidth)" } ?? "none"), current \(cur), now \(bounds)")
}
return false
}
var bounds: CGRect { CGDisplayBounds(id) }
/// Moves several of our displays at once (one configuration, so they
/// never land on each other half way), retrying for a second until
/// macOS has them all where asked.
static func arrange(_ origins: [(VirtualDisplay, CGPoint)], attempt: Int = 0) {
let pending = origins.filter { $0.0.bounds.origin != $0.1 }
guard !pending.isEmpty else { return }
for (vd, p) in origins { vd.wantOrigin = p }
var cfg: CGDisplayConfigRef?
CGBeginDisplayConfiguration(&cfg)
for (vd, p) in origins { CGConfigureDisplayOrigin(cfg, vd.id, Int32(p.x), Int32(p.y)) }
CGCompleteDisplayConfiguration(cfg, .forSession)
guard attempt < 10 else {
agentLog("arrange: \(pending.count) displays not where asked")
return
}
DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) { arrange(origins, attempt: attempt + 1) }
}
var isHiDPI: Bool { CGDisplayCopyDisplayMode(id).map { $0.pixelWidth > $0.width } ?? false }
/// In the mode asked for: HiDPI at the requested size.
var isReady: Bool {
guard let m = CGDisplayCopyDisplayMode(id) else { return false }
return m.width == size.width && m.height == size.height && m.pixelWidth == 2 * size.width
}
var screen: NSScreen? {
NSScreen.screens.first {
($0.deviceDescription[NSDeviceDescriptionKey("NSScreenNumber")] as? NSNumber)?.uint32Value == id
@@ -150,6 +223,7 @@ final class SeparateSource: CaptureSource {
var frameRect: CGRect { inner?.frameRect ?? .zero }
var displayID: CGDirectDisplayID? { display?.id }
var virtualDisplay: VirtualDisplay? { display }
func start(maxLong: Int, fps: Int, completion: @escaping (String?) -> Void) {
guard AXIsProcessTrusted() else {
@@ -170,14 +244,18 @@ final class SeparateSource: CaptureSource {
if w.isFullScreen { return completion("take the window out of full screen first") }
// A display the window's size, plus room for the menu bar, in
// points; within what a panel can show sharply.
let width = min(max(Int(frame.width.rounded()), 640), 2560) / 2 * 2
let height = min(max(Int(frame.height.rounded()) + 40, 480), 1600) / 2 * 2
// At least 800x600 points: smaller displays don't get a usable 2x
// (sharp) mode on macOS 26. A smaller window is resized to fill it,
// or cropped if it keeps its own size.
let width = min(max(Int(frame.width.rounded()), 800), 2560) / 2 * 2
let height = min(max(Int(frame.height.rounded()) + 40, 600), 1600) / 2 * 2
guard let vd = VirtualDisplay(name: app.isEmpty ? "Frame Control" : "\(app) (Frame)", width: width, height: height) else {
return completion("macOS wouldn't create a display for this window")
}
display = vd
window = w
original = frame
agentLog("separate \(windowID) (\(app)): display \(vd.id) \(width)x\(height) pt for window \(frame)")
// The new screen shows up in NSScreen a moment later; its menu bar
// decides where the window can go.
placeWindow(attempt: 0, maxLong: maxLong, fps: fps, completion: completion)
@@ -188,14 +266,22 @@ final class SeparateSource: CaptureSource {
guard !stopped, let vd = display, let w = window else { return }
// Wait for the screen to appear, and for its HiDPI mode, so the
// first frames are captured sharp.
if vd.screen == nil || !vd.isHiDPI, attempt < 30 {
if !vd.settle() || vd.screen == nil, attempt < 30 {
DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
self.placeWindow(attempt: attempt + 1, maxLong: maxLong, fps: fps, completion: completion)
}
return
}
let target = vd.usableRect
let target = vd.usableRect // read now: displays may have moved meanwhile
w.setFrame(target)
if let now = w.frame, !vd.bounds.contains(CGPoint(x: now.midX, y: now.minY + 10)), attempt < 30 {
// The arrangement changed under us: try again in a moment.
DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
self.placeWindow(attempt: attempt + 1, maxLong: maxLong, fps: fps, completion: completion)
}
return
}
agentLog("separate \(windowID): screen after \(attempt) waits, ready \(vd.isReady), display \(vd.bounds), target \(target), window now \(w.frame.map { "\($0)" } ?? "?")")
guard let now = w.frame, vd.bounds.intersects(now) else {
restore()
return completion("macOS didn't let the window move to its own display (Stage Manager can prevent this)")
+102 -6
View File
@@ -36,8 +36,8 @@ func windowsJSON() -> [[String: Any]] {
$0.layer == 0 && $0.pid != me && !skip.contains($0.app) && $0.bounds.width >= 120 && $0.bounds.height >= 80
}.map {
["id": $0.id, "src": "window:\($0.id)", "pid": $0.pid, "app": $0.app, "title": $0.title,
"w": Int($0.bounds.width), "h": Int($0.bounds.height)]
}
"w": Int($0.bounds.width), "h": Int($0.bounds.height), "x": Int($0.bounds.minX), "y": Int($0.bounds.minY)]
} // front to back, as CGWindowList gives them
}
func displaysJSON() -> [[String: Any]] {
@@ -50,7 +50,7 @@ func displaysJSON() -> [[String: Any]] {
}.map { id in
let b = CGDisplayBounds(id)
return ["id": id, "src": "display:\(id)", "name": displayName(id), "w": Int(b.width), "h": Int(b.height),
"main": CGDisplayIsMain(id) != 0]
"x": Int(b.minX), "y": Int(b.minY), "main": CGDisplayIsMain(id) != 0]
}
}
@@ -106,6 +106,8 @@ final class Session {
var onEnd: ((Session) -> Void)?
var onAck: (() -> Void)?
var onFinished: ((String) -> Void)?
/// The Mac space's layout in the room (message "arrange").
var onArrange: (([[String: Any]]) -> Void)?
private let encodeQueue = DispatchQueue(label: "frame-mac-view.encode", qos: .userInteractive)
init(id: Int, source: Source, capture: CaptureSource, ws: WebSocket, codec: Codec, fps: Int, bitsPerPixel: Double,
@@ -245,6 +247,7 @@ final class Session {
capture.onFrame = { [weak self] pb, pts, shown in self?.offer(pb, pts: pts, capture: shown) }
capture.onEnded = { [weak self] reason in
guard let self else { return }
agentLog("stream \(self.id) \(self.source.key) ended: \(reason)")
self.ws.sendJSON(["t": "closed", "reason": reason])
self.onFinished?(self.source.key)
self.end()
@@ -275,8 +278,12 @@ final class Session {
}
t.resume()
statsTimer = t
agentLog("stream \(id) \(source.key) starting")
capture.start(maxLong: maxLong, fps: fps) { [weak self] error in
guard let self else { return }
agentLog("stream \(self.id) \(self.source.key) " + (error.map { "failed: \($0) (gone: \(self.capture.gone))" }
?? "capturing \(Int(self.capture.frameRect.width))x\(Int(self.capture.frameRect.height)) pt"
+ ((self.capture as? SeparateSource)?.displayID.map { " on display \($0)" } ?? "")))
if let error {
if self.capture.gone {
// Final, like a window closing mid-stream: the viewer closes
@@ -389,6 +396,10 @@ final class Session {
guard !isStopped, let d = text.data(using: .utf8),
let m = try? JSONSerialization.jsonObject(with: d) as? [String: Any], let t = m["t"] as? String else { return }
if report(t, m) { return }
if t == "arrange", let list = m["panels"] as? [[String: Any]] {
onArrange?(list)
return
}
let x = m["x"] as? Double ?? -1, y = m["y"] as? Double ?? -1
if t == "ack" {
onAck?()
@@ -438,13 +449,22 @@ final class Session {
case "release":
Input.releaseAll(owner: id)
case "focus":
// Before keys meant for this window (the Mac space's text-size buttons).
if case .window(let wid) = source, let pid = capture.pid { Input.focus(window: wid, pid: pid) }
if case .separate(let wid) = source, let pid = capture.pid { Input.focus(window: wid, pid: pid) }
default: break
}
}
}
}
/// One line to stderr (the agent's log), with the time.
func agentLog(_ s: String) {
let f = DateFormatter()
f.dateFormat = "HH:mm:ss.SSS"
FileHandle.standardError.write(Data("\(f.string(from: Date())) \(s)\n".utf8))
}
func randomKey() -> String {
var bytes = [UInt8](repeating: 0, count: 24)
_ = SecRandomCopyBytes(kSecRandomDefault, bytes.count, &bytes)
@@ -472,11 +492,47 @@ final class Agent {
private var tickets: [String: (src: String, expiry: Date, key: String?)] = [:]
/// reconnect key -> source, until Stop for that source.
private var reconnectKeys: [String: String] = [:]
/// Space keys (ui/mac-space.html): each lists the Mac's windows and
/// displays and opens streams for any of them, until Stop All (or Stop
/// for "space"). Handed out once, for a ticket for "space".
private var spaceKeys = Set<String>()
/// Sources that ended for good (the window closed), for Frame Control.
private var finished = Set<String>()
/// Ends every stream, then exits once separated windows are back on the
/// Mac's own screens (they're moved back, then their displays go 0.3 s later).
/// Lays the separated windows' displays out around the main display the
/// way their panels sit around the desktop panel in the room, so the
/// Mac's own pointer crosses from one to the next where the panels are
/// next to each other. `list`: [{src, x}], x in metres from the desktop
/// panel's centre, right positive.
func arrange(_ list: [[String: Any]]) {
lock.lock()
let bySrc = Dictionary(sessions.values.map { ($0.source.key, $0) }, uniquingKeysWith: { a, _ in a })
lock.unlock()
var sources: [(SeparateSource, Double)] = []
for e in list {
guard let src = e["src"] as? String, let x = num(e["x"]),
let sep = bySrc[src]?.capture as? SeparateSource else { continue }
sources.append((sep, x))
}
DispatchQueue.main.async { // Separate mode's state lives on main
let items: [(vd: VirtualDisplay, x: Double)] = sources.compactMap { sep, x in sep.virtualDisplay.map { (vd: $0, x: x) } }
let main = CGDisplayBounds(CGMainDisplayID())
var origins: [(VirtualDisplay, CGPoint)] = []
var right = main.maxX, left = main.minX
for (vd, _) in items.filter({ $0.x >= 0 }).sorted(by: { $0.x < $1.x }) {
origins.append((vd, CGPoint(x: right, y: main.minY)))
right += CGFloat(vd.size.width)
}
for (vd, _) in items.filter({ $0.x < 0 }).sorted(by: { $0.x > $1.x }) {
left -= CGFloat(vd.size.width)
origins.append((vd, CGPoint(x: left, y: main.minY)))
}
VirtualDisplay.arrange(origins)
}
}
/// Idempotent: SIGTERM and the parent going away can both call it, and a
/// session already ended by /close may still be putting its window back.
private var quitting = false
@@ -524,9 +580,26 @@ final class Agent {
return key
}
if let r = req.query["r"], reconnectKeys[r] == src { return r }
if let s = req.query["s"], spaceKeys.contains(s) {
let key = randomKey()
reconnectKeys[key] = src
return key
}
return nil
}
private func isSpace(_ req: Request) -> Bool {
lock.lock(); defer { lock.unlock() }
return spaceKeys.contains(req.query["s"] ?? "")
}
/// A page next to the viewer (the Mac space and its shared script); no secrets in them.
private func sibling(_ name: String, _ type: String, _ c: HTTPConnection) {
guard let page, let body = try? Data(contentsOf: page.deletingLastPathComponent().appendingPathComponent(name))
else { return c.respond(404, text: "not found") }
c.respond(200, body: body, type: type)
}
/// The viewer has its reconnect key, so the ticket that led to it can't
/// be used again (also when the ack comes over a reconnection).
func acknowledged(key: String) {
@@ -541,6 +614,25 @@ final class Agent {
case ("GET", "/view"):
guard let page, let body = try? Data(contentsOf: page) else { return c.respond(404, text: "no viewer page") }
return c.respond(200, body: body, type: "text/html; charset=utf-8")
case ("GET", "/space"): return sibling("mac-space.html", "text/html; charset=utf-8", c)
case ("GET", "/mac-stream.js"): return sibling("mac-stream.js", "text/javascript; charset=utf-8", c)
case ("GET", "/space/key"):
// A ticket for "space", spent here: the page keeps the key in memory.
lock.lock()
let now = Date()
tickets = tickets.filter { $0.value.expiry > now }
guard let t = req.query["t"], let entry = tickets[t], entry.src == "space" else {
lock.unlock()
return c.respond(403, text: "forbidden")
}
tickets[t] = nil
let key = randomKey()
spaceKeys.insert(key)
lock.unlock()
return c.respond(json: ["key": key])
case ("GET", "/space/windows"):
guard isSpace(req) else { return c.respond(403, text: "forbidden") }
return c.respond(json: ["windows": windowsJSON(), "displays": displaysJSON(), "input": Input.allowed])
case ("GET", "/stream"):
guard let src = Source(req.query["src"] ?? "") else { return c.respond(400, text: "bad src") }
guard let key = mayStream(req, src: src.key) else { return c.respond(403, text: "forbidden") }
@@ -571,11 +663,13 @@ final class Agent {
case ("GET", "/displays"):
c.respond(json: ["displays": displaysJSON()])
case ("POST", "/ticket"):
guard let src = Source(req.query["src"] ?? "") else { return c.respond(400, text: "bad src") }
let isSpaceTicket = req.query["src"] == "space"
guard isSpaceTicket || Source(req.query["src"] ?? "") != nil else { return c.respond(400, text: "bad src") }
let key = isSpaceTicket ? "space" : Source(req.query["src"] ?? "")!.key
let t = randomKey()
lock.lock()
tickets[t] = (src.key, Date().addingTimeInterval(60), nil)
finished.remove(src.key)
tickets[t] = (key, Date().addingTimeInterval(60), nil)
finished.remove(key)
lock.unlock()
c.respond(json: ["ticket": t])
case ("POST", "/close"):
@@ -586,6 +680,7 @@ final class Agent {
let matching = sessions.values.filter { src == nil || $0.source.key == src }
reconnectKeys = reconnectKeys.filter { src != nil && $0.value != src }
tickets = tickets.filter { src != nil && $0.value.src != src } // not yet used ones too
if src == nil || src == "space" { spaceKeys.removeAll() }
lock.unlock()
for s in matching { s.ws.sendJSON(["t": "close"]) }
DispatchQueue.global().asyncAfter(deadline: .now() + 0.3) { for s in matching { s.end() } }
@@ -660,6 +755,7 @@ final class Agent {
self.lock.unlock()
}
session.onAck = { [weak self] in self?.acknowledged(key: key) }
session.onArrange = { [weak self] list in self?.arrange(list) }
session.start(maxLong: maxLong, fps: fps)
}
}
+2 -1
View File
@@ -1,6 +1,7 @@
# Website
The Frame Control website, <https://frame-control.pages.dev>, on Cloudflare Pages.
The Frame Control website, <https://framecontrol.fun>, on Cloudflare Pages.
The Pages project is `frame-control`; its fallback hostname is `frame-control.pages.dev`.
- `public/`: static pages. `/` is the landing page, `/feedback/` the feedback form, `/privacy/` the privacy note.
- `functions/api/feedback.js`: `POST /api/feedback`, which turns the form into a GitHub issue labelled `feedback`.
+30 -19
View File
@@ -25,12 +25,16 @@ async function overLimit(kv, key, limit, ttl) {
export async function onRequestPost({ request, env }) {
if (!env.GITHUB_TOKEN || !env.GITHUB_REPO) {
return json(503, { error: "Feedback isn't connected to GitHub yet. Use the GitHub link instead." });
return json(503, { error: "Feedback isn't connected to GitHub yet. Use the GitHub link instead.", retrySafe: true });
}
const origin = request.headers.get("origin");
if (origin && new URL(origin).host !== new URL(request.url).host) {
return json(403, { error: "Send feedback from the website's form." });
if (origin) {
try {
if (new URL(origin).origin !== new URL(request.url).origin) throw new Error("Origin mismatch");
} catch {
return json(403, { error: "Send feedback from the website's form." });
}
}
let input;
@@ -63,24 +67,31 @@ export async function onRequestPost({ request, env }) {
console.log(`Rate limit check failed: ${err}`);
}
const res = await fetch(`https://api.github.com/repos/${env.GITHUB_REPO}/issues`, {
method: "POST",
headers: {
authorization: `Bearer ${env.GITHUB_TOKEN}`,
accept: "application/vnd.github+json",
"x-github-api-version": "2022-11-28",
"user-agent": "frame-control-website",
"content-type": "application/json",
},
body: JSON.stringify(buildIssue(checked.value)),
});
try {
const res = await fetch(`https://api.github.com/repos/${env.GITHUB_REPO}/issues`, {
method: "POST",
headers: {
authorization: `Bearer ${env.GITHUB_TOKEN}`,
accept: "application/vnd.github+json",
"x-github-api-version": "2022-11-28",
"user-agent": "frame-control-website",
"content-type": "application/json",
},
body: JSON.stringify(buildIssue(checked.value)),
});
if (!res.ok) {
console.log(`GitHub answered ${res.status}: ${(await res.text()).slice(0, 500)}`);
return json(502, { error: "GitHub didn't accept it just now. Try again, or use the GitHub link." });
if (!res.ok) {
return json(502, { error: "GitHub did not confirm delivery. Check recent issues before sending again.", retrySafe: false });
}
const issue = await res.json();
if (!Number.isInteger(issue.number) || typeof issue.html_url !== "string"
|| issue.html_url !== `https://github.com/${env.GITHUB_REPO}/issues/${issue.number}`) {
throw new Error("Invalid issue receipt");
}
return json(201, { ok: true, number: issue.number, url: issue.html_url });
} catch {
return json(502, { error: "Delivery could not be confirmed. Check recent issues before sending again.", retrySafe: false });
}
const issue = await res.json();
return json(201, { ok: true, number: issue.number, url: issue.html_url });
}
export const onRequest = () => json(405, { error: "POST only." });
+2 -2
View File
@@ -79,8 +79,8 @@ export function buildIssue(value) {
// Our own lines go first, so nothing in the sender's text can hide them.
const lines = [
value.github
? `> Sent from the website feedback form by @${value.github}.`
: "> Sent from the website feedback form. The sender left no GitHub username, so they won't see replies here.",
? `> Sent from the website feedback form. Provided GitHub username (not verified): @${value.github}.`
: "> Sent from the website feedback form. No GitHub username provided; bookmark or subscribe to this issue to follow replies.",
"",
];
if (details.length) {
+2 -2
View File
@@ -8,8 +8,8 @@
<meta name="theme-color" content="#0e141b">
<link rel="icon" href="/favicon.png">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
</head>
<body>
+21 -6
View File
@@ -3,8 +3,8 @@
--bg: #0e141b; --bg-2: #131c26; --panel: #17222e; --panel-2: #1d2a38; --line: rgba(143, 152, 160, .16);
--text: #c7d0d8; --bright: #fff; --muted: #8f98a0; --dim: #5e6873;
--blue: #1a9fff; --link: #66c0f4; --green: #75b022; --warn: #d9a23a; --bad: #e0573c;
--action: linear-gradient(to right, #47bfff 5%, #1a44c2 95%);
--action-hi: linear-gradient(to right, #5fcaff 5%, #2458d6 95%);
--action: linear-gradient(to right, #12699e 5%, #1a44c2 95%);
--action-hi: linear-gradient(to right, #1676ad 5%, #2458d6 95%);
--radius: 14px; --wrap: 1160px;
--font: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Arial, sans-serif;
color-scheme: dark;
@@ -38,8 +38,19 @@ code { font: 14px ui-monospace, SFMono-Regular, Menlo, monospace; background: rg
.top nav a { color: var(--muted); font-size: 14.5px; font-weight: 500; }
.top nav a:hover, .top nav a[aria-current] { color: var(--bright); }
.top .end { margin-left: auto; display: flex; gap: 10px; align-items: center; }
@media (max-width: 880px) { .top nav { display: none; } }
@media (max-width: 480px) { .top .end .ghost { display: none; } }
@media (max-width: 880px) {
.top { position: static; }
.top .wrap { flex-wrap: wrap; height: auto; gap: 12px; padding-top: 12px; padding-bottom: 12px; }
.top nav { order: 3; width: 100%; margin-left: 0; gap: 16px; flex-wrap: wrap; }
.top nav a { padding: 6px 0; }
}
@media (max-width: 480px) {
.top .end .ghost { display: none; }
.top .wrap { padding-left: 16px; padding-right: 16px; gap: 8px; }
.top .brand { font-size: 12px; letter-spacing: 1px; gap: 8px; }
.top .brand img { width: 24px; height: 24px; }
.top .end .btn { padding: 0 10px; }
}
/* ---- buttons ---- */
.btn { display: inline-flex; align-items: center; justify-content: center; gap: 9px; height: 46px; padding: 0 22px; border-radius: 10px;
@@ -53,8 +64,8 @@ code { font: 14px ui-monospace, SFMono-Regular, Menlo, monospace; background: rg
.btn.ghost { background: transparent; border-color: var(--line); }
.btn.ghost:hover { border-color: rgba(143,152,160,.4); background: rgba(255,255,255,.03); }
.btn.small { height: 36px; padding: 0 14px; font-size: 14px; border-radius: 8px; }
.btn.coffee { background: #ff5e5b; box-shadow: 0 8px 24px rgba(255, 94, 91, .22); }
.btn.coffee:hover { background: #ff7471; }
.btn.coffee { background: #ff5e5b; color: #0e141b; box-shadow: 0 8px 24px rgba(255, 94, 91, .22); }
.btn.coffee:hover { background: #ff7471; color: #0e141b; }
.btn svg { width: 18px; height: 18px; flex: none; }
.btn[disabled] { opacity: .6; cursor: progress; transform: none; }
@@ -64,6 +75,7 @@ code { font: 14px ui-monospace, SFMono-Regular, Menlo, monospace; background: rg
background: radial-gradient(600px 380px at 30% 30%, rgba(26,159,255,.22), transparent 70%),
radial-gradient(520px 360px at 72% 20%, rgba(111,66,193,.2), transparent 70%); }
.hero > * { position: relative; }
.hero .cta .btn { max-width: 100%; min-height: 46px; height: auto; padding-top: 12px; padding-bottom: 12px; white-space: normal; }
.eyebrow { display: inline-block; max-width: 100%; padding: 6px 14px; border-radius: 999px; font-size: 13.5px;
color: var(--link); background: rgba(26,159,255,.1); border: 1px solid rgba(102,192,244,.22); margin-bottom: 26px; }
.eyebrow b { color: var(--bright); font-weight: 600; }
@@ -131,10 +143,13 @@ section.alt { background: var(--bg-2); border-block: 1px solid var(--line); }
.dl .need { color: var(--muted); font-size: 14px; }
.dl .links { display: flex; flex-direction: column; gap: 8px; margin-top: 16px; }
.dl .links .btn { width: 100%; }
.dl > p { margin-top: 12px; font-size: 14px; }
.dl code { overflow-wrap: anywhere; }
.dl .tag { margin-left: auto; font-size: 11.5px; font-weight: 650; letter-spacing: 1px; text-transform: uppercase; color: var(--link); }
.faq { max-width: 780px; margin: 0 auto; display: grid; gap: 10px; }
.faq details { background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 0 22px; }
.faq code { overflow-wrap: anywhere; }
.faq summary { cursor: pointer; list-style: none; padding: 18px 0; color: var(--bright); font-weight: 600; display: flex; justify-content: space-between; gap: 16px; }
.faq summary::-webkit-details-marker { display: none; }
.faq summary::after { content: "+"; color: var(--muted); font-size: 22px; line-height: 1; transition: transform .2s; }
+4 -4
View File
@@ -8,9 +8,9 @@
<meta name="theme-color" content="#0e141b">
<link rel="icon" href="/favicon.png">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<script src="/js/feedback.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
<script src="/js/feedback.js?v=20260930-deep1" defer></script>
</head>
<body>
@@ -87,7 +87,7 @@
</div>
<div class="field">
<label for="github">GitHub username <small>Optional. Add it to get notified when someone replies.</small></label>
<label for="github">GitHub username <small>Optional and not verified. Subscribe to the created issue on GitHub to follow replies.</small></label>
<input type="text" id="github" name="github" maxlength="40" autocomplete="username" placeholder="@yourname">
</div>
+39 -22
View File
@@ -11,10 +11,12 @@
<meta property="og:type" content="website">
<meta property="og:title" content="Frame Control">
<meta property="og:description" content="Manage your Valve Steam Frame from your computer. Free and open source.">
<meta property="og:image" content="/img/og.jpg">
<meta property="og:image" content="https://framecontrol.fun/img/og.jpg">
<link rel="canonical" href="https://framecontrol.fun/">
<meta property="og:url" content="https://framecontrol.fun/">
<meta name="twitter:card" content="summary_large_image">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
</head>
<body>
@@ -41,20 +43,21 @@
<main>
<section class="hero">
<div class="wrap">
<span class="eyebrow"><b>Free and open source</b><span class="sep"> · </span><span class="plats">macOS · Windows · Linux · iPhone</span></span>
<span class="eyebrow"><b>Free and open source</b><span class="sep"> · </span><span class="plats">macOS · Windows · Linux</span></span>
<h1>Your Steam Frame, <span>managed from your desk.</span></h1>
<p class="lede">See what the headset sees, install games and Android apps, move files and text across, and keep an eye on battery and status. All over SSH, with nothing to install on the Frame.</p>
<p class="lede">Install Frame Control on your computer, not your headset. See what the Frame sees, install games and Android apps, move files and text across, and check battery and status.</p>
<div class="cta">
<a class="btn primary" id="hero-download" href="#download">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M12 3v12m0 0-5-5m5 5 5-5M4 20h16"/></svg>
<span data-label>Download Frame Control</span>
<span data-label>Get started — download the app</span>
</a>
<a class="btn ghost" href="#trailer">
<svg viewBox="0 0 24 24" fill="currentColor" aria-hidden="true"><path d="M8 5.5v13a1 1 0 0 0 1.5.86l10.5-6.5a1 1 0 0 0 0-1.72L9.5 4.64A1 1 0 0 0 8 5.5z"/></svg>
Watch the trailer
</a>
</div>
<p class="fine"><span hidden>Latest: <span data-version></span> · </span><a href="#download">All platforms</a> · MIT licence · Not affiliated with Valve</p>
<p class="fine">Download → install and open → connect your headset. <a href="#setup">Setup guide</a></p>
<p class="fine"><span hidden>Latest: <span data-version></span> · </span><a href="#download">All desktop downloads</a> · <a href="#iphone">iPhone developer build</a> · Not affiliated with Valve</p>
<div class="shot">
<div class="bar"><i></i><i></i><i></i></div>
<img src="/img/home-live.jpg" width="1600" height="1000" alt="Frame Control's Home tab: a live view of the headset beside battery, storage, temperature and Wi-Fi.">
@@ -168,12 +171,12 @@
</div>
</section>
<section>
<section id="iphone">
<div class="wrap">
<div class="head">
<span class="kicker">iPhone and iPad</span>
<h2>Also in your pocket</h2>
<p>The same features on iPhone and iPad, served from the Frame itself, so there's nothing to run on a computer. Build it from the repo in Xcode.</p>
<h2>iPhone: a separate developer build</h2>
<p>Not an App Store download. This option requires a Mac and Xcode, and runs a server on the Frame instead of your computer. <a href="https://github.com/saphid/frame-control/blob/main/docs/iphone.md">Read the iPhone build and pairing guide</a>. For the simplest start, use the desktop app below.</p>
</div>
<div class="phones">
<img src="/img/phone-home.jpg" width="600" height="1304" loading="lazy" alt="Frame Control on iPhone: headset view and device status.">
@@ -186,21 +189,22 @@
<div class="wrap">
<div class="head">
<span class="kicker">Setup</span>
<h2>One password, typed once</h2>
<p>Everything else happens on your computer.</p>
<h2>From download to connected</h2>
<p>You'll need a computer and a Steam Frame, both on the same trusted network. Keep the headset awake while connecting.</p>
</div>
<div class="steps">
<div class="card">
<h3>Install and open the app</h3>
<p><a href="#download">Choose your computer's download below</a>, install it, then open <b>Frame Control</b>. You don't install the desktop app on the headset, and you don't need to clone GitHub or install Python.</p>
</div>
<div class="card">
<h3>Turn on Developer Mode</h3>
<p>On the Frame: Steam Settings → System → <b>Enable Developer Mode</b>, then <b>Set User Password</b> in the Developer section.</p>
</div>
<div class="card">
<h3>Set up the connection</h3>
<p>Open Frame Control and choose <b>Set Up Connection</b>. It finds the headset, makes an SSH key, and asks for that password once.</p>
</div>
<div class="card">
<h3>That's it</h3>
<p>The app reaches the headset whenever it's awake and on the same network. For anywhere else, use <a href="https://github.com/saphid/frame-control/blob/main/docs/tailscale.md">Tailscale</a>.</p>
<p>In Frame Control, choose <b>Set Up Connection</b> and enter the headset's password when asked. It creates an SSH key so you don't have to type it next time.</p>
<p>Headset not found? Enter its IP address from the Frame's Quick Settings. For remote access, use <a href="https://github.com/saphid/frame-control/blob/main/docs/tailscale.md">Tailscale</a>.</p>
</div>
</div>
</div>
@@ -211,7 +215,7 @@
<div class="head">
<span class="kicker">Download</span>
<h2>Get Frame Control</h2>
<p>Free, with Python and adb built in. <span hidden>Version <span data-version></span>.</span> <a href="https://github.com/saphid/frame-control/releases">Release notes</a></p>
<p>Download on your computer, then follow the opening steps on your card. Python is included; adb is included except on arm64 Linux. <span hidden>Version <span data-version></span>.</span> <a href="https://github.com/saphid/frame-control/releases">Release notes</a></p>
</div>
<div class="downloads">
<div class="dl" data-os="mac">
@@ -220,6 +224,8 @@
<div class="links">
<a class="btn primary" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-mac-arm64.dmg">Download .dmg</a>
</div>
<p><b>Open:</b> open the .dmg, drag Frame Control to Applications, then open it from Applications.</p>
<p class="need">Apple Silicon only, not Intel Macs. Unsigned and not notarized: <a href="#mac-first-launch">read the first-launch instructions</a> if macOS blocks it.</p>
</div>
<div class="dl" data-os="windows">
<div class="os"><svg viewBox="0 0 24 24" fill="currentColor"><path d="M3 5.5 10 4.5v7H3zM11 4.35 21 3v8.5H11zM3 12.5h7v7L3 18.5zM11 12.5h10V21l-10-1.4z"/></svg><h3>Windows</h3><span class="tag" hidden>Your system</span></div>
@@ -228,15 +234,21 @@
<a class="btn primary" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-Setup-x64.exe">Download installer</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-win-x64.zip">Portable .zip</a>
</div>
<p><b>Open:</b> run the installer, then launch Frame Control from Start. For the portable .zip, extract it first and run <code>Frame Control.exe</code>.</p>
<p class="need">Unsigned: <a href="#windows-first-launch">read about SmartScreen</a> before proceeding past a warning.</p>
</div>
<div class="dl" data-os="linux">
<div class="os"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="3" y="4" width="18" height="16" rx="2"/><path d="m7 9 3 3-3 3M13 15h4"/></svg><h3>Linux</h3><span class="tag" hidden>Your system</span></div>
<span class="need">x64 or arm64 · needs <code>ssh</code></span>
<p>On arm64 Linux, install <code>adb</code> from your distribution for Android apps: on Debian or Ubuntu, run <code>sudo apt install adb</code>.</p>
<div class="links">
<a class="btn primary" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-x86_64.AppImage">AppImage (x64)</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-amd64.deb">.deb (x64)</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-arm64.AppImage">AppImage (arm64)</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-arm64.deb">.deb (arm64)</a>
</div>
<p><b>Open:</b> for an AppImage, use your file manager's Properties → Permissions to allow executing it, then open it. For a .deb, install it with your package manager and launch Frame Control.</p>
<p class="need"><a href="#linux-first-launch">AppImage won't open?</a> Then <a href="#setup">connect your headset</a>.</p>
</div>
</div>
</div>
@@ -249,22 +261,27 @@
<h2>Good questions</h2>
</div>
<div class="faq">
<details>
<details id="mac-first-launch">
<summary>macOS says the app is damaged or can't be checked</summary>
<div><p>The app isn't notarized, because there's no paid Apple developer account behind it. Drag it to Applications, then clear the download quarantine once in Terminal:</p>
<div><p>The macOS download isn't notarized. Drag it to Applications, then clear the download quarantine once in Terminal:</p>
<p><code>xattr -dr com.apple.quarantine "/Applications/Frame Control.app"</code></p></div>
</details>
<details>
<details id="windows-first-launch">
<summary>Windows SmartScreen says it protected my PC</summary>
<div><p>The installer isn't code-signed. Choose <b>More info → Run anyway</b>, or use the portable .zip: unzip it anywhere and run <code>Frame Control.exe</code>.</p></div>
</details>
<details>
<summary id="linux-first-launch">Linux: the AppImage won't open</summary>
<div><p>Make the file executable in your file manager, or in Terminal run <code>chmod +x Frame-Control-linux-x86_64.AppImage</code> and then <code>./Frame-Control-linux-x86_64.AppImage</code>. On arm64, use the arm64 filename instead.</p>
<p>If FUSE is missing, use the .deb package or run the AppImage with <code>--appimage-extract-and-run</code>.</p></div>
</details>
<details>
<summary>What does it change on my headset?</summary>
<div><p>Only what you click. Installs go to your user account on the Frame, and nothing needs <code>sudo</code> except the power buttons. On your computer it adds a <code>Host frame</code> entry to <code>~/.ssh/config</code> and two SSH keys.</p></div>
</details>
<details>
<summary>Is it safe to leave Developer Mode on?</summary>
<div><p>With Developer Mode on, SSH, ADB and remote desktop are reachable on your local network. Use trusted networks, turn Developer Mode off when you don't need it, and never port-forward those ports from your router. For remote access, use Tailscale. <a href="https://github.com/saphid/frame-control#readme">Security notes</a></p></div>
<div><p>With Developer Mode on, SSH, ADB and remote desktop are reachable on your local network. Use trusted networks, turn Developer Mode off when you don't need it, and never port-forward those ports from your router. Tailscale is an advanced option; off-LAN access has not been verified. Read the <a href="https://github.com/saphid/frame-control/blob/main/docs/tailscale.md#exposure-every-port-is-on-the-tailnet">Tailscale exposure notes</a> before enabling it.</p></div>
</details>
<details>
<summary>Is this made by Valve?</summary>
+15 -8
View File
@@ -27,19 +27,21 @@ form.addEventListener("change", (e) => {
function githubUrl(data) {
const params = data.kind === "idea"
? new URLSearchParams({ template: "idea.yml", title: data.title, what: data.message })
: new URLSearchParams({ template: "bug.yml", title: data.title, description: data.message,
version: data.version, os: [data.os, data.steamos && `SteamOS ${data.steamos}`].filter(Boolean).join(", ") });
: data.kind === "bug"
? new URLSearchParams({ template: "bug.yml", title: data.title, description: data.message,
version: data.version, os: [data.os, data.steamos && `SteamOS ${data.steamos}`].filter(Boolean).join(", ") })
: new URLSearchParams({ title: data.title, body: data.message });
return `https://github.com/saphid/frame-control/issues/new?${params}`;
}
function showError(message, data) {
function showError(message, data, uncertain = false) {
errorBox.textContent = message + " ";
if (data) {
const a = document.createElement("a");
a.href = githubUrl(data);
a.href = uncertain ? "https://github.com/saphid/frame-control/issues" : githubUrl(data);
a.target = "_blank";
a.rel = "noopener";
a.textContent = "Open it on GitHub instead";
a.textContent = uncertain ? "Check recent issues before sending again" : "Open it on GitHub instead";
errorBox.append(a);
}
errorBox.hidden = false;
@@ -62,6 +64,7 @@ form.addEventListener("submit", async (e) => {
send.disabled = true;
send.textContent = "Sending…";
let uncertain = true;
try {
// The server treats anything sent sooner as a script (site/lib/feedback.js MIN_FILL_MS).
const wait = 3100 - (performance.now() - started);
@@ -72,7 +75,11 @@ form.addEventListener("submit", async (e) => {
body: JSON.stringify({ ...data, elapsed: Math.round(performance.now() - started) }),
});
const reply = await res.json().catch(() => ({}));
if (!res.ok) throw new Error(reply.error || "Something went wrong sending that.");
if (!res.ok) {
uncertain = reply.retrySafe === false || (reply.retrySafe !== true && res.status >= 500);
throw new Error(reply.error || "Something went wrong sending that.");
}
if (!reply.ok) throw new Error("Delivery could not be confirmed.");
form.hidden = true;
document.getElementById("done").hidden = false;
@@ -81,11 +88,11 @@ form.addEventListener("submit", async (e) => {
link.href = reply.url;
link.hidden = false;
document.getElementById("done-text").textContent = `Your feedback is now issue #${reply.number} on GitHub.`
+ (data.github ? " You'll be notified when someone replies." : " Bookmark it to follow along.");
+ " Bookmark it or subscribe on GitHub to follow replies.";
}
window.scrollTo({ top: 0, behavior: "smooth" });
} catch (err) {
showError(err.message || "Couldn't reach the server.", data);
showError(err.message || "Couldn't confirm delivery.", data, uncertain);
} finally {
send.disabled = false;
send.textContent = "Send feedback";
+15 -5
View File
@@ -5,8 +5,6 @@ const SITE = {
kofi: "alexsouthwell",
};
const RELEASE = `https://github.com/${SITE.repo}/releases/latest/download/`;
// Donate buttons.
for (const el of document.querySelectorAll("[data-kofi]")) {
if (SITE.kofi) el.href = `https://ko-fi.com/${SITE.kofi}`;
@@ -28,7 +26,7 @@ function detectPlatform() {
}
const PLATFORMS = {
mac: { name: "macOS", file: "Frame-Control-mac-arm64.dmg" },
mac: { name: "macOS (Apple Silicon)", file: "Frame-Control-mac-arm64.dmg" },
windows: { name: "Windows", file: "Frame-Control-Setup-x64.exe" },
linux: { name: "Linux", file: "Frame-Control-linux-x86_64.AppImage" },
"linux-arm": { name: "Linux (arm64)", file: "Frame-Control-linux-arm64.AppImage" },
@@ -37,8 +35,7 @@ const PLATFORMS = {
const platform = detectPlatform();
const hero = document.getElementById("hero-download");
if (hero && PLATFORMS[platform]) {
hero.href = RELEASE + PLATFORMS[platform].file;
hero.querySelector("[data-label]").textContent = `Download for ${PLATFORMS[platform].name}`;
hero.querySelector("[data-label]").textContent = `Get started on ${PLATFORMS[platform].name}`;
}
const card = platform && document.querySelector(`.dl[data-os="${platform.replace("-arm", "")}"]`);
if (card) {
@@ -61,3 +58,16 @@ if (versionEls.length) {
}
document.querySelectorAll("[data-year]").forEach((el) => (el.textContent = new Date().getFullYear()));
function revealLinkedAnswer(hash = location.hash) {
const target = document.getElementById(hash.slice(1));
const answer = target?.closest("details");
if (answer) answer.open = true;
}
window.addEventListener("hashchange", () => revealLinkedAnswer());
document.addEventListener("click", (event) => {
const link = event.target?.closest?.('a[href^="#"]');
if (link) revealLinkedAnswer(link.getAttribute("href"));
});
revealLinkedAnswer();
+8 -6
View File
@@ -8,8 +8,8 @@
<meta name="theme-color" content="#0e141b">
<link rel="icon" href="/favicon.png">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
</head>
<body>
@@ -38,12 +38,14 @@
<div class="head">
<span class="kicker">Privacy</span>
<h1>Privacy</h1>
<p>Short version: the app collects nothing, and the website keeps only what you choose to send.</p>
<p>Short version: the released app sends anonymous usage statistics after a first-run notice. You can turn them off. Website feedback becomes public on GitHub.</p>
</div>
<div class="faq notes">
<section><h2>The app</h2><p>Frame Control talks only to your headset (over SSH on your network), to GitHub for releases, and to Steam and F-Droid for game and app listings. It has no analytics and no accounts.</p></section>
<section><h2>The feedback form</h2><p>What you type becomes a public GitHub issue on <a href="https://github.com/saphid/frame-control/issues">saphid/frame-control</a>. To stop abuse, the form keeps a one-way hash of your IP address for about an hour to count submissions. The address itself isn't stored or published.</p></section>
<section><h2>This website</h2><p>Hosted on Cloudflare Pages. No cookies, no analytics, no trackers. The download section asks GitHub for the latest version number. Donations go through Ko-fi, under Ko-fi's own privacy policy.</p></section>
<section><h2>App usage statistics</h2><p>In v0.4.0, anonymous usage statistics are on by default after the first-run notice has been shown. They go to PostHog's US cloud and include a random persistent installation ID, app version, operating system, CPU architecture and Python version; app opens, updates, tabs used, headset SteamOS builds, and install outcomes. Public catalogue installs may include a Steam app ID, Flathub ID or F-Droid package name. Usage events exclude local filenames, paths and account data. Person profiles and location lookup are disabled, and events use a placeholder IP address; the receiving service still handles the network connection.</p><p>Turn statistics off in <b>Privacy &amp; updates</b>. <b>Show what's been sent</b> lets you inspect sent events. Turning a level off removes its unsent events, not events already received. Source checkouts do not send automatic telemetry by default. <code>DO_NOT_TRACK=1</code> or <code>FRAME_CONTROL_TELEMETRY=0</code> blocks automatic telemetry, not reports you deliberately send through <b>Report a problem</b>.</p></section>
<section><h2>Optional app reports</h2><p>Compatibility sharing and detailed error reports are off by default. Enabling compatibility sharing sends reports and tests, including earlier reports on this computer. Enabling error details sends scrubbed error messages and tracebacks. Review free text before sharing: automatic scrubbing is not a guarantee that every personal detail is removed. The app's <b>Report a problem</b> dialog deliberately sends a private report to the maintainer's PostHog project, independently of the usage-statistics setting. This differs from the public website form below. See the <a href="https://github.com/saphid/frame-control/blob/v0.4.0/docs/privacy.md">v0.4.0 privacy guide</a> for exact fields and controls.</p></section>
<section><h2>Other app connections</h2><p>Frame Control connects to your headset and to services used by the features you choose, including GitHub for releases, Steam, F-Droid and Flathub for listings or downloads, and the sources of apps you choose to install. It does not require a Frame Control account.</p></section>
<section><h2>The website feedback form</h2><p>What you type becomes a public GitHub issue on <a href="https://github.com/saphid/frame-control/issues">saphid/frame-control</a>, including any username and diagnostic details you provide. Do not include passwords, tokens or private information. To stop abuse, the form keeps a token-salted, one-way hash of your IP address in an hourly counter for up to 65 minutes, plus a non-identifying daily counter for up to 25 hours. The form does not store or publish the raw address; Cloudflare still processes network requests.</p></section>
<section><h2>This website</h2><p>Hosted on Cloudflare Pages. The site itself sets no cookies and includes no analytics or tracking scripts. Cloudflare processes requests to serve and protect it. The download section asks GitHub for the latest release. Donations go through Ko-fi, under Ko-fi's own privacy policy.</p></section>
</div>
</div>
</main>
+64
View File
@@ -0,0 +1,64 @@
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { test } from "node:test";
import vm from "node:vm";
const script = await readFile(new URL("../public/js/feedback.js", import.meta.url), "utf8");
function load(fetchResponse) {
const handlers = {};
const links = [];
const form = { elements: { os: { value: "" } }, addEventListener: (name, handler) => { handlers[name] = handler; } };
const errorBox = { hidden: true, append: (link) => links.push(link), scrollIntoView: () => {} };
const send = { disabled: false, textContent: "Send feedback" };
let elapsed = 0;
const context = {
navigator: { userAgent: "Windows" },
performance: { now: () => elapsed },
URLSearchParams,
fetch: fetchResponse,
window: { scrollTo: () => {} },
document: {
getElementById: (id) => ({ feedback: form, error: errorBox, send })[id] || {},
createElement: () => ({}),
},
FormData: class extends Map {
constructor() { super(Object.entries({ kind: "question", title: "A local question", message: "This test never sends real feedback." })); }
},
};
vm.runInNewContext(script, context);
elapsed = 4000;
return { context, handlers, links, send };
}
test("questions and general feedback are not mislabelled as bugs in GitHub fallback", () => {
const page = load();
for (const kind of ["question", "other"]) {
const url = new URL(page.context.githubUrl({ kind, title: "A local question", message: "Please help" }));
assert.equal(url.searchParams.has("template"), false);
assert.equal(url.searchParams.get("body"), "Please help");
}
assert.equal(new URL(page.context.githubUrl({ kind: "bug", title: "A bug", message: "Details" })).searchParams.get("template"), "bug.yml");
assert.equal(new URL(page.context.githubUrl({ kind: "idea", title: "An idea", message: "Details" })).searchParams.get("template"), "idea.yml");
});
test("an unconfirmed send points to recent issues rather than creating a duplicate", async () => {
const page = load(async () => ({ ok: false, status: 502, json: async () => ({ error: "Could not confirm delivery", retrySafe: false }) }));
await page.handlers.submit({ preventDefault: () => {} });
assert.equal(page.links[0].href, "https://github.com/saphid/frame-control/issues");
assert.equal(page.send.disabled, false);
});
test("a browser network failure also points to recent issues", async () => {
const page = load(async () => { throw new Error("Local offline test"); });
await page.handlers.submit({ preventDefault: () => {} });
assert.equal(page.links[0].href, "https://github.com/saphid/frame-control/issues");
});
test("known-unsent integration errors retain the prefilled issue fallback", async () => {
const page = load(async () => ({ ok: false, status: 503, json: async () => ({ error: "Not connected", retrySafe: true }) }));
await page.handlers.submit({ preventDefault: () => {} });
const url = new URL(page.links[0].href);
assert.equal(url.pathname, "/saphid/frame-control/issues/new");
assert.equal(url.searchParams.get("body"), "This test never sends real feedback.");
});
+56
View File
@@ -0,0 +1,56 @@
import assert from "node:assert/strict";
import { test } from "node:test";
import { onRequestPost } from "../functions/api/feedback.js";
const env = { GITHUB_TOKEN: "local-test-placeholder", GITHUB_REPO: "example/local-test" };
const input = { kind: "bug", title: "Local test report", message: "No external requests are made by these tests.", elapsed: 4000 };
const request = (origin, body = input) => new Request("https://framecontrol.fun/api/feedback", {
method: "POST",
headers: { "content-type": "application/json", ...(origin === undefined ? {} : { origin }) },
body: JSON.stringify(body),
});
test("malformed and opaque Origins are rejected without throwing or contacting GitHub", async (testContext) => {
const upstream = testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Unexpected external request"); });
for (const origin of ["not-a-url", "null", "https://other.example", "http://framecontrol.fun"]) {
const response = await onRequestPost({ request: request(origin), env });
assert.equal(response.status, 403, origin);
}
assert.equal(upstream.mock.callCount(), 0);
});
test("same-origin invalid JSON content is rejected before contacting GitHub", async (testContext) => {
const upstream = testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Unexpected external request"); });
const response = await onRequestPost({ request: request("https://framecontrol.fun", null), env });
assert.equal(response.status, 400);
assert.equal(upstream.mock.callCount(), 0);
});
test("missing integration settings fail without contacting GitHub", async (testContext) => {
const upstream = testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Unexpected external request"); });
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env: {} });
assert.equal(response.status, 503);
assert.equal((await response.json()).retrySafe, true);
assert.equal(upstream.mock.callCount(), 0);
});
test("GitHub network failure produces an uncertainty-aware response instead of throwing", async (testContext) => {
testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Local simulated outage"); });
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env });
assert.equal(response.status, 502);
assert.equal((await response.json()).retrySafe, false);
});
test("an unreadable successful GitHub response does not encourage duplicate submission", async (testContext) => {
testContext.mock.method(globalThis, "fetch", async () => new Response("not JSON", { status: 201 }));
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env });
assert.equal(response.status, 502);
assert.equal((await response.json()).retrySafe, false);
});
test("successful feedback returns the created issue", async (testContext) => {
testContext.mock.method(globalThis, "fetch", async () => Response.json({ number: 7, html_url: "https://github.com/example/local-test/issues/7" }, { status: 201 }));
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env });
assert.equal(response.status, 201);
assert.deepEqual(await response.json(), { ok: true, number: 7, url: "https://github.com/example/local-test/issues/7" });
});
+4 -4
View File
@@ -37,20 +37,20 @@ test("unknown kinds become other feedback", () => {
assert.equal(validate(form({ kind: "__proto__" })).value.kind, "other");
});
test("builds a labelled issue that credits a GitHub user", () => {
test("builds a labelled issue with an explicitly unverified GitHub username", () => {
const { value } = validate(form({ github: "@octocat", version: "0.3.1", steamos: "20260922" }));
const issue = buildIssue(value);
assert.equal(issue.title, "Bug report: Live view freezes");
assert.deepEqual(issue.labels, ["feedback", "bug"]);
assert.match(issue.body, /\| Frame Control version \| 0\.3\.1 \|/);
assert.match(issue.body, /^> Sent from the website feedback form by @octocat\./);
assert.match(issue.body, /Provided GitHub username \(not verified\): @octocat\./);
assert.ok(issue.body.endsWith(value.message));
});
test("anonymous feedback says replies won't reach the sender", () => {
test("anonymous feedback explains how to follow replies", () => {
const issue = buildIssue(validate(form({ kind: "other" })).value);
assert.deepEqual(issue.labels, ["feedback"]);
assert.match(issue.body, /won't see replies/);
assert.match(issue.body, /bookmark or subscribe/);
});
test("breaks mentions, issue refs and table cells in user text", () => {
+123
View File
@@ -0,0 +1,123 @@
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { test } from "node:test";
import vm from "node:vm";
const script = await readFile(new URL("../public/js/site.js", import.meta.url), "utf8");
const html = await readFile(new URL("../public/index.html", import.meta.url), "utf8");
function visit(platform, userAgent = "", maxTouchPoints = 0, hash = "") {
const label = { textContent: "Get started — download the app" };
const hero = { href: "#download", querySelector: () => label };
const answer = { open: false };
const listeners = {};
const context = {
navigator: { platform, userAgent, maxTouchPoints },
location: { hash },
window: { addEventListener: (event, handler) => { listeners[event] = handler; } },
document: {
addEventListener: (event, handler) => { listeners[`document-${event}`] = handler; },
querySelectorAll: () => [],
querySelector: () => null,
getElementById: (id) => id === "hero-download" ? hero : id === "mac-first-launch" ? { closest: () => answer } : null,
},
};
vm.runInNewContext(script, context);
return { hero, label, answer, listeners, context };
}
test("desktop hero leads to opening instructions rather than a bare package", () => {
for (const [platform, name] of [["MacIntel", "macOS (Apple Silicon)"], ["Win32", "Windows"], ["Linux x86_64", "Linux"], ["Linux aarch64", "Linux (arm64)"]]) {
const page = visit(platform);
assert.equal(page.hero.href, "#download");
assert.equal(page.label.textContent, `Get started on ${name}`);
}
});
test("phones and touch-enabled iPads are not offered a desktop package", () => {
for (const [platform, userAgent, touches] of [["iPhone", "iPhone", 1], ["MacIntel", "Macintosh", 5], ["Linux armv8l", "Android", 1]]) {
const page = visit(platform, userAgent, touches);
assert.equal(page.hero.href, "#download");
assert.equal(page.label.textContent, "Get started — download the app");
}
});
test("first-launch links open the answer on arrival and hash navigation", () => {
assert.equal(visit("", "", 0, "#mac-first-launch").answer.open, true);
const page = visit("");
page.context.location.hash = "#mac-first-launch";
page.listeners.hashchange();
assert.equal(page.answer.open, true);
assert.doesNotThrow(() => visit("", "", 0, "#missing"));
});
test("clicking the same first-launch link reopens a manually closed answer", () => {
const page = visit("", "", 0, "#mac-first-launch");
page.answer.open = false;
page.listeners["document-click"]({ target: { closest: () => ({ getAttribute: () => "#mac-first-launch" }) } });
assert.equal(page.answer.open, true);
assert.doesNotThrow(() => page.listeners["document-click"]({ target: {} }));
});
test("landing page keeps unique anchors and opening instructions for every desktop", () => {
const ids = [...html.matchAll(/\bid="([^"]+)"/g)].map((match) => match[1]);
assert.equal(new Set(ids).size, ids.length);
for (const match of html.matchAll(/href="#([^"]+)"/g)) assert.ok(ids.includes(match[1]), match[1]);
assert.match(html, /Install Frame Control on your computer, not your headset/);
assert.match(html, /drag Frame Control to Applications/);
assert.match(html, /launch Frame Control from Start/);
assert.match(html, /Properties → Permissions/);
assert.match(html, /Frame-Control-linux-arm64\.deb/);
assert.match(html, /Not an App Store download/);
});
test("sharing metadata points to the custom domain", () => {
assert.match(html, /rel="canonical" href="https:\/\/framecontrol\.fun\/"/);
assert.match(html, /property="og:url" content="https:\/\/framecontrol\.fun\/"/);
assert.match(html, /property="og:image" content="https:\/\/framecontrol\.fun\/img\/og\.jpg"/);
});
test("Linux arm64 download explains the system adb prerequisite", () => {
assert.match(html, /On arm64 Linux, install <code>adb<\/code>/);
});
test("privacy describes released telemetry and distinguishes private app reports from public feedback", async () => {
const privacy = await readFile(new URL("../public/privacy/index.html", import.meta.url), "utf8");
assert.doesNotMatch(privacy, /app collects nothing|has no analytics/);
for (const text of ["on by default", "PostHog", "random persistent installation ID", "Privacy &amp; updates", "off by default", "public GitHub issue", "v0.4.0/docs/privacy.md"]) {
assert.ok(privacy.includes(text), text);
}
});
test("compact headers retain navigation and high-contrast support text", async () => {
const css = await readFile(new URL("../public/css/site.css", import.meta.url), "utf8");
assert.doesNotMatch(css, /\.top nav\s*\{\s*display:\s*none/);
assert.match(css, /\.top nav \{ order: 3; width: 100%/);
assert.match(css, /@media \(max-width: 880px\) \{\s*\.top \{ position: static; \}/);
assert.match(css, /\.btn\.coffee \{[^}]*color: #0e141b/);
});
test("primary button gradients keep white text above 4.5:1 contrast", async () => {
const css = await readFile(new URL("../public/css/site.css", import.meta.url), "utf8");
const luminance = (channels) => channels.map((channel) => {
const normalized = channel / 255;
return normalized <= 0.04045 ? normalized / 12.92 : ((normalized + 0.055) / 1.055) ** 2.4;
}).reduce((total, channel, index) => total + channel * [0.2126, 0.7152, 0.0722][index], 0);
for (const variable of ["action", "action-hi"]) {
const value = css.match(new RegExp(`--${variable}: ([^;]+);`))[1];
const colors = [...value.matchAll(/#([0-9a-f]{6})/g)].map((match) => [0, 2, 4].map((offset) => parseInt(match[1].slice(offset, offset + 2), 16)));
assert.equal(colors.length, 2);
for (let step = 0; step <= 100; step++) {
const channels = colors[0].map((channel, index) => channel + (colors[1][index] - channel) * step / 100);
assert.ok(1.05 / (luminance(channels) + 0.05) >= 4.5, `${variable} at ${step}%`);
}
}
});
test("every page versions shared assets to avoid stale onboarding behavior", async () => {
for (const file of ["index.html", "feedback/index.html", "privacy/index.html", "404.html"]) {
const page = await readFile(new URL(`../public/${file}`, import.meta.url), "utf8");
assert.match(page, /href="\/css\/site\.css\?v=/, file);
assert.match(page, /src="\/js\/site\.js\?v=/, file);
}
});
+41 -1
View File
@@ -113,6 +113,17 @@ class Helpers(unittest.TestCase):
self.assertIn("Chromium", str(cm.exception))
self.assertTrue(calls[0].startswith("bash -s -- "))
def test_the_mac_space_opens_with_a_space_ticket(self):
calls, asked = [], []
mv = frame_macview.MacView(["ssh"], lambda remote, **kw: calls.append(remote) or "panel x", "frame")
mv.call = lambda path, **kw: asked.append((path, kw)) or {"screen": True, "ticket": "tk1"}
mv.ensure_tunnel = lambda **kw: None
mv.remote_port = 47900
out = mv.show("space")
self.assertIn(("/ticket", {"method": "POST", "src": "space"}), asked)
self.assertIn("http://127.0.0.1:47900/space?t=tk1", calls[0])
self.assertEqual(out["src"], "space")
def test_separate_windows_need_accessibility(self):
mv = frame_macview.MacView(["ssh"], lambda *a, **k: self.fail("no ssh"), "frame")
mv.call = lambda path, **kw: {"screen": True, "accessibility": False}
@@ -284,7 +295,10 @@ class Agent(unittest.TestCase):
def test_serves_the_viewer_page(self):
status, body = self.get(f"/view?k={self.token}&src=test")
self.assertEqual(status, 200)
self.assertIn(b"VideoDecoder", body)
self.assertIn(b'<script src="/mac-stream.js">', body)
status, script = self.get("/mac-stream.js")
self.assertEqual(status, 200)
self.assertIn(b"VideoDecoder", script)
def test_snapshot_needs_the_key(self):
self.assertEqual(self.get("/snapshot?display=1")[0], 403)
@@ -345,6 +359,32 @@ class Agent(unittest.TestCase):
self.get(f"/close?k={self.token}&src=test", method="POST")
self.assertEqual(WS(self.port, f"/stream?src=test&r={hello['r']}").status, 403)
def test_a_space_key_opens_any_source_until_stop_all(self):
t = self.ticket("space")
status, body = self.get(f"/space/key?t={t}")
self.assertEqual(status, 200)
key = json.loads(body)["key"]
self.assertEqual(self.get(f"/space/key?t={t}")[0], 403) # a ticket gives one key
self.assertEqual(self.get("/space/windows?s=nope")[0], 403)
listing = json.loads(self.get(f"/space/windows?s={key}")[1])
self.assertIn("windows", listing)
self.assertTrue(all("x" in d and "y" in d for d in listing["displays"]))
ws = WS(self.port, f"/stream?src=test&s={key}")
self.assertEqual(ws.status, 101)
hello = json.loads(ws.recv()[1])
self.assertEqual(hello["t"], "hello")
ws.close()
self.assertEqual(WS(self.port, f"/stream?src=test&t={t}").status, 403) # not a stream ticket
self.get(f"/close?k={self.token}", method="POST") # Stop All
self.assertEqual(self.get(f"/space/windows?s={key}")[0], 403)
self.assertEqual(WS(self.port, f"/stream?src=test&s={key}").status, 403)
self.assertEqual(WS(self.port, f"/stream?src=test&r={hello['r']}").status, 403)
def test_the_space_page_and_its_script_are_served(self):
for path in ("/space", "/mac-stream.js"):
status, _ = self.get(path)
self.assertEqual(status, 200, path)
def test_stop_revokes_tickets_not_yet_used(self):
t = self.ticket("test")
self.get(f"/close?k={self.token}&src=test", method="POST")
+30
View File
@@ -9,6 +9,7 @@ failure; FRAME_PC_INSTALLER_BUNDLE=1 checks the installer's bundle instead.
"""
import ctypes
import http.client
from http.server import ThreadingHTTPServer
import json
import os
from pathlib import Path
@@ -31,6 +32,35 @@ from frame_stream_stats import Stats
from test_macview import WS
class ViewerResourcesTests(unittest.TestCase):
def test_public_viewer_can_load_its_script_without_master_credentials(self):
httpd = ThreadingHTTPServer(('127.0.0.1', 0), agent.Handler)
httpd.agent = mock.Mock(page=ROOT / 'ui/mac-view.html', grants=agent.Grants('master'))
worker = threading.Thread(target=httpd.serve_forever, daemon=True)
worker.start()
try:
for path, content_type, expected in [
('/view', 'text/html', b'<script src="/mac-stream.js">'),
('/mac-stream.js', 'application/javascript', b'class MacStream'),
('/status', 'application/json', b'forbidden'),
('/../mac-stream.js', 'application/json', b'forbidden'),
]:
with self.subTest(path=path):
conn = http.client.HTTPConnection('127.0.0.1', httpd.server_port, timeout=3)
try:
conn.request('GET', path)
response = conn.getresponse()
self.assertEqual(response.status, 200 if path in ('/view', '/mac-stream.js') else 403)
self.assertIn(content_type, response.getheader('Content-Type'))
self.assertIn(expected, response.read())
finally:
conn.close()
finally:
httpd.shutdown()
httpd.server_close()
worker.join(timeout=3)
class GrantsTests(unittest.TestCase):
def test_source_bound_retry_ack_and_stop(self):
g = agent.Grants('master')
+13 -5
View File
@@ -386,7 +386,7 @@ class MacView:
self.launching -= 1
def _show(self, src, quality, width, height):
if src != "test" and not src.startswith(("window:", "display:", "separate:")):
if src not in ("test", "space") and not src.startswith(("window:", "display:", "separate:")):
raise MacViewError("Pick a window or display to show.")
self.shows += 1
q = QUALITY.get(quality) or QUALITY["balanced"]
@@ -411,11 +411,19 @@ class MacView:
# A single-use ticket for this source, not Frame Control's key: the URL
# is visible in the Frame's process list.
ticket = self.call("/ticket", method="POST", src=src)["ticket"]
params = {"src": src, "t": ticket, "tag": tag, "codec": q["codec"], "max": q["max"], "fps": q["fps"],
"bpp": q["bpp"]}
url = f"http://127.0.0.1:{self.remote_port}/view?{urlencode(params)}"
if src == "space":
# The Mac space (ui/mac-space.html): one WebXR page that opens the
# Mac's display and any windows pulled out of it (docs/mac-space.md).
url = f"http://127.0.0.1:{self.remote_port}/space?{urlencode({'t': ticket, 'tag': tag})}"
else:
params = {"src": src, "t": ticket, "tag": tag, "codec": q["codec"], "max": q["max"], "fps": q["fps"],
"bpp": q["bpp"]}
url = f"http://127.0.0.1:{self.remote_port}/view?{urlencode(params)}"
w, h = fit(width or 1280, height or 720)
args = " ".join(_quote(str(a)) for a in (appid, url, w, h, tag, self.viewer_profile, *self.browser_flags))
# WebXR needs Chromium XR's OpenXR feature, and its XR process needs
# the seccomp filter off on arm64 (docs/webxr-chromium.md).
xr = ["--enable-features=OpenXR", "--disable-seccomp-filter-sandbox"] if src == "space" else []
args = " ".join(_quote(str(a)) for a in (appid, url, w, h, tag, self.viewer_profile, *self.browser_flags, *xr))
try:
out = self.run("bash -s -- " + args, stdin=LAUNCH, timeout=60)
except Exception as e: # noqa: BLE001 - the server's Failure carries the Frame's words
+5
View File
@@ -498,6 +498,11 @@ class Handler(BaseHTTPRequestHandler):
return self.reply(b'frame-mac-view', content='text/plain')
if method == 'GET' and url.path == '/view':
return self.reply(agent.page.read_bytes(), content='text/html; charset=utf-8')
if method == 'GET' and url.path == '/mac-stream.js':
script = agent.page.with_name('mac-stream.js')
if not script.is_file():
return self.reply({'error': 'not found'}, 404)
return self.reply(script.read_bytes(), content='application/javascript; charset=utf-8')
if method == 'GET' and url.path == '/stream':
return self.stream(q)
if not agent.grants.master(q.get('k', self.headers.get('X-Token'))):
+13 -1
View File
@@ -494,6 +494,8 @@
.mv-perm .row { margin-top: 8px; }
.mv-live { color: var(--green, #7fd67f); font-size: 12px; }
.mv-opt { display: block; font-size: 13px; margin: 0 0 12px; }
.mv-space { align-items: center; gap: 12px; margin: 0 0 14px; }
.mv-space .sub { font-size: 13px; }
.mv-opt .sub { display: block; margin-left: 22px; font-size: 12px; }
.disp select { width: 100%; background: rgba(0,0,0,.28); color: var(--text); border: 1px solid transparent; border-radius: 3px;
padding: 8px 10px; font: inherit; font-size: 13px; }
@@ -1254,6 +1256,7 @@
<section id="shots">
<div class="shelf-head"><h2>Screenshots</h2><span class="count" id="shotCount"></span><span class="spacer"></span>
<button class="small" id="shotsRefresh">Refresh</button>
<button class="small" id="phoneShotDownloads" hidden>Downloads &amp; Photos</button>
<button class="small" id="shotsFolder" title="Open the SteamFrame folder in your Pictures">Show folder</button>
<button class="action small" id="shotsSaveNew" disabled>Save new to this computer</button>
</div>
@@ -1737,6 +1740,8 @@
<div class="sub" id="mvPcState" role="status" aria-live="polite" hidden></div>
</div>
<div id="mvMain">
<div class="row mv-space"><button class="action" data-mv="show" data-src="space" data-title="Mac space">Open Mac space</button>
<span class="sub">Your Mac's screen in the room, like Vision Pro. Drag a window's title bar off its edge to give it a place of its own.</span></div>
<label class="mv-opt"><input type="checkbox" id="mvSeparate" checked> Give each window its own display
<span class="sub">its menus and sheets come along, nothing covers it, and it goes back when you stop</span></label>
<div class="row" style="margin:0 0 12px"><label for="mvQuality">Picture quality</label>
@@ -5074,6 +5079,10 @@ api("/api/host").then(applyHostWording).catch(() => {});
// thumbs: id -> promise of an object URL, kept across reloads so a refresh
// that finds a new shot fetches only that one's thumbnail.
const shots = { list: [], sig: null, thumbs: new Map(), loading: null, gen: 0, fill: 0 };
if (window.frameApp?.screenshotDownloads) {
$("phoneShotDownloads").hidden = false;
$("phoneShotDownloads").onclick = () => window.frameApp.screenshotDownloads();
}
const STEAMVR_APPID = "250820";
const SHOTS_POLL_MS = 8000;
function shotApp(appid) {
@@ -5667,6 +5676,7 @@ async function loadMacView(restart) {
$("mvMain").hidden = false;
$("mvPcState").hidden = true;
$("mvSeparate").closest("label").hidden = pc; // separate virtual displays are a Mac feature
$("macview").querySelector(".mv-space").hidden = pc;
const live = new Set((s.streams || []).map(x => x.src));
const byStream = src => (s.streams || []).find(x => x.src === src);
$("mvCount").textContent = live.size ? `${live.size} showing${s.route === "usb" ? " · over USB-C" : ""}` : "";
@@ -5744,7 +5754,9 @@ $("macview").onclick = async e => {
await act(`Show ${b.dataset.title} in the headset`, async () => {
const r = await api("/api/macview", { action: "show", src: as, quality: $("mvQuality").value,
w: +b.dataset.w || undefined, h: +b.dataset.h || undefined });
return { message: `${b.dataset.title} is a panel in the headset now. Float it with the SteamVR dashboard.`, ...r };
const message = as === "space" ? "The Mac space is ready in the headset: press Enter Mac space there."
: `${b.dataset.title} is a panel in the headset now. Float it with the SteamVR dashboard.`;
return { message, ...r };
}, b);
} else if (b.dataset.mv === "stop") {
await act("Stop showing it", () => api("/api/macview", { action: "stop", src }), b);
+687
View File
@@ -0,0 +1,687 @@
<!doctype html>
<!-- Frame Control: the Mac space. The Mac's main display as a large panel in
the room, and any Mac window pulled out of it as a panel of its own, in
one WebXR session with passthrough behind (docs/mac-space.md). Served by
the Mac's frame-mac-view agent; streams come from ui/mac-stream.js. Without
WebXR (a desktop browser) the same scene shows in the page, with the
mouse as the laser: left button = trigger, right button = grab. -->
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Mac space</title>
<style>
html, body { margin: 0; height: 100%; background: #101216; color: #eee; font: 16px/1.45 system-ui, sans-serif; overflow: hidden; }
#gl { position: fixed; inset: 0; width: 100%; height: 100%; display: block; }
#ui { position: fixed; inset: 0; display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 18px;
text-align: center; padding: 24px; background: rgba(16,18,22,.92); }
#ui[hidden] { display: none; }
#enter { font: 600 28px system-ui, sans-serif; padding: 22px 44px; border-radius: 18px; border: 0; color: #fff;
background: #2f6fed; cursor: pointer; }
#enter:disabled { background: #444; cursor: default; }
#msg { max-width: 34em; color: #bbb; }
#hint { position: fixed; left: 12px; bottom: 10px; color: #9aa; font-size: 13px; pointer-events: none; }
</style>
</head>
<body>
<canvas id="gl"></canvas>
<div id="ui">
<div style="font-size:22px">Mac space</div>
<button id="enter" disabled>Connecting…</button>
<div id="msg">Your Mac's screen appears in front of you. Drag a window's title bar off its edge to give that window a
place of its own; drop it back on the screen to put it back.</div>
</div>
<div id="hint" hidden>Preview: left button clicks, right button (or Shift) drags a panel by its bar, wheel scrolls.</div>
<script src="/mac-stream.js"></script>
<script>
"use strict";
const q = new URLSearchParams(location.search);
const tag = q.get("tag") || "";
document.title = tag ? `Mac space [${tag}]` : "Mac space";
const $ = id => document.getElementById(id);
const say = t => { $("msg").textContent = t; };
// ---------- small 3D math (column-major 4x4, like WebGL) ----------
const V = {
add: (a, b) => [a[0] + b[0], a[1] + b[1], a[2] + b[2]],
sub: (a, b) => [a[0] - b[0], a[1] - b[1], a[2] - b[2]],
scale: (a, s) => [a[0] * s, a[1] * s, a[2] * s],
dot: (a, b) => a[0] * b[0] + a[1] * b[1] + a[2] * b[2],
len: a => Math.hypot(a[0], a[1], a[2]),
norm: a => { const l = Math.hypot(a[0], a[1], a[2]) || 1; return [a[0] / l, a[1] / l, a[2] / l]; },
};
const M = {
mul(a, b) {
const o = new Float32Array(16);
for (let c = 0; c < 4; c++) for (let r = 0; r < 4; r++) {
let s = 0;
for (let k = 0; k < 4; k++) s += a[k * 4 + r] * b[c * 4 + k];
o[c * 4 + r] = s;
}
return o;
},
// Rigid transform (rotation about Y by yaw, then position), scaled by sx, sy.
panel(pos, yaw, sx = 1, sy = 1) {
const c = Math.cos(yaw), s = Math.sin(yaw);
return new Float32Array([c * sx, 0, -s * sx, 0, 0, sy, 0, 0, s, 0, c, 0, pos[0], pos[1], pos[2], 1]);
},
point: (m, p) => [m[0] * p[0] + m[4] * p[1] + m[8] * p[2] + m[12], m[1] * p[0] + m[5] * p[1] + m[9] * p[2] + m[13],
m[2] * p[0] + m[6] * p[1] + m[10] * p[2] + m[14]],
dir: (m, d) => [m[0] * d[0] + m[4] * d[1] + m[8] * d[2], m[1] * d[0] + m[5] * d[1] + m[9] * d[2], m[2] * d[0] + m[6] * d[1] + m[10] * d[2]],
perspective(fovy, aspect, near, far) {
const f = 1 / Math.tan(fovy / 2), nf = 1 / (near - far);
return new Float32Array([f / aspect, 0, 0, 0, 0, f, 0, 0, 0, 0, (far + near) * nf, -1, 0, 0, 2 * far * near * nf, 0]);
},
// Inverse of a rigid transform (rotation + translation only).
invRigid(m) {
const o = new Float32Array(16);
for (let r = 0; r < 3; r++) for (let c = 0; c < 3; c++) o[c * 4 + r] = m[r * 4 + c];
const t = [m[12], m[13], m[14]];
o[12] = -(o[0] * t[0] + o[4] * t[1] + o[8] * t[2]);
o[13] = -(o[1] * t[0] + o[5] * t[1] + o[9] * t[2]);
o[14] = -(o[2] * t[0] + o[6] * t[1] + o[10] * t[2]);
o[15] = 1;
return o;
},
};
// ---------- panels ----------
// A panel is a Mac display or window: a video quad, a bar under it to grab,
// and a close button to the bar's left (as on Vision Pro).
const BAR_GAP = 0.03, BAR_H = 0.018, CLOSE_R = 0.013;
const panels = [];
let desktop = null;
function barOf(p) {
const w = Math.min(0.28, p.w * 0.3);
return { cx: 0, cy: -p.h / 2 - BAR_GAP, w, h: BAR_H };
}
function closeOf(p) { const b = barOf(p); return { cx: -b.w / 2 - 0.04, cy: b.cy, r: CLOSE_R }; }
// Text size buttons (A−, A+) to the bar's right, on window panels.
function zoomOf(p, dir) { const b = barOf(p); return { cx: b.w / 2 + (dir < 0 ? 0.04 : 0.075), cy: b.cy, r: CLOSE_R }; }
function placeFacing(p, pos, head) {
p.pos = pos;
// Upright, turned to face the viewer (yaw only), as windows do on Vision Pro.
p.yaw = Math.atan2(head[0] - pos[0], head[2] - pos[2]);
}
function matrixOf(p) { return M.panel(p.pos, p.yaw); }
// Where a ray meets the panel's plane, in panel metres (x right, y up), or null.
function hitPanel(p, origin, dir) {
const inv = M.invRigid(matrixOf(p));
const o = M.point(inv, origin), d = M.dir(inv, dir);
if (Math.abs(d[2]) < 1e-6) return null;
const t = -o[2] / d[2];
if (t <= 0) return null;
return { x: o[0] + d[0] * t, y: o[1] + d[1] * t, t };
}
// What a ray points at: the nearest panel part.
function pick(origin, dir) {
let best = null;
for (const p of panels) {
const h = hitPanel(p, origin, dir);
if (!h) continue;
let part = null;
const b = barOf(p), c = closeOf(p);
if (Math.abs(h.x) <= p.w / 2 && Math.abs(h.y) <= p.h / 2) part = "content";
else if (Math.abs(h.x - b.cx) <= b.w / 2 + 0.01 && Math.abs(h.y - b.cy) <= b.h / 2 + 0.012) part = "bar";
else if (Math.hypot(h.x - c.cx, h.y - c.cy) <= c.r + 0.008) part = "close";
else if (p.kind === "window") {
for (const dir of [-1, 1]) {
const z = zoomOf(p, dir);
if (Math.hypot(h.x - z.cx, h.y - z.cy) <= z.r + 0.008) part = dir < 0 ? "zoomOut" : "zoomIn";
}
}
if (part && (!best || h.t < best.t)) best = { panel: p, part, t: h.t, x: h.x, y: h.y, u: h.x / p.w + 0.5, v: 0.5 - h.y / p.h };
}
return best;
}
let spaceKey = "", layout = { windows: [], displays: [] }, mainDisplay = null;
async function refreshLayout() {
try {
const r = await fetch(`/space/windows?s=${encodeURIComponent(spaceKey)}`);
if (r.ok) layout = await r.json();
} catch (e) {}
}
// Replaceable in tests (the Mac's real stacking decides otherwise).
const hooks = { windowAt: (u, v) => windowAt(u, v) };
// The topmost Mac window under a point of the main display (u, v in 0..1).
function windowAt(u, v) {
const d = mainDisplay;
if (!d) return null;
const x = d.x + u * d.w, y = d.y + v * d.h;
return layout.windows.find(w => x >= w.x && x < w.x + w.w && y >= w.y && y < w.y + w.h) || null;
}
function openPanel(src, opts) {
const p = { src, kind: opts.kind, title: opts.title || "", w: opts.w || 1.2, h: opts.h || 0.7, pos: [0, 1.4, -1.2], yaw: 0,
zoom: { auto: true, base: 0, steps: 0 },
tex: null, texW: 0, texH: 0, pending: null, lastMove: 0, pendingMove: null };
p.stream = new MacStream({
src, space: spaceKey, max: opts.max || "2560", fps: "60",
onFrame(img, w, h, seq, decodedAt) {
// Keep only the newest picture until the next rendered frame.
if (p.pending) { p.stream.replaced(p.pending.seq, p.pending.decodedAt); p.pending.img.close(); }
p.pending = { img, w, h, seq, decodedAt };
},
onInfo(m) { if (m.title) p.title = m.title; },
onClosed() { removePanel(p, false); },
});
p.stream.start();
panels.push(p);
return p;
}
function removePanel(p, stop = true) {
const i = panels.indexOf(p);
if (i < 0) return;
panels.splice(i, 1);
if (stop) p.stream.stop(); // the Mac ends the stream; a separated window goes back
if (p.pending) { p.pending.img.close(); p.pending = null; }
if (p.tex && gl) gl.deleteTexture(p.tex);
if (p === desktop) desktop = null;
else sendArrange();
}
// Tell the Mac where the window panels are, left or right of the desktop, so
// it lays their displays out the same way: the Mac's own pointer then moves
// from the desktop into the panel beside it, as it looks in the room.
let arrangeTimer = 0;
function sendArrange() {
clearTimeout(arrangeTimer);
arrangeTimer = setTimeout(() => {
if (!desktop) return;
const inv = M.invRigid(matrixOf(desktop));
const list = panels.filter(p => p.kind === "window").map(p => ({ src: p.src, x: +M.point(inv, p.pos)[0].toFixed(3) }));
desktop.stream.send({ t: "arrange", panels: list });
}, 300);
}
// Text size: the window's own zoom (Cmd+= / Cmd+-), which browsers, editors,
// terminals and chat apps all have. Automatic by default: after a window
// panel is moved, its text follows its distance, a step per 12% nearer or
// farther than where it was first let go; the A−/A+ buttons take over by hand.
const ZOOM_STEP = 1.12;
function textSize(p, steps) {
if (!steps || !panels.includes(p)) return;
p.stream.focus();
const code = steps > 0 ? "Equal" : "Minus", key = steps > 0 ? "=" : "-";
for (let i = 0; i < Math.abs(steps); i++) {
setTimeout(() => {
p.stream.key(true, code, key, ["meta"]);
setTimeout(() => p.stream.key(false, code, key, ["meta"]), 30);
}, 120 + i * 90);
}
p.zoom.steps += steps;
}
function autoTextSize(p) {
if (p.kind !== "window" || !p.zoom.auto || !p.zoom.base) return;
const d = V.len(V.sub(p.pos, headNow.pos));
const want = Math.max(-6, Math.min(8, Math.round(Math.log(d / p.zoom.base) / Math.log(ZOOM_STEP))));
textSize(p, want - p.zoom.steps);
}
// Remembered placement of the desktop panel, in the room.
function saveDesktop() {
if (desktop) localStorage.setItem("macspace.desktop", JSON.stringify({ pos: desktop.pos, yaw: desktop.yaw, w: desktop.w }));
}
function restoreDesktop(head) {
try {
const s = JSON.parse(localStorage.getItem("macspace.desktop") || "null");
if (s && s.pos && s.pos.length === 3) { desktop.pos = s.pos; desktop.yaw = s.yaw; desktop.w = s.w; return; }
} catch (e) {}
// First time: a large screen a little below eye level, 1.2 m ahead.
const fwd = head.forward;
placeFacing(desktop, V.add(head.pos, [fwd[0] * 1.2, -0.12, fwd[2] * 1.2]), head.pos);
}
// ---------- steadying the laser ----------
// A One Euro filter (Casiez et al. 2012) per axis: heavy smoothing while the
// pointer moves slowly (hand tremor), little lag when it moves fast.
class OneEuro {
constructor(minCutoff = 1.2, beta = 0.35, dCutoff = 1) { Object.assign(this, { minCutoff, beta, dCutoff, x: null, dx: 0, t: 0 }); }
alpha(cutoff, dt) { const tau = 1 / (2 * Math.PI * cutoff); return 1 / (1 + tau / dt); }
filter(v, t) {
if (this.x === null) { this.x = v; this.t = t; return v; }
const dt = Math.max(1e-3, (t - this.t) / 1000);
this.t = t;
const d = (v - this.x) / dt;
this.dx += this.alpha(this.dCutoff, dt) * (d - this.dx);
const cutoff = this.minCutoff + this.beta * Math.abs(this.dx);
this.x += this.alpha(cutoff, dt) * (v - this.x);
return this.x;
}
}
// Smooths a pointer's ray (direction in radians-ish units, origin in metres)
// and keeps the last 200 ms, so a click can use where it pointed just before
// the trigger jerked it.
function steady(pt, origin, dir, t) {
if (!pt.f) pt.f = { d: [0, 1, 2].map(() => new OneEuro(1.0, 0.6)), o: [0, 1, 2].map(() => new OneEuro(1.5, 0.8)), hist: [] };
pt.origin = origin.map((v, i) => pt.f.o[i].filter(v, t));
pt.dir = V.norm(dir.map((v, i) => pt.f.d[i].filter(v, t)));
pt.f.hist.push({ t, origin: pt.origin, dir: pt.dir });
while (pt.f.hist.length && t - pt.f.hist[0].t > 200) pt.f.hist.shift();
}
// Where the pointer was `ago` ms ago (or now).
function pointedAgo(pt, ago) {
const h = pt.f && pt.f.hist;
if (!h || !h.length) return { origin: pt.origin, dir: pt.dir };
const t = h[h.length - 1].t - ago;
return h.find(e => e.t >= t) || h[h.length - 1];
}
// Click stabilisation: after the trigger goes down, the Mac's pointer stays
// put until the ray has clearly moved (1.2 degrees), so a click doesn't turn
// into a small drag.
const CLICK_HOLD_RAD = 1.2 * Math.PI / 180;
// ---------- pointers: controllers, hands, or the mouse ----------
// Each pointer has a ray; the trigger (select) clicks, squeeze grabs.
const pointers = new Map();
function pointerState(key) {
if (!pointers.has(key)) pointers.set(key, { key, origin: [0, 0, 0], dir: [0, 0, -1], hit: null, selecting: false,
grab: null, press: null, lastWheel: 0 });
return pointers.get(key);
}
// Called whenever a pointer's ray moves.
function pointerMoved(pt, head) {
pt.hit = pick(pt.origin, pt.dir);
if (pt.grab) return dragGrab(pt, head);
const press = pt.press;
if (press && press.panel === desktop && press.win && panels.includes(desktop)) {
// Dragging a window's title bar off the edge of the desktop: detach it.
const h = hitPanel(desktop, pt.origin, pt.dir);
const outside = !h || Math.abs(h.x) > desktop.w / 2 || Math.abs(h.y) > desktop.h / 2;
if (outside) return detach(pt, press, head);
}
const target = press ? press.panel : pt.hit && pt.hit.part === "content" ? pt.hit.panel : null;
if (!target) return;
if (press && pt.hold) {
if (Math.acos(Math.min(1, V.dot(pt.hold.dir, pt.dir))) < CLICK_HOLD_RAD) return;
pt.hold = null; // moved on purpose: it's a drag
}
const h = press ? hitPanel(target, pt.origin, pt.dir) : pt.hit;
if (!h) return;
const u = Math.min(1, Math.max(0, h.x / target.w + 0.5)), v = Math.min(1, Math.max(0, 0.5 - h.y / target.h));
if (press) press.last = { u, v };
sendMove(target, u, v);
}
// Moves go at most every 8 ms per panel.
function sendMove(p, u, v) {
const now = performance.now();
if (now - p.lastMove >= 8) { p.stream.move(u, v); p.lastMove = now; p.pendingMove = null; }
else p.pendingMove = { u, v };
}
function selectStart(pt, head) {
// Where it pointed just before the trigger pull jerked the controller.
const before = pointedAgo(pt, 80);
const hit = pick(before.origin, before.dir) || pt.hit;
if (!hit) return;
pt.hold = { dir: before.dir, until: true };
if (hit.part === "close") {
if (hit.panel === desktop) return leave();
return removePanel(hit.panel);
}
if (hit.part === "bar") return startGrab(pt, hit.panel, head);
if (hit.part === "zoomOut" || hit.part === "zoomIn") {
hit.panel.zoom.auto = false; // chosen by hand from now on
return textSize(hit.panel, hit.part === "zoomIn" ? 1 : -1);
}
// A click on the Mac. On the desktop, remember which window it started on,
// and whether it was on its title bar (the top 32 points).
const p = hit.panel, u = hit.u, v = hit.v;
let win = null;
if (p === desktop) {
const w = hooks.windowAt(u, v);
if (w && mainDisplay && (mainDisplay.y + v * mainDisplay.h) - w.y <= 32) win = w;
}
pt.press = { panel: p, win, last: { u, v }, down: { u, v } };
p.stream.down(u, v, 0);
}
function selectEnd(pt) {
const held = !!pt.hold;
pt.hold = null;
if (pt.grab && pt.grab.byTrigger) return endGrab(pt);
const press = pt.press;
pt.press = null;
// A clean click lets go where it went down.
if (press && panels.includes(press.panel)) press.panel.stream.up(held ? press.down.u : press.last.u, held ? press.down.v : press.last.v, 0);
}
function squeezeStart(pt, head) {
if (pt.hit) startGrab(pt, pt.hit.panel, head, false);
}
function squeezeEnd(pt) { if (pt.grab && !pt.grab.byTrigger) endGrab(pt); }
// Grabbing: the panel keeps its distance along the ray and its offset from
// the hit point, and turns to face the viewer. Thumbstick up/down pushes and
// pulls; left/right resizes.
function startGrab(pt, p, head, byTrigger = true) {
const h = hitPanel(p, pt.origin, pt.dir);
const dist = h ? h.t : V.len(V.sub(p.pos, pt.origin));
const at = V.add(pt.origin, V.scale(pt.dir, dist));
pt.grab = { panel: p, dist, offset: V.sub(p.pos, at), byTrigger };
panels.push(panels.splice(panels.indexOf(p), 1)[0]); // draw it last
}
function dragGrab(pt, head) {
const g = pt.grab;
if (!panels.includes(g.panel)) { pt.grab = null; return; }
const at = V.add(pt.origin, V.scale(pt.dir, g.dist));
placeFacing(g.panel, V.add(at, g.offset), head.pos);
}
function endGrab(pt) {
const g = pt.grab;
pt.grab = null;
if (!g || !panels.includes(g.panel)) return;
const p = g.panel;
sendArrange();
if (p === desktop) return saveDesktop();
// A window dropped back onto the desktop goes home, at its old text size.
if (desktop) {
const h = hitPanel(desktop, pt.origin, pt.dir);
if (h && Math.abs(h.x) <= desktop.w / 2 && Math.abs(h.y) <= desktop.h / 2 && h.t < g.dist + 0.3) {
textSize(p, -p.zoom.steps);
return setTimeout(() => removePanel(p), 150 + Math.abs(p.zoom.steps) * 90 + 200);
}
}
if (!p.zoom.base) p.zoom.base = V.len(V.sub(p.pos, headNow.pos)); // where it was pulled out to
else autoTextSize(p);
}
function thumbstick(pt, x, y, dt) {
if (pt.grab) {
const g = pt.grab;
g.dist = Math.min(6, Math.max(0.3, g.dist - y * dt * 1.5));
const s = 1 + x * dt * 0.8;
g.panel.w = Math.min(4, Math.max(0.2, g.panel.w * s));
g.panel.h = g.panel.w * (g.panel.aspect || 0.6);
return;
}
const hit = pt.hit;
if (!hit || hit.part !== "content" || (Math.abs(y) < 0.2 && Math.abs(x) < 0.2)) return;
const now = performance.now();
if (now - pt.lastWheel < 16) return;
pt.lastWheel = now;
hit.panel.stream.wheel(x * 24, y * 24, hit.u, hit.v);
}
// Pulling a window out of the desktop: finish the Mac-side drag, open the
// window as a panel of its own (on its own display on the Mac), and let it
// follow the pointer until the trigger is released.
function detach(pt, press, head) {
pt.press = null;
desktop.stream.up(press.last.u, press.last.v, 0);
const win = press.win, d = mainDisplay;
const scale = desktop.w / d.w; // metres per Mac point, as on the desktop panel
const p = openPanel(`separate:${win.id}`, { kind: "window", title: win.title || win.app, w: win.w * scale, h: (win.h + 40) * scale });
p.aspect = p.h / p.w;
const dist = Math.max(0.5, (hitPanel(desktop, pt.origin, pt.dir) || { t: 1.2 }).t - 0.15);
const at = V.add(pt.origin, V.scale(pt.dir, dist));
placeFacing(p, V.add(at, [0, -p.h / 2 + 0.03, 0]), head.pos);
pt.grab = { panel: p, dist, offset: V.sub(p.pos, at), byTrigger: true };
}
// ---------- rendering ----------
let gl = null, prog = null, quad = null, loc = {}, isGL2 = false, aniso = null;
const VS = `attribute vec2 a; uniform mat4 mvp; uniform vec2 size; varying vec2 uv; varying vec2 pm;
void main() { uv = vec2(a.x + 0.5, 0.5 - a.y); pm = a * size; gl_Position = mvp * vec4(a * size, 0.0, 1.0); }`;
const FS = `precision highp float; varying vec2 uv; varying vec2 pm; uniform sampler2D tex; uniform vec4 color;
uniform float useTex; uniform vec2 size; uniform float radius;
void main() {
vec2 d = abs(pm) - (size * 0.5 - vec2(radius));
float dist = length(max(d, 0.0)) - radius;
float a = clamp(0.5 - dist / fwidth(dist), 0.0, 1.0);
vec4 c = useTex > 0.5 ? texture2D(tex, uv) : color;
gl_FragColor = vec4(c.rgb, c.a * a * color.a);
}`;
function initGL(canvas, xrCompatible) {
// WebGL 2: mipmaps for textures of any size, so a panel smaller than its
// pixels in view is filtered down cleanly instead of shimmering.
gl = canvas.getContext("webgl2", { alpha: true, antialias: true, premultipliedAlpha: false, xrCompatible })
|| canvas.getContext("webgl", { alpha: true, antialias: true, premultipliedAlpha: false, xrCompatible });
isGL2 = typeof WebGL2RenderingContext !== "undefined" && gl instanceof WebGL2RenderingContext;
if (!isGL2) gl.getExtension("OES_standard_derivatives");
aniso = gl.getExtension("EXT_texture_filter_anisotropic");
const sh = (type, src) => {
const s = gl.createShader(type);
gl.shaderSource(s, type === gl.FRAGMENT_SHADER && !isGL2 ? "#extension GL_OES_standard_derivatives : enable\n" + src : src);
gl.compileShader(s);
if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s));
return s;
};
// WebGL 2 needs GLSL ES 3.00 for fwidth (the rounded corners).
const v3 = src => "#version 300 es\n" + src.replace(/attribute /g, "in ").replace(/varying /g, "out ");
const f3 = src => "#version 300 es\n" + src.replace(/varying /g, "in ").replace(/texture2D\(/g, "texture(")
.replace("void main() {", "out vec4 fragColor;\n void main() {").replace(/gl_FragColor/g, "fragColor");
prog = gl.createProgram();
gl.attachShader(prog, sh(gl.VERTEX_SHADER, isGL2 ? v3(VS) : VS));
gl.attachShader(prog, sh(gl.FRAGMENT_SHADER, isGL2 ? f3(FS) : FS));
gl.linkProgram(prog);
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) throw new Error(gl.getProgramInfoLog(prog));
for (const n of ["mvp", "size", "tex", "color", "useTex", "radius"]) loc[n] = gl.getUniformLocation(prog, n);
quad = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, quad);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-0.5, -0.5, 0.5, -0.5, -0.5, 0.5, 0.5, 0.5]), gl.STATIC_DRAW);
}
function upload(p) {
const f = p.pending;
if (!f) return null;
p.pending = null;
if (!p.tex) {
p.tex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, p.tex);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, isGL2 ? gl.LINEAR_MIPMAP_LINEAR : gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
if (aniso) gl.texParameterf(gl.TEXTURE_2D, aniso.TEXTURE_MAX_ANISOTROPY_EXT, Math.min(8, gl.getParameter(aniso.MAX_TEXTURE_MAX_ANISOTROPY_EXT)));
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
}
gl.bindTexture(gl.TEXTURE_2D, p.tex);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, f.img);
if (isGL2) gl.generateMipmap(gl.TEXTURE_2D);
f.img.close();
// Keep the panel's shape the picture's shape (a separated window can change size).
p.aspect = f.h / f.w;
p.h = p.w * p.aspect;
p.texW = f.w;
p.texH = f.h;
return { seq: f.seq, decodedAt: f.decodedAt, drawnAt: performance.now() };
}
function drawRect(vp, model, w, h, color, tex, radius) {
gl.uniformMatrix4fv(loc.mvp, false, M.mul(vp, model));
gl.uniform2f(loc.size, w, h);
gl.uniform4fv(loc.color, color);
gl.uniform1f(loc.useTex, tex ? 1 : 0);
gl.uniform1f(loc.radius, radius);
if (tex) gl.bindTexture(gl.TEXTURE_2D, tex);
gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);
}
function drawScene(vp) {
gl.useProgram(prog);
gl.bindBuffer(gl.ARRAY_BUFFER, quad);
const a = gl.getAttribLocation(prog, "a");
gl.enableVertexAttribArray(a);
gl.vertexAttribPointer(a, 2, gl.FLOAT, false, 0, 0);
gl.uniform1i(loc.tex, 0);
gl.enable(gl.BLEND);
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA);
gl.enable(gl.DEPTH_TEST);
const hovered = new Set([...pointers.values()].filter(p => p.hit).map(p => p.hit.panel.src + ":" + p.hit.part));
for (const p of panels) {
const m = matrixOf(p);
// A soft shadow behind, then the picture (grey until the first frame).
drawRect(vp, M.mul(m, M.panel([0.004, -0.006, -0.004], 0)), p.w + 0.012, p.h + 0.012, [0, 0, 0, 0.35], null, 0.022);
drawRect(vp, m, p.w, p.h, p.tex ? [1, 1, 1, 1] : [0.18, 0.19, 0.22, 0.95], p.tex, 0.016);
const b = barOf(p), c = closeOf(p), bh = hovered.has(p.src + ":bar"), ch = hovered.has(p.src + ":close");
drawRect(vp, M.mul(m, M.panel([b.cx, b.cy, 0], 0)), b.w, b.h, [1, 1, 1, bh ? 0.95 : 0.55], null, b.h / 2);
drawRect(vp, M.mul(m, M.panel([c.cx, c.cy, 0], 0)), c.r * 2, c.r * 2, ch ? [0.95, 0.3, 0.3, 0.95] : [1, 1, 1, 0.45], null, c.r);
if (p.kind === "window") {
for (const dir of [-1, 1]) {
const z = zoomOf(p, dir), on = hovered.has(p.src + (dir < 0 ? ":zoomOut" : ":zoomIn"));
const zm = M.mul(m, M.panel([z.cx, z.cy, 0], 0));
drawRect(vp, zm, z.r * 2, z.r * 2, [1, 1, 1, on ? 0.9 : 0.45], null, z.r);
const glyph = M.mul(zm, M.panel([0, 0, 0.001], 0));
drawRect(vp, glyph, z.r * 1.1, z.r * 0.22, [0.1, 0.1, 0.12, 1], null, 0);
if (dir > 0) drawRect(vp, glyph, z.r * 0.22, z.r * 1.1, [0.1, 0.1, 0.12, 1], null, 0);
}
}
}
// Rays, ending where they hit.
for (const pt of pointers.values()) {
const len = pt.hit ? pt.hit.t : 2;
const mid = V.add(pt.origin, V.scale(pt.dir, len / 2));
const yaw = Math.atan2(pt.dir[0], pt.dir[2]);
// A thin strip along the ray: x across (horizontal), y along.
const along = pt.dir, across = V.norm([Math.cos(yaw), 0, -Math.sin(yaw)]);
const normal = [along[1] * across[2] - along[2] * across[1], along[2] * across[0] - along[0] * across[2], along[0] * across[1] - along[1] * across[0]];
const model = new Float32Array([across[0], across[1], across[2], 0, along[0], along[1], along[2], 0,
normal[0], normal[1], normal[2], 0, mid[0], mid[1], mid[2], 1]);
drawRect(vp, model, 0.003, len, [1, 1, 1, 0.35], null, 0);
if (pt.hit) {
const m = M.mul(matrixOf(pt.hit.panel), M.panel([pt.hit.x, pt.hit.y, 0.002], 0));
drawRect(vp, m, 0.012, 0.012, [1, 1, 1, 0.9], null, 0.006);
}
}
}
function flushMoves() {
for (const p of panels) if (p.pendingMove && performance.now() - p.lastMove >= 8) sendMove(p, p.pendingMove.u, p.pendingMove.v);
}
// ---------- WebXR ----------
let xrSession = null, xrRef = null;
async function enterXR() {
const mode = (await navigator.xr.isSessionSupported("immersive-ar").catch(() => false)) ? "immersive-ar" : "immersive-vr";
xrSession = await navigator.xr.requestSession(mode, { optionalFeatures: ["local-floor", "hand-tracking"] });
const canvas = document.createElement("canvas");
initGL(canvas, true);
await gl.makeXRCompatible();
xrSession.updateRenderState({ baseLayer: new XRWebGLLayer(xrSession, gl, { alpha: true, antialias: true }) });
xrRef = await xrSession.requestReferenceSpace("local-floor").catch(() => xrSession.requestReferenceSpace("local"));
const on = (type, fn) => xrSession.addEventListener(type, ev => fn(pointerState(ev.inputSource), headNow));
on("selectstart", selectStart);
on("selectend", pt => selectEnd(pt));
on("squeezestart", squeezeStart);
on("squeezeend", pt => squeezeEnd(pt));
xrSession.addEventListener("end", () => { xrSession = null; $("ui").hidden = false; $("enter").disabled = false; say("Left the Mac space."); });
$("ui").hidden = true;
let placed = false, last = 0;
const frame = (t, xf) => {
if (!xrSession) return;
xrSession.requestAnimationFrame(frame);
const pose = xf.getViewerPose(xrRef);
if (!pose) return;
const hm = pose.transform.matrix;
headNow = { pos: [hm[12], hm[13], hm[14]], forward: V.norm([-hm[8], 0, -hm[10]]) };
if (!placed && desktop) { restoreDesktop(headNow); placed = true; }
const dt = last ? Math.min(0.1, (t - last) / 1000) : 0;
last = t;
for (const src of xrSession.inputSources) {
if (!src.targetRaySpace) continue;
const rp = xf.getPose(src.targetRaySpace, xrRef);
if (!rp) continue;
const m = rp.transform.matrix, pt = pointerState(src);
steady(pt, [m[12], m[13], m[14]], V.norm([-m[8], -m[9], -m[10]]), t);
pointerMoved(pt, headNow);
const axes = src.gamepad && src.gamepad.axes;
if (axes && axes.length >= 4) thumbstick(pt, axes[2], axes[3], dt);
}
for (const k of [...pointers.keys()]) if (typeof k === "object" && ![...xrSession.inputSources].includes(k)) pointers.delete(k);
flushMoves();
const shown = [];
for (const p of panels) { const f = upload(p); if (f) shown.push([p, f]); }
const layer = xrSession.renderState.baseLayer;
gl.bindFramebuffer(gl.FRAMEBUFFER, layer.framebuffer);
gl.clearColor(0, 0, 0, 0); // transparent: passthrough shows through
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
for (const view of pose.views) {
const vp = layer.getViewport(view);
gl.viewport(vp.x, vp.y, vp.width, vp.height);
drawScene(M.mul(view.projectionMatrix, view.transform.inverse.matrix));
}
const now = performance.now();
for (const [p, f] of shown) p.stream.presented(f.seq, f.decodedAt, f.drawnAt, now);
};
xrSession.requestAnimationFrame(frame);
}
let headNow = { pos: [0, 1.6, 0], forward: [0, 0, -1] };
// ---------- preview in a normal page (no headset) ----------
function preview() {
const canvas = $("gl");
initGL(canvas, false);
$("hint").hidden = false;
const head = { pos: [0, 1.6, 0.4], forward: [0, 0, -1] };
headNow = head;
const mouse = pointerState("mouse");
let placed = false;
const toRay = e => {
const r = canvas.getBoundingClientRect(), x = (e.clientX - r.left) / r.width * 2 - 1, y = 1 - (e.clientY - r.top) / r.height * 2;
const f = 1 / Math.tan(Math.PI / 6), aspect = r.width / r.height;
mouse.origin = head.pos.slice();
mouse.dir = V.norm([x * aspect / f, y / f, -1]);
pointerMoved(mouse, head);
};
canvas.addEventListener("pointermove", toRay);
canvas.addEventListener("pointerdown", e => {
toRay(e);
canvas.setPointerCapture(e.pointerId);
if (e.button === 2 || e.shiftKey) squeezeStart(mouse, head); else selectStart(mouse, head);
});
canvas.addEventListener("pointerup", e => {
toRay(e);
if (mouse.grab && !mouse.grab.byTrigger) squeezeEnd(mouse); else selectEnd(mouse);
});
canvas.addEventListener("contextmenu", e => e.preventDefault());
canvas.addEventListener("wheel", e => {
e.preventDefault();
if (mouse.grab) { thumbstick(mouse, 0, -Math.sign(e.deltaY) * 0.5, 0.1); return; }
if (mouse.hit && mouse.hit.part === "content") mouse.hit.panel.stream.wheel(e.deltaX, e.deltaY, mouse.hit.u, mouse.hit.v);
}, { passive: false });
const loop = () => {
requestAnimationFrame(loop);
if (!placed && desktop) { restoreDesktop(head); placed = true; }
const w = canvas.clientWidth, h = canvas.clientHeight;
if (canvas.width !== w * devicePixelRatio) { canvas.width = w * devicePixelRatio; canvas.height = h * devicePixelRatio; }
flushMoves();
const shown = [];
for (const p of panels) { const f = upload(p); if (f) shown.push([p, f]); }
gl.bindFramebuffer(gl.FRAMEBUFFER, null);
gl.viewport(0, 0, canvas.width, canvas.height);
gl.clearColor(0.06, 0.07, 0.09, 1);
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
const view = M.invRigid(M.panel(head.pos, 0));
drawScene(M.mul(M.perspective(Math.PI / 3, w / h, 0.05, 50), view));
for (const [p, f] of shown) p.stream.drawn(f.seq, f.decodedAt, f.drawnAt);
};
loop();
}
function leave() {
for (const p of [...panels]) removePanel(p);
if (xrSession) xrSession.end();
}
// ---------- start ----------
(async () => {
const ticket = q.get("t") || "";
try {
const r = await fetch(`/space/key?t=${encodeURIComponent(ticket)}`);
if (!r.ok) throw new Error(r.status === 403 ? "This link has expired. Open the Mac space again from Frame Control." : `HTTP ${r.status}`);
spaceKey = (await r.json()).key;
} catch (e) { say(String(e.message || e)); $("enter").textContent = "Can't connect"; return; }
await refreshLayout();
setInterval(refreshLayout, 2000);
mainDisplay = layout.displays.find(d => d.main) || layout.displays[0] || null;
const src = q.get("src") || (mainDisplay ? `display:${mainDisplay.id}` : "test");
desktop = openPanel(src, { kind: "desktop", w: 1.25, h: mainDisplay ? 1.25 * mainDisplay.h / mainDisplay.w : 0.7, max: "2560" });
desktop.aspect = desktop.h / desktop.w;
// For tests over DevTools: drive a pointer the way a controller would.
window.space = { panels, pointers, layout: () => layout, desktop: () => desktop, head: () => headNow, V, M, matrixOf,
pointerState, pointerMoved, selectStart, selectEnd, squeezeStart, squeezeEnd, mainDisplay: () => mainDisplay, hooks };
const xr = navigator.xr && await navigator.xr.isSessionSupported("immersive-vr").catch(() => false);
if (!xr || q.get("preview") === "1") {
$("ui").hidden = true;
return preview();
}
$("enter").disabled = false;
$("enter").textContent = "Enter Mac space";
$("enter").onclick = () => enterXR().catch(e => { say(`Couldn't start: ${e.message || e}`); });
})();
</script>
</body>
</html>
+321
View File
@@ -0,0 +1,321 @@
// Frame Control: one stream from the Mac's frame-mac-view agent, for the
// viewer pages (mac-view.html: one panel; mac-space.html: the Mac space).
// It connects over the agent's WebSocket, keeps a clock synced with the
// Mac's, decodes H.264 with WebCodecs (or JPEG), and acknowledges and
// reports on every frame: the agent paces itself on the acks
// (Controller.swift) and measures each frame's journey from the reports
// (Stats.swift). Input goes back the same way. Plain script, no modules:
// it defines window.MacStream.
"use strict";
// What this browser draws with, and how often it gives an animation frame
// when nothing else is going on (only for the numbers).
const MacEnv = (() => {
const env = { gl: "", rafHz: 0, decoder: "" };
try {
const gl = document.createElement("canvas").getContext("webgl");
const ext = gl && gl.getExtension("WEBGL_debug_renderer_info");
env.gl = ext ? gl.getParameter(ext.UNMASKED_RENDERER_WEBGL) : gl ? "webgl" : "no webgl";
} catch (e) { env.gl = "?"; }
let n = 0, t0 = 0;
const tick = t => {
if (!t0) t0 = t;
if (t - t0 < 1000) { n++; return requestAnimationFrame(tick); }
env.rafHz = Math.round(n * 1000 / (t - t0));
};
requestAnimationFrame(tick);
return env;
})();
async function macPickCodec(want) {
if (want === "jpeg" || !("VideoDecoder" in window)) return "jpeg";
try {
const r = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", optimizeForLatency: true });
if (!r.supported) return "jpeg";
// Whether this browser has a hardware H.264 decoder (for the numbers).
const hw = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", hardwareAcceleration: "prefer-hardware" })
.catch(() => ({ supported: false }));
MacEnv.decoder = hw.supported ? "h264 hardware" : "h264 software";
return "h264";
} catch (e) { return "jpeg"; }
}
function macStartCode(b, i) { return b[i] === 0 && b[i + 1] === 0 && (b[i + 2] === 1 || (b[i + 2] === 0 && b[i + 3] === 1)); }
function macSpsCodec(au) {
for (let i = 0; i + 7 < au.length; i++) {
if (!macStartCode(au, i)) continue;
const n = au[i + 2] === 1 ? i + 3 : i + 4;
if ((au[n] & 0x1f) === 7) return "avc1." + [au[n + 1], au[n + 2], au[n + 3]].map(b => b.toString(16).padStart(2, "0")).join("");
i = n;
}
return "";
}
class MacStream {
// opts: src; one of ticket (single use, from Frame Control) or space (a
// Mac space key); codec, max, fps, bpp (optional); and callbacks:
// onFrame(image, width, height, seq, decodedAt): a VideoFrame or
// ImageBitmap, now the caller's to draw and close(). Call drawn() or
// presented() once it's on screen.
// onInfo(m), onStatus(text, sticky), onError(message), onClosed(reason),
// onStats(m), onBench(m)
constructor(opts) {
this.o = opts;
this.src = opts.src || "test";
this.stats = { frames: 0, keyFrames: 0, bytes: 0, dropped: 0, codec: "", errors: [], info: null };
this.clock = { off: null, rtt: 0, samples: [], reported: 0 };
this.reconnectKey = "";
this.ws = null;
this.dec = null;
this.codec = "";
this.codecString = "";
this.needKey = true;
this.closing = false;
this.retry = 0;
this.failedStarts = 0;
this.lastError = "";
this.lastKeyAsk = 0;
this.reports = [];
this.dropsToReport = 0;
this.shownPending = null;
this.rafQueued = false;
this.inputId = 0;
this.warmMs = 0;
this.warmUntil = 0;
this.warmTimer = 0;
this.jpegBusy = false;
this.pingTimer = 0;
// Decoded, drawn and shown times go back in batches.
this.reportTimer = setInterval(() => {
if (!this.reports.length && !this.dropsToReport) return;
this.send({ t: "fd", f: this.reports, drop: this.dropsToReport });
this.reports = [];
this.dropsToReport = 0;
}, 250);
}
// ---- clock: the Mac measures each frame's journey on its own clock ----
// NTP-style: the Mac's time minus ours, from the ping with the shortest
// round trip lately (the least queueing, so the least error).
toMac(ms) { return this.clock.off === null ? null : Math.round(ms * 1000 + this.clock.off); }
onPong(m) {
const now = performance.now(), rtt = now - m.c, clock = this.clock;
clock.samples.push({ rtt, off: m.a - (m.c + now) / 2 * 1000 });
if (clock.samples.length > 16) clock.samples.shift();
const best = clock.samples.reduce((a, b) => (b.rtt < a.rtt ? b : a));
clock.off = best.off;
clock.rtt = best.rtt;
if (now - clock.reported > 1000) {
clock.reported = now;
this.send({ t: "clock", rtt: +best.rtt.toFixed(2), dec: `${MacEnv.decoder}; ${MacEnv.gl}; raf ${MacEnv.rafHz} Hz` });
}
}
startPings() {
clearInterval(this.pingTimer);
this.clock.samples = [];
for (let i = 0; i < 10; i++) setTimeout(() => this.send({ t: "ping", c: performance.now() }), i * 40);
this.pingTimer = setInterval(() => this.send({ t: "ping", c: performance.now() }), 1000);
}
// ---- reports ----
dropped() { this.stats.dropped++; this.dropsToReport++; }
// A frame drawn on a page: it's on screen from the next animation frame,
// unless another replaces it first (then it was never seen).
drawn(seq, decodedAt, drawnAt = performance.now()) {
this.stats.frames++;
if (this.shownPending) {
const p = this.shownPending;
this.reports.push([p.seq, this.toMac(p.dec), this.toMac(p.drawn), 0]);
}
this.shownPending = { seq, dec: decodedAt, drawn: drawnAt };
if (this.rafQueued) return;
this.rafQueued = true;
requestAnimationFrame(() => {
this.rafQueued = false;
const p = this.shownPending, now = performance.now();
this.shownPending = null;
if (p && this.clock.off !== null) this.reports.push([p.seq, this.toMac(p.dec), this.toMac(p.drawn), this.toMac(now)]);
});
}
// A decoded frame a newer one replaced before it was drawn.
replaced(seq, decodedAt) {
if (this.clock.off !== null) this.reports.push([seq, this.toMac(decodedAt), 0, 0]);
}
// A frame drawn into a WebXR frame: shown when that frame is.
presented(seq, decodedAt, drawnAt, shownAt) {
this.stats.frames++;
if (this.clock.off !== null) this.reports.push([seq, this.toMac(decodedAt), this.toMac(drawnAt), this.toMac(shownAt)]);
}
// ---- connection ----
send(obj) { if (this.ws && this.ws.readyState === 1) this.ws.send(JSON.stringify(obj)); }
askKeyFrame() {
const now = performance.now();
if (now - this.lastKeyAsk < 500) return;
this.lastKeyAsk = now;
this.send({ t: "key-frame" });
}
status(text, sticky) { if (this.o.onStatus) this.o.onStatus(text, sticky); }
async start() {
this.codec = await macPickCodec(this.o.codec);
this.stats.codec = this.codec;
this.connect();
}
connect() {
if (this.codec === "h264") this.makeDecoder();
this.needKey = true;
const o = this.o;
const p = new URLSearchParams({ src: this.src, codec: this.codec, max: o.max || "1920", fps: o.fps || "60" });
if (this.reconnectKey) p.set("r", this.reconnectKey);
else if (o.space) p.set("s", o.space);
else p.set("t", o.ticket || "");
if (o.bpp) p.set("bpp", o.bpp);
const ws = this.ws = new WebSocket(`ws://${location.host}/stream?${p}`);
ws.binaryType = "arraybuffer";
ws.onopen = () => { this.retry = 0; this.lastError = ""; this.startPings(); };
ws.onmessage = ev => {
if (typeof ev.data === "string") return this.control(JSON.parse(ev.data));
const now = performance.now(), b = new Uint8Array(ev.data);
this.stats.bytes += b.length;
if (b.length < 18) return;
// flags (1 = keyframe), pts µs, sequence number, input echoed; big-endian.
const v = new DataView(ev.data), isKey = (b[0] & 1) === 1, seq = v.getUint32(9);
this.send({ t: "rx", s: seq, r: this.toMac(now) });
const payload = b.subarray(17);
if (this.codec === "h264") this.onH264(isKey, seq, payload); else this.onJPEG(seq, payload);
};
ws.onclose = () => {
if (this.ws === ws) this.ws = null;
clearInterval(this.pingTimer);
if (this.closing) return;
// Refused before ever getting a key: the ticket expired or was revoked,
// and retrying can't help.
if (!this.reconnectKey && ++this.failedStarts >= 3) {
this.status("This view has expired. Press Show in Frame Control on the Mac to open it again.", true);
if (o.onClosed) o.onClosed("expired");
return;
}
// The Mac may be asleep or the tunnel restarting: keep trying.
this.retry = Math.min(this.retry + 1, 6);
// Keep the Mac's reason (a missing permission, a closed window) on screen.
this.status(this.lastError ? `${this.lastError} Retrying…` : "Lost the Mac. Reconnecting…", true);
setTimeout(() => { if (!this.closing) this.connect(); }, 500 * 2 ** this.retry);
};
}
control(m) {
const o = this.o;
if (m.t === "pong") return this.onPong(m);
if (m.t === "stats") return o.onStats && o.onStats(m);
if (m.t === "bench") return o.onBench && o.onBench(m);
if (m.t === "hello") {
this.reconnectKey = m.r;
this.send({ t: "ack" }); // the ticket is spent only now
} else if (m.t === "info") {
this.stats.info = m;
this.warmMs = Math.max(0, +m.warm || 0);
if (o.onInfo) o.onInfo(m);
} else if (m.t === "error") {
this.stats.errors.push(m.message);
this.lastError = m.message.replace(/\.?$/, ".");
if (o.onError) o.onError(m.message);
} else if (m.t === "close" || m.t === "closed") {
this.stop();
if (o.onClosed) o.onClosed(m.reason || "");
}
}
stop() {
this.closing = true;
clearInterval(this.pingTimer);
clearInterval(this.reportTimer);
clearInterval(this.warmTimer);
if (this.ws) try { this.ws.close(); } catch (e) {}
if (this.dec) try { this.dec.close(); } catch (e) {}
}
// ---- H.264 ----
makeDecoder() {
if (this.dec) try { this.dec.close(); } catch (e) {}
this.codecString = "";
this.dec = new VideoDecoder({
// The chunk's timestamp is the Mac's sequence number, to match reports up.
output: frame => {
if (this.closing) return frame.close();
this.o.onFrame(frame, frame.displayWidth, frame.displayHeight, frame.timestamp, performance.now());
},
error: e => {
this.stats.errors.push(String(e.message || e));
this.needKey = true;
if (this.closing) return;
this.makeDecoder();
this.askKeyFrame();
},
});
}
onH264(isKey, seq, au) {
const dec = this.dec;
if (isKey) {
const cs = macSpsCodec(au);
if (cs && (cs !== this.codecString || dec.state !== "configured")) {
if (dec.state === "closed") this.makeDecoder();
this.dec.configure({ codec: cs, optimizeForLatency: true, hardwareAcceleration: "no-preference" });
this.codecString = this.stats.codec = cs;
}
this.stats.keyFrames++;
}
if (this.dec.state !== "configured" || (this.needKey && !isKey)) { this.dropped(); this.askKeyFrame(); return; }
// Far behind: skip to the next keyframe rather than show old pictures late.
// A few queued frames are fine: decoding catches up faster than a keyframe
// crosses a slow link, and only the newest decoded frame gets drawn.
if (!isKey && this.dec.decodeQueueSize > 10) { this.needKey = true; this.dropped(); this.askKeyFrame(); return; }
this.needKey = false;
this.dec.decode(new EncodedVideoChunk({ type: isKey ? "key" : "delta", timestamp: seq, data: au }));
}
// ---- JPEG ----
onJPEG(seq, data) {
if (this.jpegBusy) { this.dropped(); return; }
this.jpegBusy = true;
createImageBitmap(new Blob([data], { type: "image/jpeg" }))
.then(bmp => { if (this.closing) bmp.close(); else this.o.onFrame(bmp, bmp.width, bmp.height, seq, performance.now()); })
.catch(e => this.stats.errors.push(String(e)))
.finally(() => { this.jpegBusy = false; });
}
// ---- input ----
// x, y are 0..1 across the picture. Discrete input carries an id and when
// it happened (the Mac's clock), so the Mac can tag the first frame that
// could show its effect.
stamp(m, at) {
this.keepWarm();
m.i = ++this.inputId;
m.tv = this.toMac(at || performance.now());
return m;
}
// Keep the Frame's Wi-Fi awake for a few seconds after input, when the agent asks.
keepWarm() {
if (!this.warmMs) return;
this.warmUntil = performance.now() + 5000;
if (this.warmTimer) return;
this.warmTimer = setInterval(() => {
if (performance.now() > this.warmUntil) { clearInterval(this.warmTimer); this.warmTimer = 0; return; }
this.send({ t: "w" });
}, this.warmMs);
}
move(x, y) { this.send({ t: "m", e: "move", x, y }); }
down(x, y, button = 0, at) { this.send(this.stamp({ t: "m", e: "down", b: button, x, y }, at)); }
up(x, y, button = 0) { this.send({ t: "m", e: "up", b: button, x, y }); }
wheel(dx, dy, x, y, at) { this.send(this.stamp({ t: "wheel", dx, dy, x, y }, at)); }
key(down, code, key, mods, at) {
const m = { t: "k", e: down ? "down" : "up", code, key, mods };
this.send(down ? this.stamp(m, at) : m);
}
text(s, at) { this.send(this.stamp({ t: "text", s }, at)); }
release() { this.send({ t: "release" }); }
focus() { this.send({ t: "focus" }); }
}
window.MacStream = MacStream;
window.MacEnv = MacEnv;
+47 -265
View File
@@ -24,83 +24,22 @@
<canvas id="c" width="16" height="9"></canvas>
<div id="note">Connecting to your computer…</div>
<div id="hud" hidden></div>
<script src="/mac-stream.js"></script>
<script>
"use strict";
const q = new URLSearchParams(location.search);
// t: a single-use ticket from Frame Control. After the first connection the
// Mac sends a reconnect key, kept only in memory.
const src = q.get("src") || "test", ticket = q.get("t") || "", tag = q.get("tag") || "";
let reconnectKey = "";
// Mac sends a reconnect key, which the stream keeps in memory only.
const src = q.get("src") || "test", tag = q.get("tag") || "";
const c = document.getElementById("c"), ctx = c.getContext("2d", { alpha: false, desynchronized: true });
const note = document.getElementById("note");
// Counters for Frame Control's tests (read over DevTools).
const stats = window.stats = { frames: 0, keyFrames: 0, bytes: 0, dropped: 0, codec: "", errors: [], info: null };
// ---- timing: the Mac measures each frame's journey on its own clock ----
// Clock sync, NTP-style: the Mac's time minus ours, from the ping with the
// shortest round trip lately (the least queueing, so the least error).
const clock = { off: null, rtt: 0, samples: [], reported: 0 };
function toMac(ms) { return clock.off === null ? null : Math.round(ms * 1000 + clock.off); }
function onPong(m) {
const now = performance.now(), rtt = now - m.c;
clock.samples.push({ rtt, off: m.a - (m.c + now) / 2 * 1000 });
if (clock.samples.length > 16) clock.samples.shift();
const best = clock.samples.reduce((a, b) => (b.rtt < a.rtt ? b : a));
clock.off = best.off;
clock.rtt = best.rtt;
if (now - clock.reported > 1000) {
clock.reported = now;
send({ t: "clock", rtt: +best.rtt.toFixed(2), dec: `${decoderInfo}; ${glInfo}; raf ${rafHz} Hz` });
}
let noteTimer = 0, hideNoteOnFrame = true; // "Connecting…"/"Reconnecting…" go once video flows again
function show(text, ms) {
note.textContent = text;
note.hidden = false;
clearTimeout(noteTimer);
if (ms) noteTimer = setTimeout(() => { note.hidden = true; }, ms);
}
let pingTimer = 0;
function startPings() {
clearInterval(pingTimer);
clock.samples = [];
for (let i = 0; i < 10; i++) setTimeout(() => send({ t: "ping", c: performance.now() }), i * 40);
pingTimer = setInterval(() => send({ t: "ping", c: performance.now() }), 1000);
}
// Decoded, drawn and next-animation-frame times go back in batches.
let reports = [], dropsToReport = 0, shownPending = null, rafQueued = false;
setInterval(() => {
if (!reports.length && !dropsToReport) return;
send({ t: "fd", f: reports, drop: dropsToReport });
reports = [];
dropsToReport = 0;
}, 250);
function dropped() { stats.dropped++; dropsToReport++; }
// A drawn frame is on screen from the next animation frame, unless another
// replaces it first (then it was never seen).
function shown(seq, decodedAt, drawnAt) {
if (shownPending) reports.push([shownPending.seq, toMac(shownPending.dec), toMac(shownPending.drawn), 0]);
shownPending = { seq, dec: decodedAt, drawn: drawnAt };
if (rafQueued) return;
rafQueued = true;
requestAnimationFrame(() => {
rafQueued = false;
const p = shownPending, now = performance.now();
shownPending = null;
if (p && clock.off !== null) reports.push([p.seq, toMac(p.dec), toMac(p.drawn), toMac(now)]);
});
}
let decoderInfo = "";
// What this browser draws with, and how often it gives an animation frame
// when nothing else is going on (both only for the numbers).
let glInfo = "", rafHz = 0;
try {
const gl = document.createElement("canvas").getContext("webgl");
const ext = gl && gl.getExtension("WEBGL_debug_renderer_info");
glInfo = ext ? gl.getParameter(ext.UNMASKED_RENDERER_WEBGL) : gl ? "webgl" : "no webgl";
} catch (e) { glInfo = "?"; }
(function measureRaf() {
let n = 0, t0 = 0;
const tick = t => {
if (!t0) t0 = t;
if (t - t0 < 1000) { n++; return requestAnimationFrame(tick); }
rafHz = Math.round(n * 1000 / (t - t0));
};
requestAnimationFrame(tick);
})();
const hud = document.getElementById("hud");
function showHud(on) { hud.hidden = !on; }
function ms(o) { return o && o.p50 !== undefined ? `${o.p50}/${o.p95}` : "–"; }
@@ -111,177 +50,41 @@ function onStats(m) {
`\nlatency p50/p95 ms total ${ms(m.total)}\n capture ${ms(m.capture)} queue ${ms(m.queue)} encode ${ms(m.encode)}` +
`\n network ${ms(m.network)} decode ${ms(m.decode)} draw ${ms(m.draw)}` +
`\ninput→shown p50 ${m.input ? m.input.p50.toFixed(0) + " ms" : "–"} rtt ${m.rtt} ms` +
`\nskipped ${m.skipped}/${m.captured} dropped ${m.dropped} ${decoderInfo}`;
`\nskipped ${m.skipped}/${m.captured} dropped ${m.dropped} ${MacEnv.decoder}`;
}
showHud(q.get("stats") === "1");
// Frame Control finds this window on the Frame by the tag in its title.
document.title = tag ? `Mac [${tag}]` : "Mac";
let failedStarts = 0;
let ws = null, dec = null, codecString = "", needKey = true, closing = false, retry = 0, noteTimer = 0, lastError = "";
function show(text, ms) {
note.textContent = text;
note.hidden = false;
clearTimeout(noteTimer);
if (ms) noteTimer = setTimeout(() => { note.hidden = true; }, ms);
}
function send(obj) { if (ws && ws.readyState === 1) ws.send(JSON.stringify(obj)); }
let lastKeyAsk = 0;
function askKeyFrame() {
const now = performance.now();
if (now - lastKeyAsk < 500) return;
lastKeyAsk = now;
send({ t: "key-frame" });
}
let hideNoteOnFrame = true; // "Connecting…"/"Reconnecting…" go once video flows again
function drawFrame(img, w, h, seq, decodedAt) {
if (c.width !== w || c.height !== h) { c.width = w; c.height = h; }
ctx.drawImage(img, 0, 0);
shown(seq, decodedAt, performance.now());
stats.frames++;
if (hideNoteOnFrame) { hideNoteOnFrame = false; note.hidden = true; }
}
// ---- H.264 ----
function startCode(b, i) { return b[i] === 0 && b[i + 1] === 0 && (b[i + 2] === 1 || (b[i + 2] === 0 && b[i + 3] === 1)); }
function spsCodec(au) {
for (let i = 0; i + 7 < au.length; i++) {
if (!startCode(au, i)) continue;
const n = au[i + 2] === 1 ? i + 3 : i + 4;
if ((au[n] & 0x1f) === 7) return "avc1." + [au[n + 1], au[n + 2], au[n + 3]].map(b => b.toString(16).padStart(2, "0")).join("");
i = n;
}
return "";
}
function makeDecoder() {
if (dec) try { dec.close(); } catch (e) {}
codecString = "";
dec = new VideoDecoder({
// The chunk's timestamp is the Mac's sequence number, to match reports up.
output: frame => { drawFrame(frame, frame.displayWidth, frame.displayHeight, frame.timestamp, performance.now()); frame.close(); },
error: e => {
stats.errors.push(String(e.message || e));
needKey = true;
makeDecoder();
askKeyFrame();
},
});
}
function onH264(isKey, seq, au) {
if (isKey) {
const cs = spsCodec(au);
if (cs && (cs !== codecString || dec.state !== "configured")) {
if (dec.state === "closed") makeDecoder();
dec.configure({ codec: cs, optimizeForLatency: true, hardwareAcceleration: "no-preference" });
codecString = stats.codec = cs;
}
stats.keyFrames++;
}
if (dec.state !== "configured" || (needKey && !isKey)) { dropped(); askKeyFrame(); return; }
// Far behind: skip to the next keyframe rather than show old pictures late.
// A few queued frames are fine: decoding catches up faster than a keyframe
// crosses a slow link, and only the newest decoded frame gets drawn.
if (!isKey && dec.decodeQueueSize > 10) { needKey = true; dropped(); askKeyFrame(); return; }
needKey = false;
dec.decode(new EncodedVideoChunk({ type: isKey ? "key" : "delta", timestamp: seq, data: au }));
}
// ---- JPEG ----
let jpegBusy = false;
function onJPEG(seq, data) {
if (jpegBusy) { dropped(); return; }
jpegBusy = true;
createImageBitmap(new Blob([data], { type: "image/jpeg" })).then(bmp => {
drawFrame(bmp, bmp.width, bmp.height, seq, performance.now());
bmp.close();
}).catch(e => stats.errors.push(String(e))).finally(() => { jpegBusy = false; });
}
async function pickCodec() {
const want = q.get("codec");
if (want === "jpeg") return "jpeg";
if (!("VideoDecoder" in window)) return "jpeg";
try {
const r = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", optimizeForLatency: true });
if (!r.supported) return "jpeg";
// Whether this browser has a hardware H.264 decoder (for the numbers).
const hw = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", hardwareAcceleration: "prefer-hardware" })
.catch(() => ({ supported: false }));
decoderInfo = hw.supported ? "h264 hardware" : "h264 software";
return "h264";
} catch (e) { return "jpeg"; }
}
async function connect() {
const codec = await pickCodec();
stats.codec = codec;
if (codec === "h264") makeDecoder();
needKey = true;
const p = new URLSearchParams({ src, codec, max: q.get("max") || "1920", fps: q.get("fps") || "60" });
if (reconnectKey) p.set("r", reconnectKey); else p.set("t", ticket);
if (q.get("bpp")) p.set("bpp", q.get("bpp"));
ws = new WebSocket(`ws://${location.host}/stream?${p}`);
ws.binaryType = "arraybuffer";
ws.onopen = () => { retry = 0; lastError = ""; startPings(); };
ws.onmessage = ev => {
if (typeof ev.data === "string") return control(JSON.parse(ev.data));
const now = performance.now(), b = new Uint8Array(ev.data);
stats.bytes += b.length;
if (b.length < 18) return;
// flags (1 = keyframe), pts µs, sequence number, input echoed; big-endian.
const v = new DataView(ev.data), isKey = (b[0] & 1) === 1, seq = v.getUint32(9);
send({ t: "rx", s: seq, r: toMac(now) });
const payload = b.subarray(17);
if (codec === "h264") onH264(isKey, seq, payload); else onJPEG(seq, payload);
};
ws.onclose = () => {
ws = null;
clearInterval(pingTimer);
if (closing) return;
// Refused before ever getting a key: the ticket expired or was revoked,
// and retrying can't help.
if (!reconnectKey && ++failedStarts >= 3) {
show("This view has expired. Press Show in Frame Control on your computer to open it again.");
return;
}
hideNoteOnFrame = true;
// The Mac may be asleep or the tunnel restarting: keep trying.
retry = Math.min(retry + 1, 6);
// Keep the Mac's reason (a missing permission, a closed window) on screen.
show(lastError ? `${lastError} Retrying…` : "Lost the computer. Reconnecting…");
setTimeout(connect, 500 * 2 ** retry);
};
}
function control(m) {
if (m.t === "pong") return onPong(m);
if (m.t === "stats") return onStats(m);
if (m.t === "bench") return bench(m);
if (m.t === "hello") {
reconnectKey = m.r;
send({ t: "ack" }); // the ticket is spent only now
} else if (m.t === "info") {
stats.info = m;
warmMs = Math.max(0, +m.warm || 0);
const stream = new MacStream({
src, ticket: q.get("t") || "", codec: q.get("codec"), max: q.get("max"), fps: q.get("fps"), bpp: q.get("bpp"),
onFrame(img, w, h, seq, decodedAt) {
if (c.width !== w || c.height !== h) { c.width = w; c.height = h; }
ctx.drawImage(img, 0, 0);
img.close();
stream.drawn(seq, decodedAt);
if (hideNoteOnFrame) { hideNoteOnFrame = false; note.hidden = true; }
},
onStatus(text) { hideNoteOnFrame = true; show(text); },
onInfo(m) {
const name = m.title && m.app && m.title !== m.app ? `${m.title} — ${m.app}` : (m.title || m.app || "Mac");
document.title = tag ? `${name} [${tag}]` : name;
if (!m.input) {
hideNoteOnFrame = false; // a warning, not a connection notice: let it stay its 8 s
show(m.inputMessage || "Clicks and keys need Accessibility permission on the Mac (Frame Control asks for it).", 8000);
}
} else if (m.t === "error") {
stats.errors.push(m.message);
lastError = m.message.replace(/\.?$/, ".");
show(m.message);
} else if (m.t === "close" || m.t === "closed") {
closing = true;
if (ws) ws.close();
show(m.reason ? `Stopped: ${m.reason}.` : "Stopped.");
},
onError(message) { show(message); },
onClosed(reason) {
if (reason === "expired") return;
show(reason ? `Stopped: ${reason}.` : "Stopped.");
window.close();
}
}
},
onStats,
onBench: bench,
});
// Counters for Frame Control's tests (read over DevTools).
window.stats = stream.stats;
// ---- input ----
// Where the picture sits inside the window (object-fit: contain letterboxes it).
@@ -291,39 +94,19 @@ function toPicture(e) {
const x = (e.clientX - left) / w, y = (e.clientY - top) / h;
return { x, y, inside: x >= 0 && x <= 1 && y >= 0 && y <= 1 };
}
// Discrete input carries an id and when it happened (the Mac's clock), so
// the Mac can tag the first frame that could show its effect.
let inputId = 0;
// Keep the Frame's Wi-Fi awake for a few seconds after input, when the agent asks.
let warmMs = 0, warmUntil = 0, warmTimer = 0;
function keepWarm() {
if (!warmMs) return;
warmUntil = performance.now() + 5000;
if (warmTimer) return;
warmTimer = setInterval(() => {
if (performance.now() > warmUntil) { clearInterval(warmTimer); warmTimer = 0; return; }
send({ t: "w" });
}, warmMs);
}
function stamp(m, e) {
keepWarm();
m.i = ++inputId;
m.tv = toMac(e && e.timeStamp ? e.timeStamp : performance.now());
return m;
}
// Moves go at most every 8 ms, on a timer rather than the next animation frame:
// the Frame throttles a panel's animation frames to 15-36 Hz when it thinks
// nobody is looking, which would hold a move back by up to 60 ms.
let pendingMove = null, moveQueued = false, lastMove = 0;
function flushMove() {
moveQueued = false;
if (pendingMove) { send(pendingMove); lastMove = performance.now(); }
if (pendingMove) { stream.move(pendingMove.x, pendingMove.y); lastMove = performance.now(); }
pendingMove = null;
}
addEventListener("pointermove", e => {
const p = toPicture(e);
if (!p.inside && !e.buttons) return;
pendingMove = { t: "m", e: "move", x: p.x, y: p.y };
pendingMove = p;
if (moveQueued) return;
const wait = lastMove + 8 - performance.now();
if (wait <= 0) return flushMove();
@@ -335,19 +118,19 @@ addEventListener("pointerdown", e => {
if (!p.inside) return;
if (e.pointerId !== undefined) try { c.setPointerCapture(e.pointerId); } catch (err) {}
pendingMove = null;
send(stamp({ t: "m", e: "down", b: e.button < 0 ? 0 : e.button, x: p.x, y: p.y }, e));
stream.down(p.x, p.y, e.button < 0 ? 0 : e.button, e.timeStamp);
e.preventDefault();
});
addEventListener("pointerup", e => {
const p = toPicture(e);
send({ t: "m", e: "up", b: e.button < 0 ? 0 : e.button, x: p.x, y: p.y });
stream.up(p.x, p.y, e.button < 0 ? 0 : e.button);
});
// Let go of any held button where the pointer is on the Mac, not at a corner.
addEventListener("pointercancel", () => send({ t: "release" }));
addEventListener("pointercancel", () => stream.release());
addEventListener("contextmenu", e => e.preventDefault());
addEventListener("wheel", e => {
const p = toPicture(e), unit = e.deltaMode === 1 ? 16 : e.deltaMode === 2 ? innerHeight : 1;
send(stamp({ t: "wheel", dx: e.deltaX * unit, dy: e.deltaY * unit, x: p.x, y: p.y }, e));
stream.wheel(e.deltaX * unit, e.deltaY * unit, p.x, p.y, e.timeStamp);
e.preventDefault();
}, { passive: false });
function mods(e) {
@@ -359,8 +142,7 @@ function onKey(e, down) {
if (down) showHud(hud.hidden);
return e.preventDefault();
}
const m = { t: "k", e: down ? "down" : "up", code: e.code, key: e.key, mods: mods(e) };
send(down ? stamp(m, e) : m);
stream.key(down, e.code, e.key, mods(e), e.timeStamp);
e.preventDefault();
}
@@ -373,26 +155,26 @@ function keyCode(ch) {
function bench(m) {
if (m.action === "overlay") return showHud(m.on !== 0 && m.on !== "0");
if (m.action === "click") {
const at = { x: +m.x || 0.5, y: +m.y || 0.5 };
send(stamp({ t: "m", e: "down", b: 0, ...at }));
setTimeout(() => send({ t: "m", e: "up", b: 0, ...at }), 40);
const x = +m.x || 0.5, y = +m.y || 0.5;
stream.down(x, y, 0);
setTimeout(() => stream.up(x, y, 0), 40);
} else if (m.action === "type") {
const text = String(m.text || ""), gap = +m.interval || 150;
[...text].forEach((ch, n) => setTimeout(() => {
const code = keyCode(ch);
if (!code) return send(stamp({ t: "text", s: ch }));
if (!code) return stream.text(ch);
const mods = ch !== ch.toLowerCase() ? ["shift"] : [];
send(stamp({ t: "k", e: "down", code, key: ch, mods }));
setTimeout(() => send({ t: "k", e: "up", code, key: ch, mods }), 30);
stream.key(true, code, ch, mods);
setTimeout(() => stream.key(false, code, ch, mods), 30);
}, n * gap));
}
}
addEventListener("keydown", e => onKey(e, true));
addEventListener("keyup", e => onKey(e, false));
addEventListener("blur", () => send({ t: "release" }));
addEventListener("blur", () => stream.release());
show("Connecting to your computer…");
connect();
stream.start();
</script>
</body>
</html>