Compare commits

...
Author SHA1 Message Date
saphid ae9df35442 Add Android phone companion with automatic screenshot downloads 2026-10-24 21:22:57 +11:00
saphid ce27c0e8f1 Add durable foreground iOS screenshot downloads and Photos preference 2026-10-24 16:43:44 +11:00
saphid 5277232367 Merge remote-tracking branch 'origin/main' into feat/phone-screenshot-autosync-108
# Conflicts:
#	README.md
#	ui/frame_macview.py
#	ui/index.html
#	ui/mac-view.html
2026-10-24 16:31:05 +11:00
saphid 3fdc6c46c3 Document Mac capture and live-video verification 2026-10-17 13:01:09 +11:00
saphid 9a6b48dea5 iPhone: record physical feature tests and remaining checks 2026-10-16 20:36:44 +11:00
saphid a3e2b70555 iPhone: record approved beta and phone test blocker 2026-10-16 20:11:41 +11:00
saphid 092aae24bd iPhone: document cloud signing access and the public beta link 2026-10-16 19:33:28 +11:00
saphid 7adf0805d7 iPhone: declare encryption compliance and document TestFlight releases
Declare no non-exempt encryption in the generated app, document upload and tester setup, and record the physical iPhone connection check and remaining tests. Refs #100.
2026-10-16 19:29:45 +11:00
saphidandClaude Opus 5.5 7f23acb03f iPhone: show/hide toggle on the Developer Mode password; TestFlight upload script
The setup screen's password field gets an eye button (44 pt target) that
swaps between hidden and visible text and keeps focus.

ios/scripts/testflight.sh archives with automatic signing and uploads to
App Store Connect using an API key (ASC_KEY_ID, ASC_ISSUER_ID,
ASC_KEY_PATH); the build number is the UTC time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:24:37 +11:00
saphid d9454ef099 Improve Frame Control website onboarding and audit fixes 2026-09-30 19:56:02 +10:00
saphidandClaude Opus 5.5 71a3a96dd2 Mac space: fix separated windows landing on the wrong display; steadier laser; room-matched displays; text size
From Alex's first try in the headset:
- Separated windows could end up on another window's display, so panels
  showed the wrong window or froze. New virtual displays were slotted in
  next to the main one, pushing earlier ones along while a window was being
  placed; and displays under 800x600 points never got their 2x mode.
  Now each display goes left of the leftmost one, the window is placed
  only once the display is in its 2x mode and checked afterwards, and
  displays are at least 800x600 (probed: smaller ones have no usable 2x
  mode on macOS 26). Verified: detach A, detach B, dock A, detach A again,
  each panel streams its own window at 2x.
- Laser: One Euro smoothing on the ray; a click uses where it pointed
  80 ms before the trigger pull, and the Mac pointer holds still until the
  ray moves 1.2 degrees, so clicks don't become drags.
- The Mac lays the separated windows' displays out left/right of the main
  one as their panels sit in the room, so the Mac's own pointer crosses
  into the panel beside it. Verified in the preview.
- Text size: A-/A+ on window panels send the window's own Cmd+-/Cmd+=;
  automatic by default, following distance after a move. WebGL 2 with
  mipmaps and anisotropic filtering, so distant panels don't shimmer.
- The agent logs each stream's start/end and Separate mode's steps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 23:09:15 +10:00
saphidandClaude Opus 5.5 952b660bf7 Mac space: launch Chromium XR with OpenXR on and the seccomp filter off (as its own launcher does)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 22:45:26 +10:00
saphidandClaude Opus 5.5 a4d840a148 Mac space (first version): the Mac's screen in the room, windows pulled out as their own panels
- docs/mac-space.md: what Vision Pro's Mac Virtual Display does, what the
  Frame allows (another process can't move SteamVR's panels: verified,
  PermissionDenied), and why a WebXR page comes first.
- ui/mac-stream.js: the stream protocol shared by the viewers (connect,
  clock sync, decode, acks/reports, reconnect, input); mac-view.html uses it.
- ui/mac-space.html: WebXR (passthrough behind) or an in-page preview.
  Desktop panel with a Vision Pro-style bar and close button; laser clicks,
  thumbstick scrolls, squeeze or the bar grabs; dragging a window's title
  bar off the desktop opens it on its own display as a new panel that
  follows the pointer; dropping it back sends it home.
- Agent: space tickets -> a space key (/space/key) that lists windows and
  displays (/space/windows, now with positions) and opens any stream;
  Stop All revokes it. Serves /space and /mac-stream.js.
- Frame Control: "Open Mac space" in the Tools card.

Verified on the Mac (preview in headless Chrome, lab agent): the main
display streams into the panel; a test window detached onto its own
display and streamed; dropped back, it returned to its exact frame and the
display was removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 22:24:36 +10:00
85 changed files with 6790 additions and 387 deletions

No files matched your search

+1 -1
View File
@@ -7,7 +7,7 @@ body:
value: |
**Before you start:** read [CONTRIBUTING.md](https://github.com/saphid/frame-control/blob/main/CONTRIBUTING.md).
Issues from new contributors are auto-closed by default. A maintainer reviews them and reopens worthwhile ones. The [website feedback form](https://frame-control.pages.dev/feedback/) skips that queue.
Issues from new contributors are auto-closed by default. A maintainer reviews them and reopens worthwhile ones. The [website feedback form](https://framecontrol.fun/feedback/) skips that queue.
Keep this short. If it doesn't fit on one screen, it's too long. Write in your own voice.
+1 -1
View File
@@ -1,5 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Feedback form (no GitHub account needed, skips the queue)
url: https://frame-control.pages.dev/feedback/
url: https://framecontrol.fun/feedback/
about: Bugs, ideas and questions from the website become issues here without being auto-closed.
+22
View File
@@ -0,0 +1,22 @@
name: Android phone
on:
pull_request:
paths: ['android/**', 'ui/**', 'scripts/**', 'frame/**', 'apk-catalog/**', 'ios/scripts/make_frame_bundle.py', '.github/workflows/android.yml']
push:
branches: [main, 'feat/phone-screenshot-autosync-108']
paths: ['android/**', 'ui/**', 'scripts/**', 'frame/**', 'apk-catalog/**', 'ios/scripts/make_frame_bundle.py', '.github/workflows/android.yml']
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '17'
- uses: android-actions/setup-android@v3
- uses: gradle/actions/setup-gradle@v4
- run: sdkmanager 'platforms;android-36' 'build-tools;36.0.0'
- run: ./android/gradlew -p android assembleDebug assembleDebugAndroidTest testDebugUnitTest lintDebug --no-daemon
+1 -1
View File
@@ -104,7 +104,7 @@ jobs:
'',
`Maintainers review auto-closed issues regularly and reopen worthwhile ones. Issues that do not meet the quality bar in [CONTRIBUTING.md](https://github.com/${context.repo.owner}/${context.repo.repo}/blob/${defaultBranch}/CONTRIBUTING.md) will not be reopened or receive a reply.`,
'',
'Just want to report a bug or share an idea? The [website feedback form](https://frame-control.pages.dev/feedback/) skips this queue.',
'Just want to report a bug or share an idea? The [website feedback form](https://framecontrol.fun/feedback/) skips this queue.',
'',
'If a maintainer replies `lgtmi` on one of your issues, your future issues will stay open. If a maintainer replies `lgtm`, your future issues and PRs will stay open. The command must be at the start of the reply (optionally after one or more `@username` mentions) or at the end.',
'',
+17 -1
View File
@@ -12,7 +12,7 @@ See what the headset sees, install games and Android apps, move files and text a
[![Checks](https://img.shields.io/github/actions/workflow/status/saphid/steam-frame/checks.yml?branch=main&label=checks)](https://github.com/saphid/steam-frame/actions/workflows/checks.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-66c0f4)](LICENSE)
[**Website**](https://frame-control.pages.dev) · [**Download**](#install) · [Trailer](#trailer) · [Features](#features) · [Set up the headset](#set-up-the-headset) · [Feedback](#feedback) · [Docs](#going-further)
[**Get started**](https://framecontrol.fun) · [**Download**](https://framecontrol.fun/#download) · [Trailer](#trailer) · [Features](#features) · [Set up the headset](#set-up-the-headset) · [Feedback](#feedback) · [Docs](#going-further)
<br>
@@ -116,6 +116,17 @@ already ships (sideloading a game copies Valve's own devkit scripts to
## Install
**New here? [Get Frame Control and follow the setup guide on the website](https://framecontrol.fun/#download).**
Install it on your **computer, not the headset**. You don't need to clone this
repository or install Python. Download for your computer, install and open the
app, then choose **Set Up Connection** with the Frame awake on the same network.
- **macOS:** open the `.dmg`, drag Frame Control to Applications, then open it there.
- **Windows:** run the installer and launch Frame Control from Start, or extract the portable `.zip` and run `Frame Control.exe`.
- **Linux:** make the AppImage executable and open it, or install the `.deb` and launch Frame Control.
If the app is blocked, see the first-launch notes below before proceeding.
| | Download | Needs |
|---|---|---|
| **macOS** (Apple Silicon) | [Frame-Control-mac-arm64.dmg](https://github.com/saphid/steam-frame/releases/latest/download/Frame-Control-mac-arm64.dmg) | Nothing extra |
@@ -126,6 +137,10 @@ already ships (sideloading a game copies Valve's own devkit scripts to
**iPhone and iPad:** the same features from your phone, with nothing to install on
a computer. Build it from [`ios/`](ios) in Xcode; see [docs/iphone.md](docs/iphone.md).
**Android phones (development):** a native companion is in [`android/`](android).
It can download new screenshots during a user-started sync session, with Gallery
saving enabled by default. See [build instructions and current testing limits](docs/android-phone.md).
The app brings its own Python and `adb`; SSH is built into macOS and Windows.
From 0.4 it updates itself: when a new version is published, a banner offers
**Update and restart**. It sends anonymous usage statistics, which you can turn
@@ -234,6 +249,7 @@ Frame's software fits together, all checked against a real headset and labelled
| [VR mods and custom songs](docs/mods.md) | UEVR for Unreal games from Frame Control (install, play in VR, remove), the per-game table, Beat Saber blockers |
| [SSH](docs/ssh.md) · [Streaming](docs/streaming.md) · [Files](docs/file-transfer.md) · [Panels](docs/panels.md) · [Tailscale](docs/tailscale.md) | Topic notes |
| [Frame Control for iPhone](docs/iphone.md) | The iPhone and iPad app, how it runs the server on the Frame, pairing |
| [Frame Control for Android phones](docs/android-phone.md) | Development build, pairing, screenshot downloads and foreground sync limits |
| [Recovery and OS images](docs/recovery-and-images.md) | Where to download the Frame's OS, what's inside, testing without the headset |
| [AI agents and assistant](docs/agents.md) | Key-free MCP tools, human approvals, and an opt-in assistant panel |
| [Testing](docs/testing.md) | Unit tests, end-to-end tests against a fake Frame in Docker, the headset smoke test and a Windows test VM |
+5
View File
@@ -0,0 +1,5 @@
.gradle/
**/build/
local.properties
*.keystore
*.jks
+35
View File
@@ -0,0 +1,35 @@
plugins { id 'com.android.application' }
android {
namespace 'com.saphid.framecontrol'
compileSdk 36
defaultConfig {
applicationId 'com.saphid.framecontrol'
minSdk 29
targetSdk 36
versionCode 1
versionName '0.1.0'
testInstrumentationRunner 'androidx.test.runner.AndroidJUnitRunner'
}
compileOptions { sourceCompatibility JavaVersion.VERSION_17; targetCompatibility JavaVersion.VERSION_17 }
sourceSets.main.assets.srcDir layout.buildDirectory.dir('generated/frameAssets').get().asFile
}
def frameAssets = tasks.register('frameAssets', Exec) {
workingDir rootProject.projectDir.parentFile
commandLine 'python3', 'android/scripts/make_assets.py', layout.buildDirectory.dir('generated/frameAssets').get().asFile
inputs.files(fileTree(rootProject.file('../ui')), fileTree(rootProject.file('../scripts')), fileTree(rootProject.file('../frame')), fileTree(rootProject.file('../apk-catalog')) { exclude 'data/**' })
inputs.file(rootProject.file('../ios/scripts/make_frame_bundle.py'))
inputs.file(rootProject.file('scripts/make_assets.py'))
outputs.dir layout.buildDirectory.dir('generated/frameAssets')
}
tasks.named('preBuild').configure { dependsOn frameAssets }
dependencies {
implementation 'com.github.mwiede:jsch:2.28.8'
implementation 'com.google.code.gson:gson:2.13.2'
implementation 'androidx.core:core:1.17.0'
testImplementation 'junit:junit:4.13.2'
androidTestImplementation 'androidx.test:runner:1.7.0'
androidTestImplementation 'androidx.test.ext:junit:1.3.0'
}
@@ -0,0 +1,409 @@
package com.saphid.framecontrol;
import static org.junit.Assert.*;
import android.app.Activity;
import android.app.Instrumentation;
import android.content.*;
import android.database.Cursor;
import android.graphics.*;
import android.net.Uri;
import android.provider.MediaStore;
import android.view.View;
import android.view.ViewGroup;
import android.widget.Button;
import androidx.test.ext.junit.runners.AndroidJUnit4;
import androidx.test.platform.app.InstrumentationRegistry;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.util.*;
import java.util.concurrent.atomic.AtomicReference;
import org.junit.*;
import org.junit.runner.RunWith;
@RunWith(AndroidJUnit4.class)
public class ScreenshotDeviceTest {
private Context context;
private final Set<Uri> cleanup = new LinkedHashSet<>();
@Before
public void setup() {
context = InstrumentationRegistry.getInstrumentation().getTargetContext();
}
@After
public void cleanup() {
for (Uri uri : cleanup) context.getContentResolver().delete(uri, null, null);
}
private String name(String headset, SyncEngine.Shot shot) throws Exception {
String file = shot.id.substring(shot.id.lastIndexOf('/') + 1);
int dot = file.lastIndexOf('.');
return file.substring(0, dot)
+ "_"
+ FrameConnection.hash(headset + "/" + shot.id).substring(0, 20)
+ file.substring(dot);
}
private Uri row(String headset, SyncEngine.Shot shot) throws Exception {
Uri collection = MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY);
@SuppressWarnings("deprecation")
Uri includingPending = MediaStore.setIncludePending(collection);
try (Cursor c =
context
.getContentResolver()
.query(
includingPending,
new String[] {"_id"},
MediaStore.Images.Media.DISPLAY_NAME
+ "=? AND "
+ MediaStore.Images.Media.RELATIVE_PATH
+ "=?",
new String[] {name(headset, shot), "Pictures/Frame Control/"},
null)) {
if (c != null && c.moveToFirst()) {
Uri uri = ContentUris.withAppendedId(collection, c.getLong(0));
cleanup.add(uri);
assertFalse("Duplicate Gallery row", c.moveToNext());
return uri;
}
return null;
}
}
@Test
public void actualGalleryOriginalAndAppOnlySurviveRelaunch() throws Exception {
String headset = "instrumented-" + UUID.randomUUID();
File root = new File(context.getFilesDir(), "instrumented/" + headset);
Object gate = new Object();
Bitmap bitmap = Bitmap.createBitmap(48, 32, Bitmap.Config.ARGB_8888);
bitmap.eraseColor(Color.rgb(25, 184, 143));
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, bytes);
bitmap.recycle();
byte[] png = bytes.toByteArray();
SyncEngine.Transport transport =
new SyncEngine.Transport() {
public void download(SyncEngine.Shot shot, File target) throws Exception {
Files.write(target.toPath(), png);
}
public boolean validImage(File f) {
Bitmap b = BitmapFactory.decodeFile(f.getPath());
if (b == null) return false;
b.recycle();
return true;
}
};
GalleryExporter exporter = new GalleryExporter(context, headset, gate);
SyncEngine engine = new SyncEngine(root, true);
engine.reconcile(Collections.emptyList(), transport, exporter);
SyncEngine.Shot shot =
new SyncEngine.Shot("999999999991/108/20261024000000_1.png", png.length, 1792800000);
engine.reconcile(List.of(shot), transport, exporter);
assertNull(row(headset, shot));
engine.reconcile(List.of(shot), transport, exporter);
Uri saved = row(headset, shot);
assertNotNull(saved);
try (InputStream in = context.getContentResolver().openInputStream(saved)) {
assertArrayEquals(png, read(in));
}
assertEquals("saved", engine.items().get(0).destination);
engine = new SyncEngine(root, true);
engine.reconcile(List.of(shot), transport, exporter);
assertEquals(saved, row(headset, shot));
engine.setGallery(false);
SyncEngine.Shot second =
new SyncEngine.Shot("999999999991/108/20261024000001_1.png", png.length, 1792800001);
engine.reconcile(List.of(shot, second), transport, exporter);
engine.reconcile(List.of(shot, second), transport, exporter);
engine.setGallery(true);
engine = new SyncEngine(root, true);
engine.reconcile(List.of(shot, second), transport, exporter);
assertNull(row(headset, second));
assertTrue(
engine.items().stream()
.anyMatch(
i ->
i.shot.id.equals(second.id)
&& i.destination.equals("appOnly")
&& i.file.exists()));
}
@Test
public void interruptedMediaStoreRowIsRewrittenWithoutDuplicate() throws Exception {
String headset = "pending-" + UUID.randomUUID();
SyncEngine.Shot shot =
new SyncEngine.Shot("999999999992/108/20261024000002_1.png", 4, 1792800002);
ContentValues values = new ContentValues();
values.put(MediaStore.Images.Media.DISPLAY_NAME, name(headset, shot));
values.put(MediaStore.Images.Media.RELATIVE_PATH, "Pictures/Frame Control/");
values.put(MediaStore.Images.Media.MIME_TYPE, "image/png");
values.put(MediaStore.Images.Media.IS_PENDING, 1);
Uri pending =
context
.getContentResolver()
.insert(
MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY), values);
assertNotNull(pending);
cleanup.add(pending);
File file = File.createTempFile("gallery-proof", ".png", context.getCacheDir());
byte[] original = {1, 2, 3, 4};
Files.write(file.toPath(), original);
GalleryExporter exporter = new GalleryExporter(context, headset, new Object());
assertTrue(exporter.export(shot, file, () -> true));
assertEquals(pending, row(headset, shot));
assertTrue(exporter.export(shot, file, () -> true));
assertEquals(pending, row(headset, shot));
try (InputStream in = context.getContentResolver().openInputStream(pending)) {
assertArrayEquals(original, read(in));
}
file.delete();
}
@Test
public void galleryOptOutLatchesPreviouslyPairedLibraries() throws Exception {
assertFalse("Instrumentation must own a stopped product", SyncService.running());
File files = Files.createTempDirectory(context.getCacheDir().toPath(), "libraries-").toFile();
String preferences = "library-test-" + UUID.randomUUID();
Context isolated =
new ContextWrapper(context) {
@Override
public File getFilesDir() {
return files;
}
@Override
public SharedPreferences getSharedPreferences(String name, int mode) {
return context.getSharedPreferences(preferences, mode);
}
};
try {
SyncEngine.Transport transfer =
new SyncEngine.Transport() {
public void download(SyncEngine.Shot shot, File target) throws Exception {
Files.write(target.toPath(), new byte[] {1, 2, 3, 4});
}
public boolean validImage(File file) {
return true;
}
};
SyncEngine.Exporter unavailable = (shot, file, eligible) -> false;
SyncEngine.Shot shot = new SyncEngine.Shot("123/108/20261024120000_1.png", 4, 100);
File a = SyncService.library(isolated, "headset-a");
File b = SyncService.library(isolated, "headset-b");
for (File root : List.of(a, b)) {
SyncEngine engine = new SyncEngine(root, true);
engine.reconcile(Collections.emptyList(), transfer, unavailable);
engine.reconcile(List.of(shot), transfer, unavailable);
engine.reconcile(List.of(shot), transfer, unavailable);
assertEquals("pending", engine.items().get(0).destination);
}
assertTrue(
SyncService.prefs(isolated)
.edit()
.putString("pin", "headset-b")
.putBoolean("gallery", true)
.commit());
SyncService.gallery(isolated, false);
SyncService.gallery(isolated, true);
for (File root : List.of(a, b)) {
SyncEngine engine = new SyncEngine(root, true);
assertEquals("appOnly", engine.items().get(0).destination);
engine.reconcile(
List.of(shot),
transfer,
(s, f, eligible) -> {
fail("Opted-out items must never be exported after re-pairing");
return false;
});
}
} finally {
SyncService.prefs(isolated).edit().clear().commit();
try (java.util.stream.Stream<java.nio.file.Path> paths = Files.walk(files.toPath())) {
for (java.nio.file.Path path :
paths.sorted(Comparator.reverseOrder()).collect(java.util.stream.Collectors.toList()))
Files.delete(path);
}
}
}
@Test
public void changingHeadsetsRebindsShareToTheSelectedOriginal() throws Exception {
assertFalse(SyncService.running());
String aPin = "switch-a-" + UUID.randomUUID(), bPin = "switch-b-" + UUID.randomUUID();
File a = SyncService.library(context, aPin), b = SyncService.library(context, bPin);
SharedPreferences prefs = SyncService.prefs(context);
String previousPin = prefs.getString("pin", null);
boolean previousGallery = prefs.getBoolean("gallery", true);
Instrumentation instrumentation = InstrumentationRegistry.getInstrumentation();
AtomicReference<Intent> shared = new AtomicReference<>();
Instrumentation.ActivityMonitor monitor =
new Instrumentation.ActivityMonitor() {
@Override
public Instrumentation.ActivityResult onStartActivity(Intent intent) {
if (!Intent.ACTION_CHOOSER.equals(intent.getAction())) return null;
shared.set(intent.getParcelableExtra(Intent.EXTRA_INTENT));
return new Instrumentation.ActivityResult(Activity.RESULT_CANCELED, null);
}
};
Activity activity = null;
try {
SyncEngine.Shot shot = new SyncEngine.Shot("123/108/20261024120000_1.png", 4, 100);
for (File root : List.of(a, b)) {
byte[] original = root.equals(a) ? new byte[] {1, 2, 3, 4} : new byte[] {5, 6, 7, 8};
SyncEngine.Transport transfer =
new SyncEngine.Transport() {
public void download(SyncEngine.Shot s, File file) throws Exception {
Files.write(file.toPath(), original);
}
public boolean validImage(File file) {
return true;
}
};
SyncEngine engine = new SyncEngine(root, false);
engine.reconcile(Collections.emptyList(), transfer, (s, f, e) -> false);
engine.reconcile(List.of(shot), transfer, (s, f, e) -> false);
engine.reconcile(List.of(shot), transfer, (s, f, e) -> false);
}
assertTrue(prefs.edit().putString("pin", aPin).putBoolean("gallery", false).commit());
activity =
instrumentation.startActivitySync(
new Intent(context, MainActivity.class)
.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK | Intent.FLAG_ACTIVITY_CLEAR_TASK));
instrumentation.addMonitor(monitor);
instrumentation.waitForIdleSync();
Activity screen = activity;
Runnable share =
() -> {
Button button = shareButton(screen.findViewById(android.R.id.content));
assertNotNull("The downloaded original must be reachable", button);
assertTrue(button.performClick());
};
instrumentation.runOnMainSync(share);
assertNotNull(shared.get());
Uri first = shared.get().getParcelableExtra(Intent.EXTRA_STREAM);
try (InputStream in = context.getContentResolver().openInputStream(first)) {
assertArrayEquals(new byte[] {1, 2, 3, 4}, read(in));
}
assertTrue(prefs.edit().putString("pin", bPin).commit());
long deadline = System.nanoTime() + java.util.concurrent.TimeUnit.SECONDS.toNanos(5);
Uri selected;
do {
instrumentation.runOnMainSync(share);
selected = shared.get().getParcelableExtra(Intent.EXTRA_STREAM);
if (!first.equals(selected)) break;
Thread.sleep(100);
} while (System.nanoTime() < deadline);
assertNotEquals("Share must no longer target the previous headset", first, selected);
try (InputStream in = context.getContentResolver().openInputStream(selected)) {
assertArrayEquals(new byte[] {5, 6, 7, 8}, read(in));
}
} finally {
instrumentation.removeMonitor(monitor);
Activity screen = activity;
if (screen != null) instrumentation.runOnMainSync(screen::finish);
SharedPreferences.Editor restore = prefs.edit().putBoolean("gallery", previousGallery);
if (previousPin == null) restore.remove("pin");
else restore.putString("pin", previousPin);
assertTrue(restore.commit());
for (File root : List.of(a, b)) {
if (!root.exists()) continue;
try (java.util.stream.Stream<java.nio.file.Path> paths = Files.walk(root.toPath())) {
for (java.nio.file.Path path :
paths.sorted(Comparator.reverseOrder()).collect(java.util.stream.Collectors.toList()))
Files.delete(path);
}
}
}
}
private static Button shareButton(View view) {
if (view instanceof Button && "Open / share original".contentEquals(((Button) view).getText()))
return (Button) view;
if (view instanceof ViewGroup) {
ViewGroup group = (ViewGroup) view;
for (int i = 0; i < group.getChildCount(); i++) {
Button found = shareButton(group.getChildAt(i));
if (found != null) return found;
}
}
return null;
}
/** Optional real-headset provisioning. Only this test APK exposes the hook. */
@Test
public void provisionRealHeadsetIfRequested() throws Exception {
android.os.Bundle args = InstrumentationRegistry.getArguments();
Assume.assumeTrue("Optional real-headset provisioning", "true".equals(args.getString("live")));
String host = args.getString("host"), pin = args.getString("pin");
assertNotNull(host);
assertNotNull(pin);
assertEquals(
"Probe must return the expected host key before authentication",
pin,
FrameConnection.probe(host, 22, "steamos"));
String publicKey = new PhoneIdentity().authorizedKey(SyncService.install(context));
Files.write(
new File(context.getFilesDir(), "live-test-public.pub").toPath(),
(publicKey + "\n").getBytes(StandardCharsets.US_ASCII));
assertTrue(
SyncService.prefs(context)
.edit()
.putString("host", host)
.putString("user", "steamos")
.putInt("port", 22)
.putString("pin", pin)
.commit());
}
@Test
public void verifyRealHeadsetIfRequested() throws Exception {
android.os.Bundle args = InstrumentationRegistry.getArguments();
String id = args.getString("verifyId");
Assume.assumeNotNull(id);
String pin = SyncService.prefs(context).getString("pin", null);
assertNotNull(pin);
SyncEngine engine =
new SyncEngine(
SyncService.library(context, pin),
SyncService.prefs(context).getBoolean("gallery", true));
SyncEngine.Item item =
engine.items().stream().filter(i -> i.shot.id.equals(id)).findFirst().orElseThrow();
assertTrue(item.file.exists());
String expected = args.getString("sha256");
assertNotNull(expected);
assertEquals(expected, sha(Files.readAllBytes(item.file.toPath())));
boolean gallery = "true".equals(args.getString("gallery"));
Uri saved = row(pin, item.shot);
cleanup.clear();
if (gallery) {
assertEquals("saved", item.destination);
assertNotNull(saved);
try (InputStream in = context.getContentResolver().openInputStream(saved)) {
assertEquals(expected, sha(read(in)));
}
} else {
assertEquals("appOnly", item.destination);
assertNull(saved);
}
}
private static String sha(byte[] bytes) throws Exception {
StringBuilder out = new StringBuilder();
for (byte b : java.security.MessageDigest.getInstance("SHA-256").digest(bytes))
out.append(String.format(java.util.Locale.ROOT, "%02x", b & 255));
return out.toString();
}
private static byte[] read(InputStream input) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] b = new byte[4096];
int n;
while ((n = input.read(b)) != -1) out.write(b, 0, n);
return out.toByteArray();
}
}
+19
View File
@@ -0,0 +1,19 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.CHANGE_NETWORK_STATE" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_CONNECTED_DEVICE" />
<application android:label="Frame Control" android:theme="@style/AppTheme"
android:icon="@drawable/ic_frame" android:allowBackup="false" android:dataExtractionRules="@xml/data_extraction"
android:networkSecurityConfig="@xml/network_security" android:supportsRtl="true">
<activity android:name=".MainActivity" android:exported="true" android:configChanges="orientation|screenSize|keyboardHidden">
<intent-filter><action android:name="android.intent.action.MAIN" /><category android:name="android.intent.category.LAUNCHER" /></intent-filter>
</activity>
<service android:name=".SyncService" android:exported="false" android:foregroundServiceType="connectedDevice" />
<provider android:name="androidx.core.content.FileProvider" android:authorities="${applicationId}.files" android:exported="false" android:grantUriPermissions="true">
<meta-data android:name="android.support.FILE_PROVIDER_PATHS" android:resource="@xml/file_paths" />
</provider>
</application>
</manifest>
@@ -0,0 +1,414 @@
package com.saphid.framecontrol;
import android.content.Context;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonParser;
import com.jcraft.jsch.*;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.*;
import java.util.concurrent.*;
import java.util.regex.*;
/** All headset traffic goes through one pinned, authenticated SSH connection. */
final class FrameConnection implements AutoCloseable, SyncEngine.Transport {
private final Context context;
private Session session;
private ChannelExec server;
private PipedOutputStream serverInput;
private int localPort;
private String apiKey;
private static final String HOST_ALGORITHMS = "ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256";
FrameConnection(Context context) {
this.context = context.getApplicationContext();
}
static String fingerprint(String pin) throws Exception {
return "SHA256:"
+ Base64.getEncoder()
.withoutPadding()
.encodeToString(
MessageDigest.getInstance("SHA-256").digest(Base64.getDecoder().decode(pin)));
}
static String hash(String value) throws Exception {
byte[] bytes =
MessageDigest.getInstance("SHA-256").digest(value.getBytes(StandardCharsets.UTF_8));
StringBuilder result = new StringBuilder();
for (byte b : bytes) result.append(String.format(Locale.ROOT, "%02x", b & 255));
return result.toString();
}
static String quote(String s) {
return "'" + s.replace("'", "'\\''") + "'";
}
private static Session session(JSch jsch, String host, int port, String user) throws Exception {
if (host.trim().isEmpty()
|| host.length() > 253
|| host.contains("/")
|| host.chars().anyMatch(Character::isWhitespace)
|| port < 1
|| port > 65535
|| !user.matches("[a-zA-Z0-9_-]{1,32}"))
throw new IOException("Check the headset address, port and username.");
Session s = jsch.getSession(user, host, port);
s.setConfig("StrictHostKeyChecking", "yes");
s.setConfig("server_host_key", HOST_ALGORITHMS);
s.setConfig("PreferredAuthentications", "publickey");
s.setTimeout(20000);
s.setServerAliveInterval(15000);
s.setServerAliveCountMax(2);
return s;
}
private static final class PinRepository implements HostKeyRepository {
String observed;
final String expected;
PinRepository(String expected) {
this.expected = expected;
}
@Override
public int check(String host, byte[] key) {
observed = Base64.getEncoder().encodeToString(key);
return observed.equals(expected) ? OK : CHANGED;
}
@Override
public void add(HostKey k, UserInfo u) {}
@Override
public void remove(String h, String t) {}
@Override
public void remove(String h, String t, byte[] k) {}
@Override
public String getKnownHostsRepositoryID() {
return "Frame Control pinned headset";
}
@Override
public HostKey[] getHostKey() {
return new HostKey[0];
}
@Override
public HostKey[] getHostKey(String h, String t) {
return new HostKey[0];
}
}
/** Intentionally refuses the key exchange before any authentication is sent. */
static String probe(String host, int port, String user) throws Exception {
JSch jsch = new JSch();
PinRepository pin = new PinRepository(null);
jsch.setHostKeyRepository(pin);
Session s = session(jsch, host, port, user);
try {
s.connect(10000);
} catch (JSchException e) {
if (pin.observed == null) throw e;
} finally {
s.disconnect();
}
return pin.observed;
}
static void pair(String host, int port, String user, String pin, String password, String install)
throws Exception {
JSch jsch = new JSch();
jsch.setHostKeyRepository(new PinRepository(pin));
Session s = session(jsch, host, port, user);
s.setConfig("PreferredAuthentications", "password,keyboard-interactive");
s.setPassword(password);
s.setUserInfo(
new UserInfo() {
public String getPassphrase() {
return null;
}
public String getPassword() {
return password;
}
public boolean promptPassword(String x) {
return true;
}
public boolean promptPassphrase(String x) {
return false;
}
public boolean promptYesNo(String x) {
return false;
}
public void showMessage(String x) {}
});
try {
s.connect(15000);
String key = new PhoneIdentity().authorizedKey(install);
run(
s,
"umask 077; mkdir -p ~/.ssh && chmod 700 ~/.ssh && touch ~/.ssh/authorized_keys && chmod"
+ " 600 ~/.ssh/authorized_keys && (grep -qxF "
+ quote(key)
+ " ~/.ssh/authorized_keys || printf '%s\\n' "
+ quote(key)
+ " >> ~/.ssh/authorized_keys)",
10000);
} finally {
s.disconnect();
}
// Verify the new key before storing any pairing settings.
JSch verified = new JSch();
verified.setHostKeyRepository(new PinRepository(pin));
verified.addIdentity(new PhoneIdentity(), null);
Session check = session(verified, host, port, user);
try {
check.connect(15000);
} finally {
check.disconnect();
}
}
static String run(Session s, String command, int timeout) throws Exception {
ChannelExec channel = (ChannelExec) s.openChannel("exec");
channel.setCommand(command);
channel.setInputStream(null);
channel.setErrStream(
new OutputStream() {
public void write(int b) {}
public void write(byte[] b, int offset, int length) {}
});
InputStream input = channel.getInputStream();
long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(timeout);
ByteArrayOutputStream output = new ByteArrayOutputStream();
try {
channel.connect(timeout);
while (true) {
while (input.available() > 0) {
byte[] b = new byte[Math.min(input.available(), 8192)];
int n = input.read(b);
if (n > 0) output.write(b, 0, n);
if (output.size() > 2 * 1024 * 1024)
throw new IOException("Headset command returned too much data.");
}
if (channel.isClosed() && input.available() == 0) break;
if (System.nanoTime() > deadline) throw new IOException("The headset command timed out.");
Thread.sleep(25);
}
if (channel.getExitStatus() != 0)
throw new IOException("The headset command failed (" + channel.getExitStatus() + ").");
return output.toString(StandardCharsets.UTF_8.name());
} finally {
channel.disconnect();
}
}
void connect(String host, int port, String user, String pin, String install) throws Exception {
JSch jsch = new JSch();
jsch.setHostKeyRepository(new PinRepository(pin));
jsch.addIdentity(new PhoneIdentity(), null);
session = session(jsch, host, port, user);
session.connect(15000);
String version;
try (InputStream in = context.getAssets().open("frame-bundle.version")) {
version = new String(readBounded(in, 128), StandardCharsets.US_ASCII).trim();
}
if (!version.matches("[0-9a-f]{16}")) throw new IOException("Invalid bundled server version.");
String dir = ".cache/frame-control/" + version;
String present =
run(session, "test -f " + dir + "/ui/server.py && printf ready || true", 10000);
if (!present.equals("ready")) {
run(session, "mkdir -p .cache/frame-control", 10000);
String temp = dir + ".android-" + UUID.randomUUID();
ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");
try (InputStream in = context.getAssets().open("frame-bundle.tar.gz")) {
sftp.connect(10000);
sftp.put(in, temp + ".tar.gz");
} finally {
sftp.disconnect();
}
run(
session,
"mkdir "
+ temp
+ " && tar xzf "
+ temp
+ ".tar.gz -C "
+ temp
+ " && "
+ "(test -f "
+ dir
+ "/ui/server.py && rm -rf "
+ temp
+ " || mv "
+ temp
+ " "
+ dir
+ ") && rm -f "
+ temp
+ ".tar.gz",
60000);
}
apiKey = UUID.randomUUID().toString().replace("-", "");
server = (ChannelExec) session.openChannel("exec");
serverInput = new PipedOutputStream();
server.setInputStream(new PipedInputStream(serverInput));
server.setCommand(
"cd "
+ dir
+ " && FRAME_LOCAL=1 FRAME_UI_KEY="
+ apiKey
+ " FRAME_DEVICE="
+ quote(host)
+ " FRAME_CLIENT="
+ quote(install)
+ " exec python3 -I -u -B \"$PWD/ui/server.py\" --port 0 --exit-on-eof 2>&1");
InputStream input = server.getInputStream();
server.connect(10000);
ByteArrayOutputStream startup = new ByteArrayOutputStream();
Pattern portPattern = Pattern.compile("Frame Control on http://127\\.0\\.0\\.1:([0-9]+)\\s");
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(40);
int portFound = 0;
while (System.nanoTime() < deadline) {
while (input.available() > 0) {
startup.write(input.read());
if (startup.size() > 65536) throw new IOException("The headset server did not start.");
}
Matcher m = portPattern.matcher(startup.toString(StandardCharsets.UTF_8.name()));
if (m.find()) {
portFound = Integer.parseInt(m.group(1));
break;
}
if (server.isClosed())
throw new IOException("The headset server stopped before it was ready.");
Thread.sleep(50);
}
if (portFound < 1 || portFound > 65535)
throw new IOException("The headset server did not answer in time.");
localPort = session.setPortForwardingL("127.0.0.1", 0, "127.0.0.1", portFound);
// Drain the session-bound server output; never log session keys or user data.
Thread drain =
new Thread(
() -> {
try {
byte[] b = new byte[4096];
while (input.read(b) != -1) {}
} catch (IOException ignored) {
}
},
"frame-server-output");
drain.setDaemon(true);
drain.start();
}
String page() {
return "http://127.0.0.1:" + localPort + "/?key=" + apiKey;
}
boolean connected() {
return session != null && session.isConnected() && server != null && !server.isClosed();
}
private HttpURLConnection request(String path) throws Exception {
HttpURLConnection c =
(HttpURLConnection) new URL("http://127.0.0.1:" + localPort + path).openConnection();
c.setConnectTimeout(10000);
c.setReadTimeout(20000);
c.setInstanceFollowRedirects(false);
c.setRequestProperty("X-Frame-UI", apiKey);
return c;
}
List<SyncEngine.Shot> shots() throws Exception {
HttpURLConnection c = request("/api/shots");
try (InputStream in = c.getInputStream()) {
byte[] body = readBounded(in, 2 * 1024 * 1024);
JsonArray array =
JsonParser.parseString(new String(body, StandardCharsets.UTF_8))
.getAsJsonObject()
.getAsJsonArray("shots");
List<SyncEngine.Shot> shots = new ArrayList<>();
for (JsonElement e : array) {
var s = e.getAsJsonObject();
shots.add(
new SyncEngine.Shot(
s.get("id").getAsString(), s.get("size").getAsLong(), s.get("time").getAsDouble()));
}
return shots;
} finally {
c.disconnect();
}
}
private static byte[] readBounded(InputStream input, int limit) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] b = new byte[8192];
int n;
while ((n = input.read(b)) != -1) {
if (out.size() + n > limit) throw new IOException("The headset response is too large.");
out.write(b, 0, n);
}
return out.toByteArray();
}
@Override
public void download(SyncEngine.Shot shot, File destination) throws Exception {
HttpURLConnection c =
request("/api/shots/image?id=" + URLEncoder.encode(shot.id, StandardCharsets.UTF_8.name()));
try (InputStream in = c.getInputStream();
FileOutputStream out = new FileOutputStream(destination)) {
byte[] b = new byte[65536];
long count = 0;
int n;
while ((n = in.read(b)) != -1) {
count += n;
if (count > shot.size) throw new IOException("Screenshot changed while downloading.");
out.write(b, 0, n);
}
out.getFD().sync();
} finally {
c.disconnect();
}
}
@Override
public boolean validImage(File file) {
android.graphics.BitmapFactory.Options options = new android.graphics.BitmapFactory.Options();
options.inJustDecodeBounds = true;
android.graphics.BitmapFactory.decodeFile(file.getPath(), options);
if (options.outWidth <= 0
|| options.outHeight <= 0
|| options.outWidth > 16384
|| options.outHeight > 16384
|| (long) options.outWidth * options.outHeight > 60_000_000) return false;
options.inJustDecodeBounds = false;
options.inSampleSize = Math.max(1, Math.max(options.outWidth, options.outHeight) / 2048);
android.graphics.Bitmap b = android.graphics.BitmapFactory.decodeFile(file.getPath(), options);
if (b == null) return false;
b.recycle();
return true;
}
@Override
public void close() {
try {
if (serverInput != null) serverInput.close();
} catch (IOException ignored) {
}
if (server != null) server.disconnect();
if (session != null) session.disconnect();
}
}
@@ -0,0 +1,96 @@
package com.saphid.framecontrol;
import android.content.*;
import android.database.Cursor;
import android.net.Uri;
import android.provider.MediaStore;
import java.io.*;
import java.util.function.BooleanSupplier;
/** Deterministic, app-owned MediaStore rows make interrupted exports retryable. */
final class GalleryExporter implements SyncEngine.Exporter {
private final Context context;
private final String headset;
private final Object settingsLock;
GalleryExporter(Context context, String headset, Object settingsLock) {
this.context = context.getApplicationContext();
this.headset = headset;
this.settingsLock = settingsLock;
}
@Override
public boolean export(SyncEngine.Shot shot, File file, BooleanSupplier eligible)
throws Exception {
String name = shot.id.substring(shot.id.lastIndexOf('/') + 1);
int dot = name.lastIndexOf('.');
name =
name.substring(0, dot)
+ "_"
+ FrameConnection.hash(headset + "/" + shot.id).substring(0, 20)
+ name.substring(dot);
String folder = "Pictures/Frame Control/";
ContentResolver resolver = context.getContentResolver();
Uri collection = MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY);
// API 29 excludes even app-owned pending rows unless explicitly included.
@SuppressWarnings("deprecation")
Uri recoveryCollection = MediaStore.setIncludePending(collection);
Uri row = null;
boolean pending = true;
try (Cursor cursor =
resolver.query(
recoveryCollection,
new String[] {"_id", MediaStore.Images.Media.IS_PENDING},
MediaStore.Images.Media.DISPLAY_NAME
+ "=? AND "
+ MediaStore.Images.Media.RELATIVE_PATH
+ "=? AND "
+ MediaStore.Images.Media.OWNER_PACKAGE_NAME
+ "=?",
new String[] {name, folder, context.getPackageName()},
null)) {
if (cursor != null && cursor.moveToFirst()) {
row = ContentUris.withAppendedId(collection, cursor.getLong(0));
pending = cursor.getInt(1) != 0;
}
}
// A completed row is the commit receipt, including a crash before JSON save.
if (row != null && !pending) return true;
if (!eligible.getAsBoolean()) return false;
if (row == null) {
ContentValues values = new ContentValues();
values.put(MediaStore.Images.Media.DISPLAY_NAME, name);
values.put(MediaStore.Images.Media.RELATIVE_PATH, folder);
values.put(
MediaStore.Images.Media.MIME_TYPE, shot.id.endsWith(".png") ? "image/png" : "image/jpeg");
values.put(MediaStore.Images.Media.IS_PENDING, 1);
values.put(MediaStore.Images.Media.DATE_TAKEN, (long) (shot.time * 1000));
row = resolver.insert(collection, values);
if (row == null) throw new IOException("Gallery could not create the screenshot.");
}
try {
try (InputStream in = new FileInputStream(file);
OutputStream out = resolver.openOutputStream(row, "wt")) {
if (out == null) throw new IOException("Gallery could not open the screenshot.");
byte[] b = new byte[65536];
int n;
while ((n = in.read(b)) != -1) out.write(b, 0, n);
}
synchronized (settingsLock) {
if (!eligible.getAsBoolean()) {
resolver.delete(row, null, null);
return false;
}
ContentValues done = new ContentValues();
done.put(MediaStore.Images.Media.IS_PENDING, 0);
if (resolver.update(row, done, null, null) != 1)
throw new IOException("Gallery could not finish saving the screenshot.");
}
return true;
} catch (Exception e) {
// A pending row is retained and rewritten on retry. A completed row is
// discovered before any future insert, even after process death.
throw e;
}
}
}
@@ -0,0 +1,512 @@
package com.saphid.framecontrol;
import android.Manifest;
import android.app.*;
import android.content.*;
import android.content.pm.PackageManager;
import android.graphics.Color;
import android.net.Uri;
import android.os.*;
import android.text.InputType;
import android.view.*;
import android.webkit.*;
import android.widget.*;
import androidx.core.content.FileProvider;
import java.util.*;
import java.util.concurrent.*;
public final class MainActivity extends Activity {
private final ExecutorService network = Executors.newSingleThreadExecutor();
private final Handler handler = new Handler(Looper.getMainLooper());
private LinearLayout root, downloads;
private FrameLayout content;
private ScrollView companion;
private EditText host, port, user;
private TextView status, empty;
private Switch gallery;
private Button pair, connect, stop;
private WebView web;
private String loadedPage, librarySignature = "";
private int downloadLimit = 100;
private boolean busy, visible, controls;
private ValueCallback<Uri[]> fileSelection;
private static final int INK = Color.rgb(232, 240, 238),
MUTED = Color.rgb(164, 183, 179),
ACCENT = Color.rgb(124, 216, 195);
private final Runnable refresh =
new Runnable() {
public void run() {
if (!visible) return;
update();
handler.postDelayed(this, 1500);
}
};
private int dp(int n) {
return Math.round(getResources().getDisplayMetrics().density * n);
}
private TextView text(String value, int size, int color) {
TextView v = new TextView(this);
v.setText(value);
v.setTextSize(size);
v.setTextColor(color);
v.setPadding(0, dp(6), 0, dp(6));
return v;
}
private Button button(String label, Runnable action) {
Button b = new Button(this);
b.setText(label);
b.setAllCaps(false);
b.setOnClickListener(v -> action.run());
return b;
}
private EditText field(String label, String value, int type) {
companionBody.addView(text(label, 13, MUTED));
EditText v = new EditText(this);
v.setSingleLine(true);
v.setText(value);
v.setHint(label);
v.setContentDescription(label);
v.setInputType(type);
v.setTextColor(INK);
v.setSaveEnabled(false);
companionBody.addView(v);
return v;
}
private LinearLayout companionBody;
@Override
public void onCreate(Bundle state) {
super.onCreate(state);
root = new LinearLayout(this);
root.setOrientation(LinearLayout.VERTICAL);
root.setBackgroundColor(Color.rgb(17, 24, 25));
root.setPadding(dp(20), dp(12), dp(20), dp(12));
root.setOnApplyWindowInsetsListener(
(v, insets) -> {
if (Build.VERSION.SDK_INT >= 30) {
android.graphics.Insets i =
insets.getInsets(
WindowInsets.Type.systemBars() | WindowInsets.Type.displayCutout());
v.setPadding(dp(20) + i.left, dp(12) + i.top, dp(20) + i.right, dp(12) + i.bottom);
} else
v.setPadding(
dp(20) + insets.getSystemWindowInsetLeft(),
dp(12) + insets.getSystemWindowInsetTop(),
dp(20) + insets.getSystemWindowInsetRight(),
dp(12) + insets.getSystemWindowInsetBottom());
return insets;
});
root.addView(text("FRAME CONTROL", 13, ACCENT));
root.addView(text("Your headset, closer.", 27, INK));
status = text(SyncService.state(), 14, MUTED);
status.setAccessibilityLiveRegion(View.ACCESSIBILITY_LIVE_REGION_POLITE);
root.addView(status);
LinearLayout tabs = new LinearLayout(this);
tabs.addView(
button("Screenshots", () -> show(false)), new LinearLayout.LayoutParams(0, dp(48), 1));
tabs.addView(
button("Headset controls", () -> show(true)), new LinearLayout.LayoutParams(0, dp(48), 1));
root.addView(tabs);
content = new FrameLayout(this);
root.addView(content, new LinearLayout.LayoutParams(-1, 0, 1));
companion = new ScrollView(this);
companion.setFillViewport(true);
companionBody = new LinearLayout(this);
companionBody.setOrientation(LinearLayout.VERTICAL);
companion.addView(companionBody);
SharedPreferences p = SyncService.prefs(this);
companionBody.addView(text("Connect your Steam Frame", 20, INK));
companionBody.addView(
text(
"Enable Developer Mode and connect to the same network, or use your headset’s Tailscale"
+ " address.",
14,
MUTED));
host =
field(
"Headset address",
p.getString("host", "frame.local"),
InputType.TYPE_CLASS_TEXT | InputType.TYPE_TEXT_VARIATION_URI);
port = field("SSH port", Integer.toString(p.getInt("port", 22)), InputType.TYPE_CLASS_NUMBER);
user =
field(
"SSH username",
p.getString("user", "steamos"),
InputType.TYPE_CLASS_TEXT | InputType.TYPE_TEXT_VARIATION_VISIBLE_PASSWORD);
pair = button("Pair headset", this::probe);
companionBody.addView(pair);
LinearLayout actions = new LinearLayout(this);
connect = button("Start syncing", this::startSync);
stop = button("Stop", () -> stopService(new Intent(this, SyncService.class)));
actions.addView(connect, new LinearLayout.LayoutParams(0, dp(52), 1));
actions.addView(stop, new LinearLayout.LayoutParams(0, dp(52), 1));
companionBody.addView(actions);
companionBody.addView(text("Screenshot downloads", 20, INK));
gallery = new Switch(this);
gallery.setText("Save screenshots to Gallery");
gallery.setTextColor(INK);
gallery.setTextSize(16);
gallery.setMinHeight(dp(56));
gallery.setChecked(p.getBoolean("gallery", true));
gallery.setOnCheckedChangeListener(
(b, value) -> {
try {
SyncService.gallery(this, value);
librarySignature = "";
} catch (Exception e) {
gallery.setOnCheckedChangeListener(null);
gallery.setChecked(!value);
error(SyncService.message(e));
recreate();
}
});
companionBody.addView(gallery);
companionBody.addView(
text(
"New captures download automatically while sync is running. A notification keeps the"
+ " connection visible. Stop syncing to disconnect. Android may delay transfers in"
+ " battery-saving modes.",
14,
MUTED));
companionBody.addView(
text(
"Turning Gallery off keeps new downloads in this app. Turning it on again won’t export"
+ " earlier app-only downloads.",
14,
MUTED));
empty =
text(
"Pair and start syncing. Existing headset screenshots stay on the headset; new captures"
+ " will appear here.",
14,
MUTED);
companionBody.addView(empty);
downloads = new LinearLayout(this);
downloads.setOrientation(LinearLayout.VERTICAL);
companionBody.addView(downloads);
show(false);
setContentView(root);
}
private void show(boolean useControls) {
controls = useControls;
content.removeAllViews();
if (!controls) {
content.addView(companion, new FrameLayout.LayoutParams(-1, -1));
return;
}
String page = SyncService.page();
if (page == null) {
loadedPage = null;
if (web != null) web.loadUrl("about:blank");
content.addView(text("Start syncing to open your headset’s controls.", 17, MUTED));
return;
}
if (web == null) {
web = new WebView(this);
web.setBackgroundColor(Color.rgb(17, 24, 25));
WebSettings settings = web.getSettings();
settings.setJavaScriptEnabled(true);
settings.setDomStorageEnabled(true);
settings.setAllowFileAccess(false);
settings.setAllowContentAccess(false);
settings.setMixedContentMode(WebSettings.MIXED_CONTENT_NEVER_ALLOW);
settings.setSaveFormData(false);
web.setWebViewClient(
new WebViewClient() {
@Override
public boolean shouldOverrideUrlLoading(WebView view, WebResourceRequest request) {
Uri url = request.getUrl();
Uri origin = Uri.parse(loadedPage == null ? "" : loadedPage);
if ("http".equals(url.getScheme())
&& "127.0.0.1".equals(url.getHost())
&& url.getPort() == origin.getPort()) return false;
if (request.hasGesture()
&& ("https".equals(url.getScheme()) || "http".equals(url.getScheme()))) {
try {
startActivity(new Intent(Intent.ACTION_VIEW, url));
} catch (ActivityNotFoundException ignored) {
}
}
return true;
}
});
web.setWebChromeClient(
new WebChromeClient() {
@Override
public boolean onShowFileChooser(
WebView w, ValueCallback<Uri[]> callback, FileChooserParams params) {
if (fileSelection != null) fileSelection.onReceiveValue(null);
fileSelection = callback;
Intent choose =
new Intent(Intent.ACTION_OPEN_DOCUMENT)
.setType("*/*")
.addCategory(Intent.CATEGORY_OPENABLE);
choose.putExtra(
Intent.EXTRA_ALLOW_MULTIPLE,
params.getMode() == FileChooserParams.MODE_OPEN_MULTIPLE);
try {
startActivityForResult(choose, 110);
} catch (ActivityNotFoundException e) {
fileSelection.onReceiveValue(null);
fileSelection = null;
return false;
}
return true;
}
});
web.setDownloadListener(
(url, ua, disposition, mime, length) -> {
show(false);
Toast.makeText(
this,
"New screenshot originals are in Screenshot downloads.",
Toast.LENGTH_LONG)
.show();
});
}
content.addView(web, new FrameLayout.LayoutParams(-1, -1));
if (!page.equals(loadedPage)) {
loadedPage = page;
web.loadUrl(page);
}
}
@Override
protected void onActivityResult(int request, int result, Intent data) {
super.onActivityResult(request, result, data);
if (request == 110 && fileSelection != null) {
fileSelection.onReceiveValue(WebChromeClient.FileChooserParams.parseResult(result, data));
fileSelection = null;
}
}
private void update() {
if (!busy) status.setText(SyncService.state());
boolean paired = SyncService.prefs(this).contains("pin");
boolean running = SyncService.running();
connect.setEnabled(paired && !busy && !running);
pair.setEnabled(!busy && !running);
stop.setEnabled(!busy && running);
host.setEnabled(!running && !busy);
port.setEnabled(!running && !busy);
user.setEnabled(!running && !busy);
if (controls && !Objects.equals(loadedPage, SyncService.page())) show(true);
try {
List<SyncEngine.Item> items = SyncService.items(this);
StringBuilder signature = new StringBuilder();
for (var item : items)
signature
.append(item.file.getAbsolutePath())
.append('|')
.append(item.destination)
.append('|')
.append(item.file.exists())
.append('\n');
if (signature.toString().equals(librarySignature)) return;
librarySignature = signature.toString();
downloads.removeAllViews();
empty.setVisibility(items.isEmpty() ? View.VISIBLE : View.GONE);
int shown = 0;
for (SyncEngine.Item item : items) {
if (shown++ >= downloadLimit) {
downloads.addView(
button(
"Show 100 more downloads",
() -> {
downloadLimit += 100;
librarySignature = "";
update();
}));
break;
}
String label = item.shot.id.substring(item.shot.id.lastIndexOf('/') + 1);
LinearLayout card = new LinearLayout(this);
card.setOrientation(LinearLayout.VERTICAL);
card.setPadding(0, dp(12), 0, dp(12));
card.addView(text(label, 16, INK));
String where =
item.destination.equals("saved")
? "Saved to Gallery"
: item.destination.equals("appOnly") ? "In Frame Control" : "Gallery save pending";
if (!item.file.exists()) where = "Download pending";
card.addView(text(where + " · Game " + item.shot.id.split("/")[1], 13, MUTED));
if (item.file.exists()) card.addView(button("Open / share original", () -> share(item)));
downloads.addView(card);
}
} catch (Exception e) {
empty.setVisibility(View.VISIBLE);
empty.setText("Could not read screenshot downloads: " + SyncService.message(e));
}
}
private void share(SyncEngine.Item item) {
Uri uri = FileProvider.getUriForFile(this, getPackageName() + ".files", item.file);
Intent send =
new Intent(Intent.ACTION_SEND)
.setType(item.shot.id.endsWith(".png") ? "image/png" : "image/jpeg")
.putExtra(Intent.EXTRA_STREAM, uri)
.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION);
startActivity(Intent.createChooser(send, "Screenshot original"));
}
private void startSync() {
if (Build.VERSION.SDK_INT >= 33
&& checkSelfPermission(Manifest.permission.POST_NOTIFICATIONS)
!= PackageManager.PERMISSION_GRANTED) {
requestPermissions(new String[] {Manifest.permission.POST_NOTIFICATIONS}, 108);
return;
}
startForegroundService(new Intent(this, SyncService.class));
}
@Override
public void onRequestPermissionsResult(int request, String[] permissions, int[] grants) {
super.onRequestPermissionsResult(request, permissions, grants);
if (request == 108) {
if (grants.length > 0 && grants[0] == PackageManager.PERMISSION_GRANTED)
startForegroundService(new Intent(this, SyncService.class));
else
error(
"Allow notifications in Android Settings to start a visible screenshot sync session.");
}
}
private void probe() {
final String address = host.getText().toString().trim(),
username = user.getText().toString().trim();
final int sshPort;
try {
sshPort = Integer.parseInt(port.getText().toString());
} catch (NumberFormatException e) {
error("Enter an SSH port from 1 to 65535.");
return;
}
stopService(new Intent(this, SyncService.class));
busy = true;
pair.setEnabled(false);
status.setText("Checking headset identity…");
network.execute(
() -> {
try {
String pin = FrameConnection.probe(address, sshPort, username);
String fingerprint = FrameConnection.fingerprint(pin);
runOnUiThread(
() -> {
if (isFinishing() || isDestroyed()) return;
busy = false;
new AlertDialog.Builder(this)
.setTitle("Trust this headset?")
.setMessage(
address
+ ":"
+ sshPort
+ "\n\n"
+ fingerprint
+ "\n\n"
+ "Compare this SSH fingerprint with your headset before entering its"
+ " Developer Mode password.")
.setNegativeButton("Cancel", null)
.setPositiveButton(
"Trust and pair", (d, w) -> password(address, sshPort, username, pin))
.show();
});
} catch (Exception e) {
runOnUiThread(
() -> {
busy = false;
error(SyncService.message(e));
});
}
});
}
private void password(String address, int sshPort, String username, String pin) {
EditText password = new EditText(this);
password.setInputType(InputType.TYPE_CLASS_TEXT | InputType.TYPE_TEXT_VARIATION_PASSWORD);
password.setHint("Developer Mode password");
password.setSaveEnabled(false);
new AlertDialog.Builder(this)
.setTitle("Pair with " + address)
.setMessage(
"Used once to authorize this phone’s protected SSH key. The password is not saved.")
.setView(password)
.setNegativeButton("Cancel", (d, w) -> password.setText(""))
.setPositiveButton(
"Pair",
(d, w) -> {
String secret = password.getText().toString();
password.setText("");
busy = true;
status.setText("Pairing this phone…");
network.execute(
() -> {
try {
FrameConnection.pair(
address, sshPort, username, pin, secret, SyncService.install(this));
if (!SyncService.prefs(this)
.edit()
.putString("host", address)
.putInt("port", sshPort)
.putString("user", username)
.putString("pin", pin)
.commit()) throw new java.io.IOException("Could not save pairing.");
runOnUiThread(
() -> {
busy = false;
status.setText("Paired. Start syncing when you’re ready.");
update();
});
} catch (Exception e) {
runOnUiThread(
() -> {
busy = false;
error(SyncService.message(e));
});
}
});
})
.show();
}
private void error(String message) {
if (isFinishing() || isDestroyed()) return;
new AlertDialog.Builder(this)
.setTitle("Frame Control")
.setMessage(message)
.setPositiveButton("OK", null)
.show();
}
@Override
protected void onResume() {
super.onResume();
visible = true;
if (web != null) web.onResume();
handler.post(refresh);
}
@Override
protected void onPause() {
visible = false;
handler.removeCallbacks(refresh);
if (web != null) web.onPause();
super.onPause();
}
@Override
protected void onDestroy() {
network.shutdownNow();
if (fileSelection != null) fileSelection.onReceiveValue(null);
if (web != null) {
web.destroy();
web = null;
}
super.onDestroy();
}
}
@@ -0,0 +1,115 @@
package com.saphid.framecontrol;
import android.security.keystore.KeyGenParameterSpec;
import android.security.keystore.KeyProperties;
import com.jcraft.jsch.Identity;
import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.math.BigInteger;
import java.nio.charset.StandardCharsets;
import java.security.KeyPairGenerator;
import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.Signature;
import java.security.interfaces.RSAPublicKey;
import java.util.Base64;
/** The SSH private key is a non-exportable Android Keystore signing key. */
final class PhoneIdentity implements Identity {
private static final String ALIAS = "frame-control-ssh-v1";
private final PrivateKey key;
private final byte[] publicBlob;
PhoneIdentity() throws Exception {
KeyStore store = KeyStore.getInstance("AndroidKeyStore");
store.load(null);
if (!store.containsAlias(ALIAS)) {
KeyPairGenerator generator = KeyPairGenerator.getInstance("RSA", "AndroidKeyStore");
generator.initialize(
new KeyGenParameterSpec.Builder(ALIAS, KeyProperties.PURPOSE_SIGN)
.setKeySize(3072)
.setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA512)
.setSignaturePaddings(KeyProperties.SIGNATURE_PADDING_RSA_PKCS1)
.build());
generator.generateKeyPair();
}
key = (PrivateKey) store.getKey(ALIAS, null);
RSAPublicKey publicKey = (RSAPublicKey) store.getCertificate(ALIAS).getPublicKey();
publicBlob =
blob(
"ssh-rsa".getBytes(StandardCharsets.US_ASCII),
integer(publicKey.getPublicExponent()),
integer(publicKey.getModulus()));
}
private static byte[] integer(BigInteger value) {
return value.toByteArray();
}
private static byte[] blob(byte[]... strings) throws Exception {
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
DataOutputStream out = new DataOutputStream(bytes);
for (byte[] s : strings) {
out.writeInt(s.length);
out.write(s);
}
return bytes.toByteArray();
}
String authorizedKey(String install) {
return "ssh-rsa "
+ Base64.getEncoder().encodeToString(publicBlob)
+ " frame-control-android-"
+ install;
}
@Override
public byte[] getPublicKeyBlob() {
return publicBlob.clone();
}
@Override
public byte[] getSignature(byte[] data) {
return getSignature(data, "rsa-sha2-512");
}
@Override
public byte[] getSignature(byte[] data, String algorithm) {
try {
if (!algorithm.equals("rsa-sha2-256") && !algorithm.equals("rsa-sha2-512")) return null;
Signature signer =
Signature.getInstance(
algorithm.equals("rsa-sha2-512") ? "SHA512withRSA" : "SHA256withRSA");
signer.initSign(key);
signer.update(data);
return blob(algorithm.getBytes(StandardCharsets.US_ASCII), signer.sign());
} catch (Exception e) {
return null;
}
}
@Override
public boolean setPassphrase(byte[] ignored) {
return true;
}
@Override
public String getAlgName() {
return "ssh-rsa";
}
@Override
public String getName() {
return ALIAS;
}
@Override
public boolean isEncrypted() {
return false;
}
@Override
public void clear() {
/* Keystore owns the key; disconnect never deletes it. */
}
}
@@ -0,0 +1,570 @@
package com.saphid.framecontrol;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.Strictness;
import com.google.gson.TypeAdapter;
import com.google.gson.stream.JsonReader;
import com.google.gson.stream.JsonToken;
import com.google.gson.stream.JsonWriter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.NoSuchFileException;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.function.BooleanSupplier;
import java.util.regex.Pattern;
/** One reusable engine per root. The caller supplies an account/headset-specific root. */
public final class SyncEngine {
private static final long MAX_SIZE = 30L * 1024 * 1024;
private static final Pattern ID = Pattern.compile(
"^[0-9]{1,12}/[0-9]{1,20}/[0-9]{14}_[0-9]{1,4}\\.(jpg|png)$");
private static final Gson GSON = new GsonBuilder().registerTypeAdapter(Shot.class,
new TypeAdapter<Shot>() {
@Override public void write(JsonWriter writer, Shot shot) throws IOException {
writer.beginObject();
writer.name("id").value(shot.id);
writer.name("size").value(shot.size);
writer.name("time").value(shot.time);
writer.endObject();
}
@Override public Shot read(JsonReader reader) throws IOException {
String id = null;
Long size = null;
Double time = null;
Set<String> fields = new HashSet<>();
reader.beginObject();
while (reader.hasNext()) {
String field = reader.nextName();
if (!fields.add(field)) throw new IOException("Duplicate screenshot field");
switch (field) {
case "id":
if (reader.peek() != JsonToken.STRING) throw new IOException("Invalid id");
id = reader.nextString();
break;
case "size":
if (reader.peek() != JsonToken.NUMBER) throw new IOException("Invalid size");
size = reader.nextLong();
break;
case "time":
if (reader.peek() != JsonToken.NUMBER) throw new IOException("Invalid time");
time = reader.nextDouble();
break;
default: throw new IOException("Unknown screenshot field");
}
}
reader.endObject();
if (id == null || size == null || time == null) throw new IOException("Incomplete shot");
return new Shot(id, size, time);
}
}).create();
public static final class Shot {
public final String id;
public final long size;
public final double time;
public Shot(String id, long size, double time) {
this.id = id;
this.size = size;
this.time = time;
}
}
public interface Transport {
void download(Shot shot, File destination) throws Exception;
boolean validImage(File f) throws Exception;
}
public interface Exporter {
boolean export(Shot shot, File original, BooleanSupplier eligible) throws Exception;
}
public static final class Item {
public final Shot shot;
/** Future original path; may not exist while the shot is observing or queued. */
public final File file;
public final String destination;
private Item(Shot shot, File file, String destination) {
this.shot = shot;
this.file = file;
this.destination = destination;
}
}
// Boxed fields make missing journal fields distinguishable from legitimate zero values.
private static final class Ledger {
Integer version;
Boolean initialized;
Boolean gallery;
Map<String, Receipt> records;
}
private static final class Receipt {
Shot shot;
String state;
Integer observations;
String destination;
Receipt(Shot shot, String state, int observations, String destination) {
this.shot = shot;
this.state = state;
this.observations = observations;
this.destination = destination;
}
}
/** Only external callbacks and rejected transfer contents are safe to retry per item. */
private static final class ItemFailure extends IOException {
private static final long serialVersionUID = 1L;
ItemFailure(String message, Exception cause) {
super(message, cause);
}
}
private final Object lock = new Object();
private final Path root;
private final Path originals;
private final Path journal;
private Ledger ledger;
private long generation;
private long active;
public SyncEngine(File root, boolean gallery) throws Exception {
if (root == null || Files.isSymbolicLink(root.toPath())) {
throw new IOException("Unsafe sync root");
}
this.root = root.getCanonicalFile().toPath();
List<Path> missingDirectories = new ArrayList<>();
for (Path path = this.root; path != null && !Files.exists(path); path = path.getParent()) {
missingDirectories.add(path);
}
Files.createDirectories(this.root);
for (Path directory : missingDirectories) syncDirectory(directory.getParent());
originals = this.root.resolve("originals");
journal = this.root.resolve("receipts.json");
checkPath(originals);
checkPath(journal);
if (Files.exists(originals, LinkOption.NOFOLLOW_LINKS)
&& !Files.isDirectory(originals, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("Invalid originals directory");
}
if (Files.exists(journal, LinkOption.NOFOLLOW_LINKS)) {
if (!Files.isRegularFile(journal, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("Invalid sync journal");
}
// Never recover a damaged or incomplete journal by establishing another baseline.
try (JsonReader reader = new JsonReader(new InputStreamReader(
Files.newInputStream(journal), StandardCharsets.UTF_8))) {
reader.setStrictness(Strictness.STRICT);
ledger = GSON.fromJson(reader, Ledger.class);
if (reader.peek() != JsonToken.END_DOCUMENT) {
throw new IOException("Trailing sync journal data");
}
} catch (RuntimeException e) {
throw new IOException("Corrupt sync journal", e);
}
validateLedger();
setGallery(gallery);
} else {
if (Files.exists(originals, LinkOption.NOFOLLOW_LINKS)) {
try (java.util.stream.Stream<Path> paths = Files.walk(originals)) {
if (paths.anyMatch(p -> !Files.isDirectory(p, LinkOption.NOFOLLOW_LINKS))) {
throw new IOException("Originals exist without a sync journal");
}
}
}
ledger = emptyLedger();
ledger.gallery = gallery;
persist(ledger);
}
}
public void setGallery(boolean enabled) throws Exception {
synchronized (lock) {
if (ledger.gallery == enabled) return;
Ledger next = copy();
next.gallery = enabled;
if (!enabled) {
for (Receipt receipt : next.records.values()) {
if ("pending".equals(receipt.destination)) receipt.destination = "appOnly";
}
}
commit(next);
}
}
/** A second live reconcile is rejected. stop() cancels its generation and permits a new run. */
public void reconcile(List<Shot> shots, Transport transport, Exporter exporter) throws Exception {
final long run;
synchronized (lock) {
if (active != 0) throw new IllegalStateException("Sync already in progress");
run = ++generation;
active = run;
}
try {
Map<String, Shot> listing = listing(shots);
synchronized (lock) {
if (active != run) return;
Ledger next = copy();
if (!next.initialized) {
for (Shot shot : listing.values()) {
next.records.put(shot.id, new Receipt(shot, "baseline", 0, null));
}
next.initialized = true;
commit(next);
return;
}
// Missing observations break consecutiveness, including across restarts.
for (Receipt receipt : next.records.values()) {
if (!listing.containsKey(receipt.shot.id)
&& ("observing".equals(receipt.state) || "queued".equals(receipt.state))) {
receipt.observations = 0;
}
}
for (Shot shot : listing.values()) {
Receipt receipt = next.records.get(shot.id);
if (receipt == null) {
// Latch eligibility at discovery, before waiting for a stable second sample.
next.records.put(shot.id, new Receipt(shot, "observing", 1,
next.gallery ? "pending" : "appOnly"));
} else if ("observing".equals(receipt.state) || "queued".equals(receipt.state)) {
receipt.observations = same(receipt.shot, shot)
? Math.min(2, receipt.observations + 1) : 1;
receipt.shot = shot;
if (receipt.observations == 2 && "observing".equals(receipt.state)) {
receipt.state = "queued";
}
}
}
commit(next); // Queue and its disposition reach disk before any download.
}
List<String> ids;
synchronized (lock) {
if (active != run) return;
ids = new ArrayList<>(ledger.records.keySet());
}
ItemFailure failure = null;
for (String id : ids) {
Shot download = null;
synchronized (lock) {
if (active != run) return;
Receipt receipt = ledger.records.get(id);
if ("queued".equals(receipt.state) && receipt.observations == 2) {
download = receipt.shot;
}
}
try {
if (download != null) download(run, download, transport);
export(run, id, exporter);
} catch (ItemFailure e) {
if (failure == null) failure = e;
else failure.addSuppressed(e);
}
}
// Surface retryable errors to the service after independent items have progressed.
// Untagged local I/O/journal failures escape immediately and halt the run.
if (failure != null) throw failure;
} catch (Exception e) {
synchronized (lock) {
if (active != run) return; // A late failed callback belongs to a cancelled run.
}
throw e;
} finally {
synchronized (lock) {
if (active == run) active = 0;
}
}
}
/** Snapshot of all post-baseline receipts, including downloads still pending. */
public List<Item> items() {
synchronized (lock) {
List<Item> result = new ArrayList<>();
for (Receipt receipt : ledger.records.values()) {
if (!"baseline".equals(receipt.state)) {
result.add(new Item(receipt.shot, originals.resolve(receipt.shot.id).toFile(),
receipt.destination));
}
}
result.sort(Comparator.comparingDouble((Item item) -> item.shot.time).reversed()
.thenComparing(item -> item.shot.id));
return Collections.unmodifiableList(result);
}
}
public void stop() {
synchronized (lock) {
++generation;
active = 0;
}
}
private void download(long run, Shot shot, Transport transport) throws Exception {
Path temporary;
synchronized (lock) {
if (active != run) return;
temporary = Files.createTempFile(root, ".download-", ".part");
}
try {
try {
transport.download(shot, temporary.toFile());
} catch (InterruptedException e) {
throw e;
} catch (Exception e) {
throw new ItemFailure("Download failed: " + shot.id + ": " + e.getMessage(), e);
}
synchronized (lock) {
if (active != run) return;
}
checkTransfer(temporary, shot);
boolean valid;
try {
valid = transport.validImage(temporary.toFile());
} catch (InterruptedException e) {
throw e;
} catch (Exception e) {
throw new ItemFailure("Image validation failed: " + shot.id + ": " + e.getMessage(), e);
}
if (!valid) throw new ItemFailure("Invalid image: " + shot.id, null);
checkTransfer(temporary, shot);
try (FileChannel channel = FileChannel.open(temporary, StandardOpenOption.WRITE)) {
channel.force(true);
}
synchronized (lock) {
if (active != run) return;
Path target = originals.resolve(shot.id);
createParents(target.getParent());
checkPath(target);
Files.move(temporary, target, StandardCopyOption.ATOMIC_MOVE,
StandardCopyOption.REPLACE_EXISTING);
syncDirectory(target.getParent());
Ledger next = copy();
// Read the current destination: setGallery may have changed it during the transfer.
next.records.get(shot.id).state = "downloaded";
commit(next);
}
} finally {
Files.deleteIfExists(temporary);
}
}
private void export(long run, String id, Exporter exporter) throws Exception {
Shot shot;
File original;
synchronized (lock) {
if (!eligible(run, id)) return;
Receipt receipt = ledger.records.get(id);
shot = receipt.shot;
Path path = originals.resolve(id);
checkDownload(path, shot);
original = path.toFile();
}
boolean saved;
try {
saved = exporter.export(shot, original, () -> {
synchronized (lock) {
return eligible(run, id);
}
});
} catch (InterruptedException e) {
throw e;
} catch (Exception e) {
throw new ItemFailure("Gallery export failed: " + shot.id + ": " + e.getMessage(), e);
}
synchronized (lock) {
if (active != run || !saved) return;
Ledger next = copy();
// A true result means the exporter actually saved (or recovered) the MediaStore item.
next.records.get(id).destination = "saved";
commit(next);
}
}
private boolean eligible(long run, String id) {
Receipt receipt = ledger.records.get(id);
return active == run && ledger.gallery && receipt != null
&& "downloaded".equals(receipt.state) && "pending".equals(receipt.destination);
}
private static Map<String, Shot> listing(List<Shot> shots) throws IOException {
if (shots == null) throw new IOException("Missing screenshot listing");
Map<String, Shot> result = new LinkedHashMap<>();
for (Shot shot : shots) {
validateShot(shot);
Shot previous = result.put(shot.id, shot);
if (previous != null && !same(previous, shot)) {
throw new IOException("Conflicting screenshot identity: " + shot.id);
}
}
return result;
}
private static void validateShot(Shot shot) throws IOException {
if (shot == null || shot.id == null || !ID.matcher(shot.id).matches()
|| shot.size <= 0 || shot.size > MAX_SIZE || !Double.isFinite(shot.time)) {
throw new IOException("Invalid screenshot metadata");
}
}
private static boolean same(Shot a, Shot b) {
return a.size == b.size && Double.compare(a.time, b.time) == 0;
}
private void validateLedger() throws IOException {
if (ledger == null || ledger.version == null || ledger.version != 1
|| ledger.initialized == null || ledger.gallery == null || ledger.records == null
|| (!ledger.initialized && !ledger.records.isEmpty())) {
throw new IOException("Invalid sync journal header");
}
for (Map.Entry<String, Receipt> entry : ledger.records.entrySet()) {
Receipt receipt = entry.getValue();
if (receipt == null) throw new IOException("Invalid sync receipt");
validateShot(receipt.shot);
if (!entry.getKey().equals(receipt.shot.id) || receipt.observations == null
|| receipt.observations < 0 || receipt.observations > 2) {
throw new IOException("Invalid sync receipt identity");
}
boolean baseline = "baseline".equals(receipt.state);
boolean observing = "observing".equals(receipt.state);
boolean queued = "queued".equals(receipt.state);
boolean downloaded = "downloaded".equals(receipt.state);
if ((!baseline && !observing && !queued && !downloaded)
|| (baseline && receipt.destination != null)
|| (baseline && receipt.observations != 0)
|| (observing && receipt.observations > 1)
|| (downloaded && receipt.observations != 2)
|| ((observing || queued || downloaded) && !"pending".equals(receipt.destination)
&& !"appOnly".equals(receipt.destination) && !"saved".equals(receipt.destination))
|| ((observing || queued) && "saved".equals(receipt.destination))
|| (!ledger.gallery && "pending".equals(receipt.destination))) {
throw new IOException("Invalid sync receipt state");
}
Path path = originals.resolve(receipt.shot.id);
checkPath(path);
if (downloaded) checkDownload(path, receipt.shot);
}
}
private void checkDownload(Path path, Shot shot) throws IOException {
checkPath(path);
if (!Files.isRegularFile(path, LinkOption.NOFOLLOW_LINKS) || Files.size(path) != shot.size) {
throw new IOException("Incomplete original: " + shot.id);
}
}
private void checkTransfer(Path path, Shot shot) throws IOException {
checkPath(path); // Unsafe paths and local filesystem errors remain fatal.
BasicFileAttributes attributes;
try {
attributes = Files.readAttributes(path, BasicFileAttributes.class, LinkOption.NOFOLLOW_LINKS);
} catch (NoSuchFileException e) {
throw new ItemFailure("Missing downloaded image: " + shot.id, e);
}
if (!attributes.isRegularFile() || attributes.size() != shot.size) {
throw new ItemFailure("Incomplete downloaded image: " + shot.id, null);
}
}
private void checkPath(Path path) throws IOException {
if (!path.normalize().startsWith(root)) throw new IOException("Path escaped sync root");
Path current = root;
if (Files.isSymbolicLink(current)) throw new IOException("Symlink in sync root");
for (Path component : root.relativize(path)) {
current = current.resolve(component);
if (Files.isSymbolicLink(current)) throw new IOException("Symlink in sync path");
}
}
private void createParents(Path directory) throws IOException {
checkPath(directory);
Path current = root;
for (Path component : root.relativize(directory)) {
Path parent = current;
current = current.resolve(component);
if (!Files.exists(current, LinkOption.NOFOLLOW_LINKS)) {
Files.createDirectory(current);
syncDirectory(parent);
} else if (!Files.isDirectory(current, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("Invalid originals directory");
}
}
}
private Ledger copy() {
Ledger next = emptyLedger();
next.initialized = ledger.initialized;
next.gallery = ledger.gallery;
for (Map.Entry<String, Receipt> entry : ledger.records.entrySet()) {
Receipt receipt = entry.getValue();
next.records.put(entry.getKey(), new Receipt(receipt.shot, receipt.state,
receipt.observations, receipt.destination));
}
return next;
}
private static Ledger emptyLedger() {
Ledger result = new Ledger();
result.version = 1;
result.initialized = false;
result.records = new LinkedHashMap<>();
return result;
}
private void commit(Ledger next) throws IOException {
persist(next);
ledger = next;
}
private void persist(Ledger next) throws IOException {
checkPath(journal);
Path temporary = Files.createTempFile(root, ".receipts-", ".tmp");
try {
try (FileOutputStream output = new FileOutputStream(temporary.toFile());
Writer writer = new OutputStreamWriter(output, StandardCharsets.UTF_8)) {
GSON.toJson(next, writer);
writer.flush();
output.getFD().sync();
}
Files.move(temporary, journal, StandardCopyOption.ATOMIC_MOVE,
StandardCopyOption.REPLACE_EXISTING);
syncDirectory(root);
} finally {
Files.deleteIfExists(temporary);
}
}
private static void syncDirectory(Path directory) {
// Some Android filesystems reject directory channels. File contents were already fsynced;
// the rename is atomic either way. Do not report rollback after a committed rename.
try (FileChannel channel = FileChannel.open(directory, StandardOpenOption.READ)) {
channel.force(true);
} catch (IOException | UnsupportedOperationException ignored) {
// Best available portable Java directory durability.
}
}
}
@@ -0,0 +1,234 @@
package com.saphid.framecontrol;
import android.app.*;
import android.content.*;
import android.content.pm.ServiceInfo;
import android.os.*;
import java.io.File;
import java.util.*;
import java.util.concurrent.*;
/** Explicit user-started connected-device session, with a visible Stop action. */
public final class SyncService extends Service {
static final String CHANNEL = "headset-sync";
static final Object SETTINGS_LOCK = new Object();
private static volatile SyncService instance;
private static volatile String state = "Disconnected";
private static volatile String page;
private final ExecutorService worker = Executors.newSingleThreadExecutor();
private volatile boolean stopping;
private volatile FrameConnection connection;
private volatile SyncEngine engine;
static SharedPreferences prefs(Context c) {
return c.getSharedPreferences("frame", MODE_PRIVATE);
}
static String state() {
return state;
}
static String page() {
return page;
}
static boolean running() {
SyncService service = instance;
return service != null && !service.stopping;
}
static String install(Context c) {
SharedPreferences p = prefs(c);
String id = p.getString("install", null);
if (id == null) {
id = UUID.randomUUID().toString();
if (!p.edit().putString("install", id).commit())
throw new IllegalStateException("Could not save this phone's identity.");
}
return id;
}
static File library(Context c, String pin) throws Exception {
return new File(c.getFilesDir(), "screenshots/" + FrameConnection.hash(pin));
}
static void gallery(Context c, boolean value) throws Exception {
synchronized (SETTINGS_LOCK) {
SyncService service = instance;
// Journal first: a failed disk write must not appear as a saved setting.
String pin = prefs(c).getString("pin", null);
SyncEngine live = service == null ? null : service.engine;
if (live != null) live.setGallery(value);
if (!value) {
// The setting is global. Latch the opt-out in every retained library,
// including headsets that are not currently paired or connected.
File libraries = new File(c.getFilesDir(), "screenshots");
if (libraries.exists()) {
File[] roots = libraries.listFiles();
if (roots == null) throw new java.io.IOException("Could not read screenshot libraries.");
File activeRoot = live != null && pin != null ? library(c, pin) : null;
for (File root : roots) {
if (!root.getName().matches("[0-9a-f]{64}")) continue;
if (activeRoot != null && root.equals(activeRoot)) continue;
new SyncEngine(root, false).setGallery(false);
}
}
} else if (live == null && pin != null) {
new SyncEngine(library(c, pin), true).setGallery(true);
}
if (!prefs(c).edit().putBoolean("gallery", value).commit())
throw new java.io.IOException("Could not save the Gallery setting.");
}
}
static List<SyncEngine.Item> items(Context c) throws Exception {
SyncService service = instance;
if (service != null && service.engine != null) return service.engine.items();
String pin = prefs(c).getString("pin", null);
if (pin == null) return Collections.emptyList();
synchronized (SETTINGS_LOCK) {
return new SyncEngine(library(c, pin), prefs(c).getBoolean("gallery", true)).items();
}
}
@Override
public void onCreate() {
super.onCreate();
instance = this;
stopping = false;
NotificationManager manager = getSystemService(NotificationManager.class);
manager.createNotificationChannel(
new NotificationChannel(
CHANNEL, "Headset screenshot sync", NotificationManager.IMPORTANCE_LOW));
}
private Notification notification(String message) {
PendingIntent open =
PendingIntent.getActivity(
this,
0,
new Intent(this, MainActivity.class),
PendingIntent.FLAG_IMMUTABLE | PendingIntent.FLAG_UPDATE_CURRENT);
PendingIntent stop =
PendingIntent.getService(
this,
1,
new Intent(this, SyncService.class).setAction("STOP"),
PendingIntent.FLAG_IMMUTABLE | PendingIntent.FLAG_UPDATE_CURRENT);
return new Notification.Builder(this, CHANNEL)
.setSmallIcon(com.saphid.framecontrol.R.drawable.ic_frame)
.setContentTitle("Frame Control")
.setContentText(message)
.setContentIntent(open)
.setOngoing(true)
.addAction(new Notification.Action.Builder(null, "Stop syncing", stop).build())
.build();
}
private void status(String text) {
state = text;
getSystemService(NotificationManager.class).notify(108, notification(text));
}
@Override
public int onStartCommand(Intent intent, int flags, int startId) {
if (intent != null && "STOP".equals(intent.getAction())) {
stopSelf();
return START_NOT_STICKY;
}
if (connection != null || stopping) return START_NOT_STICKY;
state = "Connecting to headset";
if (Build.VERSION.SDK_INT >= 34)
startForeground(
108, notification(state), ServiceInfo.FOREGROUND_SERVICE_TYPE_CONNECTED_DEVICE);
else startForeground(108, notification(state));
connection = new FrameConnection(this);
worker.execute(this::runSession);
return START_NOT_STICKY;
}
private void runSession() {
try {
SharedPreferences p = prefs(this);
String pin = p.getString("pin", null);
if (pin == null) throw new java.io.IOException("Pair with your headset first.");
synchronized (SETTINGS_LOCK) {
engine = new SyncEngine(library(this, pin), p.getBoolean("gallery", true));
}
while (!stopping) {
try {
FrameConnection active = new FrameConnection(this);
connection = active;
status("Connecting to headset");
active.connect(
p.getString("host", ""),
p.getInt("port", 22),
p.getString("user", "steamos"),
pin,
install(this));
if (stopping) {
active.close();
break;
}
page = active.page();
status("Syncing new screenshots");
GalleryExporter exporter = new GalleryExporter(this, pin, SETTINGS_LOCK);
while (!stopping && active.connected()) {
try {
engine.reconcile(active.shots(), active, exporter);
status("Syncing new screenshots");
} catch (Exception e) {
status("Screenshot pending · " + message(e));
}
Thread.sleep(4000);
}
if (!stopping) throw new java.io.IOException("Connection lost. Reconnecting…");
} catch (Exception e) {
page = null;
if (connection != null) connection.close();
if (stopping) break;
status("Waiting for headset · " + message(e));
Thread.sleep(10000);
}
}
} catch (InterruptedException ignored) {
Thread.currentThread().interrupt();
} catch (Exception e) {
state = message(e);
} finally {
page = null;
if (connection != null) connection.close();
stopSelf();
}
}
static String message(Exception e) {
String text = e.getMessage();
if (text == null || text.trim().isEmpty()) return "Try reconnecting.";
if (text.contains("HostKey")
|| text.contains("HostKey has been changed")
|| text.contains("reject HostKey")) return "Headset identity changed. Stop and pair again.";
if (text.contains("Auth fail")) return "Headset login refused. Pair again.";
return text.length() > 180 ? text.substring(0, 180) : text;
}
@Override
public void onDestroy() {
stopping = true;
page = null;
if (engine != null) engine.stop();
if (connection != null) connection.close();
worker.shutdownNow();
instance = null;
if (state.equals("Syncing new screenshots")
|| state.startsWith("Waiting")
|| state.startsWith("Connecting")) state = "Disconnected";
stopForeground(STOP_FOREGROUND_REMOVE);
super.onDestroy();
}
@Override
public IBinder onBind(Intent intent) {
return null;
}
}
@@ -0,0 +1,3 @@
<vector xmlns:android="http://schemas.android.com/apk/res/android" android:width="48dp" android:height="48dp" android:viewportWidth="48" android:viewportHeight="48">
<path android:fillColor="#7CD8C3" android:pathData="M8,12L40,12L44,30L34,36L26,28L22,28L14,36L4,30ZM11,17L9,27L14,29L20,23L28,23L34,29L39,27L37,17Z" />
</vector>
@@ -0,0 +1,9 @@
<resources>
<style name="AppTheme" parent="android:style/Theme.Material.NoActionBar">
<item name="android:fontFamily">sans</item>
<item name="android:colorAccent">#7CD8C3</item>
<item name="android:windowLightStatusBar">false</item>
<item name="android:windowBackground">#111819</item>
<item name="android:navigationBarColor">#111819</item>
</style>
</resources>
@@ -0,0 +1,4 @@
<data-extraction-rules>
<cloud-backup><exclude domain="root" path="." /><exclude domain="file" path="." /><exclude domain="database" path="." /><exclude domain="sharedpref" path="." /><exclude domain="external" path="." /></cloud-backup>
<device-transfer><exclude domain="root" path="." /><exclude domain="file" path="." /><exclude domain="database" path="." /><exclude domain="sharedpref" path="." /><exclude domain="external" path="." /></device-transfer>
</data-extraction-rules>
@@ -0,0 +1 @@
<paths><files-path name="screenshots" path="screenshots/" /></paths>
@@ -0,0 +1,4 @@
<network-security-config>
<base-config cleartextTrafficPermitted="false" />
<domain-config cleartextTrafficPermitted="true"><domain includeSubdomains="false">127.0.0.1</domain><domain includeSubdomains="false">localhost</domain></domain-config>
</network-security-config>
@@ -0,0 +1,789 @@
package com.saphid.framecontrol;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import static org.junit.Assert.*;
public class SyncEngineTest {
@Rule public TemporaryFolder temporary = new TemporaryFolder();
private static final String FIRST = "123/456/20261024120000_1.jpg";
private static final String SECOND = "123/456/20261024120001_2.png";
private static final SyncEngine.Shot A = shot(FIRST);
private static final SyncEngine.Shot B = shot(SECOND);
private static SyncEngine.Shot shot(String id) { return new SyncEngine.Shot(id, 4, 100.5); }
private static class Transfer implements SyncEngine.Transport {
final List<String> downloads = new ArrayList<>();
boolean valid = true;
boolean truncated;
@Override public void download(SyncEngine.Shot shot, File destination) throws Exception {
downloads.add(shot.id);
try (RandomAccessFile file = new RandomAccessFile(destination, "rw")) {
file.setLength(truncated ? shot.size - 1 : shot.size);
}
}
@Override public boolean validImage(File file) throws Exception { return valid; }
}
private static class Gallery implements SyncEngine.Exporter {
int attempts;
int saves;
boolean deny;
boolean fail;
@Override public boolean export(SyncEngine.Shot shot, File original,
java.util.function.BooleanSupplier eligible) throws Exception {
attempts++;
assertTrue(original.isFile());
if (fail) throw new IOException("Gallery unavailable");
if (deny || !eligible.getAsBoolean()) return false;
saves++;
return true;
}
}
private interface Checked { void run() throws Exception; }
private static IOException fails(Checked operation) throws Exception {
try {
operation.run();
} catch (IOException expected) {
return expected;
}
throw new AssertionError("Expected operation to reject invalid or unavailable state");
}
private static void reconcile(SyncEngine engine, Transfer transfer, Gallery gallery,
SyncEngine.Shot... shots) throws Exception {
engine.reconcile(Arrays.asList(shots), transfer, gallery);
}
private static void observed(SyncEngine engine, Transfer transfer, Gallery gallery,
SyncEngine.Shot shot) throws Exception {
reconcile(engine, transfer, gallery);
reconcile(engine, transfer, gallery, shot);
}
private static void assertDownloadPending(SyncEngine engine, String destination) {
assertEquals(1, engine.items().size());
assertEquals(destination, engine.items().get(0).destination);
assertFalse(engine.items().get(0).file.exists());
}
private static SyncEngine.Item item(SyncEngine engine, String id) {
return engine.items().stream().filter(i -> i.shot.id.equals(id)).findFirst().get();
}
private static void await(CountDownLatch latch) throws InterruptedException {
assertTrue("Timed out waiting for callback", latch.await(5, TimeUnit.SECONDS));
}
private static final class Async implements AutoCloseable {
private final AtomicReference<Throwable> failure = new AtomicReference<>();
private final Thread thread;
Async(Checked operation) {
thread = new Thread(() -> {
try { operation.run(); } catch (Throwable e) { failure.set(e); }
}, "sync-core-test");
thread.start();
}
void finish() throws Exception {
thread.join(5000);
assertFalse("Reconcile did not finish", thread.isAlive());
if (failure.get() != null) throw new AssertionError("Background reconcile failed", failure.get());
}
@Override public void close() throws Exception {
if (thread.isAlive()) thread.interrupt();
thread.join(5000);
assertFalse("Leaked reconcile thread", thread.isAlive());
}
}
@Test public void firstListingIsDurableBaselineAndNeverDownloadsHistory() throws Exception {
File root = temporary.newFolder();
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
SyncEngine engine = new SyncEngine(root, true);
reconcile(engine, transfer, gallery, A);
assertTrue(new File(root, "receipts.json").isFile());
engine = new SyncEngine(root, true);
reconcile(engine, transfer, gallery, A);
reconcile(engine, transfer, gallery, new SyncEngine.Shot(FIRST, 7, 999));
assertTrue(transfer.downloads.isEmpty());
assertTrue(engine.items().isEmpty());
}
@Test public void emptyBaselineRequiresTwoIdenticalNewObservations() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), true);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
assertTrue(transfer.downloads.isEmpty());
assertDownloadPending(engine, "pending");
reconcile(engine, transfer, gallery, A);
assertEquals(Collections.singletonList(FIRST), transfer.downloads);
assertEquals(1, gallery.saves);
assertEquals("saved", engine.items().get(0).destination);
assertEquals(4, engine.items().get(0).file.length());
}
@Test public void sizeAndMtimeChangesResetStabilityAndMissingPollBreaksIt() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
SyncEngine.Shot size = new SyncEngine.Shot(FIRST, 5, A.time);
SyncEngine.Shot time = new SyncEngine.Shot(FIRST, 5, A.time + 1);
reconcile(engine, transfer, gallery, size);
reconcile(engine, transfer, gallery, time);
reconcile(engine, transfer, gallery);
reconcile(engine, transfer, gallery, time);
assertTrue(transfer.downloads.isEmpty());
reconcile(engine, transfer, gallery, time);
assertEquals(1, transfer.downloads.size());
assertEquals(5, engine.items().get(0).file.length());
}
@Test public void identityIncludesAccountAndAppAndDuplicateRowsAreOnlyOneObservation() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
reconcile(engine, transfer, gallery, A);
SyncEngine.Shot otherAccount = shot("124/456/20261024120000_1.jpg");
SyncEngine.Shot otherApp = shot("123/457/20261024120000_1.jpg");
reconcile(engine, transfer, gallery, A, otherAccount, otherAccount, otherApp);
assertTrue(transfer.downloads.isEmpty());
reconcile(engine, transfer, gallery, otherAccount, otherApp);
assertEquals(2, transfer.downloads.size());
assertEquals(2, engine.items().size());
reconcile(engine, transfer, gallery, otherAccount, otherApp);
assertEquals(2, transfer.downloads.size());
}
@Test public void truncatedAndInvalidImagesStayQueuedUntilValidRetry() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
transfer.truncated = true;
fails(() -> reconcile(engine, transfer, gallery, A));
assertDownloadPending(engine, "pending");
assertFalse(new File(root, "originals/" + FIRST).exists());
transfer.truncated = false; transfer.valid = false;
fails(() -> reconcile(engine, transfer, gallery, A));
assertDownloadPending(engine, "pending");
assertEquals(0, gallery.attempts);
transfer.valid = true;
reconcile(engine, transfer, gallery, A);
assertEquals(3, transfer.downloads.size());
assertEquals("saved", engine.items().get(0).destination);
assertEquals(0, root.listFiles((dir, name) -> name.startsWith(".download-")).length);
}
@Test public void deniedExportRetriesWithoutDownloadingAndSavedReceiptDeduplicates() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
gallery.deny = true;
reconcile(engine, transfer, gallery, A);
assertEquals("pending", engine.items().get(0).destination);
engine = new SyncEngine(root, true);
gallery.deny = false;
reconcile(engine, transfer, gallery); // Export retry does not need the remote file still listed.
assertEquals(1, transfer.downloads.size());
assertEquals(1, gallery.saves);
reconcile(new SyncEngine(root, true), transfer, gallery, A);
assertEquals(2, gallery.attempts);
}
@Test public void thrownExportRetainsPendingReceiptAcrossRelaunch() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
gallery.fail = true;
fails(() -> reconcile(engine, transfer, gallery, A));
assertEquals("pending", engine.items().get(0).destination);
gallery.fail = false;
reconcile(new SyncEngine(root, true), transfer, gallery);
assertEquals(1, gallery.saves);
assertEquals(1, transfer.downloads.size());
}
@Test public void queuedShotAndStabilitySurviveRelaunch() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
transfer.truncated = true;
fails(() -> reconcile(engine, transfer, gallery, A));
SyncEngine relaunched = new SyncEngine(root, true);
transfer.truncated = false;
reconcile(relaunched, transfer, gallery, A);
assertEquals("appOnly", relaunched.items().get(0).destination);
assertEquals(0, gallery.attempts);
}
@Test public void firstObservationSurvivesRelaunch() throws Exception {
File root = temporary.newFolder();
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(new SyncEngine(root, false), transfer, gallery, A);
SyncEngine engine = new SyncEngine(root, false);
reconcile(engine, transfer, gallery, A);
assertEquals(1, engine.items().size());
}
@Test public void offThenOnNeverRetroExportsDownloadedOrQueuedItems() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), true);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
gallery.deny = true;
reconcile(engine, transfer, gallery, A);
engine.setGallery(false);
engine.setGallery(true);
gallery.deny = false;
reconcile(engine, transfer, gallery, A, B);
assertEquals("appOnly", engine.items().get(0).destination);
reconcile(engine, transfer, gallery, A, B);
assertEquals(1, gallery.saves); // Only the later B item.
assertEquals("appOnly", engine.items().stream().filter(i -> i.shot.id.equals(FIRST))
.findFirst().get().destination);
}
@Test public void switchingOffWhileDownloadBlockedChangesItsDisposition() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), true);
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
Transfer transfer = new Transfer() {
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
entered.countDown(); await(release); super.download(shot, file);
}
};
Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
try (Async async = new Async(() -> reconcile(engine, transfer, gallery, A))) {
try {
await(entered);
engine.setGallery(false); // Must finish while network is blocked.
engine.setGallery(true);
} finally { release.countDown(); }
async.finish();
}
assertEquals("appOnly", engine.items().get(0).destination);
assertEquals(0, gallery.attempts);
}
@Test public void switchingOffWhileExporterBlockedInvalidatesEligibility() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), true);
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
Gallery gallery = new Gallery() {
@Override public boolean export(SyncEngine.Shot shot, File original,
java.util.function.BooleanSupplier eligible) throws Exception {
assertTrue(eligible.getAsBoolean());
entered.countDown(); await(release);
assertFalse(eligible.getAsBoolean());
return false;
}
};
Transfer transfer = new Transfer();
observed(engine, transfer, gallery, A);
try (Async async = new Async(() -> reconcile(engine, transfer, gallery, A))) {
try { await(entered); engine.setGallery(false); engine.setGallery(true); }
finally { release.countDown(); }
async.finish();
}
assertEquals("appOnly", engine.items().get(0).destination);
}
@Test public void lateDownloadAfterStopCannotOverwriteFreshRunOrJournal() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
Transfer slow = new Transfer() {
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
entered.countDown(); await(release);
Files.write(file.toPath(), new byte[]{9, 9, 9, 9});
}
};
Transfer fresh = new Transfer(); Gallery gallery = new Gallery();
observed(engine, fresh, gallery, A);
try (Async async = new Async(() -> reconcile(engine, slow, gallery, A))) {
byte[] journal;
try {
await(entered); engine.stop(); engine.setGallery(false);
reconcile(engine, fresh, gallery, A);
journal = Files.readAllBytes(new File(root, "receipts.json").toPath());
} finally { release.countDown(); }
async.finish();
assertArrayEquals(journal, Files.readAllBytes(new File(root, "receipts.json").toPath()));
}
assertArrayEquals(new byte[4], Files.readAllBytes(engine.items().get(0).file.toPath()));
assertEquals("appOnly", new SyncEngine(root, true).items().get(0).destination);
}
@Test public void lateExportAfterStopDoesNotMarkFreshAppOnlyReceiptSaved() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer();
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
Gallery delayed = new Gallery() {
@Override public boolean export(SyncEngine.Shot shot, File original,
java.util.function.BooleanSupplier eligible) throws Exception {
entered.countDown(); await(release);
assertFalse(eligible.getAsBoolean());
return true; // Model a pre-stop external save whose callback arrived late.
}
};
observed(engine, transfer, delayed, A);
try (Async async = new Async(() -> reconcile(engine, transfer, delayed, A))) {
byte[] journal;
try {
await(entered); engine.stop(); engine.setGallery(false);
reconcile(engine, transfer, new Gallery(), A);
journal = Files.readAllBytes(new File(root, "receipts.json").toPath());
} finally { release.countDown(); }
async.finish();
assertArrayEquals(journal, Files.readAllBytes(new File(root, "receipts.json").toPath()));
}
assertEquals("appOnly", engine.items().get(0).destination);
}
@Test public void concurrentReconcileIsRejectedWithoutBlockingOnDownload() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), false);
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
Transfer slow = new Transfer() {
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
entered.countDown(); await(release); super.download(shot, file);
}
};
Gallery gallery = new Gallery(); observed(engine, slow, gallery, A);
try (Async async = new Async(() -> reconcile(engine, slow, gallery, A))) {
try {
await(entered);
try { reconcile(engine, new Transfer(), gallery, A); fail("Concurrent run accepted"); }
catch (IllegalStateException expected) { assertDownloadPending(engine, "appOnly"); }
} finally { release.countDown(); }
async.finish();
}
}
@Test public void corruptedAndIncompleteJournalsFailClosed() throws Exception {
for (String content : new String[]{"", "{", "null", "{}", "{\"gallery\":true}",
"{\"version\":2,\"initialized\":true,\"gallery\":true,\"records\":{}}",
"{\"version\":1,\"initialized\":true,\"gallery\":true,\"records\":{}} trailing"}) {
File root = temporary.newFolder();
File journal = new File(root, "receipts.json");
Files.write(journal.toPath(), content.getBytes(StandardCharsets.UTF_8));
fails(() -> new SyncEngine(root, true));
assertEquals(content, new String(Files.readAllBytes(journal.toPath()), StandardCharsets.UTF_8));
}
}
@Test public void damagedReceiptAndMissingOriginalFailClosed() throws Exception {
File root = temporary.newFolder();
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
SyncEngine engine = new SyncEngine(root, false);
observed(engine, transfer, gallery, A); reconcile(engine, transfer, gallery, A);
File journal = new File(root, "receipts.json");
String json = new String(Files.readAllBytes(journal.toPath()), StandardCharsets.UTF_8);
for (String damaged : new String[]{json.replace("downloaded", "unknown"),
json.replace("\"time\":100.5", "\"otherTime\":100.5"),
json.replace("\"size\":4", "\"size\":0"),
json.replace("\"time\":100.5", "\"time\":1e999")}) {
Files.write(journal.toPath(), damaged.getBytes(StandardCharsets.UTF_8));
fails(() -> new SyncEngine(root, false));
}
Files.write(journal.toPath(), json.getBytes(StandardCharsets.UTF_8));
Files.delete(engine.items().get(0).file.toPath());
fails(() -> new SyncEngine(root, false));
Files.delete(journal.toPath());
Files.write(new File(root, "originals/orphan").toPath(), new byte[]{1});
fails(() -> new SyncEngine(root, false));
}
@Test public void invalidListingCannotEstablishBaselineOrPartiallyAdvanceObservations() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
fails(() -> reconcile(engine, transfer, gallery, A, shot("../bad")));
reconcile(engine, transfer, gallery, A); // A is still baseline, not newly discovered.
reconcile(engine, transfer, gallery, A, B);
fails(() -> reconcile(engine, transfer, gallery, B, shot("../bad")));
assertTrue(transfer.downloads.isEmpty());
reconcile(engine, transfer, gallery, B);
assertEquals(Collections.singletonList(SECOND), transfer.downloads);
}
@Test public void metadataBoundsAndPathTraversalAreRejected() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
for (String id : new String[]{"../" + FIRST, "/" + FIRST, "123/../" + FIRST,
"123\\456\\20261024120000_1.jpg", "123/456/20261024120000_1.JPG",
"1234567890123/456/20261024120000_1.jpg", "123/123456789012345678901/20261024120000_1.jpg",
"123/456/2026102412000_1.jpg", "123/456/20261024120000_12345.jpg",
FIRST + "\n", FIRST + "/extra", "123/456/20261024120000_1.gif"}) {
fails(() -> reconcile(engine, transfer, gallery, shot(id)));
}
for (long size : new long[]{0, -1, 30L * 1024 * 1024 + 1, Long.MAX_VALUE}) {
fails(() -> reconcile(engine, transfer, gallery, new SyncEngine.Shot(FIRST, size, 1)));
}
for (double time : new double[]{Double.NaN, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY}) {
fails(() -> reconcile(engine, transfer, gallery, new SyncEngine.Shot(FIRST, 1, time)));
}
fails(() -> engine.reconcile(null, transfer, gallery));
fails(() -> reconcile(engine, transfer, gallery, (SyncEngine.Shot) null));
fails(() -> reconcile(engine, transfer, gallery, A, new SyncEngine.Shot(FIRST, 5, 1)));
assertTrue(transfer.downloads.isEmpty());
}
@Test public void exactMaximumMetadataAndThirtyMebibyteOriginalAreAccepted() throws Exception {
SyncEngine engine = new SyncEngine(temporary.newFolder(), false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
SyncEngine.Shot maximum = new SyncEngine.Shot(
"123456789012/12345678901234567890/20261024123456_1234.png", 30L * 1024 * 1024,
Double.MAX_VALUE);
observed(engine, transfer, gallery, maximum);
reconcile(engine, transfer, gallery, maximum);
assertEquals(maximum.size, engine.items().get(0).file.length());
assertEquals("appOnly", engine.items().get(0).destination);
}
@Test public void symbolicLinksCannotEscapeOriginalsOrReplaceJournal() throws Exception {
File root = temporary.newFolder(), outside = temporary.newFolder();
Files.createSymbolicLink(new File(root, "originals").toPath(), outside.toPath());
fails(() -> new SyncEngine(root, true));
Files.delete(new File(root, "originals").toPath());
File target = new File(outside, "receipts"); Files.write(target.toPath(), new byte[]{1});
Files.createSymbolicLink(new File(root, "receipts.json").toPath(), target.toPath());
fails(() -> new SyncEngine(root, true));
assertArrayEquals(new byte[]{1}, Files.readAllBytes(target.toPath()));
}
@Test public void nestedSymlinkIntroducedDuringDownloadCannotEscapeRoot() throws Exception {
File root = temporary.newFolder(), outside = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, false);
Transfer transfer = new Transfer() {
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
super.download(shot, file);
Files.createDirectory(new File(root, "originals").toPath());
Files.createSymbolicLink(new File(root, "originals/123").toPath(), outside.toPath());
}
};
Gallery gallery = new Gallery(); observed(engine, transfer, gallery, A);
fails(() -> reconcile(engine, transfer, gallery, A));
assertDownloadPending(engine, "appOnly");
assertEquals(0, outside.list().length);
}
@Test public void failedPreferenceWriteRollsBackGalleryAndPendingDisposition() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A); gallery.deny = true;
reconcile(engine, transfer, gallery, A);
File journal = new File(root, "receipts.json"); byte[] saved = Files.readAllBytes(journal.toPath());
Files.delete(journal.toPath()); Files.createDirectory(journal.toPath());
fails(() -> engine.setGallery(false));
assertEquals("pending", engine.items().get(0).destination);
Files.delete(journal.toPath()); Files.write(journal.toPath(), saved);
gallery.deny = false; reconcile(engine, transfer, gallery, A);
assertEquals(1, gallery.saves);
assertEquals("saved", engine.items().get(0).destination);
}
@Test public void failedQueueWriteDoesNotStartDownloadOrChangeMemory() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
File journal = new File(root, "receipts.json"); byte[] saved = Files.readAllBytes(journal.toPath());
Files.delete(journal.toPath()); Files.createDirectory(journal.toPath());
fails(() -> reconcile(engine, transfer, gallery, A));
assertTrue(transfer.downloads.isEmpty());
assertDownloadPending(engine, "appOnly");
Files.delete(journal.toPath()); Files.write(journal.toPath(), saved);
reconcile(engine, transfer, gallery, A);
assertEquals(1, engine.items().size());
}
@Test public void exportedImageWithFailedReceiptWriteRecoversThroughIdempotentExporter() throws Exception {
File root = temporary.newFolder();
File journal = new File(root, "receipts.json");
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer();
AtomicReference<byte[]> pendingJournal = new AtomicReference<>();
Gallery gallery = new Gallery() {
private boolean exists;
@Override public boolean export(SyncEngine.Shot shot, File file,
java.util.function.BooleanSupplier eligible) throws Exception {
attempts++;
if (exists) return true; // Recover the previous external save without another insert.
assertTrue(eligible.getAsBoolean());
exists = true; saves++;
pendingJournal.set(Files.readAllBytes(journal.toPath()));
Files.delete(journal.toPath()); Files.createDirectory(journal.toPath());
return true;
}
};
observed(engine, transfer, gallery, A);
fails(() -> reconcile(engine, transfer, gallery, A));
assertEquals("pending", engine.items().get(0).destination);
Files.delete(journal.toPath()); Files.write(journal.toPath(), pendingJournal.get());
SyncEngine relaunched = new SyncEngine(root, true);
reconcile(relaunched, transfer, gallery);
assertEquals("saved", relaunched.items().get(0).destination);
assertEquals(1, gallery.saves);
assertEquals(2, gallery.attempts);
assertEquals(1, transfer.downloads.size());
}
@Test public void failedBaselineWriteCannotTurnHistoryIntoNewScreenshots() throws Exception {
File root = temporary.newFolder();
File journal = new File(root, "receipts.json");
SyncEngine engine = new SyncEngine(root, false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
byte[] initialJournal = Files.readAllBytes(journal.toPath());
Files.delete(journal.toPath()); Files.createDirectory(journal.toPath());
fails(() -> reconcile(engine, transfer, gallery, A));
Files.delete(journal.toPath()); Files.write(journal.toPath(), initialJournal);
reconcile(engine, transfer, gallery, A);
reconcile(engine, transfer, gallery, A);
assertTrue(engine.items().isEmpty());
assertTrue(transfer.downloads.isEmpty());
}
@Test public void firstObservedOffStaysAppOnlyWhenEnabledBeforeStabilityIncludingRelaunch() throws Exception {
for (boolean restart : new boolean[]{false, true}) {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, false);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
assertDownloadPending(engine, "appOnly");
engine.setGallery(true);
if (restart) engine = new SyncEngine(root, true);
reconcile(engine, transfer, gallery, A);
assertTrue(engine.items().get(0).file.isFile());
assertEquals("appOnly", engine.items().get(0).destination);
assertEquals(0, gallery.attempts);
reconcile(new SyncEngine(root, true), transfer, gallery, A);
assertEquals(0, gallery.attempts);
}
}
@Test public void observingOnOffOnStaysAppOnlyBeforeStabilityIncludingRelaunch() throws Exception {
for (boolean restart : new boolean[]{false, true}) {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer(); Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
assertDownloadPending(engine, "pending");
engine.setGallery(false);
assertDownloadPending(engine, "appOnly");
if (restart) engine = new SyncEngine(root, false);
engine.setGallery(true);
if (restart) engine = new SyncEngine(root, true);
// An intervening metadata change must preserve the first observation's disposition.
SyncEngine.Shot changed = new SyncEngine.Shot(FIRST, 5, A.time + 1);
reconcile(engine, transfer, gallery, changed);
assertDownloadPending(engine, "appOnly");
reconcile(engine, transfer, gallery, changed);
assertTrue(engine.items().get(0).file.isFile());
assertEquals("appOnly", engine.items().get(0).destination);
assertEquals(0, gallery.attempts);
}
}
@Test public void permanentGalleryFailureAllowsLaterOriginalAndExportInSamePoll() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer();
IOException unavailable = new IOException("permanent Gallery failure");
Gallery gallery = new Gallery() {
@Override public boolean export(SyncEngine.Shot shot, File file,
java.util.function.BooleanSupplier eligible) throws Exception {
if (shot.id.equals(FIRST)) throw unavailable;
return super.export(shot, file, eligible);
}
};
observed(engine, transfer, gallery, A);
assertSame(unavailable, fails(() -> reconcile(engine, transfer, gallery, A)).getCause());
fails(() -> reconcile(engine, transfer, gallery, A, B));
assertFalse(item(engine, SECOND).file.exists()); // B has only one observation.
assertSame(unavailable, fails(() -> reconcile(engine, transfer, gallery, A, B)).getCause());
assertEquals(Arrays.asList(FIRST, SECOND), transfer.downloads);
assertTrue(item(engine, SECOND).file.isFile());
assertEquals("saved", item(engine, SECOND).destination);
assertEquals("pending", item(engine, FIRST).destination);
fails(() -> reconcile(engine, transfer, gallery, A, B));
assertEquals(2, transfer.downloads.size());
assertEquals(1, gallery.saves);
SyncEngine relaunched = new SyncEngine(root, true);
assertEquals("pending", item(relaunched, FIRST).destination);
assertEquals("saved", item(relaunched, SECOND).destination);
}
@Test public void permanentTransferFailureAllowsLaterOriginalInSamePollAndStaysQueued() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, false);
IOException unavailable = new IOException("permanent transfer failure");
Transfer transfer = new Transfer() {
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
if (shot.id.equals(FIRST)) { downloads.add(shot.id); throw unavailable; }
super.download(shot, file);
}
};
Gallery gallery = new Gallery();
observed(engine, transfer, gallery, A);
fails(() -> reconcile(engine, transfer, gallery, A));
fails(() -> reconcile(engine, transfer, gallery, A, B));
assertSame(unavailable, fails(() -> reconcile(engine, transfer, gallery, A, B)).getCause());
assertFalse(item(engine, FIRST).file.exists());
assertTrue(item(engine, SECOND).file.isFile());
assertEquals("appOnly", item(engine, SECOND).destination);
assertEquals(Arrays.asList(FIRST, FIRST, FIRST, SECOND), transfer.downloads);
SyncEngine relaunched = new SyncEngine(root, false);
fails(() -> reconcile(relaunched, transfer, gallery, A, B));
assertEquals(1, Collections.frequency(transfer.downloads, SECOND));
assertFalse(item(relaunched, FIRST).file.exists());
assertTrue(item(relaunched, SECOND).file.isFile());
}
@Test public void permanentTruncatedInvalidOrUndecodableImageDoesNotStarveLaterOriginal() throws Exception {
for (String mode : Arrays.asList("truncated", "invalid", "decoder")) {
SyncEngine engine = new SyncEngine(temporary.newFolder(), false);
Transfer transfer = new Transfer() {
private boolean first;
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
super.download(shot, file);
first = shot.id.equals(FIRST);
if (first && mode.equals("truncated")) {
try (RandomAccessFile contents = new RandomAccessFile(file, "rw")) {
contents.setLength(shot.size - 1);
}
}
}
@Override public boolean validImage(File file) throws Exception {
if (first && mode.equals("decoder")) throw new IOException("decoder failure");
return !first;
}
};
Gallery gallery = new Gallery();
reconcile(engine, transfer, gallery);
reconcile(engine, transfer, gallery, A, B);
IOException error = fails(() -> reconcile(engine, transfer, gallery, A, B));
assertTrue(error.getMessage().contains(FIRST));
assertEquals(Arrays.asList(FIRST, SECOND), transfer.downloads);
assertFalse(item(engine, FIRST).file.exists());
assertTrue(item(engine, SECOND).file.isFile());
fails(() -> reconcile(engine, transfer, gallery, A, B));
assertEquals(1, Collections.frequency(transfer.downloads, SECOND));
assertEquals(2, Collections.frequency(transfer.downloads, FIRST));
}
}
@Test public void fatalJournalCommitAfterRetryableFailureHaltsLaterDownloadsUntilRecovered() throws Exception {
File root = temporary.newFolder();
File journal = new File(root, "receipts.json");
SyncEngine engine = new SyncEngine(root, true);
Transfer transfer = new Transfer();
SyncEngine.Shot third = shot("123/456/20261024120002_3.jpg");
IOException unavailable = new IOException("permanent Gallery failure");
AtomicReference<byte[]> pendingJournal = new AtomicReference<>();
Gallery gallery = new Gallery() {
@Override public boolean export(SyncEngine.Shot shot, File file,
java.util.function.BooleanSupplier eligible) throws Exception {
if (shot.id.equals(FIRST)) throw unavailable;
if (shot.id.equals(SECOND) && pendingJournal.get() == null) {
assertTrue(eligible.getAsBoolean());
pendingJournal.set(Files.readAllBytes(journal.toPath()));
Files.delete(journal.toPath()); Files.createDirectory(journal.toPath());
}
return true;
}
};
reconcile(engine, transfer, gallery);
reconcile(engine, transfer, gallery, A, B, third);
IOException fatal = fails(() -> reconcile(engine, transfer, gallery, A, B, third));
assertNotSame(unavailable, fatal.getCause()); // Journal failure supersedes deferred Gallery failure.
assertTrue(journal.isDirectory());
assertEquals(Arrays.asList(FIRST, SECOND), transfer.downloads);
assertEquals("pending", item(engine, SECOND).destination);
assertFalse(item(engine, third.id).file.exists());
Files.delete(journal.toPath()); Files.write(journal.toPath(), pendingJournal.get());
SyncEngine relaunched = new SyncEngine(root, true);
assertSame(unavailable, fails(() -> reconcile(relaunched, transfer, gallery, A, B, third)).getCause());
assertEquals(Arrays.asList(FIRST, SECOND, third.id), transfer.downloads);
assertEquals("saved", item(relaunched, SECOND).destination);
assertEquals("saved", item(relaunched, third.id).destination);
assertEquals("pending", item(relaunched, FIRST).destination);
}
@Test public void fatalOriginalInstallationFailureHaltsLaterDownloads() throws Exception {
File root = temporary.newFolder();
SyncEngine engine = new SyncEngine(root, false);
Transfer transfer = new Transfer() {
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
super.download(shot, file);
if (shot.id.equals(FIRST)) Files.write(new File(root, "originals").toPath(), new byte[]{1});
}
};
Gallery gallery = new Gallery();
reconcile(engine, transfer, gallery);
reconcile(engine, transfer, gallery, A, B);
IOException fatal = fails(() -> reconcile(engine, transfer, gallery, A, B));
assertEquals("Invalid originals directory", fatal.getMessage());
assertEquals(Collections.singletonList(FIRST), transfer.downloads);
assertFalse(item(engine, FIRST).file.exists());
assertFalse(item(engine, SECOND).file.exists());
}
@Test public void lateRetryableFailureAfterStopCannotChangeFreshJournalOrSurfaceAnError() throws Exception {
File root = temporary.newFolder();
File journal = new File(root, "receipts.json");
SyncEngine engine = new SyncEngine(root, false);
Transfer fresh = new Transfer(); Gallery gallery = new Gallery();
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
Transfer slow = new Transfer() {
@Override public void download(SyncEngine.Shot shot, File file) throws Exception {
entered.countDown(); await(release);
throw new IOException("late transfer failure");
}
};
reconcile(engine, fresh, gallery);
reconcile(engine, fresh, gallery, A, B);
try (Async async = new Async(() -> reconcile(engine, slow, gallery, A, B))) {
byte[] saved;
try {
await(entered); engine.stop();
reconcile(engine, fresh, gallery, A, B);
saved = Files.readAllBytes(journal.toPath());
} finally { release.countDown(); }
async.finish();
assertArrayEquals(saved, Files.readAllBytes(journal.toPath()));
}
assertEquals(Arrays.asList(FIRST, SECOND), fresh.downloads);
assertTrue(item(engine, FIRST).file.isFile());
assertTrue(item(engine, SECOND).file.isFile());
}
}
+1
View File
@@ -0,0 +1 @@
plugins { id 'com.android.application' version '9.1.0' apply false }
+2
View File
@@ -0,0 +1,2 @@
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
android.useAndroidX=true
Binary file not shown.
+8
View File
@@ -0,0 +1,8 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-9.4.1-bin.zip
networkTimeout=10000
validateDistributionUrl=true
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionSha256Sum=2ab2958f2a1e51120c326cad6f385153bb11ee93b3c216c5fccebfdfbb7ec6cb
Vendored Executable
+248
View File
@@ -0,0 +1,248 @@
#!/bin/sh
#
# Copyright © 2015 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SPDX-License-Identifier: Apache-2.0
#
##############################################################################
#
# Gradle start up script for POSIX generated by Gradle.
#
# Important for running:
#
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
# noncompliant, but you have some other compliant shell such as ksh or
# bash, then to run this script, type that shell name before the whole
# command line, like:
#
# ksh Gradle
#
# Busybox and similar reduced shells will NOT work, because this script
# requires all of these POSIX shell features:
# * functions;
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
# * compound commands having a testable exit status, especially «case»;
# * various built-in commands including «command», «set», and «ulimit».
#
# Important for patching:
#
# (2) This script targets any POSIX shell, so it avoids extensions provided
# by Bash, Ksh, etc; in particular arrays are avoided.
#
# The "traditional" practice of packing multiple parameters into a
# space-separated string is a well documented source of bugs and security
# problems, so this is (mostly) avoided, by progressively accumulating
# options in "$@", and eventually passing that to Java.
#
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
# see the in-line comments for details.
#
# There are tweaks for specific operating systems such as AIX, CygWin,
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/2d6327017519d23b96af35865dc997fcb544fb40/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
#
##############################################################################
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
app_path=$0
# Need this for daisy-chained symlinks.
while
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
[ -h "$app_path" ]
do
ls=$( ls -ld "$app_path" )
link=${ls#*' -> '}
case $link in #(
/*) app_path=$link ;; #(
*) app_path=$APP_HOME$link ;;
esac
done
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit
# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
warn () {
echo "$*"
} >&2
die () {
echo
echo "$*"
echo
exit 1
} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD=java
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi
# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
fi
# Collect all arguments for the java command, stacking in reverse order:
# * args from the command line
# * the main class name
# * -classpath
# * -D...appname settings
# * --module-path (only if needed)
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
if "$cygwin" || "$msys" ; then
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
JAVACMD=$( cygpath --unix "$JAVACMD" )
# Now convert the arguments - kludge to limit ourselves to /bin/sh
for arg do
if
case $arg in #(
-*) false ;; # don't mess with options #(
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
[ -e "$t" ] ;; #(
*) false ;;
esac
then
arg=$( cygpath --path --ignore --mixed "$arg" )
fi
# Roll the args list around exactly as many times as the number of
# args, so each arg winds up back in the position where it started, but
# possibly modified.
#
# NB: a `for` loop captures its iteration list before it begins, so
# changing the positional parameters here affects neither the number of
# iterations, nor the values presented in `arg`.
shift # remove old arg
set -- "$@" "$arg" # push replacement arg
done
fi
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Collect all arguments for the java command:
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
# and any embedded shellness will be escaped.
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
# treated as '${Hostname}' itself on the command line.
set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
-jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
#
# In Bash we could simply go:
#
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
# set -- "${ARGS[@]}" "$@"
#
# but POSIX shell has neither arrays nor command substitution, so instead we
# post-process each arg (as a line of input to sed) to backslash-escape any
# character that might be a shell metacharacter, then use eval to reverse
# that process (while maintaining the separation between arguments), and wrap
# the whole thing up as a single "set" statement.
#
# This will of course break if any of these variables contains a newline or
# an unmatched quote.
#
eval "set -- $(
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
xargs -n1 |
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
tr '\n' ' '
)" '"$@"'
exec "$JAVACMD" "$@"
+93
View File
@@ -0,0 +1,93 @@
@rem
@rem Copyright 2015 the original author or authors.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem https://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@rem SPDX-License-Identifier: Apache-2.0
@rem
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@rem
@rem ##########################################################################
@rem Set local scope for the variables with windows NT shell
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
@rem Find java.exe
if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:findJavaFromJavaHome
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
if exist "%JAVA_EXE%" goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:execute
@rem Setup the command line
@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -jar "%APP_HOME%\gradle\wrapper\gradle-wrapper.jar" %*
:end
@rem End local scope for the variables with windows NT shell
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal
:omega
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env python3
"""Use the same reproducible headset bundle as the iPhone app."""
import pathlib
import subprocess
import sys
root = pathlib.Path(__file__).resolve().parents[2]
out = pathlib.Path(sys.argv[1]).resolve()
out.mkdir(parents=True, exist_ok=True)
version = subprocess.check_output(
[sys.executable, str(root / 'ios/scripts/make_frame_bundle.py'),
str(out / 'frame-bundle.tar.gz')], cwd=root, text=True).strip().splitlines()[-1]
(out / 'frame-bundle.version').write_text(version + '\n')
+4
View File
@@ -0,0 +1,4 @@
pluginManagement { repositories { google(); mavenCentral(); gradlePluginPortal() } }
dependencyResolutionManagement { repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS); repositories { google(); mavenCentral() } }
rootProject.name = 'FrameControl'
include ':app'
+85
View File
@@ -0,0 +1,85 @@
# Frame Control for Android phones
The native Android companion lives in `android/`. It requires Android 10 or
later and a Steam Frame with Developer Mode and SSH enabled. This is a new
implementation, separate from the headset's `frame/android` Lepton tools.
## Pair and sync
1. Enter the headset address, SSH port and username (normally `steamos`).
2. Choose **Pair headset**, compare the displayed SSH fingerprint with the
headset's identity, and accept it. Enter the Developer Mode password once.
The app installs this phone's SSH public key and verifies key login before
storing the pairing. The private key stays in Android Keystore; the
password is not saved. A changed headset host key is refused.
3. Choose **Start syncing** and allow notifications. The persistent notification
shows the active connected-device session and offers **Stop syncing**.
4. Take a new screenshot. Its original downloads into Frame Control. **Save
screenshots to Gallery** is enabled by default and adds it to
`Pictures/Frame Control`. Android 10+ lets an app create its own Gallery
items without broad photo-reading permission.
The first successful listing establishes a history baseline. Existing screenshots
are not imported automatically. The app waits for two equal size/time listings,
checks the original's byte count and image decode, and saves it atomically.
Identity includes the paired headset, Steam account, game and filename.
Turning Gallery off keeps automatic downloads inside Frame Control. Pending
exports become app-only; enabling it again does not export those earlier items.
Previously saved Gallery items stay there. Original files remain available from
**Open / share original** when disconnected. The app keeps each headset's
library separately.
Gallery export uses a deterministic app-owned MediaStore row. A pending row is
rewritten after interruption, and a completed row is recognized before inserting
another. A failed export leaves the app original available for retry. Preferences
and the final Gallery commit are serialized so a disabled setting is respected.
**Headset controls** opens the shared Frame Control web interface through a
loopback-only SSH tunnel. The server has a fresh session key and listens only on
the headset's loopback interface. The tunnel and server end with the phone's
SSH session. This initial Android shell does not implement all iPhone-specific
web bridges; native screenshot originals use the downloads screen.
## Background behavior and release boundary
Sync is an explicit, user-started Android `connectedDevice` foreground service.
It can continue after leaving the app while its notification is present and
reconnects when the headset returns. It is not a Firebase push receiver.
Android battery-saving/Doze, loss of network, force-stop or process termination
can delay or end the session. The app does not request battery-optimization
exemption, use unrelated background modes, or restart itself after force-stop.
Restart sync from the app if the session ends.
The full cross-platform request still requires actual locked-phone background
proof. iOS APNs delivery is a separate unimplemented dependency. An emulator
test is not evidence that either physical platform meets that requirement.
Do not publish this as a verified always-on screenshot feature.
## Build and verify
Use JDK 17 and an Android SDK containing platform 36/build-tools 36.0.0:
```sh
export JAVA_HOME=/path/to/jdk17
export ANDROID_HOME=/path/to/android-sdk
./android/gradlew -p android assembleDebug testDebugUnitTest lintDebug
```
The Gradle wrapper pins version 9.4.1 and its distribution checksum. The build
generates the same reproducible headset bundle as iOS, including the existing
SHA-256-checked KDE Connect package fetch. Generated assets and build outputs
are ignored. No signing key, headset password or machine-specific SDK path is
committed. `assembleDebug` produces `android/app/build/outputs/apk/debug/app-debug.apk`.
It uses a development signing identity, not a configured Play release key.
`assembleDebugAndroidTest` builds instrumentation for real MediaStore round-trip,
retry recovery and app-only behavior. Optional real-headset hooks exist only in
the separate test APK (`ScreenshotDeviceTest`), not in the product app. They
provision a public test key and permit original/Gallery hash readback without
adding a product authentication bypass. Remove the temporary headset key and
fixtures after live tests.
No Play app creation, store upload, invitation or public release is part of a
debug build. Preserve a dedicated release signing identity and verify physical
devices before a production release.
@@ -0,0 +1,168 @@
# Android phone screenshot sync evidence — 2026-10-24
Work: [#108](https://github.com/saphid/frame-control/issues/108). The user
authorized a new Android companion after the earlier APK could not be located.
This is a development build, not a production or Play release.
## Implemented
Native Android 10+ companion in `android/`, with pinned SSH transport and a
non-exportable Android Keystore phone key. Pairing checks the host fingerprint
before accepting a password and verifies key login before saving the pairing.
The shared headset server runs through a session-bound loopback tunnel.
All retained libraries receive Gallery opt-out, including previously paired
headsets. Paging keeps older offline originals reachable beyond the first 100.
A failed image or Gallery export retries independently without starving later
downloads; local persistence errors still halt immediately.
The user starts a connected-device foreground service with an ongoing Stop
notification. New screenshot originals download automatically. Gallery is ON
by default; turning it off latches pending and newly discovered items to
app-only, including across relaunch and later re-enabling. Historical captures
are baselined instead of imported. Receipts, original files, and MediaStore
recovery support interrupted downloads/exports without duplicate Gallery rows.
See [usage and background limits](../android-phone.md).
## Build and automated checks
- JDK 17, Gradle 9.4.1, AGP 9.1.0; compile/target API 36, minimum API 29.
- `assembleDebug testDebugUnitTest assembleDebugAndroidTest lintDebug` exited
zero after review fixes: `/tmp/frame-108-android-build-final-r5.log`.
- 34 JVM sync-engine tests, zero failures/errors/skips. Tests cover baseline,
stability, identity, metadata limits, destination latching, retries, atomic
receipts, corrupt state, unsafe paths, late cancellation callbacks, per-item
failure isolation and fatal persistence errors.
- API 36 arm64 emulator MediaStore tests passed: original byte round-trip,
app-only across relaunch, interrupted pending row recovery and retry dedup.
The final rebuilt APK passed **five selected native checks, no skips**:
three MediaStore/multiple-library cases plus the actual headset fixture
Gallery hash (`/tmp/frame-108-android-final-device-r4.log`, four tests), then
app-only original/no-Gallery readback
(`/tmp/frame-108-android-final-apponly-device-r4.log`, one test).
- Existing server suite: 1,333 tests, 11 skips, exit zero,
`/tmp/frame-108-android-server-tests.log`. Website: 29 passed, zero skips,
`/tmp/frame-108-android-site-tests.log`.
- Lint has no errors. Warnings include dependency/target freshness, English UI
literals, backup-version compatibility, and the shared controls' JavaScript.
## Actual headset → Android emulator → Gallery
Target: task-owned headless API 36 emulator `emulator-5588`, with the actual
Steam Frame reached over SSH. Initial baseline contained nine historical
captures and imported none. An emulator-generated public test key was installed
temporarily through the existing trusted SSH alias; no password was exposed.
The unauthenticated native host-key probe matched the trusted headset key.
Synthetic fixture IDs use a separate test account/game namespace; this verifies
new-file delivery from the real headset, not the hardware capture shortcut.
| Check | Result |
| --- | --- |
| Gallery default ON | `999999999990/108/20261024100101_1.png` automatically downloaded and receipt became `saved` |
| Source / app / Gallery SHA-256 | `c9411d718c8a3477b7c7a6eed5ab827570605fce977bdeef5aaf11d29ba88316` |
| MediaStore dedup | Exactly one matching Gallery row; native test reads its original bytes |
| Shared headset controls | Actual authenticated WebView loaded the headset app library |
| Gallery OFF, app on Home | Second fixture automatically downloaded in 4.293 seconds (one synthetic trial) |
| App-only source / app SHA-256 | `6a1513b2dc3459b457f7dd86adcde7298d99dccc31c73904d7c6698703edb961` |
| App-only independent readback | One selected native test passed: correct app original and no matching Gallery row |
| Re-enable and restart | App-only receipt and absent Gallery row preserved; first image remains saved, no duplicate |
| Open/share original | Native chooser opened for the app-only original and was canceled; nothing sent |
| Final r5 APK, Gallery ON while Home | Third new fixture `999999999990/108/20261024101413_1.png` reached Gallery in 5.288 seconds (one synthetic trial); same source/app/Gallery hash as the first fixture, one matching row |
Background readback: `/tmp/frame-108-android-background-device-r1.log`.
Home was foreground with the app service active; the screen stayed on. This
is not a locked/Doze or long-duration physical-phone test.
The final r5 background Gallery readback passed separately (one selected native
test, no skip): `/tmp/frame-108-android-final-background-gallery-r5.log`.
Sanitized [machine readback](android-phone-screenshot-sync-2026-10-24/live-proof.json)
is committed; local full readback: `/tmp/frame-108-android-live-proof.json`. Fixture source
metadata: `/tmp/frame-108-android-fixtures.json`. The optional provisioning test
passed separately. Earlier full-class instrumentation runs included two
optional skipped hooks; those are not counted as four executed scenarios.
![Native saved receipt with Gallery enabled](android-phone-screenshot-sync-2026-10-24/gallery-on.png)
![Gallery re-enabled without backfilling app-only downloads](android-phone-screenshot-sync-2026-10-24/reenabled.png)
![Native connection and Gallery default](android-phone-screenshot-sync-2026-10-24/setup.png)
![Shared controls loaded from the real headset](android-phone-screenshot-sync-2026-10-24/controls.png)
## Verification boundaries
No physical Android phone, Doze reliability, hardware screenshot latency,
password-pairing UI, Play signing identity or production upload has been
verified. Android sync uses an explicit foreground session, not Firebase.
The iOS foreground implementation has separate
[evidence](phone-screenshot-sync-2026-10-24.md); APNs, its delivery service,
notification extension and locked-phone Photos export remain unimplemented or
unverified. The user's full-feature tested-and-working release condition is
not satisfied.
T3's device tools incorrectly inferred the Android SDK from the platform-tools
Cask and could not open this emulator. The Astra worker used the app-owned
agent-device launcher with a task-local session/config and the correct SDK.
No global SDK/T3 settings or physical devices were changed.
UI verification: provider `codex`, model `gpt-6-astra`, completed task
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-ui-astra-20261024-r1`.
Report: `/tmp/frame-108-android-ui-report.md`.
## Review and final artifacts
The APK is `android/app/build/outputs/apk/debug/app-debug.apk`, signed with the
development identity. No store upload or release has been performed.
Final r6 APK SHA-256: `e6a99a35154c21293975ea165eb967d8cc7ad84565f063255c3848ada1169f6e`.
Preferred different-model review (`claudeAgent` / `claude-opus-5-5`) failed
before work with `Claude API rate limit reached. Try again later.` Task:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-opus-20261024-r1`.
Implementation and independent fresh fallback reviews used Sol; they are not
different-model evidence.
Provider `codex_secondary`, model `gpt-6.1-sol`, high, completed round 1:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-sol-20261024-r1`.
All four findings were accepted and fixed: failed-item starvation, API-29
pending-row recovery query, opt-out for other retained headset libraries, and
inaccessible originals beyond the latest 100. Report:
`/tmp/frame-108-android-review-sol-r1.md`. No CODING_STANDARDS.md was found in
the task or primary worktree; CONTRIBUTING.md was included in review context.
Provider `codex_secondary`, model `gpt-6.1-sol`, high, completed round 2:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-sol-20261024-r2`.
It cleared all four prior fixes and independently reran 34 JVM tests, then found
a library-switch cache identity issue: matching receipt fields could retain a
Share action bound to the previous headset. The refresh signature now includes
the absolute original path, including the headset library root.
The final r6 build (`assembleDebug assembleDebugAndroidTest lintDebug`) exited
zero: `/tmp/frame-108-android-build-r6.log`. A new native regression launches
the real activity, switches between two libraries with matching receipt fields
but different original bytes, and verifies that Share now selects B’s bytes.
The chooser is intercepted; nothing is sent externally. This regression and
the three existing MediaStore/multiple-library cases passed together: **four
selected tests, no skips**, `/tmp/frame-108-android-final-device-r6.log`. Core
code is unchanged from the 34 independently passing JVM tests. The r6 change
does not alter the r5 transfer/storage path exercised against the real headset.
Provider `codex_secondary`, model `gpt-6.1-sol`, high, completed round 3:
`node:delegated-task:command%3Amcp%3Ad5d74833-0742-471e-af70-a23afbe44dab%3Adelegate-task%3Aframe-108-android-review-sol-20261024-r3`.
**No actionable findings** in the frozen 25-file executable target (`android/`
and `.github/workflows/android.yml`), with the remaining share-binding finding
resolved. All executable fingerprints stayed unchanged during review. Report:
`/tmp/frame-108-android-review-sol-r3.md`. The review inspected the passing r6
native/build logs and unchanged core covered by round 2’s independent 34-test
run; it did not claim new device runs or release validation.
Cleanup removed the one exact temporary public key and all three exact synthetic
headset fixture paths, verified absent. Other authorized keys and real captures
were preserved. Readback: `/tmp/frame-108-android-cleanup.json`. The task-owned
`emulator-5588` was shut down after the live tests, then restarted only for
the r6 share-binding regression and shut down again;
no other device was stopped. Build artifacts, the AVD and local test logs remain
available for follow-up verification.
@@ -0,0 +1 @@
{"removedTestKeys": 1, "removedFixtures": ["999999999990/108/20261024100101_1.png", "999999999990/108/20261024100102_1.png", "999999999990/108/20261024101413_1.png"], "testKeyAbsent": true, "allFixturesAbsent": true}
Binary file not shown.

After

Width:  |  Height:  |  Size: 391 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 226 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 185 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 177 KiB

@@ -0,0 +1,57 @@
{
"gallery": {
"id": "999999999990/108/20261024100101_1.png",
"sha256": "c9411d718c8a3477b7c7a6eed5ab827570605fce977bdeef5aaf11d29ba88316",
"bytes": 3620,
"createdAt": 1792836168.6100879,
"receipt": {
"destination": "saved",
"observations": 2,
"shot": {
"id": "999999999990/108/20261024100101_1.png",
"size": 3620,
"time": 1792836168.6217859
},
"state": "downloaded"
},
"readbackAt": 1792836209.819923
},
"appOnly": {
"id": "999999999990/108/20261024100102_1.png",
"sha256": "6a1513b2dc3459b457f7dd86adcde7298d99dccc31c73904d7c6698703edb961",
"bytes": 3623,
"createdAt": 1792836368.616906,
"background": "Home foreground; connectedDevice service active",
"receipt": {
"destination": "appOnly",
"observations": 2,
"shot": {
"id": "999999999990/108/20261024100102_1.png",
"size": 3623,
"time": 1792836368.6332257
},
"state": "downloaded"
},
"readbackAt": 1792836372.909552,
"elapsedSeconds": 4.293
},
"finalBackgroundGallery": {
"id": "999999999990/108/20261024101413_1.png",
"sha256": "c9411d718c8a3477b7c7a6eed5ab827570605fce977bdeef5aaf11d29ba88316",
"bytes": 3620,
"createdAt": 1792836871.5926661,
"background": "Home foreground; connectedDevice service active; final r5 APK",
"receipt": {
"destination": "saved",
"observations": 2,
"shot": {
"id": "999999999990/108/20261024101413_1.png",
"size": 3620,
"time": 1792836871.6072419
},
"state": "downloaded"
},
"readbackAt": 1792836876.8808918,
"elapsedSeconds": 5.288
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 187 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 238 KiB

@@ -0,0 +1,119 @@
# iOS foreground screenshot downloads — 2026-10-24
This is implementation and proof for the foreground iOS portion of
[#108](https://github.com/saphid/frame-control/issues/108). It is not the full
requested background/cross-platform feature and is not released.
## Source and merge
Dedicated worktree: `/Users/saphid/.t3/worktrees/steam-frame/feat-phone-screenshot-autosync-108`.
Branch: `feat/phone-screenshot-autosync-108`.
First merged fetched main
[1d4c57d6](https://github.com/saphid/frame-control/commit/1d4c57d6)
into the existing Mac-space parent
[3fdc6c46](https://github.com/saphid/frame-control/commit/3fdc6c46).
Merge commit: [52772323](https://github.com/saphid/frame-control/commit/52772323).
The dirty primary checkout was left untouched. Review found and fixed the PC
helper's missing public route for the extracted `mac-stream.js` dependency.
Independent merge re-review ran 45 tests successfully, one skipped, and found
no remaining actionable findings in its seven-file target.
## Implemented behavior
- Native polling every four seconds while active, independent of the visible web tab.
- First successful listing becomes the durable baseline; only new captures queue.
- Two equal metadata observations, exact byte count and complete ImageIO decode
precede an atomic original write outside purgeable caches.
- Receipts and originals distinguish pinned headset identity, Steam account,
app ID and filename. Reconnect/relaunch resumes pending work.
- Photos is enabled by default. Add-only permission is requested in foreground.
Denied/failed exports keep originals; app-only captures never automatically
export after re-enabling the setting.
- Native downloads sheet exposes retained files through ShareLink, including
during connection failure/relaunch. Ambiguous interrupted Photos commits need
explicit recovery rather than risking an automatic duplicate.
- Backgrounding stops polling; no unrelated background mode is used.
## Automated verification
Normal signed Simulator build and full native test suite: **20 tests passed,
zero failures**, including eight screenshot tests. Result bundle:
`/tmp/frame-108-ios-all-r3.xcresult`; log: `/tmp/frame-108-ios-all-r3.log`.
Both command exit status and xcresult test count were checked. Private
DerivedData cleanup completed. Test target was the independently created
Simulator `F250F91E-2E7D-4833-B029-A275F11FDC7D` on iOS 26.5.
Screenshot tests cover baseline/stability, truncated data, restart dedup,
Photos failure/retry, changes during download/consent, cancellation,
account/app identity, corrupt-ledger failure, uncertain export handling,
late success and failure across journal replacement, default preference and
offline app-only originals. Download/export dependencies in these tests are
injected; the live test below exercises actual transport and PhotoKit.
Python iOS packaging/Foundation contracts: 5 passed. Page scripts: 5 passed.
`swiftc -typecheck` for the journal and scoped `git diff --check` passed.
## Actual headset → Simulator → Photos proof
Ran the production signed app against the real headset over its pinned SSH
connection and the actual deployed `/api/shots` and `/api/shots/image` endpoints.
Created two temporary valid PNGs under a unique synthetic Steam account/app
directory after the app had baselined the existing nine screenshots.
| Check | Observed result |
|---|---|
| Default Photos ON | Original downloaded and receipt marked `saved`; Photos contained one byte-identical original |
| Source / app copy / Photos SHA-256 | `a36eb2d497e6ecce17c767189a70b3b7e732a22309d2421cdc715cf13a7668ac` |
| Photos OFF | Second original downloaded with `appOnly`; no matching Photos original |
| Re-enable + relaunch | Second receipt stayed `appOnly`; first still had exactly one Photos original |
Machine readback: `/tmp/frame-108-live-proof.json`. Photos readback hashed files
in this dedicated Simulator's own Media/DCIM directory, independently of the
app receipt. No personal phone Photos database was accessed. This is a
synthetic capture-file test, not proof of the headset screenshot-button path,
capture latency, physical-device behavior, or background delivery.
The OFF preference was applied through Simulator defaults on restart, not a
UI tap. The production setting setter and persisted opt-out were exercised in
the native tests. Astra inspected the real native sheet in both states and
found no major clipping, overlap or actionable visual defect.
![Photos enabled and app-only setting](phone-screenshot-sync-2026-10-24/settings-proof.png)
An initial unsigned test build could not retain a stable Keychain identity;
the real transport test used normal Simulator signing. App launches with a
console attached were kept alive for the live checks.
## Review provenance
Preferred Opus routes failed before work with API rate limits. Implementation
used the current GPT-6.1 Sol session; reviews are disclosed as fresh same-model
checks, not different-model evidence.
Provider `codex`, model `gpt-6.1-sol`, high, completed:
- r1: `node:delegated-task:command%3Amcp%3Ac03980f6-1e71-4cc4-af1a-1d0961b9aaad%3Adelegate-task%3Aframe-108-ios-sync-review-20261024-r1`.
Found stale export callback persistence and missing offline access. Both fixed.
- r2: `node:delegated-task:command%3Amcp%3Ac03980f6-1e71-4cc4-af1a-1d0961b9aaad%3Adelegate-task%3Aframe-108-ios-sync-review-20261024-r2`.
No actionable findings in the complete executable diff against `52772323`
plus the four new Swift files. Report: `/tmp/frame-108-ios-sync-review-r2.md`.
Visual inspection used provider `codex`, `gpt-6-astra`, completed task
`frame-108-downloads-proof-inspection-20261024-r3`; report:
`/tmp/frame-108-downloads-proof-inspection.md`.
## Remaining release requirements
- Actual APNs delivery, notification extension/shared state, persistent headset
event source and authenticated delivery service are still unimplemented.
Main-app push capability is not configured; no usable provider credential
location was found in the bounded checks. Existing App Store Connect signing
access is not an APNs provider key.
- Physical locked/background iPhone download and Photos export remain unproven.
- A new native Android companion was subsequently authorized and implemented.
See [Android phone](../android-phone.md) for its explicit foreground-service
behavior and [Android evidence](android-phone-screenshot-sync-2026-10-24.md)
for the build and emulator checks. This does not supply physical-device proof.
- No TestFlight upload, store submission or release was performed. The user's
release condition requires the full behavior to be tested and working.
Binary file not shown.

After

Width:  |  Height:  |  Size: 403 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 103 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

+171 -2
View File
@@ -57,6 +57,25 @@ makes.
| Sleep, restart, shut down ask in a terminal | The page asks for the Developer Mode password |
| Compatibility reports kept on the computer | Kept on the Frame (`~/.local/share/Frame Control`) |
## Automatic screenshot downloads (in development)
The screenshot-download implementation on this branch copies new Steam screenshots
while Frame Control is open and connected. It starts with the first successful
screenshot listing as its baseline; older captures still use the existing Save
action. Originals remain inside the app, and **Save screenshots to Photos** is
enabled by default, subject to add-only Photos permission.
Open **Downloads & Photos** on the Screenshots page, or **Screenshot downloads**
on the native connection screen. Turn the Photos setting off to keep new downloads
only in Frame Control. Re-enabling it does not export those app-only captures.
The local list and its Share buttons work without a headset connection. Failed
Photos exports retain the original and retry while the app is open; an interrupted
Photos transaction needs a manual check before saving again to avoid duplicates.
Background notification delivery and Android support are not implemented by this
change, and it has not been released. See the [verification record](evidence/phone-screenshot-sync-2026-10-24.md)
and [remaining work](https://github.com/saphid/frame-control/issues/108).
## Building
```sh
@@ -69,6 +88,88 @@ The build packs the Frame bundle from the checkout, so the phone always runs
the page and server from the same commit. Running on a phone needs your own
signing team in Xcode (Signing & Capabilities).
## TestFlight
Join the [Frame Control beta](https://testflight.apple.com/join/9xjW91g6),
install Apple's TestFlight app, then tap **Install** or **Update** next to Frame
Control. Internal account testers can also use their invitation. The first
working phone build was version 0.1.0, build 202610140516, uploaded on
2026-10-14. On 2026-10-16, build **202610160824** was uploaded and approved for
internal and external testing. App Store Connect confirmed both builds were in
the external **Public Beta** group, with its public link enabled and a limit of
50 testers. The new build declares the encryption-compliance answer in the app
and preserves the previous build's exact server bundle.
### Upload a build
You need Xcode, XcodeGen, a paid Apple Developer Program team, an App Store
Connect app record for `com.saphid.framecontrol`, and an App Store Connect team
API key with the signing access your release setup needs. This Mac relies on
Apple's cloud-managed distribution certificates: an **App Manager** key was
rejected with `Cloud signing permission error`, so this setup needs **Admin**
access. A setup with a local distribution signing identity may be able to use
a narrower role. Create the key under **Users and Access →
Integrations → App Store Connect API**. Download it directly to
`~/.appstoreconnect/private_keys/AuthKey_<KEY_ID>.p8`, outside the repository,
and restrict it to its owner (`chmod 600`). Apple allows one download. Keep
the private key out of chat, screenshots, logs and Git.
From the repository root:
```sh
ASC_KEY_ID=YOUR_KEY_ID \
ASC_ISSUER_ID=YOUR_ISSUER_UUID \
ASC_KEY_PATH="$HOME/.appstoreconnect/private_keys/AuthKey_YOUR_KEY_ID.p8" \
TEAM_ID=YOUR_TEAM_ID \
ios/scripts/testflight.sh
```
The script regenerates the Xcode project, archives with automatic signing,
checks the archived build number, and uploads to App Store Connect. `TEAM_ID`
defaults to this project's team, `NJZUMEA4BN`. `BUILD_NUMBER` defaults to UTC
`YYYYMMDDHHMM`; wait for a new minute between uploads, or supply a unique,
higher build number. The archive, export options and export output are under
`ios/build/testflight/<BUILD_NUMBER>/` (ignored by Git).
Build from a reviewed, clean checkout. The build packs the server, web pages
and helpers from the current checkout, including uncommitted changes; it does
not limit the bundle to the last commit.
An upload succeeding means Apple received the build. Wait for it to finish
processing in **App Store Connect → Frame Control → TestFlight** and check
that it is ready for testing. `ios/project.yml` declares
`ITSAppUsesNonExemptEncryption: false`, so each generated app declares that it
uses no non-exempt encryption, matching the previous builds' export-compliance
answers. Reassess that declaration if the app's cryptography changes.
To replace a key, create and securely download the replacement, verify that it
can access this app, then revoke the old key in Users and Access. Update the
upload environment to use the replacement. Revocation does not remove an
already uploaded TestFlight build from testers' phones.
### Add testers
**Internal:** add an existing App Store Connect user with an eligible role to
an internal testing group, and add the build to that group (or enable automatic
distribution). New account users must first accept their account invitation.
Internal testing grants App Store Connect access; use external testing for
people who only need to try the app.
**External:** create an external testing group, complete Beta App Information
and review contact details, add a build and its **What to Test** notes, then
submit it for TestFlight App Review when required. Use
<https://framecontrol.fun/privacy/> for the privacy policy and
<https://framecontrol.fun/feedback/> for feedback. Review notes must explain
that the app requires a Steam Frame with Developer Mode enabled and a user
password; do not provide your own headset credentials to reviewers. Apple may
need a way to evaluate features that require the hardware.
After Apple approves the build for external testing, enable the group's public
link and choose a tester limit. Share that link to let testers join without an
App Store Connect account. Creating a group alone does not make a build public.
Do not describe the link as ready until the approved build is in the group and
the link is enabled.
## Verified
<img src="img/iphone-tabs.jpg" alt="The four tabs in the iPhone app, connected to a Frame" width="900">
@@ -106,8 +207,76 @@ Sending text to the Frame's clipboard, or copying the Frame's to the phone
(**Type on the headset → Copy the headset's clipboard**), needs the desktop
panel open in the headset, as on the desktop app.
Not yet exercised: Android display changes through podman (no Android app was
running), a real sleep/restart/shut down on the Frame, and a physical iPhone.
On a physical iPhone, 2026-10-14: Alex reported build 202610140516 was working
after pairing and trying the password visibility button. A server process
started by the phone and its SSH key on the Frame confirmed the connection.
This was a setup/connection smoke check; it did not verify every app feature.
On a physical iPhone 16 Pro running iOS 27.2, 2026-10-16: the same build launched
and automatically reconnected to the Frame. A device screenshot showed the
green connection indicator, 100% headset battery and the loaded Android
catalog. This confirms connection and rendering, not status refresh or actions
inside the app. A separate signed automation runner was installed and passed
its health check. Initial interaction attempts were blocked by the phone's
lock; testing resumed once Alex unlocked it and kept the screen awake.
<img src="img/iphone-physical-20261016.jpg" alt="Frame Control connected to the Frame on a physical iPhone, showing battery and the Android catalog" width="360">
In that continuation, still on build **202610140516**, the status header
refreshed, the Games tab loaded 14 installed titles with sizes and Play
controls, and a uniquely named 114-byte text file was selected through the
iPhone's Files picker and sent to `~/Downloads` on the Frame. An SSH readback
matched the file's contents exactly; this verifies phone-to-Frame upload,
not a download back to the phone. Both the iCloud source fixture and the
uploaded test file were removed and checked absent.
Capture returned a blank headset frame with an asleep/unworn explanation.
Save opened the native share sheet and reached the Photos permission prompt;
no completed Photos save was verified. A final Settings check showed
**Add Photos Only** enabled, without photo-library read access. A test string
sent through the clipboard control was refused with the expected prerequisite:
`plasmashell` was not running, so the desktop panel must be opened in the
headset. The phone's existing clipboard was not read. Live mode started and
reported **30 fps**, but its visible frames remained blank, so meaningful
video playback is still unverified. Android listed 10 installed apps; a
separate SSH check found zero running containers and zero Android ADB
listeners, so display-setting changes were not exercised.
<img src="img/iphone-games-physical-20261016.jpg" alt="Frame Control's Games tab on the physical iPhone, showing the installed library and Play controls" width="360">
<img src="img/iphone-live-physical-20261016.jpg" alt="Live mode reporting 30 fps on the physical iPhone with blank headset content and the page horizontally clipped" width="360">
Testing also exposed an enlarged, horizontally clipped page after text entry
and navigation. Its cause has not yet been isolated. Restarting Frame Control
restored the normal layout and automatically reconnected to the headset.
### Mac follow-up
On 2026-10-17, the installed Mac app **0.4.6** was tested with the headset
unworn. Connection/status worked, and its connection test reported
`Answered in 90 ms · SSH works`. Still capture completed but both eyes were
blank. Live video failed with `Not a video capture device` when opening
`/dev/video99`, then fell back to blank still captures. No visible scene or
video playback was verified. The desktop was inactive; no headset services
or settings were changed. Live was stopped and the existing app left on Home.
These are Mac results; they do not verify the iPhone build or establish the
cause of its earlier blank output.
<img src="img/mac-live-error-20261017.png" alt="Mac Frame Control activity log reporting Live video failed: Not a video capture device" width="900">
### Physical-phone follow-up checklist
- With the headset awake and worn, capture visible headset content, save it to
Photos, then verify live video.
- Send test text to the Frame's clipboard with its desktop panel open.
- With an Android app running, change and restore its display settings.
- Reproduce and fix the page enlargement and horizontal clipping after text
entry and navigation.
- After saving headset work, deliberately test sleep, restart and shut down.
These remaining checks are pending on a physical phone. Android display
changes through podman and real sleep/restart/shut down also remain unverified
by the Simulator checks above.
Debug builds have Simulator test hooks (`FRAME_TEST_HOST`, `FRAME_TEST_PAGE`,
`FRAME_TEST_JS`, and the tunnel URL in the app's Caches folder); release builds
+112
View File
@@ -0,0 +1,112 @@
# Mac space: the Mac's desktop and its windows, placed in the room
Design notes, started 2026-09-28. The goal is Apple Vision Pro's Mac Virtual
Display, plus something Vision Pro can't do: pulling any Mac window out
to float on its own. Labels as in [ssh.md](ssh.md): **verified**,
**documented**, **reported**, **inferred**.
## What to match (Vision Pro's Mac Virtual Display)
Researched 2026-09-28. Sources: Apple Support
[118521](https://support.apple.com/en-us/118521) and the
[Vision Pro guide](https://support.apple.com/guide/apple-vision-pro/use-mac-virtual-display-tan357ede966/visionos);
[9to5Mac](https://9to5mac.com/2024/01/30/heres-what-its-like-using-vision-pro-as-an-external-mac-display/),
[UploadVR](https://www.uploadvr.com/visionos-2-2-ultrawide-mac-virtual-display/)
and [AppleInsider](https://appleinsider.com/inside/apple-vision-pro/tips/how-to-use-mac-virtual-display-for-apple-vision-pro-with-desktop-macs-and-macbooks).
- **Connecting.** Look at the Mac and a **Connect** button floats above it.
There's also a Control Center route. The Mac's own screen goes dark
(**documented**/**reported**).
- **Placement.** The display appears where you were looking, near the Mac,
at a large-monitor size. After that it stays put in the room: moving the
laptop doesn't move it. Holding the Digital Crown recenters everything
(**documented**).
- **Handling.**
- A **window bar** under the display moves it, with pinch and drag, and
also moves it nearer or farther.
- Its bottom **corners** resize it.
- **Close** sits to the left of the bar (**documented**).
- **Modes.** Standard (16:9), Wide (21:9) and Ultrawide (32:9, about 10K
pixels across). All are curved since visionOS 2.2 (**documented**).
- **Input.**
- The Mac's keyboard and trackpad drive it.
- The pointer moves to whichever window you look at (Universal Control).
- Mac audio plays in the headset (**documented**).
- **What it can't do.** You can't pull a single Mac window out of the
virtual display (**reported**, several sources). That's our
differentiator.
## What the Frame allows
- **Moving SteamVR's panels from our own process: no** (**verified**
2026-09-28). `FindOverlay("valve.steam.desktopgame.<id>")` works from a
separate OpenVR app, but `SetOverlayTransformAbsolute` returns 12
(`VROverlayError_PermissionDenied`). The panel reported transform type 5
(dashboard tab). Only SteamVR's dashboard, which runs in `vrwebhelper`,
moves them, through its privileged `VRHTML.VROverlay`, and it has no
DevTools port. See `/tmp/fmv-space/ovprobe.py` for the probe, which stays
outside the repo.
- **WebXR in Chromium XR: yes** (**verified** 2026-09-26/27,
[webxr-chromium.md](webxr-chromium.md)).
- `immersive-vr` runs at 72–75 fps.
- Passthrough shows wherever the page leaves pixels transparent.
- Controllers are `tracked-pointer`s with `xr-standard` gamepads, and
squeeze events arrive.
- Hands are tracked with 25 joints.
- **Our own OpenVR overlays: possible, but most work.** We would own their
transforms and get laser input as mouse events for free. But the Frame
would need native H.264 decoding and texture upload.
## Decision: a WebXR "Mac space" page first
The WebXR page (`ui/mac-space.html`) holds everything, with passthrough
behind it:
- **The desktop.** A large curved panel shows the Mac's main display,
placed above the laptop. The first time, you place it yourself; after
that it's remembered in the room.
- **Windows.** Each separated window gets its own panel.
- **Grabbing.** A bar under each panel moves it (squeeze, or pinch with a
hand). Its corners resize it, and there's a close button to the bar's
left, as on Vision Pro.
- **Pulling a window out.** Grab a window's title bar on the desktop panel
and drag it off the edge. The page starts a `separate:<window id>`
stream, the window moves to its own display on the Mac, and its new
panel follows your hand until you let go. Dropping a panel back onto the
desktop panel stops the stream, and the window goes back.
- **Input.** The laser and trigger click and drag, the thumbstick scrolls,
and the Mac's keyboard types, as today.
Trade-offs:
- It's the headset's active app while you use it. Other VR apps pause, but
SteamVR's own overlays still draw on top.
- Text sharpness depends on how the page renders. WebXR quad layers (the
`layers` feature) would let SteamVR's compositor sample each panel
directly, so they're worth checking first.
## Building blocks
- **One module for the protocol.** `ui/mac-stream.js` holds the stream
protocol (connect, clock sync, decoding, acks and reports, input), shared
by `mac-view.html` (the 2D panel) and `mac-space.html`.
- **A space key from the agent.** Frame Control gets a ticket with
`src=space`, and the page trades it once for a **space key**, kept in
memory. That key lists the windows and displays (with their positions,
for picking the one under the pointer) and opens streams for any of
them. Stop All revokes it.
- **Anchoring.**
- v1: manual placement, remembered in the room (to be checked: whether
`local-floor` stays put between sessions on the Frame).
- v2: the Mac shows a marker on its screen for a moment, and the headset
finds it. That needs camera access, which is **unknown** on the Frame:
the passthrough cameras are monochrome, and `IVRTrackedCamera` hasn't
been tried.
## Open questions
- Frame rate with several software-decoded streams at once.
- Whether `local-floor` placements survive restarts.
- Whether WebXR quad layers work in Chromium XR's OpenXR path.
- Whether Chromium grants `requestSession` without a click in the page
(needed for a one-click "Open Mac space").
+20 -3
View File
@@ -1,11 +1,28 @@
# Tailscale: use the Frame from anywhere
# Tailscale: advanced remote-access setup
With Tailscale on the Frame, the `frame` SSH alias works off the home LAN, and
so does everything built on it: Frame Control, the scripts and the Mac app.
Tailscale is intended to make the `frame` SSH alias available off the home LAN,
including to Frame Control and the scripts. Off-LAN access has not yet been
verified; the checks below cover the direct LAN path only. Read the exposure
section before enabling it: services normally restricted to loopback become
reachable by other devices on your tailnet.
When the Mac and the Frame are on the same network, Tailscale connects them
directly, so there's no relay in the way (`tailscale ping frame` → `via
192.168.1.50:41641`, 20 ms).
These are source-checkout commands for macOS, not packaged-app menu actions
or Windows PowerShell commands. The connection script currently uses macOS
netcat options; this route is not supported on Linux. Install Git, OpenSSH,
zsh, Python 3 and curl on your Mac; the desktop app's bundled Python is not
available to these scripts. Then obtain the scripts:
```sh
git clone https://github.com/saphid/frame-control.git
cd frame-control
```
Enable Developer Mode on the awake headset and establish working SSH access
first using the [connection guide](frame-control.md). Run from the checkout:
```sh
scripts/tailscale-on-frame.sh # install or update, then approve the login URL
scripts/connect.sh frame.<tailnet>.ts.net # point the alias at Tailscale (the script prints this)
+24 -8
View File
@@ -7,13 +7,13 @@
objects = {
/* Begin PBXBuildFile section */
C1234567890ABCDEF0123456 /* NativeSessionTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = D1234567890ABCDEF0123456 /* NativeSessionTests.swift */; };
A1234567890ABCDEF0123456 /* NativeSessionState.swift in Sources */ = {isa = PBXBuildFile; fileRef = B1234567890ABCDEF0123456 /* NativeSessionState.swift */; };
0CF0B365CE5974753E8C3C2B /* ScreenshotJournal.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0D48FCBB02F0F2212C7DE3C1 /* ScreenshotJournal.swift */; };
0DEE50BD563B1D8C328C4C0A /* HeadsetServer.swift in Sources */ = {isa = PBXBuildFile; fileRef = EDC7BA8014DBC302D08FD397 /* HeadsetServer.swift */; };
12B21D3319BAF5AE79948560 /* FrameLink.swift in Sources */ = {isa = PBXBuildFile; fileRef = 16644E7FDA7ADD5B232EB700 /* FrameLink.swift */; };
1A07EC692B0FF723907EA77B /* WebShell.swift in Sources */ = {isa = PBXBuildFile; fileRef = D6C4E6C28315CA8729FCAAEA /* WebShell.swift */; };
41A697B9924018DA48F24A1F /* Keys.swift in Sources */ = {isa = PBXBuildFile; fileRef = 237D9AF04EEA257AB382F60E /* Keys.swift */; };
4622FE0F0D6499CD642C29A2 /* InstallLink.swift in Sources */ = {isa = PBXBuildFile; fileRef = BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */; };
46EF7595BE5B36664A11AEB5 /* NativeSessionTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8F1CAE943C2556A960890079 /* NativeSessionTests.swift */; };
476D8858DC2C9E6616B084BC /* PortForwarder.swift in Sources */ = {isa = PBXBuildFile; fileRef = A7F6ED116569D0ABABF6ED65 /* PortForwarder.swift */; };
765661DBC0E6798A27CC60DB /* RootView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9B24E1BCD4F69A24C7DEF02F /* RootView.swift */; };
78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */; };
@@ -21,8 +21,12 @@
9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */ = {isa = PBXBuildFile; fileRef = DB544223FC60A59CC3E8EF5F /* SetupView.swift */; };
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */; };
A8C7AED25A6280682FCE45DC /* Citadel in Frameworks */ = {isa = PBXBuildFile; productRef = 6BA549B6CC0A0CB847126456 /* Citadel */; };
C046F310BA857BCEC98BE00E /* ScreenshotDownloadsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5EBD10B73DF1E95F67A3B074 /* ScreenshotDownloadsView.swift */; };
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */; };
DFD89170E502260F005EEC73 /* ScreenshotDownloads.swift in Sources */ = {isa = PBXBuildFile; fileRef = 07617FD9B214626526FB1F03 /* ScreenshotDownloads.swift */; };
E2FD56BB5BC1D9AF5CDBF210 /* ScreenshotSyncTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = AB8610920C7D797E5963CF88 /* ScreenshotSyncTests.swift */; };
E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */; };
F0ECE06D5DD29F19CC41FDFB /* NativeSessionState.swift in Sources */ = {isa = PBXBuildFile; fileRef = 9E8BE9A8C421AD40A0F4105B /* NativeSessionState.swift */; };
F94D0252F8CC5854314B84B2 /* AppModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8A11F3431826A26B247C0695 /* AppModel.swift */; };
/* End PBXBuildFile section */
@@ -37,19 +41,23 @@
/* End PBXContainerItemProxy section */
/* Begin PBXFileReference section */
D1234567890ABCDEF0123456 /* NativeSessionTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionTests.swift; sourceTree = "<group>"; };
B1234567890ABCDEF0123456 /* NativeSessionState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionState.swift; sourceTree = "<group>"; };
07617FD9B214626526FB1F03 /* ScreenshotDownloads.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotDownloads.swift; sourceTree = "<group>"; };
0D48FCBB02F0F2212C7DE3C1 /* ScreenshotJournal.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotJournal.swift; sourceTree = "<group>"; };
16644E7FDA7ADD5B232EB700 /* FrameLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameLink.swift; sourceTree = "<group>"; };
1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameControlTests.swift; sourceTree = "<group>"; };
237D9AF04EEA257AB382F60E /* Keys.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Keys.swift; sourceTree = "<group>"; };
2F288DF6636A417F0CA3A6CD /* FrameFinder.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameFinder.swift; sourceTree = "<group>"; };
5EBD10B73DF1E95F67A3B074 /* ScreenshotDownloadsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotDownloadsView.swift; sourceTree = "<group>"; };
6B5B6718C5EA77FA67F6B14C /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist; path = Info.plist; sourceTree = "<group>"; };
6FBC8D0B5ED7BF1C06F99892 /* FrameControlTests.xctest */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.cfbundle; path = FrameControlTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
8A11F3431826A26B247C0695 /* AppModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppModel.swift; sourceTree = "<group>"; };
8F1CAE943C2556A960890079 /* NativeSessionTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionTests.swift; sourceTree = "<group>"; };
8F2CB550FC81C01E6BDD5A71 /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = "<group>"; };
93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FrameControlApp.swift; sourceTree = "<group>"; };
9B24E1BCD4F69A24C7DEF02F /* RootView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RootView.swift; sourceTree = "<group>"; };
9E8BE9A8C421AD40A0F4105B /* NativeSessionState.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NativeSessionState.swift; sourceTree = "<group>"; };
A7F6ED116569D0ABABF6ED65 /* PortForwarder.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PortForwarder.swift; sourceTree = "<group>"; };
AB8610920C7D797E5963CF88 /* ScreenshotSyncTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ScreenshotSyncTests.swift; sourceTree = "<group>"; };
BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InstallLink.swift; sourceTree = "<group>"; };
D6C4E6C28315CA8729FCAAEA /* WebShell.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WebShell.swift; sourceTree = "<group>"; };
DB544223FC60A59CC3E8EF5F /* SetupView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SetupView.swift; sourceTree = "<group>"; };
@@ -104,9 +112,11 @@
isa = PBXGroup;
children = (
8A11F3431826A26B247C0695 /* AppModel.swift */,
B1234567890ABCDEF0123456 /* NativeSessionState.swift */,
93C8E0D7C3F4F628941B3D5A /* FrameControlApp.swift */,
BF0FCA7117DA3ABA449B4EE0 /* InstallLink.swift */,
9E8BE9A8C421AD40A0F4105B /* NativeSessionState.swift */,
07617FD9B214626526FB1F03 /* ScreenshotDownloads.swift */,
0D48FCBB02F0F2212C7DE3C1 /* ScreenshotJournal.swift */,
);
path = App;
sourceTree = "<group>";
@@ -116,7 +126,8 @@
children = (
E81218B75FEEE47B8D8BAE20 /* ComfortNotificationTests.swift */,
1740B691F9C25E5FB6F9EFC3 /* FrameControlTests.swift */,
D1234567890ABCDEF0123456 /* NativeSessionTests.swift */,
8F1CAE943C2556A960890079 /* NativeSessionTests.swift */,
AB8610920C7D797E5963CF88 /* ScreenshotSyncTests.swift */,
);
path = FrameControlTests;
sourceTree = "<group>";
@@ -125,6 +136,7 @@
isa = PBXGroup;
children = (
9B24E1BCD4F69A24C7DEF02F /* RootView.swift */,
5EBD10B73DF1E95F67A3B074 /* ScreenshotDownloadsView.swift */,
DB544223FC60A59CC3E8EF5F /* SetupView.swift */,
);
path = Views;
@@ -266,15 +278,18 @@
buildActionMask = 2147483647;
files = (
F94D0252F8CC5854314B84B2 /* AppModel.swift in Sources */,
A1234567890ABCDEF0123456 /* NativeSessionState.swift in Sources */,
78427FC66780623F31E7501E /* FrameControlApp.swift in Sources */,
E6898C714A92D3979F73B6E1 /* FrameFinder.swift in Sources */,
12B21D3319BAF5AE79948560 /* FrameLink.swift in Sources */,
0DEE50BD563B1D8C328C4C0A /* HeadsetServer.swift in Sources */,
4622FE0F0D6499CD642C29A2 /* InstallLink.swift in Sources */,
41A697B9924018DA48F24A1F /* Keys.swift in Sources */,
F0ECE06D5DD29F19CC41FDFB /* NativeSessionState.swift in Sources */,
476D8858DC2C9E6616B084BC /* PortForwarder.swift in Sources */,
765661DBC0E6798A27CC60DB /* RootView.swift in Sources */,
DFD89170E502260F005EEC73 /* ScreenshotDownloads.swift in Sources */,
C046F310BA857BCEC98BE00E /* ScreenshotDownloadsView.swift in Sources */,
0CF0B365CE5974753E8C3C2B /* ScreenshotJournal.swift in Sources */,
9657F7BC23E3352E5AB30777 /* SetupView.swift in Sources */,
1A07EC692B0FF723907EA77B /* WebShell.swift in Sources */,
);
@@ -286,7 +301,8 @@
files = (
A0C5B00E257230A38DBD9E54 /* ComfortNotificationTests.swift in Sources */,
DC043FB74BE2D23F3A5826BF /* FrameControlTests.swift in Sources */,
C1234567890ABCDEF0123456 /* NativeSessionTests.swift in Sources */,
46EF7595BE5B36664A11AEB5 /* NativeSessionTests.swift in Sources */,
E2FD56BB5BC1D9AF5CDBF210 /* ScreenshotSyncTests.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
+14
View File
@@ -22,6 +22,8 @@ final class AppModel: ObservableObject {
/// Native recovery remains reachable even when WebKit cannot draw a page.
@Published private(set) var webRecoveryMessage: String?
@Published private(set) var webReloadID = 0
let screenshots = ScreenshotDownloads()
@Published var showScreenshotDownloads = false
func webContentFailed(at url: URL, message: String) {
guard phase == .ready(url) else { return }
@@ -45,6 +47,7 @@ final class AppModel: ObservableObject {
let saved = UserDefaults.standard.data(forKey: Self.settingsKey).flatMap { try? JSONDecoder().decode(FrameSettings.self, from: $0) }
settings = saved
phase = saved == nil ? .setup : .connecting("Connecting")
if let identity = UserDefaults.standard.string(forKey: Self.hostKeyKey) { screenshots.load(identity: identity) }
}
var deviceName: String { UIDevice.current.userInterfaceIdiom == .pad ? "iPad" : "iPhone" }
@@ -146,6 +149,7 @@ final class AppModel: ObservableObject {
/// Every connection attempt has a number; anything that finishes after a newer
/// attempt started (or the user went back to setup) closes what it made and stops.
private func invalidate() {
screenshots.stop()
attempt += 1
retrying = false
webRecoveryMessage = nil
@@ -200,6 +204,7 @@ final class AppModel: ObservableObject {
}
#endif
phase = .ready(URL(string: page)!)
startScreenshots()
// Runs at once if it stopped in the moment since the check above.
server.whenExited { [weak self] tail in
Task { @MainActor in self?.lost(mine, "Frame Control on the headset stopped. \(tail.suffix(200))") }
@@ -254,6 +259,7 @@ final class AppModel: ObservableObject {
/// Called when the app comes back to the foreground: iOS may have dropped the
/// connection, or left it looking open, while it was in the background.
func resume() {
startScreenshots()
switch phase {
case .ready:
let mine = attempt
@@ -284,6 +290,7 @@ final class AppModel: ObservableObject {
}
private func teardown() async {
screenshots.stop()
forwarder?.stop()
forwarder = nil
server = nil
@@ -298,4 +305,11 @@ final class AppModel: ObservableObject {
_ = SecRandomCopyBytes(kSecRandomDefault, bytes.count, &bytes)
return bytes.map { String(format: "%02x", $0) }.joined()
}
func suspend() { screenshots.stop() }
private func startScreenshots() {
guard case .ready(let url) = phase, let hostKey else { return }
screenshots.start(url: url, identity: hostKey)
}
}
@@ -38,6 +38,7 @@ struct FrameControlApp: App {
}
.onChange(of: scenePhase) { _, phase in
if phase == .active { model.resume() }
else if phase == .background { model.suspend() }
}
}
}
@@ -0,0 +1,157 @@
import CryptoKit
import ImageIO
import Photos
import SwiftUI
import UIKit
@MainActor
final class ScreenshotDownloads: ObservableObject {
struct Item: Identifiable {
let id: String
let file: URL
let date: Date
let status: String
}
@Published private(set) var items: [Item] = []
@Published private(set) var message = "New screenshots download while Frame Control is open."
@Published private(set) var saveToPhotos: Bool
private let defaults: UserDefaults
private let storageRoot: URL?
private static let preference = "frame.screenshots.saveToPhotos"
private var engine: ScreenshotSyncEngine?
private var task: Task<Void, Never>?
private var session: URL?
private var identity: String?
init(defaults: UserDefaults = .standard, storageRoot: URL? = nil) {
self.defaults = defaults
self.storageRoot = storageRoot
saveToPhotos = defaults.object(forKey: Self.preference) == nil || defaults.bool(forKey: Self.preference)
}
func setPhotos(_ enabled: Bool) {
do {
try engine?.setPhotos(enabled)
defaults.set(enabled, forKey: Self.preference)
saveToPhotos = enabled
refresh()
if enabled { Task { _ = await Self.photoPermission() } }
} catch { message = "Couldn't save the screenshot setting: \(error.localizedDescription)" }
}
/// Local files and settings remain available without a live SSH session.
func load(identity: String) {
guard self.identity != identity || engine == nil else { return }
do {
let digest = SHA256.hash(data: Data(identity.utf8)).map { String(format: "%02x", $0) }.joined()
let base = try storageRoot ?? FileManager.default.url(for: .applicationSupportDirectory, in: .userDomainMask,
appropriateFor: nil, create: true)
self.engine = try ScreenshotSyncEngine(root: base.appendingPathComponent("Screenshots/\(digest)", isDirectory: true),
photos: saveToPhotos)
self.identity = identity
refresh()
} catch {
engine = nil
self.identity = nil
items = []
message = "Screenshot downloads need attention: \(error.localizedDescription)"
}
}
func start(url: URL, identity: String) {
if session == url, task != nil { return }
stop()
load(identity: identity)
if let engine {
session = url
task = Task { [weak self] in
while !Task.isCancelled {
guard let self, self.session == url else { return }
if UIApplication.shared.applicationState == .active {
await self.poll(url: url, engine: engine)
}
do { try await Task.sleep(nanoseconds: 4_000_000_000) }
catch { return }
}
}
}
}
func stop() {
task?.cancel()
task = nil
engine?.stop()
session = nil
}
private func poll(url: URL, engine: ScreenshotSyncEngine) async {
do {
let data = try await Self.get("/api/shots", from: url)
struct Listing: Decodable { let shots: [HeadsetScreenshot] }
let shots = try JSONDecoder().decode(Listing.self, from: data).shots
guard !Task.isCancelled, session == url else { return }
try await engine.reconcile(shots, download: { shot in
try await Self.get("/api/shots/image", id: shot.id, from: url)
}, validate: { data in
guard let image = CGImageSourceCreateWithData(data as CFData, nil),
CGImageSourceGetStatus(image) == .statusComplete else { return false }
return CGImageSourceCreateImageAtIndex(image, 0, nil) != nil
}, export: { file, allowed in
guard !Task.isCancelled, UIApplication.shared.applicationState == .active else { throw CancellationError() }
guard await Self.photoPermission() else {
throw NSError(domain: "FrameScreenshots", code: 1,
userInfo: [NSLocalizedDescriptionKey: "Originals are kept in Frame Control. Allow Photos access in iOS Settings to save them to Photos."])
}
// Cancellation or a toggle change while the permission prompt was
// visible must not start a Photos transaction.
guard !Task.isCancelled, allowed() else { throw CancellationError() }
try await PHPhotoLibrary.shared().performChanges {
PHAssetCreationRequest.forAsset().addResource(with: .photo, fileURL: file, options: nil)
}
})
guard !Task.isCancelled, session == url else { return }
message = "New screenshots download while Frame Control is open."
} catch {
guard !Task.isCancelled, session == url else { return }
message = "\(error.localizedDescription) Downloads retry while the app is open."
}
refresh()
}
private static func get(_ path: String, id: String? = nil, from page: URL) async throws -> Data {
let original = URLComponents(url: page, resolvingAgainstBaseURL: false)
let key = original?.queryItems?.first(where: { $0.name == "key" })?.value
var parts = original!
parts.path = path
parts.queryItems = id.map { [URLQueryItem(name: "id", value: $0)] }
parts.fragment = nil
var request = URLRequest(url: parts.url!, cachePolicy: .reloadIgnoringLocalCacheData, timeoutInterval: 30)
request.setValue(key, forHTTPHeaderField: "X-Frame-UI")
let (data, response) = try await URLSession.shared.data(for: request)
guard let http = response as? HTTPURLResponse, http.statusCode == 200 else { throw URLError(.badServerResponse) }
return data
}
private static func photoPermission() async -> Bool {
var status = PHPhotoLibrary.authorizationStatus(for: .addOnly)
if status == .notDetermined, UIApplication.shared.applicationState == .active {
status = await PHPhotoLibrary.requestAuthorization(for: .addOnly)
}
return status == .authorized || status == .limited
}
private func refresh() {
guard let engine else { return }
items = engine.journal.receipts.values.filter(\.downloaded).map { receipt in
let status: String
switch receipt.destination {
case .saved: status = "Saved to Photos"
case .appOnly: status = "In Frame Control"
case .pending: status = "In Frame Control · waiting for Photos"
case .committing: status = "In Frame Control · Photos save interrupted; check Photos before saving again"
}
return Item(id: receipt.shot.id, file: engine.file(for: receipt.shot.id),
date: Date(timeIntervalSince1970: receipt.shot.time), status: status)
}.sorted { $0.date > $1.date }
}
}
@@ -0,0 +1,151 @@
import Foundation
struct HeadsetScreenshot: Codable, Equatable {
let id: String
let size: Int
let time: Double
var valid: Bool {
id.range(of: #"^[0-9]{1,12}/[0-9]{1,20}/[0-9]{14}_[0-9]{1,4}\.(jpg|png)$"#, options: .regularExpression) != nil
&& size > 0 && size <= 30 * 1024 * 1024 && time.isFinite
}
}
/// All receipt state lives beside the originals, outside purgeable caches.
/// An interrupted Photos transaction is deliberately not retried automatically:
/// add-only access cannot check whether its asset was committed before a crash.
struct ScreenshotJournal: Codable {
enum Export: String, Codable { case pending, appOnly, committing, saved }
struct Receipt: Codable {
var shot: HeadsetScreenshot
var downloaded = false
var destination: Export
}
var baseline: Set<String>?
var receipts: [String: Receipt] = [:]
mutating func observe(_ shots: [HeadsetScreenshot], photos: Bool) {
let shots = shots.filter(\.valid)
guard let baseline else {
self.baseline = Set(shots.map(\.id))
return
}
for shot in shots where !baseline.contains(shot.id) {
if var receipt = receipts[shot.id] {
if !receipt.downloaded { receipt.shot = shot; receipts[shot.id] = receipt }
} else {
receipts[shot.id] = Receipt(shot: shot, destination: photos ? .pending : .appOnly)
}
}
}
mutating func keepPendingInApp() {
for id in receipts.keys where receipts[id]?.destination == .pending || receipts[id]?.destination == .committing {
receipts[id]?.destination = .appOnly
}
}
}
/// Serialized reconciler shared by foreground polling and future event delivery.
/// Network and Photos operations are injected so failures and late callbacks can
/// be exercised against the actual persistence and state transitions.
@MainActor
final class ScreenshotSyncEngine {
let root: URL
private(set) var journal: ScreenshotJournal
private var previous: [String: HeadsetScreenshot] = [:]
private var generation = UUID()
private var running = false
private(set) var saveToPhotos: Bool
init(root: URL, photos: Bool) throws {
self.root = root
saveToPhotos = photos
let ledger = root.appendingPathComponent("receipts.json")
if FileManager.default.fileExists(atPath: ledger.path) {
// Corrupt state must never silently become a fresh receipt ledger.
journal = try JSONDecoder().decode(ScreenshotJournal.self, from: Data(contentsOf: ledger))
} else { journal = ScreenshotJournal() }
if !photos { journal.keepPendingInApp() }
try persist()
}
func stop() { generation = UUID() }
func setPhotos(_ enabled: Bool) throws {
let oldJournal = journal
let oldSetting = saveToPhotos
saveToPhotos = enabled
if !enabled { journal.keepPendingInApp() }
do { try persist() }
catch { journal = oldJournal; saveToPhotos = oldSetting; throw error }
}
func file(for id: String) -> URL { root.appendingPathComponent(id) }
private func persist() throws {
try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true)
try JSONEncoder().encode(journal).write(to: root.appendingPathComponent("receipts.json"), options: .atomic)
}
func reconcile(_ shots: [HeadsetScreenshot],
download: (HeadsetScreenshot) async throws -> Data,
validate: (Data) -> Bool,
export: (URL, () -> Bool) async throws -> Void) async throws {
guard !running else { return }
running = true
defer { running = false }
let mine = generation
func current() -> Bool { mine == generation && !Task.isCancelled }
guard current() else { return }
let valid = shots.filter(\.valid)
let observed = Dictionary(valid.map { ($0.id, $0) }, uniquingKeysWith: { _, last in last })
let stable = previous
previous = observed
journal.observe(valid, photos: saveToPhotos)
try persist() // Queue before touching an original or the photo library.
var firstError: Error?
for id in journal.receipts.keys.sorted() {
guard current(), var receipt = journal.receipts[id], receipt.shot.valid else { continue }
do {
let file = file(for: id)
if !receipt.downloaded {
// Steam may still be writing. Require two equal observations.
guard observed[id] == receipt.shot, stable[id] == receipt.shot else { continue }
let data = try await download(receipt.shot)
guard current() else { return }
guard data.count == receipt.shot.size, validate(data) else {
throw CocoaError(.fileReadCorruptFile)
}
try FileManager.default.createDirectory(at: file.deletingLastPathComponent(), withIntermediateDirectories: true)
try data.write(to: file, options: .atomic)
// Re-read destination: the user can change it during download.
receipt = journal.receipts[id]!
receipt.downloaded = true
journal.receipts[id] = receipt
try persist()
}
if receipt.destination == .pending, saveToPhotos {
journal.receipts[id]?.destination = .committing
try persist()
do {
try await export(file, { current() && self.saveToPhotos && self.journal.receipts[id]?.destination == .committing })
} catch {
guard current() else { return }
// A reported transaction failure is safe to retry. A crash
// leaves committing, which needs explicit user recovery.
if journal.receipts[id]?.destination == .committing {
journal.receipts[id]?.destination = saveToPhotos ? .pending : .appOnly
}
try persist()
throw error
}
guard current() else { return }
journal.receipts[id]?.destination = .saved
try persist()
}
} catch { if firstError == nil { firstError = error } }
}
if let firstError { throw firstError }
}
}
+3 -1
View File
@@ -31,6 +31,8 @@
</array>
<key>CFBundleVersion</key>
<string>1</string>
<key>ITSAppUsesNonExemptEncryption</key>
<false/>
<key>LSApplicationQueriesSchemes</key>
<array>
<string>ssh</string>
@@ -50,7 +52,7 @@
<key>NSLocalNetworkUsageDescription</key>
<string>Frame Control finds your Steam Frame on your network and connects to it.</string>
<key>NSPhotoLibraryAddUsageDescription</key>
<string>Frame Control saves headset captures and screenshots to your photo library when you ask it to.</string>
<string>Frame Control automatically saves new headset screenshots to Photos when Save screenshots to Photos is enabled.</string>
<key>UILaunchScreen</key>
<dict>
<key>UIColorName</key>
+9
View File
@@ -39,6 +39,15 @@ struct RootView: View {
}
.preferredColorScheme(.dark)
.tint(.frameBlue)
.safeAreaInset(edge: .bottom) {
if case .ready = model.phase {} else {
Button("Screenshot downloads") { model.showScreenshotDownloads = true }
.padding().frame(maxWidth: .infinity).background(Color.frameBackground)
}
}
.sheet(isPresented: $model.showScreenshotDownloads) {
ScreenshotDownloadsView(downloads: model.screenshots)
}
}
}
@@ -0,0 +1,40 @@
import SwiftUI
struct ScreenshotDownloadsView: View {
@ObservedObject var downloads: ScreenshotDownloads
@Environment(\.dismiss) private var dismiss
var body: some View {
NavigationStack {
List {
Section {
Toggle("Save screenshots to Photos", isOn: Binding(get: { downloads.saveToPhotos }, set: downloads.setPhotos))
} footer: {
Text("Enabled by default. Turn off to keep new downloads only in Frame Control. Existing Photos are kept. App-only downloads stay app-only when this is turned on again.")
}
Section {
Text(downloads.message)
Text("Automatic downloads begin with new screenshots after the first connection. Keep the app open and connected; background delivery is not available yet.")
.foregroundStyle(.secondary)
}
Section("Downloaded from this headset") {
if downloads.items.isEmpty { Text("No new screenshots downloaded yet.").foregroundStyle(.secondary) }
ForEach(downloads.items) { item in
HStack {
VStack(alignment: .leading, spacing: 4) {
Text(item.date, format: .dateTime.month().day().hour().minute())
Text(item.status).font(.caption).foregroundStyle(.secondary)
}
Spacer()
ShareLink(item: item.file) { Image(systemName: "square.and.arrow.up") }
.accessibilityLabel("Share screenshot")
}
}
}
}
.navigationTitle("Screenshot downloads")
.navigationBarTitleDisplayMode(.inline)
.toolbar { ToolbarItem(placement: .confirmationAction) { Button("Done") { dismiss() } } }
}
}
}
+26 -3
View File
@@ -7,6 +7,7 @@ struct SetupView: View {
@ObservedObject var model: AppModel
@StateObject private var finder = FrameFinder()
@State private var password = ""
@State private var showPassword = false
@State private var manualHost = ""
@State private var user = "steamos"
@State private var showOther = false
@@ -92,13 +93,35 @@ struct SetupView: View {
// MARK: step 2
@ViewBuilder private var passwordStep: some View {
SecureField("Developer Mode password", text: $password)
HStack(spacing: 8) {
Group {
if showPassword {
TextField("Developer Mode password", text: $password)
} else {
SecureField("Developer Mode password", text: $password)
}
}
.textContentType(.password)
.textInputAutocapitalization(.never)
.autocorrectionDisabled()
.submitLabel(.go)
.focused($passwordFocused)
.onSubmit(connect)
.padding(12)
.background(Color.black.opacity(0.28), in: RoundedRectangle(cornerRadius: 10))
.padding(.vertical, 12)
Button {
showPassword.toggle()
passwordFocused = true // keep typing after the field swaps
} label: {
Image(systemName: showPassword ? "eye.slash" : "eye").foregroundStyle(Color.frameMuted)
.frame(width: 44, height: 44)
.contentShape(Rectangle())
}
.buttonStyle(.plain)
.accessibilityLabel(showPassword ? "Hide password" : "Show password")
}
.padding(.leading, 12)
.padding(.trailing, 2)
.background(Color.black.opacity(0.28), in: RoundedRectangle(cornerRadius: 10))
Text("Haven't set one? On the Frame: Steam Settings → Developer → Set User Password. It's only used now, to let this \(model.deviceName) in; it isn't saved.")
.font(.footnote).foregroundStyle(Color.frameMuted)
Button(action: connect) {
+4
View File
@@ -59,6 +59,7 @@ struct WebShell: UIViewRepresentable {
setUpConnection: () => call("setUpConnection"),
open: (what) => call("open", what),
saveImages: (images) => call("saveImages", images),
screenshotDownloads: () => call("screenshotDownloads"),
onInstallLink: (cb) => { installCb = cb; return call("installLinkReady", window.__frameDocumentID); },
};
window.__frameInstallLink = (req) => { if (!installCb) return false; installCb(req); return true; };
@@ -162,6 +163,9 @@ struct WebShell: UIViewRepresentable {
guard !images.isEmpty else { return replyHandler(nil, "No images to save") }
share(images)
replyHandler(["message": "Choose Save Image to keep \(images.count == 1 ? "it" : "them") in Photos"], nil)
case "screenshotDownloads":
model.showScreenshotDownloads = true
replyHandler(true, nil)
case "installLinkReady":
guard let id = arg as? String, let web,
let data = try? JSONSerialization.data(withJSONObject: id, options: .fragmentsAllowed),
@@ -0,0 +1,198 @@
import Foundation
import CryptoKit
import XCTest
@testable import Frame_Control
@MainActor
final class ScreenshotSyncTests: XCTestCase {
private let old = HeadsetScreenshot(id: "1/10/20261024010000_1.jpg", size: 3, time: 1)
private let new = HeadsetScreenshot(id: "1/10/20261024010001_1.jpg", size: 3, time: 2)
private func temporary() throws -> URL {
let root = FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)
try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true)
addTeardownBlock { try? FileManager.default.removeItem(at: root) }
return root
}
func testBaselineStabilityAtomicOriginalAndRestartDedup() async throws {
let root = try temporary()
let engine = try ScreenshotSyncEngine(root: root, photos: true)
var downloads = 0, exports = 0
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in downloads += 1; return Data([1, 2, 3]) }
let save: (URL, () -> Bool) async throws -> Void = { file, allowed in
XCTAssertTrue(allowed())
XCTAssertEqual(try Data(contentsOf: file), Data([1, 2, 3]))
exports += 1
}
try await engine.reconcile([old], download: fetch, validate: { _ in true }, export: save)
try await engine.reconcile([old, new], download: fetch, validate: { _ in true }, export: save)
XCTAssertEqual(downloads, 0, "Do not import history or read a file still being written")
try await engine.reconcile([old, new, new], download: fetch, validate: { _ in true }, export: save)
XCTAssertEqual(downloads, 1)
XCTAssertEqual(exports, 1)
let restarted = try ScreenshotSyncEngine(root: root, photos: true)
try await restarted.reconcile([old, new], download: fetch, validate: { _ in true }, export: save)
XCTAssertEqual(downloads, 1)
XCTAssertEqual(exports, 1)
XCTAssertEqual(restarted.journal.receipts[new.id]?.destination, .saved)
}
func testInterruptedDownloadRetriesAndAppOnlyNeverExportsLater() async throws {
let root = try temporary()
let engine = try ScreenshotSyncEngine(root: root, photos: false)
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in Data([1, 2, 3]) }
let noExport: (URL, () -> Bool) async throws -> Void = { _, _ in XCTFail("App-only item exported") }
try await engine.reconcile([], download: fetch, validate: { _ in true }, export: noExport)
try await engine.reconcile([new], download: fetch, validate: { _ in true }, export: noExport)
do {
try await engine.reconcile([new], download: { _ in Data([1]) }, validate: { _ in true }, export: noExport)
XCTFail("Truncated original accepted")
} catch {}
XCTAssertFalse(FileManager.default.fileExists(atPath: engine.file(for: new.id).path))
let restarted = try ScreenshotSyncEngine(root: root, photos: true)
try await restarted.reconcile([new], download: fetch, validate: { _ in true }, export: noExport)
try await restarted.reconcile([new], download: fetch, validate: { _ in true }, export: noExport)
XCTAssertEqual(restarted.journal.receipts[new.id]?.destination, .appOnly)
XCTAssertTrue(restarted.journal.receipts[new.id]!.downloaded)
}
func testDestinationChangeDuringDownloadAndPermissionPrompt() async throws {
let engine = try ScreenshotSyncEngine(root: temporary(), photos: true)
let noExport: (URL, () -> Bool) async throws -> Void = { _, _ in XCTFail("Destination changed while downloading") }
try await engine.reconcile([], download: { _ in Data() }, validate: { _ in true }, export: noExport)
try await engine.reconcile([new], download: { _ in Data() }, validate: { _ in true }, export: noExport)
try await engine.reconcile([new], download: { _ in
try engine.setPhotos(false)
try engine.setPhotos(true)
return Data([1, 2, 3])
}, validate: { _ in true }, export: noExport)
XCTAssertEqual(engine.journal.receipts[new.id]?.destination, .appOnly)
let other = HeadsetScreenshot(id: "2/10/20261024010001_1.jpg", size: 3, time: 2)
try await engine.reconcile([new, other], download: { _ in Data() }, validate: { _ in true }, export: noExport)
do {
try await engine.reconcile([new, other], download: { _ in Data([1, 2, 3]) }, validate: { _ in true }, export: { _, allowed in
try engine.setPhotos(false)
try engine.setPhotos(true)
XCTAssertFalse(allowed(), "Re-enabling must not revive an export cancelled during consent")
throw CancellationError()
})
} catch {}
XCTAssertEqual(engine.journal.receipts[other.id]?.destination, .appOnly)
}
func testCancellationRejectsLateOriginalAndIdentityIncludesAccountAndApp() async throws {
let engine = try ScreenshotSyncEngine(root: temporary(), photos: false)
let variants = [new, HeadsetScreenshot(id: "2/10/20261024010001_1.jpg", size: 3, time: 2),
HeadsetScreenshot(id: "1/20/20261024010001_1.jpg", size: 3, time: 2)]
let save: (URL, () -> Bool) async throws -> Void = { _, _ in XCTFail("Unexpected export") }
try await engine.reconcile([], download: { _ in Data() }, validate: { _ in true }, export: save)
try await engine.reconcile(variants, download: { _ in Data() }, validate: { _ in true }, export: save)
try await engine.reconcile(variants, download: { _ in engine.stop(); return Data([1, 2, 3]) }, validate: { _ in true }, export: save)
XCTAssertEqual(engine.journal.receipts.count, 3)
XCTAssertTrue(engine.journal.receipts.values.allSatisfy { !$0.downloaded })
XCTAssertNotEqual(engine.file(for: variants[0].id), engine.file(for: variants[1].id))
XCTAssertNotEqual(engine.file(for: variants[0].id), engine.file(for: variants[2].id))
}
func testUncertainPhotosCommitIsNotDuplicatedAndCorruptLedgerFailsClosed() async throws {
let root = try temporary()
var journal = ScreenshotJournal()
journal.observe([], photos: true)
journal.observe([new], photos: true)
journal.receipts[new.id]?.downloaded = true
journal.receipts[new.id]?.destination = .committing
try JSONEncoder().encode(journal).write(to: root.appendingPathComponent("receipts.json"))
let engine = try ScreenshotSyncEngine(root: root, photos: true)
try await engine.reconcile([new], download: { _ in XCTFail(); return Data() }, validate: { _ in true }, export: { _, _ in XCTFail("Possible duplicate Photos asset") })
XCTAssertEqual(engine.journal.receipts[new.id]?.destination, .committing)
try Data("broken".utf8).write(to: root.appendingPathComponent("receipts.json"))
XCTAssertThrowsError(try ScreenshotSyncEngine(root: root, photos: true))
XCTAssertFalse(HeadsetScreenshot(id: "../../private.jpg", size: 3, time: 2).valid)
}
func testPhotosFailureRetainsOriginalAndRetriesOnlyExport() async throws {
let root = try temporary()
let engine = try ScreenshotSyncEngine(root: root, photos: true)
var downloads = 0, exports = 0
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in downloads += 1; return Data([1, 2, 3]) }
let denied: (URL, () -> Bool) async throws -> Void = { _, _ in throw CocoaError(.fileWriteNoPermission) }
try await engine.reconcile([], download: fetch, validate: { _ in true }, export: denied)
try await engine.reconcile([new], download: fetch, validate: { _ in true }, export: denied)
do {
try await engine.reconcile([new], download: fetch, validate: { _ in true }, export: denied)
XCTFail("Denied export reported success")
} catch {}
XCTAssertEqual(downloads, 1)
XCTAssertEqual(engine.journal.receipts[new.id]?.destination, .pending)
XCTAssertEqual(try Data(contentsOf: engine.file(for: new.id)), Data([1, 2, 3]))
let restarted = try ScreenshotSyncEngine(root: root, photos: true)
try await restarted.reconcile([], download: fetch, validate: { _ in true }, export: { _, allowed in
XCTAssertTrue(allowed()); exports += 1
})
XCTAssertEqual(downloads, 1)
XCTAssertEqual(exports, 1)
XCTAssertEqual(restarted.journal.receipts[new.id]?.destination, .saved)
}
func testLateExportSuccessAndFailureCannotOverwriteReplacementJournal() async throws {
for succeeds in [true, false] {
let root = try temporary()
let oldEngine = try ScreenshotSyncEngine(root: root, photos: true)
let fetch: (HeadsetScreenshot) async throws -> Data = { _ in Data([1, 2, 3]) }
let noop: (URL, () -> Bool) async throws -> Void = { _, _ in }
try await oldEngine.reconcile([], download: fetch, validate: { _ in true }, export: noop)
try await oldEngine.reconcile([new], download: fetch, validate: { _ in true }, export: noop)
var suspended: CheckedContinuation<Void, Never>?
let task = Task {
try await oldEngine.reconcile([new], download: fetch, validate: { _ in true }, export: { _, allowed in
await withCheckedContinuation { suspended = $0 }
XCTAssertFalse(allowed(), "Stopped export must lose eligibility")
if !succeeds { throw CocoaError(.fileWriteNoPermission) }
})
}
while suspended == nil { await Task.yield() }
oldEngine.stop()
let replacement = try ScreenshotSyncEngine(root: root, photos: false)
let other = HeadsetScreenshot(id: "2/10/20261024010001_1.jpg", size: 3, time: 3)
try await replacement.reconcile([new, other], download: fetch, validate: { _ in true }, export: noop)
try await replacement.reconcile([new, other], download: fetch, validate: { _ in true }, export: noop)
suspended!.resume()
try await task.value
let reloaded = try ScreenshotSyncEngine(root: root, photos: true)
XCTAssertEqual(reloaded.journal.receipts.count, 2)
XCTAssertEqual(reloaded.journal.receipts[other.id]?.destination, .appOnly)
XCTAssertTrue(reloaded.journal.receipts[other.id]!.downloaded)
XCTAssertEqual(reloaded.journal.receipts[new.id]?.destination, .appOnly)
}
}
func testDefaultPreferenceAndOfflineAppOnlyLibraryAfterRelaunch() throws {
let root = try temporary()
let suite = "frame-screenshot-test-\(UUID().uuidString)"
let defaults = try XCTUnwrap(UserDefaults(suiteName: suite))
defer { defaults.removePersistentDomain(forName: suite) }
let downloads = ScreenshotDownloads(defaults: defaults, storageRoot: root)
XCTAssertTrue(downloads.saveToPhotos)
downloads.setPhotos(false)
let identity = "test-headset-pinned-key"
let digest = SHA256.hash(data: Data(identity.utf8)).map { String(format: "%02x", $0) }.joined()
let folder = root.appendingPathComponent("Screenshots/\(digest)")
try FileManager.default.createDirectory(at: folder, withIntermediateDirectories: true)
var ledger = ScreenshotJournal()
ledger.observe([], photos: false)
ledger.observe([new], photos: false)
ledger.receipts[new.id]?.downloaded = true
try JSONEncoder().encode(ledger).write(to: folder.appendingPathComponent("receipts.json"))
let file = folder.appendingPathComponent(new.id)
try FileManager.default.createDirectory(at: file.deletingLastPathComponent(), withIntermediateDirectories: true)
try Data([1, 2, 3]).write(to: file)
let restarted = ScreenshotDownloads(defaults: defaults, storageRoot: root)
XCTAssertFalse(restarted.saveToPhotos)
restarted.load(identity: identity) // No URL, connection or network task.
XCTAssertEqual(restarted.items.count, 1)
XCTAssertEqual(restarted.items[0].status, "In Frame Control")
XCTAssertEqual(try Data(contentsOf: restarted.items[0].file), Data([1, 2, 3]))
}
}
+2 -1
View File
@@ -34,6 +34,7 @@ targets:
path: FrameControl/Info.plist
properties:
CFBundleDisplayName: Frame Control
ITSAppUsesNonExemptEncryption: false
UILaunchScreen:
UIColorName: ""
UISupportedInterfaceOrientations: [UIInterfaceOrientationPortrait, UIInterfaceOrientationLandscapeLeft, UIInterfaceOrientationLandscapeRight]
@@ -41,7 +42,7 @@ targets:
UIUserInterfaceStyle: Dark
NSLocalNetworkUsageDescription: Frame Control finds your Steam Frame on your network and connects to it.
NSBonjourServices: [_steamos-devkit._tcp]
NSPhotoLibraryAddUsageDescription: Frame Control saves headset captures and screenshots to your photo library when you ask it to.
NSPhotoLibraryAddUsageDescription: Frame Control automatically saves new headset screenshots to Photos when Save screenshots to Photos is enabled.
NSAppTransportSecurity:
NSAllowsLocalNetworking: true
LSApplicationQueriesSchemes: [ssh, sftp, steamlink, rdp]
+60
View File
@@ -0,0 +1,60 @@
#!/bin/zsh
# Archive Frame Control and upload it to TestFlight.
#
# Needs a paid Apple Developer Program team and an App Store Connect API key
# (Users and Access → Integrations → App Store Connect API). This Mac needs
# Admin access for Apple's cloud-managed distribution signing; App Manager
# alone was rejected with "Cloud signing permission error". See docs/iphone.md.
#
# ASC_KEY_ID=ABC123 ASC_ISSUER_ID=uuid ASC_KEY_PATH=~/.appstoreconnect/AuthKey_ABC123.p8 \
# ios/scripts/testflight.sh
#
# TEAM_ID defaults to the team in this Mac's signing certificate. The build
# number is the current UTC time, so every upload is higher than the last.
set -euo pipefail
: "${ASC_KEY_ID:?set ASC_KEY_ID}" "${ASC_ISSUER_ID:?set ASC_ISSUER_ID}" "${ASC_KEY_PATH:?set ASC_KEY_PATH}"
TEAM_ID="${TEAM_ID:-NJZUMEA4BN}"
BUILD_NUMBER="${BUILD_NUMBER:-$(date -u +%Y%m%d%H%M)}"
# Resolve the key before changing directory; expand a quoted leading ~/.
key_path="${ASC_KEY_PATH/#\~\//$HOME/}"
key_path="${key_path:A}"
[[ -f "$key_path" ]] || { echo "no API key at $key_path" >&2; exit 1; }
cd "${0:A:h}/.."
out="build/testflight/$BUILD_NUMBER"
mkdir -p "$out"
auth=(-allowProvisioningUpdates
-authenticationKeyPath "$key_path"
-authenticationKeyID "$ASC_KEY_ID"
-authenticationKeyIssuerID "$ASC_ISSUER_ID")
xcodegen generate
xcodebuild archive -project FrameControl.xcodeproj -scheme FrameControl \
-configuration Release -destination generic/platform=iOS \
-archivePath "$out/FrameControl.xcarchive" \
DEVELOPMENT_TEAM="$TEAM_ID" CODE_SIGN_STYLE=Automatic \
CURRENT_PROJECT_VERSION="$BUILD_NUMBER" "${auth[@]}"
# Check the archived build number, not just the argument passed.
archived=$(plutil -extract CFBundleVersion raw \
"$out/FrameControl.xcarchive/Products/Applications/Frame Control.app/Info.plist")
[[ "$archived" == "$BUILD_NUMBER" ]] || { echo "archived build $archived, expected $BUILD_NUMBER" >&2; exit 1; }
cat > "$out/ExportOptions.plist" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
<key>method</key><string>app-store-connect</string>
<key>destination</key><string>upload</string>
<key>teamID</key><string>$TEAM_ID</string>
<key>signingStyle</key><string>automatic</string>
<key>manageAppVersionAndBuildNumber</key><false/>
</dict></plist>
EOF
xcodebuild -exportArchive -archivePath "$out/FrameControl.xcarchive" \
-exportOptionsPlist "$out/ExportOptions.plist" -exportPath "$out/export" "${auth[@]}"
echo "Uploaded build $BUILD_NUMBER. It shows in TestFlight once App Store Connect finishes processing."
+94 -8
View File
@@ -18,11 +18,15 @@ final class VirtualDisplay {
static let refreshRate = Double(ProcessInfo.processInfo.environment["FRAME_MAC_VIEW_VD_HZ"] ?? "") ?? 60
private var display: CGVirtualDisplay?
let id: CGDirectDisplayID
/// The size asked for, in points (the display starts in another mode and
/// switches a moment later).
let size: (width: Int, height: Int)
/// Removal is deferred while the Mac's screen sleeps, and an identity
/// can't be reused until it's gone, so each display gets a fresh serial.
private static var serial = (UInt32(truncatingIfNeeded: ProcessInfo.processInfo.processIdentifier) & 0xFFFF) << 12
init?(name: String, width: Int, height: Int) {
size = (width, height)
var made: CGVirtualDisplay?
for _ in 0..<8 where made == nil {
VirtualDisplay.serial &+= 1
@@ -40,6 +44,9 @@ final class VirtualDisplay {
let s = CGVirtualDisplaySettings()
s.hiDPI = 1
let hz = VirtualDisplay.refreshRate
// Both modes: macOS then starts the display at 2x by itself, as long
// as it's at least 800x600 points; smaller, its 2x mode is never
// usable (probed 2026-09-28 on macOS 26.5.2).
s.modes = [CGVirtualDisplayMode(width: UInt32(2 * width), height: UInt32(2 * height), refreshRate: hz)!,
CGVirtualDisplayMode(width: UInt32(width), height: UInt32(height), refreshRate: hz)!]
if vd.apply(s), vd.displayID != 0 { made = vd }
@@ -47,23 +54,89 @@ final class VirtualDisplay {
guard let made else { return nil }
display = made
id = made.displayID
// Pick the HiDPI mode: `width` points drawn with twice the pixels.
}
/// Where it should sit: left of everything else, touching the leftmost
/// display. Otherwise macOS slots each new display in next to the main
/// one and pushes earlier ones along, and a window placed a moment before
/// ends up on the wrong display (seen 2026-09-28 with two windows).
private var wantOrigin: CGPoint?
private var modesSeen = -1
private var modeAskedAt = Date.distantPast
/// Applies the HiDPI mode and the position once macOS has registered the
/// display (right after creation both fail: no modes yet, and error 1001
/// for the origin). True once both are in place. Call until it is.
func settle() -> Bool {
if wantOrigin == nil {
var ids = [CGDirectDisplayID](repeating: 0, count: 32)
var n: UInt32 = 0
CGGetOnlineDisplayList(32, &ids, &n)
let others = ids.prefix(Int(n)).filter { $0 != id }.map { CGDisplayBounds($0) }
let left = others.map(\.minX).min() ?? 0
let top = others.first(where: { $0.minX == left })?.minY ?? 0
wantOrigin = CGPoint(x: left - CGFloat(size.width), y: top)
}
let placed = bounds.origin == wantOrigin
if isReady && placed { return true }
let (width, height) = size
// The HiDPI mode: `width` points drawn with twice the pixels.
let modes = (CGDisplayCopyAllDisplayModes(id, [kCGDisplayShowDuplicateLowResolutionModes: true] as CFDictionary)
as? [CGDisplayMode]) ?? []
if let hidpi = modes.first(where: {
let hidpi = modes.first(where: {
$0.width == width && $0.height == height && $0.pixelWidth == 2 * width && $0.refreshRate == VirtualDisplay.refreshRate
}) ?? modes.first(where: { $0.width == width && $0.height == height && $0.pixelWidth == 2 * width }) {
}) ?? modes.first(where: { $0.width == width && $0.height == height && $0.pixelWidth == 2 * width })
// Separately: a refused position would take the mode change down with it.
// A mode switch takes a moment; asking again restarts it.
if !isReady, let hidpi, Date().timeIntervalSince(modeAskedAt) > 1.5 {
modeAskedAt = Date()
let r = CGDisplaySetDisplayMode(id, hidpi, nil) // only if macOS didn't start it at 2x
agentLog("display \(id): set \(hidpi.width)x\(hidpi.height)@\(hidpi.pixelWidth) -> \(r.rawValue), "
+ "usable \(hidpi.isUsableForDesktopGUI()), now \(CGDisplayCopyDisplayMode(id).map { "\($0.width)x\($0.height)@\($0.pixelWidth)" } ?? "?")")
}
if !placed, let o = wantOrigin {
var cfg: CGDisplayConfigRef?
CGBeginDisplayConfiguration(&cfg)
CGConfigureDisplayWithDisplayMode(cfg, id, hidpi, nil)
CGConfigureDisplayOrigin(cfg, id, Int32(o.x), Int32(o.y))
CGCompleteDisplayConfiguration(cfg, .forSession)
}
if modesSeen != modes.count {
modesSeen = modes.count
let cur = CGDisplayCopyDisplayMode(id).map { "\($0.width)x\($0.height)@\($0.pixelWidth)" } ?? "?"
agentLog("display \(id): \(modes.count) modes [\(modes.map { "\($0.width)x\($0.height)@\($0.pixelWidth)\($0.isUsableForDesktopGUI() ? "u" : "")" }.joined(separator: " "))], "
+ "hidpi \(hidpi.map { "\($0.width)x\($0.height)@\($0.pixelWidth)" } ?? "none"), current \(cur), now \(bounds)")
}
return false
}
var bounds: CGRect { CGDisplayBounds(id) }
/// Moves several of our displays at once (one configuration, so they
/// never land on each other half way), retrying for a second until
/// macOS has them all where asked.
static func arrange(_ origins: [(VirtualDisplay, CGPoint)], attempt: Int = 0) {
let pending = origins.filter { $0.0.bounds.origin != $0.1 }
guard !pending.isEmpty else { return }
for (vd, p) in origins { vd.wantOrigin = p }
var cfg: CGDisplayConfigRef?
CGBeginDisplayConfiguration(&cfg)
for (vd, p) in origins { CGConfigureDisplayOrigin(cfg, vd.id, Int32(p.x), Int32(p.y)) }
CGCompleteDisplayConfiguration(cfg, .forSession)
guard attempt < 10 else {
agentLog("arrange: \(pending.count) displays not where asked")
return
}
DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) { arrange(origins, attempt: attempt + 1) }
}
var isHiDPI: Bool { CGDisplayCopyDisplayMode(id).map { $0.pixelWidth > $0.width } ?? false }
/// In the mode asked for: HiDPI at the requested size.
var isReady: Bool {
guard let m = CGDisplayCopyDisplayMode(id) else { return false }
return m.width == size.width && m.height == size.height && m.pixelWidth == 2 * size.width
}
var screen: NSScreen? {
NSScreen.screens.first {
($0.deviceDescription[NSDeviceDescriptionKey("NSScreenNumber")] as? NSNumber)?.uint32Value == id
@@ -150,6 +223,7 @@ final class SeparateSource: CaptureSource {
var frameRect: CGRect { inner?.frameRect ?? .zero }
var displayID: CGDirectDisplayID? { display?.id }
var virtualDisplay: VirtualDisplay? { display }
func start(maxLong: Int, fps: Int, completion: @escaping (String?) -> Void) {
guard AXIsProcessTrusted() else {
@@ -170,14 +244,18 @@ final class SeparateSource: CaptureSource {
if w.isFullScreen { return completion("take the window out of full screen first") }
// A display the window's size, plus room for the menu bar, in
// points; within what a panel can show sharply.
let width = min(max(Int(frame.width.rounded()), 640), 2560) / 2 * 2
let height = min(max(Int(frame.height.rounded()) + 40, 480), 1600) / 2 * 2
// At least 800x600 points: smaller displays don't get a usable 2x
// (sharp) mode on macOS 26. A smaller window is resized to fill it,
// or cropped if it keeps its own size.
let width = min(max(Int(frame.width.rounded()), 800), 2560) / 2 * 2
let height = min(max(Int(frame.height.rounded()) + 40, 600), 1600) / 2 * 2
guard let vd = VirtualDisplay(name: app.isEmpty ? "Frame Control" : "\(app) (Frame)", width: width, height: height) else {
return completion("macOS wouldn't create a display for this window")
}
display = vd
window = w
original = frame
agentLog("separate \(windowID) (\(app)): display \(vd.id) \(width)x\(height) pt for window \(frame)")
// The new screen shows up in NSScreen a moment later; its menu bar
// decides where the window can go.
placeWindow(attempt: 0, maxLong: maxLong, fps: fps, completion: completion)
@@ -188,14 +266,22 @@ final class SeparateSource: CaptureSource {
guard !stopped, let vd = display, let w = window else { return }
// Wait for the screen to appear, and for its HiDPI mode, so the
// first frames are captured sharp.
if vd.screen == nil || !vd.isHiDPI, attempt < 30 {
if !vd.settle() || vd.screen == nil, attempt < 30 {
DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
self.placeWindow(attempt: attempt + 1, maxLong: maxLong, fps: fps, completion: completion)
}
return
}
let target = vd.usableRect
let target = vd.usableRect // read now: displays may have moved meanwhile
w.setFrame(target)
if let now = w.frame, !vd.bounds.contains(CGPoint(x: now.midX, y: now.minY + 10)), attempt < 30 {
// The arrangement changed under us: try again in a moment.
DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
self.placeWindow(attempt: attempt + 1, maxLong: maxLong, fps: fps, completion: completion)
}
return
}
agentLog("separate \(windowID): screen after \(attempt) waits, ready \(vd.isReady), display \(vd.bounds), target \(target), window now \(w.frame.map { "\($0)" } ?? "?")")
guard let now = w.frame, vd.bounds.intersects(now) else {
restore()
return completion("macOS didn't let the window move to its own display (Stage Manager can prevent this)")
+102 -6
View File
@@ -36,8 +36,8 @@ func windowsJSON() -> [[String: Any]] {
$0.layer == 0 && $0.pid != me && !skip.contains($0.app) && $0.bounds.width >= 120 && $0.bounds.height >= 80
}.map {
["id": $0.id, "src": "window:\($0.id)", "pid": $0.pid, "app": $0.app, "title": $0.title,
"w": Int($0.bounds.width), "h": Int($0.bounds.height)]
}
"w": Int($0.bounds.width), "h": Int($0.bounds.height), "x": Int($0.bounds.minX), "y": Int($0.bounds.minY)]
} // front to back, as CGWindowList gives them
}
func displaysJSON() -> [[String: Any]] {
@@ -50,7 +50,7 @@ func displaysJSON() -> [[String: Any]] {
}.map { id in
let b = CGDisplayBounds(id)
return ["id": id, "src": "display:\(id)", "name": displayName(id), "w": Int(b.width), "h": Int(b.height),
"main": CGDisplayIsMain(id) != 0]
"x": Int(b.minX), "y": Int(b.minY), "main": CGDisplayIsMain(id) != 0]
}
}
@@ -106,6 +106,8 @@ final class Session {
var onEnd: ((Session) -> Void)?
var onAck: (() -> Void)?
var onFinished: ((String) -> Void)?
/// The Mac space's layout in the room (message "arrange").
var onArrange: (([[String: Any]]) -> Void)?
private let encodeQueue = DispatchQueue(label: "frame-mac-view.encode", qos: .userInteractive)
init(id: Int, source: Source, capture: CaptureSource, ws: WebSocket, codec: Codec, fps: Int, bitsPerPixel: Double,
@@ -245,6 +247,7 @@ final class Session {
capture.onFrame = { [weak self] pb, pts, shown in self?.offer(pb, pts: pts, capture: shown) }
capture.onEnded = { [weak self] reason in
guard let self else { return }
agentLog("stream \(self.id) \(self.source.key) ended: \(reason)")
self.ws.sendJSON(["t": "closed", "reason": reason])
self.onFinished?(self.source.key)
self.end()
@@ -275,8 +278,12 @@ final class Session {
}
t.resume()
statsTimer = t
agentLog("stream \(id) \(source.key) starting")
capture.start(maxLong: maxLong, fps: fps) { [weak self] error in
guard let self else { return }
agentLog("stream \(self.id) \(self.source.key) " + (error.map { "failed: \($0) (gone: \(self.capture.gone))" }
?? "capturing \(Int(self.capture.frameRect.width))x\(Int(self.capture.frameRect.height)) pt"
+ ((self.capture as? SeparateSource)?.displayID.map { " on display \($0)" } ?? "")))
if let error {
if self.capture.gone {
// Final, like a window closing mid-stream: the viewer closes
@@ -389,6 +396,10 @@ final class Session {
guard !isStopped, let d = text.data(using: .utf8),
let m = try? JSONSerialization.jsonObject(with: d) as? [String: Any], let t = m["t"] as? String else { return }
if report(t, m) { return }
if t == "arrange", let list = m["panels"] as? [[String: Any]] {
onArrange?(list)
return
}
let x = m["x"] as? Double ?? -1, y = m["y"] as? Double ?? -1
if t == "ack" {
onAck?()
@@ -438,13 +449,22 @@ final class Session {
case "release":
Input.releaseAll(owner: id)
case "focus":
// Before keys meant for this window (the Mac space's text-size buttons).
if case .window(let wid) = source, let pid = capture.pid { Input.focus(window: wid, pid: pid) }
if case .separate(let wid) = source, let pid = capture.pid { Input.focus(window: wid, pid: pid) }
default: break
}
}
}
}
/// One line to stderr (the agent's log), with the time.
func agentLog(_ s: String) {
let f = DateFormatter()
f.dateFormat = "HH:mm:ss.SSS"
FileHandle.standardError.write(Data("\(f.string(from: Date())) \(s)\n".utf8))
}
func randomKey() -> String {
var bytes = [UInt8](repeating: 0, count: 24)
_ = SecRandomCopyBytes(kSecRandomDefault, bytes.count, &bytes)
@@ -472,11 +492,47 @@ final class Agent {
private var tickets: [String: (src: String, expiry: Date, key: String?)] = [:]
/// reconnect key -> source, until Stop for that source.
private var reconnectKeys: [String: String] = [:]
/// Space keys (ui/mac-space.html): each lists the Mac's windows and
/// displays and opens streams for any of them, until Stop All (or Stop
/// for "space"). Handed out once, for a ticket for "space".
private var spaceKeys = Set<String>()
/// Sources that ended for good (the window closed), for Frame Control.
private var finished = Set<String>()
/// Ends every stream, then exits once separated windows are back on the
/// Mac's own screens (they're moved back, then their displays go 0.3 s later).
/// Lays the separated windows' displays out around the main display the
/// way their panels sit around the desktop panel in the room, so the
/// Mac's own pointer crosses from one to the next where the panels are
/// next to each other. `list`: [{src, x}], x in metres from the desktop
/// panel's centre, right positive.
func arrange(_ list: [[String: Any]]) {
lock.lock()
let bySrc = Dictionary(sessions.values.map { ($0.source.key, $0) }, uniquingKeysWith: { a, _ in a })
lock.unlock()
var sources: [(SeparateSource, Double)] = []
for e in list {
guard let src = e["src"] as? String, let x = num(e["x"]),
let sep = bySrc[src]?.capture as? SeparateSource else { continue }
sources.append((sep, x))
}
DispatchQueue.main.async { // Separate mode's state lives on main
let items: [(vd: VirtualDisplay, x: Double)] = sources.compactMap { sep, x in sep.virtualDisplay.map { (vd: $0, x: x) } }
let main = CGDisplayBounds(CGMainDisplayID())
var origins: [(VirtualDisplay, CGPoint)] = []
var right = main.maxX, left = main.minX
for (vd, _) in items.filter({ $0.x >= 0 }).sorted(by: { $0.x < $1.x }) {
origins.append((vd, CGPoint(x: right, y: main.minY)))
right += CGFloat(vd.size.width)
}
for (vd, _) in items.filter({ $0.x < 0 }).sorted(by: { $0.x > $1.x }) {
left -= CGFloat(vd.size.width)
origins.append((vd, CGPoint(x: left, y: main.minY)))
}
VirtualDisplay.arrange(origins)
}
}
/// Idempotent: SIGTERM and the parent going away can both call it, and a
/// session already ended by /close may still be putting its window back.
private var quitting = false
@@ -524,9 +580,26 @@ final class Agent {
return key
}
if let r = req.query["r"], reconnectKeys[r] == src { return r }
if let s = req.query["s"], spaceKeys.contains(s) {
let key = randomKey()
reconnectKeys[key] = src
return key
}
return nil
}
private func isSpace(_ req: Request) -> Bool {
lock.lock(); defer { lock.unlock() }
return spaceKeys.contains(req.query["s"] ?? "")
}
/// A page next to the viewer (the Mac space and its shared script); no secrets in them.
private func sibling(_ name: String, _ type: String, _ c: HTTPConnection) {
guard let page, let body = try? Data(contentsOf: page.deletingLastPathComponent().appendingPathComponent(name))
else { return c.respond(404, text: "not found") }
c.respond(200, body: body, type: type)
}
/// The viewer has its reconnect key, so the ticket that led to it can't
/// be used again (also when the ack comes over a reconnection).
func acknowledged(key: String) {
@@ -541,6 +614,25 @@ final class Agent {
case ("GET", "/view"):
guard let page, let body = try? Data(contentsOf: page) else { return c.respond(404, text: "no viewer page") }
return c.respond(200, body: body, type: "text/html; charset=utf-8")
case ("GET", "/space"): return sibling("mac-space.html", "text/html; charset=utf-8", c)
case ("GET", "/mac-stream.js"): return sibling("mac-stream.js", "text/javascript; charset=utf-8", c)
case ("GET", "/space/key"):
// A ticket for "space", spent here: the page keeps the key in memory.
lock.lock()
let now = Date()
tickets = tickets.filter { $0.value.expiry > now }
guard let t = req.query["t"], let entry = tickets[t], entry.src == "space" else {
lock.unlock()
return c.respond(403, text: "forbidden")
}
tickets[t] = nil
let key = randomKey()
spaceKeys.insert(key)
lock.unlock()
return c.respond(json: ["key": key])
case ("GET", "/space/windows"):
guard isSpace(req) else { return c.respond(403, text: "forbidden") }
return c.respond(json: ["windows": windowsJSON(), "displays": displaysJSON(), "input": Input.allowed])
case ("GET", "/stream"):
guard let src = Source(req.query["src"] ?? "") else { return c.respond(400, text: "bad src") }
guard let key = mayStream(req, src: src.key) else { return c.respond(403, text: "forbidden") }
@@ -571,11 +663,13 @@ final class Agent {
case ("GET", "/displays"):
c.respond(json: ["displays": displaysJSON()])
case ("POST", "/ticket"):
guard let src = Source(req.query["src"] ?? "") else { return c.respond(400, text: "bad src") }
let isSpaceTicket = req.query["src"] == "space"
guard isSpaceTicket || Source(req.query["src"] ?? "") != nil else { return c.respond(400, text: "bad src") }
let key = isSpaceTicket ? "space" : Source(req.query["src"] ?? "")!.key
let t = randomKey()
lock.lock()
tickets[t] = (src.key, Date().addingTimeInterval(60), nil)
finished.remove(src.key)
tickets[t] = (key, Date().addingTimeInterval(60), nil)
finished.remove(key)
lock.unlock()
c.respond(json: ["ticket": t])
case ("POST", "/close"):
@@ -586,6 +680,7 @@ final class Agent {
let matching = sessions.values.filter { src == nil || $0.source.key == src }
reconnectKeys = reconnectKeys.filter { src != nil && $0.value != src }
tickets = tickets.filter { src != nil && $0.value.src != src } // not yet used ones too
if src == nil || src == "space" { spaceKeys.removeAll() }
lock.unlock()
for s in matching { s.ws.sendJSON(["t": "close"]) }
DispatchQueue.global().asyncAfter(deadline: .now() + 0.3) { for s in matching { s.end() } }
@@ -660,6 +755,7 @@ final class Agent {
self.lock.unlock()
}
session.onAck = { [weak self] in self?.acknowledged(key: key) }
session.onArrange = { [weak self] list in self?.arrange(list) }
session.start(maxLong: maxLong, fps: fps)
}
}
+2 -1
View File
@@ -1,6 +1,7 @@
# Website
The Frame Control website, <https://frame-control.pages.dev>, on Cloudflare Pages.
The Frame Control website, <https://framecontrol.fun>, on Cloudflare Pages.
The Pages project is `frame-control`; its fallback hostname is `frame-control.pages.dev`.
- `public/`: static pages. `/` is the landing page, `/feedback/` the feedback form, `/privacy/` the privacy note.
- `functions/api/feedback.js`: `POST /api/feedback`, which turns the form into a GitHub issue labelled `feedback`.
+30 -19
View File
@@ -25,12 +25,16 @@ async function overLimit(kv, key, limit, ttl) {
export async function onRequestPost({ request, env }) {
if (!env.GITHUB_TOKEN || !env.GITHUB_REPO) {
return json(503, { error: "Feedback isn't connected to GitHub yet. Use the GitHub link instead." });
return json(503, { error: "Feedback isn't connected to GitHub yet. Use the GitHub link instead.", retrySafe: true });
}
const origin = request.headers.get("origin");
if (origin && new URL(origin).host !== new URL(request.url).host) {
return json(403, { error: "Send feedback from the website's form." });
if (origin) {
try {
if (new URL(origin).origin !== new URL(request.url).origin) throw new Error("Origin mismatch");
} catch {
return json(403, { error: "Send feedback from the website's form." });
}
}
let input;
@@ -63,24 +67,31 @@ export async function onRequestPost({ request, env }) {
console.log(`Rate limit check failed: ${err}`);
}
const res = await fetch(`https://api.github.com/repos/${env.GITHUB_REPO}/issues`, {
method: "POST",
headers: {
authorization: `Bearer ${env.GITHUB_TOKEN}`,
accept: "application/vnd.github+json",
"x-github-api-version": "2022-11-28",
"user-agent": "frame-control-website",
"content-type": "application/json",
},
body: JSON.stringify(buildIssue(checked.value)),
});
try {
const res = await fetch(`https://api.github.com/repos/${env.GITHUB_REPO}/issues`, {
method: "POST",
headers: {
authorization: `Bearer ${env.GITHUB_TOKEN}`,
accept: "application/vnd.github+json",
"x-github-api-version": "2022-11-28",
"user-agent": "frame-control-website",
"content-type": "application/json",
},
body: JSON.stringify(buildIssue(checked.value)),
});
if (!res.ok) {
console.log(`GitHub answered ${res.status}: ${(await res.text()).slice(0, 500)}`);
return json(502, { error: "GitHub didn't accept it just now. Try again, or use the GitHub link." });
if (!res.ok) {
return json(502, { error: "GitHub did not confirm delivery. Check recent issues before sending again.", retrySafe: false });
}
const issue = await res.json();
if (!Number.isInteger(issue.number) || typeof issue.html_url !== "string"
|| issue.html_url !== `https://github.com/${env.GITHUB_REPO}/issues/${issue.number}`) {
throw new Error("Invalid issue receipt");
}
return json(201, { ok: true, number: issue.number, url: issue.html_url });
} catch {
return json(502, { error: "Delivery could not be confirmed. Check recent issues before sending again.", retrySafe: false });
}
const issue = await res.json();
return json(201, { ok: true, number: issue.number, url: issue.html_url });
}
export const onRequest = () => json(405, { error: "POST only." });
+2 -2
View File
@@ -79,8 +79,8 @@ export function buildIssue(value) {
// Our own lines go first, so nothing in the sender's text can hide them.
const lines = [
value.github
? `> Sent from the website feedback form by @${value.github}.`
: "> Sent from the website feedback form. The sender left no GitHub username, so they won't see replies here.",
? `> Sent from the website feedback form. Provided GitHub username (not verified): @${value.github}.`
: "> Sent from the website feedback form. No GitHub username provided; bookmark or subscribe to this issue to follow replies.",
"",
];
if (details.length) {
+2 -2
View File
@@ -8,8 +8,8 @@
<meta name="theme-color" content="#0e141b">
<link rel="icon" href="/favicon.png">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
</head>
<body>
+21 -6
View File
@@ -3,8 +3,8 @@
--bg: #0e141b; --bg-2: #131c26; --panel: #17222e; --panel-2: #1d2a38; --line: rgba(143, 152, 160, .16);
--text: #c7d0d8; --bright: #fff; --muted: #8f98a0; --dim: #5e6873;
--blue: #1a9fff; --link: #66c0f4; --green: #75b022; --warn: #d9a23a; --bad: #e0573c;
--action: linear-gradient(to right, #47bfff 5%, #1a44c2 95%);
--action-hi: linear-gradient(to right, #5fcaff 5%, #2458d6 95%);
--action: linear-gradient(to right, #12699e 5%, #1a44c2 95%);
--action-hi: linear-gradient(to right, #1676ad 5%, #2458d6 95%);
--radius: 14px; --wrap: 1160px;
--font: "Inter", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Arial, sans-serif;
color-scheme: dark;
@@ -38,8 +38,19 @@ code { font: 14px ui-monospace, SFMono-Regular, Menlo, monospace; background: rg
.top nav a { color: var(--muted); font-size: 14.5px; font-weight: 500; }
.top nav a:hover, .top nav a[aria-current] { color: var(--bright); }
.top .end { margin-left: auto; display: flex; gap: 10px; align-items: center; }
@media (max-width: 880px) { .top nav { display: none; } }
@media (max-width: 480px) { .top .end .ghost { display: none; } }
@media (max-width: 880px) {
.top { position: static; }
.top .wrap { flex-wrap: wrap; height: auto; gap: 12px; padding-top: 12px; padding-bottom: 12px; }
.top nav { order: 3; width: 100%; margin-left: 0; gap: 16px; flex-wrap: wrap; }
.top nav a { padding: 6px 0; }
}
@media (max-width: 480px) {
.top .end .ghost { display: none; }
.top .wrap { padding-left: 16px; padding-right: 16px; gap: 8px; }
.top .brand { font-size: 12px; letter-spacing: 1px; gap: 8px; }
.top .brand img { width: 24px; height: 24px; }
.top .end .btn { padding: 0 10px; }
}
/* ---- buttons ---- */
.btn { display: inline-flex; align-items: center; justify-content: center; gap: 9px; height: 46px; padding: 0 22px; border-radius: 10px;
@@ -53,8 +64,8 @@ code { font: 14px ui-monospace, SFMono-Regular, Menlo, monospace; background: rg
.btn.ghost { background: transparent; border-color: var(--line); }
.btn.ghost:hover { border-color: rgba(143,152,160,.4); background: rgba(255,255,255,.03); }
.btn.small { height: 36px; padding: 0 14px; font-size: 14px; border-radius: 8px; }
.btn.coffee { background: #ff5e5b; box-shadow: 0 8px 24px rgba(255, 94, 91, .22); }
.btn.coffee:hover { background: #ff7471; }
.btn.coffee { background: #ff5e5b; color: #0e141b; box-shadow: 0 8px 24px rgba(255, 94, 91, .22); }
.btn.coffee:hover { background: #ff7471; color: #0e141b; }
.btn svg { width: 18px; height: 18px; flex: none; }
.btn[disabled] { opacity: .6; cursor: progress; transform: none; }
@@ -64,6 +75,7 @@ code { font: 14px ui-monospace, SFMono-Regular, Menlo, monospace; background: rg
background: radial-gradient(600px 380px at 30% 30%, rgba(26,159,255,.22), transparent 70%),
radial-gradient(520px 360px at 72% 20%, rgba(111,66,193,.2), transparent 70%); }
.hero > * { position: relative; }
.hero .cta .btn { max-width: 100%; min-height: 46px; height: auto; padding-top: 12px; padding-bottom: 12px; white-space: normal; }
.eyebrow { display: inline-block; max-width: 100%; padding: 6px 14px; border-radius: 999px; font-size: 13.5px;
color: var(--link); background: rgba(26,159,255,.1); border: 1px solid rgba(102,192,244,.22); margin-bottom: 26px; }
.eyebrow b { color: var(--bright); font-weight: 600; }
@@ -131,10 +143,13 @@ section.alt { background: var(--bg-2); border-block: 1px solid var(--line); }
.dl .need { color: var(--muted); font-size: 14px; }
.dl .links { display: flex; flex-direction: column; gap: 8px; margin-top: 16px; }
.dl .links .btn { width: 100%; }
.dl > p { margin-top: 12px; font-size: 14px; }
.dl code { overflow-wrap: anywhere; }
.dl .tag { margin-left: auto; font-size: 11.5px; font-weight: 650; letter-spacing: 1px; text-transform: uppercase; color: var(--link); }
.faq { max-width: 780px; margin: 0 auto; display: grid; gap: 10px; }
.faq details { background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 0 22px; }
.faq code { overflow-wrap: anywhere; }
.faq summary { cursor: pointer; list-style: none; padding: 18px 0; color: var(--bright); font-weight: 600; display: flex; justify-content: space-between; gap: 16px; }
.faq summary::-webkit-details-marker { display: none; }
.faq summary::after { content: "+"; color: var(--muted); font-size: 22px; line-height: 1; transition: transform .2s; }
+4 -4
View File
@@ -8,9 +8,9 @@
<meta name="theme-color" content="#0e141b">
<link rel="icon" href="/favicon.png">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<script src="/js/feedback.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
<script src="/js/feedback.js?v=20260930-deep1" defer></script>
</head>
<body>
@@ -87,7 +87,7 @@
</div>
<div class="field">
<label for="github">GitHub username <small>Optional. Add it to get notified when someone replies.</small></label>
<label for="github">GitHub username <small>Optional and not verified. Subscribe to the created issue on GitHub to follow replies.</small></label>
<input type="text" id="github" name="github" maxlength="40" autocomplete="username" placeholder="@yourname">
</div>
+39 -22
View File
@@ -11,10 +11,12 @@
<meta property="og:type" content="website">
<meta property="og:title" content="Frame Control">
<meta property="og:description" content="Manage your Valve Steam Frame from your computer. Free and open source.">
<meta property="og:image" content="/img/og.jpg">
<meta property="og:image" content="https://framecontrol.fun/img/og.jpg">
<link rel="canonical" href="https://framecontrol.fun/">
<meta property="og:url" content="https://framecontrol.fun/">
<meta name="twitter:card" content="summary_large_image">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
</head>
<body>
@@ -41,20 +43,21 @@
<main>
<section class="hero">
<div class="wrap">
<span class="eyebrow"><b>Free and open source</b><span class="sep"> · </span><span class="plats">macOS · Windows · Linux · iPhone</span></span>
<span class="eyebrow"><b>Free and open source</b><span class="sep"> · </span><span class="plats">macOS · Windows · Linux</span></span>
<h1>Your Steam Frame, <span>managed from your desk.</span></h1>
<p class="lede">See what the headset sees, install games and Android apps, move files and text across, and keep an eye on battery and status. All over SSH, with nothing to install on the Frame.</p>
<p class="lede">Install Frame Control on your computer, not your headset. See what the Frame sees, install games and Android apps, move files and text across, and check battery and status.</p>
<div class="cta">
<a class="btn primary" id="hero-download" href="#download">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M12 3v12m0 0-5-5m5 5 5-5M4 20h16"/></svg>
<span data-label>Download Frame Control</span>
<span data-label>Get started — download the app</span>
</a>
<a class="btn ghost" href="#trailer">
<svg viewBox="0 0 24 24" fill="currentColor" aria-hidden="true"><path d="M8 5.5v13a1 1 0 0 0 1.5.86l10.5-6.5a1 1 0 0 0 0-1.72L9.5 4.64A1 1 0 0 0 8 5.5z"/></svg>
Watch the trailer
</a>
</div>
<p class="fine"><span hidden>Latest: <span data-version></span> · </span><a href="#download">All platforms</a> · MIT licence · Not affiliated with Valve</p>
<p class="fine">Download → install and open → connect your headset. <a href="#setup">Setup guide</a></p>
<p class="fine"><span hidden>Latest: <span data-version></span> · </span><a href="#download">All desktop downloads</a> · <a href="#iphone">iPhone developer build</a> · Not affiliated with Valve</p>
<div class="shot">
<div class="bar"><i></i><i></i><i></i></div>
<img src="/img/home-live.jpg" width="1600" height="1000" alt="Frame Control's Home tab: a live view of the headset beside battery, storage, temperature and Wi-Fi.">
@@ -168,12 +171,12 @@
</div>
</section>
<section>
<section id="iphone">
<div class="wrap">
<div class="head">
<span class="kicker">iPhone and iPad</span>
<h2>Also in your pocket</h2>
<p>The same features on iPhone and iPad, served from the Frame itself, so there's nothing to run on a computer. Build it from the repo in Xcode.</p>
<h2>iPhone: a separate developer build</h2>
<p>Not an App Store download. This option requires a Mac and Xcode, and runs a server on the Frame instead of your computer. <a href="https://github.com/saphid/frame-control/blob/main/docs/iphone.md">Read the iPhone build and pairing guide</a>. For the simplest start, use the desktop app below.</p>
</div>
<div class="phones">
<img src="/img/phone-home.jpg" width="600" height="1304" loading="lazy" alt="Frame Control on iPhone: headset view and device status.">
@@ -186,21 +189,22 @@
<div class="wrap">
<div class="head">
<span class="kicker">Setup</span>
<h2>One password, typed once</h2>
<p>Everything else happens on your computer.</p>
<h2>From download to connected</h2>
<p>You'll need a computer and a Steam Frame, both on the same trusted network. Keep the headset awake while connecting.</p>
</div>
<div class="steps">
<div class="card">
<h3>Install and open the app</h3>
<p><a href="#download">Choose your computer's download below</a>, install it, then open <b>Frame Control</b>. You don't install the desktop app on the headset, and you don't need to clone GitHub or install Python.</p>
</div>
<div class="card">
<h3>Turn on Developer Mode</h3>
<p>On the Frame: Steam Settings → System → <b>Enable Developer Mode</b>, then <b>Set User Password</b> in the Developer section.</p>
</div>
<div class="card">
<h3>Set up the connection</h3>
<p>Open Frame Control and choose <b>Set Up Connection</b>. It finds the headset, makes an SSH key, and asks for that password once.</p>
</div>
<div class="card">
<h3>That's it</h3>
<p>The app reaches the headset whenever it's awake and on the same network. For anywhere else, use <a href="https://github.com/saphid/frame-control/blob/main/docs/tailscale.md">Tailscale</a>.</p>
<p>In Frame Control, choose <b>Set Up Connection</b> and enter the headset's password when asked. It creates an SSH key so you don't have to type it next time.</p>
<p>Headset not found? Enter its IP address from the Frame's Quick Settings. For remote access, use <a href="https://github.com/saphid/frame-control/blob/main/docs/tailscale.md">Tailscale</a>.</p>
</div>
</div>
</div>
@@ -211,7 +215,7 @@
<div class="head">
<span class="kicker">Download</span>
<h2>Get Frame Control</h2>
<p>Free, with Python and adb built in. <span hidden>Version <span data-version></span>.</span> <a href="https://github.com/saphid/frame-control/releases">Release notes</a></p>
<p>Download on your computer, then follow the opening steps on your card. Python is included; adb is included except on arm64 Linux. <span hidden>Version <span data-version></span>.</span> <a href="https://github.com/saphid/frame-control/releases">Release notes</a></p>
</div>
<div class="downloads">
<div class="dl" data-os="mac">
@@ -220,6 +224,8 @@
<div class="links">
<a class="btn primary" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-mac-arm64.dmg">Download .dmg</a>
</div>
<p><b>Open:</b> open the .dmg, drag Frame Control to Applications, then open it from Applications.</p>
<p class="need">Apple Silicon only, not Intel Macs. Unsigned and not notarized: <a href="#mac-first-launch">read the first-launch instructions</a> if macOS blocks it.</p>
</div>
<div class="dl" data-os="windows">
<div class="os"><svg viewBox="0 0 24 24" fill="currentColor"><path d="M3 5.5 10 4.5v7H3zM11 4.35 21 3v8.5H11zM3 12.5h7v7L3 18.5zM11 12.5h10V21l-10-1.4z"/></svg><h3>Windows</h3><span class="tag" hidden>Your system</span></div>
@@ -228,15 +234,21 @@
<a class="btn primary" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-Setup-x64.exe">Download installer</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-win-x64.zip">Portable .zip</a>
</div>
<p><b>Open:</b> run the installer, then launch Frame Control from Start. For the portable .zip, extract it first and run <code>Frame Control.exe</code>.</p>
<p class="need">Unsigned: <a href="#windows-first-launch">read about SmartScreen</a> before proceeding past a warning.</p>
</div>
<div class="dl" data-os="linux">
<div class="os"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="3" y="4" width="18" height="16" rx="2"/><path d="m7 9 3 3-3 3M13 15h4"/></svg><h3>Linux</h3><span class="tag" hidden>Your system</span></div>
<span class="need">x64 or arm64 · needs <code>ssh</code></span>
<p>On arm64 Linux, install <code>adb</code> from your distribution for Android apps: on Debian or Ubuntu, run <code>sudo apt install adb</code>.</p>
<div class="links">
<a class="btn primary" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-x86_64.AppImage">AppImage (x64)</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-amd64.deb">.deb (x64)</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-arm64.AppImage">AppImage (arm64)</a>
<a class="btn ghost" href="https://github.com/saphid/frame-control/releases/latest/download/Frame-Control-linux-arm64.deb">.deb (arm64)</a>
</div>
<p><b>Open:</b> for an AppImage, use your file manager's Properties → Permissions to allow executing it, then open it. For a .deb, install it with your package manager and launch Frame Control.</p>
<p class="need"><a href="#linux-first-launch">AppImage won't open?</a> Then <a href="#setup">connect your headset</a>.</p>
</div>
</div>
</div>
@@ -249,22 +261,27 @@
<h2>Good questions</h2>
</div>
<div class="faq">
<details>
<details id="mac-first-launch">
<summary>macOS says the app is damaged or can't be checked</summary>
<div><p>The app isn't notarized, because there's no paid Apple developer account behind it. Drag it to Applications, then clear the download quarantine once in Terminal:</p>
<div><p>The macOS download isn't notarized. Drag it to Applications, then clear the download quarantine once in Terminal:</p>
<p><code>xattr -dr com.apple.quarantine "/Applications/Frame Control.app"</code></p></div>
</details>
<details>
<details id="windows-first-launch">
<summary>Windows SmartScreen says it protected my PC</summary>
<div><p>The installer isn't code-signed. Choose <b>More info → Run anyway</b>, or use the portable .zip: unzip it anywhere and run <code>Frame Control.exe</code>.</p></div>
</details>
<details>
<summary id="linux-first-launch">Linux: the AppImage won't open</summary>
<div><p>Make the file executable in your file manager, or in Terminal run <code>chmod +x Frame-Control-linux-x86_64.AppImage</code> and then <code>./Frame-Control-linux-x86_64.AppImage</code>. On arm64, use the arm64 filename instead.</p>
<p>If FUSE is missing, use the .deb package or run the AppImage with <code>--appimage-extract-and-run</code>.</p></div>
</details>
<details>
<summary>What does it change on my headset?</summary>
<div><p>Only what you click. Installs go to your user account on the Frame, and nothing needs <code>sudo</code> except the power buttons. On your computer it adds a <code>Host frame</code> entry to <code>~/.ssh/config</code> and two SSH keys.</p></div>
</details>
<details>
<summary>Is it safe to leave Developer Mode on?</summary>
<div><p>With Developer Mode on, SSH, ADB and remote desktop are reachable on your local network. Use trusted networks, turn Developer Mode off when you don't need it, and never port-forward those ports from your router. For remote access, use Tailscale. <a href="https://github.com/saphid/frame-control#readme">Security notes</a></p></div>
<div><p>With Developer Mode on, SSH, ADB and remote desktop are reachable on your local network. Use trusted networks, turn Developer Mode off when you don't need it, and never port-forward those ports from your router. Tailscale is an advanced option; off-LAN access has not been verified. Read the <a href="https://github.com/saphid/frame-control/blob/main/docs/tailscale.md#exposure-every-port-is-on-the-tailnet">Tailscale exposure notes</a> before enabling it.</p></div>
</details>
<details>
<summary>Is this made by Valve?</summary>
+15 -8
View File
@@ -27,19 +27,21 @@ form.addEventListener("change", (e) => {
function githubUrl(data) {
const params = data.kind === "idea"
? new URLSearchParams({ template: "idea.yml", title: data.title, what: data.message })
: new URLSearchParams({ template: "bug.yml", title: data.title, description: data.message,
version: data.version, os: [data.os, data.steamos && `SteamOS ${data.steamos}`].filter(Boolean).join(", ") });
: data.kind === "bug"
? new URLSearchParams({ template: "bug.yml", title: data.title, description: data.message,
version: data.version, os: [data.os, data.steamos && `SteamOS ${data.steamos}`].filter(Boolean).join(", ") })
: new URLSearchParams({ title: data.title, body: data.message });
return `https://github.com/saphid/frame-control/issues/new?${params}`;
}
function showError(message, data) {
function showError(message, data, uncertain = false) {
errorBox.textContent = message + " ";
if (data) {
const a = document.createElement("a");
a.href = githubUrl(data);
a.href = uncertain ? "https://github.com/saphid/frame-control/issues" : githubUrl(data);
a.target = "_blank";
a.rel = "noopener";
a.textContent = "Open it on GitHub instead";
a.textContent = uncertain ? "Check recent issues before sending again" : "Open it on GitHub instead";
errorBox.append(a);
}
errorBox.hidden = false;
@@ -62,6 +64,7 @@ form.addEventListener("submit", async (e) => {
send.disabled = true;
send.textContent = "Sending…";
let uncertain = true;
try {
// The server treats anything sent sooner as a script (site/lib/feedback.js MIN_FILL_MS).
const wait = 3100 - (performance.now() - started);
@@ -72,7 +75,11 @@ form.addEventListener("submit", async (e) => {
body: JSON.stringify({ ...data, elapsed: Math.round(performance.now() - started) }),
});
const reply = await res.json().catch(() => ({}));
if (!res.ok) throw new Error(reply.error || "Something went wrong sending that.");
if (!res.ok) {
uncertain = reply.retrySafe === false || (reply.retrySafe !== true && res.status >= 500);
throw new Error(reply.error || "Something went wrong sending that.");
}
if (!reply.ok) throw new Error("Delivery could not be confirmed.");
form.hidden = true;
document.getElementById("done").hidden = false;
@@ -81,11 +88,11 @@ form.addEventListener("submit", async (e) => {
link.href = reply.url;
link.hidden = false;
document.getElementById("done-text").textContent = `Your feedback is now issue #${reply.number} on GitHub.`
+ (data.github ? " You'll be notified when someone replies." : " Bookmark it to follow along.");
+ " Bookmark it or subscribe on GitHub to follow replies.";
}
window.scrollTo({ top: 0, behavior: "smooth" });
} catch (err) {
showError(err.message || "Couldn't reach the server.", data);
showError(err.message || "Couldn't confirm delivery.", data, uncertain);
} finally {
send.disabled = false;
send.textContent = "Send feedback";
+15 -5
View File
@@ -5,8 +5,6 @@ const SITE = {
kofi: "alexsouthwell",
};
const RELEASE = `https://github.com/${SITE.repo}/releases/latest/download/`;
// Donate buttons.
for (const el of document.querySelectorAll("[data-kofi]")) {
if (SITE.kofi) el.href = `https://ko-fi.com/${SITE.kofi}`;
@@ -28,7 +26,7 @@ function detectPlatform() {
}
const PLATFORMS = {
mac: { name: "macOS", file: "Frame-Control-mac-arm64.dmg" },
mac: { name: "macOS (Apple Silicon)", file: "Frame-Control-mac-arm64.dmg" },
windows: { name: "Windows", file: "Frame-Control-Setup-x64.exe" },
linux: { name: "Linux", file: "Frame-Control-linux-x86_64.AppImage" },
"linux-arm": { name: "Linux (arm64)", file: "Frame-Control-linux-arm64.AppImage" },
@@ -37,8 +35,7 @@ const PLATFORMS = {
const platform = detectPlatform();
const hero = document.getElementById("hero-download");
if (hero && PLATFORMS[platform]) {
hero.href = RELEASE + PLATFORMS[platform].file;
hero.querySelector("[data-label]").textContent = `Download for ${PLATFORMS[platform].name}`;
hero.querySelector("[data-label]").textContent = `Get started on ${PLATFORMS[platform].name}`;
}
const card = platform && document.querySelector(`.dl[data-os="${platform.replace("-arm", "")}"]`);
if (card) {
@@ -61,3 +58,16 @@ if (versionEls.length) {
}
document.querySelectorAll("[data-year]").forEach((el) => (el.textContent = new Date().getFullYear()));
function revealLinkedAnswer(hash = location.hash) {
const target = document.getElementById(hash.slice(1));
const answer = target?.closest("details");
if (answer) answer.open = true;
}
window.addEventListener("hashchange", () => revealLinkedAnswer());
document.addEventListener("click", (event) => {
const link = event.target?.closest?.('a[href^="#"]');
if (link) revealLinkedAnswer(link.getAttribute("href"));
});
revealLinkedAnswer();
+8 -6
View File
@@ -8,8 +8,8 @@
<meta name="theme-color" content="#0e141b">
<link rel="icon" href="/favicon.png">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="stylesheet" href="/css/site.css">
<script src="/js/site.js" defer></script>
<link rel="stylesheet" href="/css/site.css?v=20260930-deep1">
<script src="/js/site.js?v=20260930-deep1" defer></script>
</head>
<body>
@@ -38,12 +38,14 @@
<div class="head">
<span class="kicker">Privacy</span>
<h1>Privacy</h1>
<p>Short version: the app collects nothing, and the website keeps only what you choose to send.</p>
<p>Short version: the released app sends anonymous usage statistics after a first-run notice. You can turn them off. Website feedback becomes public on GitHub.</p>
</div>
<div class="faq notes">
<section><h2>The app</h2><p>Frame Control talks only to your headset (over SSH on your network), to GitHub for releases, and to Steam and F-Droid for game and app listings. It has no analytics and no accounts.</p></section>
<section><h2>The feedback form</h2><p>What you type becomes a public GitHub issue on <a href="https://github.com/saphid/frame-control/issues">saphid/frame-control</a>. To stop abuse, the form keeps a one-way hash of your IP address for about an hour to count submissions. The address itself isn't stored or published.</p></section>
<section><h2>This website</h2><p>Hosted on Cloudflare Pages. No cookies, no analytics, no trackers. The download section asks GitHub for the latest version number. Donations go through Ko-fi, under Ko-fi's own privacy policy.</p></section>
<section><h2>App usage statistics</h2><p>In v0.4.0, anonymous usage statistics are on by default after the first-run notice has been shown. They go to PostHog's US cloud and include a random persistent installation ID, app version, operating system, CPU architecture and Python version; app opens, updates, tabs used, headset SteamOS builds, and install outcomes. Public catalogue installs may include a Steam app ID, Flathub ID or F-Droid package name. Usage events exclude local filenames, paths and account data. Person profiles and location lookup are disabled, and events use a placeholder IP address; the receiving service still handles the network connection.</p><p>Turn statistics off in <b>Privacy &amp; updates</b>. <b>Show what's been sent</b> lets you inspect sent events. Turning a level off removes its unsent events, not events already received. Source checkouts do not send automatic telemetry by default. <code>DO_NOT_TRACK=1</code> or <code>FRAME_CONTROL_TELEMETRY=0</code> blocks automatic telemetry, not reports you deliberately send through <b>Report a problem</b>.</p></section>
<section><h2>Optional app reports</h2><p>Compatibility sharing and detailed error reports are off by default. Enabling compatibility sharing sends reports and tests, including earlier reports on this computer. Enabling error details sends scrubbed error messages and tracebacks. Review free text before sharing: automatic scrubbing is not a guarantee that every personal detail is removed. The app's <b>Report a problem</b> dialog deliberately sends a private report to the maintainer's PostHog project, independently of the usage-statistics setting. This differs from the public website form below. See the <a href="https://github.com/saphid/frame-control/blob/v0.4.0/docs/privacy.md">v0.4.0 privacy guide</a> for exact fields and controls.</p></section>
<section><h2>Other app connections</h2><p>Frame Control connects to your headset and to services used by the features you choose, including GitHub for releases, Steam, F-Droid and Flathub for listings or downloads, and the sources of apps you choose to install. It does not require a Frame Control account.</p></section>
<section><h2>The website feedback form</h2><p>What you type becomes a public GitHub issue on <a href="https://github.com/saphid/frame-control/issues">saphid/frame-control</a>, including any username and diagnostic details you provide. Do not include passwords, tokens or private information. To stop abuse, the form keeps a token-salted, one-way hash of your IP address in an hourly counter for up to 65 minutes, plus a non-identifying daily counter for up to 25 hours. The form does not store or publish the raw address; Cloudflare still processes network requests.</p></section>
<section><h2>This website</h2><p>Hosted on Cloudflare Pages. The site itself sets no cookies and includes no analytics or tracking scripts. Cloudflare processes requests to serve and protect it. The download section asks GitHub for the latest release. Donations go through Ko-fi, under Ko-fi's own privacy policy.</p></section>
</div>
</div>
</main>
+64
View File
@@ -0,0 +1,64 @@
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { test } from "node:test";
import vm from "node:vm";
const script = await readFile(new URL("../public/js/feedback.js", import.meta.url), "utf8");
function load(fetchResponse) {
const handlers = {};
const links = [];
const form = { elements: { os: { value: "" } }, addEventListener: (name, handler) => { handlers[name] = handler; } };
const errorBox = { hidden: true, append: (link) => links.push(link), scrollIntoView: () => {} };
const send = { disabled: false, textContent: "Send feedback" };
let elapsed = 0;
const context = {
navigator: { userAgent: "Windows" },
performance: { now: () => elapsed },
URLSearchParams,
fetch: fetchResponse,
window: { scrollTo: () => {} },
document: {
getElementById: (id) => ({ feedback: form, error: errorBox, send })[id] || {},
createElement: () => ({}),
},
FormData: class extends Map {
constructor() { super(Object.entries({ kind: "question", title: "A local question", message: "This test never sends real feedback." })); }
},
};
vm.runInNewContext(script, context);
elapsed = 4000;
return { context, handlers, links, send };
}
test("questions and general feedback are not mislabelled as bugs in GitHub fallback", () => {
const page = load();
for (const kind of ["question", "other"]) {
const url = new URL(page.context.githubUrl({ kind, title: "A local question", message: "Please help" }));
assert.equal(url.searchParams.has("template"), false);
assert.equal(url.searchParams.get("body"), "Please help");
}
assert.equal(new URL(page.context.githubUrl({ kind: "bug", title: "A bug", message: "Details" })).searchParams.get("template"), "bug.yml");
assert.equal(new URL(page.context.githubUrl({ kind: "idea", title: "An idea", message: "Details" })).searchParams.get("template"), "idea.yml");
});
test("an unconfirmed send points to recent issues rather than creating a duplicate", async () => {
const page = load(async () => ({ ok: false, status: 502, json: async () => ({ error: "Could not confirm delivery", retrySafe: false }) }));
await page.handlers.submit({ preventDefault: () => {} });
assert.equal(page.links[0].href, "https://github.com/saphid/frame-control/issues");
assert.equal(page.send.disabled, false);
});
test("a browser network failure also points to recent issues", async () => {
const page = load(async () => { throw new Error("Local offline test"); });
await page.handlers.submit({ preventDefault: () => {} });
assert.equal(page.links[0].href, "https://github.com/saphid/frame-control/issues");
});
test("known-unsent integration errors retain the prefilled issue fallback", async () => {
const page = load(async () => ({ ok: false, status: 503, json: async () => ({ error: "Not connected", retrySafe: true }) }));
await page.handlers.submit({ preventDefault: () => {} });
const url = new URL(page.links[0].href);
assert.equal(url.pathname, "/saphid/frame-control/issues/new");
assert.equal(url.searchParams.get("body"), "This test never sends real feedback.");
});
+56
View File
@@ -0,0 +1,56 @@
import assert from "node:assert/strict";
import { test } from "node:test";
import { onRequestPost } from "../functions/api/feedback.js";
const env = { GITHUB_TOKEN: "local-test-placeholder", GITHUB_REPO: "example/local-test" };
const input = { kind: "bug", title: "Local test report", message: "No external requests are made by these tests.", elapsed: 4000 };
const request = (origin, body = input) => new Request("https://framecontrol.fun/api/feedback", {
method: "POST",
headers: { "content-type": "application/json", ...(origin === undefined ? {} : { origin }) },
body: JSON.stringify(body),
});
test("malformed and opaque Origins are rejected without throwing or contacting GitHub", async (testContext) => {
const upstream = testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Unexpected external request"); });
for (const origin of ["not-a-url", "null", "https://other.example", "http://framecontrol.fun"]) {
const response = await onRequestPost({ request: request(origin), env });
assert.equal(response.status, 403, origin);
}
assert.equal(upstream.mock.callCount(), 0);
});
test("same-origin invalid JSON content is rejected before contacting GitHub", async (testContext) => {
const upstream = testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Unexpected external request"); });
const response = await onRequestPost({ request: request("https://framecontrol.fun", null), env });
assert.equal(response.status, 400);
assert.equal(upstream.mock.callCount(), 0);
});
test("missing integration settings fail without contacting GitHub", async (testContext) => {
const upstream = testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Unexpected external request"); });
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env: {} });
assert.equal(response.status, 503);
assert.equal((await response.json()).retrySafe, true);
assert.equal(upstream.mock.callCount(), 0);
});
test("GitHub network failure produces an uncertainty-aware response instead of throwing", async (testContext) => {
testContext.mock.method(globalThis, "fetch", async () => { throw new Error("Local simulated outage"); });
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env });
assert.equal(response.status, 502);
assert.equal((await response.json()).retrySafe, false);
});
test("an unreadable successful GitHub response does not encourage duplicate submission", async (testContext) => {
testContext.mock.method(globalThis, "fetch", async () => new Response("not JSON", { status: 201 }));
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env });
assert.equal(response.status, 502);
assert.equal((await response.json()).retrySafe, false);
});
test("successful feedback returns the created issue", async (testContext) => {
testContext.mock.method(globalThis, "fetch", async () => Response.json({ number: 7, html_url: "https://github.com/example/local-test/issues/7" }, { status: 201 }));
const response = await onRequestPost({ request: request("https://framecontrol.fun"), env });
assert.equal(response.status, 201);
assert.deepEqual(await response.json(), { ok: true, number: 7, url: "https://github.com/example/local-test/issues/7" });
});
+4 -4
View File
@@ -37,20 +37,20 @@ test("unknown kinds become other feedback", () => {
assert.equal(validate(form({ kind: "__proto__" })).value.kind, "other");
});
test("builds a labelled issue that credits a GitHub user", () => {
test("builds a labelled issue with an explicitly unverified GitHub username", () => {
const { value } = validate(form({ github: "@octocat", version: "0.3.1", steamos: "20260922" }));
const issue = buildIssue(value);
assert.equal(issue.title, "Bug report: Live view freezes");
assert.deepEqual(issue.labels, ["feedback", "bug"]);
assert.match(issue.body, /\| Frame Control version \| 0\.3\.1 \|/);
assert.match(issue.body, /^> Sent from the website feedback form by @octocat\./);
assert.match(issue.body, /Provided GitHub username \(not verified\): @octocat\./);
assert.ok(issue.body.endsWith(value.message));
});
test("anonymous feedback says replies won't reach the sender", () => {
test("anonymous feedback explains how to follow replies", () => {
const issue = buildIssue(validate(form({ kind: "other" })).value);
assert.deepEqual(issue.labels, ["feedback"]);
assert.match(issue.body, /won't see replies/);
assert.match(issue.body, /bookmark or subscribe/);
});
test("breaks mentions, issue refs and table cells in user text", () => {
+123
View File
@@ -0,0 +1,123 @@
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import { test } from "node:test";
import vm from "node:vm";
const script = await readFile(new URL("../public/js/site.js", import.meta.url), "utf8");
const html = await readFile(new URL("../public/index.html", import.meta.url), "utf8");
function visit(platform, userAgent = "", maxTouchPoints = 0, hash = "") {
const label = { textContent: "Get started — download the app" };
const hero = { href: "#download", querySelector: () => label };
const answer = { open: false };
const listeners = {};
const context = {
navigator: { platform, userAgent, maxTouchPoints },
location: { hash },
window: { addEventListener: (event, handler) => { listeners[event] = handler; } },
document: {
addEventListener: (event, handler) => { listeners[`document-${event}`] = handler; },
querySelectorAll: () => [],
querySelector: () => null,
getElementById: (id) => id === "hero-download" ? hero : id === "mac-first-launch" ? { closest: () => answer } : null,
},
};
vm.runInNewContext(script, context);
return { hero, label, answer, listeners, context };
}
test("desktop hero leads to opening instructions rather than a bare package", () => {
for (const [platform, name] of [["MacIntel", "macOS (Apple Silicon)"], ["Win32", "Windows"], ["Linux x86_64", "Linux"], ["Linux aarch64", "Linux (arm64)"]]) {
const page = visit(platform);
assert.equal(page.hero.href, "#download");
assert.equal(page.label.textContent, `Get started on ${name}`);
}
});
test("phones and touch-enabled iPads are not offered a desktop package", () => {
for (const [platform, userAgent, touches] of [["iPhone", "iPhone", 1], ["MacIntel", "Macintosh", 5], ["Linux armv8l", "Android", 1]]) {
const page = visit(platform, userAgent, touches);
assert.equal(page.hero.href, "#download");
assert.equal(page.label.textContent, "Get started — download the app");
}
});
test("first-launch links open the answer on arrival and hash navigation", () => {
assert.equal(visit("", "", 0, "#mac-first-launch").answer.open, true);
const page = visit("");
page.context.location.hash = "#mac-first-launch";
page.listeners.hashchange();
assert.equal(page.answer.open, true);
assert.doesNotThrow(() => visit("", "", 0, "#missing"));
});
test("clicking the same first-launch link reopens a manually closed answer", () => {
const page = visit("", "", 0, "#mac-first-launch");
page.answer.open = false;
page.listeners["document-click"]({ target: { closest: () => ({ getAttribute: () => "#mac-first-launch" }) } });
assert.equal(page.answer.open, true);
assert.doesNotThrow(() => page.listeners["document-click"]({ target: {} }));
});
test("landing page keeps unique anchors and opening instructions for every desktop", () => {
const ids = [...html.matchAll(/\bid="([^"]+)"/g)].map((match) => match[1]);
assert.equal(new Set(ids).size, ids.length);
for (const match of html.matchAll(/href="#([^"]+)"/g)) assert.ok(ids.includes(match[1]), match[1]);
assert.match(html, /Install Frame Control on your computer, not your headset/);
assert.match(html, /drag Frame Control to Applications/);
assert.match(html, /launch Frame Control from Start/);
assert.match(html, /Properties → Permissions/);
assert.match(html, /Frame-Control-linux-arm64\.deb/);
assert.match(html, /Not an App Store download/);
});
test("sharing metadata points to the custom domain", () => {
assert.match(html, /rel="canonical" href="https:\/\/framecontrol\.fun\/"/);
assert.match(html, /property="og:url" content="https:\/\/framecontrol\.fun\/"/);
assert.match(html, /property="og:image" content="https:\/\/framecontrol\.fun\/img\/og\.jpg"/);
});
test("Linux arm64 download explains the system adb prerequisite", () => {
assert.match(html, /On arm64 Linux, install <code>adb<\/code>/);
});
test("privacy describes released telemetry and distinguishes private app reports from public feedback", async () => {
const privacy = await readFile(new URL("../public/privacy/index.html", import.meta.url), "utf8");
assert.doesNotMatch(privacy, /app collects nothing|has no analytics/);
for (const text of ["on by default", "PostHog", "random persistent installation ID", "Privacy &amp; updates", "off by default", "public GitHub issue", "v0.4.0/docs/privacy.md"]) {
assert.ok(privacy.includes(text), text);
}
});
test("compact headers retain navigation and high-contrast support text", async () => {
const css = await readFile(new URL("../public/css/site.css", import.meta.url), "utf8");
assert.doesNotMatch(css, /\.top nav\s*\{\s*display:\s*none/);
assert.match(css, /\.top nav \{ order: 3; width: 100%/);
assert.match(css, /@media \(max-width: 880px\) \{\s*\.top \{ position: static; \}/);
assert.match(css, /\.btn\.coffee \{[^}]*color: #0e141b/);
});
test("primary button gradients keep white text above 4.5:1 contrast", async () => {
const css = await readFile(new URL("../public/css/site.css", import.meta.url), "utf8");
const luminance = (channels) => channels.map((channel) => {
const normalized = channel / 255;
return normalized <= 0.04045 ? normalized / 12.92 : ((normalized + 0.055) / 1.055) ** 2.4;
}).reduce((total, channel, index) => total + channel * [0.2126, 0.7152, 0.0722][index], 0);
for (const variable of ["action", "action-hi"]) {
const value = css.match(new RegExp(`--${variable}: ([^;]+);`))[1];
const colors = [...value.matchAll(/#([0-9a-f]{6})/g)].map((match) => [0, 2, 4].map((offset) => parseInt(match[1].slice(offset, offset + 2), 16)));
assert.equal(colors.length, 2);
for (let step = 0; step <= 100; step++) {
const channels = colors[0].map((channel, index) => channel + (colors[1][index] - channel) * step / 100);
assert.ok(1.05 / (luminance(channels) + 0.05) >= 4.5, `${variable} at ${step}%`);
}
}
});
test("every page versions shared assets to avoid stale onboarding behavior", async () => {
for (const file of ["index.html", "feedback/index.html", "privacy/index.html", "404.html"]) {
const page = await readFile(new URL(`../public/${file}`, import.meta.url), "utf8");
assert.match(page, /href="\/css\/site\.css\?v=/, file);
assert.match(page, /src="\/js\/site\.js\?v=/, file);
}
});
+41 -1
View File
@@ -113,6 +113,17 @@ class Helpers(unittest.TestCase):
self.assertIn("Chromium", str(cm.exception))
self.assertTrue(calls[0].startswith("bash -s -- "))
def test_the_mac_space_opens_with_a_space_ticket(self):
calls, asked = [], []
mv = frame_macview.MacView(["ssh"], lambda remote, **kw: calls.append(remote) or "panel x", "frame")
mv.call = lambda path, **kw: asked.append((path, kw)) or {"screen": True, "ticket": "tk1"}
mv.ensure_tunnel = lambda **kw: None
mv.remote_port = 47900
out = mv.show("space")
self.assertIn(("/ticket", {"method": "POST", "src": "space"}), asked)
self.assertIn("http://127.0.0.1:47900/space?t=tk1", calls[0])
self.assertEqual(out["src"], "space")
def test_separate_windows_need_accessibility(self):
mv = frame_macview.MacView(["ssh"], lambda *a, **k: self.fail("no ssh"), "frame")
mv.call = lambda path, **kw: {"screen": True, "accessibility": False}
@@ -284,7 +295,10 @@ class Agent(unittest.TestCase):
def test_serves_the_viewer_page(self):
status, body = self.get(f"/view?k={self.token}&src=test")
self.assertEqual(status, 200)
self.assertIn(b"VideoDecoder", body)
self.assertIn(b'<script src="/mac-stream.js">', body)
status, script = self.get("/mac-stream.js")
self.assertEqual(status, 200)
self.assertIn(b"VideoDecoder", script)
def test_snapshot_needs_the_key(self):
self.assertEqual(self.get("/snapshot?display=1")[0], 403)
@@ -345,6 +359,32 @@ class Agent(unittest.TestCase):
self.get(f"/close?k={self.token}&src=test", method="POST")
self.assertEqual(WS(self.port, f"/stream?src=test&r={hello['r']}").status, 403)
def test_a_space_key_opens_any_source_until_stop_all(self):
t = self.ticket("space")
status, body = self.get(f"/space/key?t={t}")
self.assertEqual(status, 200)
key = json.loads(body)["key"]
self.assertEqual(self.get(f"/space/key?t={t}")[0], 403) # a ticket gives one key
self.assertEqual(self.get("/space/windows?s=nope")[0], 403)
listing = json.loads(self.get(f"/space/windows?s={key}")[1])
self.assertIn("windows", listing)
self.assertTrue(all("x" in d and "y" in d for d in listing["displays"]))
ws = WS(self.port, f"/stream?src=test&s={key}")
self.assertEqual(ws.status, 101)
hello = json.loads(ws.recv()[1])
self.assertEqual(hello["t"], "hello")
ws.close()
self.assertEqual(WS(self.port, f"/stream?src=test&t={t}").status, 403) # not a stream ticket
self.get(f"/close?k={self.token}", method="POST") # Stop All
self.assertEqual(self.get(f"/space/windows?s={key}")[0], 403)
self.assertEqual(WS(self.port, f"/stream?src=test&s={key}").status, 403)
self.assertEqual(WS(self.port, f"/stream?src=test&r={hello['r']}").status, 403)
def test_the_space_page_and_its_script_are_served(self):
for path in ("/space", "/mac-stream.js"):
status, _ = self.get(path)
self.assertEqual(status, 200, path)
def test_stop_revokes_tickets_not_yet_used(self):
t = self.ticket("test")
self.get(f"/close?k={self.token}&src=test", method="POST")
+30
View File
@@ -9,6 +9,7 @@ failure; FRAME_PC_INSTALLER_BUNDLE=1 checks the installer's bundle instead.
"""
import ctypes
import http.client
from http.server import ThreadingHTTPServer
import json
import os
from pathlib import Path
@@ -31,6 +32,35 @@ from frame_stream_stats import Stats
from test_macview import WS
class ViewerResourcesTests(unittest.TestCase):
def test_public_viewer_can_load_its_script_without_master_credentials(self):
httpd = ThreadingHTTPServer(('127.0.0.1', 0), agent.Handler)
httpd.agent = mock.Mock(page=ROOT / 'ui/mac-view.html', grants=agent.Grants('master'))
worker = threading.Thread(target=httpd.serve_forever, daemon=True)
worker.start()
try:
for path, content_type, expected in [
('/view', 'text/html', b'<script src="/mac-stream.js">'),
('/mac-stream.js', 'application/javascript', b'class MacStream'),
('/status', 'application/json', b'forbidden'),
('/../mac-stream.js', 'application/json', b'forbidden'),
]:
with self.subTest(path=path):
conn = http.client.HTTPConnection('127.0.0.1', httpd.server_port, timeout=3)
try:
conn.request('GET', path)
response = conn.getresponse()
self.assertEqual(response.status, 200 if path in ('/view', '/mac-stream.js') else 403)
self.assertIn(content_type, response.getheader('Content-Type'))
self.assertIn(expected, response.read())
finally:
conn.close()
finally:
httpd.shutdown()
httpd.server_close()
worker.join(timeout=3)
class GrantsTests(unittest.TestCase):
def test_source_bound_retry_ack_and_stop(self):
g = agent.Grants('master')
+13 -5
View File
@@ -386,7 +386,7 @@ class MacView:
self.launching -= 1
def _show(self, src, quality, width, height):
if src != "test" and not src.startswith(("window:", "display:", "separate:")):
if src not in ("test", "space") and not src.startswith(("window:", "display:", "separate:")):
raise MacViewError("Pick a window or display to show.")
self.shows += 1
q = QUALITY.get(quality) or QUALITY["balanced"]
@@ -411,11 +411,19 @@ class MacView:
# A single-use ticket for this source, not Frame Control's key: the URL
# is visible in the Frame's process list.
ticket = self.call("/ticket", method="POST", src=src)["ticket"]
params = {"src": src, "t": ticket, "tag": tag, "codec": q["codec"], "max": q["max"], "fps": q["fps"],
"bpp": q["bpp"]}
url = f"http://127.0.0.1:{self.remote_port}/view?{urlencode(params)}"
if src == "space":
# The Mac space (ui/mac-space.html): one WebXR page that opens the
# Mac's display and any windows pulled out of it (docs/mac-space.md).
url = f"http://127.0.0.1:{self.remote_port}/space?{urlencode({'t': ticket, 'tag': tag})}"
else:
params = {"src": src, "t": ticket, "tag": tag, "codec": q["codec"], "max": q["max"], "fps": q["fps"],
"bpp": q["bpp"]}
url = f"http://127.0.0.1:{self.remote_port}/view?{urlencode(params)}"
w, h = fit(width or 1280, height or 720)
args = " ".join(_quote(str(a)) for a in (appid, url, w, h, tag, self.viewer_profile, *self.browser_flags))
# WebXR needs Chromium XR's OpenXR feature, and its XR process needs
# the seccomp filter off on arm64 (docs/webxr-chromium.md).
xr = ["--enable-features=OpenXR", "--disable-seccomp-filter-sandbox"] if src == "space" else []
args = " ".join(_quote(str(a)) for a in (appid, url, w, h, tag, self.viewer_profile, *self.browser_flags, *xr))
try:
out = self.run("bash -s -- " + args, stdin=LAUNCH, timeout=60)
except Exception as e: # noqa: BLE001 - the server's Failure carries the Frame's words
+5
View File
@@ -498,6 +498,11 @@ class Handler(BaseHTTPRequestHandler):
return self.reply(b'frame-mac-view', content='text/plain')
if method == 'GET' and url.path == '/view':
return self.reply(agent.page.read_bytes(), content='text/html; charset=utf-8')
if method == 'GET' and url.path == '/mac-stream.js':
script = agent.page.with_name('mac-stream.js')
if not script.is_file():
return self.reply({'error': 'not found'}, 404)
return self.reply(script.read_bytes(), content='application/javascript; charset=utf-8')
if method == 'GET' and url.path == '/stream':
return self.stream(q)
if not agent.grants.master(q.get('k', self.headers.get('X-Token'))):
+13 -1
View File
@@ -494,6 +494,8 @@
.mv-perm .row { margin-top: 8px; }
.mv-live { color: var(--green, #7fd67f); font-size: 12px; }
.mv-opt { display: block; font-size: 13px; margin: 0 0 12px; }
.mv-space { align-items: center; gap: 12px; margin: 0 0 14px; }
.mv-space .sub { font-size: 13px; }
.mv-opt .sub { display: block; margin-left: 22px; font-size: 12px; }
.disp select { width: 100%; background: rgba(0,0,0,.28); color: var(--text); border: 1px solid transparent; border-radius: 3px;
padding: 8px 10px; font: inherit; font-size: 13px; }
@@ -1254,6 +1256,7 @@
<section id="shots">
<div class="shelf-head"><h2>Screenshots</h2><span class="count" id="shotCount"></span><span class="spacer"></span>
<button class="small" id="shotsRefresh">Refresh</button>
<button class="small" id="phoneShotDownloads" hidden>Downloads &amp; Photos</button>
<button class="small" id="shotsFolder" title="Open the SteamFrame folder in your Pictures">Show folder</button>
<button class="action small" id="shotsSaveNew" disabled>Save new to this computer</button>
</div>
@@ -1737,6 +1740,8 @@
<div class="sub" id="mvPcState" role="status" aria-live="polite" hidden></div>
</div>
<div id="mvMain">
<div class="row mv-space"><button class="action" data-mv="show" data-src="space" data-title="Mac space">Open Mac space</button>
<span class="sub">Your Mac's screen in the room, like Vision Pro. Drag a window's title bar off its edge to give it a place of its own.</span></div>
<label class="mv-opt"><input type="checkbox" id="mvSeparate" checked> Give each window its own display
<span class="sub">its menus and sheets come along, nothing covers it, and it goes back when you stop</span></label>
<div class="row" style="margin:0 0 12px"><label for="mvQuality">Picture quality</label>
@@ -5074,6 +5079,10 @@ api("/api/host").then(applyHostWording).catch(() => {});
// thumbs: id -> promise of an object URL, kept across reloads so a refresh
// that finds a new shot fetches only that one's thumbnail.
const shots = { list: [], sig: null, thumbs: new Map(), loading: null, gen: 0, fill: 0 };
if (window.frameApp?.screenshotDownloads) {
$("phoneShotDownloads").hidden = false;
$("phoneShotDownloads").onclick = () => window.frameApp.screenshotDownloads();
}
const STEAMVR_APPID = "250820";
const SHOTS_POLL_MS = 8000;
function shotApp(appid) {
@@ -5667,6 +5676,7 @@ async function loadMacView(restart) {
$("mvMain").hidden = false;
$("mvPcState").hidden = true;
$("mvSeparate").closest("label").hidden = pc; // separate virtual displays are a Mac feature
$("macview").querySelector(".mv-space").hidden = pc;
const live = new Set((s.streams || []).map(x => x.src));
const byStream = src => (s.streams || []).find(x => x.src === src);
$("mvCount").textContent = live.size ? `${live.size} showing${s.route === "usb" ? " · over USB-C" : ""}` : "";
@@ -5744,7 +5754,9 @@ $("macview").onclick = async e => {
await act(`Show ${b.dataset.title} in the headset`, async () => {
const r = await api("/api/macview", { action: "show", src: as, quality: $("mvQuality").value,
w: +b.dataset.w || undefined, h: +b.dataset.h || undefined });
return { message: `${b.dataset.title} is a panel in the headset now. Float it with the SteamVR dashboard.`, ...r };
const message = as === "space" ? "The Mac space is ready in the headset: press Enter Mac space there."
: `${b.dataset.title} is a panel in the headset now. Float it with the SteamVR dashboard.`;
return { message, ...r };
}, b);
} else if (b.dataset.mv === "stop") {
await act("Stop showing it", () => api("/api/macview", { action: "stop", src }), b);
+687
View File
@@ -0,0 +1,687 @@
<!doctype html>
<!-- Frame Control: the Mac space. The Mac's main display as a large panel in
the room, and any Mac window pulled out of it as a panel of its own, in
one WebXR session with passthrough behind (docs/mac-space.md). Served by
the Mac's frame-mac-view agent; streams come from ui/mac-stream.js. Without
WebXR (a desktop browser) the same scene shows in the page, with the
mouse as the laser: left button = trigger, right button = grab. -->
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Mac space</title>
<style>
html, body { margin: 0; height: 100%; background: #101216; color: #eee; font: 16px/1.45 system-ui, sans-serif; overflow: hidden; }
#gl { position: fixed; inset: 0; width: 100%; height: 100%; display: block; }
#ui { position: fixed; inset: 0; display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 18px;
text-align: center; padding: 24px; background: rgba(16,18,22,.92); }
#ui[hidden] { display: none; }
#enter { font: 600 28px system-ui, sans-serif; padding: 22px 44px; border-radius: 18px; border: 0; color: #fff;
background: #2f6fed; cursor: pointer; }
#enter:disabled { background: #444; cursor: default; }
#msg { max-width: 34em; color: #bbb; }
#hint { position: fixed; left: 12px; bottom: 10px; color: #9aa; font-size: 13px; pointer-events: none; }
</style>
</head>
<body>
<canvas id="gl"></canvas>
<div id="ui">
<div style="font-size:22px">Mac space</div>
<button id="enter" disabled>Connecting…</button>
<div id="msg">Your Mac's screen appears in front of you. Drag a window's title bar off its edge to give that window a
place of its own; drop it back on the screen to put it back.</div>
</div>
<div id="hint" hidden>Preview: left button clicks, right button (or Shift) drags a panel by its bar, wheel scrolls.</div>
<script src="/mac-stream.js"></script>
<script>
"use strict";
const q = new URLSearchParams(location.search);
const tag = q.get("tag") || "";
document.title = tag ? `Mac space [${tag}]` : "Mac space";
const $ = id => document.getElementById(id);
const say = t => { $("msg").textContent = t; };
// ---------- small 3D math (column-major 4x4, like WebGL) ----------
const V = {
add: (a, b) => [a[0] + b[0], a[1] + b[1], a[2] + b[2]],
sub: (a, b) => [a[0] - b[0], a[1] - b[1], a[2] - b[2]],
scale: (a, s) => [a[0] * s, a[1] * s, a[2] * s],
dot: (a, b) => a[0] * b[0] + a[1] * b[1] + a[2] * b[2],
len: a => Math.hypot(a[0], a[1], a[2]),
norm: a => { const l = Math.hypot(a[0], a[1], a[2]) || 1; return [a[0] / l, a[1] / l, a[2] / l]; },
};
const M = {
mul(a, b) {
const o = new Float32Array(16);
for (let c = 0; c < 4; c++) for (let r = 0; r < 4; r++) {
let s = 0;
for (let k = 0; k < 4; k++) s += a[k * 4 + r] * b[c * 4 + k];
o[c * 4 + r] = s;
}
return o;
},
// Rigid transform (rotation about Y by yaw, then position), scaled by sx, sy.
panel(pos, yaw, sx = 1, sy = 1) {
const c = Math.cos(yaw), s = Math.sin(yaw);
return new Float32Array([c * sx, 0, -s * sx, 0, 0, sy, 0, 0, s, 0, c, 0, pos[0], pos[1], pos[2], 1]);
},
point: (m, p) => [m[0] * p[0] + m[4] * p[1] + m[8] * p[2] + m[12], m[1] * p[0] + m[5] * p[1] + m[9] * p[2] + m[13],
m[2] * p[0] + m[6] * p[1] + m[10] * p[2] + m[14]],
dir: (m, d) => [m[0] * d[0] + m[4] * d[1] + m[8] * d[2], m[1] * d[0] + m[5] * d[1] + m[9] * d[2], m[2] * d[0] + m[6] * d[1] + m[10] * d[2]],
perspective(fovy, aspect, near, far) {
const f = 1 / Math.tan(fovy / 2), nf = 1 / (near - far);
return new Float32Array([f / aspect, 0, 0, 0, 0, f, 0, 0, 0, 0, (far + near) * nf, -1, 0, 0, 2 * far * near * nf, 0]);
},
// Inverse of a rigid transform (rotation + translation only).
invRigid(m) {
const o = new Float32Array(16);
for (let r = 0; r < 3; r++) for (let c = 0; c < 3; c++) o[c * 4 + r] = m[r * 4 + c];
const t = [m[12], m[13], m[14]];
o[12] = -(o[0] * t[0] + o[4] * t[1] + o[8] * t[2]);
o[13] = -(o[1] * t[0] + o[5] * t[1] + o[9] * t[2]);
o[14] = -(o[2] * t[0] + o[6] * t[1] + o[10] * t[2]);
o[15] = 1;
return o;
},
};
// ---------- panels ----------
// A panel is a Mac display or window: a video quad, a bar under it to grab,
// and a close button to the bar's left (as on Vision Pro).
const BAR_GAP = 0.03, BAR_H = 0.018, CLOSE_R = 0.013;
const panels = [];
let desktop = null;
function barOf(p) {
const w = Math.min(0.28, p.w * 0.3);
return { cx: 0, cy: -p.h / 2 - BAR_GAP, w, h: BAR_H };
}
function closeOf(p) { const b = barOf(p); return { cx: -b.w / 2 - 0.04, cy: b.cy, r: CLOSE_R }; }
// Text size buttons (A−, A+) to the bar's right, on window panels.
function zoomOf(p, dir) { const b = barOf(p); return { cx: b.w / 2 + (dir < 0 ? 0.04 : 0.075), cy: b.cy, r: CLOSE_R }; }
function placeFacing(p, pos, head) {
p.pos = pos;
// Upright, turned to face the viewer (yaw only), as windows do on Vision Pro.
p.yaw = Math.atan2(head[0] - pos[0], head[2] - pos[2]);
}
function matrixOf(p) { return M.panel(p.pos, p.yaw); }
// Where a ray meets the panel's plane, in panel metres (x right, y up), or null.
function hitPanel(p, origin, dir) {
const inv = M.invRigid(matrixOf(p));
const o = M.point(inv, origin), d = M.dir(inv, dir);
if (Math.abs(d[2]) < 1e-6) return null;
const t = -o[2] / d[2];
if (t <= 0) return null;
return { x: o[0] + d[0] * t, y: o[1] + d[1] * t, t };
}
// What a ray points at: the nearest panel part.
function pick(origin, dir) {
let best = null;
for (const p of panels) {
const h = hitPanel(p, origin, dir);
if (!h) continue;
let part = null;
const b = barOf(p), c = closeOf(p);
if (Math.abs(h.x) <= p.w / 2 && Math.abs(h.y) <= p.h / 2) part = "content";
else if (Math.abs(h.x - b.cx) <= b.w / 2 + 0.01 && Math.abs(h.y - b.cy) <= b.h / 2 + 0.012) part = "bar";
else if (Math.hypot(h.x - c.cx, h.y - c.cy) <= c.r + 0.008) part = "close";
else if (p.kind === "window") {
for (const dir of [-1, 1]) {
const z = zoomOf(p, dir);
if (Math.hypot(h.x - z.cx, h.y - z.cy) <= z.r + 0.008) part = dir < 0 ? "zoomOut" : "zoomIn";
}
}
if (part && (!best || h.t < best.t)) best = { panel: p, part, t: h.t, x: h.x, y: h.y, u: h.x / p.w + 0.5, v: 0.5 - h.y / p.h };
}
return best;
}
let spaceKey = "", layout = { windows: [], displays: [] }, mainDisplay = null;
async function refreshLayout() {
try {
const r = await fetch(`/space/windows?s=${encodeURIComponent(spaceKey)}`);
if (r.ok) layout = await r.json();
} catch (e) {}
}
// Replaceable in tests (the Mac's real stacking decides otherwise).
const hooks = { windowAt: (u, v) => windowAt(u, v) };
// The topmost Mac window under a point of the main display (u, v in 0..1).
function windowAt(u, v) {
const d = mainDisplay;
if (!d) return null;
const x = d.x + u * d.w, y = d.y + v * d.h;
return layout.windows.find(w => x >= w.x && x < w.x + w.w && y >= w.y && y < w.y + w.h) || null;
}
function openPanel(src, opts) {
const p = { src, kind: opts.kind, title: opts.title || "", w: opts.w || 1.2, h: opts.h || 0.7, pos: [0, 1.4, -1.2], yaw: 0,
zoom: { auto: true, base: 0, steps: 0 },
tex: null, texW: 0, texH: 0, pending: null, lastMove: 0, pendingMove: null };
p.stream = new MacStream({
src, space: spaceKey, max: opts.max || "2560", fps: "60",
onFrame(img, w, h, seq, decodedAt) {
// Keep only the newest picture until the next rendered frame.
if (p.pending) { p.stream.replaced(p.pending.seq, p.pending.decodedAt); p.pending.img.close(); }
p.pending = { img, w, h, seq, decodedAt };
},
onInfo(m) { if (m.title) p.title = m.title; },
onClosed() { removePanel(p, false); },
});
p.stream.start();
panels.push(p);
return p;
}
function removePanel(p, stop = true) {
const i = panels.indexOf(p);
if (i < 0) return;
panels.splice(i, 1);
if (stop) p.stream.stop(); // the Mac ends the stream; a separated window goes back
if (p.pending) { p.pending.img.close(); p.pending = null; }
if (p.tex && gl) gl.deleteTexture(p.tex);
if (p === desktop) desktop = null;
else sendArrange();
}
// Tell the Mac where the window panels are, left or right of the desktop, so
// it lays their displays out the same way: the Mac's own pointer then moves
// from the desktop into the panel beside it, as it looks in the room.
let arrangeTimer = 0;
function sendArrange() {
clearTimeout(arrangeTimer);
arrangeTimer = setTimeout(() => {
if (!desktop) return;
const inv = M.invRigid(matrixOf(desktop));
const list = panels.filter(p => p.kind === "window").map(p => ({ src: p.src, x: +M.point(inv, p.pos)[0].toFixed(3) }));
desktop.stream.send({ t: "arrange", panels: list });
}, 300);
}
// Text size: the window's own zoom (Cmd+= / Cmd+-), which browsers, editors,
// terminals and chat apps all have. Automatic by default: after a window
// panel is moved, its text follows its distance, a step per 12% nearer or
// farther than where it was first let go; the A−/A+ buttons take over by hand.
const ZOOM_STEP = 1.12;
function textSize(p, steps) {
if (!steps || !panels.includes(p)) return;
p.stream.focus();
const code = steps > 0 ? "Equal" : "Minus", key = steps > 0 ? "=" : "-";
for (let i = 0; i < Math.abs(steps); i++) {
setTimeout(() => {
p.stream.key(true, code, key, ["meta"]);
setTimeout(() => p.stream.key(false, code, key, ["meta"]), 30);
}, 120 + i * 90);
}
p.zoom.steps += steps;
}
function autoTextSize(p) {
if (p.kind !== "window" || !p.zoom.auto || !p.zoom.base) return;
const d = V.len(V.sub(p.pos, headNow.pos));
const want = Math.max(-6, Math.min(8, Math.round(Math.log(d / p.zoom.base) / Math.log(ZOOM_STEP))));
textSize(p, want - p.zoom.steps);
}
// Remembered placement of the desktop panel, in the room.
function saveDesktop() {
if (desktop) localStorage.setItem("macspace.desktop", JSON.stringify({ pos: desktop.pos, yaw: desktop.yaw, w: desktop.w }));
}
function restoreDesktop(head) {
try {
const s = JSON.parse(localStorage.getItem("macspace.desktop") || "null");
if (s && s.pos && s.pos.length === 3) { desktop.pos = s.pos; desktop.yaw = s.yaw; desktop.w = s.w; return; }
} catch (e) {}
// First time: a large screen a little below eye level, 1.2 m ahead.
const fwd = head.forward;
placeFacing(desktop, V.add(head.pos, [fwd[0] * 1.2, -0.12, fwd[2] * 1.2]), head.pos);
}
// ---------- steadying the laser ----------
// A One Euro filter (Casiez et al. 2012) per axis: heavy smoothing while the
// pointer moves slowly (hand tremor), little lag when it moves fast.
class OneEuro {
constructor(minCutoff = 1.2, beta = 0.35, dCutoff = 1) { Object.assign(this, { minCutoff, beta, dCutoff, x: null, dx: 0, t: 0 }); }
alpha(cutoff, dt) { const tau = 1 / (2 * Math.PI * cutoff); return 1 / (1 + tau / dt); }
filter(v, t) {
if (this.x === null) { this.x = v; this.t = t; return v; }
const dt = Math.max(1e-3, (t - this.t) / 1000);
this.t = t;
const d = (v - this.x) / dt;
this.dx += this.alpha(this.dCutoff, dt) * (d - this.dx);
const cutoff = this.minCutoff + this.beta * Math.abs(this.dx);
this.x += this.alpha(cutoff, dt) * (v - this.x);
return this.x;
}
}
// Smooths a pointer's ray (direction in radians-ish units, origin in metres)
// and keeps the last 200 ms, so a click can use where it pointed just before
// the trigger jerked it.
function steady(pt, origin, dir, t) {
if (!pt.f) pt.f = { d: [0, 1, 2].map(() => new OneEuro(1.0, 0.6)), o: [0, 1, 2].map(() => new OneEuro(1.5, 0.8)), hist: [] };
pt.origin = origin.map((v, i) => pt.f.o[i].filter(v, t));
pt.dir = V.norm(dir.map((v, i) => pt.f.d[i].filter(v, t)));
pt.f.hist.push({ t, origin: pt.origin, dir: pt.dir });
while (pt.f.hist.length && t - pt.f.hist[0].t > 200) pt.f.hist.shift();
}
// Where the pointer was `ago` ms ago (or now).
function pointedAgo(pt, ago) {
const h = pt.f && pt.f.hist;
if (!h || !h.length) return { origin: pt.origin, dir: pt.dir };
const t = h[h.length - 1].t - ago;
return h.find(e => e.t >= t) || h[h.length - 1];
}
// Click stabilisation: after the trigger goes down, the Mac's pointer stays
// put until the ray has clearly moved (1.2 degrees), so a click doesn't turn
// into a small drag.
const CLICK_HOLD_RAD = 1.2 * Math.PI / 180;
// ---------- pointers: controllers, hands, or the mouse ----------
// Each pointer has a ray; the trigger (select) clicks, squeeze grabs.
const pointers = new Map();
function pointerState(key) {
if (!pointers.has(key)) pointers.set(key, { key, origin: [0, 0, 0], dir: [0, 0, -1], hit: null, selecting: false,
grab: null, press: null, lastWheel: 0 });
return pointers.get(key);
}
// Called whenever a pointer's ray moves.
function pointerMoved(pt, head) {
pt.hit = pick(pt.origin, pt.dir);
if (pt.grab) return dragGrab(pt, head);
const press = pt.press;
if (press && press.panel === desktop && press.win && panels.includes(desktop)) {
// Dragging a window's title bar off the edge of the desktop: detach it.
const h = hitPanel(desktop, pt.origin, pt.dir);
const outside = !h || Math.abs(h.x) > desktop.w / 2 || Math.abs(h.y) > desktop.h / 2;
if (outside) return detach(pt, press, head);
}
const target = press ? press.panel : pt.hit && pt.hit.part === "content" ? pt.hit.panel : null;
if (!target) return;
if (press && pt.hold) {
if (Math.acos(Math.min(1, V.dot(pt.hold.dir, pt.dir))) < CLICK_HOLD_RAD) return;
pt.hold = null; // moved on purpose: it's a drag
}
const h = press ? hitPanel(target, pt.origin, pt.dir) : pt.hit;
if (!h) return;
const u = Math.min(1, Math.max(0, h.x / target.w + 0.5)), v = Math.min(1, Math.max(0, 0.5 - h.y / target.h));
if (press) press.last = { u, v };
sendMove(target, u, v);
}
// Moves go at most every 8 ms per panel.
function sendMove(p, u, v) {
const now = performance.now();
if (now - p.lastMove >= 8) { p.stream.move(u, v); p.lastMove = now; p.pendingMove = null; }
else p.pendingMove = { u, v };
}
function selectStart(pt, head) {
// Where it pointed just before the trigger pull jerked the controller.
const before = pointedAgo(pt, 80);
const hit = pick(before.origin, before.dir) || pt.hit;
if (!hit) return;
pt.hold = { dir: before.dir, until: true };
if (hit.part === "close") {
if (hit.panel === desktop) return leave();
return removePanel(hit.panel);
}
if (hit.part === "bar") return startGrab(pt, hit.panel, head);
if (hit.part === "zoomOut" || hit.part === "zoomIn") {
hit.panel.zoom.auto = false; // chosen by hand from now on
return textSize(hit.panel, hit.part === "zoomIn" ? 1 : -1);
}
// A click on the Mac. On the desktop, remember which window it started on,
// and whether it was on its title bar (the top 32 points).
const p = hit.panel, u = hit.u, v = hit.v;
let win = null;
if (p === desktop) {
const w = hooks.windowAt(u, v);
if (w && mainDisplay && (mainDisplay.y + v * mainDisplay.h) - w.y <= 32) win = w;
}
pt.press = { panel: p, win, last: { u, v }, down: { u, v } };
p.stream.down(u, v, 0);
}
function selectEnd(pt) {
const held = !!pt.hold;
pt.hold = null;
if (pt.grab && pt.grab.byTrigger) return endGrab(pt);
const press = pt.press;
pt.press = null;
// A clean click lets go where it went down.
if (press && panels.includes(press.panel)) press.panel.stream.up(held ? press.down.u : press.last.u, held ? press.down.v : press.last.v, 0);
}
function squeezeStart(pt, head) {
if (pt.hit) startGrab(pt, pt.hit.panel, head, false);
}
function squeezeEnd(pt) { if (pt.grab && !pt.grab.byTrigger) endGrab(pt); }
// Grabbing: the panel keeps its distance along the ray and its offset from
// the hit point, and turns to face the viewer. Thumbstick up/down pushes and
// pulls; left/right resizes.
function startGrab(pt, p, head, byTrigger = true) {
const h = hitPanel(p, pt.origin, pt.dir);
const dist = h ? h.t : V.len(V.sub(p.pos, pt.origin));
const at = V.add(pt.origin, V.scale(pt.dir, dist));
pt.grab = { panel: p, dist, offset: V.sub(p.pos, at), byTrigger };
panels.push(panels.splice(panels.indexOf(p), 1)[0]); // draw it last
}
function dragGrab(pt, head) {
const g = pt.grab;
if (!panels.includes(g.panel)) { pt.grab = null; return; }
const at = V.add(pt.origin, V.scale(pt.dir, g.dist));
placeFacing(g.panel, V.add(at, g.offset), head.pos);
}
function endGrab(pt) {
const g = pt.grab;
pt.grab = null;
if (!g || !panels.includes(g.panel)) return;
const p = g.panel;
sendArrange();
if (p === desktop) return saveDesktop();
// A window dropped back onto the desktop goes home, at its old text size.
if (desktop) {
const h = hitPanel(desktop, pt.origin, pt.dir);
if (h && Math.abs(h.x) <= desktop.w / 2 && Math.abs(h.y) <= desktop.h / 2 && h.t < g.dist + 0.3) {
textSize(p, -p.zoom.steps);
return setTimeout(() => removePanel(p), 150 + Math.abs(p.zoom.steps) * 90 + 200);
}
}
if (!p.zoom.base) p.zoom.base = V.len(V.sub(p.pos, headNow.pos)); // where it was pulled out to
else autoTextSize(p);
}
function thumbstick(pt, x, y, dt) {
if (pt.grab) {
const g = pt.grab;
g.dist = Math.min(6, Math.max(0.3, g.dist - y * dt * 1.5));
const s = 1 + x * dt * 0.8;
g.panel.w = Math.min(4, Math.max(0.2, g.panel.w * s));
g.panel.h = g.panel.w * (g.panel.aspect || 0.6);
return;
}
const hit = pt.hit;
if (!hit || hit.part !== "content" || (Math.abs(y) < 0.2 && Math.abs(x) < 0.2)) return;
const now = performance.now();
if (now - pt.lastWheel < 16) return;
pt.lastWheel = now;
hit.panel.stream.wheel(x * 24, y * 24, hit.u, hit.v);
}
// Pulling a window out of the desktop: finish the Mac-side drag, open the
// window as a panel of its own (on its own display on the Mac), and let it
// follow the pointer until the trigger is released.
function detach(pt, press, head) {
pt.press = null;
desktop.stream.up(press.last.u, press.last.v, 0);
const win = press.win, d = mainDisplay;
const scale = desktop.w / d.w; // metres per Mac point, as on the desktop panel
const p = openPanel(`separate:${win.id}`, { kind: "window", title: win.title || win.app, w: win.w * scale, h: (win.h + 40) * scale });
p.aspect = p.h / p.w;
const dist = Math.max(0.5, (hitPanel(desktop, pt.origin, pt.dir) || { t: 1.2 }).t - 0.15);
const at = V.add(pt.origin, V.scale(pt.dir, dist));
placeFacing(p, V.add(at, [0, -p.h / 2 + 0.03, 0]), head.pos);
pt.grab = { panel: p, dist, offset: V.sub(p.pos, at), byTrigger: true };
}
// ---------- rendering ----------
let gl = null, prog = null, quad = null, loc = {}, isGL2 = false, aniso = null;
const VS = `attribute vec2 a; uniform mat4 mvp; uniform vec2 size; varying vec2 uv; varying vec2 pm;
void main() { uv = vec2(a.x + 0.5, 0.5 - a.y); pm = a * size; gl_Position = mvp * vec4(a * size, 0.0, 1.0); }`;
const FS = `precision highp float; varying vec2 uv; varying vec2 pm; uniform sampler2D tex; uniform vec4 color;
uniform float useTex; uniform vec2 size; uniform float radius;
void main() {
vec2 d = abs(pm) - (size * 0.5 - vec2(radius));
float dist = length(max(d, 0.0)) - radius;
float a = clamp(0.5 - dist / fwidth(dist), 0.0, 1.0);
vec4 c = useTex > 0.5 ? texture2D(tex, uv) : color;
gl_FragColor = vec4(c.rgb, c.a * a * color.a);
}`;
function initGL(canvas, xrCompatible) {
// WebGL 2: mipmaps for textures of any size, so a panel smaller than its
// pixels in view is filtered down cleanly instead of shimmering.
gl = canvas.getContext("webgl2", { alpha: true, antialias: true, premultipliedAlpha: false, xrCompatible })
|| canvas.getContext("webgl", { alpha: true, antialias: true, premultipliedAlpha: false, xrCompatible });
isGL2 = typeof WebGL2RenderingContext !== "undefined" && gl instanceof WebGL2RenderingContext;
if (!isGL2) gl.getExtension("OES_standard_derivatives");
aniso = gl.getExtension("EXT_texture_filter_anisotropic");
const sh = (type, src) => {
const s = gl.createShader(type);
gl.shaderSource(s, type === gl.FRAGMENT_SHADER && !isGL2 ? "#extension GL_OES_standard_derivatives : enable\n" + src : src);
gl.compileShader(s);
if (!gl.getShaderParameter(s, gl.COMPILE_STATUS)) throw new Error(gl.getShaderInfoLog(s));
return s;
};
// WebGL 2 needs GLSL ES 3.00 for fwidth (the rounded corners).
const v3 = src => "#version 300 es\n" + src.replace(/attribute /g, "in ").replace(/varying /g, "out ");
const f3 = src => "#version 300 es\n" + src.replace(/varying /g, "in ").replace(/texture2D\(/g, "texture(")
.replace("void main() {", "out vec4 fragColor;\n void main() {").replace(/gl_FragColor/g, "fragColor");
prog = gl.createProgram();
gl.attachShader(prog, sh(gl.VERTEX_SHADER, isGL2 ? v3(VS) : VS));
gl.attachShader(prog, sh(gl.FRAGMENT_SHADER, isGL2 ? f3(FS) : FS));
gl.linkProgram(prog);
if (!gl.getProgramParameter(prog, gl.LINK_STATUS)) throw new Error(gl.getProgramInfoLog(prog));
for (const n of ["mvp", "size", "tex", "color", "useTex", "radius"]) loc[n] = gl.getUniformLocation(prog, n);
quad = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, quad);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([-0.5, -0.5, 0.5, -0.5, -0.5, 0.5, 0.5, 0.5]), gl.STATIC_DRAW);
}
function upload(p) {
const f = p.pending;
if (!f) return null;
p.pending = null;
if (!p.tex) {
p.tex = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, p.tex);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, isGL2 ? gl.LINEAR_MIPMAP_LINEAR : gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
if (aniso) gl.texParameterf(gl.TEXTURE_2D, aniso.TEXTURE_MAX_ANISOTROPY_EXT, Math.min(8, gl.getParameter(aniso.MAX_TEXTURE_MAX_ANISOTROPY_EXT)));
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
}
gl.bindTexture(gl.TEXTURE_2D, p.tex);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, f.img);
if (isGL2) gl.generateMipmap(gl.TEXTURE_2D);
f.img.close();
// Keep the panel's shape the picture's shape (a separated window can change size).
p.aspect = f.h / f.w;
p.h = p.w * p.aspect;
p.texW = f.w;
p.texH = f.h;
return { seq: f.seq, decodedAt: f.decodedAt, drawnAt: performance.now() };
}
function drawRect(vp, model, w, h, color, tex, radius) {
gl.uniformMatrix4fv(loc.mvp, false, M.mul(vp, model));
gl.uniform2f(loc.size, w, h);
gl.uniform4fv(loc.color, color);
gl.uniform1f(loc.useTex, tex ? 1 : 0);
gl.uniform1f(loc.radius, radius);
if (tex) gl.bindTexture(gl.TEXTURE_2D, tex);
gl.drawArrays(gl.TRIANGLE_STRIP, 0, 4);
}
function drawScene(vp) {
gl.useProgram(prog);
gl.bindBuffer(gl.ARRAY_BUFFER, quad);
const a = gl.getAttribLocation(prog, "a");
gl.enableVertexAttribArray(a);
gl.vertexAttribPointer(a, 2, gl.FLOAT, false, 0, 0);
gl.uniform1i(loc.tex, 0);
gl.enable(gl.BLEND);
gl.blendFunc(gl.SRC_ALPHA, gl.ONE_MINUS_SRC_ALPHA);
gl.enable(gl.DEPTH_TEST);
const hovered = new Set([...pointers.values()].filter(p => p.hit).map(p => p.hit.panel.src + ":" + p.hit.part));
for (const p of panels) {
const m = matrixOf(p);
// A soft shadow behind, then the picture (grey until the first frame).
drawRect(vp, M.mul(m, M.panel([0.004, -0.006, -0.004], 0)), p.w + 0.012, p.h + 0.012, [0, 0, 0, 0.35], null, 0.022);
drawRect(vp, m, p.w, p.h, p.tex ? [1, 1, 1, 1] : [0.18, 0.19, 0.22, 0.95], p.tex, 0.016);
const b = barOf(p), c = closeOf(p), bh = hovered.has(p.src + ":bar"), ch = hovered.has(p.src + ":close");
drawRect(vp, M.mul(m, M.panel([b.cx, b.cy, 0], 0)), b.w, b.h, [1, 1, 1, bh ? 0.95 : 0.55], null, b.h / 2);
drawRect(vp, M.mul(m, M.panel([c.cx, c.cy, 0], 0)), c.r * 2, c.r * 2, ch ? [0.95, 0.3, 0.3, 0.95] : [1, 1, 1, 0.45], null, c.r);
if (p.kind === "window") {
for (const dir of [-1, 1]) {
const z = zoomOf(p, dir), on = hovered.has(p.src + (dir < 0 ? ":zoomOut" : ":zoomIn"));
const zm = M.mul(m, M.panel([z.cx, z.cy, 0], 0));
drawRect(vp, zm, z.r * 2, z.r * 2, [1, 1, 1, on ? 0.9 : 0.45], null, z.r);
const glyph = M.mul(zm, M.panel([0, 0, 0.001], 0));
drawRect(vp, glyph, z.r * 1.1, z.r * 0.22, [0.1, 0.1, 0.12, 1], null, 0);
if (dir > 0) drawRect(vp, glyph, z.r * 0.22, z.r * 1.1, [0.1, 0.1, 0.12, 1], null, 0);
}
}
}
// Rays, ending where they hit.
for (const pt of pointers.values()) {
const len = pt.hit ? pt.hit.t : 2;
const mid = V.add(pt.origin, V.scale(pt.dir, len / 2));
const yaw = Math.atan2(pt.dir[0], pt.dir[2]);
// A thin strip along the ray: x across (horizontal), y along.
const along = pt.dir, across = V.norm([Math.cos(yaw), 0, -Math.sin(yaw)]);
const normal = [along[1] * across[2] - along[2] * across[1], along[2] * across[0] - along[0] * across[2], along[0] * across[1] - along[1] * across[0]];
const model = new Float32Array([across[0], across[1], across[2], 0, along[0], along[1], along[2], 0,
normal[0], normal[1], normal[2], 0, mid[0], mid[1], mid[2], 1]);
drawRect(vp, model, 0.003, len, [1, 1, 1, 0.35], null, 0);
if (pt.hit) {
const m = M.mul(matrixOf(pt.hit.panel), M.panel([pt.hit.x, pt.hit.y, 0.002], 0));
drawRect(vp, m, 0.012, 0.012, [1, 1, 1, 0.9], null, 0.006);
}
}
}
function flushMoves() {
for (const p of panels) if (p.pendingMove && performance.now() - p.lastMove >= 8) sendMove(p, p.pendingMove.u, p.pendingMove.v);
}
// ---------- WebXR ----------
let xrSession = null, xrRef = null;
async function enterXR() {
const mode = (await navigator.xr.isSessionSupported("immersive-ar").catch(() => false)) ? "immersive-ar" : "immersive-vr";
xrSession = await navigator.xr.requestSession(mode, { optionalFeatures: ["local-floor", "hand-tracking"] });
const canvas = document.createElement("canvas");
initGL(canvas, true);
await gl.makeXRCompatible();
xrSession.updateRenderState({ baseLayer: new XRWebGLLayer(xrSession, gl, { alpha: true, antialias: true }) });
xrRef = await xrSession.requestReferenceSpace("local-floor").catch(() => xrSession.requestReferenceSpace("local"));
const on = (type, fn) => xrSession.addEventListener(type, ev => fn(pointerState(ev.inputSource), headNow));
on("selectstart", selectStart);
on("selectend", pt => selectEnd(pt));
on("squeezestart", squeezeStart);
on("squeezeend", pt => squeezeEnd(pt));
xrSession.addEventListener("end", () => { xrSession = null; $("ui").hidden = false; $("enter").disabled = false; say("Left the Mac space."); });
$("ui").hidden = true;
let placed = false, last = 0;
const frame = (t, xf) => {
if (!xrSession) return;
xrSession.requestAnimationFrame(frame);
const pose = xf.getViewerPose(xrRef);
if (!pose) return;
const hm = pose.transform.matrix;
headNow = { pos: [hm[12], hm[13], hm[14]], forward: V.norm([-hm[8], 0, -hm[10]]) };
if (!placed && desktop) { restoreDesktop(headNow); placed = true; }
const dt = last ? Math.min(0.1, (t - last) / 1000) : 0;
last = t;
for (const src of xrSession.inputSources) {
if (!src.targetRaySpace) continue;
const rp = xf.getPose(src.targetRaySpace, xrRef);
if (!rp) continue;
const m = rp.transform.matrix, pt = pointerState(src);
steady(pt, [m[12], m[13], m[14]], V.norm([-m[8], -m[9], -m[10]]), t);
pointerMoved(pt, headNow);
const axes = src.gamepad && src.gamepad.axes;
if (axes && axes.length >= 4) thumbstick(pt, axes[2], axes[3], dt);
}
for (const k of [...pointers.keys()]) if (typeof k === "object" && ![...xrSession.inputSources].includes(k)) pointers.delete(k);
flushMoves();
const shown = [];
for (const p of panels) { const f = upload(p); if (f) shown.push([p, f]); }
const layer = xrSession.renderState.baseLayer;
gl.bindFramebuffer(gl.FRAMEBUFFER, layer.framebuffer);
gl.clearColor(0, 0, 0, 0); // transparent: passthrough shows through
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
for (const view of pose.views) {
const vp = layer.getViewport(view);
gl.viewport(vp.x, vp.y, vp.width, vp.height);
drawScene(M.mul(view.projectionMatrix, view.transform.inverse.matrix));
}
const now = performance.now();
for (const [p, f] of shown) p.stream.presented(f.seq, f.decodedAt, f.drawnAt, now);
};
xrSession.requestAnimationFrame(frame);
}
let headNow = { pos: [0, 1.6, 0], forward: [0, 0, -1] };
// ---------- preview in a normal page (no headset) ----------
function preview() {
const canvas = $("gl");
initGL(canvas, false);
$("hint").hidden = false;
const head = { pos: [0, 1.6, 0.4], forward: [0, 0, -1] };
headNow = head;
const mouse = pointerState("mouse");
let placed = false;
const toRay = e => {
const r = canvas.getBoundingClientRect(), x = (e.clientX - r.left) / r.width * 2 - 1, y = 1 - (e.clientY - r.top) / r.height * 2;
const f = 1 / Math.tan(Math.PI / 6), aspect = r.width / r.height;
mouse.origin = head.pos.slice();
mouse.dir = V.norm([x * aspect / f, y / f, -1]);
pointerMoved(mouse, head);
};
canvas.addEventListener("pointermove", toRay);
canvas.addEventListener("pointerdown", e => {
toRay(e);
canvas.setPointerCapture(e.pointerId);
if (e.button === 2 || e.shiftKey) squeezeStart(mouse, head); else selectStart(mouse, head);
});
canvas.addEventListener("pointerup", e => {
toRay(e);
if (mouse.grab && !mouse.grab.byTrigger) squeezeEnd(mouse); else selectEnd(mouse);
});
canvas.addEventListener("contextmenu", e => e.preventDefault());
canvas.addEventListener("wheel", e => {
e.preventDefault();
if (mouse.grab) { thumbstick(mouse, 0, -Math.sign(e.deltaY) * 0.5, 0.1); return; }
if (mouse.hit && mouse.hit.part === "content") mouse.hit.panel.stream.wheel(e.deltaX, e.deltaY, mouse.hit.u, mouse.hit.v);
}, { passive: false });
const loop = () => {
requestAnimationFrame(loop);
if (!placed && desktop) { restoreDesktop(head); placed = true; }
const w = canvas.clientWidth, h = canvas.clientHeight;
if (canvas.width !== w * devicePixelRatio) { canvas.width = w * devicePixelRatio; canvas.height = h * devicePixelRatio; }
flushMoves();
const shown = [];
for (const p of panels) { const f = upload(p); if (f) shown.push([p, f]); }
gl.bindFramebuffer(gl.FRAMEBUFFER, null);
gl.viewport(0, 0, canvas.width, canvas.height);
gl.clearColor(0.06, 0.07, 0.09, 1);
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
const view = M.invRigid(M.panel(head.pos, 0));
drawScene(M.mul(M.perspective(Math.PI / 3, w / h, 0.05, 50), view));
for (const [p, f] of shown) p.stream.drawn(f.seq, f.decodedAt, f.drawnAt);
};
loop();
}
function leave() {
for (const p of [...panels]) removePanel(p);
if (xrSession) xrSession.end();
}
// ---------- start ----------
(async () => {
const ticket = q.get("t") || "";
try {
const r = await fetch(`/space/key?t=${encodeURIComponent(ticket)}`);
if (!r.ok) throw new Error(r.status === 403 ? "This link has expired. Open the Mac space again from Frame Control." : `HTTP ${r.status}`);
spaceKey = (await r.json()).key;
} catch (e) { say(String(e.message || e)); $("enter").textContent = "Can't connect"; return; }
await refreshLayout();
setInterval(refreshLayout, 2000);
mainDisplay = layout.displays.find(d => d.main) || layout.displays[0] || null;
const src = q.get("src") || (mainDisplay ? `display:${mainDisplay.id}` : "test");
desktop = openPanel(src, { kind: "desktop", w: 1.25, h: mainDisplay ? 1.25 * mainDisplay.h / mainDisplay.w : 0.7, max: "2560" });
desktop.aspect = desktop.h / desktop.w;
// For tests over DevTools: drive a pointer the way a controller would.
window.space = { panels, pointers, layout: () => layout, desktop: () => desktop, head: () => headNow, V, M, matrixOf,
pointerState, pointerMoved, selectStart, selectEnd, squeezeStart, squeezeEnd, mainDisplay: () => mainDisplay, hooks };
const xr = navigator.xr && await navigator.xr.isSessionSupported("immersive-vr").catch(() => false);
if (!xr || q.get("preview") === "1") {
$("ui").hidden = true;
return preview();
}
$("enter").disabled = false;
$("enter").textContent = "Enter Mac space";
$("enter").onclick = () => enterXR().catch(e => { say(`Couldn't start: ${e.message || e}`); });
})();
</script>
</body>
</html>
+321
View File
@@ -0,0 +1,321 @@
// Frame Control: one stream from the Mac's frame-mac-view agent, for the
// viewer pages (mac-view.html: one panel; mac-space.html: the Mac space).
// It connects over the agent's WebSocket, keeps a clock synced with the
// Mac's, decodes H.264 with WebCodecs (or JPEG), and acknowledges and
// reports on every frame: the agent paces itself on the acks
// (Controller.swift) and measures each frame's journey from the reports
// (Stats.swift). Input goes back the same way. Plain script, no modules:
// it defines window.MacStream.
"use strict";
// What this browser draws with, and how often it gives an animation frame
// when nothing else is going on (only for the numbers).
const MacEnv = (() => {
const env = { gl: "", rafHz: 0, decoder: "" };
try {
const gl = document.createElement("canvas").getContext("webgl");
const ext = gl && gl.getExtension("WEBGL_debug_renderer_info");
env.gl = ext ? gl.getParameter(ext.UNMASKED_RENDERER_WEBGL) : gl ? "webgl" : "no webgl";
} catch (e) { env.gl = "?"; }
let n = 0, t0 = 0;
const tick = t => {
if (!t0) t0 = t;
if (t - t0 < 1000) { n++; return requestAnimationFrame(tick); }
env.rafHz = Math.round(n * 1000 / (t - t0));
};
requestAnimationFrame(tick);
return env;
})();
async function macPickCodec(want) {
if (want === "jpeg" || !("VideoDecoder" in window)) return "jpeg";
try {
const r = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", optimizeForLatency: true });
if (!r.supported) return "jpeg";
// Whether this browser has a hardware H.264 decoder (for the numbers).
const hw = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", hardwareAcceleration: "prefer-hardware" })
.catch(() => ({ supported: false }));
MacEnv.decoder = hw.supported ? "h264 hardware" : "h264 software";
return "h264";
} catch (e) { return "jpeg"; }
}
function macStartCode(b, i) { return b[i] === 0 && b[i + 1] === 0 && (b[i + 2] === 1 || (b[i + 2] === 0 && b[i + 3] === 1)); }
function macSpsCodec(au) {
for (let i = 0; i + 7 < au.length; i++) {
if (!macStartCode(au, i)) continue;
const n = au[i + 2] === 1 ? i + 3 : i + 4;
if ((au[n] & 0x1f) === 7) return "avc1." + [au[n + 1], au[n + 2], au[n + 3]].map(b => b.toString(16).padStart(2, "0")).join("");
i = n;
}
return "";
}
class MacStream {
// opts: src; one of ticket (single use, from Frame Control) or space (a
// Mac space key); codec, max, fps, bpp (optional); and callbacks:
// onFrame(image, width, height, seq, decodedAt): a VideoFrame or
// ImageBitmap, now the caller's to draw and close(). Call drawn() or
// presented() once it's on screen.
// onInfo(m), onStatus(text, sticky), onError(message), onClosed(reason),
// onStats(m), onBench(m)
constructor(opts) {
this.o = opts;
this.src = opts.src || "test";
this.stats = { frames: 0, keyFrames: 0, bytes: 0, dropped: 0, codec: "", errors: [], info: null };
this.clock = { off: null, rtt: 0, samples: [], reported: 0 };
this.reconnectKey = "";
this.ws = null;
this.dec = null;
this.codec = "";
this.codecString = "";
this.needKey = true;
this.closing = false;
this.retry = 0;
this.failedStarts = 0;
this.lastError = "";
this.lastKeyAsk = 0;
this.reports = [];
this.dropsToReport = 0;
this.shownPending = null;
this.rafQueued = false;
this.inputId = 0;
this.warmMs = 0;
this.warmUntil = 0;
this.warmTimer = 0;
this.jpegBusy = false;
this.pingTimer = 0;
// Decoded, drawn and shown times go back in batches.
this.reportTimer = setInterval(() => {
if (!this.reports.length && !this.dropsToReport) return;
this.send({ t: "fd", f: this.reports, drop: this.dropsToReport });
this.reports = [];
this.dropsToReport = 0;
}, 250);
}
// ---- clock: the Mac measures each frame's journey on its own clock ----
// NTP-style: the Mac's time minus ours, from the ping with the shortest
// round trip lately (the least queueing, so the least error).
toMac(ms) { return this.clock.off === null ? null : Math.round(ms * 1000 + this.clock.off); }
onPong(m) {
const now = performance.now(), rtt = now - m.c, clock = this.clock;
clock.samples.push({ rtt, off: m.a - (m.c + now) / 2 * 1000 });
if (clock.samples.length > 16) clock.samples.shift();
const best = clock.samples.reduce((a, b) => (b.rtt < a.rtt ? b : a));
clock.off = best.off;
clock.rtt = best.rtt;
if (now - clock.reported > 1000) {
clock.reported = now;
this.send({ t: "clock", rtt: +best.rtt.toFixed(2), dec: `${MacEnv.decoder}; ${MacEnv.gl}; raf ${MacEnv.rafHz} Hz` });
}
}
startPings() {
clearInterval(this.pingTimer);
this.clock.samples = [];
for (let i = 0; i < 10; i++) setTimeout(() => this.send({ t: "ping", c: performance.now() }), i * 40);
this.pingTimer = setInterval(() => this.send({ t: "ping", c: performance.now() }), 1000);
}
// ---- reports ----
dropped() { this.stats.dropped++; this.dropsToReport++; }
// A frame drawn on a page: it's on screen from the next animation frame,
// unless another replaces it first (then it was never seen).
drawn(seq, decodedAt, drawnAt = performance.now()) {
this.stats.frames++;
if (this.shownPending) {
const p = this.shownPending;
this.reports.push([p.seq, this.toMac(p.dec), this.toMac(p.drawn), 0]);
}
this.shownPending = { seq, dec: decodedAt, drawn: drawnAt };
if (this.rafQueued) return;
this.rafQueued = true;
requestAnimationFrame(() => {
this.rafQueued = false;
const p = this.shownPending, now = performance.now();
this.shownPending = null;
if (p && this.clock.off !== null) this.reports.push([p.seq, this.toMac(p.dec), this.toMac(p.drawn), this.toMac(now)]);
});
}
// A decoded frame a newer one replaced before it was drawn.
replaced(seq, decodedAt) {
if (this.clock.off !== null) this.reports.push([seq, this.toMac(decodedAt), 0, 0]);
}
// A frame drawn into a WebXR frame: shown when that frame is.
presented(seq, decodedAt, drawnAt, shownAt) {
this.stats.frames++;
if (this.clock.off !== null) this.reports.push([seq, this.toMac(decodedAt), this.toMac(drawnAt), this.toMac(shownAt)]);
}
// ---- connection ----
send(obj) { if (this.ws && this.ws.readyState === 1) this.ws.send(JSON.stringify(obj)); }
askKeyFrame() {
const now = performance.now();
if (now - this.lastKeyAsk < 500) return;
this.lastKeyAsk = now;
this.send({ t: "key-frame" });
}
status(text, sticky) { if (this.o.onStatus) this.o.onStatus(text, sticky); }
async start() {
this.codec = await macPickCodec(this.o.codec);
this.stats.codec = this.codec;
this.connect();
}
connect() {
if (this.codec === "h264") this.makeDecoder();
this.needKey = true;
const o = this.o;
const p = new URLSearchParams({ src: this.src, codec: this.codec, max: o.max || "1920", fps: o.fps || "60" });
if (this.reconnectKey) p.set("r", this.reconnectKey);
else if (o.space) p.set("s", o.space);
else p.set("t", o.ticket || "");
if (o.bpp) p.set("bpp", o.bpp);
const ws = this.ws = new WebSocket(`ws://${location.host}/stream?${p}`);
ws.binaryType = "arraybuffer";
ws.onopen = () => { this.retry = 0; this.lastError = ""; this.startPings(); };
ws.onmessage = ev => {
if (typeof ev.data === "string") return this.control(JSON.parse(ev.data));
const now = performance.now(), b = new Uint8Array(ev.data);
this.stats.bytes += b.length;
if (b.length < 18) return;
// flags (1 = keyframe), pts µs, sequence number, input echoed; big-endian.
const v = new DataView(ev.data), isKey = (b[0] & 1) === 1, seq = v.getUint32(9);
this.send({ t: "rx", s: seq, r: this.toMac(now) });
const payload = b.subarray(17);
if (this.codec === "h264") this.onH264(isKey, seq, payload); else this.onJPEG(seq, payload);
};
ws.onclose = () => {
if (this.ws === ws) this.ws = null;
clearInterval(this.pingTimer);
if (this.closing) return;
// Refused before ever getting a key: the ticket expired or was revoked,
// and retrying can't help.
if (!this.reconnectKey && ++this.failedStarts >= 3) {
this.status("This view has expired. Press Show in Frame Control on the Mac to open it again.", true);
if (o.onClosed) o.onClosed("expired");
return;
}
// The Mac may be asleep or the tunnel restarting: keep trying.
this.retry = Math.min(this.retry + 1, 6);
// Keep the Mac's reason (a missing permission, a closed window) on screen.
this.status(this.lastError ? `${this.lastError} Retrying…` : "Lost the Mac. Reconnecting…", true);
setTimeout(() => { if (!this.closing) this.connect(); }, 500 * 2 ** this.retry);
};
}
control(m) {
const o = this.o;
if (m.t === "pong") return this.onPong(m);
if (m.t === "stats") return o.onStats && o.onStats(m);
if (m.t === "bench") return o.onBench && o.onBench(m);
if (m.t === "hello") {
this.reconnectKey = m.r;
this.send({ t: "ack" }); // the ticket is spent only now
} else if (m.t === "info") {
this.stats.info = m;
this.warmMs = Math.max(0, +m.warm || 0);
if (o.onInfo) o.onInfo(m);
} else if (m.t === "error") {
this.stats.errors.push(m.message);
this.lastError = m.message.replace(/\.?$/, ".");
if (o.onError) o.onError(m.message);
} else if (m.t === "close" || m.t === "closed") {
this.stop();
if (o.onClosed) o.onClosed(m.reason || "");
}
}
stop() {
this.closing = true;
clearInterval(this.pingTimer);
clearInterval(this.reportTimer);
clearInterval(this.warmTimer);
if (this.ws) try { this.ws.close(); } catch (e) {}
if (this.dec) try { this.dec.close(); } catch (e) {}
}
// ---- H.264 ----
makeDecoder() {
if (this.dec) try { this.dec.close(); } catch (e) {}
this.codecString = "";
this.dec = new VideoDecoder({
// The chunk's timestamp is the Mac's sequence number, to match reports up.
output: frame => {
if (this.closing) return frame.close();
this.o.onFrame(frame, frame.displayWidth, frame.displayHeight, frame.timestamp, performance.now());
},
error: e => {
this.stats.errors.push(String(e.message || e));
this.needKey = true;
if (this.closing) return;
this.makeDecoder();
this.askKeyFrame();
},
});
}
onH264(isKey, seq, au) {
const dec = this.dec;
if (isKey) {
const cs = macSpsCodec(au);
if (cs && (cs !== this.codecString || dec.state !== "configured")) {
if (dec.state === "closed") this.makeDecoder();
this.dec.configure({ codec: cs, optimizeForLatency: true, hardwareAcceleration: "no-preference" });
this.codecString = this.stats.codec = cs;
}
this.stats.keyFrames++;
}
if (this.dec.state !== "configured" || (this.needKey && !isKey)) { this.dropped(); this.askKeyFrame(); return; }
// Far behind: skip to the next keyframe rather than show old pictures late.
// A few queued frames are fine: decoding catches up faster than a keyframe
// crosses a slow link, and only the newest decoded frame gets drawn.
if (!isKey && this.dec.decodeQueueSize > 10) { this.needKey = true; this.dropped(); this.askKeyFrame(); return; }
this.needKey = false;
this.dec.decode(new EncodedVideoChunk({ type: isKey ? "key" : "delta", timestamp: seq, data: au }));
}
// ---- JPEG ----
onJPEG(seq, data) {
if (this.jpegBusy) { this.dropped(); return; }
this.jpegBusy = true;
createImageBitmap(new Blob([data], { type: "image/jpeg" }))
.then(bmp => { if (this.closing) bmp.close(); else this.o.onFrame(bmp, bmp.width, bmp.height, seq, performance.now()); })
.catch(e => this.stats.errors.push(String(e)))
.finally(() => { this.jpegBusy = false; });
}
// ---- input ----
// x, y are 0..1 across the picture. Discrete input carries an id and when
// it happened (the Mac's clock), so the Mac can tag the first frame that
// could show its effect.
stamp(m, at) {
this.keepWarm();
m.i = ++this.inputId;
m.tv = this.toMac(at || performance.now());
return m;
}
// Keep the Frame's Wi-Fi awake for a few seconds after input, when the agent asks.
keepWarm() {
if (!this.warmMs) return;
this.warmUntil = performance.now() + 5000;
if (this.warmTimer) return;
this.warmTimer = setInterval(() => {
if (performance.now() > this.warmUntil) { clearInterval(this.warmTimer); this.warmTimer = 0; return; }
this.send({ t: "w" });
}, this.warmMs);
}
move(x, y) { this.send({ t: "m", e: "move", x, y }); }
down(x, y, button = 0, at) { this.send(this.stamp({ t: "m", e: "down", b: button, x, y }, at)); }
up(x, y, button = 0) { this.send({ t: "m", e: "up", b: button, x, y }); }
wheel(dx, dy, x, y, at) { this.send(this.stamp({ t: "wheel", dx, dy, x, y }, at)); }
key(down, code, key, mods, at) {
const m = { t: "k", e: down ? "down" : "up", code, key, mods };
this.send(down ? this.stamp(m, at) : m);
}
text(s, at) { this.send(this.stamp({ t: "text", s }, at)); }
release() { this.send({ t: "release" }); }
focus() { this.send({ t: "focus" }); }
}
window.MacStream = MacStream;
window.MacEnv = MacEnv;
+47 -265
View File
@@ -24,83 +24,22 @@
<canvas id="c" width="16" height="9"></canvas>
<div id="note">Connecting to your computer…</div>
<div id="hud" hidden></div>
<script src="/mac-stream.js"></script>
<script>
"use strict";
const q = new URLSearchParams(location.search);
// t: a single-use ticket from Frame Control. After the first connection the
// Mac sends a reconnect key, kept only in memory.
const src = q.get("src") || "test", ticket = q.get("t") || "", tag = q.get("tag") || "";
let reconnectKey = "";
// Mac sends a reconnect key, which the stream keeps in memory only.
const src = q.get("src") || "test", tag = q.get("tag") || "";
const c = document.getElementById("c"), ctx = c.getContext("2d", { alpha: false, desynchronized: true });
const note = document.getElementById("note");
// Counters for Frame Control's tests (read over DevTools).
const stats = window.stats = { frames: 0, keyFrames: 0, bytes: 0, dropped: 0, codec: "", errors: [], info: null };
// ---- timing: the Mac measures each frame's journey on its own clock ----
// Clock sync, NTP-style: the Mac's time minus ours, from the ping with the
// shortest round trip lately (the least queueing, so the least error).
const clock = { off: null, rtt: 0, samples: [], reported: 0 };
function toMac(ms) { return clock.off === null ? null : Math.round(ms * 1000 + clock.off); }
function onPong(m) {
const now = performance.now(), rtt = now - m.c;
clock.samples.push({ rtt, off: m.a - (m.c + now) / 2 * 1000 });
if (clock.samples.length > 16) clock.samples.shift();
const best = clock.samples.reduce((a, b) => (b.rtt < a.rtt ? b : a));
clock.off = best.off;
clock.rtt = best.rtt;
if (now - clock.reported > 1000) {
clock.reported = now;
send({ t: "clock", rtt: +best.rtt.toFixed(2), dec: `${decoderInfo}; ${glInfo}; raf ${rafHz} Hz` });
}
let noteTimer = 0, hideNoteOnFrame = true; // "Connecting…"/"Reconnecting…" go once video flows again
function show(text, ms) {
note.textContent = text;
note.hidden = false;
clearTimeout(noteTimer);
if (ms) noteTimer = setTimeout(() => { note.hidden = true; }, ms);
}
let pingTimer = 0;
function startPings() {
clearInterval(pingTimer);
clock.samples = [];
for (let i = 0; i < 10; i++) setTimeout(() => send({ t: "ping", c: performance.now() }), i * 40);
pingTimer = setInterval(() => send({ t: "ping", c: performance.now() }), 1000);
}
// Decoded, drawn and next-animation-frame times go back in batches.
let reports = [], dropsToReport = 0, shownPending = null, rafQueued = false;
setInterval(() => {
if (!reports.length && !dropsToReport) return;
send({ t: "fd", f: reports, drop: dropsToReport });
reports = [];
dropsToReport = 0;
}, 250);
function dropped() { stats.dropped++; dropsToReport++; }
// A drawn frame is on screen from the next animation frame, unless another
// replaces it first (then it was never seen).
function shown(seq, decodedAt, drawnAt) {
if (shownPending) reports.push([shownPending.seq, toMac(shownPending.dec), toMac(shownPending.drawn), 0]);
shownPending = { seq, dec: decodedAt, drawn: drawnAt };
if (rafQueued) return;
rafQueued = true;
requestAnimationFrame(() => {
rafQueued = false;
const p = shownPending, now = performance.now();
shownPending = null;
if (p && clock.off !== null) reports.push([p.seq, toMac(p.dec), toMac(p.drawn), toMac(now)]);
});
}
let decoderInfo = "";
// What this browser draws with, and how often it gives an animation frame
// when nothing else is going on (both only for the numbers).
let glInfo = "", rafHz = 0;
try {
const gl = document.createElement("canvas").getContext("webgl");
const ext = gl && gl.getExtension("WEBGL_debug_renderer_info");
glInfo = ext ? gl.getParameter(ext.UNMASKED_RENDERER_WEBGL) : gl ? "webgl" : "no webgl";
} catch (e) { glInfo = "?"; }
(function measureRaf() {
let n = 0, t0 = 0;
const tick = t => {
if (!t0) t0 = t;
if (t - t0 < 1000) { n++; return requestAnimationFrame(tick); }
rafHz = Math.round(n * 1000 / (t - t0));
};
requestAnimationFrame(tick);
})();
const hud = document.getElementById("hud");
function showHud(on) { hud.hidden = !on; }
function ms(o) { return o && o.p50 !== undefined ? `${o.p50}/${o.p95}` : "–"; }
@@ -111,177 +50,41 @@ function onStats(m) {
`\nlatency p50/p95 ms total ${ms(m.total)}\n capture ${ms(m.capture)} queue ${ms(m.queue)} encode ${ms(m.encode)}` +
`\n network ${ms(m.network)} decode ${ms(m.decode)} draw ${ms(m.draw)}` +
`\ninput→shown p50 ${m.input ? m.input.p50.toFixed(0) + " ms" : "–"} rtt ${m.rtt} ms` +
`\nskipped ${m.skipped}/${m.captured} dropped ${m.dropped} ${decoderInfo}`;
`\nskipped ${m.skipped}/${m.captured} dropped ${m.dropped} ${MacEnv.decoder}`;
}
showHud(q.get("stats") === "1");
// Frame Control finds this window on the Frame by the tag in its title.
document.title = tag ? `Mac [${tag}]` : "Mac";
let failedStarts = 0;
let ws = null, dec = null, codecString = "", needKey = true, closing = false, retry = 0, noteTimer = 0, lastError = "";
function show(text, ms) {
note.textContent = text;
note.hidden = false;
clearTimeout(noteTimer);
if (ms) noteTimer = setTimeout(() => { note.hidden = true; }, ms);
}
function send(obj) { if (ws && ws.readyState === 1) ws.send(JSON.stringify(obj)); }
let lastKeyAsk = 0;
function askKeyFrame() {
const now = performance.now();
if (now - lastKeyAsk < 500) return;
lastKeyAsk = now;
send({ t: "key-frame" });
}
let hideNoteOnFrame = true; // "Connecting…"/"Reconnecting…" go once video flows again
function drawFrame(img, w, h, seq, decodedAt) {
if (c.width !== w || c.height !== h) { c.width = w; c.height = h; }
ctx.drawImage(img, 0, 0);
shown(seq, decodedAt, performance.now());
stats.frames++;
if (hideNoteOnFrame) { hideNoteOnFrame = false; note.hidden = true; }
}
// ---- H.264 ----
function startCode(b, i) { return b[i] === 0 && b[i + 1] === 0 && (b[i + 2] === 1 || (b[i + 2] === 0 && b[i + 3] === 1)); }
function spsCodec(au) {
for (let i = 0; i + 7 < au.length; i++) {
if (!startCode(au, i)) continue;
const n = au[i + 2] === 1 ? i + 3 : i + 4;
if ((au[n] & 0x1f) === 7) return "avc1." + [au[n + 1], au[n + 2], au[n + 3]].map(b => b.toString(16).padStart(2, "0")).join("");
i = n;
}
return "";
}
function makeDecoder() {
if (dec) try { dec.close(); } catch (e) {}
codecString = "";
dec = new VideoDecoder({
// The chunk's timestamp is the Mac's sequence number, to match reports up.
output: frame => { drawFrame(frame, frame.displayWidth, frame.displayHeight, frame.timestamp, performance.now()); frame.close(); },
error: e => {
stats.errors.push(String(e.message || e));
needKey = true;
makeDecoder();
askKeyFrame();
},
});
}
function onH264(isKey, seq, au) {
if (isKey) {
const cs = spsCodec(au);
if (cs && (cs !== codecString || dec.state !== "configured")) {
if (dec.state === "closed") makeDecoder();
dec.configure({ codec: cs, optimizeForLatency: true, hardwareAcceleration: "no-preference" });
codecString = stats.codec = cs;
}
stats.keyFrames++;
}
if (dec.state !== "configured" || (needKey && !isKey)) { dropped(); askKeyFrame(); return; }
// Far behind: skip to the next keyframe rather than show old pictures late.
// A few queued frames are fine: decoding catches up faster than a keyframe
// crosses a slow link, and only the newest decoded frame gets drawn.
if (!isKey && dec.decodeQueueSize > 10) { needKey = true; dropped(); askKeyFrame(); return; }
needKey = false;
dec.decode(new EncodedVideoChunk({ type: isKey ? "key" : "delta", timestamp: seq, data: au }));
}
// ---- JPEG ----
let jpegBusy = false;
function onJPEG(seq, data) {
if (jpegBusy) { dropped(); return; }
jpegBusy = true;
createImageBitmap(new Blob([data], { type: "image/jpeg" })).then(bmp => {
drawFrame(bmp, bmp.width, bmp.height, seq, performance.now());
bmp.close();
}).catch(e => stats.errors.push(String(e))).finally(() => { jpegBusy = false; });
}
async function pickCodec() {
const want = q.get("codec");
if (want === "jpeg") return "jpeg";
if (!("VideoDecoder" in window)) return "jpeg";
try {
const r = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", optimizeForLatency: true });
if (!r.supported) return "jpeg";
// Whether this browser has a hardware H.264 decoder (for the numbers).
const hw = await VideoDecoder.isConfigSupported({ codec: "avc1.640028", hardwareAcceleration: "prefer-hardware" })
.catch(() => ({ supported: false }));
decoderInfo = hw.supported ? "h264 hardware" : "h264 software";
return "h264";
} catch (e) { return "jpeg"; }
}
async function connect() {
const codec = await pickCodec();
stats.codec = codec;
if (codec === "h264") makeDecoder();
needKey = true;
const p = new URLSearchParams({ src, codec, max: q.get("max") || "1920", fps: q.get("fps") || "60" });
if (reconnectKey) p.set("r", reconnectKey); else p.set("t", ticket);
if (q.get("bpp")) p.set("bpp", q.get("bpp"));
ws = new WebSocket(`ws://${location.host}/stream?${p}`);
ws.binaryType = "arraybuffer";
ws.onopen = () => { retry = 0; lastError = ""; startPings(); };
ws.onmessage = ev => {
if (typeof ev.data === "string") return control(JSON.parse(ev.data));
const now = performance.now(), b = new Uint8Array(ev.data);
stats.bytes += b.length;
if (b.length < 18) return;
// flags (1 = keyframe), pts µs, sequence number, input echoed; big-endian.
const v = new DataView(ev.data), isKey = (b[0] & 1) === 1, seq = v.getUint32(9);
send({ t: "rx", s: seq, r: toMac(now) });
const payload = b.subarray(17);
if (codec === "h264") onH264(isKey, seq, payload); else onJPEG(seq, payload);
};
ws.onclose = () => {
ws = null;
clearInterval(pingTimer);
if (closing) return;
// Refused before ever getting a key: the ticket expired or was revoked,
// and retrying can't help.
if (!reconnectKey && ++failedStarts >= 3) {
show("This view has expired. Press Show in Frame Control on your computer to open it again.");
return;
}
hideNoteOnFrame = true;
// The Mac may be asleep or the tunnel restarting: keep trying.
retry = Math.min(retry + 1, 6);
// Keep the Mac's reason (a missing permission, a closed window) on screen.
show(lastError ? `${lastError} Retrying…` : "Lost the computer. Reconnecting…");
setTimeout(connect, 500 * 2 ** retry);
};
}
function control(m) {
if (m.t === "pong") return onPong(m);
if (m.t === "stats") return onStats(m);
if (m.t === "bench") return bench(m);
if (m.t === "hello") {
reconnectKey = m.r;
send({ t: "ack" }); // the ticket is spent only now
} else if (m.t === "info") {
stats.info = m;
warmMs = Math.max(0, +m.warm || 0);
const stream = new MacStream({
src, ticket: q.get("t") || "", codec: q.get("codec"), max: q.get("max"), fps: q.get("fps"), bpp: q.get("bpp"),
onFrame(img, w, h, seq, decodedAt) {
if (c.width !== w || c.height !== h) { c.width = w; c.height = h; }
ctx.drawImage(img, 0, 0);
img.close();
stream.drawn(seq, decodedAt);
if (hideNoteOnFrame) { hideNoteOnFrame = false; note.hidden = true; }
},
onStatus(text) { hideNoteOnFrame = true; show(text); },
onInfo(m) {
const name = m.title && m.app && m.title !== m.app ? `${m.title} — ${m.app}` : (m.title || m.app || "Mac");
document.title = tag ? `${name} [${tag}]` : name;
if (!m.input) {
hideNoteOnFrame = false; // a warning, not a connection notice: let it stay its 8 s
show(m.inputMessage || "Clicks and keys need Accessibility permission on the Mac (Frame Control asks for it).", 8000);
}
} else if (m.t === "error") {
stats.errors.push(m.message);
lastError = m.message.replace(/\.?$/, ".");
show(m.message);
} else if (m.t === "close" || m.t === "closed") {
closing = true;
if (ws) ws.close();
show(m.reason ? `Stopped: ${m.reason}.` : "Stopped.");
},
onError(message) { show(message); },
onClosed(reason) {
if (reason === "expired") return;
show(reason ? `Stopped: ${reason}.` : "Stopped.");
window.close();
}
}
},
onStats,
onBench: bench,
});
// Counters for Frame Control's tests (read over DevTools).
window.stats = stream.stats;
// ---- input ----
// Where the picture sits inside the window (object-fit: contain letterboxes it).
@@ -291,39 +94,19 @@ function toPicture(e) {
const x = (e.clientX - left) / w, y = (e.clientY - top) / h;
return { x, y, inside: x >= 0 && x <= 1 && y >= 0 && y <= 1 };
}
// Discrete input carries an id and when it happened (the Mac's clock), so
// the Mac can tag the first frame that could show its effect.
let inputId = 0;
// Keep the Frame's Wi-Fi awake for a few seconds after input, when the agent asks.
let warmMs = 0, warmUntil = 0, warmTimer = 0;
function keepWarm() {
if (!warmMs) return;
warmUntil = performance.now() + 5000;
if (warmTimer) return;
warmTimer = setInterval(() => {
if (performance.now() > warmUntil) { clearInterval(warmTimer); warmTimer = 0; return; }
send({ t: "w" });
}, warmMs);
}
function stamp(m, e) {
keepWarm();
m.i = ++inputId;
m.tv = toMac(e && e.timeStamp ? e.timeStamp : performance.now());
return m;
}
// Moves go at most every 8 ms, on a timer rather than the next animation frame:
// the Frame throttles a panel's animation frames to 15-36 Hz when it thinks
// nobody is looking, which would hold a move back by up to 60 ms.
let pendingMove = null, moveQueued = false, lastMove = 0;
function flushMove() {
moveQueued = false;
if (pendingMove) { send(pendingMove); lastMove = performance.now(); }
if (pendingMove) { stream.move(pendingMove.x, pendingMove.y); lastMove = performance.now(); }
pendingMove = null;
}
addEventListener("pointermove", e => {
const p = toPicture(e);
if (!p.inside && !e.buttons) return;
pendingMove = { t: "m", e: "move", x: p.x, y: p.y };
pendingMove = p;
if (moveQueued) return;
const wait = lastMove + 8 - performance.now();
if (wait <= 0) return flushMove();
@@ -335,19 +118,19 @@ addEventListener("pointerdown", e => {
if (!p.inside) return;
if (e.pointerId !== undefined) try { c.setPointerCapture(e.pointerId); } catch (err) {}
pendingMove = null;
send(stamp({ t: "m", e: "down", b: e.button < 0 ? 0 : e.button, x: p.x, y: p.y }, e));
stream.down(p.x, p.y, e.button < 0 ? 0 : e.button, e.timeStamp);
e.preventDefault();
});
addEventListener("pointerup", e => {
const p = toPicture(e);
send({ t: "m", e: "up", b: e.button < 0 ? 0 : e.button, x: p.x, y: p.y });
stream.up(p.x, p.y, e.button < 0 ? 0 : e.button);
});
// Let go of any held button where the pointer is on the Mac, not at a corner.
addEventListener("pointercancel", () => send({ t: "release" }));
addEventListener("pointercancel", () => stream.release());
addEventListener("contextmenu", e => e.preventDefault());
addEventListener("wheel", e => {
const p = toPicture(e), unit = e.deltaMode === 1 ? 16 : e.deltaMode === 2 ? innerHeight : 1;
send(stamp({ t: "wheel", dx: e.deltaX * unit, dy: e.deltaY * unit, x: p.x, y: p.y }, e));
stream.wheel(e.deltaX * unit, e.deltaY * unit, p.x, p.y, e.timeStamp);
e.preventDefault();
}, { passive: false });
function mods(e) {
@@ -359,8 +142,7 @@ function onKey(e, down) {
if (down) showHud(hud.hidden);
return e.preventDefault();
}
const m = { t: "k", e: down ? "down" : "up", code: e.code, key: e.key, mods: mods(e) };
send(down ? stamp(m, e) : m);
stream.key(down, e.code, e.key, mods(e), e.timeStamp);
e.preventDefault();
}
@@ -373,26 +155,26 @@ function keyCode(ch) {
function bench(m) {
if (m.action === "overlay") return showHud(m.on !== 0 && m.on !== "0");
if (m.action === "click") {
const at = { x: +m.x || 0.5, y: +m.y || 0.5 };
send(stamp({ t: "m", e: "down", b: 0, ...at }));
setTimeout(() => send({ t: "m", e: "up", b: 0, ...at }), 40);
const x = +m.x || 0.5, y = +m.y || 0.5;
stream.down(x, y, 0);
setTimeout(() => stream.up(x, y, 0), 40);
} else if (m.action === "type") {
const text = String(m.text || ""), gap = +m.interval || 150;
[...text].forEach((ch, n) => setTimeout(() => {
const code = keyCode(ch);
if (!code) return send(stamp({ t: "text", s: ch }));
if (!code) return stream.text(ch);
const mods = ch !== ch.toLowerCase() ? ["shift"] : [];
send(stamp({ t: "k", e: "down", code, key: ch, mods }));
setTimeout(() => send({ t: "k", e: "up", code, key: ch, mods }), 30);
stream.key(true, code, ch, mods);
setTimeout(() => stream.key(false, code, ch, mods), 30);
}, n * gap));
}
}
addEventListener("keydown", e => onKey(e, true));
addEventListener("keyup", e => onKey(e, false));
addEventListener("blur", () => send({ t: "release" }));
addEventListener("blur", () => stream.release());
show("Connecting to your computer…");
connect();
stream.start();
</script>
</body>
</html>