saphid--frame-control/docs/bug-intake.md

Authenticated Frame Control bug intake

This service is solely for https://github.com/saphid/frame-control, PostHog project 343535. The receiver never calls an agent or external API. The local dispatcher can wake only its operator-configured signed T3 webhook when a job exists. It is separate from the shipped desktop app and uses Python 3.9+ stdlib, matching repository tests. Existing .github/workflows/checks.yml discovers its tests. Existing release workflow builds tag-triggered draft releases; scripts/publish-release.sh publishes tested drafts. The deployment uses a signature-protected T3 scheduler endpoint; no GitHub workflow trigger is introduced.

Trust boundary and deployment prerequisites

A public PostHog capture key is not proof that a report came from the app. $lib, OS, and version are relevance evidence only, all forgeable by a public ingestion client. Even signed reports remain untrusted data and may contain prompt injection or private contact/log data.

Before this deployment, project 343535 had only GeoIP transformation and an hourly email-only bug alert. The live deployment now uses a native template-webhook destination with encrypted signing_secret, service-owned credentials, a signed T3 scheduler, and persistent Tailscale Funnel HTTPS ingress. The existing email alert is preserved. No bridge or private API poller is required. Configure the destination body to the fixed envelope below: literal repository and project, with the PostHog event supplying uuid, event, and properties. Use the destination's JSON serialization for the event/properties, never string interpolation of unescaped report text. Do not put signing keys in app telemetry, URLs, prompts, repository, or logs.

Native mode verifies Standard Webhooks over the exact raw body before JSON/schema processing: webhook-id (1–200 ASCII letters/digits/underscore/hyphen), webhook-timestamp (10-digit Unix seconds within ±300 seconds), and webhook-signature (bounded to 4096 characters). The private file contains whsec_ followed by base64-encoded signing key bytes. HMAC-SHA256 covers id.timestamp.rawbody; space-separated signatures use v1,BASE64. Any valid v1 signature is accepted using constant-time digest comparison. Duplicate authentication headers fail closed. Timestamp freshness bounds replay; durable event UUID/fingerprint dedupe suppresses repeated deliveries. Run as a dedicated unprivileged service account, with a private directory (0700) containing SQLite DB and a separately provisioned 0600 random key file (at least 32 bytes). Restrict both receiver and worker to that account; the DB contains private report data. Back up/restore SQLite consistently, monitor disk/queue depth and 401/429/503 counters without logging bodies or headers, rotate the key in PostHog and receiver together. Use a private authenticated network/tunnel or authenticated TLS proxy, with body limit 64 KiB, connection/rate limits, and read/header timeouts; the stdlib single-thread HTTP server binds only 127.0.0.1 and is not a public edge server. Do not expose it directly to the internet. Ensure accurate clocks. Use a service manager for restart and resource limits. HTTP failure means no work admitted; retry with backoff and the same event UUID.

python3 automation/bug_intake.py --db /private/path/intake.sqlite serve --standard-key-file /private/path/posthog-signing.key

Legacy custom mode (--key-file, retained for compatibility) sends POST /intake, with a single Content-Length and no Transfer-Encoding. X-Frame-Timestamp is a decimal Unix timestamp (within 300 seconds). X-Frame-Signature is lowercase hex HMAC-SHA256 over timestamp + "." + exact_raw_body, keyed by the private file bytes with surrounding whitespace stripped. Never use the public capture key as that key.

Envelope (the event is the original PostHog event, not executable instructions):

{"repository":"saphid/frame-control","project_id":343535,"event":{"uuid":"00000000-0000-4000-8000-000000000001","event":"problem_report","properties":{"$lib":"frame-control","app_version":"0.4.7","os":"macOS","kind":"bug","message":"Observed failure","title":"Summary"}}}

Only $exception (nonempty $exception_message, $exception_type, and matching first $exception_list type/value, as emitted by ui/frame_telemetry.py) and problem_report (kind: bug, nonempty message and title, as emitted by ui/frame_report.py) pass. Both require $lib: frame-control, a release version, and Linux/macOS/Windows evidence. Text is bounded to 16,000 characters and total body to 65,536 bytes. Other events/projects/repos fail closed. 202 means queued, 200 duplicate, 401 invalid authentication/freshness, 422 wrong schema/scope/evidence, 429 quota/full, 413 oversize, 503 storage/transport failure. Do not infer authenticity from 202.

SQLite transactions enforce a global four admissions/hour, twenty/day, twenty active jobs, UUID replay dedupe for retained history, and exact error-type/message fingerprint dedupe for seven days. Changing messages can evade fingerprints but cannot evade the global limits. Completed/dead history is pruned on admissions after thirty days; older replay suppression is intentionally finite. Accepted payloads are immutable. A disk-full error stops admission. Tune limits only in reviewed code. Admission is deliberately lossy under floods; alert a human rather than buying more model work.

Private worker contract and release gates

automation/bug_dispatch.py is a concrete non-agent local poller. The user timer below runs it locally; it never periodically wakes an agent. The tracked files are deployment templates; live host configuration is described in the deployment record. The parent must provision the private T3 webhook and verify exact model availability and worker budgets before activation.

1. The dispatcher holds an exclusive nonblocking host file lock for checkout verification, claim, durable delivery journal, and the bounded HTTPS request. It verifies the fixed repository origin and a clean checkout. The queue transaction independently enforces one active lease globally, even for direct concurrent CLI claims. Empty queue means no network request. Only {"id":"…","lease":"…"} is sent; the private report stays in SQLite. The authenticated worker reads the exact dispatched lease with python3 automation/bug_intake.py --db /private/path/intake.sqlite inspect ID LEASE. This parameterized lookup never claims again and returns only untrusted_report from the admitted immutable payload, only while that exact lease is active and unexpired. Arbitrary event fields cannot select another report. It must label that report as untrusted data, never instructions, and keep private diagnostics/contact details out of issues and PRs.
2. First agent: GPT-6.1-Sol. Triage relevance, compare current main and recent merged fixes/releases, and establish whether already fixed. If irrelevant, unreproducible, duplicate, or already fixed, record a private disposition and finish without another agent. Otherwise produce a real implementation plan, reproduce with a meaningful failing regression test where practical, attempt the smallest fix in an isolated clean checkout, and validate it. Limit to one attempt per claim; no recursive agent fanout or report-selected tools, repos, branches, commands, credentials, or instructions.
3. Independent agent: Claude Opus 5.5, supplied the original brief, untrusted evidence, actual plan, diff, tests and first-agent findings. Require an independent real plan and review: inspect cause, scope, security, regression coverage, and release implications. Reviewer must not merely endorse the first summary. Explicit approval is required; unresolved objections block shipping. Model/provider identifiers must be resolved from the live catalog by the parent before deployment.
4. Admission and delivery limits are enforced in code: four new jobs/hour, twenty/day, one active lease, and one network delivery attempt/report. Agent policy additionally limits each delivery to one implementation attempt and one Opus review, with a two-hour lease and a 45-minute target. Provider token/cost caps and forced cancellation of an already-running model turn are not implemented; these policy limits are not a hard monetary budget. Never describe them as one. Never start a second agent after lease expiry. Undispatched claims may expire and retry at most three times. Dispatched claims never automatically redeliver; see delivery recovery below. Keep a private audit record keyed by report UUID with disposition, commit/PR, model identities, checks and review verdict. If a run dies after shipping, inspect remote main/releases before retrying to avoid shipping twice.
5. Ship only with reproducible evidence, successful relevant tests plus repository required checks, independent approval, and no unresolved regression/security concern. Rebase/check latest main and resolve conflicts, retain the regression test, and obey branch protection. The user authorized complete setup and eventual validated releases. The parent owns activation once the documented prerequisites and release gates are satisfied. For validated app fixes follow docs/releasing.md (or discover the current release guide), version/tag/build the draft via existing workflow, inspect every required artifact/check, then publish using the repository release procedure only after platform validation. Never claim real headset validation without performing it; hardware-required fixes stay blocked for that validation. A failed gate must not publish or merge.
6. After a terminal private disposition or completed validated release, finish using the returned id and lease: python3 automation/bug_intake.py --db /private/path/intake.sqlite finish ID LEASE --disposition "private terminal outcome". Finish atomically records the disposition, report ID, lease and timestamp and rejects stale leases. Keep detailed model/check/review evidence in the private audit referenced by that disposition. Inspect dead letters and held deliveries manually; never automatically replay a dispatched job. There is no remote claim/finish route.

Before activation, test end-to-end with synthetic staging data: bad/missing signatures, wrong project/repo, non-bug events, forged app evidence, oversized body, duplicate/restart, concurrent flood, no-job/no-token behavior, finite dispatcher budget, lease expiry, storage failure, model availability and release failure gates. Keep the receiver private throughout.

python3 -m unittest discover -s tests -p test_bug_intake.py -v

Local deployment templates

Install the reviewed clean checkout at %h/Services/frame-control-intake, and use the user units in automation/systemd/. The service fixes the script/working directory and the private config path %h/.config/frame-control-intake/dispatch.json; its only writable application directory is %h/.local/state/frame-control-intake. Create those private directories (0700) as the dedicated account. Provision this config as 0600, replacing paths/URL with actual operator-controlled values (no secrets belong in this example):

{
  "repository": "saphid/frame-control",
  "checkout": "/home/SERVICE_ACCOUNT/Services/frame-control-intake",
  "database": "/home/SERVICE_ACCOUNT/.local/state/frame-control-intake/intake.sqlite",
  "lock": "/home/SERVICE_ACCOUNT/.local/state/frame-control-intake/dispatch.lock",
  "webhook_url": "https://OPERATOR_CONFIGURED_T3_HOST/PRIVATE_WEBHOOK_PATH",
  "key_file": "/home/SERVICE_ACCOUNT/.config/frame-control-intake/t3-webhook.key"
}

The operator must separately provision the signing key (owned regular file, 0600, at least 32 bytes after whitespace stripping); keys are provisioned separately from this code. Configure T3 webhook signature verification with header x-hub-signature-256, encoding hex, prefix sha256=, and the same key. This is HMAC-SHA256 of the exact raw JSON body, without the receiver's timestamp-dot prefix. The URL must be a private preconfigured authenticated T3 endpoint, never a value from a report. HTTPS certificate validation is enabled; redirects are never followed. Network reachability/privacy, T3 signature configuration, and job/lease consumer mapping require staging verification. No T3 tools or model APIs run in the polling process.

After provisioning prerequisites, the parent/operator can install the three user unit files, reload the user manager, and enable the timer. Confirm user-manager persistence/lingering as appropriate to the host. Do not activate before a clean deployed checkout, native PostHog signing, private network, signing key, worker DB access, actual model availability, budgets and release gates are verified. PostHog, 1Password and login setup remain parent-owned. The service has a 60-second process limit and the HTTPS connection a 15-second timeout; the parent must independently enforce agent/model budgets and lease expiry.

Delivery recovery and audit

A dispatches row is committed as uncertain before sending. Each report has at most one dispatcher delivery attempt, including across restarts. A crash before sending can therefore require manual recovery. A 2xx response records accepted, another status records rejected, and transport ambiguity remains uncertain; none is retried automatically. Response bodies, URLs, headers and reports are never logged. Successful HTTP acceptance is not proof of successful worker completion. The worker must validate the lease and dedupe by report ID before spending tokens or shipping.

While a dispatched lease is active, all additional claims stop. On expiry it becomes held, which also stops all further claims globally until an operator reconciles T3 delivery/run state and any remote commits/releases. finish only accepts the current unexpired lease. For an expired held job, an operator must record the private disposition and explicitly resolve its state in a transaction after reconciliation; there is intentionally no automatic replay/reset command. Preserve its dispatch row and report for investigation. Do not delete the delivery journal to force a retry. Dispatch rows persist even when terminal report payloads age out after thirty days; back up the database privately. Undispatched failed claims retain the existing three-claim bound. This conservative availability tradeoff prevents unlimited ambiguous webhook delivery and duplicate agent spending.

Live deployment record — 2026-10-20 UTC

Receiver and 60-second local dispatcher timer are enabled on lxso2, with user lingering enabled. Clean checkout: /home/saphid/Services/frame-control-intake. Private config: /home/saphid/.config/frame-control-intake/ (0700; keys/config 0600). Queue/audit: /home/saphid/.local/state/frame-control-intake/. No credentials are tracked.

• Public signed ingress: https://lxso2.tail4e5636.ts.net:10000/intake, persistent Tailscale Funnel to loopback port 8789. Existing routes are preserved.
• Enabled PostHog destination: 01a1569a-46b1-0000-17f5-d4516b740dd2, project 343535, “Frame Control — authenticated bug triage”. Filters require $lib=frame-control, and $exception or problem_report with kind=bug. No historical backfill.
• Enabled signed T3 webhook task: frame-control-authenticated-bug-dispatch-v1, launching a fresh independent worktree/thread per delivery (no setup-thread binding); GPT-6.1-Sol (codex/gpt-6.1-sol). Actual worker instructions are mirrored in worker-policy.md. Endpoint/key are private in dispatch config. Maximum accepted delivery age is five minutes.
• Native PostHog signed invocation, public signature/replay smoke, and authenticated dispatch smoke all completed. Three synthetic jobs are done, with no application coding/delegation. Unsigned requests returned 401; duplicate admission created no second job. Local empty queue performs no agent/network dispatch.
• Focused suite: 18 passing; compilation, diff whitespace and systemd unit verification passed. Full suite: 1011 tests, 12 skipped, one headset-removal race failure (test_headset_removed_elsewhere_mid_install_reaches_nothing); subsequent complete test_link.py rerun passed all 81 tests. This is not an entirely passing full-suite run.
• Independent security review found slow-trickle starvation; fixed with an absolute socket shutdown deadline and a real socket regression test. Review is not Opus release approval.
• Claude Opus 5.5 native review attempts hit provider rate limits; an OpenRouter attempt reported unknown provider. No substitute model approval, main merge or release occurred. Live workers must stop shipping when Opus cannot approve. Model/turn/privacy gates remain agent policy, not enforced token or monetary caps.

Fleet audit (read-only): lxso2, lxso1, MacBook Pro, and nursedroid had no existing matching PostHog/Frame Control bug intake. Nursedroid scheduler was checked at ~/.t3code/userdata/statev2.sqlite. lxso3 required an additional Tailscale login check; MacBook Air denied SSH, so those hosts remain unverified. Unrelated scheduled tasks were left untouched. One central receiver is deployed, with no duplicate per-host agents.

Independent unattended proof — 2026-10-20 01:24–01:25 UTC

Scheduler binding was changed through the T3 scheduler API to boundThreadId=null: each signed delivery now creates an independent top-level GPT-6.1-Sol worktree, without requiring the setup conversation. The stored model remains codex/gpt-6.1-sol, full-access runtime.

PostHog's native destination invocation sent synthetic report 4028bb1d-42fb-48a5-a4e5-959b21803770 at 01:24:31. The existing systemd timer dispatched it at 01:25:00 (HTTP 202; T3 journal signature verified). T3 automatically created worker thread:project:74cd7f55-05d4-49bd-8d6d-8719bad00083:eeb6c52e-d3c0-4a25-afa4-2fd8a039599e, with no parent thread, in its own worktree. That worker independently validated the lease, read the report, wrote a 0600 private audit and finished deployment-smoke at 01:25:21; its run completed at 01:25:25 with zero pending requests. The setup agent only observed the job and did not claim or finish it. No approval, human input, coding, or review delegation was needed. A separate invalid-signature T3 request returned 403 without a worker wake.

Private correlated evidence is at ~/.local/state/frame-control-intake/audit/unattended-proof-2026-10-20.json, including admission, dispatch, completion, signature result, model and worker identity. This demonstrates unattended transport and terminal worker execution, not an application fix/release or operation while the T3 host is shut down. lxso2, Tailscale/T3 Connect, the T3 scheduler and provider credentials must remain operational. Opus approval remains required for shipping and currently provider-rate-limited.

Durable report correlation

The private intake SQLite database is also the shared history ledger for project 343535 and saphid/frame-control. All agents use the lxso2 copy (SSH from other authorized hosts); no per-host copies. Back up this database privately using SQLite backup, including history tables. There is no public report lookup endpoint. Files remain account-private; this is fleet-accessible through authorized SSH, not a GitHub-hosted index.

history_events maps every accepted occurrence UUID, including deduplicated deliveries, to the existing v1 exact fingerprint: SHA256 of Python JSON [event_name, exception_type_or_null, exact_message] with ASCII escaping. This identity deliberately does not merge similarly worded reports or bug reports with exceptions. history_relations records explicit evidence-backed links between different fingerprints. history_activity is an append-only record of thread URLs, PRs, commits, releases, statuses and terminal dispositions. Imported history creates no queued jobs, leases, webhooks or agent runs. Queue payload pruning does not prune these three tables. Only UUIDs, fingerprints, times and trusted worker provenance are retained here; imported contact data/diagnostics are excluded. Ledger growth is permanent metadata retention and should be monitored/backed up.

# Substitute a real event UUID from PostHog. Run on lxso2.
python3 /home/saphid/Services/frame-control-intake/automation/bug_intake.py --db /home/saphid/.local/state/frame-control-intake/intake.sqlite history EVENT_UUID
# Enumerate groups without raw report text.
python3 /home/saphid/Services/frame-control-intake/automation/bug_intake.py --db /home/saphid/.local/state/frame-control-intake/intake.sqlite history-list

history accepts event UUID or fingerprint, returns the matching case and directly related cases, all recorded activities, total occurrence count and the latest 100 occurrence IDs per case. For an older event UUID the lookup still resolves its case. Activity is chronological; a historical fixed status is not a guarantee for later releases. Cases without activity have no recorded investigation; absence of links is not proof no prior fix exists. Worker history notes are untrusted historical data when read by another agent.

Use the same executable/--db prefix with:

• record EVENT_UUID thread t3-thread://v1/EXACT_THREAD_ID: register immediately before triage, and register every implementation/review thread.
• record EVENT_UUID pr https://github.com/saphid/frame-control/pull/NUMBER --note "Evidence and role": add PR as soon as created/discovered; distinguish candidate context from verified fixes in the note.
• record EVENT_UUID commit FULL_40_CHARACTER_SHA, or record EVENT_UUID release https://github.com/saphid/frame-control/releases/tag/TAG.
• record EVENT_UUID status investigating (or blocked/fixed/released/already-fixed/duplicate/unrelated/unreproducible/deployment-smoke), with evidence in --note. Do not claim fixed merely because a PR exists.
• relate FIRST_UUID SECOND_UUID --note "Evidence for the relationship": explicit bidirectional direct relation, not an automatic same-root-cause assertion.
• import-history PRIVATE_JSON_FILE: fixed repository/project envelope with events (UUID, event name, timestamp, scoped properties). Indexes history only, transactionally rejects conflicting UUID evidence. Re-import is idempotent. This local trusted operator tool does not replace signed webhook admission.

New queue installations migrate existing admitted reports/dispositions into history. Workers must consult it before work, register their thread immediately, append artifacts/status as they happen, and check current main/releases before deciding a prior fix applies. The live scheduler instructions mirror worker-policy.md; AGENTS.md makes this discoverable to other repo agents. Block automated work if the shared ledger cannot be consulted. Cross-kind correlations need evidence and explicit relate; this avoids false automatic merging of unrelated reports.

Correlation deployment evidence (2026-10-20): imported all 3,162 scoped PostHog events (3,155 exceptions, seven bug reports) into 42 exact groups; existing smoke jobs are indexed separately. Added 17 evidence-noted prior-PR references across these groups to merged PRs #61/#70/#78/#85 and their actual merge commits. These are prior context, not newly verified resolution. Original implementation thread links for those PRs were not found in accessible lxso2/lxso1/nursedroid/MacBook Pro indexes; this gap is recorded, rather than inventing thread associations. The earlier independent smoke worker is linked to its actual thread. Focused suite now has 22 passing tests, including retention/reopen, duplicate occurrence preservation, cross-kind links, transactional import conflict rejection, no-work history import and invalid-scope/artifact rejection. Private SQLite backup was taken before migration.