docs: full reverse-engineering writeup + RE toolbench

- docs/REVERSE_ENGINEERING.md: library extraction, scan-out ring, pan,
  chroma plane (V-first words), kernel-side notes, PLD delivery, timeline
- tools/: dumplib payload + sprx/xref hunters + frame renderers (own code only)
This commit is contained in:
fstat13 committed 2026-10-15 14:49:47 +02:00
1 parent 76a6408586
commit 2a58cbcb43
14 files changed
+3076

No files matched your search

+168
View File
@@ -0,0 +1,168 @@
# PSCapture — Reverse Engineering Notes
Everything in this document was obtained on a **personally owned, jailbroken
PS5** for interoperability purposes (reading the console's own frame buffer to
use it as a capture device). **No Sony binaries are distributed in this
repository** — the extracted libraries below stay on the researcher's machine;
the `tools/` folder contains only our own code, and `tools/dumplib/` is the
payload that extracts the libraries **from your own console** so you can
reproduce every finding yourself.
---
## 1. Extracting the system libraries (`tools/dumplib/`)
Method: a payload running under the jailbreak asks the kernel for the module
list of a live process and dumps each module to `/data` (FTP out).
Getting there took two false walls:
1. **`0xCC` fill** — reading a module image through the process' user mappings
yields trap padding for unloaded segments. The fix: read the *file image*
(the module as loaded from its source file), not the mapped view.
2. **`kernel_dynlib_handle` semantics** — the handle index counts modules per
process; querying the wrong process silently fails. The reliable recipe:
`kernel_dynlib_handle(pid, "libXxx.sprx") → handle → dump via the kernel's
own copy primitives`.
What a full pass pulls from the console (kept locally, catalogued here):
| File | Why it mattered |
|---|---|
| `libSceAvcap2.sprx`, `libSceAvcap2mvr.sprx` | the "capture" service family |
| `libSceVideoOut.sprx`, `libSceVideoOutSecondary.sprx` | video output |
| `libSceVideoRecording.sprx` | recording service (128 MB session file) |
| `libSceJpegEnc.sprx` | the hardware JPEG encoder |
| `libSceScreenShot.sprx` | system screenshot API |
| `libSceAvSetting.sprx`, `libSceAgc.sprx`, `libSceVenc.sprx`, `libSceVdecCore.native.sprx` | AV pipeline |
| `libSceIpmi.sprx`, `libSceMbus.sprx`, `libSceLncUtil.sprx` | IPC plumbing used by the services |
| `libSceSysmodule.sprx`, `libSceSystemService.sprx` | module loading / services |
| `libkernel.sprx`, `libkernel_sys.sprx` | syscall surface |
| `daemon/SceSysAvControl.elf` | **the key**: the AV control daemon |
| `daemon/AgcCompositor.elf`, `daemon/SceSysCore.elf` | companions used for cross-checks |
Loaded-library inventory (which process has what) is produced by the same
payload (`recon` style listing).
---
## 2. Where the pixels are
The picture never travels through any "capture" API that apps can call
(that was the *other* project — the one with 128 MB session files and the IPMI
wall). Instead, the display controller (**DCE**) continuously writes the
scan-out frame into a ring buffer in memory, and a system daemon exposes the
geometry of that buffer.
### The scan-out ring
- Ring virtual address: `0x200200000`, ring data at `+0x4200000`.
- Four surfaces, `SURF_STRIDE = 0x600000` (6 MB) apart, written at 60 Hz.
- Each surface holds the frame as **P010** (10-bit), row stride `3840` bytes
(1920 px × 2 bytes).
- The daemon structure (`ScaninControl` @ `0x4f1958` inside
`SceSysAvControl.elf`, vtable `0x4e5078`) exposes:
- `+0x528` / `+0x530`: physical addresses of the two tap buffers,
- `+0x550` / `+0x554`: dimensions,
- `+0x55c`: flag, `+0x560`: slot table `{u64 w; u64 h; u32 pid; u32 tapPoint} × 3`,
- `getScaninBufferOffset` (`0x435a30`) returns the buffer paddr and the VA.
### P010 decoding
10-bit samples store the useful byte at **odd** indices; extracting the odd
bytes yields a clean 8-bit plane (the "even = low bits = noise" mistake was
made once and is documented so it is not repeated).
### The pan ("dimension")
The scan-out base inside the ring **drifts**: the frame's first pixel does not
sit at a fixed offset. Reading at a fixed offset yields the classic
"image rotates and slides" artifact.
- Exact metric (v10): continuity across a cut. For a candidate cut `c*`, the
strip `[row r, col 0..]` must match `[row r-1, col c*..]`; minimize
`Σ |Y[r][j*2+1] − Y[r-1][(c+j)*2+1]|` over ~64 sampled rows.
- Pan conversion: `pan = ((1920 − c) % 1920) * 2` — no `+960` offset.
- Twin ambiguity: two minima 1920 bytes apart; the usable pair on this unit is
`scan + 1920`. **Field-verified constant: pan 2230** (kept fixed; drift is
~2 B/s and below visibility). A user-controlled fine shift (PC-side) handles
the last few pixels.
### The chroma plane (the hard one)
- The chroma data starts at byte **`0x3EA300` (4 104 960)** — *not* after the
luma plane. The region before it is a dead zone.
- **In this plane the 16-bit words are V-first**: the pair order must be
swapped when building NV12 (verified byte-to-byte: error 15.5 → 1.7 →
0.06 against a reference frame with the same content).
- Valid rows: 527 of 540; the tail is not yet written on read — replicate the
last good row (the famous "green bar" was the tail being read into the gap).
- The historical v10 build read 11 rows later (`0x3F4800 + pan`): same
palette, wrong position — that is why "v10 colors looked right but red-ish".
### Color & geometry verification loop
A reference frame (NV12 dump) plus tiny offline tools:
- `tools/p010png.py` / `tools/nv12png.py` — deterministic renderers (same
math as the driver's YUY2 path, so screenshots match what Discord shows);
- byte-level diffs against a stored reference frame (mean |Δ| per plane);
- anchors for alignment: the centered title text of the setup screen
(993 px → 959 px after correction, on a 1920-wide frame);
- the correction itself: a cyclic horizontal shift, **even** values only
(an odd shift re-labels U/V pairs and hue-inverts the image).
---
## 3. Kernel-side notes (why the payload is careful)
- **Reads of kernel memory** go through a page-table walk (validation of every
translation step) and 8-byte reads via the direct-map alias. **Never**
`kernel_copyout` on a kernel address: a fault there is a kernel panic
(learned the hard way, once).
- **Ring mapping trick**: a cacheable user mapping of the DCE physical range
is built by writing one PML4 entry pointing at our own fresh page-table
subtree (the daemon's own PDE flags replicated: write-back, no PCD/PWT).
This is what makes bulk frame reads fast (4 GB/s instead of 45 MB/s through
the syscall path).
- **Quiescence**: each surface is being rewritten at 60 Hz; the reader copies,
waits 2 ms, compares, and moves on if the surface changed — avoids tearing.
- **Service setter**: `sceAvControlChangeScanInMode` is never called (it can
wedge the display engine and, in the worst case, the host app).
## 4. Running code with kernel privileges (delivery)
- **PLD format**: `eb 0b 'P' 'L' 'D'` + u64 `code_size`, then code, then data.
A loader on port **9019** maps `code_size` bytes executable and the rest RW
behind it, then calls the entry. This is how the ecosystem's kernel payloads
ship — and how further experiments (USB NIC driver) are delivered.
- Kernel modules compile against a vendored **FreeBSD 9.0** header set (the
Orbis kernel base, `__FreeBSD_version 900044`); per-firmware offset tables
(e.g. `allproc`, IDT, sysentvec for 13.60) exist in the scene and were
cross-checked.
## 5. Tools in this repo
| Tool | Purpose |
|---|---|
| `tools/dumplib/` | payload that extracts system libraries from your own console |
| `tools/sprx.py` | sprx ELF lister/extractor (exports, imports, sections) |
| `tools/callsites.py`, `tools/scan_refs.py` | xref hunt in disassembly (RIP-relative call sites) |
| `tools/whoimports.py`, `tools/stubmap.py` | import graph across the dumped libraries |
| `tools/p010png.py`, `tools/nv12png.py` | deterministic frame renderers (P010 / NV12) |
| `tools/flat.py`, `tools/asmwin.py` | binary utilities used during the hunt |
## 6. Timeline (condensed)
1. Capture through the official recording stack → dead end (session files,
IPMI client wall).
2. Direct read of the DCE ring → correct pixels, but the image "rotated":
the pan was discovered.
3. Pan metric, twin resolution, fixed 2230 — geometry solved.
4. Chroma plane hunt: base, dead zones, V-first words, tail replication —
colors solved (byte-verified).
5. Ring mapped cacheable → 1080p at ~19-35 fps over plain TCP to the PC.
6. Kernel-side delivery studied (PLD, 9019) → foundation for the next step:
a real USB network driver for full-rate/60 fps streaming.
*Reported, reproduced, and maintained by **fstat13**.*