steam-frame/Pixelplanet--Pixel-VR-Player
1
0 Fork 0
mirror of https://github.com/Pixelplanet/Pixel-VR-Player.git synced 2026-10-26 05:00:23 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity
Files
a6be5eb60dfc3cd88afeb528da4db89df4fa63ee
Pixelplanet--Pixel-VR-Player/TODO.md
T

22 KiB
Raw Blame History

Pixel VR Player — TODO & Handoff

Native C++/OpenXR/Vulkan/FFmpeg VR video player for the Valve Steam Frame (Snapdragon 8 Gen 3, Adreno 750 / Turnip, SteamOS/aarch64). This file is the handoff for the next agent. Deep device/build facts live in repo memory /memories/repo/steamframe-facts.md — read that too.

Active backlog (2026-10-20 — Completed)

  • Virtual Keyboard Tilt Inversion, Raycast Auto-Focus & UI Hide Refinements
    • Symptom 1: Keyboard spawned tilted with the top edge pitched forward (slanted away from user).
    • Resolution 1: Corrected tilt angle to -0.38f\ (-21.8 degrees). The top edge slants backward away from user; face normal tilts upward directly toward the user's eyes.
    • Symptom 2: Laser beam shot 2.5m past the keyboard with no cursor dot on the keys; laser wasn't focused.
    • Resolution 2: Added keyboard plane collision test in \VulkanRenderer::updatePointer()\ using \KeyboardModalData::transform. Laser ray terminates directly on the keyboard surface with glowing cursor dot on hovered keys. Also ensured active controller laser turns on automatically when keyboard opens, and guarded background menus from eating keyboard trigger clicks.
    • Symptom 3: Trigger pull while HUD dismissed didn't reactivate the UI unless hitting a hidden button; empty-space clicks re-dismissed the UI in the same frame.
    • Resolution 3: Recorded \wasHudDismissed\ at frame start and guarded dock interactions with !wasHudDismissed. Trigger click cleanly un-hides the HUD without clicking hidden buttons or re-dismissing in empty space.
    • Symptom 4: Analog grip squeeze was overly sensitive and accidentally reactivated the dismissed UI.
    • Resolution 4: Ignored soft analog grip (\in.gripDown\ and \BtnGrip); only digital full grip click stop (\in.gripFullPressDown) un-hides the HUD.

Active backlog (2026-10-19 — Completed)

  • 1. Fix Virtual Keyboard Laser Pointer Focus & Tracking

    • Symptom: Virtual keyboard opened, but laser pointer disappeared or wasn't focused on the keyboard; user had to squeeze/press the controller grip fully to activate it. Also, the keyboard centered where the Steam menu was centered when the app launched instead of where the user centered their view.
    • Root Cause:
      • SteamVR's virtual keyboard overlay key is VRKeyboard_uid0 (from CEF webhelper), not valve.steam.gamepadui.keyboard. Previous lookups failed, so MakeOverlaysInteractiveIfVisible and VROverlayInputMethod_Mouse were never applied to the actual keyboard overlay.
      • VulkanRenderer::updatePointer() was returning early (if (keyboardModal_.visible) return;), which prevented raycasts to the modal plane and left pointerActive_ false with no cursor dot.
      • SetKeyboardTransformAbsolute was never called with the user's current centered view orientation/position, so SteamVR defaulted to the launch standing origin.
    • Resolution:
      • Implemented dynamic overlay discovery across handles 1..128 and known keys (VRKeyboard*, valve.steam.gamepadui.keyboard, system.keyboard).
      • Configured MakeOverlaysInteractiveIfVisible, VROverlayInputMethod_Mouse, and SendVRDiscreteScrollEvents on the keyboard overlay.
      • Calculated world matrix in TrackingUniverseStanding directly from the user's HMD pose (or OpenXR headPos and headForward): placed 1.05m in front of eyes, 0.38m below eye level, tilted 28° up toward the eyes.
      • Applied transform via SetKeyboardTransformAbsolute(vr::TrackingUniverseStanding, &kbMat) and SetOverlayTransformAbsolute.
      • Updated VulkanRenderer::updatePointer() to raycast against the modal plane when keyboardModal_.visible is active, ensuring sharp laser pointer focus and a visible cursor dot.
      • Guarded VrKeyboard::shutdown() to prevent crash on exit.

  • 2. Controller Movement UI Dismissal Fix

    • Symptom: When the user clicked "Hide UI" or clicked outside controls, any hand movement immediately reactivated the UI.
    • Requirement: Once dismissed, the UI must stay hidden regardless of controller movement/rotation, and only reactivate when the user explicitly presses a physical button on the controller.
    • Resolution:
      • In App::run(), when hudDismissed_ is true, controller movement (moveDist) and rotation (rotDiff) are blocked from clearing hudDismissed_ or setting activity = true.
      • Only physical button presses (in.togglePlay, in.browserToggle, in.recenter, in.buttonDown[...], in.gripDown, in.gripFullPressDown, in.select outside cooldown, or thumbstick movement) reactivate the HUD.

  • 2. Strict IPv4 Validation & Live Sanitization

    • Symptom: Host field accepted invalid characters, spaces, and did not strip leading zeros.
    • Resolution:
      • Created src/util/ipv4.hpp with sanitizeIpv4() and isValidIpv4().
      • Live typing sanitization: automatically strips non-digits/dots, discards duplicate dots, clamps octets to 255, drops leading zeros in each octet (e.g. 001 -> 1, 010 -> 10, 00 -> 0), limits to 4 octets, and preserves trailing dots while typing.
      • Added automated unit test suite tests/test_ipv4.cpp and verified all test cases pass on the device.
      • Verified strict isValidIpv4() validation before saving an SMB share in saveSmbModal().

  • 3. Fix ??? Glyphs (Unicode Star Elimination)

    • Symptom: Sidebar and header button displayed ??? instead of the star icon.
    • Root Cause: FreeType SDF text atlas was initialized only for ASCII codepoints 32..127. UTF-8 multi-byte sequence ★ (\xe2\x98\x85) produced 3 replacement ? characters.
    • Resolution: Replaced all UTF-8 star glyphs with clean ASCII strings: "Bookmarks", "[ Bookmarked ]", and "[ + Bookmark ]". Adjusted header breadcrumb max width to 0.46f to prevent collision with the header button.

  • 4. Rename Quick Path to Bookmarks & Implement Multi-Bookmark Overview (bookmarks://)

    • Requirement: Rename Quick Path to Bookmarks, and have the Bookmarks sidebar tab open an overview showing all bookmarked folders and files in regular browser views.
    • Resolution:
      • Added Bookmark struct and multi-bookmark storage in Browser with persistent JSON serialization to ~/.config/pixelvr/bookmarks.json.
      • Implemented bookmarks:// virtual directory navigation displaying all bookmarked paths with full cards (BigCards, MiniCards) and CompactList support, thumbnails, and direct activation.
      • Sidebar subtab 4 navigates to bookmarks:// with proper active tab highlighting.
      • Header button toggles bookmark for the current open directory ([ + Bookmark ] / [ Bookmarked ]).
      • Action Menu (Y button) provides "Add to Bookmarks" / "Remove from Bookmarks" / "Go to Bookmarks", and in bookmarks:// provides "Open Bookmark" / "Remove Bookmark".
      • Updated Settings Menu shortcut label to "Bookmarks Shortcut".

  • 5. Seamless 1-Click Update System & Streamlined Distribution

    • Requirement: Provide easy, friction-free updating so users do not need to memorize terminal commands or worry about losing their settings.
    • Resolution:
      • Added standalone local updater tools/update.sh and --update flag to tools/install.sh.
      • Added KDE Plasma Update Pixel VR Player.desktop desktop shortcut deployed automatically to ~/Desktop/.
      • Added remote Wi-Fi 1-click updaters for PC users (tools/update-from-pc.bat for Windows and tools/update-from-pc.sh for macOS/Linux).
      • Updated tools/package-release.sh to package updater scripts in the release tarball.
      • Updated README.md with comprehensive update instructions across all platforms.

  • 6. Fix UI Dismissal Flicker and Recessed Island Alignment

    • Symptom: Clicking outside the controls or clicking the Hide UI button caused the UI to briefly flicker for 1 frame and immediately reappear. The Hide UI button was also visually misaligned (half outside the dark island).
    • Root Cause:
      • in.selectHeld and trigger activity handling in App::tick() continuously reset hudDismissed_ = false and set activity = true on subsequent frames of the trigger pull, fighting the dismissal call.
      • Controller activity anchors were not reset on dismissal, and there was no dismissal grace cooldown, causing slight hand tremor during trigger release to immediately re-trigger movement thresholds.
      • The center transport recessed island background in VulkanRenderer was rendered at cx = -0.05f, hw = 0.30f, clipping through the center of the Hide UI button at +0.26f.
    • Resolution:
      • Created App::dismissHud() which sets hudDismissed_ = true, records hudDismissTime_, zeroes out activity, hides the control bar and laser ray, and snaps controller activity anchors to their exact current positions.
      • Added a 500ms dismissal cooldown preventing accidental re-engagement from trigger release tremors.
      • Blocked in.select and in.selectHeld from clearing hudDismissed_ while the HUD is dismissed.
      • Re-centered the transport recessed island background to cx = 0.0f, hw = 0.34f, creating perfect 34mm symmetrical padding for all 5 buttons (Stop, SeekBack, PlayPause, SeekFwd, HideHud).
      • Added hover tooltip pills for [Hide UI] and [Stop].

Carried over & Completed

  • A. Full-pitch recenter (usable lying down) — reorient screen + controls + 360/180 to full head orientation, not just horizon yaw.
  • B. Real controller models (glTF) rendered with depth.
  • C. Dynamic controller button labels from the user's bindings.

Completed (2026-10-18):

  • I. Laser auto-release when pressing Steam button (unfocused session gates controller pointer).
  • J. Native SteamVR / SteamOS Virtual Keyboard integration (OpenVR GamepadTextInput via dlopen).
  • K. Protected SMB credentials manager: hardware-keyed AES-GCM encryption, strict permissions, full CRUD.
  • L. Settings > Network & SMB Shares: manual IP/hostname input, authentication settings, auto-listed in Network browser.
  • M. UI Follow Mode: Mode 0 (Rock-solid World Lock default, zero drift/tilt) and Mode 1 (Smooth Follow).
  • N. Player UI dismissal: [ ✕ Hide ] button, empty space click-outside dismissal, B/Y toggle, and micro-tremor filtering (>0.06m threshold).
  • O. Quick Path Whirligig-style bookmarking header button + R3 controller shortcut button.

1. Build / deploy / test (the loop that works)

Build NATIVELY ON THE DEVICE (cross-compiling risks a libstdc++/FFmpeg SONAME mismatch — device is GCC 15 + FFmpeg n7.0 libavcodec.so.61).

  • Device: steamos@192.168.178.61 (mDNS frame may not resolve from WSL — use IP).
  • SSH key (WSL): $HOME/.config/steamos-devkit/devkit_rsa (from Windows %LOCALAPPDATA%\steamos-devkit\steamos-devkit\devkit_rsa).
  • All commands run from Windows via wsl -d Ubuntu -- bash -lc '...'.

Deploy one-liner (rsync → build → deploy):

wsl -d Ubuntu -- bash -lc 'rsync -az -e "ssh -i $HOME/.config/steamos-devkit/devkit_rsa -o BatchMode=yes" \
  --exclude "build*" --exclude dist --exclude sysroot --exclude .git --exclude .vscode \
  "/mnt/c/Projects/Pixel VR Player/" steamos@192.168.178.61:~/pixelvr-src/ && \
  ssh -i "$HOME/.config/steamos-devkit/devkit_rsa" -o BatchMode=yes steamos@192.168.178.61 \
  "cd ~/pixelvr-src && cmake --build build --parallel 2>&1 | tail -3; \
   pkill -x pixelvr.real 2>/dev/null; sleep 1; \
   cp build/bin/pixelvr ~/devkit-game/PixelVRPlayer/pixelvr.real; \
   cp build/shaders/*.spv ~/devkit-game/PixelVRPlayer/shaders/; echo DEPLOYED"'

(First time only: cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo.)

Fast compile-check without the device (WSL x86): cmake --build ~/pixelvr-build --parallel (configure once: cmake -S . -B ~/pixelvr-build -G Ninja -DPIXELVR_ENABLE_MEDIA=ON).

Launch (needs headset AWAKE — see gotchas): ssh … "python3 ~/devkit-utils/steam-devkit-rpc run-game gameid=PixelVRPlayer"

  • run-game sometimes BLOCKS → run it in an async terminal, read logs with a separate ssh. App logs to ~/pixelvr-run.log (wrapper sets PIXELVR_LOG=debug).
  • ~/devkit-game/PixelVRPlayer/pixelvr is a sh WRAPPER; real binary is pixelvr.real. Initial clip = path in ~/media/which.txt.

Headless diagnostics (NO headset needed):

  • pixelvr --probe <path-or-smb-url> → opens media, samples 5 frames, prints chroma min/max/mean (mean ~128 good, 0 = broken). Also tools/probe_smb.sh, tools/chroma_check.py.
  • pixelvr --xrext → lists OpenXR runtime instance extensions.

Gotchas (bit us repeatedly):

  • NEVER pkill -f pixelvr from the ssh cmd — the pattern matches the ssh session's own args and kills it (no output, exit 1). Use pkill -x pixelvr.real.
  • Triple-nested quoting (WSL '…' > ssh "…" > python/labels) breaks. Put scripts in a file and scp them (see tools/chroma_check.py).
  • Intermediate shell vars in bash -lc don't expand reliably — INLINE "$HOME/.config/…" and steamos@192.168.178.61 every time.
  • The run_in_terminal tool rewrites &&/|| to ;.
  • Video frames only decode with the headset WORN — remote launches with the HMD asleep reach "Session running" but process_video never runs. Use --probe for headless decode testing.
  • OpenXR apiVersion MUST be XR_MAKE_VERSION(1,0,0) (SteamVR rejects 1.1 with XrResult(-4)).

2. DONE (deployed & verified)

  • VR playback: HW decode (h264_v4l2m2m → NV12), PulseAudio audio, MSAA 4x.
  • GREEN BUG FIXED — libswscale drops chroma on NV12→NV12 on this platform. Fix in media_engine.cpp process_video: when srcFmt==AV_PIX_FMT_NV12, repack planes with a direct memcpy honoring linesize (skip sws; sws only for real conversions like SW yuv420p). Verified via --probe (chroma 0/255/mean126).
  • SMB streaming by default — media_engine opens smb:// via a custom AVIOContext backed by libsmbclient (SmbIO, guest auth). Browser activate() streams smb:// directly; downloadFocused() (A) is the explicit download. Verified headless (streamed a 15 GB file off the NAS).
  • In-VR file browser (FreeType text, local + SMB scan + avahi/mDNS discovery).
  • Controls remapped: A/X=play, B/Y=menu, Trigger=select, Grip=recenter, thumbstick=nav/seek/distance.
  • Transport HUD world-locked below the flat video (was head-locked), auto-hide.
  • Tracked controllers — OpenXR aim-pose actions + per-hand action spaces (xr_context.cpp); XrInputState.controllers[2]. Laser pointer + cursor + (basic) controller stub in vulkan_renderer.cpp. Pointer raycasts the browser panel → hover row; Trigger selects. Math helpers added in math.hpp.
  • Media control bar — point-and-click << · play/pause · >> · Files below the flat video; always-visible laser (2.5 m, snaps to UI hit). Files button opens the browser via the pointer (no reliance on face-button bindings).

3. TODO — BUGS (from latest in-headset testing)

B1. Video fringing / white-pixel aliasing when the screen is farther away

  • Symptom: high-res video shows white-pixel fringing around objects when the screen is a bit further; gone when moved much closer.
  • Cause: the NV12 video textures have NO mipmaps (mipLevels=1) and the sampler is linear-only → minification aliasing when the plane is small on screen.
  • Fix: create luma/chroma images with a full mip chain; generate mipmaps after each upload (vkCmdBlitImage down the chain in uploadCmd_ — R8/R8G8 support LINEAR blit on Turnip); sampler mipmapMode=LINEAR, maxLod=mipCount, and enable anisotropy (samplerAnisotropy feature + maxAnisotropy). Files: vulkan_renderer.cpp createVideoResources (mipLevels + image views), updateVideoTexture (blit mip gen), createSamplerAndDescriptors (sampler). Watch the per-frame blit cost; only regenerate when a new frame is uploaded.

B2. High-res 360/180 video won't play + stale frame + controls disappear

  • Symptom: a ~3600p video (meant to be 180°/360°) shows the LAST FRAME of the previous test clip and never plays; and the control bar/transport vanish (they only render for ProjectionMode::Flat).
  • Likely causes / actions:
    1. Decode fails at that resolution (HW decoder session limits, or HEVC — see K1). Run pixelvr --probe <file> to see codec/res and whether frames decode. Probably need a software-decode fallback for too-high-res / HEVC, or reject with a message. (Iris HW decode is fragile ≥6K and floods POLLERR on 4K HEVC).
    2. Stale frame on failure: when media_.open() fails, the renderer keeps the last uploaded frame (hasVideo_ stays true). app.loadMedia must detect the failure, clear the video (add a renderer_.clearVideo()), and show an error.
    3. Controls in 360/180 (DONE): Added floating, gaze-anchored control panel & transport HUD for 360/180 at distance 1.6 m, tilted towards eye level, fully raycastable by the laser pointer with clickable transport buttons.

B3. SMB browser scrolling is buggy (feedback loop) — DONE

  • Fixed: decoupled pointer hover from selection and scroll offset. Added browserHovered_ for laser aiming and browserFirst_ for stable windowing. Clamped pointer row mapping to the currently-visible window. Added a sleek, visible scrollbar track and thumb indicator on the right side of the panel.

4. TODO — FEATURES

F1. Real controller models (XR_EXT_render_model) — user's top ask

Runtime supports XR_EXT_render_model + XR_EXT_interaction_render_model (verified via --xrext). Our pinned OpenXR SDK 1.1.43 only declares the older XR_FB_render_model (which this runtime does NOT support). Phased plan:

  1. Bump OpenXR SDK — CMakeLists.txt PIXELVR_OPENXR_TAG (currently release-1.1.43) → a release that declares XR_EXT_render_model (verify by grepping the fetched openxr.h for xrCreateRenderModelEXT). Or hand-declare it.
  2. Add a DEPTH buffer to the render pass (3D models must self-occlude; current UI is painter-ordered, no depth). Add depth image/attachment + clear; give the model pipeline depth test+write, leave the flat UI pipelines depth-test-off.
  3. Lit model pipeline — pos+normal vertex input + N·L shader (model.vert/frag), register in cmake/CompileShaders + CMakeLists shader list.
  4. Load models — enable exts; enumerate per-controller interaction render models; fetch the glTF/GLB asset; parse with cgltf (single-header) → positions/normals/ indices (bake node transforms); render at the grip pose each frame. Skip textures first (flat gray + lighting). Big + blind → iterate in-headset.
  • Also consider adding the native /interaction_profiles/valve/frame_controller bindings (runtime lists XR_VALVE_frame_controller_interaction) for correct Frame button mapping — but a single wrong component path fails the WHOLE profile suggestion (graceful fallback), so get all paths right or skip.

F2. Better button / UI graphics — DONE

  • Replaced text labels (<<, II, >>, Files) with GPU-procedural Signed Distance Field (SDF) vector icons rendered in shaders/ui.frag (Rewind, Play, Pause, Fast-Forward, Folder/Files, Settings Gear, Recenter Target, Reticle Cursor).
  • Upgraded button styling to rounded glass capsules with antialiased borders and glowing hover states.
  • Transport HUD redesigned as a rounded pill progress track with glowing fill and thumb position dot.
  • Laser pointer cursor upgraded from a flat quad to a high-tech circular reticle.

F3. Controller remap settings UI — DONE

  • Implemented SettingsMenu model (src/ui/settings_menu.{hpp,cpp}) and pointer-driven in-VR Settings panel overlay.
  • Users can view and cycle button bindings (Play/Pause, Menu, Select, Recenter) and player options (Projection Mode, Stereo 3D Mode, Screen Distance, Reset View).
  • Settings automatically persist to ~/.config/pixelvr/settings.conf across restarts.

F4. Variable-speed joystick fast-forward/rewind

  • Replace the discrete ±10 s edge-triggered seek (thumbstick X) with CONTINUOUS seek whose rate scales with deflection: rate = sign(x) * (deadzone(|x|))^2 * maxRate, applied per-frame (* dt). Show the seek speed on the HUD. Files: app.cpp (per- frame dt, continuous seek), media_engine (seekRelative exists; may want a scrub that doesn't thrash the decoder — coalesce seeks).

5. Known issues / notes

  • K1. HEVC 4K HW decode is broken — hevc_v4l2m2m floods output POLLERR + STREAMOFF on 2160p HEVC (e.g. the NAS "…2160p HEVC….mkv"); only a few frames decode. Many NAS movies are 4K HEVC. Needs SW fallback for HEVC/4K (overlaps B2).
  • Sharpness — partly the mipmap issue (B1); 1080p on a big VR screen is inherently soft (magnification). Re-judge after B1.
  • The pixelvr deploy target is still a wrapper (pixelvr → pixelvr.real, reads ~/media/which.txt). Restore a plain binary for shipping.
  • Temporary diagnostics: --probe / --xrext modes in main.cpp (harmless, keep or remove later). The green-bug per-frame diagnostics were already removed from media_engine.cpp.

6. Source map (where things live)

  • src/xr/xr_context.{hpp,cpp} — OpenXR instance/session/actions; XrInputState (buttons + controllers[2] aim poses); aim-pose action + action spaces; renderFrame.
  • src/gfx/vulkan_renderer.{hpp,cpp} — all Vulkan. NV12 pipeline, MSAA, pipelines (quad/sphere/ui/text), transport HUD, control bar, browser panel, controller lasers/ cursor, updatePointer raycast, screen placement (screenPivot_/Forward_/Yaw_/Distance_).
  • src/media/media_engine.{hpp,cpp} — FFmpeg PIMPL; HW/SW decode; process_video (NV12 direct-copy fix); SMB AVIO streaming; pause/seek/position; PulseAudio.
  • src/ui/browser.{hpp,cpp} — file browser model (local + SMB + mDNS, async download).
  • src/app/app.{hpp,cpp} — run loop, input handling, browser + control-bar wiring.
  • src/main.cpp — entry + --probe / --xrext diag modes.
  • src/util/math.hpp — Mat4/Quat/Vec3 + ray/billboard helpers.
  • shaders/ — quad (NV12 YUV→RGB), sphere (360/180), ui (solid), text (glyph atlas).
  • CMakeLists.txt — PIXELVR_OPENXR_TAG, shader list, FFmpeg/FreeType/smbclient/pulse.
Reference in new issue View Git Blame Copy Permalink