MyLuxy--overdrive/ps5-react/docs/VIDEO.md

Video

<Video> plays a file or an http(s) stream full screen or in any box: H.264, HEVC (including 10-bit HDR) and VP9 up to 3840×2160 through the PS5's hardware decoder, and what FFmpeg decodes in software up to 1080p. Audio plays through the app's sound mixer and is the clock the picture follows.

import {useRef} from 'react';
import {Video} from '@ps5-react/core';

function Player({url, onClose}) {
  const video = useRef(null);
  return (
    <Video ref={video} source={{uri: url}} className="absolute inset-0"
      onLoad={({duration, naturalSize}) => console.log(duration, naturalSize)}
      onProgress={({currentTime}) => console.log(currentTime)}
      onEnd={onClose} onError={({error}) => console.log(error)} />
  );
}

Turn it on in app.json; streams also need networking:

{"video": true, "networking": true, "filesystemAccess": "console"}

Without "video": true the app links no FFmpeg (about 11 MB less in the PS5 build) and <Video> reports how to turn it on. apps/video-lab plays everything in its media/ folder and measures it.

Props

Prop	Meaning
source	An app path (/app0/..., /download0/...) or an http(s) URL, or {uri, headers} with request headers
paused, muted, volume (0–1), repeat	Playback controls; change them at any time
resizeMode	'contain' (default, letterboxed in black), 'cover' or 'stretch'
upscale	A network for animation that doubles a picture shown bigger than its source, before it is scaled to the box: 'artcnn' (or true; ArtCNN C4F16, the luma only, chroma stays bilinear) or 'anime4k-s', -m, -l, -vl, -ul (Anime4K Upscale CNN x2, on R'G'B'). Its passes are split over two display frames, so the picture shows one display frame (17 ms) late. Needs compute shaders (the PS5); elsewhere it scales as before. A model compiles the first time it is used (0.5 to 0.8 s on the PS5). One upscaled video at a time
compare	{upscale, split}: right of split (0 to 1 across the picture) the picture goes through the model upscale (any upscale value, or false/'' for none) instead of the element's own, so two models, or a model and none, show side by side on the same frame. That model runs whole in the display frame its picture first shows in, on top of the element's own model: a heavy pair drops frames while playing. A paused picture is upscaled once per model. Leave it out to end the comparison
startTime	Seconds to start at, for resuming
audioDelay	Seconds (-2 to 2) the sound plays later than the picture, to match a TV that shows the picture late; negative for earlier
audio	false decodes no audio and leaves the mixer to other players (default true)
audioOutput	'pcm' (default) decodes the track to the mixer; 'auto' sends AC-3, E-AC-3 and DTS to the TV or receiver undecoded when its EDID lists them; 'passthrough' sends them whatever the EDID says. See Audio output
onLoad({duration, currentTime, naturalSize, audioTracks})	Once, when the first picture or sound is ready
onProgress({currentTime, playableDuration, seekableDuration})	Every progressUpdateInterval ms (250) while playing
onBuffer({isBuffering}), onSeek({currentTime}), onEnd(), onError({error})	State changes
subtitles, subtitleTrack	Text subtitle files to add as tracks ([{uri, language, title}], read when the video opens), and the track shown: an index from subtitleTracks, or -1 (default) for none
onSubtitle(text)	Whenever the subtitle text showing changes; '' when none shows
onStatus(status)	The player's full status (below) whenever its state, pause, seek or track changes

The ref has seek(seconds), selectAudioTrack(index) (an index from audioTracks, or -1 for none) and getStatus(). The status has state ('loading', 'ready', 'buffering', 'ended', 'failed'), position, duration, buffered, width, height, videoCodec, audioCodec, decoder ('sceVideodec2' or 'ffmpeg'), audioTracks, audioTrack, shownFrames, droppedFrames, lateMs (how far, on average, a picture is shown past its time), audioClock (whether audio is the clock), audioPassthrough (whether the sink decodes the track), subtitleTracks ({index, language, title, codec, external}), subtitleTrack and subtitle.

Subtitles

Text subtitles show as the app draws them: onSubtitle gives the text showing now, its lines separated by \n, and the app renders it over the video like any other text. They come from the file's own text tracks (SubRip, ASS/SSA, mov_text, WebVTT), listed in subtitleTracks by their stream index, or from separate files (.srt, .ass, .vtt, by path or URL) added with subtitles, listed from index 1000 and loaded off the render thread. ASS styling and positioning are dropped; only the text is kept. Bitmap subtitles (PGS, DVD) are not read: a server can burn them into the picture. Choosing a track of the file reads the file again from the playhead, so its cues before the current read-ahead show.

<Video> is an <Image layer> the presenter fills with the picture: it is drawn by the GPU beneath the interface, and anything drawn over it (controls, gradients, text) composites as over any layer (IMAGES.md). Like any layer it needs a host that draws layers; both hosts do.

What plays

	PS5	Desktop preview
H.264 Baseline, Main, High, 8-bit	hardware, up to 3840×2176	FFmpeg
HEVC Main, Main 10	hardware, up to 3840×2176	FFmpeg
VP9 profile 0	hardware, up to 3840×2176	FFmpeg
Other H.264 (Hi10P, 4:2:2, 4:4:4), VP9 profile 2, MPEG-4 Part 2, MPEG-2, VC-1, MJPEG	FFmpeg, up to 1920×1088	FFmpeg
AV1	no (FFmpeg's decoder needs hardware; no dav1d yet)	no
Containers	MP4/MOV, Matroska/WebM, MPEG-TS, MPEG-PS, AVI, FLV, HLS	same
Audio	AAC, AC-3, E-AC-3, MP3, MP2, Opus, Vorbis, FLAC, ALAC, TrueHD, DTS, PCM	same

Bigger than 1080p and not taken by the hardware fails with an error that says so: a 4K picture decoded in software runs the app's heap out of memory. One video per hardware decoder plays at a time (one H.264, one HEVC, one VP9); another one of the same codec decodes in software when it fits.

Decoded audio of any layout and rate is mixed down to the mixer's 48 kHz stereo. AC-3, E-AC-3 and DTS can instead go to the TV or receiver undecoded, surround and all: see Audio output.

HDR10 and HLG are tone-mapped to SDR in the shader (BT.2100 to BT.709, reference white at 203 nits, a soft shoulder): the presenter outputs 8-bit SDR, so the TV does not switch to HDR. Dolby Vision profile 5 is not supported; other Dolby Vision streams play their HDR10 base layer.

Audio output

audioOutput picks where the track goes:

'pcm' (default)	Decoded and mixed down to 48 kHz stereo, with the app's sounds
'auto'	AC-3, E-AC-3 and DTS at 48 kHz pass through to the sink when its EDID lists the codec (an E-AC-3 decoder counts for AC-3, a DTS-HD one for DTS); everything else is decoded
'passthrough'	AC-3, E-AC-3 and DTS at 48 kHz pass through whatever the EDID says (for a sink whose EDID is wrong or unreadable); the rest is decoded

Passed through, the frames are packed as IEC 61937 bursts, byte for byte as FFmpeg's spdif muxer writes them, and the PS5 sends them on the HDMI output its disc player uses (sceAudioOutExConfigureOutput and sceAudioOutExOpen): AC-3 and DTS on a 48 kHz carrier, E-AC-3 on 192 kHz. DTS sends the core only: a DTS-HD MA or DTS:X track plays as its DTS core (5.1 at most), and DTS Express, which has no core, is decoded. TrueHD is always decoded: it does not pass through. If the console refuses the output, the track is decoded instead. Changing audioOutput reads the track again from the playhead, as changing the track does; status.audioPassthrough says which happened.

While a track passes through:

• Clock. The bursts the output has read are the clock, as decoded samples are, so pause, seek and buffering work the same. A seek starts the sound at the frame the target falls in (at most 32 ms before it), since a burst cannot be cut. The sink decodes after the console, which may delay the sound a frame or more; audioDelay corrects what remains.
• Volume. A bitstream has no volume: volume does nothing, the sink's volume is the one. muted (or volume 0) sends silence.
• Switching. Opening the output switches the HDMI audio mode, so the sink may take a moment to lock on, and so may switching between AC-3, E-AC-3 and DTS. A seek or pause keeps the output open. Pausing cuts a burst short, which the sink drops: a frame's sound (32 ms) may go missing there.
• App sounds. The app's sounds still play through the mixer's stereo output. Whether the console mixes them in while its HDMI output carries a bitstream is not established yet.

AudioOutput.capabilities() reads the connected sink's EDID now, so an app can decide what to ask a media server for before it opens a video:

import {AudioOutput} from '@ps5-react/core';

const sink = AudioOutput.capabilities();
// {known: true, name: 'AVR-X1700H',
//  formats: [{codec: 'pcm', channels: 8, sampleRates: [32000, ..., 192000], bitDepths: [16, 20, 24]},
//            {codec: 'ac3', channels: 6, sampleRates: [32000, 44100, 48000], maxBitrate: 640000},
//            {codec: 'eac3', channels: 8, sampleRates: [44100, 48000]}, {codec: 'truehd', ...}, ...],
//  passthrough: ['ac3', 'eac3', 'dts'], pcmChannels: 8}

Field	Meaning
known	An EDID was read. false on the desktop preview (and then formats and passthrough are empty)
name	The sink's EDID name, '' when it has none
formats	Every codec the EDID's audio descriptors list, merged per codec: codec ('pcm', 'ac3', 'eac3', 'dts', 'dtshd', 'truehd', 'aac', 'mp3', 'mp2', ...), the most channels, sampleRates in Hz, bitDepths for PCM, maxBitrate in bit/s where the descriptor gives one. A TV with only the EDID's basic audio flag lists two-channel PCM
passthrough	The codecs this host sends to this sink undecoded with audioOutput: 'auto': 'ac3', 'eac3', 'dts'
pcmChannels	The most PCM channels the sink takes (2 when it lists none). Decoded audio is still stereo

It reads the EDID through sceVideoOutGetHdmiRawEdid_ at each call (a request to a system service, not yet timed): call it when needed, not every frame. Through an AV receiver the EDID is the receiver's. Not yet validated on hardware inside an app: see HARDWARE.md.

How it plays

• Threads. Each player demuxes on one thread and decodes video and audio on two more; the render thread only picks the picture due and presents it. This is the one exception to "engine work on the render thread": decoding a 4K frame takes longer than a frame. Players are closed off the render thread too.
• Clock. The audio the mixer has played, less its output latency, is the clock; without audio the wall clock is. At each display frame the player shows the picture due within half a frame, at most one new one per display frame unless it is behind: a 60 fps video on a 60 Hz display shows each picture once. A display at 59.94 Hz still has to drop one picture of a 60.00 fps video about every 17 seconds.
• Read-ahead. 4 seconds of a local file, 30 seconds of a stream (at most 160 MiB), so a network stall plays through; FFmpeg's http reconnects. A stream that runs dry is 'buffering': the clock and the sound stop, and play again once 2 seconds are read ahead. When FFmpeg stops reconnecting, the player opens the stream again every 2 seconds where it broke off, for up to five minutes, then fails.
• End. Sound that ends before the picture hands the clock to the wall clock, so the last pictures still play.
• Hardware decoding (PS5). At launch, before the app's filesystem elevation, the host loads libSceVideodec2 and prepares a decoder each for H.264, HEVC and VP9, sized for 4K: a compute queue, its memory and workspace. After elevation the module will not load and none of that can be had. The decoders themselves are created the first time a stream needs one, because creating one starts threads and the elevation helper takes only a process with a single thread; creating one after elevation works. Between streams a decoder is reset. It writes each picture into one of sixteen frame buffers the player reads directly; a buffer is reused once the player has shown its picture. The player keeps eight decoded pictures ahead: a 4K keyframe holds the decoder for up to about 60 ms, three and a half display frames. A hidden VP9 frame (alt-ref) is decoded for reference and never shown. See licenses/video-decoder-NOTICE.txt for whose findings this follows.
• Upload. On the PS5 a frame goes to the GPU as one storage buffer the shader reads bytes from (about 1.5 ms for a 4K frame); the console's 2D texture uploads run at a fraction of that speed. The desktop uploads plane textures.

Memory (PS5)

A video app's param.json asks for the most flexible memory the console gives, 1 GiB instead of 448 MiB, and the app heap leaves 448 MiB of it outside: the three 4K decoders' workspaces take about 160 MiB. The heap is then 384 MiB. The frame buffers and decoder memory are direct memory (about 390 MiB for frames, of more than 9 GiB free).

Testing

apps/video-lab lists the files in its media/ folder (not in the repository; on the PS5, the title folder's media/) and the URLs in media/urls.txt (Label | URL per line), and shows the status on screen. A PS5 test deploy with dev/playlist.txt (file seconds [seek-to] per line) plays each file for that long, logs a line per file and quits. With a video open the PS5 host logs each player's state, decoder, shown and dropped frames and queue depths every two seconds, and the upload volume and time.

Cost (PS5)

The whole process, measured with getrusage while playing (the PS5 log line has it every two seconds): about 1.0 to 1.2 cores busy for 3840×2160 60 fps H.264, 0.8 for 3840×2160 24 fps HEVC 10-bit, 0.18 for 320×240 MPEG-4 decoded by FFmpeg, out of the console's eight cores. Most of the 4K cost is moving each picture to the GPU, 12 to 25 MB a frame.

Not yet

• HDR10 output: the scanout has to be registered as 10-bit BT.2100 PQ (EVO Player does it on AGC), and the OpenGL layer that owns it offers no way to ask for that.
• Bitmap subtitles (PGS, DVD), ASS styling.
• AV1 (dav1d), Dolby Vision profile 5.
• Multichannel PCM: decoded surround is mixed down to stereo; only passthrough carries surround.
• TrueHD and DTS-HD MA / DTS:X passthrough (DTS sends its core).
• Matching the display's refresh rate to the film (24p judder at 60 Hz).