Swordpdf--ps5-framegen/docs/algorithm.md

# The algorithm

A generated frame is the picture at the moment halfway between two frames of the game: the previous one (A) and the current one (B).
For a pixel `x` of it and a motion `v` (half of the way the content moves from A to B), the content at `x` is where A shows it at
`x + v` and where B shows it at `x - v`. If `v` is right, those two samples agree, and their blend is the pixel. Everything below is
about finding a `v` that is right, and about what to show where there is none.

All of it runs in three groups of compute passes a frame, recorded by `interpolator::record()` (`src/ps5_framegen.cpp`).

## 1. Optical flow (FSR3's, unchanged)

The passes `of_*`: the luma of each frame, a 7-level pyramid of it, a scene change detection from luma histograms, and a block search
from the coarsest level down with a filter and an upscale at each level (FSR3's "v5" optical flow). The result is one motion per 8x8
block of B, in whole pixels, pointing from B to A.

Two changes, both in the callback headers:

- **Scene changes:** FSR3 counted the first 5 frames after a reset, and the 4 frames after a detected scene change, as scene
  changes, with no flow and no interpolation. Now only the frame of the change itself (and the first frame after a reset) is.
  The motion comes back a frame after a cut instead of four, so there is no judder after every cut.
- **Portability:** the search's reductions run in shared memory (`FFX_FG_PORTABLE_WAVE`), not in wave operations that assume
  32 or 64 lanes. Software devices have 4.

## 2. The vector field (FSR3's)

`fi_setup` and `fi_vector_field` push each block's motion, halved, to where the block is halfway. Where two blocks land on the same
place, the faster one wins. FSR3 keeps this field to sample a motion at any pixel. It is one candidate here among several.

## 3. The interpolation (`computeFrameinterpolationQuad`, one 2x2 quad a thread)

### Candidates

| Index | Candidate | Bias | Why |
|---|---|---|---|
| (zero) | no motion | 0.008 | the HUD, text, still scenery, a character the camera follows |
| 0 | the vector field, blended | 0 | FSR3's own answer |
| 1-4 | the field's 4 nearest cells, one by one | 0.004 | a blend of two motions at an edge is neither |
| 5 | the optical flow of the quad's block | 0.004 | the field may have put another block's motion here |
| 6-9 | the optical flow of the 4 blocks around it | 0.006 | at the edge of something moving, its own motion and the background's are both nearby |
| 10 | the quad's motion in the last generated frame | 0.004 | motion goes on; the choice stays the same from frame to frame |
| 11 | the quad's best one, refined | as its source | sub-pixel precision, see below |

A motion found twice, within a quarter of a pixel, is tried once. A motion under half a pixel counts as no motion.

### Score

For each candidate and pixel, the error is the sum of the three channels' differences between the two samples, from 0 to 3. The
score adds half of the quad's mean error, so that the quad decides together where its pixels cannot tell, and the candidate's bias.
Ties go to motion: a moving pattern can match "no motion" for a frame or two.

Penalties added to the bias:

- **Consistency (`FgUnlike`, 0.12):** what the candidate takes from B at `x - v`, and from A at `x + v`, should have moved by
  `2v` according to the optical flow there. The most alike of the 2x2 nearest blocks counts, which is lenient where two motions
  meet. Tolerance: 3 px plus 15% of the motion.
  - The reason: two dark samples agree just as well as the right ones. In GTA IV a pixel of Niko's ear took the background's
    motion from the block beside it, because A's dark wall and B's dark hair agreed better than the ear under the head's motion
    (the ear's shading changes as the head moves). The ear came out with a black speck.
- **Off screen (`FG_BORDER`, 0.06):** when one of the two samples falls off the screen, there is nothing to compare, so the
  sample on the screen is used alone at a middling score.
  - This is the strip that comes in or goes out at the edge of the screen as the camera turns. Before, that strip filled with
    whatever else happened to agree.

### Sub-pixel refinement

The flow is in whole pixels, so half vectors come in steps of half a pixel. Along a sharp edge moving a fraction of a pixel, the two
samples miss each other by up to a quarter of a pixel each, and the graffiti on GTA IV's walls shimmered along every line.

The quad's best candidate is tried half a pixel to each side on each axis. A parabola through the three errors gives the best
fraction, and that motion becomes one more candidate. Quads that already agree within 0.03 are left as they are.

### Choosing, per pixel

- **The best candidate:** each pixel takes its own best (the lowest score). If it agrees (error under `FG_MATCH`, 0.12), the pixel
  is the blend of its two samples.
- **Own match (m73f):** the quad's choice may not agree at one of its pixels while another motion does, at that pixel alone. That
  pixel belongs to what that motion moves, for example an ear in a quad that is mostly background, so it takes that motion. It is
  not an edge.
- **Edges:** where nothing agrees, something in front covers or uncovers what moves behind it. Of the best moving candidate's two
  samples, the one that another motion explains belongs to what is in front, so the other one is shown. "Another motion" means no
  motion, or the best moving candidate that differs by more than 2 px. When neither or both are explained, the pixel is the blend
  (FSR3's own fallback). A soft ramp between `FG_MATCH` and `FG_MATCH + FG_MATCH_SOFT` keeps this from flickering.
- **Still history:** a pixel that has not changed for 8 frames in a row (`rw_static_history`) shows no motion, whatever the
  candidates say.
  - The reason: inside a thin letter of a subtitle over a moving background, a moving candidate's two samples both land on the
    background and agree just as well. A single pair of frames cannot tell that from a pattern moving by its own period (a fence,
    bricks). The history can, because that coincidence does not last.
  - The history is cleared when the interpolator starts. Before m74 it started with whatever the memory held, and random pixels
    of moving things were held still.

### Other moments than halfway

With several frames generated (`set_generated`), the frame at moment `t` (0 is the previous frame, 1 the current one) takes A at
`x + 2t v` and B at `x - 2(1 - t) v`. At `t = 0.5` that is exactly `x + v` and `x - v`. The blend weighs the nearer frame more. The
vector field stays FSR3's halfway one for every moment: its candidates are a little off away from the middle, and the other
candidates are not tied to a moment. The still history and the quads' motions are updated by the first frame of the group only.

### Debug view

With `dispatchFlags` bit 2 (`set_debug_view(true)`), the output shows the decisions instead of the picture:

| Colour | Decision |
|---|---|
| grey | no motion |
| white | still by the history |
| green | a motion agreed |
| red / blue | an edge where A's / B's sample was kept |
| magenta | an edge, blended |

Built with `-DFG_DEBUG_VECTORS`, it shows the chosen motion in red and green and the candidate index in blue, for scripts.

## What was tried and dropped

| Tried | Result |
|---|---|
| A median of the edge pixels (m73f's cleanup pass) | The specks it was for were not edge pixels; it changed little else and cost a pass |
| A local search for quads where nothing agrees (for hands swinging as a character walks) | Spurious matches elsewhere made the whole frame worse, and the hands, which also rotate, were not recovered |
| Skipping the candidates for quads that did not change | A pattern moving by its own period looks unchanged too: the moving checkerboard of the "all" scene came out three times worse |
| Stopping after the vector field when it agrees | Cost precision, saved little |

They are still in the source behind test defines (`FG_QUIET_SKIP`, `FG_FIELD_EARLY`), or described in the history of RPCS3-PS5.

## Known limits

- **Small fast things that turn as they move** (hands, a thrown object): the flow's 8x8 blocks follow what is around them, and no
  single motion fits a rotating hand. They come out fragmented or faded in the generated frame. The real frames before and after
  are untouched, so at 60 fps or more it is hard to see.
- **What a camera turn uncovers at the screen's edge:** the flow has nothing there (the content was off screen in A). The on-screen
  sample is used when the right motion is among the candidates.
- **Without depth there is no order between two motions:** where two layers move differently, the edge test guesses which one is in
  front from which sample another motion explains.