diff --git a/docs/reference.md b/docs/reference.md index 8721b75..7b4d0e7 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -276,12 +276,12 @@ In gaze mode the 3D mouse's pointer goes where you look, and the mouse or the ke Deferred: it costs a lot of the headset's CPU and needs more work, so `install.sh` doesn't offer it. It still builds and runs, installed by hand, for working on it. -Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode a pinch clicks where you look when it opens; hold it and move the hand to correct the pointer first. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1`, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (a pinch clicks, a middle pinch right-clicks or scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". +Your hands show over the screens: where a tracked hand is between an eye and a screen, ft-screens lets that eye see the room through the screen. The same tracker detects pinches and grips, and with `POINTER_HANDS=1` in `~/.config/frametop.conf` they work the pointer. In gaze mode your eyes aim and your hands click, Vision Pro style: a pinch clicks where you were looking as it began, and the hand ray stays off (`POINTER_GAZE_HAND_RAY=1` brings it back beside the gaze). Pinch, hold and move the hand a few centimetres to nudge the pointer slowly onto the target, then let go to click there; the correction teaches the gaze tracker, as the mouse's do. Pinch and hold still for half a second to press and drag with the hand. A middle pinch right-clicks the same way. Without gaze mode a pinch is a press like the mouse's button, so a held pinch drags. A grip (closing the hand) presses and drags. To install it: `hands/run.sh install`. With `POINTER_HANDS=1` and gaze mode off, a bare hand raised and pointing also shows a ray that drives the pointer like a controller's laser (a pinch clicks, a middle pinch right-clicks or scrolls, reaching out and in pushes and pulls a carried panel), a controller used recently takes over its own hand only, and during a paused VR game hand tracking is parked rather than stopped, waking when the dashboard opens. Details: [hands/README.md](../hands/README.md), "Pinches and grips in the pointer". - `ft-camd` borrows XRService's camera buffers and publishes the four IR tracking cameras to `/run/user/UID/frametop-hands/cam-ring`. It runs on the host as `frametop-camd.service`, with file capabilities that `hands/run.sh install` sets through sudo, and it drops them once set up. A rebuild clears them: `hands/run.sh caps`. The Hand Recorder's installer (`hands/rec/install.sh`) sets them the same way. `hands/run.sh uncaps` takes them back while neither the services nor the Hand Recorder is installed. `hands/run.sh uninstall` and `hands/rec/install.sh uninstall` run it after removing their own part, and `uninstall.sh` always takes them back. - `ft-hands` runs in the `dev` container as `frametop-hands.service`. It finds and triangulates the hands, and publishes `hands` (read by ft-screens' cutouts) and `gestures` (pinches and grips, read by the pointer helper) next to the ring. - `HANDS_MODE` says when they run: `off` (the default) only after `ft-handsctl on`, until `ft-handsctl off` or SteamVR stops; `frametop` while the Frametop desktop runs; `always` whenever SteamVR runs, in games too. Frametop Input Settings' Hands page sets it and applies it at once (`hands/handsmode.py`), next to the hand cutouts' margin (`HANDS_MATTE_MARGIN`, live). The install links `ft-handsctl` into `~/.local/bin`. `ft-handsctl status` and `ft-handsctl log` (or `hands/run.sh status` and `log`) show how they're doing, `ft-handsctl cutouts on|off` turns just the cutouts off, and `ft-handsctl gestures` shows pinches and grips live. -- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (`auto`, the default: ft-hands tells from the hands when some SteamVR restart has swapped the side cameras' names, and fixes them; `0` or `1` force them, and `hands/tools/check_sides.py --ring` tells which is right), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, `POINTER_PINCH_TYPING`, and the hand ray's `POINTER_HAND_RAY`, `POINTER_RAY_DEADZONE`, `POINTER_RAY_PUSH`, `POINTER_RAY_SCROLL`, `POINTER_RAY_RELEASE`, `POINTER_RAY_GRACE`, its smoothing and taps (`POINTER_RAY_SMOOTH_MINCUTOFF`, `POINTER_RAY_SMOOTH_BETA`, `POINTER_RAY_SMOOTH_HOLD`, `POINTER_RAY_STEADY`, `POINTER_RAY_TAP`, `POINTER_RAY_TAP_DEADZONE`), and ft-screens' near grabs (`HANDS_GRAB_SMOOTH_MINCUTOFF`, `HANDS_GRAB_SMOOTH_BETA`, `HANDS_GRAB_DEADZONE`). The example config explains each. +- Settings in `~/.config/frametop.conf`: `HANDS_SWAP_SIDES` (`auto`, the default: ft-hands tells from the hands when some SteamVR restart has swapped the side cameras' names, and fixes them; `0` or `1` force them, and `hands/tools/check_sides.py --ring` tells which is right), `HANDS_CPUS`, the cameras it tracks with (`HANDS_CAMERAS`, `HANDS_BRIGHT`, `HANDS_BRIGHT_ON`, `HANDS_BRIGHT_OFF`, `HANDS_COLOR_LEFT`, `HANDS_COLOR_CROP`), and the pointer helper's `POINTER_HANDS`, `POINTER_PINCH_GAIN`, `POINTER_PINCH_DEADZONE`, `POINTER_GRIP_GAIN`, `POINTER_GRIP_BELOW`, `POINTER_PINCH_TYPING`, the gaze pinch's `POINTER_GAZE_HAND_RAY`, `POINTER_GAZE_PINCH_DEADZONE`, and `POINTER_GAZE_PINCH_GAIN`, the hand ray's `POINTER_HAND_RAY`, `POINTER_RAY_DEADZONE`, `POINTER_RAY_PUSH`, `POINTER_RAY_SCROLL`, `POINTER_RAY_RELEASE`, `POINTER_RAY_GRACE`, its smoothing and taps (`POINTER_RAY_SMOOTH_MINCUTOFF`, `POINTER_RAY_SMOOTH_BETA`, `POINTER_RAY_SMOOTH_HOLD`, `POINTER_RAY_STEADY`, `POINTER_RAY_TAP`, `POINTER_RAY_TAP_DEADZONE`), and ft-screens' near grabs (`HANDS_GRAB_SMOOTH_MINCUTOFF`, `HANDS_GRAB_SMOOTH_BETA`, `HANDS_GRAB_DEADZONE`). The example config explains each. Details, options, and the recording and replay tools are in [hands/README.md](../hands/README.md). diff --git a/hands/README.md b/hands/README.md index df7821c..39e112f 100644 --- a/hands/README.md +++ b/hands/README.md @@ -220,9 +220,9 @@ ft-hands detects a pinch per hand (`track/pinch.h`) and publishes it to the gest With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/ft-pointer.cpp`) reads the gestures file every frame. It's off by default. -**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`). +**The hand ray** (`POINTER_HAND_RAY=1`, the default with `POINTER_HANDS`): a bare hand raised in front and pointing, palm away, shows a ray from it, Quest-style (`include/fh_ray.h`: from a shoulder estimated from the head, through the knuckles), and the ray drives the pointer's virtual controller, so it works on SteamVR's dashboard, Steam's pages, overlays and Frametop's panels like a controller's laser. With the ray up, a pinch clicks (a held pinch drags), a middle pinch right-clicks (held and moved, it scrolls), and while a pinch carries a panel, reaching out pushes it away and pulling in brings it near. With the hand low or the palm toward you, there's no ray, and pinches and grips work as below. In gaze mode there's no ray at all unless `POINTER_GAZE_HAND_RAY=1`. A hand holding a controller, or whose controller was used in the last seconds, has no ray and no gestures (the helper's handoff, `pointer/helper/handoff.h`). The rules, timings and measurements are at the top of `ft-pointer.cpp` ("Handoff", "Hand ray") and in frame-hands' `notes/hand-interaction.md`. `pointer/tools/rayeval` runs the ray over a replay (`ft-handreplay --ray`). -- **Pinch to click.** In gaze mode a pinch works like the mouse's press: the pointer stops where the gaze put it, and the click comes when the pinch opens, where the pointer is then. A quick tap clicks where you looked. Held, the pinching hand moves the pointer to correct the gaze, and the correction is a lesson for the gaze tracker, as with the mouse. +- **Pinch to click, in gaze mode.** Your eyes aim and your hands click, Vision Pro style, and the hand ray stays off (`POINTER_GAZE_HAND_RAY`). A pinch locks the pointer where you were looking just before the fingers began to close (75 ms before ft-hands saw the pinch, since the eyes often move on as the hand closes), and a quick tap clicks there. Pinch, hold and move the hand to nudge the pointer: nothing moves until the hand has moved 4 degrees (about 3 cm at arm's length), then the pointer follows at 0.3 times the hand's movement. Letting go clicks where the pointer is, and the correction is a lesson for the gaze tracker, as with the mouse. Held still for `POINTER_GAZE_HOLD` (0.5 s) instead, the pinch is a real press at that point, and the hand drags at `POINTER_GRIP_GAIN`. A middle pinch is a right click the same way. The rules and timings are in `pointer/helper/gazehand.h`. - **Without gaze mode,** a pinch is a real press, like the mouse's button: pressed when it closes, released when it opens, and while it's held the hand drags the pointer. A tap is still a click where the pointer is. - **Grip to drag.** Closing the hand presses where the pointer is, the hand moves the pointer, and opening the hand releases. So a title bar moves its window, a panel's grab bar carries the panel, and text gets selected. - A pinch ended by losing the hand, or by a grip taking over, doesn't click. @@ -231,8 +231,11 @@ With `POINTER_HANDS=1`, the pointer helper (`pointer/helper/ft-pointer.cpp`) rea Settings in `~/.config/frametop.conf`: - `POINTER_HANDS` (0): 1 turns pinches and grips on. -- `POINTER_PINCH_GAIN` (0.5): a held pinch moves the pointer this many times the hand's angle, seen from the eye. Under 1 gives precision. -- `POINTER_PINCH_DEADZONE` (1.5): how many degrees the pinching hand moves before the pointer does, so a tap's jitter and the pinch point shifting as the fingers close don't move it. +- `POINTER_PINCH_GAIN` (0.5): without gaze mode, a held pinch moves the pointer this many times the hand's angle, seen from the eye. Under 1 gives precision. +- `POINTER_PINCH_DEADZONE` (1.5): without gaze mode, how many degrees the pinching hand moves before the pointer does, so a tap's jitter and the pinch point shifting as the fingers close don't move it. +- `POINTER_GAZE_HAND_RAY` (0): in gaze mode, 1 keeps the hand ray beside the gaze (a raised, pointing hand's pinches are then ray pinches). +- `POINTER_GAZE_PINCH_DEADZONE` (4): in gaze mode, how many degrees a held pinch moves before it nudges the pointer. A tap's jitter and a hand drifting while you look stay inside it. +- `POINTER_GAZE_PINCH_GAIN` (0.3): in gaze mode, past the dead zone the pointer moves this many times the hand's angle. Low, so a nudge is a fine correction. - `POINTER_GRIP_GAIN` (1): a grip moves the pointer this many times the hand's angle. - `POINTER_GRIP_BELOW` (0.35): grips that begin more than this many metres below the eyes are ignored, because hands resting on a desk curl like a loose fist. Pinches have no such limit: deliberate ones sat 0.35-0.45 m below the eyes with the elbow resting. - `POINTER_PINCH_TYPING` (1): no pinch begins within this many seconds of a key press, because typing touches thumb to index. diff --git a/input-settings/ft_input_settings.py b/input-settings/ft_input_settings.py index 7d061e3..c7f1499 100644 --- a/input-settings/ft_input_settings.py +++ b/input-settings/ft_input_settings.py @@ -169,6 +169,9 @@ GAZE_SETTINGS = [ ("POINTER_GAZE_NUDGE_MAX", "Largest correction to learn", 55, 1, 110, 1, "°"), ("POINTER_GAZE_HOLD", "Hold still to drag", 0.5, 0.1, 2.0, 0.05, "s"), ("POINTER_GAZE_SHOW", "Dot shows after moving", 1.0, 0.0, 5.0, 0.1, "s"), + # with hand tracking (POINTER_HANDS): how far a held pinch moves before it nudges, and how fast + ("POINTER_GAZE_PINCH_DEADZONE", "Pinch nudge starts after", 4.0, 1.0, 10.0, 0.5, "°"), + ("POINTER_GAZE_PINCH_GAIN", "Pinch nudge speed", 0.3, 0.1, 1.0, 0.05, "×"), ] # In gaze mode, what the mouse's left button does (POINTER_GAZE_MOUSE). GAZE_MOUSE = {"precision": "Gaze precision: hold to steer with the mouse, release to click", diff --git a/input-settings/main.qml b/input-settings/main.qml index 282ca27..26b57a0 100644 --- a/input-settings/main.qml +++ b/input-settings/main.qml @@ -1273,7 +1273,10 @@ Kirigami.ApplicationWindow { + "with the button held and let go there. Hold still to drag: hold a press this long without moving " + "to drag something instead. Look away to hand back: how far from the pointer you look before the " + "gaze takes it back from the mouse. Largest correction to learn: a click corrected further than " - + "this isn't learned; the tracker is that far off, so the quick check runs instead. Eye tracker: SteamVR's, or our own " + + "this isn't learned; the tracker is that far off, so the quick check runs instead. With hand tracking, " + + "a pinch clicks where you look; pinch, hold and move the hand to nudge the pointer first. Pinch nudge " + + "starts after: how far the hand moves before the pointer does. Pinch nudge speed: how far the pointer " + + "moves for each degree of hand movement after that. Eye tracker: SteamVR's, or our own " + "(gaze/tracker), which keeps its own calibration. Eye bias: the gaze " + "combines both eyes, since their errors partly cancel; Left or Right counts that eye twice as " + "much, and Auto weights each by how far off it was at your recent nudges." diff --git a/pointer/helper/ft-pointer.cpp b/pointer/helper/ft-pointer.cpp index ccf62a3..a52be3a 100644 --- a/pointer/helper/ft-pointer.cpp +++ b/pointer/helper/ft-pointer.cpp @@ -240,16 +240,14 @@ // // Hands (POINTER_HANDS, off by default; needs hand tracking, hands/): ft-hands publishes // pinches and grips (hands/include/fh_gestures.h), read here every frame. -// A pinch works like gaze mode's mouse press: the pointer stops (where the gaze put it), and -// the click comes when the pinch opens, where the pointer is then. A quick tap clicks where -// you looked. Held, the pinch's hand moves the pointer, for correcting the gaze: past -// POINTER_PINCH_DEADZONE (1.5 deg of hand movement, seen from the eye; a tap's jitter and the -// pinch point shifting as the fingers close stay inside it), at POINTER_PINCH_GAIN (0.5: half -// the hand's angle, for precision). A correction is a lesson for the gaze tracker, as with the -// mouse. A pinch ended by losing the hand (or by a grip taking over) doesn't click. -// Without gaze mode, a pinch is a real press instead, like the mouse's button: pressed when -// it closes and released when it opens, and held, its hand drags the pointer (at the same gain -// and past the same dead zone). A tap is still a click where the pointer is. That's what the +// In gaze mode a pinch clicks where you looked, and held and moved, its hand nudges the pointer +// first (see "Gaze pinch" below). A pinch ended by losing the hand (or by a grip taking over) +// doesn't click. +// Without gaze mode, a pinch is a real press, like the mouse's button: pressed when it closes +// and released when it opens, and held, its hand drags the pointer: past POINTER_PINCH_DEADZONE +// (1.5 deg of hand movement, seen from the eye; a tap's jitter and the pinch point shifting as the +// fingers close stay inside it), at POINTER_PINCH_GAIN (0.5: half the hand's angle, for +// precision). A tap is still a click where the pointer is. That's what the // gaze probe's Click practice wants: it has its own gaze dot, frozen by the press and dragged // by the pointer's movement until the release. // A grip (closing the hand) is a press and drag: the press where the pointer is, then the @@ -346,6 +344,32 @@ // aren't allowed lets it go at once, unless the mouse is in use. In gaze mode the gaze takes the // pointer back when the ray goes. // +// Gaze pinch (gaze mode with POINTER_HANDS; gazehand.h has the rules and the timings): the eyes aim +// and the hands click, Vision Pro style. In gaze mode the hand ray stays off unless +// POINTER_GAZE_HAND_RAY=1 (then a raised, pointing hand's pinches are still ray pinches, as above), +// so pinches and middle pinches are gaze pinches. ft-screens' near poke and grab, and its palm-up +// Steam menu, are the same as without gaze (its near claims come first). +// A pinch holds its press back at the gaze from just before the fingers began to close (the gaze +// lock: the newest gaze that came 75 ms or more before the begin's capture), and the pointer goes +// there, off the gaze. A tap clicks there when it opens. Held and moved, the hand nudges the +// pointer, with low sensitivity: nothing moves until the pinch point has moved +// POINTER_GAZE_PINCH_DEADZONE (4 deg, seen from the eye; about 3 cm at arm's length), then the +// pointer follows at POINTER_GAZE_PINCH_GAIN (0.3 times the hand's angle) from where it is, so it +// doesn't jump as the nudge engages. The release clicks where the pointer is, and the correction +// (the locked gaze to the click) is a lesson for the gaze tracker, or past POINTER_GAZE_NUDGE_MAX a +// quick check, as with the mouse. +// Held still (inside the dead zone) for POINTER_GAZE_HOLD (0.5 s), the pinch becomes a real press +// at the locked point, and the hand drags (at POINTER_GRIP_GAIN, 1, past 0.5 deg) until it opens. +// The hand drives the drag, not the gaze, as the mouse and the head do in gaze mode's other holds: +// the eyes jump about and look ahead to where a thing should go, so a dragged window would jump +// with them, and a drop would land wherever you last glanced. A pinch that has nudged only clicks. +// A middle pinch is the right button the same way: a right click at the locked gaze, nudged +// first if moved; it never becomes a press. No pinch begins within POINTER_PINCH_TYPING of a key, +// and the rules for all gestures hold (hands allowed, a controller on that side, the near claims). +// Each pinch logs one line: "pinch right: clicked at the gaze on frametop.screen.0 (gaze 80 ms +// before, held 0.18 s)", "nudged 2.4 deg, then clicked on ... (a lesson; ...)", "became a press on +// ... (held still 0.50 s; ...), dragged 6.0 deg, let go", or why not. +// // Placement (for layout): SteamVR keeps a floating panel's position inside the // dashboard, where nothing outside can set it, so the helper carries panels like a user // would. It measures the panel (md::ScanPanel), aims the device at its grab bar @@ -402,11 +426,14 @@ // POINTER_RAY_RELEASE (2 s), POINTER_RAY_GRACE (0.5 s), POINTER_RAY_SMOOTH_MINCUTOFF (0.8 Hz), // POINTER_RAY_SMOOTH_BETA (0.06), POINTER_RAY_SMOOTH_HOLD (0.5 Hz), POINTER_RAY_STEADY (1), // POINTER_RAY_TAP (0.3 s), POINTER_RAY_TAP_DEADZONE (6 deg): the hand ray, above. +// POINTER_GAZE_HAND_RAY (0), POINTER_GAZE_PINCH_DEADZONE (4 deg), POINTER_GAZE_PINCH_GAIN (0.3): gaze +// pinches, above. // POINTER_GAZE_MOUSE (precision), POINTER_HEAD_DEADZONE (0.5 deg), POINTER_KEY_TAP (0.25 s), // POINTER_ROLE (right): gaze precision and keyboard clicks, above. #include #include "../../hands/include/fh_ray.h" +#include "gazehand.h" #include "handoff.h" #include "handsin.h" #include "rayowner.h" @@ -831,6 +858,10 @@ int main() { fhray::Params rayParams; bool rayParamsNew = true; double rayTap = 0.3, rayTapDeadzone = 6.0; + // Gaze pinches (see "Gaze pinch" at the top): POINTER_GAZE_HAND_RAY, POINTER_GAZE_PINCH_DEADZONE, + // POINTER_GAZE_PINCH_GAIN. + bool gazeHandRay = false; + double gazePinchDeadzone = 4.0, gazePinchGain = 0.3; // Gaze precision (see the top): POINTER_GAZE_MOUSE (precision: the mouse's left button // holds back its press in gaze mode; direct: it clicks at once), POINTER_ROLE. bool gazeMousePrecision = true; @@ -883,6 +914,9 @@ int main() { rayParamsNew = true; rayTap = std::clamp(ConfDouble(conf, "POINTER_RAY_TAP", 0.3), 0.0, 2.0); rayTapDeadzone = std::clamp(ConfDouble(conf, "POINTER_RAY_TAP_DEADZONE", 6.0), 0.0, 30.0); + gazeHandRay = ConfDouble(conf, "POINTER_GAZE_HAND_RAY", 0) != 0; + gazePinchDeadzone = std::clamp(ConfDouble(conf, "POINTER_GAZE_PINCH_DEADZONE", 4.0), 0.0, 15.0); + gazePinchGain = std::clamp(ConfDouble(conf, "POINTER_GAZE_PINCH_GAIN", 0.3), 0.05, 2.0); const auto gm = conf.find("POINTER_GAZE_MOUSE"); gazeMousePrecision = gm == conf.end() || gm->second != "direct"; const auto mm = conf.find("POINTER_GAZE_MOUSE_MOVE"); @@ -1089,7 +1123,15 @@ int main() { double h0 = 0, d0 = 0, scrollX = 0, scrollY = 0; Vec3 origin; double refYaw = 0, refPitch = 0, startYaw = 0, startPitch = 0, lastYaw = 0, lastPitch = 0; + gazehand::Pinch gaze; // a gaze pinch (see "Gaze pinch" at the top), when gaze.On() } hold; + // Gaze pinches (see "Gaze pinch" at the top): the gaze as it came, for the lock; a click one is + // due (its log line goes out with the click, gazeClickSide: its side); why the held one ended; + // what pressLeft sent the gaze service ("a lesson", "the quick check", or ""). + gazehand::GazeHistory gazeHist; + gazehand::Pinch gazeClick; + int gazeClickSide = 0; + std::string gazeEndWhy, lessonSaid; // "precision|gazedrag 1|0" from the relay, done in the frame (see Holds). struct DevicePress { std::string source; @@ -1242,6 +1284,7 @@ int main() { // ft-screens sends the keyboard to the panel clicked last; it sees clicks on // its own screens, but only we know when one lands on another panel. SendTo(out, "ft_screens", "click " + (lastHit.empty() ? std::string("-") : lastHit)); + lessonSaid.clear(); // A click after nudging the gaze-placed pointer: the nudge is a lesson, or past // POINTER_GAZE_NUDGE_MAX, a quick check (see the top). if (gazeOn && nudging && !gazeOwns && havePoint && Clock::now() - nudgeAt < std::chrono::seconds(10) && @@ -1255,6 +1298,7 @@ int main() { else std::snprintf(msg, sizeof msg, "recheck %.0f", off); SendTo(out, "ft_gazed", msg); + lessonSaid = off <= gazeNudgeMax ? "a lesson" : "the quick check"; if (debug) std::printf("gaze %s (nudged %.2f deg, off %.2f)\n", msg, nudgeMoved, off); if (debug) std::fflush(stdout); } @@ -1654,6 +1698,7 @@ int main() { double g[4]; if (std::sscanf(buf, "gz %lf %lf %lf %lf", &g[0], &g[1], &g[2], &g[3]) == 4) { gz = {g[0], g[1], g[2], g[3], Clock::now()}; + gazeHist.Add({secsOf(gz.at), g[0], g[1], g[2], g[3]}); // for a gaze pinch's lock continue; } // The gaze calibration panel (see the top): presses answer it instead of clicking. @@ -2074,7 +2119,8 @@ int main() { // The hand ray (see "Hand ray" at the top): each side's hand, when a new frame came. const bool mouseIdle = tnow - lastMouse >= std::chrono::seconds(1); auto mayPoint = [&](int s) { - return rayOn && handsAllowed && mouseIdle && arbiter.State(s) == handoff::Owner::Hand; + return gazehand::RayAllowed(rayOn, gazeOn, gazeHandRay) && handsAllowed && mouseIdle && + arbiter.State(s) == handoff::Owner::Hand; }; if (rayParamsNew) { rayParamsNew = false; @@ -2103,7 +2149,7 @@ int main() { // from 20 cm behind the index tip along the finger meets one within 25 cm (the tip // within 5 cm of it). Only while the ray could show, so it costs nothing otherwise. f.near = tnow - nearAt < std::chrono::milliseconds(200) && nearSide[s]; - if (!f.near && active && (rays[s].Shown() || rays[s].LastTests().show)) { + if (!f.near && active && mayPoint(s) && (rays[s].Shown() || rays[s].LastTests().show)) { const Vec3 tip = h->pts[8], fd = Normalize(h->pts[8] - h->pts[7]), from = tip - fd * 0.2; for (const auto &[key, handle] : handles) { if (!visible[key] || sceneGraph[key]) continue; @@ -2166,7 +2212,20 @@ int main() { hp = std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI; return true; }; + // A gaze pinch ended (see "Gaze pinch" at the top): its log line now, or a click's once it goes out. + auto gazePinchEnds = [&](bool lost) { + if (!lost && !hold.pressed && aimHeld) { + gazeClick = hold.gaze, gazeClickSide = hold.side; + } else { + const std::string why = gazeEndWhy.empty() ? "the gaze calibration panel came up" : gazeEndWhy; + std::printf("%s %s: %s\n", hold.gaze.Right() ? "middle pinch" : "pinch", hold.side ? "right" : "left", + hold.gaze.Says(secsOf(tnow), lost, why, "", "").c_str()); + std::fflush(stdout); + } + gazeEndWhy.clear(); + }; auto endHold = [&](bool lost) { + if (hold.gaze.On()) gazePinchEnds(lost); if (hold.pressed) { if (leftHeld) releaseLeft(); } else if (aimHeld) { @@ -2237,10 +2296,11 @@ int main() { if (hold.src != Src::None) { // A grip takes over a pinch on the way to it (closing the hand passes through // one); otherwise one hold at a time. - if (hold.src != Src::Hand || !grip || hold.grip) { + if (hold.src != Src::Hand || !grip || hold.grip || hold.pressed) { if (!grip) pinchLog(side, 0, "ignored (no ray up; a press is already held)"); return; } + if (hold.gaze.On()) gazeEndWhy = "a grip took over", gazePinchEnds(true); // its log line aimHeld = aimHand = false; hold = {}; } @@ -2473,12 +2533,54 @@ int main() { return r >= 0 ? r : s; }; auto canRay = [&](int rs) { return rays[rs].CanShowForPinch(secsOf(tnow), rayGrace); }; + // A gaze pinch (see "Gaze pinch" at the top): kind 0 a pinch, 2 a middle pinch (the right button). + // The press is held back at the gaze locked from just before the fingers began to close. + auto beginGazePinch = [&](int side, int kind, const fh_pinch_t &g) { + handUsed = tnow; + gazehand::Situation sit; + sit.busy = hold.src != Src::None || leftHeld || aimHeld || clickPress || clickRelease; + sit.active = active; + sit.sinceKey = std::chrono::duration(tnow - lastTyping).count(); + sit.typingHold = typingHold; + if (const char *why = gazehand::Refused(sit)) { + if (!sit.busy && !active) wake(tnow); // the first gesture only wakes the pointer + pinchLog(side, kind, why); + return; + } + vr::HmdMatrix34_t head; + if (!poses.At(g.begin_ns, head)) return; + Hold h; + h.src = Src::Hand, h.side = side, h.right = kind == 2, h.origin = Position(head); + if (!handAngles(g.begin_point, g.begin_ns, h.origin, h.refYaw, h.refPitch)) return; + gazehand::Params gp; + gp.deadzone = gazePinchDeadzone, gp.gain = gazePinchGain, gp.hold = gazeHold, gp.dragGain = gripGain; + gazehand::GazeSample gs; + double lockAge = -1; + vr::HmdMatrix34_t gazeHead; + const bool locked = gazeHist.Lock(double(g.begin_ns) / 1e9, gp, gs, lockAge) && + poses.At(uint64_t(gs.t * 1e9), gazeHead); + if (locked) { + const Vec3 d = Rotate(gazeHead, Direction(gs.hy, gs.hp)); + yaw = std::atan2(-d.x, -d.z) * 180 / M_PI; + pitch = std::clamp(std::asin(std::clamp(d.y, -1.0, 1.0)) * 180 / M_PI, -85.0, 85.0); + anchor = eye, anchored = true; + } + hold = h; + startHold(false); + // the lesson's look is the locked one (ft-gazed finds it in what it sent) + if (locked) nudging = true, nudgeRawHy = gs.rhy, nudgeRawHp = gs.rhp, nudgeHead = gazeHead; + hold.gaze.Begin(gp, secsOf(tnow), yaw, pitch, h.refYaw, h.refPitch, kind == 2, locked ? lockAge : -1); + if (debug) std::printf("hand %s %s began at the gaze\n", side ? "right" : "left", kind ? "middle pinch" : "pinch"); + if (debug) std::fflush(stdout); + }; auto startGesture = [&](int s, int kind, const fh_pinch_t &g, bool ray) { const int rs = rayFor(s, g); // (and still: a ray that went for good while ft-screens was asked isn't; a grip needs it up) ray = ray && (kind == 1 ? rays[rs].Shown() : canRay(rs)); if (ray && kind != 1) { beginRayHold(s, rs, kind, g); + } else if (gazeOn && kind != 1) { + beginGazePinch(s, kind, g); } else if (ray || kind == 2) { // a fist with the ray up, or a middle pinch without it: nothing if (kind == 2) pinchLog(s, kind, "ignored (no ray up)"); @@ -2490,11 +2592,14 @@ int main() { auto holdingKind = [&](int s, int kind) { if (hold.side != s) return false; if (hold.src == Src::Ray) return hold.rayKind == kind; - return hold.src == Src::Hand && hold.grip == (kind == 1); + return hold.src == Src::Hand && hold.grip == (kind == 1) && hold.right == (kind == 2); }; auto endGesture = [&](bool lost, const char *why) { if (hold.src == Src::Ray) endRayHold(lost, why); - else endHold(lost); + else { + if (hold.gaze.On()) gazeEndWhy = why; + endHold(lost); + } }; if (rayHoldCancel) { // the driving ray went (a controller, the mouse, hand input off): its hold ends, as lost rayHoldCancel = false; @@ -2571,10 +2676,17 @@ int main() { // The held gesture's hand moves the pointer: past the dead zone (a tap's jitter, // the pinch point shifting as the fingers close), then at the gain, from there. if (hold.src == Src::Hand) { - const fh_pinch_t &g = hold.grip ? hg.grip[hold.side] : hg.pinch[hold.side]; + const fh_pinch_t &g = hold.grip ? hg.grip[hold.side] : hold.right ? hg.pinch2[hold.side] : hg.pinch[hold.side]; double hy, hp; - if ((g.flags & FH_PINCH_TRACKED) && handAngles(g.point, hg.capture_ns, hold.origin, hy, hp)) - steer(hy, hp, hold.grip ? 0.5 : pinchDeadzone, hold.grip ? gripGain : pinchGain); + if ((g.flags & FH_PINCH_TRACKED) && handAngles(g.point, hg.capture_ns, hold.origin, hy, hp)) { + if (!hold.gaze.On()) { + steer(hy, hp, hold.grip ? 0.5 : pinchDeadzone, hold.grip ? gripGain : pinchGain); + } else if (hold.gaze.Hand(hy, hp)) { // a gaze pinch: nudged, or dragged once pressed + yaw = hold.gaze.Yaw(), pitch = hold.gaze.Pitch(); + if (!hold.pressed) nudgeMoved = hold.gaze.Moved(); + lastMove = tnow; + } + } } // A middle pinch held and moved: scrolling, the stick deflected by the hand's displacement // from the begin (0.5 cm dead zone, full at 4 cm): up and down, and across the ray. @@ -3124,6 +3236,16 @@ int main() { if (debug) std::printf("hold key %s: pressed (held still)\n", hold.right ? "right" : "left"); if (debug) std::fflush(stdout); } + // A gaze pinch held still for POINTER_GAZE_HOLD: a real press at the locked gaze, then the hand + // drags (see "Gaze pinch" at the top). + if (hold.src == Src::Hand && hold.gaze.On() && aimHeld && hold.gaze.Promote(secsOf(tnow))) { + aimHeld = aimHand = false; + hold.pressed = true; + gazeBack = gazeOn; + pressRight = false; + pressLeft(); + hold.gaze.pressHit = lastHit; + } if (clickPress) { clickPress = false; pressLeft(); @@ -3133,6 +3255,13 @@ int main() { clickRelease = false; releaseLeft(); } + // A gaze pinch's click went out (or was dropped with the pointer): its log line. + if (gazeClick.On() && !clickPress) { + std::printf("%s %s: %s\n", gazeClick.Right() ? "middle pinch" : "pinch", gazeClickSide ? "right" : "left", + gazeClick.Says(secsOf(tnow), !clickRelease, "the pointer went", lastHit, lessonSaid).c_str()); + std::fflush(stdout); + gazeClick = {}; + } controllerButtons.Poll( [&](const char *button, bool down) { diff --git a/pointer/helper/gazehand.h b/pointer/helper/gazehand.h new file mode 100644 index 0000000..d192d34 --- /dev/null +++ b/pointer/helper/gazehand.h @@ -0,0 +1,206 @@ +// Gaze + hands: with gaze mode and POINTER_HANDS both on, the eyes aim and the hands click (Vision +// Pro style), and a pinch held and moved nudges the pointer with low sensitivity. The pointer helper's +// side of "Gaze pinch" in ft-pointer.cpp. Header-only and free of OpenVR, so +// pointer/tests/gazehand-test.cpp can drive it with made-up gaze and hand samples. +// +// Ray: in gaze mode the hand ray stays off (POINTER_GAZE_HAND_RAY=0, the default), so every pinch is +// a gaze pinch; 1 keeps the ray beside the gaze (RayAllowed). +// +// Gaze lock: where a pinch clicks. ft-hands sees a pinch once the tips are within begin_m, and by the +// time it's read (its latency, and up to 60 ms waiting for ft-screens' near report) the eyes may have +// moved on: to the next target, or down to the hand. So the click goes where you looked just before +// the fingers began to close: the newest gaze sample that came at least lead (75 ms) before the +// begin's capture time. ft-gazed's samples come with the eye tracker's own delay on top, so the look +// itself is a little older still, and its fixation lock (the running mean of the fixation) holds +// steady through a fixation, so a sample anywhere in it will do. None within stale (0.15 s) before +// that (a blink, the service stopped): the oldest one between then and the begin; none at all: no +// lock, and the pointer stays where the gaze last put it. +// +// Nudge: the pinch point's direction from the eye (in the room, from where the eye was at the +// begin), against where it was at the begin. Nothing moves until it has moved deadzone (4 deg, +// POINTER_GAZE_PINCH_DEADZONE; about 3 cm at arm's length): a tap's jitter, the pinch point shifting +// as the fingers close, and a hand drifting while you look don't nudge. Past it, the pointer follows +// at gain (0.3, POINTER_GAZE_PINCH_GAIN) times the hand's angle from that point on, so it doesn't +// jump when the nudge engages. +// +// Hold: a pinch held inside the dead zone for hold (POINTER_GAZE_HOLD, 0.5 s) becomes a real press +// at the locked point, and the hand drags from there: past dragDeadzone (0.5 deg) at dragGain +// (POINTER_GRIP_GAIN, 1). A pinch that has nudged only clicks, as the mouse's held-back press does +// once it has moved. A middle pinch (the right button) never becomes a press. +// +// Times are seconds on one steady clock (CLOCK_MONOTONIC: ft-hands' capture times as seconds). +// Angles are degrees: the gaze's head-relative as ft-gazed sends it (yaw +left, pitch +up), the +// pointer's and the hand's in the room. +#pragma once + +#include +#include +#include +#include +#include + +namespace gazehand { + +struct Params { + double deadzone = 4.0; // POINTER_GAZE_PINCH_DEADZONE: hand degrees before a nudge + double gain = 0.3; // POINTER_GAZE_PINCH_GAIN: pointer degrees per hand degree + double hold = 0.5; // POINTER_GAZE_HOLD: held still this long, a real press + double dragDeadzone = 0.5; // after the press (the grip's) + double dragGain = 1.0; // POINTER_GRIP_GAIN + double lead = 0.075; // gaze lock: the newest sample at least this long before the begin + double stale = 0.15; // ... and at most this much older than that +}; + +// The hand ray may show: with gaze mode on, only with POINTER_GAZE_HAND_RAY=1. +inline bool RayAllowed(bool rayOn, bool gazeOn, bool gazeHandRay) { return rayOn && (!gazeOn || gazeHandRay); } + +// Why a gaze pinch doesn't begin, for its log line, or nullptr. The helper has already checked +// what holds for every gesture (hands allowed, a controller on that side, ft-screens' near claims). +struct Situation { + bool busy = false; // a press or a hold is already going (the mouse's, a key's, a hand's) + bool active = true; // the pointer is awake (the first gesture only wakes it) + double sinceKey = 1e9; // seconds since the relay's last "typing" + double typingHold = 1.0; // POINTER_PINCH_TYPING +}; +inline const char *Refused(const Situation &s) { + if (s.busy) return "ignored (a press is already held)"; + if (!s.active) return "it woke the pointer, no click"; + if (s.sinceKey < s.typingHold) return "ignored (typing)"; + return nullptr; +} + +// One gaze sample as it came from ft-gazed: corrected and raw, head-relative; t is when it came. +struct GazeSample { + double t = 0, hy = 0, hp = 0, rhy = 0, rhp = 0; +}; + +class GazeHistory { +public: + void Add(const GazeSample &s) { + h_.push_back(s); + while (!h_.empty() && h_.front().t < s.t - 1.0) h_.pop_front(); + } + // The gaze a pinch that began at tBegin clicks at (see "Gaze lock" at the top). age: how long + // before the begin it came. + bool Lock(double tBegin, const Params &p, GazeSample &out, double &age) const { + const double want = tBegin - p.lead; + const GazeSample *pick = nullptr; + for (auto it = h_.rbegin(); it != h_.rend(); ++it) { + if (it->t > want + 1e-9) continue; + if (it->t >= want - p.stale - 1e-9) pick = &*it; + break; + } + if (!pick) + for (const GazeSample &s : h_) + if (s.t >= want - p.stale - 1e-9 && s.t <= tBegin + 1e-9) { + pick = &s; + break; + } + if (!pick) return false; + out = *pick, age = tBegin - pick->t; + return true; + } + void Clear() { h_.clear(); } + +private: + std::deque h_; +}; + +// One gaze pinch, from its begin to its end. +class Pinch { +public: + enum class Phase { Off, Aim, Nudge, Press }; + + // A pinch began at t: the pointer locked at (yaw, pitch), the pinch point seen from the eye at + // (hy, hp). right: a middle pinch. lockAge: the gaze lock's age (s), < 0 without one. + void Begin(const Params &p, double t, double yaw, double pitch, double hy, double hp, bool right, double lockAge) { + *this = Pinch(); + p_ = p, phase_ = Phase::Aim, right_ = right, t0_ = t, lockAge_ = lockAge; + yaw_ = yaw, pitch_ = pitch; + refY_ = lastHy_ = hy, refP_ = lastHp_ = hp; + } + bool On() const { return phase_ != Phase::Off; } + Phase State() const { return phase_; } + bool Right() const { return right_; } + double Yaw() const { return yaw_; } + double Pitch() const { return pitch_; } + double Moved() const { return moved_; } // the nudge's path (deg), before any press + double Dragged() const { return dragged_; } // the drag's path (deg), after the press + double LockAge() const { return lockAge_; } + double Began() const { return t0_; } + std::string pressHit; // what the press landed on (the helper's, for the log) + + // A hands frame: where the pinch point is now, seen from the eye at the begin. True when the + // pointer moved (Yaw, Pitch). + bool Hand(double hy, double hp) { + if (phase_ == Phase::Off) return false; + lastHy_ = hy, lastHp_ = hp; + const bool press = phase_ == Phase::Press; + const double dy = std::remainder(hy - refY_, 360.0), dp = hp - refP_; + if (!engaged_) { + if (std::hypot(dy, dp) <= (press ? p_.dragDeadzone : p_.deadzone)) return false; + engaged_ = true; + refY_ = hy, refP_ = hp; // from here, so the pointer doesn't jump + startY_ = yaw_, startP_ = pitch_; + if (!press) phase_ = Phase::Nudge; + return false; + } + const double gain = press ? p_.dragGain : p_.gain; + const double ny = std::remainder(startY_ + gain * dy, 360.0); + const double np = std::clamp(startP_ + gain * dp, -85.0, 85.0); + const double step = std::hypot(std::remainder(ny - yaw_, 360.0), np - pitch_); + if (step <= 0) return false; + (press ? dragged_ : moved_) += step; + yaw_ = ny, pitch_ = np; + return true; + } + + // Each frame: true once, when a pinch held inside the dead zone for hold becomes a real press. + bool Promote(double t) { + if (phase_ != Phase::Aim || right_ || t - t0_ < p_.hold - 1e-9) return false; + phase_ = Phase::Press; + engaged_ = false; + refY_ = lastHy_, refP_ = lastHp_; // the drag counts from the hand now + pressAt_ = t; + return true; + } + + // What it did, for its log line ("pinch right: " + this). lost: it ended without a click (why + // says what); hit: what the click landed on; lesson: what the helper sent the gaze service for + // it ("a lesson", "the quick check", or "" for nothing). + std::string Says(double t, bool lost, const std::string &why, const std::string &hit, const std::string &lesson) const { + char b[320]; + const std::string on = hit.empty() ? "nothing" : hit; + char gaze[48] = "no fresh gaze: where the pointer was"; + if (lockAge_ >= 0) std::snprintf(gaze, sizeof gaze, "gaze %.0f ms before", lockAge_ * 1000); + const double held = t - t0_; + const char *button = right_ ? "right click" : "clicked"; + if (phase_ == Phase::Press) { + const std::string pressed = pressHit.empty() ? "nothing" : pressHit; + char drag[40] = ""; + if (dragged_ > 0) std::snprintf(drag, sizeof drag, "dragged %.1f deg, ", dragged_); + std::snprintf(b, sizeof b, "became a press on %s (held still %.2f s; %s), %slet go%s%s (held %.2f s)", + pressed.c_str(), pressAt_ - t0_, gaze, drag, lost ? ": " : "", lost ? why.c_str() : "", held); + } else if (lost) { + std::snprintf(b, sizeof b, "no click: %s (%s, held %.2f s)", why.c_str(), gaze, held); + } else if (moved_ < 0.2) { + std::snprintf(b, sizeof b, "%s at the gaze on %s (%s, held %.2f s)", button, on.c_str(), gaze, held); + } else { + std::snprintf(b, sizeof b, "nudged %.1f deg, then %s on %s (%s; %s, held %.2f s)", moved_, + right_ ? "a right click" : "clicked", on.c_str(), + lesson.empty() ? "not learned" : lesson.c_str(), gaze, held); + } + return b; + } + +private: + Params p_; + Phase phase_ = Phase::Off; + bool right_ = false, engaged_ = false; + double t0_ = 0, pressAt_ = 0, lockAge_ = -1; + double yaw_ = 0, pitch_ = 0, startY_ = 0, startP_ = 0; + double refY_ = 0, refP_ = 0, lastHy_ = 0, lastHp_ = 0; + double moved_ = 0, dragged_ = 0; +}; + +} // namespace gazehand diff --git a/pointer/tests/gazehand-test.cpp b/pointer/tests/gazehand-test.cpp new file mode 100644 index 0000000..ad07a8a --- /dev/null +++ b/pointer/tests/gazehand-test.cpp @@ -0,0 +1,228 @@ +// Unit tests for gaze pinches (pointer/helper/gazehand.h): the gaze lock, the nudge's dead zone and +// gain, the press after a still hold, the typing guard and the ray switch, with made-up gaze at +// ft-gazed's 90 Hz and hands at ft-hands' 30 Hz. +// Build and run: just test-c (or g++ -std=c++17 -I. pointer/tests/gazehand-test.cpp). +#include "pointer/helper/gazehand.h" + +#include +#include +#include + +using namespace gazehand; + +static int failures = 0; +#define CHECK(cond, ...) \ + do { \ + if (!(cond)) { \ + std::printf("FAIL %s:%d: ", __FILE__, __LINE__); \ + std::printf(__VA_ARGS__); \ + std::printf("\n"); \ + ++failures; \ + } \ + } while (0) + +namespace { + +constexpr double kGaze = 1 / 90.0; +constexpr double kFrame = 1 / 30.0; + +bool Has(const std::string &s, const char *what) { return s.find(what) != std::string::npos; } + +// Gaze at (hy, hp) from t0 up to t1, at 90 Hz; raw a degree off the corrected. +void Look(GazeHistory &h, double t0, double t1, double hy, double hp) { + for (double t = t0; t < t1 - 1e-9; t += kGaze) h.Add({t, hy, hp, hy + 1, hp}); +} + +// The live failure this is for: you look at A, start pinching, and the eyes are on their way to the +// next thing before ft-hands calls the pinch. The click goes to A. +void TestLockTakesTheLookBeforeThePinch() { + GazeHistory h; + Look(h, 0, 1.0, 10, 0); // A + Look(h, 1.0, 1.4, 25, -10); // B, from 1.0 s + GazeSample s; + double age = 0; + // the begin's capture at 1.05: 50 ms after the eyes left A + CHECK(h.Lock(1.05, Params(), s, age), "a lock"); + CHECK(s.hy == 10 && s.hp == 0, "at A, not B: %.1f %.1f", s.hy, s.hp); + CHECK(age >= 0.075 - 1e-9 && age < 0.075 + kGaze + 1e-9, "the newest at least 75 ms before: %.3f", age); + CHECK(s.rhy == 11, "with its raw gaze (the lesson's look): %.1f", s.rhy); + // a pinch well into the fixation on B clicks B + CHECK(h.Lock(1.3, Params(), s, age) && s.hy == 25, "later: B (%.1f)", s.hy); +} + +void TestLockWithoutFreshGaze() { + GazeHistory h; + Look(h, 0, 0.5, 10, 0); + GazeSample s; + double age = 0; + CHECK(!h.Lock(1.0, Params(), s, age), "the gaze stopped half a second before: no lock"); + // a blink: nothing from 0.5 to 0.98, then the eyes are back: the first sample after it + Look(h, 0.98, 1.1, 12, 1); + CHECK(h.Lock(1.0, Params(), s, age), "back just before the begin: a lock"); + CHECK(s.hy == 12 && std::fabs(s.t - 0.98) < 1e-6, "the first sample back: %.1f at %.3f", s.hy, s.t); + GazeHistory none; + CHECK(!none.Lock(1.0, Params(), s, age), "no gaze at all: no lock"); +} + +// A tap: the hand jitters (and the pinch point shifts as the fingers close) inside the dead zone. +void TestTapClicksAtTheLock() { + Pinch p; + p.Begin(Params(), 1.0, 30, 5, -20, -35, false, 0.08); + double t = 1.0; + const double jitter[] = {0.4, -0.6, 1.1, 1.8, 2.2, 1.5}; // degrees, along both axes + for (double j : jitter) { + t += kFrame; + CHECK(!p.Hand(-20 + j, -35 - j * 0.5), "jitter doesn't move it"); + CHECK(!p.Promote(t), "a tap isn't a press"); + } + CHECK(p.Yaw() == 30 && p.Pitch() == 5 && p.Moved() == 0, "still at the lock: %.2f %.2f", p.Yaw(), p.Pitch()); + CHECK(p.State() == Pinch::Phase::Aim, "still aiming"); + const std::string says = p.Says(t, false, "", "frametop.screen.0", ""); + CHECK(Has(says, "clicked at the gaze on frametop.screen.0") && Has(says, "gaze 80 ms before"), "%s", says.c_str()); +} + +// Slow drift under the dead zone (a hand sagging while you read) never nudges. +void TestDriftUnderTheDeadZone() { + Pinch p; + p.Begin(Params(), 0, 0, 0, -10, -30, false, 0.08); + for (int i = 0; i <= 40; ++i) p.Hand(-10 + i * 0.06, -30 - i * 0.075); // 3.9 deg over 1.3 s at 30 Hz, down and in + CHECK(p.Yaw() == 0 && p.Pitch() == 0 && p.Moved() == 0, "no nudge: %.3f %.3f", p.Yaw(), p.Pitch()); +} + +// Past the dead zone: the pointer follows at the gain, from where it was (no jump). +void TestNudgePastTheDeadZone() { + Params prm; + Pinch p; + p.Begin(prm, 0, 30, 5, -20, -35, false, 0.08); + double hy = -20, lastYaw = 30; + double maxStep = 0, engagedAt = 0; + // the hand moves left (yaw +) half a degree a frame, 10 degrees in all + for (int i = 1; i <= 20; ++i) { + hy += 0.5; + p.Hand(hy, -35); + maxStep = std::max(maxStep, std::fabs(p.Yaw() - lastYaw)); + lastYaw = p.Yaw(); + if (p.State() == Pinch::Phase::Nudge && engagedAt == 0) engagedAt = hy; + } + CHECK(std::fabs(engagedAt - (-20 + prm.deadzone + 0.5)) < 1e-9, "engaged just past the dead zone: at %.2f", engagedAt); + CHECK(maxStep <= prm.gain * 0.5 + 1e-9, "no jump: the biggest step %.3f deg (gain x 0.5)", maxStep); + // from the engagement (hand at -15.5) to -10: 5.5 deg of hand, 1.65 of pointer + CHECK(std::fabs(p.Yaw() - (30 + prm.gain * (hy - engagedAt))) < 1e-9, "at the gain: %.3f", p.Yaw()); + CHECK(p.Pitch() == 5, "only across: %.3f", p.Pitch()); + // the click: at the corrected point, a correction the helper sends as a lesson (>= 0.2 deg moved) + CHECK(p.Moved() >= 0.2 && std::fabs(p.Moved() - (p.Yaw() - 30)) < 1e-9, "moved %.3f", p.Moved()); + CHECK(!p.Promote(10), "a pinch that nudged never becomes a press"); + std::string says = p.Says(1.2, false, "", "frametop.screen.1", "a lesson"); + CHECK(Has(says, "nudged 1.6 deg, then clicked on frametop.screen.1 (a lesson;"), "%s", says.c_str()); + says = p.Says(1.2, false, "", "frametop.screen.1", "the quick check"); + CHECK(Has(says, "(the quick check;"), "%s", says.c_str()); + says = p.Says(1.2, false, "", "frametop.screen.1", ""); + CHECK(Has(says, "(not learned;"), "%s", says.c_str()); + // back again: the pointer comes back with the hand + for (int i = 0; i < 11; ++i) hy -= 0.5, p.Hand(hy, -35); + CHECK(std::fabs(p.Yaw() - 30) < 1e-9, "back where it engaged: %.3f", p.Yaw()); +} + +void TestGainAndDeadZoneSettings() { + Params prm; + prm.deadzone = 1.0, prm.gain = 0.5; + Pinch p; + p.Begin(prm, 0, 0, 0, 0, 0, false, -1); + p.Hand(0, -1.2); // engages + p.Hand(0, -3.2); + CHECK(std::fabs(p.Pitch() + 1.0) < 1e-9, "POINTER_GAZE_PINCH_GAIN 0.5: %.3f", p.Pitch()); + const std::string says = p.Says(1, false, "", "", ""); + CHECK(Has(says, "no fresh gaze: where the pointer was") && Has(says, "on nothing"), "%s", says.c_str()); +} + +// Held still for POINTER_GAZE_HOLD: a real press at the lock, then the hand drags at the grip's gain. +void TestHoldStillBecomesAPress() { + Params prm; + Pinch p; + p.Begin(prm, 0, 30, 5, -20, -35, false, 0.08); + double t = 0; + bool pressed = false; + int presses = 0; + while (t < 1.0) { + t += 0.008; + if (std::fmod(t, kFrame) < 0.008) p.Hand(-20 + 1.0, -35 + 0.5); // still, a little off the begin + if (p.Promote(t)) { + ++presses; + CHECK(!pressed && t >= prm.hold - 1e-9 && t < prm.hold + 0.009, "pressed at %.3f s", t); + pressed = true; + } + } + CHECK(presses == 1, "one press: %d", presses); + CHECK(p.State() == Pinch::Phase::Press && p.Yaw() == 30 && p.Pitch() == 5, "at the lock"); + // the drag counts from the hand at the press: inside 0.5 deg nothing, then gain 1 + p.Hand(-19 + 0.4, -34.5); + CHECK(p.Yaw() == 30, "the drag's dead zone"); + p.Hand(-19 + 0.6, -34.5); // engages + p.Hand(-19 + 6.6, -34.5); + CHECK(std::fabs(p.Yaw() - 36) < 1e-9 && p.Moved() == 0 && std::fabs(p.Dragged() - 6) < 1e-9, + "dragged at gain 1: %.3f (moved %.3f, dragged %.3f)", p.Yaw(), p.Moved(), p.Dragged()); + p.pressHit = "frametop.float.3"; + std::string says = p.Says(1.4, false, "", "", ""); + CHECK(Has(says, "became a press on frametop.float.3 (held still 0.50 s;") && Has(says, "dragged 6.0 deg, let go (held 1.40 s)"), + "%s", says.c_str()); + says = p.Says(1.4, true, "its side has a controller in use", "", ""); + CHECK(Has(says, "let go: its side has a controller in use"), "%s", says.c_str()); +} + +void TestMiddlePinch() { + Pinch p; + p.Begin(Params(), 0, 10, 0, 0, -30, true, 0.09); + CHECK(!p.Promote(2.0), "a middle pinch never becomes a press"); + std::string says = p.Says(2.0, false, "", "frametop.screen.0", ""); + CHECK(Has(says, "right click at the gaze on frametop.screen.0"), "%s", says.c_str()); + for (double hy = 0; hy >= -10; hy -= 0.5) p.Hand(hy, -30); + says = p.Says(2.0, false, "", "frametop.screen.0", "a lesson"); + CHECK(Has(says, "then a right click on"), "%s", says.c_str()); + says = p.Says(2.0, true, "ft-hands lost the hand, or a grip took the pinch", "", ""); + CHECK(Has(says, "no click: ft-hands lost the hand"), "%s", says.c_str()); +} + +void TestRefused() { + Situation s; + CHECK(Refused(s) == nullptr, "nothing in the way"); + s.sinceKey = 0.3; + CHECK(Refused(s) && std::string(Refused(s)) == "ignored (typing)", "a key 0.3 s ago"); + s.sinceKey = 1.2; + CHECK(Refused(s) == nullptr, "a key 1.2 s ago"); + s.typingHold = 0; + s.sinceKey = 0; + CHECK(Refused(s) == nullptr, "POINTER_PINCH_TYPING=0: no guard"); + s = Situation(); + s.active = false, s.sinceKey = 0.1; + CHECK(std::string(Refused(s)) == "it woke the pointer, no click", "asleep: it only wakes"); + s.busy = true; + CHECK(std::string(Refused(s)) == "ignored (a press is already held)", "busy first"); +} + +void TestRaySwitch() { + CHECK(RayAllowed(true, false, false), "no gaze mode: the ray"); + CHECK(!RayAllowed(true, true, false), "gaze mode: no ray (POINTER_GAZE_HAND_RAY=0)"); + CHECK(RayAllowed(true, true, true), "gaze mode with POINTER_GAZE_HAND_RAY=1: the ray"); + CHECK(!RayAllowed(false, false, true), "POINTER_HAND_RAY=0: never"); +} + +} // namespace + +int main() { + TestLockTakesTheLookBeforeThePinch(); + TestLockWithoutFreshGaze(); + TestTapClicksAtTheLock(); + TestDriftUnderTheDeadZone(); + TestNudgePastTheDeadZone(); + TestGainAndDeadZoneSettings(); + TestHoldStillBecomesAPress(); + TestMiddlePinch(); + TestRefused(); + TestRaySwitch(); + if (failures) { + std::printf("%d failures\n", failures); + return 1; + } + std::printf("gazehand-test: ok\n"); + return 0; +} diff --git a/session/frametop.conf.example b/session/frametop.conf.example index 5c74e5b..5ce4fb9 100644 --- a/session/frametop.conf.example +++ b/session/frametop.conf.example @@ -65,8 +65,8 @@ HANDS_GRAB_DEADZONE=0.01 # ... and moves the panel only once it's this far (m) HANDS_NEAR=1 # Frametop panels: poke to click, pinch the bar, edges or corners to move or resize (1 = when the pointer helper says a hand is bare | 0 = off | force = always, for testing) HANDS_PALM_MENU=1 # hands: turn a bare hand palm up toward your face and look at it, and a ring shows above the palm; pinch with that hand to open the Steam menu (or close the dashboard). 0 = off POINTER_HANDS=0 # 1 = pinches click (on the release; hold and move to nudge the pointer first) and grips press and drag, with hand tracking -POINTER_PINCH_GAIN=0.5 # hands: the pointer moves this times the pinching hand's angle (precision) -POINTER_PINCH_DEADZONE=1.5 # hands: degrees the pinching hand moves before the pointer does (a tap's jitter) +POINTER_PINCH_GAIN=0.5 # hands without gaze mode: a held pinch drags the pointer this times the hand's angle (precision) +POINTER_PINCH_DEADZONE=1.5 # hands without gaze mode: degrees the pinching hand moves before the pointer does (a tap's jitter) POINTER_GRIP_GAIN=1 # hands: the pointer moves this times the gripping hand's angle POINTER_GRIP_BELOW=0.35 # hands: grips beginning more than this (m) below the eyes are ignored (hands on a desk) POINTER_PINCH_TYPING=1 # hands: no pinch within this many seconds of a key (typing touches thumb to index) @@ -82,6 +82,9 @@ POINTER_RAY_SMOOTH_HOLD=0.5 # hand ray: the still smoothing while a pinch is hel POINTER_RAY_STEADY=1 # hand ray: while a pinch is held, the ray pivots on where the shoulder was at the press, so looking around doesn't move what you drag POINTER_RAY_TAP=0.3 # hand ray: for this many seconds after a pinch begins ... POINTER_RAY_TAP_DEADZONE=6 # ... the ray turns this many degrees before the press drags (a quick tap stays a click) +POINTER_GAZE_HAND_RAY=0 # hands in gaze mode: 0 = no hand ray, your eyes aim and a pinch clicks where you look | 1 = a raised, pointing hand's ray works beside the gaze +POINTER_GAZE_PINCH_DEADZONE=4 # hands in gaze mode: degrees (seen from the eye; about 3 cm at arm's length) a held pinch moves before it nudges the pointer; held still for POINTER_GAZE_HOLD it's a press, to drag +POINTER_GAZE_PINCH_GAIN=0.3 # hands in gaze mode: past that, the pointer moves this times the pinching hand's angle (low, for small corrections) SHARE_KEYS=0 # 1 = keys of keyboards grabbed for the desktop also go to @frametop_keys, for hotkey tools (any local process can listen) DISPLAY_OFF_MIN=0 # minutes without use before frametop-power turns the displays off, even if the headset seems worn (0 = never) DISPLAY_MOVE_MM=5 # displays off: the headset or a controller moving this far (mm) within 10 s counts as use