FEnigma/docs/map_vision_plan.md

# Map screenshot vision pipeline: status and next-step plan

Goal: given a screenshot of the in-game map view (not the typewriter text), detect
enemy unit markers (red diamonds per the in-game legend) and resolve each one's
position (large grid + small grid) automatically, for import as targets
(position + id only, type stays `TargetType.UNKNOWN` for this first pass).

This is a *second*, entirely separate OCR/vision pipeline from `ocr.py`'s text
pipeline. `ocr.py`'s module docstring already flags this as future/out-of-scope
work; this doc is that future work's design record.

## Where the code lives right now

`docs/map_vision_wip.py` in this repo is a copy of the working prototype, as of
the end of this exploration session. It is **not wired into the app** and not
`src/fenigma/map_vision.py` yet — it's scratchpad-quality (developed and tested
against three sample screenshots via ad-hoc test scripts, not a proper test
suite). Treat it as a strong starting point, not finished code: variable names,
error handling, and docstrings need a pass before it belongs in `src/`.

## What's validated and working

1. **Grid label OCR** (`find_grid_labels`): sliding-window Tesseract sweep
   across the image, filtered to the `[A-T](10|[1-9])` pattern
   (e.g. `M8`, `Q10`). Works reasonably well on clean labels; struggles when a
   colored hatch-line overlay crosses directly through a label's glyphs
   (`L7` misread as `AF`, `N8` as `NWS`) — tried desaturating before OCR,
   didn't help (recoloring the hatch line neutral still leaves a shape gap in
   the letter). **Needs real inpainting** (fill the interrupted stroke from
   surrounding pixels) to fix, not yet done.

2. **Pitch estimation** (`estimate_pitch`): derives cell pitch (both x and y)
   from the actual pixel spacing between same-row / same-column labels. No
   hardcoded pitch constant anywhere — this was an explicit, correct call-out
   mid-session (an earlier version hardcoded `pitch=749` from eyeballing one
   image; that doesn't generalize and was thrown out).

3. **Corner/intersection matching** (`find_crossing`, `cross_kernel`): given a
   predicted pixel position for a grid-line intersection, finds the real one
   nearby. This went through several broken iterations before landing on the
   current approach — worth remembering *why* each earlier attempt failed,
   so they don't get reinvented:
   - Plain local brightness-percentile search: worked on one lucky isolated
     line, completely unreliable elsewhere (confirmed via zoomed crop: a
     claimed "corner" for `L8` sat in flat background, nowhere near any real
     line).
   - Generic `cv2.goodFeaturesToTrack`: fires on *any* strong corner, so it
     reliably found the label's own text glyphs or a nearby diamond marker's
     vertex instead of the grid intersection. Fixed by masking out the
     label's own bounding box, then *also* masking out anything with high
     HSV saturation (markers and hatching are deliberately colored; the grid
     itself is neutral gray/white — a generalizable distinction, not
     per-image tuning).
   - Even with masking, a generic corner detector still isn't picking the
     right *kind* of corner (a line's endpoint kink looks the same as a true
     4-way crossing to it). Fixed by replacing it entirely with
     `cross_kernel`: a matched filter shaped like a bright `+` (positive
     along both a horizontal and vertical arm through the center, negative
     in the four quadrant gaps) — a lone single-direction line only lights
     up one arm and scores far below a true crossing. This is the piece
     that actually made precise matches possible (verified to land within a
     couple pixels of a manually-confirmed true corner).
   - The matched-filter kernel size and the search-window size must both be
     computed **from the current image's own pitch**, never a fixed pixel
     constant — a kernel/window tuned for a 750px cell is meaningless on a
     150px cell. `find_crossing` takes `pitch_x, pitch_y` and derives both
     from them.
   - Candidate selection within the window: originally multiplied the
     matched-filter response by a tight Gaussian prior (centered on the
     predicted position) *before* taking the max — this let a weak,
     coincidentally-central false response beat a real, stronger crossing
     nearby ("prior is too aggressive, matches predicted center" was the
     exact bug report). Fixed by widening the Gaussian (`sigma = 1.5 *
     window radius`) and only blending it in lightly (`0.85 + 0.15*prior`)
     so the actual filter response does most of the selecting; the prior
     now only functions as a mild tie-breaker plus a final plausibility
     check on the winner, not the primary selection mechanism.
   - The "plausible region" is a Gaussian, so it should be *drawn* as a
     circle in diagnostics, not a rectangle — a rectangle visually implies a
     hard cutoff that doesn't reflect the actual model. Fixed in the debug
     visualization.

4. **Whole-grid crossing prediction** (`span_grid_crossings`): once we have
   *any* labels (even 2-3), the grid is regular, so predict and test every
   crossing across the visible frame, not just the ones adjacent to a label
   that happened to OCR cleanly. This gives far more correspondence points
   than "one per successfully-read label." Validated end to end on the
   zoomed-out strategic-overview screenshot: 4 labels → 10 predicted
   crossings → 4 matched → homography fit with all 4 as RANSAC inliers.

5. **Homography fit** (`fit_grid_homography`): `cv2.findHomography(...,
   cv2.RANSAC, 15.0)`, requires 4+ points with real geometric diversity (2+
   distinct columns AND 2+ distinct rows — `has_diversity`) before even
   attempting a fit, refusing collinear/degenerate input rather than
   producing garbage. This replaced an earlier rigid-rotation-only
   (translation + single theta) model once it became clear the real screen
   has genuine perspective/keystone distortion (parallel lines don't stay
   parallel), confirmed by directly measuring the same line's x-position at
   two widely-separated y-values and finding a real, consistent ~2-4° drift,
   not noise.

## The known remaining bug, and the planned fix

**Bug**: with exactly 4 matched points, `cv2.findHomography` always fits them
*exactly* — 0 residual and "4/4 inliers" is true by construction and doesn't
mean the fit is actually good. Confirmed visually: a 4-point fit on 4 points
that happened to form a lopsided "staircase" in grid-space (missing two
corners of what should've been a proper 2×2 block) produced a visibly
skewed parallelogram instead of a rectangle, even though every individual
point matched its real intersection correctly. The 4 points were individually
right; the *set* was too small and too oddly-shaped to constrain the fit
meaningfully.

Root cause of *why* only 4/10 predicted crossings matched: `span_grid_crossings`
predicts every crossing from **one single reference label** using one global
pitch value for the whole image. Since perspective distortion is real, that
single global (origin, pitch) pair drifts further from the truth the farther a
predicted crossing is from the reference — a jump of several cells accumulates
several cells' worth of drift before the search window even starts looking.
(Some other misses were legitimately unfindable — busy photo texture with no
clean line at that exact spot, confirmed via a zoomed crop — but the long-range
extrapolation drift is the fixable, systematic part.)

### Planned fix: grid-growing (BFS) instead of batch prediction from one origin

Don't predict the whole grid from one fixed point. Walk outward one cell at a
time from every confirmed point, correcting the local estimate as you go:

1. **Seeds**: every label that OCR'd *and* corner-matched successfully is a
   confirmed `(grid_col, grid_row) -> (pixel_x, pixel_y)` point. Multiple
   seeds, not just the single highest-confidence label.
2. **Expand one cell at a time**: from each confirmed point, only ever
   predict its *immediate* neighbor (one cell in one of the 4 directions) —
   never extrapolate further than one cell from something already confirmed.
3. **Prefer local spacing over the global average**: if two confirmed points
   already share a row (for a column step) or column (for a row step), use
   *their* measured spacing to predict the next one out — that's the real
   local pitch right there. Only fall back to the global pitch estimate for
   the very first step away from a seed, where no local measurement exists
   yet.
4. **Search, confirm, repeat**: run the same `find_crossing` search at that
   one-cell-away prediction. Success → add to the confirmed set, push onto
   the expansion frontier. Failure → that one edge just stops there, doesn't
   block expansion from other confirmed points nearby.
5. Keep expanding (a plain BFS/queue over grid coordinates, with a
   visited/attempted set so failed edges aren't retried forever) until the
   frontier is empty. Feed every confirmed point into the homography fit —
   likely dozens of points instead of 4, each individually short-range and
   therefore much less exposed to long-range perspective drift, with enough
   redundancy that the residual/inlier check from RANSAC actually means
   something instead of being a vacuous exact-fit.

This is a genuine restructure of `span_grid_crossings`'s control flow (batch
prediction → BFS), not a parameter tweak. Implementing it is the next step
when this work resumes.

## Also still open (lower priority than the BFS fix)

- Label OCR robustness where hatching crosses the glyph (needs inpainting).
- Once the grid calibration is reliably robust: red-diamond blob detection
  (color threshold + connected components) — not started.
- Matching each detected blob to its `#N` id label via nearby OCR — not
  started.
- Wiring into `app.py`: a new button, merging results through the existing
  `_merge_targets`-style flow as `TargetType.UNKNOWN` — not started.