# Map screenshot vision pipeline: status and next-step plan Goal: given a screenshot of the in-game map view (not the typewriter text), detect enemy unit markers (red diamonds per the in-game legend) and resolve each one's position (large grid + small grid) automatically, for import as targets (position + id only, type stays `TargetType.UNKNOWN` for this first pass). This is a *second*, entirely separate OCR/vision pipeline from `ocr.py`'s text pipeline. `ocr.py`'s module docstring already flags this as future/out-of-scope work; this doc is that future work's design record. ## Where the code lives right now `docs/map_vision_wip.py` in this repo is a copy of the working prototype, as of the end of this exploration session. It is **not wired into the app** and not `src/fenigma/map_vision.py` yet — it's scratchpad-quality (developed and tested against three sample screenshots via ad-hoc test scripts, not a proper test suite). Treat it as a strong starting point, not finished code: variable names, error handling, and docstrings need a pass before it belongs in `src/`. ## What's validated and working 1. **Grid label OCR** (`find_grid_labels`): sliding-window Tesseract sweep across the image, filtered to the `[A-T](10|[1-9])` pattern (e.g. `M8`, `Q10`). Works reasonably well on clean labels; struggles when a colored hatch-line overlay crosses directly through a label's glyphs (`L7` misread as `AF`, `N8` as `NWS`) — tried desaturating before OCR, didn't help (recoloring the hatch line neutral still leaves a shape gap in the letter). **Needs real inpainting** (fill the interrupted stroke from surrounding pixels) to fix, not yet done. 2. **Pitch estimation** (`estimate_pitch`): derives cell pitch (both x and y) from the actual pixel spacing between same-row / same-column labels. No hardcoded pitch constant anywhere — this was an explicit, correct call-out mid-session (an earlier version hardcoded `pitch=749` from eyeballing one image; that doesn't generalize and was thrown out). 3. **Corner/intersection matching** (`find_crossing`, `cross_kernel`): given a predicted pixel position for a grid-line intersection, finds the real one nearby. This went through several broken iterations before landing on the current approach — worth remembering *why* each earlier attempt failed, so they don't get reinvented: - Plain local brightness-percentile search: worked on one lucky isolated line, completely unreliable elsewhere (confirmed via zoomed crop: a claimed "corner" for `L8` sat in flat background, nowhere near any real line). - Generic `cv2.goodFeaturesToTrack`: fires on *any* strong corner, so it reliably found the label's own text glyphs or a nearby diamond marker's vertex instead of the grid intersection. Fixed by masking out the label's own bounding box, then *also* masking out anything with high HSV saturation (markers and hatching are deliberately colored; the grid itself is neutral gray/white — a generalizable distinction, not per-image tuning). - Even with masking, a generic corner detector still isn't picking the right *kind* of corner (a line's endpoint kink looks the same as a true 4-way crossing to it). Fixed by replacing it entirely with `cross_kernel`: a matched filter shaped like a bright `+` (positive along both a horizontal and vertical arm through the center, negative in the four quadrant gaps) — a lone single-direction line only lights up one arm and scores far below a true crossing. This is the piece that actually made precise matches possible (verified to land within a couple pixels of a manually-confirmed true corner). - The matched-filter kernel size and the search-window size must both be computed **from the current image's own pitch**, never a fixed pixel constant — a kernel/window tuned for a 750px cell is meaningless on a 150px cell. `find_crossing` takes `pitch_x, pitch_y` and derives both from them. - Candidate selection within the window: originally multiplied the matched-filter response by a tight Gaussian prior (centered on the predicted position) *before* taking the max — this let a weak, coincidentally-central false response beat a real, stronger crossing nearby ("prior is too aggressive, matches predicted center" was the exact bug report). Fixed by widening the Gaussian (`sigma = 1.5 * window radius`) and only blending it in lightly (`0.85 + 0.15*prior`) so the actual filter response does most of the selecting; the prior now only functions as a mild tie-breaker plus a final plausibility check on the winner, not the primary selection mechanism. - The "plausible region" is a Gaussian, so it should be *drawn* as a circle in diagnostics, not a rectangle — a rectangle visually implies a hard cutoff that doesn't reflect the actual model. Fixed in the debug visualization. 4. **Whole-grid crossing prediction** (`span_grid_crossings`): once we have *any* labels (even 2-3), the grid is regular, so predict and test every crossing across the visible frame, not just the ones adjacent to a label that happened to OCR cleanly. This gives far more correspondence points than "one per successfully-read label." Validated end to end on the zoomed-out strategic-overview screenshot: 4 labels → 10 predicted crossings → 4 matched → homography fit with all 4 as RANSAC inliers. 5. **Homography fit** (`fit_grid_homography`): `cv2.findHomography(..., cv2.RANSAC, 15.0)`, requires 4+ points with real geometric diversity (2+ distinct columns AND 2+ distinct rows — `has_diversity`) before even attempting a fit, refusing collinear/degenerate input rather than producing garbage. This replaced an earlier rigid-rotation-only (translation + single theta) model once it became clear the real screen has genuine perspective/keystone distortion (parallel lines don't stay parallel), confirmed by directly measuring the same line's x-position at two widely-separated y-values and finding a real, consistent ~2-4° drift, not noise. ## The known remaining bug, and the planned fix **Bug**: with exactly 4 matched points, `cv2.findHomography` always fits them *exactly* — 0 residual and "4/4 inliers" is true by construction and doesn't mean the fit is actually good. Confirmed visually: a 4-point fit on 4 points that happened to form a lopsided "staircase" in grid-space (missing two corners of what should've been a proper 2×2 block) produced a visibly skewed parallelogram instead of a rectangle, even though every individual point matched its real intersection correctly. The 4 points were individually right; the *set* was too small and too oddly-shaped to constrain the fit meaningfully. Root cause of *why* only 4/10 predicted crossings matched: `span_grid_crossings` predicts every crossing from **one single reference label** using one global pitch value for the whole image. Since perspective distortion is real, that single global (origin, pitch) pair drifts further from the truth the farther a predicted crossing is from the reference — a jump of several cells accumulates several cells' worth of drift before the search window even starts looking. (Some other misses were legitimately unfindable — busy photo texture with no clean line at that exact spot, confirmed via a zoomed crop — but the long-range extrapolation drift is the fixable, systematic part.) ### Planned fix: grid-growing (BFS) instead of batch prediction from one origin Don't predict the whole grid from one fixed point. Walk outward one cell at a time from every confirmed point, correcting the local estimate as you go: 1. **Seeds**: every label that OCR'd *and* corner-matched successfully is a confirmed `(grid_col, grid_row) -> (pixel_x, pixel_y)` point. Multiple seeds, not just the single highest-confidence label. 2. **Expand one cell at a time**: from each confirmed point, only ever predict its *immediate* neighbor (one cell in one of the 4 directions) — never extrapolate further than one cell from something already confirmed. 3. **Prefer local spacing over the global average**: if two confirmed points already share a row (for a column step) or column (for a row step), use *their* measured spacing to predict the next one out — that's the real local pitch right there. Only fall back to the global pitch estimate for the very first step away from a seed, where no local measurement exists yet. 4. **Search, confirm, repeat**: run the same `find_crossing` search at that one-cell-away prediction. Success → add to the confirmed set, push onto the expansion frontier. Failure → that one edge just stops there, doesn't block expansion from other confirmed points nearby. 5. Keep expanding (a plain BFS/queue over grid coordinates, with a visited/attempted set so failed edges aren't retried forever) until the frontier is empty. Feed every confirmed point into the homography fit — likely dozens of points instead of 4, each individually short-range and therefore much less exposed to long-range perspective drift, with enough redundancy that the residual/inlier check from RANSAC actually means something instead of being a vacuous exact-fit. This is a genuine restructure of `span_grid_crossings`'s control flow (batch prediction → BFS), not a parameter tweak. Implementing it is the next step when this work resumes. ## Also still open (lower priority than the BFS fix) - Label OCR robustness where hatching crosses the glyph (needs inpainting). - Once the grid calibration is reliably robust: red-diamond blob detection (color threshold + connected components) — not started. - Matching each detected blob to its `#N` id label via nearby OCR — not started. - Wiring into `app.py`: a new button, merging results through the existing `_merge_targets`-style flow as `TargetType.UNKNOWN` — not started.