diff --git a/docs/map_vision_plan.md b/docs/map_vision_plan.md new file mode 100644 index 0000000..224fbf1 --- /dev/null +++ b/docs/map_vision_plan.md @@ -0,0 +1,165 @@ +# Map screenshot vision pipeline: status and next-step plan + +Goal: given a screenshot of the in-game map view (not the typewriter text), detect +enemy unit markers (red diamonds per the in-game legend) and resolve each one's +position (large grid + small grid) automatically, for import as targets +(position + id only, type stays `TargetType.UNKNOWN` for this first pass). + +This is a *second*, entirely separate OCR/vision pipeline from `ocr.py`'s text +pipeline. `ocr.py`'s module docstring already flags this as future/out-of-scope +work; this doc is that future work's design record. + +## Where the code lives right now + +`docs/map_vision_wip.py` in this repo is a copy of the working prototype, as of +the end of this exploration session. It is **not wired into the app** and not +`src/fenigma/map_vision.py` yet — it's scratchpad-quality (developed and tested +against three sample screenshots via ad-hoc test scripts, not a proper test +suite). Treat it as a strong starting point, not finished code: variable names, +error handling, and docstrings need a pass before it belongs in `src/`. + +## What's validated and working + +1. **Grid label OCR** (`find_grid_labels`): sliding-window Tesseract sweep + across the image, filtered to the `[A-T](10|[1-9])` pattern + (e.g. `M8`, `Q10`). Works reasonably well on clean labels; struggles when a + colored hatch-line overlay crosses directly through a label's glyphs + (`L7` misread as `AF`, `N8` as `NWS`) — tried desaturating before OCR, + didn't help (recoloring the hatch line neutral still leaves a shape gap in + the letter). **Needs real inpainting** (fill the interrupted stroke from + surrounding pixels) to fix, not yet done. + +2. **Pitch estimation** (`estimate_pitch`): derives cell pitch (both x and y) + from the actual pixel spacing between same-row / same-column labels. No + hardcoded pitch constant anywhere — this was an explicit, correct call-out + mid-session (an earlier version hardcoded `pitch=749` from eyeballing one + image; that doesn't generalize and was thrown out). + +3. **Corner/intersection matching** (`find_crossing`, `cross_kernel`): given a + predicted pixel position for a grid-line intersection, finds the real one + nearby. This went through several broken iterations before landing on the + current approach — worth remembering *why* each earlier attempt failed, + so they don't get reinvented: + - Plain local brightness-percentile search: worked on one lucky isolated + line, completely unreliable elsewhere (confirmed via zoomed crop: a + claimed "corner" for `L8` sat in flat background, nowhere near any real + line). + - Generic `cv2.goodFeaturesToTrack`: fires on *any* strong corner, so it + reliably found the label's own text glyphs or a nearby diamond marker's + vertex instead of the grid intersection. Fixed by masking out the + label's own bounding box, then *also* masking out anything with high + HSV saturation (markers and hatching are deliberately colored; the grid + itself is neutral gray/white — a generalizable distinction, not + per-image tuning). + - Even with masking, a generic corner detector still isn't picking the + right *kind* of corner (a line's endpoint kink looks the same as a true + 4-way crossing to it). Fixed by replacing it entirely with + `cross_kernel`: a matched filter shaped like a bright `+` (positive + along both a horizontal and vertical arm through the center, negative + in the four quadrant gaps) — a lone single-direction line only lights + up one arm and scores far below a true crossing. This is the piece + that actually made precise matches possible (verified to land within a + couple pixels of a manually-confirmed true corner). + - The matched-filter kernel size and the search-window size must both be + computed **from the current image's own pitch**, never a fixed pixel + constant — a kernel/window tuned for a 750px cell is meaningless on a + 150px cell. `find_crossing` takes `pitch_x, pitch_y` and derives both + from them. + - Candidate selection within the window: originally multiplied the + matched-filter response by a tight Gaussian prior (centered on the + predicted position) *before* taking the max — this let a weak, + coincidentally-central false response beat a real, stronger crossing + nearby ("prior is too aggressive, matches predicted center" was the + exact bug report). Fixed by widening the Gaussian (`sigma = 1.5 * + window radius`) and only blending it in lightly (`0.85 + 0.15*prior`) + so the actual filter response does most of the selecting; the prior + now only functions as a mild tie-breaker plus a final plausibility + check on the winner, not the primary selection mechanism. + - The "plausible region" is a Gaussian, so it should be *drawn* as a + circle in diagnostics, not a rectangle — a rectangle visually implies a + hard cutoff that doesn't reflect the actual model. Fixed in the debug + visualization. + +4. **Whole-grid crossing prediction** (`span_grid_crossings`): once we have + *any* labels (even 2-3), the grid is regular, so predict and test every + crossing across the visible frame, not just the ones adjacent to a label + that happened to OCR cleanly. This gives far more correspondence points + than "one per successfully-read label." Validated end to end on the + zoomed-out strategic-overview screenshot: 4 labels → 10 predicted + crossings → 4 matched → homography fit with all 4 as RANSAC inliers. + +5. **Homography fit** (`fit_grid_homography`): `cv2.findHomography(..., + cv2.RANSAC, 15.0)`, requires 4+ points with real geometric diversity (2+ + distinct columns AND 2+ distinct rows — `has_diversity`) before even + attempting a fit, refusing collinear/degenerate input rather than + producing garbage. This replaced an earlier rigid-rotation-only + (translation + single theta) model once it became clear the real screen + has genuine perspective/keystone distortion (parallel lines don't stay + parallel), confirmed by directly measuring the same line's x-position at + two widely-separated y-values and finding a real, consistent ~2-4° drift, + not noise. + +## The known remaining bug, and the planned fix + +**Bug**: with exactly 4 matched points, `cv2.findHomography` always fits them +*exactly* — 0 residual and "4/4 inliers" is true by construction and doesn't +mean the fit is actually good. Confirmed visually: a 4-point fit on 4 points +that happened to form a lopsided "staircase" in grid-space (missing two +corners of what should've been a proper 2×2 block) produced a visibly +skewed parallelogram instead of a rectangle, even though every individual +point matched its real intersection correctly. The 4 points were individually +right; the *set* was too small and too oddly-shaped to constrain the fit +meaningfully. + +Root cause of *why* only 4/10 predicted crossings matched: `span_grid_crossings` +predicts every crossing from **one single reference label** using one global +pitch value for the whole image. Since perspective distortion is real, that +single global (origin, pitch) pair drifts further from the truth the farther a +predicted crossing is from the reference — a jump of several cells accumulates +several cells' worth of drift before the search window even starts looking. +(Some other misses were legitimately unfindable — busy photo texture with no +clean line at that exact spot, confirmed via a zoomed crop — but the long-range +extrapolation drift is the fixable, systematic part.) + +### Planned fix: grid-growing (BFS) instead of batch prediction from one origin + +Don't predict the whole grid from one fixed point. Walk outward one cell at a +time from every confirmed point, correcting the local estimate as you go: + +1. **Seeds**: every label that OCR'd *and* corner-matched successfully is a + confirmed `(grid_col, grid_row) -> (pixel_x, pixel_y)` point. Multiple + seeds, not just the single highest-confidence label. +2. **Expand one cell at a time**: from each confirmed point, only ever + predict its *immediate* neighbor (one cell in one of the 4 directions) — + never extrapolate further than one cell from something already confirmed. +3. **Prefer local spacing over the global average**: if two confirmed points + already share a row (for a column step) or column (for a row step), use + *their* measured spacing to predict the next one out — that's the real + local pitch right there. Only fall back to the global pitch estimate for + the very first step away from a seed, where no local measurement exists + yet. +4. **Search, confirm, repeat**: run the same `find_crossing` search at that + one-cell-away prediction. Success → add to the confirmed set, push onto + the expansion frontier. Failure → that one edge just stops there, doesn't + block expansion from other confirmed points nearby. +5. Keep expanding (a plain BFS/queue over grid coordinates, with a + visited/attempted set so failed edges aren't retried forever) until the + frontier is empty. Feed every confirmed point into the homography fit — + likely dozens of points instead of 4, each individually short-range and + therefore much less exposed to long-range perspective drift, with enough + redundancy that the residual/inlier check from RANSAC actually means + something instead of being a vacuous exact-fit. + +This is a genuine restructure of `span_grid_crossings`'s control flow (batch +prediction → BFS), not a parameter tweak. Implementing it is the next step +when this work resumes. + +## Also still open (lower priority than the BFS fix) + +- Label OCR robustness where hatching crosses the glyph (needs inpainting). +- Once the grid calibration is reliably robust: red-diamond blob detection + (color threshold + connected components) — not started. +- Matching each detected blob to its `#N` id label via nearby OCR — not + started. +- Wiring into `app.py`: a new button, merging results through the existing + `_merge_targets`-style flow as `TargetType.UNKNOWN` — not started. diff --git a/docs/map_vision_wip.py b/docs/map_vision_wip.py new file mode 100644 index 0000000..1a845f6 --- /dev/null +++ b/docs/map_vision_wip.py @@ -0,0 +1,236 @@ +"""Generalized (no per-image constants) map-screenshot grid calibration. + +find_grid_labels() -> OCR sweep for visible large-grid labels. +estimate_pitch() -> derive cell pitch from label-to-label spacing. +find_label_corner()-> per-label corner search via a real corner detector + (cv2.goodFeaturesToTrack), window scaled to pitch. +fit_grid_homography() -> RANSAC homography from however many corners + were found, refusing to fit degenerate/thin data. +""" +import re +import cv2 +import numpy as np +import pytesseract + +LARGE_X = "ABCDEFGHIJKLMNOPQRST" +LABEL_RE = re.compile(r'^([A-T])(10|[1-9])$') +ANCHOR_FRAC_X = 0.10 +ANCHOR_FRAC_Y = 0.15 + + +def desaturate(rgb): + """Replace saturated (colored) pixels with a neutral gray of the same + brightness. The hatching overlay and unit markers are drawn in + deliberately saturated colors while the grid + its labels are neutral + white/cream on a grayscale photo, hatch lines crossing straight + through a label glyph otherwise corrupt its shape enough to break + OCR (seen: 'L7' -> 'AF', 'N8' -> 'NWS').""" + hsv = cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV) + sat = hsv[:, :, 1] + gray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY) + out = gray.copy() + return out, sat + + +def find_grid_labels(gray, band_height=160, band_step=60, scale=3.0, min_conf=30): + H, W = gray.shape + config = "--psm 11 -c tessedit_char_whitelist=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789" + found = {} + for y0 in range(0, H, band_step): + y1 = min(y0 + band_height, H) + band = gray[y0:y1, :] + big = cv2.resize(band, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC) + data = pytesseract.image_to_data(big, output_type=pytesseract.Output.DICT, config=config) + for i, txt in enumerate(data['text']): + t = txt.strip() + m = LABEL_RE.match(t) + if not m or int(data['conf'][i]) < min_conf: + continue + conf = int(data['conf'][i]) + key = t + if key not in found or conf > found[key]['conf']: + found[key] = dict( + text=t, letter=m.group(1), number=int(m.group(2)), + x=data['left'][i]/scale, y=y0 + data['top'][i]/scale, + w=data['width'][i]/scale, h=data['height'][i]/scale, conf=conf, + ) + return list(found.values()) + + +def estimate_pitch(labels): + xs_by_row, ys_by_col = {}, {} + for lb in labels: + xs_by_row.setdefault(lb['number'], []).append((LARGE_X.index(lb['letter']), lb['x']+lb['w']/2)) + ys_by_col.setdefault(lb['letter'], []).append((lb['number'], lb['y']+lb['h']/2)) + px = [] + for pts in xs_by_row.values(): + pts.sort() + for (ci,xi),(cj,xj) in zip(pts, pts[1:]): + if cj > ci: + px.append((xj-xi)/(cj-ci)) + py = [] + for pts in ys_by_col.values(): + pts.sort() + for (ri,yi),(rj,yj) in zip(pts, pts[1:]): + if rj != ri: + py.append(abs(yj-yi)/abs(rj-ri)) + pitch_x = float(np.median(px)) if px else None + pitch_y = float(np.median(py)) if py else None + if pitch_x and not pitch_y: + pitch_y = pitch_x + if pitch_y and not pitch_x: + pitch_x = pitch_y + return pitch_x, pitch_y + + +def cross_kernel(size, arm_width): + """Matched filter for a bright axis-aligned line CROSSING in both + directions through the center, not just any corner-like feature. A + generic corner detector (Shi-Tomasi/Harris) fires just as happily on + a diamond marker's vertex or a single line's endpoint kink as on a + real grid intersection, this is specific to the one shape we + actually want: positive along both the horizontal and vertical arm, + negative in the four quadrant gaps between them (a lone single- + direction line only lights up one arm and loses on the other three + quadrants plus the missing arm, scoring far below a true crossing). + size/arm_width are in PIXELS, computed by the caller from the + current image's own pitch, never a fixed constant, a 25px kernel + tuned for a 750px cell is meaningless on a 150px cell.""" + size = max(int(size) | 1, 9) # odd, sane minimum + arm_width = max(int(round(arm_width)), 1) + k = np.full((size, size), -1.0, dtype=np.float32) + c = size // 2 + half = arm_width // 2 + k[c-half:c+half+1, :] = 1.0 + k[:, c-half:c+half+1] = 1.0 + k -= k.mean() + k /= np.abs(k).sum() + return k + + +def find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, window_frac=0.12, + label_bbox=None, label_margin=6, sat_thresh=60, min_score=0.12): + """Locate the grid-line intersection nearest this label's anchor + guess, by convolving a cross/intersection matched filter (see + cross_kernel()) against a local, saturation-masked, label-masked + window, then taking the response peak, weighted by a Gaussian + falloff in distance from the guess so a stronger-but-farther false + cross elsewhere in the window doesn't win over the real, closer one. + window_frac is deliberately small: the anchor guess (from label + position + pitch, itself derived from real label spacing) should + already be close, a small margin covers its own slop without + covering enough area to catch an unrelated intersection.""" + H, W = gray.shape + # search window and match kernel both scale off THIS image's pitch, + # not fixed pixel constants, so this works whether a cell is 150px or + # 750px across. Kernel arm needs to reach far enough to genuinely + # distinguish 'line extends in this direction' from noise, but must + # stay smaller than the window it slides within. + wx, wy = max(window_frac*pitch_x, 12), max(window_frac*pitch_y, 12) + x0, x1 = max(0,int(guess_x-wx)), min(W,int(guess_x+wx)) + y0, y1 = max(0,int(guess_y-wy)), min(H,int(guess_y+wy)) + kernel_size = max(int(0.9 * min(wx, wy)), 7) + kernel_arm = max(pitch_x, pitch_y) * 0.01 + cross = cross_kernel(kernel_size, kernel_arm) + blur_kernel = max(int(kernel_arm * 4) | 1, 5) + if x1-x0 < cross.shape[1] or y1-y0 < cross.shape[0]: + return None, (guess_x, guess_y), (x0,y0,x1,y1) + roi = gray[y0:y1, x0:x1].astype(np.float32) + baseline = cv2.medianBlur(gray[y0:y1, x0:x1], blur_kernel).astype(np.float32) + excess = np.clip(roi - baseline, 0, 60) + sat_roi = hsv_sat[y0:y1, x0:x1] + excess[sat_roi > sat_thresh] = 0 + if label_bbox is not None: + lx0 = int(label_bbox['x']) - label_margin - x0 + ly0 = int(label_bbox['y']) - label_margin - y0 + lx1 = int(label_bbox['x'] + label_bbox['w']) + label_margin - x0 + ly1 = int(label_bbox['y'] + label_bbox['h']) + label_margin - y0 + lx0, ly0 = max(0, lx0), max(0, ly0) + lx1, ly1 = min(excess.shape[1], lx1), min(excess.shape[0], ly1) + if lx1 > lx0 and ly1 > ly0: + excess[ly0:ly1, lx0:lx1] = 0 + + response = cv2.filter2D(excess, -1, cross) + half = cross.shape[0] // 2 + response[:half, :] = -1e9; response[-half:, :] = -1e9 + response[:, :half] = -1e9; response[:, -half:] = -1e9 + + # let the matched-filter response do the actual selecting (find the + # strongest genuine cross in the window), the Gaussian prior only + # nudges among near-tied candidates and sanity-checks the winner + # isn't implausibly far from the guess, it was previously multiplied + # straight into the per-pixel score, which let a weak-but-central + # false response beat a real, stronger crossing nearby. + yy, xx = np.mgrid[0:response.shape[0], 0:response.shape[1]].astype(np.float32) + gx, gy = guess_x - x0, guess_y - y0 + sigma = 1.5 * min(wx, wy) + prior = np.exp(-((xx-gx)**2 + (yy-gy)**2) / (2*sigma**2)) + weighted = response * (0.85 + 0.15*prior) + py, px = np.unravel_index(np.argmax(weighted), weighted.shape) + peak_response = response[py, px] + if peak_response < min_score * np.abs(cross).sum() * 60: + return None, (guess_x, guess_y), (x0,y0,x1,y1) + return (float(px+x0), float(py+y0)), (guess_x, guess_y), (x0,y0,x1,y1) + + +def find_label_corner(gray, hsv_sat, label, pitch_x, pitch_y, **kw): + guess_x = label['x'] - ANCHOR_FRAC_X*pitch_x + guess_y = label['y'] - ANCHOR_FRAC_Y*pitch_y + return find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, label_bbox=label, **kw) + + +def span_grid_crossings(gray, hsv_sat, labels, pitch_x, pitch_y, margin_cells=1): + """Once we have a rough pitch/origin from however many labels OCR'd + (even just 2-3), the whole grid is regular, so predict and directly + test EVERY crossing across the visible frame, not just the ones next + to a label that happened to be readable. Returns (ideal_pts, + img_pts, debug) for every crossing that matched; a bad/outlier label + just contributes points that RANSAC discards downstream rather than + limiting how much of the grid we ever attempt. + + dc = column offset in cells from the reference label. dr = offset in + cells DOWN the image (increasing y) from the reference, so it moves + opposite to row number (row 8 sits above row 7 on screen): the ideal + row coordinate is -(ref_number - dr) = dr - ref_number, matching the + (col, -row_number) convention used everywhere else in this module.""" + H, W = gray.shape + if not labels: + return [], [], [] + ref = max(labels, key=lambda lb: lb['conf']) + ref_col = LARGE_X.index(ref['letter']) + ref_corner_x = ref['x'] - ANCHOR_FRAC_X*pitch_x + ref_corner_y = ref['y'] - ANCHOR_FRAC_Y*pitch_y + + dc_lo = int(np.floor((0 - ref_corner_x) / pitch_x)) - margin_cells + dc_hi = int(np.ceil((W - ref_corner_x) / pitch_x)) + margin_cells + dr_lo = int(np.floor((0 - ref_corner_y) / pitch_y)) - margin_cells + dr_hi = int(np.ceil((H - ref_corner_y) / pitch_y)) + margin_cells + + ideal_pts, img_pts, debug = [], [], [] + for dc in range(dc_lo, dc_hi + 1): + for dr in range(dr_lo, dr_hi + 1): + gx = ref_corner_x + dc * pitch_x + gy = ref_corner_y + dr * pitch_y + if not (0 <= gx < W and 0 <= gy < H): + continue + corner, guess, window = find_crossing(gray, hsv_sat, gx, gy, pitch_x, pitch_y) + debug.append((guess, corner, window)) + if corner is not None: + ideal_pts.append((ref_col + dc, dr - ref['number'])) + img_pts.append(corner) + return ideal_pts, img_pts, debug + + +def has_diversity(ideal_pts): + cols = set(p[0] for p in ideal_pts) + rows = set(p[1] for p in ideal_pts) + return len(cols) >= 2 and len(rows) >= 2 + + +def fit_grid_homography(ideal_pts, img_pts): + if len(ideal_pts) < 4 or not has_diversity(ideal_pts): + return None + ideal = np.array(ideal_pts, dtype=np.float32) + img = np.array(img_pts, dtype=np.float32) + Hmat, mask = cv2.findHomography(ideal, img, cv2.RANSAC, 15.0) + return Hmat, mask