dodox/FEnigma
dodox/FEnigma
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: map-vision pipeline status + BFS grid-growing plan

Browse Source 
Session produced a working prototype (docs/map_vision_wip.py, not
wired into the app yet) for the image-based map-screenshot marker
pipeline: label OCR, pitch estimation from label spacing, a matched-
filter grid-intersection detector, whole-grid crossing prediction, and
RANSAC homography fitting, each validated against real screenshots
with diagnostic images along the way.

Found a real remaining bug before pausing: crossings are currently
all predicted from one single reference label using one global pitch,
so predictions drift with distance from that reference under genuine
perspective distortion (confirmed: ~2-4 degree measured tilt, not
noise). Documented the fix (BFS grid-growing: expand one cell at a
time from every confirmed point, using local rather than global
spacing) as the next step, not yet implemented.
This commit is contained in:
Dominik Moritz Roth 2026-08-09 14:49:40 +02:00
parent b48a4eb759
commit c5bcbd3bce
2 changed files with 401 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

165
docs/map_vision_plan.md Normal file
View File

@ -0,0 +1,165 @@
# Map screenshot vision pipeline: status and next-step plan
Goal: given a screenshot of the in-game map view (not the typewriter text), detect
enemy unit markers (red diamonds per the in-game legend) and resolve each one's
position (large grid + small grid) automatically, for import as targets
(position + id only, type stays `TargetType.UNKNOWN` for this first pass).
This is a *second*, entirely separate OCR/vision pipeline from `ocr.py`'s text
pipeline. `ocr.py`'s module docstring already flags this as future/out-of-scope
work; this doc is that future work's design record.
## Where the code lives right now
`docs/map_vision_wip.py` in this repo is a copy of the working prototype, as of
the end of this exploration session. It is **not wired into the app** and not
`src/fenigma/map_vision.py` yet — it's scratchpad-quality (developed and tested
against three sample screenshots via ad-hoc test scripts, not a proper test
suite). Treat it as a strong starting point, not finished code: variable names,
error handling, and docstrings need a pass before it belongs in `src/`.
## What's validated and working
1. **Grid label OCR** (`find_grid_labels`): sliding-window Tesseract sweep
across the image, filtered to the `[A-T](10|[1-9])` pattern
(e.g. `M8`, `Q10`). Works reasonably well on clean labels; struggles when a
colored hatch-line overlay crosses directly through a label's glyphs
(`L7` misread as `AF`, `N8` as `NWS`) — tried desaturating before OCR,
didn't help (recoloring the hatch line neutral still leaves a shape gap in
the letter). **Needs real inpainting** (fill the interrupted stroke from
surrounding pixels) to fix, not yet done.
2. **Pitch estimation** (`estimate_pitch`): derives cell pitch (both x and y)
from the actual pixel spacing between same-row / same-column labels. No
hardcoded pitch constant anywhere — this was an explicit, correct call-out
mid-session (an earlier version hardcoded `pitch=749` from eyeballing one
image; that doesn't generalize and was thrown out).
3. **Corner/intersection matching** (`find_crossing`, `cross_kernel`): given a
predicted pixel position for a grid-line intersection, finds the real one
nearby. This went through several broken iterations before landing on the
current approach — worth remembering *why* each earlier attempt failed,
so they don't get reinvented:
- Plain local brightness-percentile search: worked on one lucky isolated
line, completely unreliable elsewhere (confirmed via zoomed crop: a
claimed "corner" for `L8` sat in flat background, nowhere near any real
line).
- Generic `cv2.goodFeaturesToTrack`: fires on *any* strong corner, so it
reliably found the label's own text glyphs or a nearby diamond marker's
vertex instead of the grid intersection. Fixed by masking out the
label's own bounding box, then *also* masking out anything with high
HSV saturation (markers and hatching are deliberately colored; the grid
itself is neutral gray/white — a generalizable distinction, not
per-image tuning).
- Even with masking, a generic corner detector still isn't picking the
right *kind* of corner (a line's endpoint kink looks the same as a true
4-way crossing to it). Fixed by replacing it entirely with
`cross_kernel`: a matched filter shaped like a bright `+` (positive
along both a horizontal and vertical arm through the center, negative
in the four quadrant gaps) — a lone single-direction line only lights
up one arm and scores far below a true crossing. This is the piece
that actually made precise matches possible (verified to land within a
couple pixels of a manually-confirmed true corner).
- The matched-filter kernel size and the search-window size must both be
computed **from the current image's own pitch**, never a fixed pixel
constant — a kernel/window tuned for a 750px cell is meaningless on a
150px cell. `find_crossing` takes `pitch_x, pitch_y` and derives both
from them.
- Candidate selection within the window: originally multiplied the
matched-filter response by a tight Gaussian prior (centered on the
predicted position) *before* taking the max — this let a weak,
coincidentally-central false response beat a real, stronger crossing
nearby ("prior is too aggressive, matches predicted center" was the
exact bug report). Fixed by widening the Gaussian (`sigma = 1.5 *
window radius`) and only blending it in lightly (`0.85 + 0.15*prior`)
so the actual filter response does most of the selecting; the prior
now only functions as a mild tie-breaker plus a final plausibility
check on the winner, not the primary selection mechanism.
- The "plausible region" is a Gaussian, so it should be *drawn* as a
circle in diagnostics, not a rectangle — a rectangle visually implies a
hard cutoff that doesn't reflect the actual model. Fixed in the debug
visualization.
4. **Whole-grid crossing prediction** (`span_grid_crossings`): once we have
*any* labels (even 2-3), the grid is regular, so predict and test every
crossing across the visible frame, not just the ones adjacent to a label
that happened to OCR cleanly. This gives far more correspondence points
than "one per successfully-read label." Validated end to end on the
zoomed-out strategic-overview screenshot: 4 labels → 10 predicted
crossings → 4 matched → homography fit with all 4 as RANSAC inliers.
5. **Homography fit** (`fit_grid_homography`): `cv2.findHomography(...,
cv2.RANSAC, 15.0)`, requires 4+ points with real geometric diversity (2+
distinct columns AND 2+ distinct rows — `has_diversity`) before even
attempting a fit, refusing collinear/degenerate input rather than
producing garbage. This replaced an earlier rigid-rotation-only
(translation + single theta) model once it became clear the real screen
has genuine perspective/keystone distortion (parallel lines don't stay
parallel), confirmed by directly measuring the same line's x-position at
two widely-separated y-values and finding a real, consistent ~2-4° drift,
not noise.
## The known remaining bug, and the planned fix
**Bug**: with exactly 4 matched points, `cv2.findHomography` always fits them
*exactly* — 0 residual and "4/4 inliers" is true by construction and doesn't
mean the fit is actually good. Confirmed visually: a 4-point fit on 4 points
that happened to form a lopsided "staircase" in grid-space (missing two
corners of what should've been a proper 2×2 block) produced a visibly
skewed parallelogram instead of a rectangle, even though every individual
point matched its real intersection correctly. The 4 points were individually
right; the *set* was too small and too oddly-shaped to constrain the fit
meaningfully.
Root cause of *why* only 4/10 predicted crossings matched: `span_grid_crossings`
predicts every crossing from **one single reference label** using one global
pitch value for the whole image. Since perspective distortion is real, that
single global (origin, pitch) pair drifts further from the truth the farther a
predicted crossing is from the reference — a jump of several cells accumulates
several cells' worth of drift before the search window even starts looking.
(Some other misses were legitimately unfindable — busy photo texture with no
clean line at that exact spot, confirmed via a zoomed crop — but the long-range
extrapolation drift is the fixable, systematic part.)
### Planned fix: grid-growing (BFS) instead of batch prediction from one origin
Don't predict the whole grid from one fixed point. Walk outward one cell at a
time from every confirmed point, correcting the local estimate as you go:
1. **Seeds**: every label that OCR'd *and* corner-matched successfully is a
confirmed `(grid_col, grid_row) -> (pixel_x, pixel_y)` point. Multiple
seeds, not just the single highest-confidence label.
2. **Expand one cell at a time**: from each confirmed point, only ever
predict its *immediate* neighbor (one cell in one of the 4 directions) —
never extrapolate further than one cell from something already confirmed.
3. **Prefer local spacing over the global average**: if two confirmed points
already share a row (for a column step) or column (for a row step), use
*their* measured spacing to predict the next one out — that's the real
local pitch right there. Only fall back to the global pitch estimate for
the very first step away from a seed, where no local measurement exists
yet.
4. **Search, confirm, repeat**: run the same `find_crossing` search at that
one-cell-away prediction. Success → add to the confirmed set, push onto
the expansion frontier. Failure → that one edge just stops there, doesn't
block expansion from other confirmed points nearby.
5. Keep expanding (a plain BFS/queue over grid coordinates, with a
visited/attempted set so failed edges aren't retried forever) until the
frontier is empty. Feed every confirmed point into the homography fit —
likely dozens of points instead of 4, each individually short-range and
therefore much less exposed to long-range perspective drift, with enough
redundancy that the residual/inlier check from RANSAC actually means
something instead of being a vacuous exact-fit.
This is a genuine restructure of `span_grid_crossings`'s control flow (batch
prediction → BFS), not a parameter tweak. Implementing it is the next step
when this work resumes.
## Also still open (lower priority than the BFS fix)
- Label OCR robustness where hatching crosses the glyph (needs inpainting).
- Once the grid calibration is reliably robust: red-diamond blob detection
(color threshold + connected components) — not started.
- Matching each detected blob to its `#N` id label via nearby OCR — not
started.
- Wiring into `app.py`: a new button, merging results through the existing
`_merge_targets`-style flow as `TargetType.UNKNOWN` — not started.

236
docs/map_vision_wip.py Normal file
View File

@ -0,0 +1,236 @@
"""Generalized (no per-image constants) map-screenshot grid calibration.
find_grid_labels() -> OCR sweep for visible large-grid labels.
estimate_pitch() -> derive cell pitch from label-to-label spacing.
find_label_corner()-> per-label corner search via a real corner detector
(cv2.goodFeaturesToTrack), window scaled to pitch.
fit_grid_homography() -> RANSAC homography from however many corners
were found, refusing to fit degenerate/thin data.
"""
import re
import cv2
import numpy as np
import pytesseract
LARGE_X = "ABCDEFGHIJKLMNOPQRST"
LABEL_RE = re.compile(r'^([A-T])(10|[1-9])$')
ANCHOR_FRAC_X = 0.10
ANCHOR_FRAC_Y = 0.15
def desaturate(rgb):
"""Replace saturated (colored) pixels with a neutral gray of the same
brightness. The hatching overlay and unit markers are drawn in
deliberately saturated colors while the grid + its labels are neutral
white/cream on a grayscale photo, hatch lines crossing straight
through a label glyph otherwise corrupt its shape enough to break
OCR (seen: 'L7' -> 'AF', 'N8' -> 'NWS')."""
hsv = cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV)
sat = hsv[:, :, 1]
gray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
out = gray.copy()
return out, sat
def find_grid_labels(gray, band_height=160, band_step=60, scale=3.0, min_conf=30):
H, W = gray.shape
config = "--psm 11 -c tessedit_char_whitelist=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
found = {}
for y0 in range(0, H, band_step):
y1 = min(y0 + band_height, H)
band = gray[y0:y1, :]
big = cv2.resize(band, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC)
data = pytesseract.image_to_data(big, output_type=pytesseract.Output.DICT, config=config)
for i, txt in enumerate(data['text']):
t = txt.strip()
m = LABEL_RE.match(t)
if not m or int(data['conf'][i]) < min_conf:
continue
conf = int(data['conf'][i])
key = t
if key not in found or conf > found[key]['conf']:
found[key] = dict(
text=t, letter=m.group(1), number=int(m.group(2)),
x=data['left'][i]/scale, y=y0 + data['top'][i]/scale,
w=data['width'][i]/scale, h=data['height'][i]/scale, conf=conf,
)
return list(found.values())
def estimate_pitch(labels):
xs_by_row, ys_by_col = {}, {}
for lb in labels:
xs_by_row.setdefault(lb['number'], []).append((LARGE_X.index(lb['letter']), lb['x']+lb['w']/2))
ys_by_col.setdefault(lb['letter'], []).append((lb['number'], lb['y']+lb['h']/2))
px = []
for pts in xs_by_row.values():
pts.sort()
for (ci,xi),(cj,xj) in zip(pts, pts[1:]):
if cj > ci:
px.append((xj-xi)/(cj-ci))
py = []
for pts in ys_by_col.values():
pts.sort()
for (ri,yi),(rj,yj) in zip(pts, pts[1:]):
if rj != ri:
py.append(abs(yj-yi)/abs(rj-ri))
pitch_x = float(np.median(px)) if px else None
pitch_y = float(np.median(py)) if py else None
if pitch_x and not pitch_y:
pitch_y = pitch_x
if pitch_y and not pitch_x:
pitch_x = pitch_y
return pitch_x, pitch_y
def cross_kernel(size, arm_width):
"""Matched filter for a bright axis-aligned line CROSSING in both
directions through the center, not just any corner-like feature. A
generic corner detector (Shi-Tomasi/Harris) fires just as happily on
a diamond marker's vertex or a single line's endpoint kink as on a
real grid intersection, this is specific to the one shape we
actually want: positive along both the horizontal and vertical arm,
negative in the four quadrant gaps between them (a lone single-
direction line only lights up one arm and loses on the other three
quadrants plus the missing arm, scoring far below a true crossing).
size/arm_width are in PIXELS, computed by the caller from the
current image's own pitch, never a fixed constant, a 25px kernel
tuned for a 750px cell is meaningless on a 150px cell."""
size = max(int(size) | 1, 9) # odd, sane minimum
arm_width = max(int(round(arm_width)), 1)
k = np.full((size, size), -1.0, dtype=np.float32)
c = size // 2
half = arm_width // 2
k[c-half:c+half+1, :] = 1.0
k[:, c-half:c+half+1] = 1.0
k -= k.mean()
k /= np.abs(k).sum()
return k
def find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, window_frac=0.12,
label_bbox=None, label_margin=6, sat_thresh=60, min_score=0.12):
"""Locate the grid-line intersection nearest this label's anchor
guess, by convolving a cross/intersection matched filter (see
cross_kernel()) against a local, saturation-masked, label-masked
window, then taking the response peak, weighted by a Gaussian
falloff in distance from the guess so a stronger-but-farther false
cross elsewhere in the window doesn't win over the real, closer one.
window_frac is deliberately small: the anchor guess (from label
position + pitch, itself derived from real label spacing) should
already be close, a small margin covers its own slop without
covering enough area to catch an unrelated intersection."""
H, W = gray.shape
# search window and match kernel both scale off THIS image's pitch,
# not fixed pixel constants, so this works whether a cell is 150px or
# 750px across. Kernel arm needs to reach far enough to genuinely
# distinguish 'line extends in this direction' from noise, but must
# stay smaller than the window it slides within.
wx, wy = max(window_frac*pitch_x, 12), max(window_frac*pitch_y, 12)
x0, x1 = max(0,int(guess_x-wx)), min(W,int(guess_x+wx))
y0, y1 = max(0,int(guess_y-wy)), min(H,int(guess_y+wy))
kernel_size = max(int(0.9 * min(wx, wy)), 7)
kernel_arm = max(pitch_x, pitch_y) * 0.01
cross = cross_kernel(kernel_size, kernel_arm)
blur_kernel = max(int(kernel_arm * 4) | 1, 5)
if x1-x0 < cross.shape[1] or y1-y0 < cross.shape[0]:
return None, (guess_x, guess_y), (x0,y0,x1,y1)
roi = gray[y0:y1, x0:x1].astype(np.float32)
baseline = cv2.medianBlur(gray[y0:y1, x0:x1], blur_kernel).astype(np.float32)
excess = np.clip(roi - baseline, 0, 60)
sat_roi = hsv_sat[y0:y1, x0:x1]
excess[sat_roi > sat_thresh] = 0
if label_bbox is not None:
lx0 = int(label_bbox['x']) - label_margin - x0
ly0 = int(label_bbox['y']) - label_margin - y0
lx1 = int(label_bbox['x'] + label_bbox['w']) + label_margin - x0
ly1 = int(label_bbox['y'] + label_bbox['h']) + label_margin - y0
lx0, ly0 = max(0, lx0), max(0, ly0)
lx1, ly1 = min(excess.shape[1], lx1), min(excess.shape[0], ly1)
if lx1 > lx0 and ly1 > ly0:
excess[ly0:ly1, lx0:lx1] = 0
response = cv2.filter2D(excess, -1, cross)
half = cross.shape[0] // 2
response[:half, :] = -1e9; response[-half:, :] = -1e9
response[:, :half] = -1e9; response[:, -half:] = -1e9
# let the matched-filter response do the actual selecting (find the
# strongest genuine cross in the window), the Gaussian prior only
# nudges among near-tied candidates and sanity-checks the winner
# isn't implausibly far from the guess, it was previously multiplied
# straight into the per-pixel score, which let a weak-but-central
# false response beat a real, stronger crossing nearby.
yy, xx = np.mgrid[0:response.shape[0], 0:response.shape[1]].astype(np.float32)
gx, gy = guess_x - x0, guess_y - y0
sigma = 1.5 * min(wx, wy)
prior = np.exp(-((xx-gx)**2 + (yy-gy)**2) / (2*sigma**2))
weighted = response * (0.85 + 0.15*prior)
py, px = np.unravel_index(np.argmax(weighted), weighted.shape)
peak_response = response[py, px]
if peak_response < min_score * np.abs(cross).sum() * 60:
return None, (guess_x, guess_y), (x0,y0,x1,y1)
return (float(px+x0), float(py+y0)), (guess_x, guess_y), (x0,y0,x1,y1)
def find_label_corner(gray, hsv_sat, label, pitch_x, pitch_y, **kw):
guess_x = label['x'] - ANCHOR_FRAC_X*pitch_x
guess_y = label['y'] - ANCHOR_FRAC_Y*pitch_y
return find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, label_bbox=label, **kw)
def span_grid_crossings(gray, hsv_sat, labels, pitch_x, pitch_y, margin_cells=1):
"""Once we have a rough pitch/origin from however many labels OCR'd
(even just 2-3), the whole grid is regular, so predict and directly
test EVERY crossing across the visible frame, not just the ones next
to a label that happened to be readable. Returns (ideal_pts,
img_pts, debug) for every crossing that matched; a bad/outlier label
just contributes points that RANSAC discards downstream rather than
limiting how much of the grid we ever attempt.
dc = column offset in cells from the reference label. dr = offset in
cells DOWN the image (increasing y) from the reference, so it moves
opposite to row number (row 8 sits above row 7 on screen): the ideal
row coordinate is -(ref_number - dr) = dr - ref_number, matching the
(col, -row_number) convention used everywhere else in this module."""
H, W = gray.shape
if not labels:
return [], [], []
ref = max(labels, key=lambda lb: lb['conf'])
ref_col = LARGE_X.index(ref['letter'])
ref_corner_x = ref['x'] - ANCHOR_FRAC_X*pitch_x
ref_corner_y = ref['y'] - ANCHOR_FRAC_Y*pitch_y
dc_lo = int(np.floor((0 - ref_corner_x) / pitch_x)) - margin_cells
dc_hi = int(np.ceil((W - ref_corner_x) / pitch_x)) + margin_cells
dr_lo = int(np.floor((0 - ref_corner_y) / pitch_y)) - margin_cells
dr_hi = int(np.ceil((H - ref_corner_y) / pitch_y)) + margin_cells
ideal_pts, img_pts, debug = [], [], []
for dc in range(dc_lo, dc_hi + 1):
for dr in range(dr_lo, dr_hi + 1):
gx = ref_corner_x + dc * pitch_x
gy = ref_corner_y + dr * pitch_y
if not (0 <= gx < W and 0 <= gy < H):
continue
corner, guess, window = find_crossing(gray, hsv_sat, gx, gy, pitch_x, pitch_y)
debug.append((guess, corner, window))
if corner is not None:
ideal_pts.append((ref_col + dc, dr - ref['number']))
img_pts.append(corner)
return ideal_pts, img_pts, debug
def has_diversity(ideal_pts):
cols = set(p[0] for p in ideal_pts)
rows = set(p[1] for p in ideal_pts)
return len(cols) >= 2 and len(rows) >= 2
def fit_grid_homography(ideal_pts, img_pts):
if len(ideal_pts) < 4 or not has_diversity(ideal_pts):
return None
ideal = np.array(ideal_pts, dtype=np.float32)
img = np.array(img_pts, dtype=np.float32)
Hmat, mask = cv2.findHomography(ideal, img, cv2.RANSAC, 15.0)
return Hmat, mask