docs: map-vision pipeline status + BFS grid-growing plan
Session produced a working prototype (docs/map_vision_wip.py, not wired into the app yet) for the image-based map-screenshot marker pipeline: label OCR, pitch estimation from label spacing, a matched- filter grid-intersection detector, whole-grid crossing prediction, and RANSAC homography fitting, each validated against real screenshots with diagnostic images along the way. Found a real remaining bug before pausing: crossings are currently all predicted from one single reference label using one global pitch, so predictions drift with distance from that reference under genuine perspective distortion (confirmed: ~2-4 degree measured tilt, not noise). Documented the fix (BFS grid-growing: expand one cell at a time from every confirmed point, using local rather than global spacing) as the next step, not yet implemented.
This commit is contained in:
parent
b48a4eb759
commit
c5bcbd3bce
165
docs/map_vision_plan.md
Normal file
165
docs/map_vision_plan.md
Normal file
@ -0,0 +1,165 @@
|
|||||||
|
# Map screenshot vision pipeline: status and next-step plan
|
||||||
|
|
||||||
|
Goal: given a screenshot of the in-game map view (not the typewriter text), detect
|
||||||
|
enemy unit markers (red diamonds per the in-game legend) and resolve each one's
|
||||||
|
position (large grid + small grid) automatically, for import as targets
|
||||||
|
(position + id only, type stays `TargetType.UNKNOWN` for this first pass).
|
||||||
|
|
||||||
|
This is a *second*, entirely separate OCR/vision pipeline from `ocr.py`'s text
|
||||||
|
pipeline. `ocr.py`'s module docstring already flags this as future/out-of-scope
|
||||||
|
work; this doc is that future work's design record.
|
||||||
|
|
||||||
|
## Where the code lives right now
|
||||||
|
|
||||||
|
`docs/map_vision_wip.py` in this repo is a copy of the working prototype, as of
|
||||||
|
the end of this exploration session. It is **not wired into the app** and not
|
||||||
|
`src/fenigma/map_vision.py` yet — it's scratchpad-quality (developed and tested
|
||||||
|
against three sample screenshots via ad-hoc test scripts, not a proper test
|
||||||
|
suite). Treat it as a strong starting point, not finished code: variable names,
|
||||||
|
error handling, and docstrings need a pass before it belongs in `src/`.
|
||||||
|
|
||||||
|
## What's validated and working
|
||||||
|
|
||||||
|
1. **Grid label OCR** (`find_grid_labels`): sliding-window Tesseract sweep
|
||||||
|
across the image, filtered to the `[A-T](10|[1-9])` pattern
|
||||||
|
(e.g. `M8`, `Q10`). Works reasonably well on clean labels; struggles when a
|
||||||
|
colored hatch-line overlay crosses directly through a label's glyphs
|
||||||
|
(`L7` misread as `AF`, `N8` as `NWS`) — tried desaturating before OCR,
|
||||||
|
didn't help (recoloring the hatch line neutral still leaves a shape gap in
|
||||||
|
the letter). **Needs real inpainting** (fill the interrupted stroke from
|
||||||
|
surrounding pixels) to fix, not yet done.
|
||||||
|
|
||||||
|
2. **Pitch estimation** (`estimate_pitch`): derives cell pitch (both x and y)
|
||||||
|
from the actual pixel spacing between same-row / same-column labels. No
|
||||||
|
hardcoded pitch constant anywhere — this was an explicit, correct call-out
|
||||||
|
mid-session (an earlier version hardcoded `pitch=749` from eyeballing one
|
||||||
|
image; that doesn't generalize and was thrown out).
|
||||||
|
|
||||||
|
3. **Corner/intersection matching** (`find_crossing`, `cross_kernel`): given a
|
||||||
|
predicted pixel position for a grid-line intersection, finds the real one
|
||||||
|
nearby. This went through several broken iterations before landing on the
|
||||||
|
current approach — worth remembering *why* each earlier attempt failed,
|
||||||
|
so they don't get reinvented:
|
||||||
|
- Plain local brightness-percentile search: worked on one lucky isolated
|
||||||
|
line, completely unreliable elsewhere (confirmed via zoomed crop: a
|
||||||
|
claimed "corner" for `L8` sat in flat background, nowhere near any real
|
||||||
|
line).
|
||||||
|
- Generic `cv2.goodFeaturesToTrack`: fires on *any* strong corner, so it
|
||||||
|
reliably found the label's own text glyphs or a nearby diamond marker's
|
||||||
|
vertex instead of the grid intersection. Fixed by masking out the
|
||||||
|
label's own bounding box, then *also* masking out anything with high
|
||||||
|
HSV saturation (markers and hatching are deliberately colored; the grid
|
||||||
|
itself is neutral gray/white — a generalizable distinction, not
|
||||||
|
per-image tuning).
|
||||||
|
- Even with masking, a generic corner detector still isn't picking the
|
||||||
|
right *kind* of corner (a line's endpoint kink looks the same as a true
|
||||||
|
4-way crossing to it). Fixed by replacing it entirely with
|
||||||
|
`cross_kernel`: a matched filter shaped like a bright `+` (positive
|
||||||
|
along both a horizontal and vertical arm through the center, negative
|
||||||
|
in the four quadrant gaps) — a lone single-direction line only lights
|
||||||
|
up one arm and scores far below a true crossing. This is the piece
|
||||||
|
that actually made precise matches possible (verified to land within a
|
||||||
|
couple pixels of a manually-confirmed true corner).
|
||||||
|
- The matched-filter kernel size and the search-window size must both be
|
||||||
|
computed **from the current image's own pitch**, never a fixed pixel
|
||||||
|
constant — a kernel/window tuned for a 750px cell is meaningless on a
|
||||||
|
150px cell. `find_crossing` takes `pitch_x, pitch_y` and derives both
|
||||||
|
from them.
|
||||||
|
- Candidate selection within the window: originally multiplied the
|
||||||
|
matched-filter response by a tight Gaussian prior (centered on the
|
||||||
|
predicted position) *before* taking the max — this let a weak,
|
||||||
|
coincidentally-central false response beat a real, stronger crossing
|
||||||
|
nearby ("prior is too aggressive, matches predicted center" was the
|
||||||
|
exact bug report). Fixed by widening the Gaussian (`sigma = 1.5 *
|
||||||
|
window radius`) and only blending it in lightly (`0.85 + 0.15*prior`)
|
||||||
|
so the actual filter response does most of the selecting; the prior
|
||||||
|
now only functions as a mild tie-breaker plus a final plausibility
|
||||||
|
check on the winner, not the primary selection mechanism.
|
||||||
|
- The "plausible region" is a Gaussian, so it should be *drawn* as a
|
||||||
|
circle in diagnostics, not a rectangle — a rectangle visually implies a
|
||||||
|
hard cutoff that doesn't reflect the actual model. Fixed in the debug
|
||||||
|
visualization.
|
||||||
|
|
||||||
|
4. **Whole-grid crossing prediction** (`span_grid_crossings`): once we have
|
||||||
|
*any* labels (even 2-3), the grid is regular, so predict and test every
|
||||||
|
crossing across the visible frame, not just the ones adjacent to a label
|
||||||
|
that happened to OCR cleanly. This gives far more correspondence points
|
||||||
|
than "one per successfully-read label." Validated end to end on the
|
||||||
|
zoomed-out strategic-overview screenshot: 4 labels → 10 predicted
|
||||||
|
crossings → 4 matched → homography fit with all 4 as RANSAC inliers.
|
||||||
|
|
||||||
|
5. **Homography fit** (`fit_grid_homography`): `cv2.findHomography(...,
|
||||||
|
cv2.RANSAC, 15.0)`, requires 4+ points with real geometric diversity (2+
|
||||||
|
distinct columns AND 2+ distinct rows — `has_diversity`) before even
|
||||||
|
attempting a fit, refusing collinear/degenerate input rather than
|
||||||
|
producing garbage. This replaced an earlier rigid-rotation-only
|
||||||
|
(translation + single theta) model once it became clear the real screen
|
||||||
|
has genuine perspective/keystone distortion (parallel lines don't stay
|
||||||
|
parallel), confirmed by directly measuring the same line's x-position at
|
||||||
|
two widely-separated y-values and finding a real, consistent ~2-4° drift,
|
||||||
|
not noise.
|
||||||
|
|
||||||
|
## The known remaining bug, and the planned fix
|
||||||
|
|
||||||
|
**Bug**: with exactly 4 matched points, `cv2.findHomography` always fits them
|
||||||
|
*exactly* — 0 residual and "4/4 inliers" is true by construction and doesn't
|
||||||
|
mean the fit is actually good. Confirmed visually: a 4-point fit on 4 points
|
||||||
|
that happened to form a lopsided "staircase" in grid-space (missing two
|
||||||
|
corners of what should've been a proper 2×2 block) produced a visibly
|
||||||
|
skewed parallelogram instead of a rectangle, even though every individual
|
||||||
|
point matched its real intersection correctly. The 4 points were individually
|
||||||
|
right; the *set* was too small and too oddly-shaped to constrain the fit
|
||||||
|
meaningfully.
|
||||||
|
|
||||||
|
Root cause of *why* only 4/10 predicted crossings matched: `span_grid_crossings`
|
||||||
|
predicts every crossing from **one single reference label** using one global
|
||||||
|
pitch value for the whole image. Since perspective distortion is real, that
|
||||||
|
single global (origin, pitch) pair drifts further from the truth the farther a
|
||||||
|
predicted crossing is from the reference — a jump of several cells accumulates
|
||||||
|
several cells' worth of drift before the search window even starts looking.
|
||||||
|
(Some other misses were legitimately unfindable — busy photo texture with no
|
||||||
|
clean line at that exact spot, confirmed via a zoomed crop — but the long-range
|
||||||
|
extrapolation drift is the fixable, systematic part.)
|
||||||
|
|
||||||
|
### Planned fix: grid-growing (BFS) instead of batch prediction from one origin
|
||||||
|
|
||||||
|
Don't predict the whole grid from one fixed point. Walk outward one cell at a
|
||||||
|
time from every confirmed point, correcting the local estimate as you go:
|
||||||
|
|
||||||
|
1. **Seeds**: every label that OCR'd *and* corner-matched successfully is a
|
||||||
|
confirmed `(grid_col, grid_row) -> (pixel_x, pixel_y)` point. Multiple
|
||||||
|
seeds, not just the single highest-confidence label.
|
||||||
|
2. **Expand one cell at a time**: from each confirmed point, only ever
|
||||||
|
predict its *immediate* neighbor (one cell in one of the 4 directions) —
|
||||||
|
never extrapolate further than one cell from something already confirmed.
|
||||||
|
3. **Prefer local spacing over the global average**: if two confirmed points
|
||||||
|
already share a row (for a column step) or column (for a row step), use
|
||||||
|
*their* measured spacing to predict the next one out — that's the real
|
||||||
|
local pitch right there. Only fall back to the global pitch estimate for
|
||||||
|
the very first step away from a seed, where no local measurement exists
|
||||||
|
yet.
|
||||||
|
4. **Search, confirm, repeat**: run the same `find_crossing` search at that
|
||||||
|
one-cell-away prediction. Success → add to the confirmed set, push onto
|
||||||
|
the expansion frontier. Failure → that one edge just stops there, doesn't
|
||||||
|
block expansion from other confirmed points nearby.
|
||||||
|
5. Keep expanding (a plain BFS/queue over grid coordinates, with a
|
||||||
|
visited/attempted set so failed edges aren't retried forever) until the
|
||||||
|
frontier is empty. Feed every confirmed point into the homography fit —
|
||||||
|
likely dozens of points instead of 4, each individually short-range and
|
||||||
|
therefore much less exposed to long-range perspective drift, with enough
|
||||||
|
redundancy that the residual/inlier check from RANSAC actually means
|
||||||
|
something instead of being a vacuous exact-fit.
|
||||||
|
|
||||||
|
This is a genuine restructure of `span_grid_crossings`'s control flow (batch
|
||||||
|
prediction → BFS), not a parameter tweak. Implementing it is the next step
|
||||||
|
when this work resumes.
|
||||||
|
|
||||||
|
## Also still open (lower priority than the BFS fix)
|
||||||
|
|
||||||
|
- Label OCR robustness where hatching crosses the glyph (needs inpainting).
|
||||||
|
- Once the grid calibration is reliably robust: red-diamond blob detection
|
||||||
|
(color threshold + connected components) — not started.
|
||||||
|
- Matching each detected blob to its `#N` id label via nearby OCR — not
|
||||||
|
started.
|
||||||
|
- Wiring into `app.py`: a new button, merging results through the existing
|
||||||
|
`_merge_targets`-style flow as `TargetType.UNKNOWN` — not started.
|
||||||
236
docs/map_vision_wip.py
Normal file
236
docs/map_vision_wip.py
Normal file
@ -0,0 +1,236 @@
|
|||||||
|
"""Generalized (no per-image constants) map-screenshot grid calibration.
|
||||||
|
|
||||||
|
find_grid_labels() -> OCR sweep for visible large-grid labels.
|
||||||
|
estimate_pitch() -> derive cell pitch from label-to-label spacing.
|
||||||
|
find_label_corner()-> per-label corner search via a real corner detector
|
||||||
|
(cv2.goodFeaturesToTrack), window scaled to pitch.
|
||||||
|
fit_grid_homography() -> RANSAC homography from however many corners
|
||||||
|
were found, refusing to fit degenerate/thin data.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
import pytesseract
|
||||||
|
|
||||||
|
LARGE_X = "ABCDEFGHIJKLMNOPQRST"
|
||||||
|
LABEL_RE = re.compile(r'^([A-T])(10|[1-9])$')
|
||||||
|
ANCHOR_FRAC_X = 0.10
|
||||||
|
ANCHOR_FRAC_Y = 0.15
|
||||||
|
|
||||||
|
|
||||||
|
def desaturate(rgb):
|
||||||
|
"""Replace saturated (colored) pixels with a neutral gray of the same
|
||||||
|
brightness. The hatching overlay and unit markers are drawn in
|
||||||
|
deliberately saturated colors while the grid + its labels are neutral
|
||||||
|
white/cream on a grayscale photo, hatch lines crossing straight
|
||||||
|
through a label glyph otherwise corrupt its shape enough to break
|
||||||
|
OCR (seen: 'L7' -> 'AF', 'N8' -> 'NWS')."""
|
||||||
|
hsv = cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV)
|
||||||
|
sat = hsv[:, :, 1]
|
||||||
|
gray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
|
||||||
|
out = gray.copy()
|
||||||
|
return out, sat
|
||||||
|
|
||||||
|
|
||||||
|
def find_grid_labels(gray, band_height=160, band_step=60, scale=3.0, min_conf=30):
|
||||||
|
H, W = gray.shape
|
||||||
|
config = "--psm 11 -c tessedit_char_whitelist=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
|
||||||
|
found = {}
|
||||||
|
for y0 in range(0, H, band_step):
|
||||||
|
y1 = min(y0 + band_height, H)
|
||||||
|
band = gray[y0:y1, :]
|
||||||
|
big = cv2.resize(band, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC)
|
||||||
|
data = pytesseract.image_to_data(big, output_type=pytesseract.Output.DICT, config=config)
|
||||||
|
for i, txt in enumerate(data['text']):
|
||||||
|
t = txt.strip()
|
||||||
|
m = LABEL_RE.match(t)
|
||||||
|
if not m or int(data['conf'][i]) < min_conf:
|
||||||
|
continue
|
||||||
|
conf = int(data['conf'][i])
|
||||||
|
key = t
|
||||||
|
if key not in found or conf > found[key]['conf']:
|
||||||
|
found[key] = dict(
|
||||||
|
text=t, letter=m.group(1), number=int(m.group(2)),
|
||||||
|
x=data['left'][i]/scale, y=y0 + data['top'][i]/scale,
|
||||||
|
w=data['width'][i]/scale, h=data['height'][i]/scale, conf=conf,
|
||||||
|
)
|
||||||
|
return list(found.values())
|
||||||
|
|
||||||
|
|
||||||
|
def estimate_pitch(labels):
|
||||||
|
xs_by_row, ys_by_col = {}, {}
|
||||||
|
for lb in labels:
|
||||||
|
xs_by_row.setdefault(lb['number'], []).append((LARGE_X.index(lb['letter']), lb['x']+lb['w']/2))
|
||||||
|
ys_by_col.setdefault(lb['letter'], []).append((lb['number'], lb['y']+lb['h']/2))
|
||||||
|
px = []
|
||||||
|
for pts in xs_by_row.values():
|
||||||
|
pts.sort()
|
||||||
|
for (ci,xi),(cj,xj) in zip(pts, pts[1:]):
|
||||||
|
if cj > ci:
|
||||||
|
px.append((xj-xi)/(cj-ci))
|
||||||
|
py = []
|
||||||
|
for pts in ys_by_col.values():
|
||||||
|
pts.sort()
|
||||||
|
for (ri,yi),(rj,yj) in zip(pts, pts[1:]):
|
||||||
|
if rj != ri:
|
||||||
|
py.append(abs(yj-yi)/abs(rj-ri))
|
||||||
|
pitch_x = float(np.median(px)) if px else None
|
||||||
|
pitch_y = float(np.median(py)) if py else None
|
||||||
|
if pitch_x and not pitch_y:
|
||||||
|
pitch_y = pitch_x
|
||||||
|
if pitch_y and not pitch_x:
|
||||||
|
pitch_x = pitch_y
|
||||||
|
return pitch_x, pitch_y
|
||||||
|
|
||||||
|
|
||||||
|
def cross_kernel(size, arm_width):
|
||||||
|
"""Matched filter for a bright axis-aligned line CROSSING in both
|
||||||
|
directions through the center, not just any corner-like feature. A
|
||||||
|
generic corner detector (Shi-Tomasi/Harris) fires just as happily on
|
||||||
|
a diamond marker's vertex or a single line's endpoint kink as on a
|
||||||
|
real grid intersection, this is specific to the one shape we
|
||||||
|
actually want: positive along both the horizontal and vertical arm,
|
||||||
|
negative in the four quadrant gaps between them (a lone single-
|
||||||
|
direction line only lights up one arm and loses on the other three
|
||||||
|
quadrants plus the missing arm, scoring far below a true crossing).
|
||||||
|
size/arm_width are in PIXELS, computed by the caller from the
|
||||||
|
current image's own pitch, never a fixed constant, a 25px kernel
|
||||||
|
tuned for a 750px cell is meaningless on a 150px cell."""
|
||||||
|
size = max(int(size) | 1, 9) # odd, sane minimum
|
||||||
|
arm_width = max(int(round(arm_width)), 1)
|
||||||
|
k = np.full((size, size), -1.0, dtype=np.float32)
|
||||||
|
c = size // 2
|
||||||
|
half = arm_width // 2
|
||||||
|
k[c-half:c+half+1, :] = 1.0
|
||||||
|
k[:, c-half:c+half+1] = 1.0
|
||||||
|
k -= k.mean()
|
||||||
|
k /= np.abs(k).sum()
|
||||||
|
return k
|
||||||
|
|
||||||
|
|
||||||
|
def find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, window_frac=0.12,
|
||||||
|
label_bbox=None, label_margin=6, sat_thresh=60, min_score=0.12):
|
||||||
|
"""Locate the grid-line intersection nearest this label's anchor
|
||||||
|
guess, by convolving a cross/intersection matched filter (see
|
||||||
|
cross_kernel()) against a local, saturation-masked, label-masked
|
||||||
|
window, then taking the response peak, weighted by a Gaussian
|
||||||
|
falloff in distance from the guess so a stronger-but-farther false
|
||||||
|
cross elsewhere in the window doesn't win over the real, closer one.
|
||||||
|
window_frac is deliberately small: the anchor guess (from label
|
||||||
|
position + pitch, itself derived from real label spacing) should
|
||||||
|
already be close, a small margin covers its own slop without
|
||||||
|
covering enough area to catch an unrelated intersection."""
|
||||||
|
H, W = gray.shape
|
||||||
|
# search window and match kernel both scale off THIS image's pitch,
|
||||||
|
# not fixed pixel constants, so this works whether a cell is 150px or
|
||||||
|
# 750px across. Kernel arm needs to reach far enough to genuinely
|
||||||
|
# distinguish 'line extends in this direction' from noise, but must
|
||||||
|
# stay smaller than the window it slides within.
|
||||||
|
wx, wy = max(window_frac*pitch_x, 12), max(window_frac*pitch_y, 12)
|
||||||
|
x0, x1 = max(0,int(guess_x-wx)), min(W,int(guess_x+wx))
|
||||||
|
y0, y1 = max(0,int(guess_y-wy)), min(H,int(guess_y+wy))
|
||||||
|
kernel_size = max(int(0.9 * min(wx, wy)), 7)
|
||||||
|
kernel_arm = max(pitch_x, pitch_y) * 0.01
|
||||||
|
cross = cross_kernel(kernel_size, kernel_arm)
|
||||||
|
blur_kernel = max(int(kernel_arm * 4) | 1, 5)
|
||||||
|
if x1-x0 < cross.shape[1] or y1-y0 < cross.shape[0]:
|
||||||
|
return None, (guess_x, guess_y), (x0,y0,x1,y1)
|
||||||
|
roi = gray[y0:y1, x0:x1].astype(np.float32)
|
||||||
|
baseline = cv2.medianBlur(gray[y0:y1, x0:x1], blur_kernel).astype(np.float32)
|
||||||
|
excess = np.clip(roi - baseline, 0, 60)
|
||||||
|
sat_roi = hsv_sat[y0:y1, x0:x1]
|
||||||
|
excess[sat_roi > sat_thresh] = 0
|
||||||
|
if label_bbox is not None:
|
||||||
|
lx0 = int(label_bbox['x']) - label_margin - x0
|
||||||
|
ly0 = int(label_bbox['y']) - label_margin - y0
|
||||||
|
lx1 = int(label_bbox['x'] + label_bbox['w']) + label_margin - x0
|
||||||
|
ly1 = int(label_bbox['y'] + label_bbox['h']) + label_margin - y0
|
||||||
|
lx0, ly0 = max(0, lx0), max(0, ly0)
|
||||||
|
lx1, ly1 = min(excess.shape[1], lx1), min(excess.shape[0], ly1)
|
||||||
|
if lx1 > lx0 and ly1 > ly0:
|
||||||
|
excess[ly0:ly1, lx0:lx1] = 0
|
||||||
|
|
||||||
|
response = cv2.filter2D(excess, -1, cross)
|
||||||
|
half = cross.shape[0] // 2
|
||||||
|
response[:half, :] = -1e9; response[-half:, :] = -1e9
|
||||||
|
response[:, :half] = -1e9; response[:, -half:] = -1e9
|
||||||
|
|
||||||
|
# let the matched-filter response do the actual selecting (find the
|
||||||
|
# strongest genuine cross in the window), the Gaussian prior only
|
||||||
|
# nudges among near-tied candidates and sanity-checks the winner
|
||||||
|
# isn't implausibly far from the guess, it was previously multiplied
|
||||||
|
# straight into the per-pixel score, which let a weak-but-central
|
||||||
|
# false response beat a real, stronger crossing nearby.
|
||||||
|
yy, xx = np.mgrid[0:response.shape[0], 0:response.shape[1]].astype(np.float32)
|
||||||
|
gx, gy = guess_x - x0, guess_y - y0
|
||||||
|
sigma = 1.5 * min(wx, wy)
|
||||||
|
prior = np.exp(-((xx-gx)**2 + (yy-gy)**2) / (2*sigma**2))
|
||||||
|
weighted = response * (0.85 + 0.15*prior)
|
||||||
|
py, px = np.unravel_index(np.argmax(weighted), weighted.shape)
|
||||||
|
peak_response = response[py, px]
|
||||||
|
if peak_response < min_score * np.abs(cross).sum() * 60:
|
||||||
|
return None, (guess_x, guess_y), (x0,y0,x1,y1)
|
||||||
|
return (float(px+x0), float(py+y0)), (guess_x, guess_y), (x0,y0,x1,y1)
|
||||||
|
|
||||||
|
|
||||||
|
def find_label_corner(gray, hsv_sat, label, pitch_x, pitch_y, **kw):
|
||||||
|
guess_x = label['x'] - ANCHOR_FRAC_X*pitch_x
|
||||||
|
guess_y = label['y'] - ANCHOR_FRAC_Y*pitch_y
|
||||||
|
return find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, label_bbox=label, **kw)
|
||||||
|
|
||||||
|
|
||||||
|
def span_grid_crossings(gray, hsv_sat, labels, pitch_x, pitch_y, margin_cells=1):
|
||||||
|
"""Once we have a rough pitch/origin from however many labels OCR'd
|
||||||
|
(even just 2-3), the whole grid is regular, so predict and directly
|
||||||
|
test EVERY crossing across the visible frame, not just the ones next
|
||||||
|
to a label that happened to be readable. Returns (ideal_pts,
|
||||||
|
img_pts, debug) for every crossing that matched; a bad/outlier label
|
||||||
|
just contributes points that RANSAC discards downstream rather than
|
||||||
|
limiting how much of the grid we ever attempt.
|
||||||
|
|
||||||
|
dc = column offset in cells from the reference label. dr = offset in
|
||||||
|
cells DOWN the image (increasing y) from the reference, so it moves
|
||||||
|
opposite to row number (row 8 sits above row 7 on screen): the ideal
|
||||||
|
row coordinate is -(ref_number - dr) = dr - ref_number, matching the
|
||||||
|
(col, -row_number) convention used everywhere else in this module."""
|
||||||
|
H, W = gray.shape
|
||||||
|
if not labels:
|
||||||
|
return [], [], []
|
||||||
|
ref = max(labels, key=lambda lb: lb['conf'])
|
||||||
|
ref_col = LARGE_X.index(ref['letter'])
|
||||||
|
ref_corner_x = ref['x'] - ANCHOR_FRAC_X*pitch_x
|
||||||
|
ref_corner_y = ref['y'] - ANCHOR_FRAC_Y*pitch_y
|
||||||
|
|
||||||
|
dc_lo = int(np.floor((0 - ref_corner_x) / pitch_x)) - margin_cells
|
||||||
|
dc_hi = int(np.ceil((W - ref_corner_x) / pitch_x)) + margin_cells
|
||||||
|
dr_lo = int(np.floor((0 - ref_corner_y) / pitch_y)) - margin_cells
|
||||||
|
dr_hi = int(np.ceil((H - ref_corner_y) / pitch_y)) + margin_cells
|
||||||
|
|
||||||
|
ideal_pts, img_pts, debug = [], [], []
|
||||||
|
for dc in range(dc_lo, dc_hi + 1):
|
||||||
|
for dr in range(dr_lo, dr_hi + 1):
|
||||||
|
gx = ref_corner_x + dc * pitch_x
|
||||||
|
gy = ref_corner_y + dr * pitch_y
|
||||||
|
if not (0 <= gx < W and 0 <= gy < H):
|
||||||
|
continue
|
||||||
|
corner, guess, window = find_crossing(gray, hsv_sat, gx, gy, pitch_x, pitch_y)
|
||||||
|
debug.append((guess, corner, window))
|
||||||
|
if corner is not None:
|
||||||
|
ideal_pts.append((ref_col + dc, dr - ref['number']))
|
||||||
|
img_pts.append(corner)
|
||||||
|
return ideal_pts, img_pts, debug
|
||||||
|
|
||||||
|
|
||||||
|
def has_diversity(ideal_pts):
|
||||||
|
cols = set(p[0] for p in ideal_pts)
|
||||||
|
rows = set(p[1] for p in ideal_pts)
|
||||||
|
return len(cols) >= 2 and len(rows) >= 2
|
||||||
|
|
||||||
|
|
||||||
|
def fit_grid_homography(ideal_pts, img_pts):
|
||||||
|
if len(ideal_pts) < 4 or not has_diversity(ideal_pts):
|
||||||
|
return None
|
||||||
|
ideal = np.array(ideal_pts, dtype=np.float32)
|
||||||
|
img = np.array(img_pts, dtype=np.float32)
|
||||||
|
Hmat, mask = cv2.findHomography(ideal, img, cv2.RANSAC, 15.0)
|
||||||
|
return Hmat, mask
|
||||||
Loading…
Reference in New Issue
Block a user