Add the map-table vision pipeline
Reads a screenshot of the in-game map table and recovers where on the board it is looking. The scene is a flat table under a perspective camera, so a single homography describes grid<->screen exactly; that is fitted from line evidence (LSD segments, vanishing-point RANSAC, 1-D lattice fits). Line evidence alone cannot finish the job: the 1km cells and the 100m subgrid are locally identical, so it fixes neither scale, axis assignment, axis direction nor phase. Those come from the cell labels -- and the labels are never *detected*, they are correlated where the grid says they must be (9% in from a cell's left edge, 6% down from its top). Blob detection on aerial-photo terrain finds texture, not glyphs; correlating a known template at a known place has no such failure mode. The same score then ranks the geometry hypotheses, since a wrong lattice puts the crop where no label is, so one number resolves scale, axis assignment, direction, phase and anchor together. Markers are found by hostile/friendly colour plus an IoU test against an ideal inscribed diamond, which cut a red-lit shot from 43 false positives to 2 while preserving every hand-verified count. Unit-type classification is present but not yet reliable, and returns unknown rather than guessing. Measured over the fixture set: 7 of 10 solve, each with 100% of its ground-truth points in the correct cell (85/112 overall), residual spread 0.005-0.033 cells. The other three reject cleanly; none has ever produced a plausible-but-wrong grid. Across 122 typewriter screenshots solve() accepted none, which is what makes clipboard routing safe. Fixtures: 10 map shots with hand-transcribed ground truth, plus 10 typewriter shots spanning 262-5366px for routing and future OCR tests. Map shots wider than 2400px (the pipeline's own maximum working width) were downscaled with their coordinates rescaled to match; re-measured afterwards, the results are identical. Typewriter shots stay at native resolution because OCR needs the text legible. Drops docs/map_vision_plan.md and its WIP prototype: the design now lives in the module docstring, next to the code it describes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
512a0a41b4
commit
ff3b89c41b
2
.gitignore
vendored
2
.gitignore
vendored
@ -3,3 +3,5 @@ __pycache__/
|
||||
captures/*.png
|
||||
.venv/
|
||||
GameAssets
|
||||
# tools/eval_map_vision.py renders its overlays here
|
||||
build/
|
||||
|
||||
BIN
assets/fonts/CourierPrime-Regular.ttf
Normal file
BIN
assets/fonts/CourierPrime-Regular.ttf
Normal file
Binary file not shown.
50
assets/fonts/OFL.txt
Normal file
50
assets/fonts/OFL.txt
Normal file
@ -0,0 +1,50 @@
|
||||
Copyright (c) 2013, Quote-Unquote Apps (http://quoteunquoteapps.com), with Reserved Font Name Courier Prime.
|
||||
|
||||
This Font Software is licensed under the SIL Open Font License, Version 1.1. This license is copied below, and is also available with a FAQ at: http://scripts.sil.org/OFL
|
||||
|
||||
-----------------------------------------------------------
|
||||
|
||||
SIL OPEN FONT LICENSE
|
||||
Version 1.1 - 26 February 2007
|
||||
|
||||
-----------------------------------------------------------
|
||||
|
||||
PREAMBLE
|
||||
|
||||
The goals of the Open Font License (OFL) are to stimulate worldwide development of collaborative font projects, to support the font creation efforts of academic and linguistic communities, and to provide a free and open framework in which fonts may be shared and improved in partnership with others.
|
||||
|
||||
The OFL allows the licensed fonts to be used, studied, modified and redistributed freely as long as they are not sold by themselves. The fonts, including any derivative works, can be bundled, embedded, redistributed and/or sold with any software provided that any reserved names are not used by derivative works. The fonts and derivatives, however, cannot be released under any other type of license. The requirement for fonts to remain under this license does not apply to any document created using the fonts or their derivatives.
|
||||
|
||||
DEFINITIONS
|
||||
|
||||
"Font Software" refers to the set of files released by the Copyright Holder(s) under this license and clearly marked as such. This may include source files, build scripts and documentation.
|
||||
|
||||
"Reserved Font Name" refers to any names specified as such after the copyright statement(s).
|
||||
|
||||
"Original Version" refers to the collection of Font Software components as distributed by the Copyright Holder(s).
|
||||
|
||||
"Modified Version" refers to any derivative made by adding to, deleting, or substituting -- in part or in whole -- any of the components of the Original Version, by changing formats or by porting the Font Software to a new environment.
|
||||
|
||||
"Author" refers to any designer, engineer, programmer, technical writer or other person who contributed to the Font Software.
|
||||
|
||||
PERMISSION & CONDITIONS
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of the Font Software, to use, study, copy, merge, embed, modify, redistribute, and sell modified and unmodified copies of the Font Software, subject to the following conditions:
|
||||
|
||||
1) Neither the Font Software nor any of its individual components, in Original or Modified Versions, may be sold by itself.
|
||||
|
||||
2) Original or Modified Versions of the Font Software may be bundled, redistributed and/or sold with any software, provided that each copy contains the above copyright notice and this license. These can be included either as stand-alone text files, human-readable headers or in the appropriate machine-readable metadata fields within text or binary files as long as those fields can be easily viewed by the user.
|
||||
|
||||
3) No Modified Version of the Font Software may use the Reserved Font Name(s) unless explicit written permission is granted by the corresponding Copyright Holder. This restriction only applies to the primary font name as presented to the users.
|
||||
|
||||
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font Software shall not be used to promote, endorse or advertise any Modified Version, except to acknowledge the contribution(s) of the Copyright Holder(s) and the Author(s) or with their explicit written permission.
|
||||
|
||||
5) The Font Software, modified or unmodified, in part or in whole, must be distributed entirely under this license, and must not be distributed under any other license. The requirement for fonts to remain under this license does not apply to any document created using the Font Software.
|
||||
|
||||
TERMINATION
|
||||
|
||||
This license becomes null and void if any of the above conditions are not met.
|
||||
|
||||
DISCLAIMER
|
||||
|
||||
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE.
|
||||
@ -1,165 +0,0 @@
|
||||
# Map screenshot vision pipeline: status and next-step plan
|
||||
|
||||
Goal: given a screenshot of the in-game map view (not the typewriter text), detect
|
||||
enemy unit markers (red diamonds per the in-game legend) and resolve each one's
|
||||
position (large grid + small grid) automatically, for import as targets
|
||||
(position + id only, type stays `TargetType.UNKNOWN` for this first pass).
|
||||
|
||||
This is a *second*, entirely separate OCR/vision pipeline from `ocr.py`'s text
|
||||
pipeline. `ocr.py`'s module docstring already flags this as future/out-of-scope
|
||||
work; this doc is that future work's design record.
|
||||
|
||||
## Where the code lives right now
|
||||
|
||||
`docs/map_vision_wip.py` in this repo is a copy of the working prototype, as of
|
||||
the end of this exploration session. It is **not wired into the app** and not
|
||||
`src/fenigma/map_vision.py` yet — it's scratchpad-quality (developed and tested
|
||||
against three sample screenshots via ad-hoc test scripts, not a proper test
|
||||
suite). Treat it as a strong starting point, not finished code: variable names,
|
||||
error handling, and docstrings need a pass before it belongs in `src/`.
|
||||
|
||||
## What's validated and working
|
||||
|
||||
1. **Grid label OCR** (`find_grid_labels`): sliding-window Tesseract sweep
|
||||
across the image, filtered to the `[A-T](10|[1-9])` pattern
|
||||
(e.g. `M8`, `Q10`). Works reasonably well on clean labels; struggles when a
|
||||
colored hatch-line overlay crosses directly through a label's glyphs
|
||||
(`L7` misread as `AF`, `N8` as `NWS`) — tried desaturating before OCR,
|
||||
didn't help (recoloring the hatch line neutral still leaves a shape gap in
|
||||
the letter). **Needs real inpainting** (fill the interrupted stroke from
|
||||
surrounding pixels) to fix, not yet done.
|
||||
|
||||
2. **Pitch estimation** (`estimate_pitch`): derives cell pitch (both x and y)
|
||||
from the actual pixel spacing between same-row / same-column labels. No
|
||||
hardcoded pitch constant anywhere — this was an explicit, correct call-out
|
||||
mid-session (an earlier version hardcoded `pitch=749` from eyeballing one
|
||||
image; that doesn't generalize and was thrown out).
|
||||
|
||||
3. **Corner/intersection matching** (`find_crossing`, `cross_kernel`): given a
|
||||
predicted pixel position for a grid-line intersection, finds the real one
|
||||
nearby. This went through several broken iterations before landing on the
|
||||
current approach — worth remembering *why* each earlier attempt failed,
|
||||
so they don't get reinvented:
|
||||
- Plain local brightness-percentile search: worked on one lucky isolated
|
||||
line, completely unreliable elsewhere (confirmed via zoomed crop: a
|
||||
claimed "corner" for `L8` sat in flat background, nowhere near any real
|
||||
line).
|
||||
- Generic `cv2.goodFeaturesToTrack`: fires on *any* strong corner, so it
|
||||
reliably found the label's own text glyphs or a nearby diamond marker's
|
||||
vertex instead of the grid intersection. Fixed by masking out the
|
||||
label's own bounding box, then *also* masking out anything with high
|
||||
HSV saturation (markers and hatching are deliberately colored; the grid
|
||||
itself is neutral gray/white — a generalizable distinction, not
|
||||
per-image tuning).
|
||||
- Even with masking, a generic corner detector still isn't picking the
|
||||
right *kind* of corner (a line's endpoint kink looks the same as a true
|
||||
4-way crossing to it). Fixed by replacing it entirely with
|
||||
`cross_kernel`: a matched filter shaped like a bright `+` (positive
|
||||
along both a horizontal and vertical arm through the center, negative
|
||||
in the four quadrant gaps) — a lone single-direction line only lights
|
||||
up one arm and scores far below a true crossing. This is the piece
|
||||
that actually made precise matches possible (verified to land within a
|
||||
couple pixels of a manually-confirmed true corner).
|
||||
- The matched-filter kernel size and the search-window size must both be
|
||||
computed **from the current image's own pitch**, never a fixed pixel
|
||||
constant — a kernel/window tuned for a 750px cell is meaningless on a
|
||||
150px cell. `find_crossing` takes `pitch_x, pitch_y` and derives both
|
||||
from them.
|
||||
- Candidate selection within the window: originally multiplied the
|
||||
matched-filter response by a tight Gaussian prior (centered on the
|
||||
predicted position) *before* taking the max — this let a weak,
|
||||
coincidentally-central false response beat a real, stronger crossing
|
||||
nearby ("prior is too aggressive, matches predicted center" was the
|
||||
exact bug report). Fixed by widening the Gaussian (`sigma = 1.5 *
|
||||
window radius`) and only blending it in lightly (`0.85 + 0.15*prior`)
|
||||
so the actual filter response does most of the selecting; the prior
|
||||
now only functions as a mild tie-breaker plus a final plausibility
|
||||
check on the winner, not the primary selection mechanism.
|
||||
- The "plausible region" is a Gaussian, so it should be *drawn* as a
|
||||
circle in diagnostics, not a rectangle — a rectangle visually implies a
|
||||
hard cutoff that doesn't reflect the actual model. Fixed in the debug
|
||||
visualization.
|
||||
|
||||
4. **Whole-grid crossing prediction** (`span_grid_crossings`): once we have
|
||||
*any* labels (even 2-3), the grid is regular, so predict and test every
|
||||
crossing across the visible frame, not just the ones adjacent to a label
|
||||
that happened to OCR cleanly. This gives far more correspondence points
|
||||
than "one per successfully-read label." Validated end to end on the
|
||||
zoomed-out strategic-overview screenshot: 4 labels → 10 predicted
|
||||
crossings → 4 matched → homography fit with all 4 as RANSAC inliers.
|
||||
|
||||
5. **Homography fit** (`fit_grid_homography`): `cv2.findHomography(...,
|
||||
cv2.RANSAC, 15.0)`, requires 4+ points with real geometric diversity (2+
|
||||
distinct columns AND 2+ distinct rows — `has_diversity`) before even
|
||||
attempting a fit, refusing collinear/degenerate input rather than
|
||||
producing garbage. This replaced an earlier rigid-rotation-only
|
||||
(translation + single theta) model once it became clear the real screen
|
||||
has genuine perspective/keystone distortion (parallel lines don't stay
|
||||
parallel), confirmed by directly measuring the same line's x-position at
|
||||
two widely-separated y-values and finding a real, consistent ~2-4° drift,
|
||||
not noise.
|
||||
|
||||
## The known remaining bug, and the planned fix
|
||||
|
||||
**Bug**: with exactly 4 matched points, `cv2.findHomography` always fits them
|
||||
*exactly* — 0 residual and "4/4 inliers" is true by construction and doesn't
|
||||
mean the fit is actually good. Confirmed visually: a 4-point fit on 4 points
|
||||
that happened to form a lopsided "staircase" in grid-space (missing two
|
||||
corners of what should've been a proper 2×2 block) produced a visibly
|
||||
skewed parallelogram instead of a rectangle, even though every individual
|
||||
point matched its real intersection correctly. The 4 points were individually
|
||||
right; the *set* was too small and too oddly-shaped to constrain the fit
|
||||
meaningfully.
|
||||
|
||||
Root cause of *why* only 4/10 predicted crossings matched: `span_grid_crossings`
|
||||
predicts every crossing from **one single reference label** using one global
|
||||
pitch value for the whole image. Since perspective distortion is real, that
|
||||
single global (origin, pitch) pair drifts further from the truth the farther a
|
||||
predicted crossing is from the reference — a jump of several cells accumulates
|
||||
several cells' worth of drift before the search window even starts looking.
|
||||
(Some other misses were legitimately unfindable — busy photo texture with no
|
||||
clean line at that exact spot, confirmed via a zoomed crop — but the long-range
|
||||
extrapolation drift is the fixable, systematic part.)
|
||||
|
||||
### Planned fix: grid-growing (BFS) instead of batch prediction from one origin
|
||||
|
||||
Don't predict the whole grid from one fixed point. Walk outward one cell at a
|
||||
time from every confirmed point, correcting the local estimate as you go:
|
||||
|
||||
1. **Seeds**: every label that OCR'd *and* corner-matched successfully is a
|
||||
confirmed `(grid_col, grid_row) -> (pixel_x, pixel_y)` point. Multiple
|
||||
seeds, not just the single highest-confidence label.
|
||||
2. **Expand one cell at a time**: from each confirmed point, only ever
|
||||
predict its *immediate* neighbor (one cell in one of the 4 directions) —
|
||||
never extrapolate further than one cell from something already confirmed.
|
||||
3. **Prefer local spacing over the global average**: if two confirmed points
|
||||
already share a row (for a column step) or column (for a row step), use
|
||||
*their* measured spacing to predict the next one out — that's the real
|
||||
local pitch right there. Only fall back to the global pitch estimate for
|
||||
the very first step away from a seed, where no local measurement exists
|
||||
yet.
|
||||
4. **Search, confirm, repeat**: run the same `find_crossing` search at that
|
||||
one-cell-away prediction. Success → add to the confirmed set, push onto
|
||||
the expansion frontier. Failure → that one edge just stops there, doesn't
|
||||
block expansion from other confirmed points nearby.
|
||||
5. Keep expanding (a plain BFS/queue over grid coordinates, with a
|
||||
visited/attempted set so failed edges aren't retried forever) until the
|
||||
frontier is empty. Feed every confirmed point into the homography fit —
|
||||
likely dozens of points instead of 4, each individually short-range and
|
||||
therefore much less exposed to long-range perspective drift, with enough
|
||||
redundancy that the residual/inlier check from RANSAC actually means
|
||||
something instead of being a vacuous exact-fit.
|
||||
|
||||
This is a genuine restructure of `span_grid_crossings`'s control flow (batch
|
||||
prediction → BFS), not a parameter tweak. Implementing it is the next step
|
||||
when this work resumes.
|
||||
|
||||
## Also still open (lower priority than the BFS fix)
|
||||
|
||||
- Label OCR robustness where hatching crosses the glyph (needs inpainting).
|
||||
- Once the grid calibration is reliably robust: red-diamond blob detection
|
||||
(color threshold + connected components) — not started.
|
||||
- Matching each detected blob to its `#N` id label via nearby OCR — not
|
||||
started.
|
||||
- Wiring into `app.py`: a new button, merging results through the existing
|
||||
`_merge_targets`-style flow as `TargetType.UNKNOWN` — not started.
|
||||
@ -1,236 +0,0 @@
|
||||
"""Generalized (no per-image constants) map-screenshot grid calibration.
|
||||
|
||||
find_grid_labels() -> OCR sweep for visible large-grid labels.
|
||||
estimate_pitch() -> derive cell pitch from label-to-label spacing.
|
||||
find_label_corner()-> per-label corner search via a real corner detector
|
||||
(cv2.goodFeaturesToTrack), window scaled to pitch.
|
||||
fit_grid_homography() -> RANSAC homography from however many corners
|
||||
were found, refusing to fit degenerate/thin data.
|
||||
"""
|
||||
import re
|
||||
import cv2
|
||||
import numpy as np
|
||||
import pytesseract
|
||||
|
||||
LARGE_X = "ABCDEFGHIJKLMNOPQRST"
|
||||
LABEL_RE = re.compile(r'^([A-T])(10|[1-9])$')
|
||||
ANCHOR_FRAC_X = 0.10
|
||||
ANCHOR_FRAC_Y = 0.15
|
||||
|
||||
|
||||
def desaturate(rgb):
|
||||
"""Replace saturated (colored) pixels with a neutral gray of the same
|
||||
brightness. The hatching overlay and unit markers are drawn in
|
||||
deliberately saturated colors while the grid + its labels are neutral
|
||||
white/cream on a grayscale photo, hatch lines crossing straight
|
||||
through a label glyph otherwise corrupt its shape enough to break
|
||||
OCR (seen: 'L7' -> 'AF', 'N8' -> 'NWS')."""
|
||||
hsv = cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV)
|
||||
sat = hsv[:, :, 1]
|
||||
gray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
|
||||
out = gray.copy()
|
||||
return out, sat
|
||||
|
||||
|
||||
def find_grid_labels(gray, band_height=160, band_step=60, scale=3.0, min_conf=30):
|
||||
H, W = gray.shape
|
||||
config = "--psm 11 -c tessedit_char_whitelist=ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
|
||||
found = {}
|
||||
for y0 in range(0, H, band_step):
|
||||
y1 = min(y0 + band_height, H)
|
||||
band = gray[y0:y1, :]
|
||||
big = cv2.resize(band, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC)
|
||||
data = pytesseract.image_to_data(big, output_type=pytesseract.Output.DICT, config=config)
|
||||
for i, txt in enumerate(data['text']):
|
||||
t = txt.strip()
|
||||
m = LABEL_RE.match(t)
|
||||
if not m or int(data['conf'][i]) < min_conf:
|
||||
continue
|
||||
conf = int(data['conf'][i])
|
||||
key = t
|
||||
if key not in found or conf > found[key]['conf']:
|
||||
found[key] = dict(
|
||||
text=t, letter=m.group(1), number=int(m.group(2)),
|
||||
x=data['left'][i]/scale, y=y0 + data['top'][i]/scale,
|
||||
w=data['width'][i]/scale, h=data['height'][i]/scale, conf=conf,
|
||||
)
|
||||
return list(found.values())
|
||||
|
||||
|
||||
def estimate_pitch(labels):
|
||||
xs_by_row, ys_by_col = {}, {}
|
||||
for lb in labels:
|
||||
xs_by_row.setdefault(lb['number'], []).append((LARGE_X.index(lb['letter']), lb['x']+lb['w']/2))
|
||||
ys_by_col.setdefault(lb['letter'], []).append((lb['number'], lb['y']+lb['h']/2))
|
||||
px = []
|
||||
for pts in xs_by_row.values():
|
||||
pts.sort()
|
||||
for (ci,xi),(cj,xj) in zip(pts, pts[1:]):
|
||||
if cj > ci:
|
||||
px.append((xj-xi)/(cj-ci))
|
||||
py = []
|
||||
for pts in ys_by_col.values():
|
||||
pts.sort()
|
||||
for (ri,yi),(rj,yj) in zip(pts, pts[1:]):
|
||||
if rj != ri:
|
||||
py.append(abs(yj-yi)/abs(rj-ri))
|
||||
pitch_x = float(np.median(px)) if px else None
|
||||
pitch_y = float(np.median(py)) if py else None
|
||||
if pitch_x and not pitch_y:
|
||||
pitch_y = pitch_x
|
||||
if pitch_y and not pitch_x:
|
||||
pitch_x = pitch_y
|
||||
return pitch_x, pitch_y
|
||||
|
||||
|
||||
def cross_kernel(size, arm_width):
|
||||
"""Matched filter for a bright axis-aligned line CROSSING in both
|
||||
directions through the center, not just any corner-like feature. A
|
||||
generic corner detector (Shi-Tomasi/Harris) fires just as happily on
|
||||
a diamond marker's vertex or a single line's endpoint kink as on a
|
||||
real grid intersection, this is specific to the one shape we
|
||||
actually want: positive along both the horizontal and vertical arm,
|
||||
negative in the four quadrant gaps between them (a lone single-
|
||||
direction line only lights up one arm and loses on the other three
|
||||
quadrants plus the missing arm, scoring far below a true crossing).
|
||||
size/arm_width are in PIXELS, computed by the caller from the
|
||||
current image's own pitch, never a fixed constant, a 25px kernel
|
||||
tuned for a 750px cell is meaningless on a 150px cell."""
|
||||
size = max(int(size) | 1, 9) # odd, sane minimum
|
||||
arm_width = max(int(round(arm_width)), 1)
|
||||
k = np.full((size, size), -1.0, dtype=np.float32)
|
||||
c = size // 2
|
||||
half = arm_width // 2
|
||||
k[c-half:c+half+1, :] = 1.0
|
||||
k[:, c-half:c+half+1] = 1.0
|
||||
k -= k.mean()
|
||||
k /= np.abs(k).sum()
|
||||
return k
|
||||
|
||||
|
||||
def find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, window_frac=0.12,
|
||||
label_bbox=None, label_margin=6, sat_thresh=60, min_score=0.12):
|
||||
"""Locate the grid-line intersection nearest this label's anchor
|
||||
guess, by convolving a cross/intersection matched filter (see
|
||||
cross_kernel()) against a local, saturation-masked, label-masked
|
||||
window, then taking the response peak, weighted by a Gaussian
|
||||
falloff in distance from the guess so a stronger-but-farther false
|
||||
cross elsewhere in the window doesn't win over the real, closer one.
|
||||
window_frac is deliberately small: the anchor guess (from label
|
||||
position + pitch, itself derived from real label spacing) should
|
||||
already be close, a small margin covers its own slop without
|
||||
covering enough area to catch an unrelated intersection."""
|
||||
H, W = gray.shape
|
||||
# search window and match kernel both scale off THIS image's pitch,
|
||||
# not fixed pixel constants, so this works whether a cell is 150px or
|
||||
# 750px across. Kernel arm needs to reach far enough to genuinely
|
||||
# distinguish 'line extends in this direction' from noise, but must
|
||||
# stay smaller than the window it slides within.
|
||||
wx, wy = max(window_frac*pitch_x, 12), max(window_frac*pitch_y, 12)
|
||||
x0, x1 = max(0,int(guess_x-wx)), min(W,int(guess_x+wx))
|
||||
y0, y1 = max(0,int(guess_y-wy)), min(H,int(guess_y+wy))
|
||||
kernel_size = max(int(0.9 * min(wx, wy)), 7)
|
||||
kernel_arm = max(pitch_x, pitch_y) * 0.01
|
||||
cross = cross_kernel(kernel_size, kernel_arm)
|
||||
blur_kernel = max(int(kernel_arm * 4) | 1, 5)
|
||||
if x1-x0 < cross.shape[1] or y1-y0 < cross.shape[0]:
|
||||
return None, (guess_x, guess_y), (x0,y0,x1,y1)
|
||||
roi = gray[y0:y1, x0:x1].astype(np.float32)
|
||||
baseline = cv2.medianBlur(gray[y0:y1, x0:x1], blur_kernel).astype(np.float32)
|
||||
excess = np.clip(roi - baseline, 0, 60)
|
||||
sat_roi = hsv_sat[y0:y1, x0:x1]
|
||||
excess[sat_roi > sat_thresh] = 0
|
||||
if label_bbox is not None:
|
||||
lx0 = int(label_bbox['x']) - label_margin - x0
|
||||
ly0 = int(label_bbox['y']) - label_margin - y0
|
||||
lx1 = int(label_bbox['x'] + label_bbox['w']) + label_margin - x0
|
||||
ly1 = int(label_bbox['y'] + label_bbox['h']) + label_margin - y0
|
||||
lx0, ly0 = max(0, lx0), max(0, ly0)
|
||||
lx1, ly1 = min(excess.shape[1], lx1), min(excess.shape[0], ly1)
|
||||
if lx1 > lx0 and ly1 > ly0:
|
||||
excess[ly0:ly1, lx0:lx1] = 0
|
||||
|
||||
response = cv2.filter2D(excess, -1, cross)
|
||||
half = cross.shape[0] // 2
|
||||
response[:half, :] = -1e9; response[-half:, :] = -1e9
|
||||
response[:, :half] = -1e9; response[:, -half:] = -1e9
|
||||
|
||||
# let the matched-filter response do the actual selecting (find the
|
||||
# strongest genuine cross in the window), the Gaussian prior only
|
||||
# nudges among near-tied candidates and sanity-checks the winner
|
||||
# isn't implausibly far from the guess, it was previously multiplied
|
||||
# straight into the per-pixel score, which let a weak-but-central
|
||||
# false response beat a real, stronger crossing nearby.
|
||||
yy, xx = np.mgrid[0:response.shape[0], 0:response.shape[1]].astype(np.float32)
|
||||
gx, gy = guess_x - x0, guess_y - y0
|
||||
sigma = 1.5 * min(wx, wy)
|
||||
prior = np.exp(-((xx-gx)**2 + (yy-gy)**2) / (2*sigma**2))
|
||||
weighted = response * (0.85 + 0.15*prior)
|
||||
py, px = np.unravel_index(np.argmax(weighted), weighted.shape)
|
||||
peak_response = response[py, px]
|
||||
if peak_response < min_score * np.abs(cross).sum() * 60:
|
||||
return None, (guess_x, guess_y), (x0,y0,x1,y1)
|
||||
return (float(px+x0), float(py+y0)), (guess_x, guess_y), (x0,y0,x1,y1)
|
||||
|
||||
|
||||
def find_label_corner(gray, hsv_sat, label, pitch_x, pitch_y, **kw):
|
||||
guess_x = label['x'] - ANCHOR_FRAC_X*pitch_x
|
||||
guess_y = label['y'] - ANCHOR_FRAC_Y*pitch_y
|
||||
return find_crossing(gray, hsv_sat, guess_x, guess_y, pitch_x, pitch_y, label_bbox=label, **kw)
|
||||
|
||||
|
||||
def span_grid_crossings(gray, hsv_sat, labels, pitch_x, pitch_y, margin_cells=1):
|
||||
"""Once we have a rough pitch/origin from however many labels OCR'd
|
||||
(even just 2-3), the whole grid is regular, so predict and directly
|
||||
test EVERY crossing across the visible frame, not just the ones next
|
||||
to a label that happened to be readable. Returns (ideal_pts,
|
||||
img_pts, debug) for every crossing that matched; a bad/outlier label
|
||||
just contributes points that RANSAC discards downstream rather than
|
||||
limiting how much of the grid we ever attempt.
|
||||
|
||||
dc = column offset in cells from the reference label. dr = offset in
|
||||
cells DOWN the image (increasing y) from the reference, so it moves
|
||||
opposite to row number (row 8 sits above row 7 on screen): the ideal
|
||||
row coordinate is -(ref_number - dr) = dr - ref_number, matching the
|
||||
(col, -row_number) convention used everywhere else in this module."""
|
||||
H, W = gray.shape
|
||||
if not labels:
|
||||
return [], [], []
|
||||
ref = max(labels, key=lambda lb: lb['conf'])
|
||||
ref_col = LARGE_X.index(ref['letter'])
|
||||
ref_corner_x = ref['x'] - ANCHOR_FRAC_X*pitch_x
|
||||
ref_corner_y = ref['y'] - ANCHOR_FRAC_Y*pitch_y
|
||||
|
||||
dc_lo = int(np.floor((0 - ref_corner_x) / pitch_x)) - margin_cells
|
||||
dc_hi = int(np.ceil((W - ref_corner_x) / pitch_x)) + margin_cells
|
||||
dr_lo = int(np.floor((0 - ref_corner_y) / pitch_y)) - margin_cells
|
||||
dr_hi = int(np.ceil((H - ref_corner_y) / pitch_y)) + margin_cells
|
||||
|
||||
ideal_pts, img_pts, debug = [], [], []
|
||||
for dc in range(dc_lo, dc_hi + 1):
|
||||
for dr in range(dr_lo, dr_hi + 1):
|
||||
gx = ref_corner_x + dc * pitch_x
|
||||
gy = ref_corner_y + dr * pitch_y
|
||||
if not (0 <= gx < W and 0 <= gy < H):
|
||||
continue
|
||||
corner, guess, window = find_crossing(gray, hsv_sat, gx, gy, pitch_x, pitch_y)
|
||||
debug.append((guess, corner, window))
|
||||
if corner is not None:
|
||||
ideal_pts.append((ref_col + dc, dr - ref['number']))
|
||||
img_pts.append(corner)
|
||||
return ideal_pts, img_pts, debug
|
||||
|
||||
|
||||
def has_diversity(ideal_pts):
|
||||
cols = set(p[0] for p in ideal_pts)
|
||||
rows = set(p[1] for p in ideal_pts)
|
||||
return len(cols) >= 2 and len(rows) >= 2
|
||||
|
||||
|
||||
def fit_grid_homography(ideal_pts, img_pts):
|
||||
if len(ideal_pts) < 4 or not has_diversity(ideal_pts):
|
||||
return None
|
||||
ideal = np.array(ideal_pts, dtype=np.float32)
|
||||
img = np.array(img_pts, dtype=np.float32)
|
||||
Hmat, mask = cv2.findHomography(ideal, img, cv2.RANSAC, 15.0)
|
||||
return Hmat, mask
|
||||
@ -5,3 +5,4 @@
|
||||
Pillow
|
||||
numpy
|
||||
pytesseract
|
||||
opencv-python-headless
|
||||
|
||||
908
src/fenigma/map_vision.py
Normal file
908
src/fenigma/map_vision.py
Normal file
@ -0,0 +1,908 @@
|
||||
"""Recover the game grid from a screenshot of the map table, and read the
|
||||
unit markers off it.
|
||||
|
||||
This is a second, separate pipeline from `ocr.py`'s typewriter-text OCR.
|
||||
See `docs/map_vision_plan.md` for the design record, the measurements
|
||||
behind it, and the approaches that were tried and rejected.
|
||||
|
||||
The scene is a *flat* table viewed by a perspective camera, so a single
|
||||
homography describes grid-to-screen exactly. The grid, its per-cell labels
|
||||
and the markers are drawn on the table surface, so they are visible whether
|
||||
or not the aerial photo covers that part of the map.
|
||||
|
||||
Shape of the solution:
|
||||
|
||||
1. Line evidence -> two projective line families -> candidate lattices.
|
||||
Abundant and cheap, but it can only ever give a lattice up to unknown
|
||||
scale (1 km cells and the 100 m subgrid look identical), axis
|
||||
assignment, direction and phase.
|
||||
2. Those discrete unknowns, plus the absolute anchor, are resolved by
|
||||
READING CELL LABELS -- and the labels are not detected. Once a
|
||||
candidate lattice exists we know exactly where a label must be (9% of
|
||||
a cell in from the left, 6% down from the top), so we crop there and
|
||||
correlate the glyphs rendered in the game's own font. Detection was
|
||||
tried five different ways and always returned aerial-photo texture
|
||||
instead of glyphs; correlating a known template at a known place does
|
||||
not have that failure mode.
|
||||
3. The correlation score also *ranks the lattice candidates*: a wrong
|
||||
lattice puts the crop where no label is, so it scores low. One number
|
||||
therefore selects scale, axis assignment, direction, phase and anchor
|
||||
together.
|
||||
|
||||
Measured on the 10 fixtures in tests/fixtures/map_shots: solves 7 of them,
|
||||
with 100% of each solved shot's annotated points landing in the correct cell
|
||||
(85 of 112 overall) and a residual spread of 0.005-0.033 cells. The other
|
||||
three are rejected rather than guessed at, and no fixture has ever produced
|
||||
a plausible-but-wrong grid. Rejection is a supported outcome -- a silently
|
||||
misplaced target is far worse than a refusal.
|
||||
|
||||
Over the 122 typewriter screenshots this was checked against, solve()
|
||||
accepted none, which is what makes it safe to route clipboard images through
|
||||
it (see looks_like_map for the cheap pre-filter).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import cv2
|
||||
except ImportError as exc: # pragma: no cover
|
||||
raise ImportError(
|
||||
"map_vision needs opencv (pip install opencv-python-headless)") from exc
|
||||
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
LARGE_X = "ABCDEFGHIJKLMNOPQRST"
|
||||
COLS, ROWS = 20, 10
|
||||
FONT_PATH = Path(__file__).resolve().parents[2] / "assets" / "fonts" / "CourierPrime-Regular.ttf"
|
||||
|
||||
WORK_W = 1500 # working resolution; screenshots vary 700..6880 px wide
|
||||
CELL_PX = 240 # canonical size a rectified cell is warped to
|
||||
CELL_MARGIN = 0.18 # rectify beyond the cell bounds, see rectify_cell()
|
||||
PAD_L, PAD_T = 0.09, 0.06 # label padding inside its cell (game constant)
|
||||
LABEL_ACCEPT = 0.62 # per-read confidence; measured: correct reads 0.73-0.87,
|
||||
# wrong reads 0.40-0.56, so this sits inside the gap
|
||||
MIN_LABEL_VOTES = 2 # one label has no error detection: a misread shifts the
|
||||
# whole board with nothing to contradict it
|
||||
FLIPS = ((False, False), (True, False), (False, True), (True, True))
|
||||
SWAP = np.array([[0, 1, 0], [1, 0, 0], [0, 0, 1]], np.float64)
|
||||
LABELS = [f"{c}{n}" for c in LARGE_X for n in range(1, 11)]
|
||||
|
||||
|
||||
# --------------------------------------------------------------- evidence
|
||||
|
||||
def load(path, work_w=None) -> np.ndarray:
|
||||
img = cv2.imread(str(path), cv2.IMREAD_COLOR)
|
||||
if img is None:
|
||||
raise ValueError(f"cannot read image: {path}")
|
||||
return downscale(img, work_w)
|
||||
|
||||
|
||||
def downscale(img, work_w=None) -> np.ndarray:
|
||||
h, w = img.shape[:2]
|
||||
s = min(1.0, (work_w or WORK_W) / w)
|
||||
if s < 1.0:
|
||||
img = cv2.resize(img, (int(w * s), int(h * s)), interpolation=cv2.INTER_AREA)
|
||||
return img
|
||||
|
||||
|
||||
def ridge(img: np.ndarray) -> np.ndarray:
|
||||
"""Bright, neutral, thin line structures; coloured overlays suppressed.
|
||||
|
||||
Two things this must get right:
|
||||
|
||||
* the colour mask is relative to the scene's OWN illuminant. One fixture
|
||||
is lit bright red, and an absolute saturation cut masks the entire
|
||||
table, grid included.
|
||||
* several top-hat kernel sizes, not one. A top-hat kernel must be LARGER
|
||||
than the structure it keeps or it hollows it out, and grid lines run
|
||||
from ~2px when the whole table is in frame to ~15px when a single cell
|
||||
fills it.
|
||||
"""
|
||||
lab = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
|
||||
lum = lab[:, :, 0]
|
||||
a = lab[:, :, 1].astype(np.float32) - float(np.median(lab[:, :, 1]))
|
||||
b = lab[:, :, 2].astype(np.float32) - float(np.median(lab[:, :, 2]))
|
||||
chroma = np.sqrt(a * a + b * b)
|
||||
acc = np.zeros(lum.shape, np.float32)
|
||||
for k in (7, 15, 31, 51):
|
||||
el = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (k, k))
|
||||
acc = np.maximum(acc, cv2.morphologyEx(lum, cv2.MORPH_TOPHAT, el).astype(np.float32))
|
||||
acc[chroma > 16.0] = 0
|
||||
return (np.clip(acc, 0, 55) / 55.0 * 255).astype(np.uint8)
|
||||
|
||||
|
||||
def segments(ev: np.ndarray, min_len: float) -> np.ndarray:
|
||||
lines = cv2.createLineSegmentDetector().detect(ev)[0]
|
||||
if lines is None:
|
||||
return np.zeros((0, 4), np.float32)
|
||||
L = lines.reshape(-1, 4)
|
||||
return L[np.hypot(L[:, 2] - L[:, 0], L[:, 3] - L[:, 1]) >= min_len]
|
||||
|
||||
|
||||
def _homog_lines(L):
|
||||
p1 = np.c_[L[:, 0], L[:, 1], np.ones(len(L))]
|
||||
p2 = np.c_[L[:, 2], L[:, 3], np.ones(len(L))]
|
||||
ln = np.cross(p1, p2)
|
||||
return ln / (np.linalg.norm(ln[:, :2], axis=1, keepdims=True) + 1e-9)
|
||||
|
||||
|
||||
def _vp_ransac(L, allowed, iters=4000, tol_deg=1.5, seed=0):
|
||||
idx = np.flatnonzero(allowed)
|
||||
if len(idx) < 3:
|
||||
return np.zeros(len(L), bool)
|
||||
rng = np.random.default_rng(seed)
|
||||
ln = _homog_lines(L)
|
||||
mid = np.c_[(L[:, 0] + L[:, 2]) / 2, (L[:, 1] + L[:, 3]) / 2]
|
||||
ang = np.arctan2(L[:, 3] - L[:, 1], L[:, 2] - L[:, 0])
|
||||
tol = np.deg2rad(tol_deg)
|
||||
best = np.zeros(len(L), bool)
|
||||
for _ in range(iters):
|
||||
i, j = rng.choice(idx, 2, replace=False)
|
||||
v = np.cross(ln[i], ln[j])
|
||||
if abs(v[2]) < 1e-9:
|
||||
continue
|
||||
vx, vy = v[0] / v[2], v[1] / v[2]
|
||||
want = np.arctan2(vy - mid[:, 1], vx - mid[:, 0])
|
||||
diff = np.abs((want - ang + np.pi / 2) % np.pi - np.pi / 2)
|
||||
inl = (diff < tol) & allowed
|
||||
if inl.sum() > best.sum():
|
||||
best = inl
|
||||
return best
|
||||
|
||||
|
||||
def two_families(L, min_sep_deg=20.0):
|
||||
"""The two pencils of grid lines.
|
||||
|
||||
The second family is required to be angularly DISTINCT from the first.
|
||||
Simply re-running RANSAC on the leftovers lets both passes lock onto the
|
||||
same family and report two "families" a degree apart. Orientation is
|
||||
also undirected, so angles are compared as doubled angles -- otherwise
|
||||
+89 and -89 degrees look like opposites instead of neighbours.
|
||||
"""
|
||||
th2 = 2.0 * np.arctan2(L[:, 3] - L[:, 1], L[:, 2] - L[:, 0])
|
||||
inA = _vp_ransac(L, np.ones(len(L), bool), seed=1)
|
||||
if inA.sum() == 0:
|
||||
return inA, np.zeros(len(L), bool)
|
||||
mA = np.arctan2(np.median(np.sin(th2[inA])), np.median(np.cos(th2[inA])))
|
||||
d = np.abs(np.angle(np.exp(1j * (th2 - mA)))) / 2.0
|
||||
inB = _vp_ransac(L, (~inA) & (d > np.deg2rad(min_sep_deg)), seed=2)
|
||||
return inA, inB
|
||||
|
||||
|
||||
def _fit_vp(L, inl):
|
||||
_, _, Vt = np.linalg.svd(_homog_lines(L[inl]))
|
||||
v = Vt[-1]
|
||||
return v / (v[2] if abs(v[2]) > 1e-12 else 1e-12)
|
||||
|
||||
|
||||
# ---------------------------------------------------------- lattice fitting
|
||||
|
||||
def _dir_of(Hp, sub):
|
||||
ds = []
|
||||
for (x1, y1, x2, y2) in sub:
|
||||
p = Hp @ np.array([[x1, x2], [y1, y2], [1.0, 1.0]])
|
||||
if np.any(np.abs(p[2]) < 1e-9):
|
||||
continue
|
||||
p = p[:2] / p[2]
|
||||
d = p[:, 1] - p[:, 0]
|
||||
n = np.linalg.norm(d)
|
||||
if n > 1e-9:
|
||||
d = d / n
|
||||
ds.append(d if d[0] >= 0 else -d)
|
||||
if not ds:
|
||||
return None
|
||||
d = np.median(np.array(ds), axis=0)
|
||||
return d / (np.linalg.norm(d) + 1e-12)
|
||||
|
||||
|
||||
def rectify_candidates(vpA, vpB, L, inA, inB, shape):
|
||||
"""Maps that turn the perspective lattice into an axis-aligned regular
|
||||
one, so spacing becomes a 2-parameter fit instead of a projective one.
|
||||
|
||||
Both a projective and an affine variant are offered. Insisting on the
|
||||
projective one is wrong: with a near-overhead camera the vanishing
|
||||
points are far away and ill-conditioned, so the horizon estimate is
|
||||
noise and the "horizon crosses the frame" guard fires on the *easiest*
|
||||
inputs. Mild perspective must be the easy case.
|
||||
"""
|
||||
out = []
|
||||
cand_hp = []
|
||||
horizon = np.cross(vpA, vpB)
|
||||
if abs(horizon[2]) > 1e-9:
|
||||
hz = horizon / horizon[2]
|
||||
Hp = np.array([[1, 0, 0], [0, 1, 0], [hz[0], hz[1], 1.0]], np.float64)
|
||||
h, w = shape[:2]
|
||||
corners = np.array([[0, w, w, 0], [0, 0, h, h], [1, 1, 1, 1]], np.float64)
|
||||
ws = (Hp @ corners)[2]
|
||||
if np.all(np.abs(ws) > 1e-6) and not (ws.min() < 0 < ws.max()):
|
||||
cand_hp.append(Hp)
|
||||
cand_hp.append(np.eye(3))
|
||||
for Hp in cand_hp:
|
||||
dA, dB = _dir_of(Hp, L[inA]), _dir_of(Hp, L[inB])
|
||||
if dA is None or dB is None or abs(float(np.cross(dA, dB))) < 0.05:
|
||||
continue
|
||||
Ha = np.eye(3)
|
||||
Ha[:2, :2] = np.linalg.inv(np.column_stack([dA, dB]))
|
||||
out.append(Ha @ Hp)
|
||||
return out
|
||||
|
||||
|
||||
def fit_lattice_1d(pos, min_occupancy=0.5, top=6):
|
||||
"""Fit pos ~ phase + spacing * k for unknown integers k.
|
||||
|
||||
Two traps, both hit for real:
|
||||
|
||||
* spacing -> 0 fits ANY set of positions: every value lands within
|
||||
tolerance of some multiple of a tiny spacing, so maximising inlier
|
||||
count collapses to a degenerate near-zero spacing. The guard is
|
||||
OCCUPANCY -- the fraction of integer slots between the extreme
|
||||
indices that are actually populated. A true grid fills nearly all of
|
||||
them.
|
||||
* fitting over every position lets ONE misdetected line (a film-strip
|
||||
edge, a dotted front line) drag spacing and phase, producing a grid
|
||||
visibly off by a line. So refit on inliers only.
|
||||
|
||||
Returns up to `top` candidates as (n_inliers, spacing, phase, occupancy),
|
||||
for the caller to choose between jointly across both axes.
|
||||
"""
|
||||
pos = np.sort(np.asarray(pos, np.float64))
|
||||
if len(pos) < 3:
|
||||
return []
|
||||
diffs = np.diff(pos)
|
||||
diffs = diffs[diffs > 1e-9]
|
||||
if len(diffs) == 0:
|
||||
return []
|
||||
cands = {float(np.median(diffs))}
|
||||
for d in diffs:
|
||||
for div in (1, 2, 3):
|
||||
cands.add(d / div)
|
||||
out = []
|
||||
for s0 in sorted(cands):
|
||||
s, phase = s0, pos[0]
|
||||
if s <= 1e-9:
|
||||
continue
|
||||
for _ in range(3):
|
||||
k = np.round((pos - phase) / s)
|
||||
sol, *_ = np.linalg.lstsq(np.column_stack([np.ones(len(pos)), k]),
|
||||
pos, rcond=None)
|
||||
phase, s = float(sol[0]), float(sol[1])
|
||||
if s <= 1e-9:
|
||||
break
|
||||
if s <= 1e-9:
|
||||
continue
|
||||
inl = np.abs(pos - (phase + np.round((pos - phase) / s) * s)) < 0.2 * s
|
||||
if inl.sum() < 3:
|
||||
continue
|
||||
for _ in range(3): # refit on inliers only
|
||||
kk = np.round((pos[inl] - phase) / s)
|
||||
if len(np.unique(kk)) < 2:
|
||||
break
|
||||
sol, *_ = np.linalg.lstsq(np.column_stack([np.ones(int(inl.sum())), kk]),
|
||||
pos[inl], rcond=None)
|
||||
p2, s2 = float(sol[0]), float(sol[1])
|
||||
if s2 <= 1e-9:
|
||||
break
|
||||
phase, s = p2, s2
|
||||
nxt = np.abs(pos - (phase + np.round((pos - phase) / s) * s)) < 0.2 * s
|
||||
if nxt.sum() < 3 or np.array_equal(nxt, inl):
|
||||
break
|
||||
inl = nxt
|
||||
k = np.round((pos - phase) / s)
|
||||
inl = np.abs(pos - (phase + k * s)) < 0.2 * s
|
||||
if inl.sum() < 3:
|
||||
continue
|
||||
ks = np.unique(k[inl])
|
||||
slots = ks.max() - ks.min() + 1
|
||||
occ = len(ks) / slots if slots > 0 else 0.0
|
||||
if occ < min_occupancy:
|
||||
continue
|
||||
out.append((int(inl.sum()), float(s), float(phase), float(occ)))
|
||||
out.sort(key=lambda r: (-r[0], -r[1]))
|
||||
keep = []
|
||||
for r in out:
|
||||
if all(abs(r[1] - k[1]) > 0.03 * max(r[1], k[1]) for k in keep):
|
||||
keep.append(r)
|
||||
if len(keep) >= top:
|
||||
break
|
||||
return keep
|
||||
|
||||
|
||||
def _cluster(vals, tol):
|
||||
vals = np.sort(np.asarray(vals, np.float64))
|
||||
out, grp = [], [vals[0]]
|
||||
for v in vals[1:]:
|
||||
if v - grp[-1] <= tol:
|
||||
grp.append(v)
|
||||
else:
|
||||
out.append(float(np.mean(grp)))
|
||||
grp = [v]
|
||||
out.append(float(np.mean(grp)))
|
||||
return np.array(out)
|
||||
|
||||
|
||||
def cell_steps(H, shape):
|
||||
"""Pixel length of a one-index step along each lattice axis, measured at
|
||||
the CENTRE of the frame -- not at index (0,0), which is usually far
|
||||
off-screen and, under perspective, a wildly different scale."""
|
||||
h, w = shape[:2]
|
||||
c = np.linalg.inv(H) @ np.array([w / 2.0, h / 2.0, 1.0])
|
||||
if abs(c[2]) < 1e-12:
|
||||
return None
|
||||
ci, cj = c[0] / c[2], c[1] / c[2]
|
||||
q = H @ np.array([[ci, ci + 1, ci], [cj, cj, cj + 1], [1, 1, 1.0]])
|
||||
if np.any(np.abs(q[2]) < 1e-12):
|
||||
return None
|
||||
q = q[:2] / q[2]
|
||||
return (float(np.linalg.norm(q[:, 1] - q[:, 0])),
|
||||
float(np.linalg.norm(q[:, 2] - q[:, 0])))
|
||||
|
||||
|
||||
def lattice_candidates(img, L, inA, inB, top=6):
|
||||
"""Candidate homographies mapping lattice index -> image pixels.
|
||||
|
||||
Includes BOTH axis assignments: the two line families are unordered, and
|
||||
getting this wrong yields the true cell transposed.
|
||||
|
||||
The two axes are also chosen jointly, not independently: cells are square
|
||||
on the table and the camera is near overhead, so a reconstructed cell must
|
||||
come out roughly rectangular on screen. Choosing per-axis lets one axis
|
||||
lock to the 1 km grid while the other locks to the 100 m subgrid, giving a
|
||||
geometrically impossible 10:1 cell.
|
||||
"""
|
||||
vpA, vpB = _fit_vp(L, inA), _fit_vp(L, inB)
|
||||
out = []
|
||||
for Hr in rectify_candidates(vpA, vpB, L, inA, inB, img.shape):
|
||||
posA, posB = [], []
|
||||
for sub, axis, acc in ((L[inA], 1, posA), (L[inB], 0, posB)):
|
||||
for (x1, y1, x2, y2) in sub:
|
||||
p = Hr @ np.array([[x1, x2], [y1, y2], [1.0, 1.0]])
|
||||
if np.any(np.abs(p[2]) < 1e-9):
|
||||
continue
|
||||
p = p[:2] / p[2]
|
||||
acc.append(float(np.mean(p[axis])))
|
||||
if len(posA) < 3 or len(posB) < 3:
|
||||
continue
|
||||
cA = fit_lattice_1d(_cluster(posA, 0.01 * max(np.ptp(posA), 1e-9)))
|
||||
cB = fit_lattice_1d(_cluster(posB, 0.01 * max(np.ptp(posB), 1e-9)))
|
||||
for nA, sA, pA, _oa in cA:
|
||||
for nB, sB, pB, _ob in cB:
|
||||
K = np.array([[sB, 0, pB], [0, sA, pA], [0, 0, 1.0]])
|
||||
H = np.linalg.inv(Hr) @ K
|
||||
st = cell_steps(H, img.shape)
|
||||
if st is None or min(st) < 2.0 or not (0.5 <= st[0] / st[1] <= 2.0):
|
||||
continue
|
||||
for swap in (False, True):
|
||||
HH = H @ SWAP if swap else H
|
||||
out.append((HH, nA + nB, cell_steps(HH, img.shape) or st))
|
||||
out.sort(key=lambda r: -r[1])
|
||||
return out[:top]
|
||||
|
||||
|
||||
# ------------------------------------------------------------- label reading
|
||||
|
||||
_TEMPLATES: dict = {}
|
||||
|
||||
|
||||
def glyph_template(label: str, height: int):
|
||||
"""The label as the game draws it: cream glyphs with a heavy dark
|
||||
outline. The outline is what makes correlation discriminative against
|
||||
aerial-photo texture, which has plenty of bright blobs but nothing
|
||||
ringed in near-black."""
|
||||
key = (label, height)
|
||||
if key in _TEMPLATES:
|
||||
return _TEMPLATES[key]
|
||||
font = ImageFont.truetype(str(FONT_PATH), int(height))
|
||||
pad = int(height * 0.6)
|
||||
im = Image.new("L", (int(height * 5) + pad, int(height * 2) + pad), 0)
|
||||
ImageDraw.Draw(im).text((pad // 2, pad // 4), label, font=font, fill=255,
|
||||
stroke_width=max(1, int(height * 0.10)), stroke_fill=0)
|
||||
a = np.array(im)
|
||||
ys, xs = np.nonzero(a > 40)
|
||||
if len(xs) == 0:
|
||||
_TEMPLATES[key] = None
|
||||
return None
|
||||
a = a[max(0, ys.min() - 2):ys.max() + 3, max(0, xs.min() - 2):xs.max() + 3]
|
||||
_TEMPLATES[key] = a
|
||||
return a
|
||||
|
||||
|
||||
def rectify_cell(img, H, i, j):
|
||||
"""Warp one lattice cell to a canonical square, with a margin.
|
||||
|
||||
The margin matters: the lattice phase can be off by ~10% of a cell, and
|
||||
an exact-bounds warp would clip a label near the cell edge -- a clipped
|
||||
glyph correlates with nothing. Reading a slightly larger region tolerates
|
||||
that instead of requiring the phase to be perfect.
|
||||
"""
|
||||
m = CELL_MARGIN
|
||||
src = np.array([[i - m, j - m], [i + 1 + m, j - m],
|
||||
[i + 1 + m, j + 1 + m], [i - m, j + 1 + m]], np.float64)
|
||||
q = H @ np.vstack([src.T, np.ones(4)])
|
||||
if np.any(np.abs(q[2]) < 1e-9):
|
||||
return None
|
||||
n = int(CELL_PX * (1 + 2 * m))
|
||||
canon = np.array([[0, 0], [n, 0], [n, n], [0, n]], np.float32)
|
||||
M = cv2.getPerspectiveTransform((q[:2] / q[2]).T.astype(np.float32), canon)
|
||||
return cv2.warpPerspective(img, M, (n, n), flags=cv2.INTER_LINEAR)
|
||||
|
||||
|
||||
def read_cell_label(cell_gray, glyph_fracs=(0.10, 0.13, 0.17)):
|
||||
"""Which label best explains the pixels where a label must be?
|
||||
|
||||
No detection: the grid fixes the label's position and size, so this
|
||||
correlates every candidate label there and takes the best. Sliding the
|
||||
template over a slightly larger crop absorbs the residual phase error.
|
||||
All four cell orientations are tried, because the lattice axes have
|
||||
arbitrary direction and the label may land in any corner, mirrored.
|
||||
|
||||
-> (label, score) with score in [-1, 1]; measured, correct reads score
|
||||
0.73-0.87 and wrong ones 0.40-0.56.
|
||||
"""
|
||||
n = cell_gray.shape[0]
|
||||
best = (None, -1.0)
|
||||
x0 = int(max(0, (CELL_MARGIN - 0.03) * CELL_PX))
|
||||
x1 = int(min(n, (CELL_MARGIN + 0.50) * CELL_PX))
|
||||
y0 = int(max(0, (CELL_MARGIN - 0.05) * CELL_PX))
|
||||
y1 = int(min(n, (CELL_MARGIN + 0.30) * CELL_PX))
|
||||
for fx, fy in FLIPS:
|
||||
v = cell_gray
|
||||
if fx:
|
||||
v = v[:, ::-1]
|
||||
if fy:
|
||||
v = v[::-1, :]
|
||||
patch = np.ascontiguousarray(v[y0:y1, x0:x1])
|
||||
if patch.shape[0] < 12 or patch.shape[1] < 12:
|
||||
continue
|
||||
for gf in glyph_fracs:
|
||||
h = max(8, int(gf * CELL_PX))
|
||||
for lab in LABELS:
|
||||
t = glyph_template(lab, h)
|
||||
if t is None or t.shape[0] >= patch.shape[0] or t.shape[1] >= patch.shape[1]:
|
||||
continue
|
||||
sc = float(cv2.matchTemplate(patch, t, cv2.TM_CCOEFF_NORMED).max())
|
||||
if sc > best[1]:
|
||||
best = (lab, sc)
|
||||
return best
|
||||
|
||||
|
||||
def visible_cells(H, shape, limit=6):
|
||||
"""Lattice cells whose centre is on screen, nearest the frame centre
|
||||
first (least perspective distortion, so the easiest to read)."""
|
||||
h, w = shape[:2]
|
||||
q = np.linalg.inv(H) @ np.array([[0, w, w, 0], [0, 0, h, h], [1, 1, 1, 1]], np.float64)
|
||||
ok = np.abs(q[2]) > 1e-9
|
||||
if ok.sum() < 3:
|
||||
return []
|
||||
ij = q[:2, ok] / q[2, ok]
|
||||
cells = []
|
||||
for i in range(int(np.floor(ij[0].min())), int(np.ceil(ij[0].max()))):
|
||||
for j in range(int(np.floor(ij[1].min())), int(np.ceil(ij[1].max()))):
|
||||
c = H @ np.array([i + 0.5, j + 0.5, 1.0])
|
||||
if abs(c[2]) < 1e-9:
|
||||
continue
|
||||
x, y = c[0] / c[2], c[1] / c[2]
|
||||
if 0 <= x < w and 0 <= y < h:
|
||||
cells.append((float(np.hypot(x - w / 2, y - h / 2)), i, j))
|
||||
cells.sort()
|
||||
return [(i, j) for _d, i, j in cells[:limit]]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- solving
|
||||
|
||||
class GridSolution:
|
||||
"""A solved grid: lattice homography plus the discrete mapping from
|
||||
lattice index to game cell."""
|
||||
|
||||
def __init__(self, H, si, sj, du, dv, votes, reads, steps):
|
||||
self.H, self.si, self.sj = H, si, sj
|
||||
self.du, self.dv = du, dv
|
||||
self.votes, self.reads, self.steps = votes, reads, steps
|
||||
|
||||
def lattice_to_grid(self):
|
||||
"""Affine 3x3 taking lattice coords (u, v) to continuous game grid
|
||||
coords (col, row), col in [0, 20] and row in [1, 11].
|
||||
|
||||
The +1 for a negative sign is not cosmetic. With si = -1, increasing u
|
||||
means decreasing col, so the cell that lattice index i opens at u = i
|
||||
is entered from its RIGHT edge, and si*u + du alone runs from col+1
|
||||
down to col across it -- floor() would name the neighbour for the whole
|
||||
cell. Offsetting by 1 makes the fraction always grow in the direction
|
||||
col/row grow, which is also what the app's Coord means by its sub-cell
|
||||
x/y (see models.Coord.as_fraction), so both signs agree with it.
|
||||
"""
|
||||
return np.array([[self.si, 0.0, self.du + (0 if self.si > 0 else 1)],
|
||||
[0.0, self.sj, self.dv + (0 if self.sj > 0 else 1)],
|
||||
[0.0, 0.0, 1.0]])
|
||||
|
||||
def grid_of(self, x, y):
|
||||
"""-> continuous (col, row) for a pixel in working-resolution coords."""
|
||||
q = self.lattice_to_grid() @ np.linalg.inv(self.H) @ np.array([x, y, 1.0])
|
||||
if abs(q[2]) < 1e-12:
|
||||
return None
|
||||
return float(q[0] / q[2]), float(q[1] / q[2])
|
||||
|
||||
def cell_of(self, x, y):
|
||||
"""-> ("J8", sub_x, sub_y) for a pixel in working-resolution coords."""
|
||||
g = self.grid_of(x, y)
|
||||
if g is None:
|
||||
return None
|
||||
colf, rowf = g
|
||||
col, row = int(np.floor(colf)), int(np.floor(rowf))
|
||||
if not (0 <= col < COLS and 1 <= row <= ROWS):
|
||||
return None
|
||||
return (f"{LARGE_X[col]}{row}",
|
||||
int(np.clip((colf - col) * 10, 0, 9)),
|
||||
int(np.clip((rowf - row) * 10, 0, 9)))
|
||||
|
||||
|
||||
def format_coord(cell) -> str:
|
||||
"""("K8", 0, 3) -> "K8 0:3", matching how the game writes coordinates."""
|
||||
if cell is None:
|
||||
return "?"
|
||||
return f"{cell[0]} {cell[1]}:{cell[2]}"
|
||||
|
||||
|
||||
def solve(img):
|
||||
"""-> (GridSolution, None) or (None, reason)."""
|
||||
L = segments(ridge(img), min_len=0.04 * img.shape[1])
|
||||
inA, inB = two_families(L)
|
||||
if inA.sum() < 3 or inB.sum() < 3:
|
||||
return None, "too few grid line families"
|
||||
cands = lattice_candidates(img, L, inA, inB)
|
||||
if not cands:
|
||||
return None, "no plausible lattice"
|
||||
best = None
|
||||
for H, _ninl, steps in cands:
|
||||
cells = visible_cells(H, img.shape)
|
||||
reads = []
|
||||
for (i, j) in cells:
|
||||
cell = rectify_cell(img, H, i, j)
|
||||
if cell is None:
|
||||
continue
|
||||
lab, sc = read_cell_label(cv2.cvtColor(cell, cv2.COLOR_BGR2GRAY))
|
||||
if lab and sc >= LABEL_ACCEPT:
|
||||
reads.append((i, j, lab, sc))
|
||||
if len(reads) < MIN_LABEL_VOTES:
|
||||
continue
|
||||
for si in (1, -1):
|
||||
for sj in (1, -1):
|
||||
votes = {}
|
||||
for i, j, lab, sc in reads:
|
||||
key = (LARGE_X.index(lab[0]) - si * i, int(lab[1:]) - sj * j)
|
||||
v = votes.setdefault(key, [0, 0.0])
|
||||
v[0] += 1
|
||||
v[1] += sc
|
||||
for (du, dv), (cnt, tot) in votes.items():
|
||||
if cnt < MIN_LABEL_VOTES:
|
||||
continue
|
||||
# extent prior: every visible cell must be a real map cell
|
||||
inrange = all(0 <= si * i + du < COLS and 1 <= sj * j + dv <= ROWS
|
||||
for (i, j) in cells)
|
||||
score = tot + cnt + (1.5 if inrange else -1.5)
|
||||
if best is None or score > best[0]:
|
||||
best = (score, H, si, sj, du, dv, cnt, len(reads), steps)
|
||||
if best is None:
|
||||
return None, (f"no confident label read "
|
||||
f"(need {MIN_LABEL_VOTES} at >={LABEL_ACCEPT})")
|
||||
_s, H, si, sj, du, dv, cnt, nreads, steps = best
|
||||
return GridSolution(H, si, sj, du, dv, cnt, nreads, steps), None
|
||||
|
||||
|
||||
def solution_from_correspondences(pairs):
|
||||
"""Build a solution from explicit grid<->pixel correspondences.
|
||||
|
||||
`pairs` is [((col, row), (x, y)), ...] with at least 4 entries, where
|
||||
(col, row) are CONTINUOUS grid coordinates: col 0..20 increasing with the
|
||||
letters, row 1..11 as the game numbers them. This is the manual override
|
||||
path -- four dragged cell corners plus that cell's label fully determine
|
||||
the homography, so it works even when line detection or label reading
|
||||
fail completely.
|
||||
|
||||
Four correspondences is the minimum: a homography has 8 degrees of
|
||||
freedom and each point contributes 2 equations. Three points would only
|
||||
fix an affine map, and under real perspective a square's image is a
|
||||
general quadrilateral, so the fourth corner is genuinely not implied by
|
||||
the other three.
|
||||
"""
|
||||
if len(pairs) < 4:
|
||||
raise ValueError("a homography needs at least 4 correspondences")
|
||||
src = np.array([[p[0][0], p[0][1]] for p in pairs], np.float64)
|
||||
dst = np.array([[p[1][0], p[1][1]] for p in pairs], np.float64)
|
||||
H, _ = cv2.findHomography(src.reshape(-1, 1, 2), dst.reshape(-1, 1, 2), 0)
|
||||
if H is None:
|
||||
raise ValueError("degenerate correspondences")
|
||||
# identity discrete mapping: the grid coords were given directly
|
||||
return GridSolution(H, 1, 1, 0, 0, votes=len(pairs), reads=len(pairs),
|
||||
steps=cell_steps(H, (1, 1)) or (1.0, 1.0))
|
||||
|
||||
|
||||
def centre_cell_quad(sol, shape):
|
||||
"""The centre-most fully-visible cell, as handles for manual correction.
|
||||
|
||||
-> (label, [(x, y) x4], [(col, row) x4]) with the two lists in matching
|
||||
order, so a UI can seed four draggable handles from the automatic
|
||||
solution and refit through solution_from_correspondences() as they move.
|
||||
"""
|
||||
h, w = shape[:2]
|
||||
cells = visible_cells(sol.H, shape, limit=1)
|
||||
if not cells:
|
||||
return None
|
||||
i, j = cells[0]
|
||||
corners_ij = [(i, j), (i + 1, j), (i + 1, j + 1), (i, j + 1)]
|
||||
q = sol.H @ np.array([[c[0] for c in corners_ij],
|
||||
[c[1] for c in corners_ij], [1, 1, 1, 1.0]])
|
||||
if np.any(np.abs(q[2]) < 1e-9):
|
||||
return None
|
||||
px = [(float(x), float(y)) for x, y in (q[:2] / q[2]).T]
|
||||
g = sol.lattice_to_grid() @ np.array([[c[0] for c in corners_ij],
|
||||
[c[1] for c in corners_ij], [1, 1, 1, 1.0]])
|
||||
grid = [(float(a), float(b)) for a, b in (g[:2] / g[2]).T]
|
||||
centre = sol.cell_of(*((np.array(px[0]) + np.array(px[2])) / 2))
|
||||
return (centre[0] if centre else None), px, grid
|
||||
|
||||
|
||||
MAP_KM_W, MAP_KM_H = 20.0, 10.0
|
||||
|
||||
|
||||
def warp_to_map(img, sol, px_per_km=100):
|
||||
"""Rectify a screenshot into map space, ready to composite under the app's
|
||||
own grid.
|
||||
|
||||
Returns (BGRA array, px_per_km). Only the region the screenshot actually
|
||||
covers is opaque; everything else is transparent, so a partial view of the
|
||||
table does not blank out the rest of the map.
|
||||
|
||||
Warping once into map space -- rather than transforming while drawing --
|
||||
keeps the renderer simple: Cairo has no projective transform, but once the
|
||||
image is in map space a plain scale and translate places it.
|
||||
"""
|
||||
out_w, out_h = int(MAP_KM_W * px_per_km), int(MAP_KM_H * px_per_km)
|
||||
# Map space is pixels over the whole board: x = col * px_per_km rightward,
|
||||
# y measured DOWN while row counts UP, so row 1 (the game's bottom row)
|
||||
# lands at the bottom edge. Row is 1-based here and 0-based in the app's
|
||||
# Coord.as_fraction, hence the extra +px_per_km.
|
||||
grid_to_map = np.array([[px_per_km, 0.0, 0.0],
|
||||
[0.0, -px_per_km, out_h + px_per_km],
|
||||
[0.0, 0.0, 1.0]])
|
||||
# sol.H alone only reaches LATTICE coords; the discrete mapping (si, sj,
|
||||
# du, dv) is what pins those to named cells, and leaving it out put the
|
||||
# screenshot in the wrong place for every automatically solved grid.
|
||||
M = grid_to_map @ sol.lattice_to_grid() @ np.linalg.inv(sol.H)
|
||||
bgra = cv2.cvtColor(img, cv2.COLOR_BGR2BGRA)
|
||||
bgra[:, :, 3] = 255
|
||||
return cv2.warpPerspective(bgra, M, (out_w, out_h), flags=cv2.INTER_LINEAR,
|
||||
borderMode=cv2.BORDER_CONSTANT,
|
||||
borderValue=(0, 0, 0, 0)), px_per_km
|
||||
|
||||
|
||||
RETRY_WORK_W = 2400
|
||||
|
||||
|
||||
GATE_MIN_FAMILY = 8
|
||||
|
||||
|
||||
def looks_like_map(img) -> bool:
|
||||
"""Cheap routing test: is this a map-table screenshot or writer text?
|
||||
|
||||
Purely a latency optimisation, NOT a correctness gate -- `solve()` is the
|
||||
real decision, and measured over 122 writer screenshots it accepts zero of
|
||||
them. This just avoids paying solve()'s ~10-20s on text pastes.
|
||||
|
||||
The discriminating feature is the number of lines in the SMALLER line
|
||||
family: measured, map screenshots have >=9 (median 15) while writer
|
||||
screenshots that reach this point have a median of 5 (lower quartile 3).
|
||||
Angular separation and implied cell size do not separate the two at all.
|
||||
|
||||
Deliberately looser than solve(): a false positive here only costs time,
|
||||
while a false negative would silently route a map to the text pipeline.
|
||||
"""
|
||||
L = segments(ridge(img), min_len=0.04 * img.shape[1])
|
||||
inA, inB = two_families(L)
|
||||
if min(int(inA.sum()), int(inB.sum())) < GATE_MIN_FAMILY:
|
||||
return False
|
||||
return bool(lattice_candidates(img, L, inA, inB))
|
||||
|
||||
|
||||
def solve_path(path):
|
||||
"""Solve a screenshot on disk, retrying at higher working resolution.
|
||||
|
||||
A very wide screenshot downscaled to WORK_W can leave cells so small that
|
||||
the label is only a handful of pixels across, which no filtering recovers
|
||||
-- measured, one fixture's best label score goes 0.54 -> 0.65 (past the
|
||||
accept threshold) purely from working at 2400px instead of 1500px. The
|
||||
retry only pays that cost when the first pass actually fails.
|
||||
|
||||
-> (GridSolution, image_used, None) or (None, image_used, reason)
|
||||
"""
|
||||
img = load(path)
|
||||
sol, err = solve(img)
|
||||
if sol is not None:
|
||||
return sol, img, None
|
||||
raw = cv2.imread(str(path), cv2.IMREAD_COLOR)
|
||||
if raw is not None and raw.shape[1] > 1.4 * WORK_W:
|
||||
big = downscale(raw, RETRY_WORK_W)
|
||||
sol2, err2 = solve(big)
|
||||
if sol2 is not None:
|
||||
return sol2, big, None
|
||||
err = err2 or err
|
||||
return None, img, err
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- markers
|
||||
|
||||
# Grey-world illuminant normalisation was tried here and removed. It did cut
|
||||
# one fixture's false positives (33 -> 5) by neutralising red stage lighting,
|
||||
# but it made the worst case worse: cancelling the red cast also restores the
|
||||
# cyan front-line ribbon to full saturation, so the ribbon then fires the
|
||||
# friendly-marker mask (43 -> 50 spurious markers). The false positives are
|
||||
# overlay geometry, not lighting, so they need a shape test, not a colour fix.
|
||||
|
||||
|
||||
# ---- unit type classification against the game's own marker icons --------
|
||||
|
||||
_ICON_BANK: dict = {}
|
||||
ICON_SIZE = 64
|
||||
DIAMOND_IOU = 0.64 # blob-vs-ideal-diamond overlap needed to be a marker.
|
||||
# Swept against verified counts: 0.64 keeps every shot
|
||||
# confirmed correct by hand (5/2/2/3 markers) while cutting
|
||||
# ribbon+hatching false positives from 43 to 2 on the worst
|
||||
# fixture. Loosening to 0.50 regains one real marker on one
|
||||
# shot but quadruples the false positives.
|
||||
SYMBOL_KEEP = 0.52 # central fraction of the marker that carries the symbol
|
||||
|
||||
|
||||
def _icon_dir(side):
|
||||
base = Path(__file__).resolve().parents[2] / "assets" / "icons" / "targets"
|
||||
return base / ("friendly" if side == "friendly" else "enemy")
|
||||
|
||||
|
||||
def icon_bank(side):
|
||||
"""Every marker icon as a normalised grayscale patch.
|
||||
|
||||
The shipped icons are the complete marker -- coloured diamond plus the
|
||||
black inner symbol -- which is exactly what is drawn on the table, so a
|
||||
detected marker can be matched against them directly. They differ ONLY in
|
||||
the inner symbol, so the comparison is effectively on that symbol.
|
||||
"""
|
||||
if side in _ICON_BANK:
|
||||
return _ICON_BANK[side]
|
||||
entries = []
|
||||
d = _icon_dir(side)
|
||||
for f in sorted(d.glob("*.png")):
|
||||
raw = cv2.imread(str(f), cv2.IMREAD_UNCHANGED)
|
||||
if raw is None:
|
||||
continue
|
||||
if raw.shape[2] == 4:
|
||||
a = raw[:, :, 3:4].astype(np.float32) / 255.0
|
||||
rgb = raw[:, :, :3].astype(np.float32)
|
||||
raw = (rgb * a + 128.0 * (1 - a)).astype(np.uint8)
|
||||
g = cv2.cvtColor(raw, cv2.COLOR_BGR2GRAY)
|
||||
g = cv2.resize(g, (ICON_SIZE, ICON_SIZE), interpolation=cv2.INTER_AREA)
|
||||
name = f.stem.split("_", 1)[1] if "_" in f.stem else f.stem
|
||||
entries.append((name, _zscore(_inner(g))))
|
||||
_ICON_BANK[side] = entries
|
||||
return entries
|
||||
|
||||
|
||||
def _inner(patch):
|
||||
"""The central part of a marker, where the only discriminative content is.
|
||||
|
||||
Every icon is the SAME diamond and differs only in the small black symbol
|
||||
inside it, so correlating whole markers lets the identical diamond edges
|
||||
dominate the score and swamp the signal -- which is why classification
|
||||
returned near-tied scores (margins under 0.05) and therefore almost always
|
||||
None.
|
||||
"""
|
||||
n = patch.shape[0]
|
||||
k = int(n * (1 - SYMBOL_KEEP) / 2)
|
||||
return patch[k:n - k, k:n - k]
|
||||
|
||||
|
||||
def _zscore(patch):
|
||||
p = patch.astype(np.float32)
|
||||
p -= p.mean()
|
||||
sd = float(p.std())
|
||||
return p / sd if sd > 1e-6 else p
|
||||
|
||||
|
||||
def classify_marker(img, box, side, pad=0.22, min_margin=0.08, min_score=0.55):
|
||||
"""Which unit icon is this marker? -> (name, score, margin).
|
||||
|
||||
`name` is None when the best match does not beat the runner-up by
|
||||
`min_margin`; the markers are small on screen and several icons differ
|
||||
only in fine detail, so an unconfident answer must stay unknown rather
|
||||
than become a wrong unit type.
|
||||
"""
|
||||
x, y, w, h = box
|
||||
m = int(pad * max(w, h))
|
||||
x0, y0 = max(0, x - m), max(0, y - m)
|
||||
x1, y1 = min(img.shape[1], x + w + m), min(img.shape[0], y + h + m)
|
||||
crop = img[y0:y1, x0:x1]
|
||||
if crop.size == 0 or min(crop.shape[:2]) < 8:
|
||||
return None, 0.0, 0.0
|
||||
g = cv2.resize(cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY),
|
||||
(ICON_SIZE, ICON_SIZE), interpolation=cv2.INTER_AREA)
|
||||
q = _zscore(_inner(g))
|
||||
scores = [(float((q * t).mean()), name) for name, t in icon_bank(side)]
|
||||
if not scores:
|
||||
return None, 0.0, 0.0
|
||||
scores.sort(reverse=True)
|
||||
best, second = scores[0], (scores[1] if len(scores) > 1 else (0.0, None))
|
||||
margin = best[0] - second[0]
|
||||
# An absolute floor as well as a margin. With only the margin, matching
|
||||
# collapsed onto one class ("Ship") for nearly every marker at scores of
|
||||
# 0.29-0.46 -- confidently wrong, which is worse than admitting ignorance,
|
||||
# because a wrong unit type looks like real intel. Markers are only ~30px
|
||||
# across at the working resolution; cropping the symbol from the NATIVE
|
||||
# resolution screenshot is the fix, not a lower threshold.
|
||||
ok = margin >= min_margin and best[0] >= min_score
|
||||
return (best[1] if ok else None), best[0], margin
|
||||
|
||||
|
||||
def marker_masks(img):
|
||||
"""Hostile (pink/red) and friendly (cyan) marker colours."""
|
||||
hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
|
||||
h, s, v = hsv[:, :, 0], hsv[:, :, 1], hsv[:, :, 2]
|
||||
hostile = ((h <= 10) | (h >= 168)) & (s > 55) & (v > 95)
|
||||
friendly = (h >= 82) & (h <= 105) & (s > 55) & (v > 110)
|
||||
k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3))
|
||||
return [cv2.morphologyEx(m.astype(np.uint8) * 255, cv2.MORPH_CLOSE, k)
|
||||
for m in (hostile, friendly)]
|
||||
|
||||
|
||||
def diamonds(mask, cell_px):
|
||||
"""Marker-sized, marker-shaped blobs.
|
||||
|
||||
The markers scale with the map, so a solved grid tells us how big one
|
||||
must be (~0.14 of a cell). A diamond also fills about half its bounding
|
||||
box, which rejects the long thin territory hatching and front-line
|
||||
ribbons that share the markers' colours.
|
||||
"""
|
||||
want = 0.14 * cell_px
|
||||
lo, hi = 0.55 * want, 1.9 * want
|
||||
n, lab, stats, cent = cv2.connectedComponentsWithStats(mask, 8)
|
||||
out = []
|
||||
for i in range(1, n):
|
||||
x, y, w, h, a = stats[i]
|
||||
if not (lo <= max(w, h) <= hi) or min(w, h) < 0.4 * lo:
|
||||
continue
|
||||
if not (0.55 <= w / h <= 1.8):
|
||||
continue
|
||||
if not (0.30 <= a / float(w * h) <= 0.85):
|
||||
continue
|
||||
# Actually test for a DIAMOND. A bounding-box fill ratio near 0.5 is
|
||||
# not enough: a chunk of the territory hatching or of a front-line
|
||||
# ribbon hits the same ratio and the same colour, which is where the
|
||||
# tens of spurious markers came from. Compare the blob against an
|
||||
# ideal diamond inscribed in its own bounding box.
|
||||
blob = (lab[y:y + h, x:x + w] == i)
|
||||
ideal = np.zeros((h, w), np.uint8)
|
||||
cv2.fillConvexPoly(ideal, np.array(
|
||||
[[w // 2, 0], [w - 1, h // 2], [w // 2, h - 1], [0, h // 2]], np.int32), 1)
|
||||
ideal = ideal.astype(bool)
|
||||
union = int(np.logical_or(blob, ideal).sum())
|
||||
if union == 0:
|
||||
continue
|
||||
if int(np.logical_and(blob, ideal).sum()) / union < DIAMOND_IOU:
|
||||
continue
|
||||
out.append((float(cent[i][0]), float(cent[i][1]), (int(x), int(y), int(w), int(h))))
|
||||
return out
|
||||
|
||||
|
||||
def find_markers(img, sol):
|
||||
"""-> list of dicts: side, unit, label, sub_x, sub_y, coord, centre, box."""
|
||||
cell = max(sol.steps)
|
||||
found = []
|
||||
for side, mask in zip(("hostile", "friendly"), marker_masks(img)):
|
||||
for (cx, cy, box) in diamonds(mask, cell):
|
||||
c = sol.cell_of(cx, cy)
|
||||
if c is None:
|
||||
continue
|
||||
unit, score, margin = classify_marker(img, box, side)
|
||||
found.append(dict(side=side, unit=unit, unit_score=score,
|
||||
unit_margin=margin, label=c[0], sub_x=c[1],
|
||||
sub_y=c[2], coord=format_coord(c),
|
||||
centre=(cx, cy), box=box))
|
||||
return found
|
||||
36
tests/fixtures/README.md
vendored
Normal file
36
tests/fixtures/README.md
vendored
Normal file
@ -0,0 +1,36 @@
|
||||
# Test fixtures
|
||||
|
||||
Real screenshots of the game, used as regression data. They are the game
|
||||
author's work, not covered by this repo's MIT license (see `/LICENSE`).
|
||||
|
||||
## `map_shots/` + `map_shots_gt.json`
|
||||
|
||||
The evaluation set for the map-grid solver (`src/fenigma/map_vision.py`), scored
|
||||
by `tools/eval_map_vision.py`. The JSON holds hand-transcribed cell labels at
|
||||
native pixel positions; its own header comment explains the format and the
|
||||
9%/6% label-padding constant.
|
||||
|
||||
Shots wider than 2400px were downscaled to 2400px, and their ground-truth
|
||||
coordinates rescaled with them. 2400 is `map_vision.RETRY_WORK_W`, the widest
|
||||
the pipeline ever works at, so nothing the code can actually read was lost.
|
||||
Measured after the downscale: the same 7 of 10 solve, 100% of their points land
|
||||
in the correct cell, residual spread unchanged.
|
||||
|
||||
`too_hard/` holds shots that are permanent rejections; see its own README.
|
||||
|
||||
## `writer_shots/`
|
||||
|
||||
Typewriter/field-log screenshots. Two uses:
|
||||
|
||||
- Measuring false positives in the map-vs-text routing gate
|
||||
(`map_vision.looks_like_map` / `solve`). Over the full 122-shot set the cheap
|
||||
gate false-positived on 6% and `solve()` accepted **none**.
|
||||
- OCR regression material for `ocr.py`.
|
||||
|
||||
Kept at NATIVE resolution deliberately: the routing gate only ever sees 1500px,
|
||||
but OCR needs the text legible, so these must not be downscaled.
|
||||
|
||||
Ten shots are committed, chosen to span the capture-scale range (262px to
|
||||
5366px wide) since scale is what both the gate and OCR are sensitive to. The
|
||||
false-positive numbers above were measured on all 122; this subset is a
|
||||
regression guard, not the measurement.
|
||||
BIN
tests/fixtures/map_shots/01.png
vendored
Normal file
BIN
tests/fixtures/map_shots/01.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.1 MiB |
BIN
tests/fixtures/map_shots/02.png
vendored
Normal file
BIN
tests/fixtures/map_shots/02.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.5 MiB |
BIN
tests/fixtures/map_shots/03.png
vendored
Normal file
BIN
tests/fixtures/map_shots/03.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.8 MiB |
BIN
tests/fixtures/map_shots/04.png
vendored
Normal file
BIN
tests/fixtures/map_shots/04.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.5 MiB |
BIN
tests/fixtures/map_shots/05.png
vendored
Normal file
BIN
tests/fixtures/map_shots/05.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.2 MiB |
BIN
tests/fixtures/map_shots/07.png
vendored
Normal file
BIN
tests/fixtures/map_shots/07.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.5 MiB |
BIN
tests/fixtures/map_shots/08.png
vendored
Normal file
BIN
tests/fixtures/map_shots/08.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.3 MiB |
BIN
tests/fixtures/map_shots/09.png
vendored
Normal file
BIN
tests/fixtures/map_shots/09.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.2 MiB |
BIN
tests/fixtures/map_shots/11.png
vendored
Normal file
BIN
tests/fixtures/map_shots/11.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.4 MiB |
BIN
tests/fixtures/map_shots/13.png
vendored
Normal file
BIN
tests/fixtures/map_shots/13.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 2.8 MiB |
BIN
tests/fixtures/map_shots/too_hard/06.png
vendored
Normal file
BIN
tests/fixtures/map_shots/too_hard/06.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.1 MiB |
BIN
tests/fixtures/map_shots/too_hard/12.png
vendored
Normal file
BIN
tests/fixtures/map_shots/too_hard/12.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 501 KiB |
BIN
tests/fixtures/map_shots/too_hard/14.png
vendored
Normal file
BIN
tests/fixtures/map_shots/too_hard/14.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 4.5 MiB |
14
tests/fixtures/map_shots/too_hard/README.md
vendored
Normal file
14
tests/fixtures/map_shots/too_hard/README.md
vendored
Normal file
@ -0,0 +1,14 @@
|
||||
# Screenshots we deliberately do not try to solve
|
||||
|
||||
Kept for the record, excluded from the evaluation set. These fail for reasons
|
||||
that are properties of the screenshot, not of the algorithm, so working around
|
||||
them would mean guessing:
|
||||
|
||||
- `06.png`, `14.png` — exactly ONE grid label visible. A single label cannot be
|
||||
cross-checked, so a misread would silently shift the whole board with nothing
|
||||
to contradict it. Two mutually consistent labels is the minimum safe anchor.
|
||||
- `12.png` — 710x594 native. Too few pixels per cell for the label glyphs to
|
||||
correlate; measured, the best label score stays ~0.44 at every working
|
||||
resolution, so it is not a tuning problem.
|
||||
|
||||
"Too zoomed in" and "too low resolution" are legitimate hard rejections.
|
||||
640
tests/fixtures/map_shots_gt.json
vendored
Normal file
640
tests/fixtures/map_shots_gt.json
vendored
Normal file
@ -0,0 +1,640 @@
|
||||
{
|
||||
"_comment": [
|
||||
"Ground truth for the map-vision fixtures, transcribed by hand from the",
|
||||
"screenshots. Each entry is [cell_label, x, y] in NATIVE pixel coordinates",
|
||||
"of the corresponding file in map_shots/, where (x, y) is roughly the",
|
||||
"centre of the drawn cell label glyphs.",
|
||||
"",
|
||||
"The invariant being asserted is simply: the pixel (x, y) lies inside the",
|
||||
"map cell named by cell_label. That is enough to catch every failure mode",
|
||||
"seen so far (wrong lattice scale, wrong integer offset, badly misfitted",
|
||||
"homography) without needing sub-pixel corner annotation.",
|
||||
"",
|
||||
"Positions were read off a ruler overlay by eye, so treat them as accurate",
|
||||
"to roughly +/-15 native px. The cell IDENTITIES are exact.",
|
||||
"",
|
||||
"Game UI constant, useful for the estimator: a cell's label is drawn with",
|
||||
"about 9% of the cell size as padding from the cell's left edge and 6% from",
|
||||
"its top edge, so label_top_left - (0.09, 0.06) * cell_size lands on the",
|
||||
"cell's top-left corner."
|
||||
],
|
||||
"01.png": [
|
||||
[
|
||||
"G10",
|
||||
899,
|
||||
294
|
||||
],
|
||||
[
|
||||
"H10",
|
||||
996,
|
||||
294
|
||||
],
|
||||
[
|
||||
"I10",
|
||||
1083,
|
||||
294
|
||||
],
|
||||
[
|
||||
"J10",
|
||||
1168,
|
||||
294
|
||||
],
|
||||
[
|
||||
"K10",
|
||||
1270,
|
||||
294
|
||||
],
|
||||
[
|
||||
"L10",
|
||||
1355,
|
||||
294
|
||||
],
|
||||
[
|
||||
"M10",
|
||||
1446,
|
||||
294
|
||||
],
|
||||
[
|
||||
"N10",
|
||||
1537,
|
||||
294
|
||||
],
|
||||
[
|
||||
"O10",
|
||||
1628,
|
||||
294
|
||||
],
|
||||
[
|
||||
"P10",
|
||||
1716,
|
||||
294
|
||||
],
|
||||
[
|
||||
"G9",
|
||||
882,
|
||||
366
|
||||
],
|
||||
[
|
||||
"I9",
|
||||
1072,
|
||||
366
|
||||
],
|
||||
[
|
||||
"K9",
|
||||
1266,
|
||||
366
|
||||
],
|
||||
[
|
||||
"M9",
|
||||
1456,
|
||||
366
|
||||
],
|
||||
[
|
||||
"O9",
|
||||
1650,
|
||||
366
|
||||
],
|
||||
[
|
||||
"H8",
|
||||
965,
|
||||
445
|
||||
],
|
||||
[
|
||||
"J8",
|
||||
1168,
|
||||
445
|
||||
],
|
||||
[
|
||||
"L8",
|
||||
1370,
|
||||
445
|
||||
],
|
||||
[
|
||||
"N8",
|
||||
1572,
|
||||
445
|
||||
],
|
||||
[
|
||||
"H7",
|
||||
953,
|
||||
536
|
||||
],
|
||||
[
|
||||
"J7",
|
||||
1166,
|
||||
536
|
||||
],
|
||||
[
|
||||
"L7",
|
||||
1379,
|
||||
536
|
||||
],
|
||||
[
|
||||
"N7",
|
||||
1592,
|
||||
536
|
||||
],
|
||||
[
|
||||
"H6",
|
||||
935,
|
||||
637
|
||||
],
|
||||
[
|
||||
"J6",
|
||||
1166,
|
||||
637
|
||||
],
|
||||
[
|
||||
"L6",
|
||||
1391,
|
||||
637
|
||||
],
|
||||
[
|
||||
"N6",
|
||||
1619,
|
||||
637
|
||||
]
|
||||
],
|
||||
"02.png": [
|
||||
[
|
||||
"I9",
|
||||
54,
|
||||
28
|
||||
],
|
||||
[
|
||||
"J9",
|
||||
329,
|
||||
28
|
||||
],
|
||||
[
|
||||
"K9",
|
||||
608,
|
||||
30
|
||||
],
|
||||
[
|
||||
"L9",
|
||||
880,
|
||||
28
|
||||
],
|
||||
[
|
||||
"M9",
|
||||
1155,
|
||||
28
|
||||
],
|
||||
[
|
||||
"I8",
|
||||
37,
|
||||
257
|
||||
],
|
||||
[
|
||||
"J8",
|
||||
320,
|
||||
257
|
||||
],
|
||||
[
|
||||
"K8",
|
||||
620,
|
||||
257
|
||||
],
|
||||
[
|
||||
"L8",
|
||||
902,
|
||||
257
|
||||
],
|
||||
[
|
||||
"M8",
|
||||
1197,
|
||||
257
|
||||
],
|
||||
[
|
||||
"I7",
|
||||
11,
|
||||
517
|
||||
],
|
||||
[
|
||||
"J7",
|
||||
316,
|
||||
517
|
||||
],
|
||||
[
|
||||
"K7",
|
||||
626,
|
||||
517
|
||||
],
|
||||
[
|
||||
"L7",
|
||||
936,
|
||||
517
|
||||
],
|
||||
[
|
||||
"M7",
|
||||
1248,
|
||||
517
|
||||
],
|
||||
[
|
||||
"J6",
|
||||
313,
|
||||
805
|
||||
],
|
||||
[
|
||||
"K6",
|
||||
638,
|
||||
805
|
||||
],
|
||||
[
|
||||
"L6",
|
||||
968,
|
||||
805
|
||||
],
|
||||
[
|
||||
"M6",
|
||||
1295,
|
||||
805
|
||||
]
|
||||
],
|
||||
"03.png": [
|
||||
[
|
||||
"H8",
|
||||
224,
|
||||
229
|
||||
],
|
||||
[
|
||||
"I8",
|
||||
856,
|
||||
229
|
||||
],
|
||||
[
|
||||
"J8",
|
||||
1496,
|
||||
229
|
||||
],
|
||||
[
|
||||
"K8",
|
||||
2136,
|
||||
229
|
||||
],
|
||||
[
|
||||
"H7",
|
||||
216,
|
||||
853
|
||||
],
|
||||
[
|
||||
"I7",
|
||||
896,
|
||||
853
|
||||
],
|
||||
[
|
||||
"J7",
|
||||
1555,
|
||||
853
|
||||
],
|
||||
[
|
||||
"K7",
|
||||
2224,
|
||||
853
|
||||
]
|
||||
],
|
||||
"04.png": [
|
||||
[
|
||||
"N8",
|
||||
243,
|
||||
180
|
||||
],
|
||||
[
|
||||
"O8",
|
||||
1119,
|
||||
180
|
||||
],
|
||||
[
|
||||
"P8",
|
||||
1996,
|
||||
180
|
||||
],
|
||||
[
|
||||
"N7",
|
||||
232,
|
||||
1038
|
||||
],
|
||||
[
|
||||
"O7",
|
||||
1135,
|
||||
1038
|
||||
],
|
||||
[
|
||||
"P7",
|
||||
2051,
|
||||
1038
|
||||
]
|
||||
],
|
||||
"05.png": [
|
||||
[
|
||||
"M3",
|
||||
261,
|
||||
213
|
||||
],
|
||||
[
|
||||
"N3",
|
||||
1160,
|
||||
213
|
||||
]
|
||||
],
|
||||
"07.png": [
|
||||
[
|
||||
"J8",
|
||||
460,
|
||||
285
|
||||
],
|
||||
[
|
||||
"K8",
|
||||
1221,
|
||||
285
|
||||
],
|
||||
[
|
||||
"L8",
|
||||
1973,
|
||||
285
|
||||
],
|
||||
[
|
||||
"J7",
|
||||
456,
|
||||
1036
|
||||
],
|
||||
[
|
||||
"K7",
|
||||
1256,
|
||||
1036
|
||||
],
|
||||
[
|
||||
"L7",
|
||||
2053,
|
||||
1036
|
||||
]
|
||||
],
|
||||
"08.png": [
|
||||
[
|
||||
"L8",
|
||||
184,
|
||||
417
|
||||
],
|
||||
[
|
||||
"M8",
|
||||
1085,
|
||||
417
|
||||
],
|
||||
[
|
||||
"N8",
|
||||
1983,
|
||||
417
|
||||
],
|
||||
[
|
||||
"L7",
|
||||
176,
|
||||
1341
|
||||
],
|
||||
[
|
||||
"M7",
|
||||
1121,
|
||||
1341
|
||||
]
|
||||
],
|
||||
"09.png": [
|
||||
[
|
||||
"O9",
|
||||
1348,
|
||||
180
|
||||
],
|
||||
[
|
||||
"P9",
|
||||
1520,
|
||||
180
|
||||
],
|
||||
[
|
||||
"Q9",
|
||||
1692,
|
||||
180
|
||||
],
|
||||
[
|
||||
"M8",
|
||||
993,
|
||||
336
|
||||
],
|
||||
[
|
||||
"N8",
|
||||
1172,
|
||||
336
|
||||
],
|
||||
[
|
||||
"O8",
|
||||
1352,
|
||||
336
|
||||
],
|
||||
[
|
||||
"P8",
|
||||
1536,
|
||||
336
|
||||
],
|
||||
[
|
||||
"Q8",
|
||||
1718,
|
||||
336
|
||||
],
|
||||
[
|
||||
"M7",
|
||||
982,
|
||||
513
|
||||
],
|
||||
[
|
||||
"N7",
|
||||
1175,
|
||||
513
|
||||
],
|
||||
[
|
||||
"O7",
|
||||
1366,
|
||||
513
|
||||
],
|
||||
[
|
||||
"P7",
|
||||
1557,
|
||||
513
|
||||
],
|
||||
[
|
||||
"Q7",
|
||||
1748,
|
||||
513
|
||||
]
|
||||
],
|
||||
"11.png": [
|
||||
[
|
||||
"J9",
|
||||
174,
|
||||
133
|
||||
],
|
||||
[
|
||||
"K9",
|
||||
996,
|
||||
133
|
||||
],
|
||||
[
|
||||
"L9",
|
||||
1803,
|
||||
133
|
||||
],
|
||||
[
|
||||
"J8",
|
||||
159,
|
||||
956
|
||||
],
|
||||
[
|
||||
"K8",
|
||||
999,
|
||||
956
|
||||
],
|
||||
[
|
||||
"L8",
|
||||
1832,
|
||||
956
|
||||
]
|
||||
],
|
||||
"13.png": [
|
||||
[
|
||||
"L4",
|
||||
182,
|
||||
213
|
||||
],
|
||||
[
|
||||
"M4",
|
||||
509,
|
||||
197
|
||||
],
|
||||
[
|
||||
"O4",
|
||||
1129,
|
||||
157
|
||||
],
|
||||
[
|
||||
"P4",
|
||||
1425,
|
||||
133
|
||||
],
|
||||
[
|
||||
"Q4",
|
||||
1711,
|
||||
117
|
||||
],
|
||||
[
|
||||
"L3",
|
||||
146,
|
||||
441
|
||||
],
|
||||
[
|
||||
"M3",
|
||||
511,
|
||||
412
|
||||
],
|
||||
[
|
||||
"N3",
|
||||
863,
|
||||
388
|
||||
],
|
||||
[
|
||||
"O3",
|
||||
1209,
|
||||
359
|
||||
],
|
||||
[
|
||||
"P3",
|
||||
1538,
|
||||
329
|
||||
],
|
||||
[
|
||||
"L2",
|
||||
106,
|
||||
724
|
||||
],
|
||||
[
|
||||
"M2",
|
||||
518,
|
||||
687
|
||||
],
|
||||
[
|
||||
"N2",
|
||||
914,
|
||||
651
|
||||
],
|
||||
[
|
||||
"O2",
|
||||
1304,
|
||||
615
|
||||
],
|
||||
[
|
||||
"P2",
|
||||
1664,
|
||||
580
|
||||
],
|
||||
[
|
||||
"L1",
|
||||
37,
|
||||
1100
|
||||
],
|
||||
[
|
||||
"M1",
|
||||
521,
|
||||
1053
|
||||
],
|
||||
[
|
||||
"N1",
|
||||
985,
|
||||
1005
|
||||
],
|
||||
[
|
||||
"O1",
|
||||
1428,
|
||||
953
|
||||
],
|
||||
[
|
||||
"P1",
|
||||
1852,
|
||||
910
|
||||
]
|
||||
],
|
||||
"_excluded": {
|
||||
"note": "moved to map_shots/too_hard/, see its README",
|
||||
"06.png": [
|
||||
[
|
||||
"L5",
|
||||
542,
|
||||
616
|
||||
]
|
||||
],
|
||||
"12.png": [
|
||||
[
|
||||
"M3",
|
||||
151,
|
||||
160
|
||||
],
|
||||
[
|
||||
"N3",
|
||||
535,
|
||||
136
|
||||
],
|
||||
[
|
||||
"M2",
|
||||
163,
|
||||
469
|
||||
],
|
||||
[
|
||||
"N2",
|
||||
580,
|
||||
440
|
||||
]
|
||||
],
|
||||
"14.png": [
|
||||
[
|
||||
"J7",
|
||||
1557,
|
||||
1659
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
BIN
tests/fixtures/writer_shots/01.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/01.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 386 KiB |
BIN
tests/fixtures/writer_shots/02.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/02.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1008 KiB |
BIN
tests/fixtures/writer_shots/03.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/03.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.6 MiB |
BIN
tests/fixtures/writer_shots/04.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/04.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 2.1 MiB |
BIN
tests/fixtures/writer_shots/05.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/05.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 7.3 MiB |
BIN
tests/fixtures/writer_shots/06.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/06.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 637 KiB |
BIN
tests/fixtures/writer_shots/07.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/07.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 3.7 MiB |
BIN
tests/fixtures/writer_shots/08.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/08.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
BIN
tests/fixtures/writer_shots/09.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/09.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 858 KiB |
BIN
tests/fixtures/writer_shots/10.png
vendored
Normal file
BIN
tests/fixtures/writer_shots/10.png
vendored
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
159
tools/eval_map_vision.py
Normal file
159
tools/eval_map_vision.py
Normal file
@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evaluate map_vision against the hand annotations, and render overlays.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python tools/eval_map_vision.py # score the fixtures
|
||||
.venv/bin/python tools/eval_map_vision.py shot.png [...] # try your own images
|
||||
|
||||
Two scores are reported, because the lenient one hid a real error:
|
||||
|
||||
cell fraction of annotated points landing in the correct cell. Too
|
||||
forgiving on its own: the annotations sit near cell centres, so a
|
||||
grid wrong by a whole line still passes.
|
||||
spread every annotated point should sit at (col + a, row + b) in recovered
|
||||
grid coordinates for ONE constant (a, b) -- the label's offset
|
||||
inside its cell. So fit that constant and report the spread of the
|
||||
residual, in cell units. A skewed or off-by-one-line grid shows up
|
||||
here even when every point is nominally in the right cell.
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
||||
from fenigma import map_vision as mv # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SHOTS = ROOT / "tests" / "fixtures" / "map_shots"
|
||||
GT_PATH = ROOT / "tests" / "fixtures" / "map_shots_gt.json"
|
||||
|
||||
|
||||
def draw(img, sol, markers):
|
||||
vis = img.copy()
|
||||
h, w = vis.shape[:2]
|
||||
q = np.linalg.inv(sol.H) @ np.array([[0, w, w, 0], [0, 0, h, h], [1, 1, 1, 1]], np.float64)
|
||||
ij = q[:2] / q[2]
|
||||
for i in range(int(np.floor(ij[0].min())) - 1, int(np.ceil(ij[0].max())) + 2):
|
||||
for j in range(int(np.floor(ij[1].min())) - 1, int(np.ceil(ij[1].max())) + 2):
|
||||
col, row = sol.si * i + sol.du, sol.sj * j + sol.dv
|
||||
if not (0 <= col < mv.COLS and 1 <= row <= mv.ROWS):
|
||||
continue
|
||||
p = sol.H @ np.array([[i, i + 1, i + 1, i], [j, j, j + 1, j + 1], [1, 1, 1, 1.0]])
|
||||
if np.any(np.abs(p[2]) < 1e-9):
|
||||
continue
|
||||
p = (p[:2] / p[2]).T
|
||||
cv2.polylines(vis, [p.astype(np.int32)], True, (0, 255, 255), 2, cv2.LINE_AA)
|
||||
lx = i + (mv.PAD_L if sol.si > 0 else 1 - mv.PAD_L)
|
||||
ly = j + (mv.PAD_T if sol.sj > 0 else 1 - mv.PAD_T)
|
||||
t = sol.H @ np.array([lx, ly, 1.0])
|
||||
if abs(t[2]) < 1e-9:
|
||||
continue
|
||||
x, y = int(t[0] / t[2]), int(t[1] / t[2])
|
||||
f = max(0.45, min(2.0, float(np.linalg.norm(p[1] - p[0])) / 190.0))
|
||||
txt = f"{mv.LARGE_X[col]}{row}"
|
||||
cv2.putText(vis, txt, (x, y), cv2.FONT_HERSHEY_SIMPLEX, f, (0, 0, 0), int(f * 5) + 2)
|
||||
cv2.putText(vis, txt, (x, y), cv2.FONT_HERSHEY_SIMPLEX, f, (0, 255, 0), int(f * 2) + 1)
|
||||
for m in markers:
|
||||
x, y, bw, bh = m["box"]
|
||||
c = (0, 0, 255) if m["side"] == "hostile" else (255, 210, 0)
|
||||
cv2.rectangle(vis, (x, y), (x + bw, y + bh), c, 2)
|
||||
txt = m["coord"] + (f' {m["unit"]}' if m.get("unit") else "")
|
||||
cv2.putText(vis, txt, (x, y - 6), cv2.FONT_HERSHEY_SIMPLEX, 0.55, (0, 0, 0), 4)
|
||||
cv2.putText(vis, txt, (x, y - 6), cv2.FONT_HERSHEY_SIMPLEX, 0.55, c, 2)
|
||||
return vis
|
||||
|
||||
|
||||
def run_one(path, outdir):
|
||||
"""Solve a single arbitrary screenshot (no ground truth needed)."""
|
||||
sol, img, err = mv.solve_path(path)
|
||||
if sol is None:
|
||||
print(f"{path.name}: REJECTED - {err}")
|
||||
return 1
|
||||
markers = mv.find_markers(img, sol)
|
||||
print(f"{path.name}: solved, {sol.votes} label votes, "
|
||||
f"cell {sol.steps[0]:.0f}x{sol.steps[1]:.0f}px, {len(markers)} markers")
|
||||
for m in markers:
|
||||
print(f" {m['side']:<8} {m['coord']:<9} {m.get('unit') or 'unknown type'}")
|
||||
out = outdir / f"solved_{path.stem}.png"
|
||||
cv2.imwrite(str(out), draw(img, sol, markers))
|
||||
print(f" overlay: {out}")
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
args = [a for a in sys.argv[1:]]
|
||||
outdir = ROOT / "build" / "map_vision"
|
||||
images = [Path(a) for a in args if Path(a).suffix.lower() in (".png", ".jpg", ".jpeg")]
|
||||
if images:
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
rc = 0
|
||||
for p in images:
|
||||
rc |= run_one(p, outdir)
|
||||
return rc
|
||||
outdir = Path(args[0]) if args else outdir
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
gt = json.loads(GT_PATH.read_text())
|
||||
names = sorted(k for k in gt if k.endswith(".png"))
|
||||
tiles, ok_t, n_t = [], 0, 0
|
||||
print(f"{'shot':<8} {'pts':>4} {'cell':>6} {'spread':>7} {'votes':>5} "
|
||||
f"{'mk':>3} status")
|
||||
for name in names:
|
||||
path = SHOTS / name
|
||||
native_w = cv2.imread(str(path), cv2.IMREAD_REDUCED_COLOR_8).shape[1] * 8
|
||||
pts = gt[name]
|
||||
n_t += len(pts)
|
||||
sol, img, err = mv.solve_path(path)
|
||||
s = img.shape[1] / native_w
|
||||
if sol is None:
|
||||
print(f"{name:<8} {len(pts):>4} {'-':>6} {'-':>7} {'-':>5} {'-':>3} REJECT: {err}")
|
||||
vis = img.copy()
|
||||
cv2.putText(vis, f"{name} REJECTED", (12, 34), cv2.FONT_HERSHEY_SIMPLEX,
|
||||
1.0, (0, 0, 255), 3)
|
||||
cv2.putText(vis, err[:56], (12, 66), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 255), 2)
|
||||
tiles.append(vis)
|
||||
cv2.imwrite(str(outdir / f"rejected_{name}"), vis)
|
||||
continue
|
||||
offs, right = [], 0
|
||||
for lab, x, y in pts:
|
||||
got = sol.cell_of(x * s, y * s)
|
||||
if got and got[0] == lab:
|
||||
right += 1
|
||||
g = sol.grid_of(x * s, y * s)
|
||||
if g is None:
|
||||
continue
|
||||
offs.append((g[0] - mv.LARGE_X.index(lab[0]), g[1] - int(lab[1:])))
|
||||
ok_t += right
|
||||
o = np.array(offs)
|
||||
spread = float(np.sqrt(((o - o.mean(axis=0)) ** 2).sum(axis=1).mean()))
|
||||
markers = mv.find_markers(img, sol)
|
||||
verdict = "GOOD" if spread < 0.08 else ("SKEWED" if spread < 0.3 else "BAD")
|
||||
print(f"{name:<8} {len(pts):>4} {right/len(pts):>5.0%} {spread:>7.3f} "
|
||||
f"{sol.votes:>5} {len(markers):>3} {verdict}")
|
||||
vis = draw(img, sol, markers)
|
||||
cv2.putText(vis, f"{name} {len(markers)} markers spread {spread:.3f}",
|
||||
(12, 34), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 3)
|
||||
tiles.append(vis)
|
||||
cv2.imwrite(str(outdir / f"solved_{name}"), vis)
|
||||
for m in markers:
|
||||
print(f" {m['side']:<8} {m['coord']:<9} "
|
||||
f"{str(m.get('unit')):<26} s={m['unit_score']:.2f} d={m['unit_margin']:.3f}")
|
||||
print(f"\nTOTAL {ok_t}/{n_t} annotated points in the correct cell "
|
||||
f"({ok_t / max(n_t, 1):.0%})")
|
||||
TW, TH, cols = 860, 520, 3
|
||||
rows = (len(tiles) + cols - 1) // cols
|
||||
sheet = np.zeros((TH * rows, TW * cols, 3), np.uint8)
|
||||
for i, t in enumerate(tiles):
|
||||
hh, ww = t.shape[:2]
|
||||
sc = min(TW / ww, TH / hh)
|
||||
t2 = cv2.resize(t, (int(ww * sc), int(hh * sc)))
|
||||
y, x = (i // cols) * TH, (i % cols) * TW
|
||||
sheet[y:y + t2.shape[0], x:x + t2.shape[1]] = t2
|
||||
cv2.imwrite(str(outdir / "sheet.png"), sheet)
|
||||
print(f"wrote {outdir / 'sheet.png'}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main() or 0)
|
||||
Loading…
Reference in New Issue
Block a user