Add the map-table vision pipeline

Reads a screenshot of the in-game map table and recovers where on the board
it is looking. The scene is a flat table under a perspective camera, so a
single homography describes grid<->screen exactly; that is fitted from line
evidence (LSD segments, vanishing-point RANSAC, 1-D lattice fits).

Line evidence alone cannot finish the job: the 1km cells and the 100m
subgrid are locally identical, so it fixes neither scale, axis assignment,
axis direction nor phase. Those come from the cell labels -- and the labels
are never *detected*, they are correlated where the grid says they must be
(9% in from a cell's left edge, 6% down from its top). Blob detection on
aerial-photo terrain finds texture, not glyphs; correlating a known template
at a known place has no such failure mode. The same score then ranks the
geometry hypotheses, since a wrong lattice puts the crop where no label is,
so one number resolves scale, axis assignment, direction, phase and anchor
together.

Markers are found by hostile/friendly colour plus an IoU test against an
ideal inscribed diamond, which cut a red-lit shot from 43 false positives to
2 while preserving every hand-verified count. Unit-type classification is
present but not yet reliable, and returns unknown rather than guessing.

Measured over the fixture set: 7 of 10 solve, each with 100% of its
ground-truth points in the correct cell (85/112 overall), residual spread
0.005-0.033 cells. The other three reject cleanly; none has ever produced a
plausible-but-wrong grid. Across 122 typewriter screenshots solve() accepted
none, which is what makes clipboard routing safe.

Fixtures: 10 map shots with hand-transcribed ground truth, plus 10
typewriter shots spanning 262-5366px for routing and future OCR tests. Map
shots wider than 2400px (the pipeline's own maximum working width) were
downscaled with their coordinates rescaled to match; re-measured afterwards,
the results are identical. Typewriter shots stay at native resolution
because OCR needs the text legible.

Drops docs/map_vision_plan.md and its WIP prototype: the design now lives in
the module docstring, next to the code it describes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-10 21:41:33 +02:00
co-authored by Claude Opus 5
parent 512a0a41b4
commit ff3b89c41b
34 changed files with 1810 additions and 401 deletions
+36
View File
@@ -0,0 +1,36 @@
# Test fixtures
Real screenshots of the game, used as regression data. They are the game
author's work, not covered by this repo's MIT license (see `/LICENSE`).
## `map_shots/` + `map_shots_gt.json`
The evaluation set for the map-grid solver (`src/fenigma/map_vision.py`), scored
by `tools/eval_map_vision.py`. The JSON holds hand-transcribed cell labels at
native pixel positions; its own header comment explains the format and the
9%/6% label-padding constant.
Shots wider than 2400px were downscaled to 2400px, and their ground-truth
coordinates rescaled with them. 2400 is `map_vision.RETRY_WORK_W`, the widest
the pipeline ever works at, so nothing the code can actually read was lost.
Measured after the downscale: the same 7 of 10 solve, 100% of their points land
in the correct cell, residual spread unchanged.
`too_hard/` holds shots that are permanent rejections; see its own README.
## `writer_shots/`
Typewriter/field-log screenshots. Two uses:
- Measuring false positives in the map-vs-text routing gate
(`map_vision.looks_like_map` / `solve`). Over the full 122-shot set the cheap
gate false-positived on 6% and `solve()` accepted **none**.
- OCR regression material for `ocr.py`.
Kept at NATIVE resolution deliberately: the routing gate only ever sees 1500px,
but OCR needs the text legible, so these must not be downscaled.
Ten shots are committed, chosen to span the capture-scale range (262px to
5366px wide) since scale is what both the gate and OCR are sensitive to. The
false-positive numbers above were measured on all 122; this subset is a
regression guard, not the measurement.
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.1 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.8 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.5 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.5 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.3 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.2 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.4 MiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.8 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 501 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 4.5 MiB

+14
View File
@@ -0,0 +1,14 @@
# Screenshots we deliberately do not try to solve
Kept for the record, excluded from the evaluation set. These fail for reasons
that are properties of the screenshot, not of the algorithm, so working around
them would mean guessing:
- `06.png`, `14.png` — exactly ONE grid label visible. A single label cannot be
cross-checked, so a misread would silently shift the whole board with nothing
to contradict it. Two mutually consistent labels is the minimum safe anchor.
- `12.png` — 710x594 native. Too few pixels per cell for the label glyphs to
correlate; measured, the best label score stays ~0.44 at every working
resolution, so it is not a tuning problem.
"Too zoomed in" and "too low resolution" are legitimate hard rejections.
+640
View File
@@ -0,0 +1,640 @@
{
"_comment": [
"Ground truth for the map-vision fixtures, transcribed by hand from the",
"screenshots. Each entry is [cell_label, x, y] in NATIVE pixel coordinates",
"of the corresponding file in map_shots/, where (x, y) is roughly the",
"centre of the drawn cell label glyphs.",
"",
"The invariant being asserted is simply: the pixel (x, y) lies inside the",
"map cell named by cell_label. That is enough to catch every failure mode",
"seen so far (wrong lattice scale, wrong integer offset, badly misfitted",
"homography) without needing sub-pixel corner annotation.",
"",
"Positions were read off a ruler overlay by eye, so treat them as accurate",
"to roughly +/-15 native px. The cell IDENTITIES are exact.",
"",
"Game UI constant, useful for the estimator: a cell's label is drawn with",
"about 9% of the cell size as padding from the cell's left edge and 6% from",
"its top edge, so label_top_left - (0.09, 0.06) * cell_size lands on the",
"cell's top-left corner."
],
"01.png": [
[
"G10",
899,
294
],
[
"H10",
996,
294
],
[
"I10",
1083,
294
],
[
"J10",
1168,
294
],
[
"K10",
1270,
294
],
[
"L10",
1355,
294
],
[
"M10",
1446,
294
],
[
"N10",
1537,
294
],
[
"O10",
1628,
294
],
[
"P10",
1716,
294
],
[
"G9",
882,
366
],
[
"I9",
1072,
366
],
[
"K9",
1266,
366
],
[
"M9",
1456,
366
],
[
"O9",
1650,
366
],
[
"H8",
965,
445
],
[
"J8",
1168,
445
],
[
"L8",
1370,
445
],
[
"N8",
1572,
445
],
[
"H7",
953,
536
],
[
"J7",
1166,
536
],
[
"L7",
1379,
536
],
[
"N7",
1592,
536
],
[
"H6",
935,
637
],
[
"J6",
1166,
637
],
[
"L6",
1391,
637
],
[
"N6",
1619,
637
]
],
"02.png": [
[
"I9",
54,
28
],
[
"J9",
329,
28
],
[
"K9",
608,
30
],
[
"L9",
880,
28
],
[
"M9",
1155,
28
],
[
"I8",
37,
257
],
[
"J8",
320,
257
],
[
"K8",
620,
257
],
[
"L8",
902,
257
],
[
"M8",
1197,
257
],
[
"I7",
11,
517
],
[
"J7",
316,
517
],
[
"K7",
626,
517
],
[
"L7",
936,
517
],
[
"M7",
1248,
517
],
[
"J6",
313,
805
],
[
"K6",
638,
805
],
[
"L6",
968,
805
],
[
"M6",
1295,
805
]
],
"03.png": [
[
"H8",
224,
229
],
[
"I8",
856,
229
],
[
"J8",
1496,
229
],
[
"K8",
2136,
229
],
[
"H7",
216,
853
],
[
"I7",
896,
853
],
[
"J7",
1555,
853
],
[
"K7",
2224,
853
]
],
"04.png": [
[
"N8",
243,
180
],
[
"O8",
1119,
180
],
[
"P8",
1996,
180
],
[
"N7",
232,
1038
],
[
"O7",
1135,
1038
],
[
"P7",
2051,
1038
]
],
"05.png": [
[
"M3",
261,
213
],
[
"N3",
1160,
213
]
],
"07.png": [
[
"J8",
460,
285
],
[
"K8",
1221,
285
],
[
"L8",
1973,
285
],
[
"J7",
456,
1036
],
[
"K7",
1256,
1036
],
[
"L7",
2053,
1036
]
],
"08.png": [
[
"L8",
184,
417
],
[
"M8",
1085,
417
],
[
"N8",
1983,
417
],
[
"L7",
176,
1341
],
[
"M7",
1121,
1341
]
],
"09.png": [
[
"O9",
1348,
180
],
[
"P9",
1520,
180
],
[
"Q9",
1692,
180
],
[
"M8",
993,
336
],
[
"N8",
1172,
336
],
[
"O8",
1352,
336
],
[
"P8",
1536,
336
],
[
"Q8",
1718,
336
],
[
"M7",
982,
513
],
[
"N7",
1175,
513
],
[
"O7",
1366,
513
],
[
"P7",
1557,
513
],
[
"Q7",
1748,
513
]
],
"11.png": [
[
"J9",
174,
133
],
[
"K9",
996,
133
],
[
"L9",
1803,
133
],
[
"J8",
159,
956
],
[
"K8",
999,
956
],
[
"L8",
1832,
956
]
],
"13.png": [
[
"L4",
182,
213
],
[
"M4",
509,
197
],
[
"O4",
1129,
157
],
[
"P4",
1425,
133
],
[
"Q4",
1711,
117
],
[
"L3",
146,
441
],
[
"M3",
511,
412
],
[
"N3",
863,
388
],
[
"O3",
1209,
359
],
[
"P3",
1538,
329
],
[
"L2",
106,
724
],
[
"M2",
518,
687
],
[
"N2",
914,
651
],
[
"O2",
1304,
615
],
[
"P2",
1664,
580
],
[
"L1",
37,
1100
],
[
"M1",
521,
1053
],
[
"N1",
985,
1005
],
[
"O1",
1428,
953
],
[
"P1",
1852,
910
]
],
"_excluded": {
"note": "moved to map_shots/too_hard/, see its README",
"06.png": [
[
"L5",
542,
616
]
],
"12.png": [
[
"M3",
151,
160
],
[
"N3",
535,
136
],
[
"M2",
163,
469
],
[
"N2",
580,
440
]
],
"14.png": [
[
"J7",
1557,
1659
]
]
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 386 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1008 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 7.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 637 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 858 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 120 KiB