diff --git a/CLAUDE.md b/CLAUDE.md index a5c733c..353a898 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,12 +43,12 @@ Run as `puga ...` (e.g. `puga scan --min-n 3`, `puga plan pull `); - `tools/state.py sync|show` refreshes `empire/state/company.yaml` from live FIO (cash, permits, buildings, real production efficiency, storage, ships). - `tools/plan_push.py` builds/validates PRUNplanner plans from YAML specs (`plans/examples/`, `empire/plans/`). Dry run by default; `--apply` only after the player says yes; only `[PuGa]`-named plans are created/updated/deleted (`delete `, only when asked); `list` shows the account's plans, `pull -o file.yaml` reads a plan back into a YAML spec including edits made in the PRUNplanner UI. Workflow: push a plan, the player refines it in the UI and saves, then `pull` / `simulate --uuid` read it back and you continue from there. - `tools/simulate.py [--off EXT,SME] [--basis real|uni30|vwap30|ask|...]` replica of PRUNplanner's simulator (efficiency, workforce, material I/O, profit). Flows and efficiency verified exact vs screenshots; profit within ~2%. The NEW CAPEX line = planned minus already built (from state), the honest payback; ignore the plan-level ROI (PRUNplanner always adds a core module). `--uuid` reads the SAVED plan from the account (UI edits must be saved first). -- `tools/scan.py` DEPTH-AWARE recipe scan (use this): buildings the market could absorb, ROI for our own size, patient prices, ask-walked inputs. Fully staffed AND understaffed variants by default (`--staffing`), freight netted out (`--trip-cost`, `--cargo`), HQ/COGC/experts/faction, `--planet ID` adds that planet's extraction, fertility and COGC (a planet not in state is a new base: no HQ, permits+1), `--deprec 60` prices demolish-later. `--min-n` is ONLY a noise filter on market capacity (use 3, ideally 10); ROI is always for our own size (`--own`, default 1 building), never at the filtered scale. Below 1 lets sub-building junk in. `--json rows.json` feeds `persistence.py`. Library: `puga/saturation.py`. +- `tools/scan.py` DEPTH-AWARE recipe scan (use this): buildings the market could absorb, ROI for our own size, patient prices, ask-walked inputs. Fully staffed AND understaffed variants by default (`--staffing`), freight netted out (`--trip-cost`, `--cargo`), HQ/COGC/experts/faction, `--planet ID` adds that planet's extraction, fertility and COGC (a planet not in state is a new base: no HQ, permits+1), `--deprec 60 [--reclaim-frac 0.65]` prices demolish-later: amortizes the UNRECLAIMED capex (not full capex - real FIO reclaim data shows 50-86% comes back even at near-100% condition, see `docs/decisions.md`) over the holding period. `--min-n` is ONLY a noise filter on market capacity (use 3, ideally 10); ROI is always for our own size (`--own`, default 1 building), never at the filtered scale. Below 1 lets sub-building junk in. `--json rows.json` feeds `persistence.py`. `--sort area` ranks by profit/day per tile of BUILDING area (FIO `AreaCost`, not hab area) - once an existing base's area cap binds, this is the real allocation criterion, not %-yield/payback (a higher-%% recipe needing much more area can lose to a lower-%% one that fits more densely). `--max-impact 10` (stage 2 only) flags `[exit-thin]` when the LIVE bid book can't absorb even 1 day of 1 building's output within that %% (`puga.saturation.max_units_at_impact`) - an INSTANT-DUMP/panic-exit liquidity check, not production price impact (that's what `market`/`lim` already measure via real 30-day flow); a single snapshot (orders change constantly), so it's a WARNING, not folded into `market`/`lim`. Library: `puga/saturation.py`. - `tools/history.py TICKER` monthly margin history of the recipe producing TICKER; `tools/persistence.py rows.json` re-prices scan rows over history (mean ROI 7/14/30/90/180d, payback, net gain over 7 and 14 days, % days profitable). ALWAYS check persistence before recommending: current margins are often a spike. - `tools/buy.py --days N` cash-aware shopping list: construction gap (plan vs empire state's built buildings) plus N days of the plan's NET operating stock (via `simulate()`, so self-produced inputs like AL net against consumption), priced at real fill cost, checked against cash/reserve. Pass a plan covering the WHOLE base for restocking, not just a new addition (see `docs/library.md`). -- `tools/sell.py TICKER QTY [--tight]` where to post an ask, expected hours to clear, and a self-contained tranche split: aggressive (undercut) tranche capped at the median 30d-volume quantile (or 80th pct with `--tight`, for when a payment is imminent and stockout risk outweighs a few points of margin), patient tranche priced just under the next competitor tier. `--show-bid` for the instant-bid comparison (off by default; usually worse). +- `tools/sell.py TICKER QTY [--tight]` where to post an ask, expected hours to clear, and a self-contained tranche split: aggressive (undercut) tranche capped at the median 30d-volume quantile (or 80th pct with `--tight`, for when a payment is imminent and stockout risk outweighs a few points of margin), patient tranche priced just under the next competitor tier. `--show-bid` for the instant-bid comparison (off by default; usually worse). `--tiers [--rate R]` answers "is it ever worth pricing above the current market": combines queue-depth days (real resting quantity, within the traded range) with an EMPIRICAL first-passage estimate from the ticker's own multi-year price history (no random-walk assumption - a driftless-walk closed form was tried first and backtested wrong, see `docs/decisions.md`'s "Optimal ask price" entry), via max (the binding constraint). `rate` should be the player's REALIZED capital growth rate (read off their equity-growth chart, default 0.05), not a theoretical build ROI. Library: `puga/history.py::empirical_touch()`. - `tools/network.py ` models a MULTI-BASE chain (e.g. mine LST at Nike, ship it, make BSE at Deimos): simulates each base, nets a transferred material's producer-surplus against consumer-need, charges real freight only on what's moved (`puga/network.py`). Spec and worked example in `plans/chains/`. `cm_free: true` per base for a new founding covered by a Core Module Kit. profit/day per base is the WHOLE plan, not the marginal addition - diff against the base's plan without the addition for a true marginal ROI. -- `tools/whatif.py [--add BUILDING:AMOUNT[:RECIPE]] [--infra HB1:+1] [--after after.yaml]` marginal payback of adding to a base you already run (whole-plan diff, same rule as buy/runway - pass the WHOLE base, not just the addition). `--add` mutates a copy of `before` in-memory (reuses an existing building's recipe if omitted, e.g. `--add POL:1`); `--after` takes a second full spec instead. Flags any workforce tier the addition newly overloads (vs. a pre-existing one, which it won't flag) - this is what caught the PP2/pioneer-housing and 2nd-CHP/pioneer-housing surprises by hand before this tool existed. Charges marginal freight too (real fuel model, CX-to-planet, via `puga.shipping`): only the EXTRA round trips this addition needs beyond the base's existing traffic, so a small addition riding along on already-necessary hauls correctly costs ~0, while one that meaningfully adds to shipped tonnage doesn't get a free pass - this is what would catch a recipe whose ROI looks great only because its freight cost was left out. +- `tools/whatif.py [--add BUILDING:AMOUNT[:RECIPE]] [--infra HB1:+1] [--after after.yaml]` marginal payback of adding to a base you already run (whole-plan diff, same rule as buy/runway - pass the WHOLE base, not just the addition). `--add` mutates a copy of `before` in-memory (reuses an existing building's recipe if omitted, e.g. `--add POL:1`); `--after` takes a second full spec instead. Flags any workforce tier the addition newly overloads (vs. a pre-existing one, which it won't flag) - this is what caught the PP2/pioneer-housing and 2nd-CHP/pioneer-housing surprises by hand before this tool existed. Charges marginal freight too (real fuel model, CX-to-planet, via `puga.shipping`): a CONTINUOUS amortized rate (extra tonnage/cargo-hold x round-trip cost), scaling with the addition's own tonnage, not a step function - a small addition adds a small continuous freight cost, not exactly zero, and a recipe whose ROI looks great only because its freight cost was left out still gets caught. - `tools/runway.py [--no-inbound]` days of runway per material before a base stalls: real NET consumption via `simulate()` (self-produced inputs excluded), base storage, PLUS (by default) any ship currently flying to that planet, auto-detected from live flight data (`tools/state.py` tags each ship with its flight origin/destination). Pass the WHOLE base plan (see `docs/library.md`). - `tools/undervalued.py [--cx AI1] [--min-vol 5] [--top 20]` manufactured goods currently priced below implied production cost - buy-and-hold-for-reversion candidates, not build candidates. **Expected ROI is much lower than production builds** (~0.10-0.24%/day backtested, vs 14-29%/day for the builds this toolkit recommends) - only worth it for excess idle capital once better-yielding builds are exhausted. Library: `puga/undervalued.py`. - `tools/chain.py TICKER` make-vs-buy cost tree plus sourcing depth of inputs. `--consumers` flips it: what recipes EAT this ticker as an input, with each output's market depth - for "is there a deeper downstream product worth chaining into" questions (e.g. checking BOR/BSE/BBH for further processing). @@ -56,6 +56,7 @@ Run as `puga ...` (e.g. `puga scan --min-n 3`, `puga plan pull `); - `tools/route.py ` jump-distance route (not a PRUNplanner feature - built from FIO's `/systemstars` graph, Dijkstra by straight-line ly since that's what FTL fuel/time scale with, jump count reported alongside). Accepts planet natural IDs, system natural IDs, or names. Library: `puga/route.py`. - `tools/shiptrip.py SHIP FROM TO [--v0 X --v1 Y]` the simple ship-trip tool: give a ship registration + start + stop, it works out jumps/charges/landing fuel itself via `tools/route.py`'s jump graph and reports fuel cost (defaults to max sliders). Direct-jump routes only - an unconnected pair means gateway-only, and it tells you exactly what to read off the live APEX Ship Flight Control screen (the LOCK fee - genuinely not available via FIO/PRUNplanner, confirmed by inspecting both) and what to run instead (`tools/shipcost.py`). Warns if the given ship's live specs (FIO `/ship/ships/{user}`) don't match the calibrated hull/reactor. - `tools/shipcost.py [--jumps N --charges N] [--v0 X --v1 Y] [--gateways N --lock-fee AMOUNT:CURRENCY] [--to PLANET]` the explicit/manual version: fuel/AIC cost of a ship trip from physics parameters you already know (jump count, slider values, gateway LOCK fees read off the live game) - use `tools/shiptrip.py` instead if you just want "ship X from A to B". Landing fuel is a measured-planet lookup via `--to`, independent of whether the route used jumps or gateways. Both tools scoped to this account's one hull/reactor (blueprint BP-STRT-0000, shared by both ships AVI-07JFB/AVI-07JFC); open caveats (v0 relative to current fuel load, landing fuel confounded with gravity/distance, no second hull to cross-check) are in `puga/shipping.py`'s module docstring - don't trust magnitudes beyond what's there. Raw calibration data (APEX DOM reads, 2026-10-09): `empire/data/shipping_calibration/`. Also wired into `tools/network.py`'s freight cost (direct-jump transfers use the real model; gateway transfers need an explicit `trip_cost` in the chain YAML, else a 9250 AIC fallback). Library: `puga/shipping.py` (`trip_fuel()`, `round_trip_cost()`). +- `tools/parkcash.py [--min-spread X] [--min-trade-days N] [--rate R] [--top N]` where to park idle cash as a buy-low/sell-high position (not a build) while ships are busy or cash has no better use right now - scans every ticker for a big bid/ask spread AND real trading liquidity (trade-days/gap over a calendar window - a huge spread with few trading days is a stale-quote trap, confirmed by hand: AST showed a 130% spread but its resting ask had gone stale after a 60% real price crash). Round-trip wait/value is EMPIRICAL (`puga.history.empirical_touch`, same model as `tools/sell.py --tiers`), not an assumed random walk - a driftless-walk version was tried and backtested wrong (overstated reachability 2-4x vs 3 years of real data, see `docs/decisions.md`). This dropped the whole universe's top pick from ~94%/day to ~1%/day with real waits of 19-194+ days - treat any number from the old model as stale. `--rate` defaults to 0.0 (matches the idle-cash premise; pass your realized growth rate if this cash IS earmarked) and shows up as a separate `discounted_pct` column (margin% x real odds both legs ever happen) alongside the simple non-compounding `daily_pct` used for ranking. Library: `puga/parkcash.py`. - `tools/lmscan.py PLANET [--direction source|destination] [--min-margin X]` live Local Market hauling-ad scan (FIO, public/no-auth - real player-posted freight jobs, not the CONTD player-contract system), ranked by margin after real fuel cost (`puga.shipping.leg_cost`). One-way cost only - margin assumes the next job is picked up near the destination, not an empty flight home. A small measured gateway-fee table covers ANT<->HRT/BEN; other gateway routes can't be priced (LOCK fee isn't available via any API). Library: `puga/contracts.py`. - `tools/price.py` prices across exchanges with VWAP, volume, fill price for a quantity; `tools/book.py` order-book ladder. - `tools/prun_scan.py`, `tools/prun_cxarb.py` legacy top-of-book scans; overstate thin markets. Baseline only. `tools/arb.py` replacement is on the roadmap. diff --git a/docs/decisions.md b/docs/decisions.md index 049582f..6c28741 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -17,6 +17,144 @@ - If a bigger-model spawn is refused, tell the player and ask them to run the review. - Routine tool building is not reviewed by a bigger model. +## 2026-10-10 Optimal ask price: first-passage-time (Black-Scholes), not order-book depth +**SUPERSEDED same day - see "Black-Scholes model backtested wrong" below. The closed form here +(`daily_log_vol()`, a driftless-random-walk assumption) is no longer what `tools/sell.py --tiers` +uses; kept for the reasoning trail, not as current guidance.** +- The naive model ("units ahead of me in the book ÷ historical daily volume = days to clear") + is WRONG: it assumes the book is static (no one else reprices, cancels, or adds orders while + you wait) and that historical trade volume at normal prices still applies at a resting ask far + outside the recently-traded range. Tried first, broke immediately: fed a fat-finger 1,000,000 + AIC ask the same reachability as the real book, "confirming" it as optimal. +- Correct model: whether the market ever pays price P is governed by price DIFFUSION, not queue + depth. Model ln(price) as a driftless random walk with the ticker's own measured daily + log-return volatility sigma (deliberately no drift - same "don't model trend" rule as + `puga/parkcash.py`), and ask for the discounted value of the first time it diffuses up to P. + Closed form (standard Brownian-motion hitting-time Laplace transform): + y = ln(P/s0), rho = ln(1 + rate), D(P) = P * exp(-(y/sigma) * sqrt(2*rho)) for P > s0 + D(P) = P for P <= s0 + `rate` is the player's own REALIZED capital growth rate (opportunity cost of the delay - see + below), not a theoretical marginal-build ROI. +- ln(D) is LINEAR in y, so there is no interior optimum: D is strictly decreasing past the + current price (sell near top of book / at the last traded price wins) whenever + `sigma < sqrt(2*rho)`, and rises without bound (a degenerate corner case, not real free money) + otherwise. Checked against this account's real tickers (AL 2.5%/day, BHP 8.1%/day, DEC + 10.8%/day vol, all against a 5%/day rate -> threshold ~31%/day): every one is comfortably in + the "sell now dominates" regime. Treat a result where climbing pays as a signal to double + check sigma/rate inputs, not as license to post a speculative high ask. +- Implementation: `puga.history.daily_log_vol()` (pooled variance estimator, scales each + consecutive trading-day gap's squared log-return by its elapsed days so sparse tickers don't + understate how far price can move between trades) + `tools/sell.py --tiers --rate R`. +- This also resolved the earlier "what discount rate" question: use the player's REALIZED equity + growth rate (read off their own in-game equity-growth chart, ~5%/day as of 2026-10-10), not the + scan/build tools' marginal per-building ROI figures (14-29%/day) - those are a ceiling on one + marginal building at current scale (capped by permits, workforce-arrival delay, and market- + depth saturation the moment you scale), not what a freed-up dollar actually compounds at. The + realized rate already nets out all of that friction, which is exactly what a discount rate for + "should I wait for a better price" needs to reflect. + +## 2026-10-10 Black-Scholes closed form backtested wrong - empirical model instead +- Tested the driftless-random-walk assumption above against this account's real exchange history + (BHP: 802 trading days, sigma 7.39%/day; AL: 995 days, sigma 4.42%/day). Model predicts + P(touch level k*sigma*sqrt(T)) = 2*(1-Phi(k)), independent of T - the scale-invariance signature + of a true random walk. Measured the real empirical touch frequency for the same (k, T) pairs: + + | k | model pred | BHP empirical (T=7/14/30/60d) | AL empirical (T=7/14/30/60d) | + |-----|-----------:|----------------------------------|-----------------------------------| + | 0.5 | 61.7% | 16.6% / 18.0% / 17.5% / 20.1% | 32.6% / 33.9% / 32.4% / 40.1% | + | 1.0 | 31.7% | 8.8% / 8.0% / 8.2% / 7.9% | 9.2% / 10.2% / 10.8% / 16.7% | + | 1.5 | 13.4% | 4.1% / 3.8% / 0.1% / 0.0% | 3.9% / 3.9% / 4.4% / 2.7% | + | 2.0 | 4.6% | 2.0% / 1.2% / 0.0% / 0.0% | 1.9% / 1.5% / 2.0% / 0.0% | + | 3.0 | 0.3% | 0.5% / 0.0% / 0.0% / 0.0% | 0.7% / 0.5% / 0.0% / 0.0% | + + Model overstates touch probability by roughly 2-4x at low-moderate k, consistently across both + tickers. Scale-invariance partially holds at low k (flat across T, locally random-walk-like + short-term) but breaks down at higher k/longer T (empirical rate COLLAPSES faster than the model + past k=1.5, instead of staying flat) - the signature of mean reversion: real prices are anchored + to production cost/supply-demand equilibrium, not free-floating, so they wander less far over + longer horizons than a pure random walk would. +- The 2-4x figure overstates how much of this is mean reversion specifically: touches were + measured on daily VWAPs, a discrete sample of a continuous path, and the model's prediction + assumes continuous monitoring. Discrete sampling can only miss touches (an intraday spike that + reverts before the next VWAP looks like "no touch"), biasing the measured gap upward especially + at short horizons (k=0.5, T=7 - standard discrete-barrier corrections put this at roughly + 0.58*sigma*sqrt(step) per sampling interval, shrinking as the horizon grows). The k>=1.5/T>=30 + COLLAPSE is not explained by discretization (a roughly constant/shrinking effect, not a + progressive one) - that part is real mean reversion. +- Consequence: the model's absolute numbers are upper bounds, not honest estimates - directionally + conservative (it already said "don't climb the book," and the true probability of ever + profitably climbing is even lower than it claimed) but not quantitatively trustworthy. +- Replaced with `puga.history.empirical_touch()`: a non-parametric estimator measuring first- + passage directly from the ticker's own full multi-year history (every historical start day, + first time the price actually moved by the target amount, right-censored samples contribute 0 + rather than being excluded) - no assumed distribution, so whatever mean-reversion/fat-tails/etc. + the real data has shows up directly. Inherits the same daily-VWAP discretization conservatism as + the backtest above (a resting ask can fill on an intraday high the VWAP series never shows) - so + it also leans a little more toward "sell now" than fully warranted, not corrected for. Used by + `tools/sell.py --tiers` (combined with queue-depth as a deterministic shift of the hitting time, + not a max - ranking by the resulting hit_factor, not a point estimate of median days) and + `puga/parkcash.py` (two sequential legs: buy-dip, + sell-recovery). +- Refined further: the unconditional version above still averages over EVERY historical start day, + which answers "how often does this ticker move by y from an arbitrary day" - but mean reversion + specifically means the answer depends on where price is NOW relative to its own recent trend. + Conditioned on state instead (`empirical_touch(..., condition_on_state=True)`, the default): + only start days whose z-score against a trailing 60-day mean is within +/-0.5 of TODAY's are + used, falls back to every start day if there's not enough history to compute today's z-score or + the match band is empty. Stays non-parametric (no fitted mean-reversion speed/strength). Real + effect confirmed on DEC (`tools/parkcash.py`): round-trip wait dropped from 19 days + (unconditional) to 3 days once conditioned on DEC's actual current state - a big, real + difference, not a rounding adjustment. + +## 2026-10-10 Price-impact cap: real book depth, not a flat 25%-of-flow guess +- `puga/saturation.py`'s `S_SHARE=0.25` capped planned output at a flat 25% of a ticker's traded + flow, regardless of how deep or thin that specific market's standing bid book actually is - + never estimated how much price actually falls as supply rises, flagged by review. +- Tried first: a day-over-day regression of d(log price) on d(log volume) per ticker, the + textbook first-order elasticity estimate. Real data killed it: BHP -0.009, AL +0.004, + DEC +0.038, KV +0.007 - near zero and inconsistently signed. Cause: daily volume spikes in this + market are mostly DEMAND-driven (more buyers -> more volume AND flat/higher price), which + swamps and often reverses the sign of the SUPPLY-side effect we actually want. Shipping this + would have been worse than the flat guess (could tell you adding supply raises price). +- Fix: `puga.saturation.max_units_at_impact()` walks the LIVE bid book (same mechanics as + `puga.market.walk()`) and returns how many units you could sell before the realized price + degrades more than `--max-impact`%% (default 10) below vwap7/bid - a real, ticker-specific + measure, no regression, no confounding. +- Caveat, caught immediately when first wired as a hard `min()` cap alongside the flow-based + check: a live order book is a SINGLE SNAPSHOT - standing bids get added/withdrawn constantly, + same class of problem already found and fixed twice this session (`sell.py`'s queue model, + `parkcash.py`'s driftless walk), and we have no historical book-snapshot log to tell a + structurally-thin book from an unlucky instant. Using it as a hard cap let one noisy snapshot + silently zero out `n_lim` and hide a recipe entirely (happened to KV: flow-based n_lim 10.6 -> + 0.0 purely from the snapshot). The 30-day flow figure (`tref_precise`) is the more reliable + signal precisely because it aggregates many real days, not one instant. +- Final design: `n_lim`/filtering stays flow-only (unchanged, reliable). `max_units_at_impact()` + surfaces as an `[exit-thin]` WARNING flag instead (can the live book absorb an instant dump of 1 + day of 1 building's output right now, if cash were needed immediately?) - a panic-exit liquidity + check, NOT a production price-impact measure (that's `market`/`lim`, which price via patient + asks, the real selling mechanism) - visible, not silently filtering. Caught real signal on + several recipes (KV, SC, HER, TRN, NL, RG all flagged) without hiding any of them. + +## 2026-10-10 Demolition value: flat reclaim fraction, not a time-decay curve +- `tools/scan.py --deprec` assumed building value decays LINEARLY to 0 over ~60 days (a guess, + never checked against real data). Checked it: pulled this account's live FIO reclaimables + (`/rain/userplanetbuildingreclaimables/{user}`) for all 8 owned building types, normalized by + built-count, and compared to real construction cost (FIO `BuildingCosts` at live ask prices). +- Result: reclaim fraction is 50-86% (CHP 68.7, EXT 86.5, HB1 58.3, HB2 50.9, HWP 73.1, POL 78.0, + SME 74.6; CM a low outlier at 7.4%, likely a core-module-specific rule) - but EVERY one of these + buildings is at ~99.7-99.9% live condition (this account's base is young). If reclaim really + decayed from ~100% toward 0 over 60 days, a near-perfect-condition building should reclaim close + to 100%, not 50-86%. This contradicts the decay-curve premise and is more consistent with a + roughly FLAT, material-dependent reclaim fraction that doesn't move much with condition in the + normal operating range. +- Can't fully settle the shape - no aged/low-condition building exists in this account to check + the other end of the curve, so this is suggestive, not conclusive (single base, narrow condition + range, small sample). +- Fix: `scan.py --deprec` now amortizes `capex * (1 - --reclaim-frac)` (the UNRECLAIMED portion) + over the holding period, not full capex. `--reclaim-frac` defaults to 0.65 (the mean of the 7 + non-CM observations) - still a rough placeholder, not a calibrated curve; re-check once an aged + building exists to compare against. + ## 2026-10-09 Drop "ROI/day %" terminology - `profit/day ÷ new_capex` (what tools were calling "ROI/day X%") is just 1/payback_days rewritten as a percentage. It is NOT a compoundable rate of return - there's no reinvestment mechanism, and every opportunity here is capacity/market-limited (saturation), so it can't be extrapolated or annualized like an interest rate. - Report **payback in days** as the primary number everywhere. A "%/day" figure, if shown at all, must be labelled as simple/non-compounding and never compared to a real yield/interest rate. diff --git a/docs/library.md b/docs/library.md index 3cbed35..e7cfa1f 100644 --- a/docs/library.md +++ b/docs/library.md @@ -39,9 +39,15 @@ EOF per-tier need table), `HAB_CAP`/`HAB_AREA`, `TIERS`. This is what you use for "what if I understaff this building" or "what's the real wage bill for N heads" — the CLI tools only expose fixed slices of this. -- **`puga.saturation`** — `tref`, `n_out`, `p_patient`, `effective_supply`, `is_thin`: the - depth/persistence primitives behind `scan.py`. Reuse these instead of re-deriving a "how many - buildings can the market take" estimate by hand. +- **`puga.saturation`** — `tref` (cheap, Poisson-bounded, for stage-1 screening), `tref_precise` + (real daily-volume variance, for a bounded candidate set), `n_out` (flow x queue, NO demand + term - thin resting-bid books don't disqualify a market sold into via patient asks), `p_patient`, + `effective_supply`, `is_thin` (demand still checked here, independently), `max_units_at_impact` + (real live bid-book depth cap - how many units before realized price degrades past a threshold; + replaced a flat 25%-of-flow guess after a naive price-vs-volume elasticity regression on real + data came back near-zero/wrong-signed, see `docs/decisions.md`): the depth/persistence + primitives behind `scan.py`. Reuse these instead of re-deriving a "how many buildings can the + market take" estimate by hand. - **`puga.simulate`** — `simulate(plan, recipes, buildings, resources, fertility, price, ...)`: the PRUNplanner-simulator replica. Reuse this for anything plan-shaped rather than reimplementing efficiency/workforce/flows again. @@ -63,6 +69,25 @@ EOF (ship+from+to, auto-computes everything via `puga.route`), `tools/shipcost.py` (explicit jumps/sliders/gateway-fees, for when you already know the breakdown). Raw calibration data: `empire/data/shipping_calibration/`. +- **`puga.parkcash`** — `scan(cx="AI1", min_spread_pct=20.0, days=30, rate=0.05)`: every ticker + with a live quote and a big bid/ask spread, scored on real trading liquidity + (`puga.history.trading_days()`/`trade_gaps_days()`) and an EMPIRICAL round-trip estimate + (`puga.history.empirical_touch()`, two sequential legs: dip to bid, recover to ask) - no + random-walk assumption. CLI: `tools/parkcash.py`. See the module docstring for open gaps (no fee + data; the two legs are treated as independent, an approximation). +- **`puga.history`** — `daily_prices(tk, cx)`, `trading_days(tk, cx, days)` (real calendar-window + trading days, not "last N rows" - a sparse ticker needs this distinction), `trade_gaps_days(dates)`, + `full_series(tk, cx)` (all available history, ascending), `empirical_touch(series, y, rate)` + (non-parametric first-passage stats for a target log-move `y` - hit_factor/touched_frac/ + median_days, estimated directly from how often/fast the ticker's own real history has made + moves of that size, by default only drawing on historical days in a similar state to today + (trailing-60d z-score within +/-0.5, `condition_on_state=True`) since mean reversion's effect + depends on where price is now, not just the move size. Optional `horizon` (days, off by + default): bounds both eligibility and the search window to a fixed number of days so every + start-day sample gets the same fair window, instead of recent start days being biased by + running out of data to search. Replaced an earlier driftless-random-walk formula that + backtested wrong, see `docs/decisions.md`). Shared by `tools/history.py`, `tools/sell.py`, `tools/undervalued.py`, + `puga.parkcash`. - **`puga.contracts`** — `scan(planet, direction, sf_price, ff_price, v0=1.0, v1=1.0)`: live Local Market hauling ads (FIO `/localmarket/shipping/{source,destination}`), each enriched with real one-way fuel cost (`puga.shipping.leg_cost`) and margin. CLI: `tools/lmscan.py`. diff --git a/docs/roadmap.md b/docs/roadmap.md index 2349600..be239da 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,34 +1,46 @@ # Roadmap -Legend: [ ] todo, [x] done. Build order matters; each step is usable on its own. +Open work only - what's already built/fixed is discoverable from `CLAUDE.md`'s Tools list, +`docs/library.md`, `docs/decisions.md`, and module docstrings, not repeated here. -1. [x] Repo skeleton, git (master), venv, CLAUDE.md, mechanics from PRUNplanner. -2. [x] `puga/fio.py`: cached FIO client (TTL per endpoint, keys from `.env`), plus `puga/prunplanner.py` for `api.prunplanner.org/data/*`. Inspect what `exchanges/` adds (VWAP?) vs FIO `/exchange/all`. -3. [x] (price + book tools done; cxpc trend history not needed, PRUNplanner has vwap7/30) `puga/market.py` + `tools/price.py`, `tools/book.py`: prices across all CX, trends from cxpc, order-book walk for real fill price at quantity N. -4. [x] `puga/econ.py` (22 tests pass, expected values from PRUNplanner's suite): efficiency stack, workforce consumption, hab LP, extraction (port the formulas from `docs/mechanics.md`, with tests against PRUNplanner values). -5. [x] (v1 done; see saturation-design.md for refinements) `tools/scan.py` (replaces prun_scan.py): depth-aware ROI. Per recipe report: buildings the market can absorb (output demand + traded volume, input supply), ROI at fill price for N buildings, absorbable profit/day. Filter `--min-buildings`. Both CX-local and universe. -5b. [ ] `tools/arb.py` (replaces prun_cxarb.py): inter-CX arbitrage on `market.walk` at real fill size; capital required, fuel/shipping cost from state, profit per trip within cash. Reason: old script uses top-of-book spread x min(supply,demand); on 2 of 4 top hits it overstated profit 7 to 11x (WAI, SCN). -6. [x] `tools/chain.py`: full bill of materials, make-vs-buy per node. -7. [x] `tools/whatif.py`: marginal ROI of adding a building to a base you already run (whole-plan diff via `simulate()`, built dict from `state/company.yaml`; flags newly-caused workforce overloads). -8. [x] `tools/planet.py`: rank planets for a resource (FIO bulk `Resources`/`BuildRequirements` data), founding cost from real environment-extras materials, jump distance via `tools/route.py`. `tools/found.py` (full founding workflow/writeback) still todo. -13b. [x] `puga/network.py` + `tools/network.py`: multi-base chain modelling (per-base simulate, net transfers against real freight, `cm_free` for kits). Worked example: `plans/chains/nike_deimos_lst.yaml` (Nike LST -> Deimos PP2/BSE). Known gap: reports whole-plan profit per base, not the marginal addition; caller must diff manually. -9. [x] `tools/route.py`: jump graph (FIO `/systemstars`, Dijkstra by straight-line ly, jump count alongside). `tools/haul.py` (fuel cost, profit per trip) still todo. -9c. [x] `puga/shipping.py` + `tools/shipcost.py` (explicit params) + `tools/shiptrip.py` (just ship/from/to) + `puga.shipping.round_trip_cost()` wired into `tools/network.py`'s freight cost: real ship flight fuel/cost model (STL cruise + FTL jump/charge, linear in the fuel-usage/reactor-usage sliders, additive/no cross-term; gateway hops free-fuel + manual LOCK fee read off the live game, not lookupable via any FIO/PRUNplanner endpoint - checked; landing fuel a measured per-planet lookup). Calibrated 2026-10-09 on this account's one hull/reactor (blueprint BP-STRT-0000, shared by both ships AVI-07JFB/AVI-07JFC, confirmed identical specs via FIO) via a controlled 4x4-slider-grid sweep across 5 real destinations, read from the live APEX Ship Flight Control DOM. Raw data: `empire/data/shipping_calibration/`. Open gaps (see `puga/shipping.py` docstring): departure-side gravity dependence untested (every calibration DEP left the same origin), landing fuel vs gravity/distance confounded (n=3), no second hull/reactor to cross-check generalization. `tools/scan.py --trip-cost` still uses the flat 9250 AIC placeholder (not yet wired to the real model - needs a per-recipe destination planet, not just a CX code). -9d. [x] `puga/contracts.py` + `tools/lmscan.py`: live Local Market hauling-ad scan (FIO `/localmarket/shipping/{source,destination}`, public/no-auth - real player-posted freight jobs, not the CONTD player-contract system) ranked by margin after real fuel cost (`puga.shipping.leg_cost`, one-way, with a small measured gateway-fee table `KNOWN_GATEWAY_ROUTES` for ANT<->HRT/BEN). Ship cargo hold confirmed 500t/500m3 via FIO for both ships. -9b. [x] (dry-run default, --apply after user yes) `tools/plan_push.py`: create/update `[PuGa]` plans in PRUNplanner from YAML (Api-Key auth; see decisions.md for guardrails). -10. [x] `tools/state.py sync` (sites, production efficiency, storage, ships, cash, permits): inventory, production, ships into `state/`. -11. [ ] Docs to fill: `docs/apex.md`, `docs/fio.md`, `docs/playbook.md` (question type -> tool), `docs/decisions.md`. +## Todo +- [ ] `tools/arb.py`: inter-CX arbitrage at real fill size (replaces `prun_cxarb.py`, which + overstated profit 7-11x on 2/4 top hits). +- [ ] `tools/haul.py`: fuel cost / profit per trip (route.py has the jump graph). +- [ ] `tools/portfolio.py`: profit-maximizing set of 1-building opportunities under shared + housing/capex/permits. Ties into backlog #2 (profit/permit ranking, needs this tool to mean + anything - permits bind at the whole-base level, not per-recipe). +- [ ] Docs to fill: `docs/apex.md`, `docs/fio.md`, `docs/playbook.md`. -13. [x] `tools/buy.py`: cash-aware shopping list (construction gap vs empire state's built count, plus N days of net operating stock via simulate(), priced at real fill cost, checked against cash/reserve). -14. [x] `tools/sell.py`: ask-post-price + expected-clear-time advisor from 30d traded-volume percentiles, vs hitting the bids now. - -15. [x] `tools/runway.py`: days-of-runway per material from simulate()'s net consumption, base storage plus auto-detected inbound ship cargo (tools/state.py now tags each ship with its live flight origin/destination, and each STORE with its planet, needed once a 2nd base existed). - -12. [ ] `tools/portfolio.py`: choose the set of 1-building opportunities for one base that maximises total profit given shared whole-building housing (HB1/HB2/...), capex, permits and (for demolish-later) 60-day value decay. Persistence check exists (`tools/persistence.py`); wire it into the ranking. -14. [x] `tools/undervalued.py`: trading (not building) strategy - manufactured goods priced below implied production cost, candidates for a buy-and-hold-for-reversion bet. Backtested 2026-10-09 (326 output tickers, 287 deduplicated dip-onset episodes, 112 distinct tickers): +3mo forward return mean +21.3%/median +8.9%/73.2% hit rate at >=10 units/day volume - roughly 0.10-0.24%/day, far below the 14-29%/day marginal ROI on production builds found this session. Only worth excess idle capital, not a primary strategy. +## Backlog: review findings, still open (spot-checked against code) +1. [ ] `scan.py` payback ignores working capital (input stock/output inventory/consumables) - can + rival building capex for smelting-type recipes. `buy.py` already computes this number. +2. [ ] No profit/permit ranking (permits bind at the base level, not per-recipe) - needs + `tools/portfolio.py`'s cross-recipe/whole-base view. +3. [ ] `scan.py` doesn't charge degradation (capex/180/day) like `simulate.py` does - small but real + inconsistency between tools ranking the same thing. +4. [ ] No input-side flow cap (buying a big share of an input's flow should price-impact like + selling output does) - documented as absent, not modelled. +5. [ ] `persistence.py --lag 3` default is too low given `docs/mechanics.md`'s weekly workforce + distribution - real lag is 3-10 days. +6. [ ] Possible forward-fill bias if any tool reindexes `daily_prices()` onto a full calendar grid + (it only emits trade days) - would understate volatility for sparse tickers. Not yet confirmed + whether `persistence.py`/`undervalued.py` actually do this. +7. [ ] `undervalued.py`'s backtest (+21.3%/3mo) has no control group - meaningless without the + universe's unconditional return over the same window. +8. [ ] `persistence.py` measures past margin variance of a market we were NOT in - can't see the + dominant real risk (our own entry + the next player's compressing margin). +9. [ ] `undervalued.py::manufacturing_candidates()` keeps the first recipe seen per output, not + the cheapest (`seen.setdefault`). +10. [ ] `scan.py` hab cost ignores combined habs (HBB/HBC/HBM/HBL), which can be cheaper per head. +11. [ ] `scan.py --trip-cost` still defaults to the flat 9250 placeholder despite the real + `puga/shipping.py` model existing and already used by `whatif.py`. +12. [ ] `sell.py`'s `s0` is the day's VWAP, not its high - `--tiers` touch probability is slightly + pessimistic (same discretization conservatism as `empirical_touch()`, see `docs/decisions.md`). +13. [ ] `empirical_touch()`'s optional `horizon` (fixed-window right-censoring) exists but isn't + wired into any tool's default call yet. ## Open questions -- PRUNplanner auth is known (`Authorization: Api-Key `). FIO REST and FIO API are separate services with separate keys (separate services with separate keys; an earlier "one API" reading was wrong). Keys are in `.env`. Verified 2026-09-18: FIO_REST_KEY works on rest.fnar.net as `Authorization: ` (own /sites, /storage etc. readable); FIO_API_KEY gives 401 there, host unknown. PRUNplanner key works (plans 0, empires 1, cx prefs 1 on his account). Still unknown: the OpenAPI spec at https://doc.fnar.net/api.json has no securitySchemes. Probe with read-only calls. -- (resolved) PRUNplanner `exchanges/` has vwap_daily/7d/30d and traded volume; merged into `puga/market.py`. Trend history via cxpc still todo. -- Extraction: concentration to daily rate formula. -- (resolved 2026-09-25) Real price-band filtering (`NarrowPriceBandLow/High`, `WidePriceBandLow/High`) implemented in `puga/saturation.py` (`in_band`, `effective_demand`, `effective_supply` band param) and wired into `tools/scan.py` stage 2. Was flagged as an unimplemented refinement in saturation-design.md since day one; the player noticed a stale out-of-band bid in a live screenshot and asked the right question. +- FIO REST and FIO API are separate services/keys (`.env`). FIO_REST_KEY works on rest.fnar.net; + FIO_API_KEY's host is unknown; PRUNplanner's Api-Key works. `doc.fnar.net/api.json` has no + securitySchemes.