feat: SQLAlchemy ORM models, filing cache incremental fetch, yfinance price cache
- Replace db/schema.sql + raw sqlite3 with SQLAlchemy ORM (db/models.py) - Filing, Signal, PriceCache models with proper indexes - db/db.py uses SQLAlchemy sessions throughout; no raw SQL strings - Add PriceCache table: stores daily close prices per ticker - backtest._fetch_prices checks DB first; skips yfinance for completed ranges - New data persisted via upsert_prices() - get_cached_prices() / upsert_prices() added to db.py - EDGAR poller incremental fetch: get_latest_filed_date() returns newest filed_date in DB; fetch_and_store_new_filings skips entries older than that cutoff before even checking accession_exists - Add get_signals_for_backtest() to db.py; backtest no longer opens its own sqlite3 connection - requirements.txt: add sqlalchemy>=2.0.0 Co-authored-by: dodox <dodox@users.noreply.local>
This commit is contained in:
@@ -7,7 +7,7 @@ from lxml import etree, html
|
||||
|
||||
import config
|
||||
from ingestion.form4_parser import parse_form4
|
||||
from db.db import insert_filing, accession_exists
|
||||
from db.db import accession_exists, get_latest_filed_date, insert_filing
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -82,7 +82,13 @@ def fetch_and_store_new_filings() -> list[dict]:
|
||||
logger.error(f"Failed to fetch EDGAR index: {e}")
|
||||
return new_filings
|
||||
|
||||
latest_in_db = get_latest_filed_date()
|
||||
|
||||
for _index_url, accession, filed_date in entries:
|
||||
if latest_in_db and filed_date and filed_date < latest_in_db:
|
||||
logger.debug(f"Skipping {accession}: filed_date {filed_date} older than latest in DB {latest_in_db}")
|
||||
continue
|
||||
|
||||
if accession_exists(accession):
|
||||
continue
|
||||
|
||||
|
||||
Reference in New Issue
Block a user