dodox
d97aac0e17
Backend: receipt upload -> LLM vision extraction (OpenAI-compatible, provider-agnostic), item review/edit, per-group splitting against Cospend projects/members, highlight+upload via WebDAV, public share link, bill creation via Cospend's OCS API (verified against real source, not just doc summaries). Frontend: capture -> review -> group -> summary flow as an installable PWA. install.sh / run.sh (venv + npm, tmux session) instead of Docker, per the ~/Projects/gain pattern.
118 lines
6.3 KiB
Markdown
118 lines
6.3 KiB
Markdown
# wgBill — plan
|
||
|
||
Webapp to replace the manual "upload to NC → share link → paste into Cospend
|
||
bill title" flow with: photo → extracted items → pick which ones (per
|
||
splitting group) → highlighted photo → uploaded + shared → Cospend bill(s)
|
||
created automatically.
|
||
|
||
## Confirmed via API research
|
||
|
||
- **WebDAV** (`PUT /remote.php/dav/files/<user>/<path>`) — uploads the
|
||
receipt/highlighted image. Auth via NC app password (Basic auth).
|
||
- **OCS Share API** (`POST /ocs/v2.php/apps/files_sharing/api/v1/shares`,
|
||
`shareType=3`) — creates the public link, response includes ready `url`.
|
||
- **Cospend API** — verified against the actual source (`appinfo/routes.php`
|
||
+ `lib/Controller/ApiController.php` + `lib/Db/{Project,Member}.php` in
|
||
`julien-nc/cospend-nc`), not just doc summaries:
|
||
- It's an OCS API, same family as the Share API — full paths are under
|
||
`/ocs/v2.php/apps/cospend/api/v1/...`, need `OCS-APIRequest: true`.
|
||
- `GET /projects` → `getLocalProjects` — lists the authenticated user's
|
||
projects (`{id, name, ...}`). No hardcoded project id needed; the app
|
||
lets you pick.
|
||
- `GET /projects/{id}/members` → `{id, name, ...}` (member `id` is
|
||
numeric).
|
||
- `POST /projects/{id}/bills` → `createBill(projectId, date, what, payer,
|
||
payedFor, amount, ..., comment, ...)` — **camelCase** params (not the
|
||
old IHateMoney-style snake_case an earlier doc summary suggested);
|
||
`payer`/`payedFor` are numeric member ids, `payedFor` is a
|
||
comma-separated string of them. Returns just the new bill's **id**
|
||
(int), not a bill object.
|
||
**No native attachment/link field on a bill.** Confirms the current manual
|
||
trick (put the link in the bill title) is the only integration point — the
|
||
app does the same. Cospend's UI only recognizes a bill as "has an
|
||
attachment" when the link is in `what` (the title), not `comment` — a link
|
||
in `comment` is just plain text. So: link goes in `what`, itemized
|
||
breakdown goes in `comment`.
|
||
- CORS ruled out doing this as a pure static frontend talking to NC directly
|
||
from a different origin (WebDAV PUT and Login Flow v2 both fight the
|
||
browser here) — decided to just build a small backend instead.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
[PWA frontend] --(HTTPS, JSON)--> [backend] --(WebDAV / OCS / Cospend API)--> [Nextcloud]
|
||
|
|
||
+--(OpenAI-compatible chat completions)--> [vision LLM]
|
||
```
|
||
|
||
- **Backend**: Python + Flask + SQLite. Holds NC app password and the
|
||
vision-provider API key as server config (env vars), never exposed to the
|
||
client. SQLite holds the per-receipt session (parsed items, image path,
|
||
group selections) — durable across a backend restart, unlike an in-memory
|
||
store, and trivial to inspect/debug directly.
|
||
- **Frontend**: Vite + React + TypeScript, PWA (installable, camera access
|
||
via `<input capture>` / `getUserMedia`).
|
||
- **Vision extraction**: any OpenAI-compatible endpoint. Config = `base_url`
|
||
+ `api_key` + `model`. Works unmodified for OpenAI; for Gemini point
|
||
`base_url` at `https://generativelanguage.googleapis.com/v1beta/openai/`.
|
||
Switch/compare providers via env var, no code change.
|
||
|
||
## Data flow
|
||
|
||
1. **Capture** — frontend takes/picks a photo, uploads to backend.
|
||
2. **Extract** — backend sends image to the vision LLM with a prompt asking
|
||
for strict JSON: `{ items: [{ id, label, price, bbox: [x,y,w,h] }] }`
|
||
(bbox in normalized 0–1 coords, for the highlight step later). Backend
|
||
stores the original image on disk and the parsed items in a SQLite
|
||
`receipts` table, keyed by a random session id; a periodic cleanup (or
|
||
just TTL-on-read) drops old sessions/images.
|
||
3. **Review** — frontend shows the parsed list, editable (OCR/LLM extraction
|
||
won't be perfect — fix a mis-read price, merge a split line, add a missed
|
||
item, adjust a bbox by dragging).
|
||
4. **Group & select** — user selects items into one or more groups; each
|
||
group gets its own "split with" member list (loaded from the relevant
|
||
Cospend project's members) and payer. Same UI, run N times per receipt.
|
||
5. **Highlight** — backend (or client canvas) draws boxes over selected
|
||
items' bboxes on the original image, one output image per group.
|
||
6. **Per group**, backend:
|
||
- `PUT` highlighted image via WebDAV to a configured folder.
|
||
- Creates a public share link via OCS Share API.
|
||
- `createBill` in Cospend: amount = sum of group's item prices, payer,
|
||
payedFor = group's members, title (`what`) includes the share link
|
||
(so Cospend's UI shows it as an attachment), comment = itemized list.
|
||
7. Frontend shows confirmation + links to the created bill(s).
|
||
|
||
## Config (env vars)
|
||
|
||
- `NC_BASE_URL`, `NC_USERNAME`, `NC_APP_PASSWORD`, `NC_UPLOAD_FOLDER`
|
||
- `COSPEND_PROJECT_ID` (or a picker if you use more than one project)
|
||
- `LLM_BASE_URL`, `LLM_API_KEY`, `LLM_MODEL`
|
||
- Single-user/household tool — one shared backend config, not per-user
|
||
login. If later you want it multi-tenant (each housemate logging in as
|
||
themselves), that's a bigger change (real NC OAuth/session per user) —
|
||
intentionally deferred, not needed for v1.
|
||
|
||
## Open items to confirm before/while building
|
||
|
||
- Cospend project/member selection is handled live via the API (project
|
||
picker + member list fetched per request), not hardcoded — resolved.
|
||
- Where the backend actually runs — decided: no Docker for now, plain
|
||
`install.sh`/`run.sh` (venv + npm, tmux session with a backend/frontend
|
||
window each), matching the pattern used in ~/Projects/gain.
|
||
- Bbox-based highlighting depends on the vision model returning usable
|
||
coordinates — some models are much better at this than others; may need a
|
||
fallback (e.g. user manually drags a highlight box) if a given provider's
|
||
bboxes are unreliable. Worth testing with your candidate receipts early.
|
||
|
||
## Build order
|
||
|
||
1. Scaffold backend (Flask + SQLite) with the three NC/Cospend calls wired
|
||
to a real test project — verify the WebDAV→share→bill chain works end to
|
||
end with a hardcoded fake bill first, before touching OCR.
|
||
2. Add the LLM extraction endpoint, test against a few real receipt photos,
|
||
compare providers.
|
||
3. Scaffold frontend (Vite/React/PWA), wire capture → review → group →
|
||
submit against the backend.
|
||
4. Highlighting (canvas draw on selected bboxes).
|
||
5. Polish: error handling for OCR misses, mobile camera UX, PWA install.
|