wgBill/PLAN.md

wgBill — plan

Webapp to replace the manual "upload to NC → share link → paste into Cospend bill title" flow with: photo → extracted items → pick which ones (per splitting group) → highlighted photo → uploaded + shared → Cospend bill(s) created automatically.

Confirmed via API research

• WebDAV (PUT /remote.php/dav/files/<user>/<path>) — uploads the receipt/highlighted image. Auth via NC app password (Basic auth).
• OCS Share API (POST /ocs/v2.php/apps/files_sharing/api/v1/shares, shareType=3) — creates the public link, response includes ready url.
• Cospend API — verified against the actual source (appinfo/routes.php
  • lib/Controller/ApiController.php + lib/Db/{Project,Member}.php in julien-nc/cospend-nc), not just doc summaries:
  • It's an OCS API, same family as the Share API — full paths are under /ocs/v2.php/apps/cospend/api/v1/..., need OCS-APIRequest: true.
  • GET /projects → getLocalProjects — lists the authenticated user's projects ({id, name, ...}). No hardcoded project id needed; the app lets you pick.
  • GET /projects/{id}/members → {id, name, ...} (member id is numeric).
  • POST /projects/{id}/bills → createBill(projectId, date, what, payer, payedFor, amount, ..., comment, ...) — camelCase params (not the old IHateMoney-style snake_case an earlier doc summary suggested); payer/payedFor are numeric member ids, payedFor is a comma-separated string of them. Returns just the new bill's id (int), not a bill object. No native attachment/link field on a bill. Confirms the current manual trick (put the link in the bill title) is the only integration point — the app does the same. Cospend's UI only recognizes a bill as "has an attachment" when the link is in what (the title), not comment — a link in comment is just plain text. So: link goes in what, itemized breakdown goes in comment.
• CORS ruled out doing this as a pure static frontend talking to NC directly from a different origin (WebDAV PUT and Login Flow v2 both fight the browser here) — decided to just build a small backend instead.

Architecture

[PWA frontend]  --(HTTPS, JSON)-->  [backend]  --(WebDAV / OCS / Cospend API)-->  [Nextcloud]
                                        |
                                        +--(OpenAI-compatible chat completions)--> [vision LLM]

• Backend: Python + Flask + SQLite. Holds NC app password and the vision-provider API key as server config (env vars), never exposed to the client. SQLite holds the per-receipt session (parsed items, image path, group selections) — durable across a backend restart, unlike an in-memory store, and trivial to inspect/debug directly.
• Frontend: Vite + React + TypeScript, PWA (installable, camera access via <input capture> / getUserMedia).
• Vision extraction: any OpenAI-compatible endpoint. Config = base_url
  • api_key + model. Works unmodified for OpenAI; for Gemini point base_url at https://generativelanguage.googleapis.com/v1beta/openai/. Switch/compare providers via env var, no code change.

Data flow

1. Capture — frontend takes/picks a photo, uploads to backend.
2. Extract — backend sends image to the vision LLM with a prompt asking for strict JSON: { items: [{ id, label, price, bbox: [x,y,w,h] }] } (bbox in normalized 0–1 coords, for the highlight step later). Backend stores the original image on disk and the parsed items in a SQLite receipts table, keyed by a random session id; a periodic cleanup (or just TTL-on-read) drops old sessions/images.
3. Review — frontend shows the parsed list, editable (OCR/LLM extraction won't be perfect — fix a mis-read price, merge a split line, add a missed item, adjust a bbox by dragging).
4. Group & select — user selects items into one or more groups; each group gets its own "split with" member list (loaded from the relevant Cospend project's members) and payer. Same UI, run N times per receipt.
5. Highlight — backend (or client canvas) draws boxes over selected items' bboxes on the original image, one output image per group.
6. Per group, backend:
   • PUT highlighted image via WebDAV to a configured folder.
   • Creates a public share link via OCS Share API.
   • createBill in Cospend: amount = sum of group's item prices, payer, payedFor = group's members, title (what) includes the share link (so Cospend's UI shows it as an attachment), comment = itemized list.
7. Frontend shows confirmation + links to the created bill(s).

Config (env vars)

• NC_BASE_URL, NC_USERNAME, NC_APP_PASSWORD, NC_UPLOAD_FOLDER
• COSPEND_PROJECT_ID (or a picker if you use more than one project)
• LLM_BASE_URL, LLM_API_KEY, LLM_MODEL
• Single-user/household tool — one shared backend config, not per-user login. If later you want it multi-tenant (each housemate logging in as themselves), that's a bigger change (real NC OAuth/session per user) — intentionally deferred, not needed for v1.

Open items to confirm before/while building

• Cospend project/member selection is handled live via the API (project picker + member list fetched per request), not hardcoded — resolved.
• Where the backend actually runs — decided: no Docker for now, plain install.sh/run.sh (venv + npm, tmux session with a backend/frontend window each), matching the pattern used in ~/Projects/gain.
• Bbox-based highlighting depends on the vision model returning usable coordinates — some models are much better at this than others; may need a fallback (e.g. user manually drags a highlight box) if a given provider's bboxes are unreliable. Worth testing with your candidate receipts early.

Build order

1. Scaffold backend (Flask + SQLite) with the three NC/Cospend calls wired to a real test project — verify the WebDAV→share→bill chain works end to end with a hardcoded fake bill first, before touching OCR.
2. Add the LLM extraction endpoint, test against a few real receipt photos, compare providers.
3. Scaffold frontend (Vite/React/PWA), wire capture → review → group → submit against the backend.
4. Highlighting (canvas draw on selected bboxes).
5. Polish: error handling for OCR misses, mobile camera UX, PWA install.