phase-1: optimize code

2026-06-04 01:36:28 -04:00
parent 225b88ea4f
commit 5a4b4aa6d1
89 changed files with 11189 additions and 845 deletions
@@ -0,0 +1,464 @@
+# CLAUDE.md
+
+Guidance for working in this repository.
+
+## Overview
+
+`market-screener` is a Node.js project with two modes:
+
+1. **CLI** — screens stocks, ETFs, and bonds via `npm start`, generates HTML reports
+2. **Fastify API server** — powers the SvelteKit dashboard in the `ui/` subdirectory
+
+Every asset is scored under two lenses:
+
+- **Market-Adjusted** — gates derived from live Yahoo benchmarks (SPY P/E, XLK P/E, XLRE yield, LQD spread). Reflects what is acceptable in today's market.
+- **Fundamental** — strict Graham/value-investing gates from `ScoringConfig`. Reflects genuine value regardless of market conditions.
+
+The comparison produces a **Signal** (Strong Buy / Momentum / Speculation / Neutral / Avoid).
+
+ES module project (`"type": "module"`); use `import`/`export`, not `require`.
+
+---
+
+## Commands
+
+```bash
+npm install                                   # install dependencies
+npm run dev                                   # start API server (port 3000) + SvelteKit UI (port 5173) together
+npm run server                                # API server only (port 3000)
+npm start                                     # CLI: Yahoo news → catalyst tickers → screener-report.html
+npm start -- watch                            # CLI: default watchlist
+npm start -- AAPL MSFT VOO                    # CLI: specific tickers
+npm run finance                               # CLI: portfolio advice + SimpleFIN → finance-report.html
+npm test                                      # run all unit tests (node:test, zero external deps)
+npm run test:watch                            # watch mode — uses verbose spec reporter
+npm run format                                # format all src/bin/tests with Prettier
+npm run format:check                          # check formatting without writing (used in CI/pre-commit)
+npm run ui:install                            # install UI dependencies (ui/ subdirectory)
+```
+
+`npm run dev` runs both the API server and the SvelteKit UI (in `ui/`) concurrently. Run `npm run ui:install` once before first use.
+
+---
+
+## Project Structure
+
+```
+bin/
+  screen.js              ← CLI screener entry point
+  finance.js             ← CLI personal finance entry point
+  import-portfolio.js    ← broker CSV importer
+  server.js              ← Fastify API server entry point
+
+scripts/
+  summary-reporter.js    ← custom node:test reporter (silent on pass, summary line at end)
+
+prompts/
+  catalyst-analysis.md   ← daily catalyst analysis playbook (LLM prompt + workflow)
+
+src/
+  config/
+    ScoringConfig.js     ← CREDIT_RATING_SCALE + ScoringRules (single source of truth)
+    constants.js         ← SIGNAL, ASSET_TYPE, SECTOR, SCORE_MODE, REGIME, SIGNAL_ORDER
+
+  market/                ← Yahoo Finance data layer
+    YahooClient.js       ← wraps yahoo-finance2 v3, retry + backoff
+    BenchmarkProvider.js ← fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD → marketContext
+    MarketRegime.js      ← derives INFLATED gate overrides from live benchmarks + rate regime
+
+  screener/              ← core screening domain
+    ScreenerEngine.js    ← orchestrates: fetch → score × 2. Methods: screenTickers() (pure data),
+                           screenWithProgress() (CLI with stdout). Accepts { logger } option.
+    DataMapper.js        ← normalises Yahoo payload → flat asset data object
+                           NOTE: uses trailingPE (not forwardPE). Preserves negative FCF.
+                           Infers bond duration from category string. Maps ETF volume.
+    RuleMerger.js        ← merges base rules + sector overrides + MarketRegime (INFLATED mode)
+    Chunker.js           ← splits ticker list into batches
+    assets/
+      Asset.js           ← abstract base: ticker, currentPrice, type, formatting helpers
+      Stock.js           ← metrics + _mapToStandardSector (8 sectors detected)
+      Etf.js             ← metrics: expenseRatio, yield, volume, fiveYearReturn, totalAssets
+      Bond.js            ← metrics: ytm, duration, creditRating, creditRatingNumeric
+    scorers/
+      StockScorer.js     ← gate checks + weighted registry (ROE, opMargin, margin, peg, rev, fcf)
+      EtfScorer.js       ← expense gate + registry (cost, yield, volume, fiveYearReturn)
+      BondScorer.js      ← credit gate + spread/duration scoring
+
+  analyst/
+    CatalystAnalyst.js   ← fetches Yahoo Finance news, extracts relatedTickers. Accepts { logger }.
+    LLMAnalyst.js        ← uses Claude Haiku (ANTHROPIC_API_KEY) to analyze headlines → summary,
+                           sentiment (BULLISH/NEUTRAL/BEARISH), affectedIndustries, relatedTickers.
+                           Returns null gracefully if API key is not set. Accepts { logger }.
+
+  calls/
+    MarketCallStore.js   ← persists quarterly market thesis entries to market-calls.json.
+                           Each call stores: title, quarter, date, thesis, tickers[], snapshot{}
+                           (price + signal per ticker at creation time). CRUD: list/get/create/delete.
+
+  finance/
+    clients/
+      SimpleFINClient.js ← claims setup token → access URL, fetches /accounts via Basic Auth header
+                           (NOT embedded credentials in URL). Accepts { logger, onAccessUrlClaimed }.
+    PersonalFinanceAnalyzer.js ← net worth, cash vs investments, spending by category
+    PortfolioAdvisor.js  ← cross-references holdings with screener signals → hold/sell/add advice
+
+  reporters/
+    HtmlReporter.js      ← render() → HTML string (server), generate() → writes file (CLI)
+    FinanceReporter.js   ← render() → HTML string (server), generate() → writes file (CLI)
+
+  server/
+    app.js               ← Fastify app factory (buildApp). Registers CORS + routes.
+    routes/
+      screener.js        ← POST /api/screen, GET /api/screen/catalysts
+                           Serializes asset.getDisplayMetrics() before JSON response.
+      finance.js         ← GET /api/finance/portfolio, GET /api/finance/market-context
+      calls.js           ← CRUD for market calls + GET /api/calls/calendar (earnings/dividend events)
+
+ui/                      ← SvelteKit dashboard (lives inside this repo, not a separate repo)
+  src/routes/
+    +page.svelte         ← main screener UI
+    calls/               ← market calls list + detail views
+    portfolio/           ← portfolio advice view
+    safe-buys/           ← filtered strong-buy view
+
+market-calls.json        ← persisted market thesis calls (written by MarketCallStore)
+portfolio.json           ← user's holdings: ticker, shares, costBasis, source, type
+.env                     ← SIMPLEFIN_ACCESS_URL or SIMPLEFIN_SETUP_TOKEN, ANTHROPIC_API_KEY
+```
+
+---
+
+## Data Flow
+
+```
+Yahoo Finance API
+  ↓
+BenchmarkProvider      — fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD
+                         builds marketContext { sp500Price, riskFreeRate, vixLevel,
+                         rateRegime, volatilityRegime, benchmarks { marketPE, techPE, reitYield, igSpread } }
+  ↓
+DataMapper             — normalises raw Yahoo payload → flat data object with type (STOCK/ETF/BOND)
+                         uses trailingPE as primary; preserves negative FCF yield; infers bond duration
+  ↓
+Asset subclass         — Stock / Etf / Bond holds metrics + getDisplayMetrics()
+  ↓
+RuleMerger × 2        — FUNDAMENTAL mode: ScoringConfig as-is (Graham-style)
+                         INFLATED mode:   sector override + MarketRegime live gate overrides
+  ↓
+Scorer × 2            — StockScorer / EtfScorer / BondScorer, fully stateless
+  ↓
+ScreenerEngine         — derives Signal from comparing both verdicts
+  ↓
+  ├── CLI path:   screenWithProgress() → HtmlReporter.generate() → screener-report.html
+  └── API path:   screenTickers() → JSON (with serialized displayMetrics) → SvelteKit UI
+```
+
+---
+
+## API Routes (Fastify)
+
+| Method | Path | Description |
+|---|---|---|
+| GET | `/health` | Health check |
+| POST | `/api/screen` | Screen tickers. Body: `{ tickers: string[] }`. Returns `{ STOCK, ETF, BOND, ERROR, marketContext }` with `asset.displayMetrics` pre-serialized |
+| GET | `/api/screen/catalysts` | Yahoo news → `{ tickers, stories }` |
+| GET | `/api/finance/portfolio` | Portfolio advice + optional SimpleFIN data |
+| GET | `/api/finance/market-context` | Live benchmark data only |
+| GET | `/api/calls` | List all market calls (newest first) |
+| GET | `/api/calls/:id` | Get one call + re-screened current prices for comparison |
+| POST | `/api/analyze` | Fetch Yahoo news for specific tickers + run LLM analysis. Body: `{ tickers: string[] }`. Returns `{ analysis }` |
+| POST | `/api/calls` | Create a market call; snapshots current prices. Body: `{ title, quarter, thesis, tickers[], date? }` |
+| DELETE | `/api/calls/:id` | Delete a market call |
+| GET | `/api/calls/calendar` | Earnings + dividend calendar. Query: `?tickers=AAPL,MSFT` (omit for all call tickers) |
+
+CORS is configured for `CLIENT_ORIGIN` env var (default `http://localhost:5173`).
+
+---
+
+## Scoring Modes
+
+| Mode | P/E Gate (general) | P/E Gate (tech) | Source |
+|---|---|---|---|
+| FUNDAMENTAL | 15x | 35x | ScoringConfig (true Graham) |
+| INFLATED | S&P 500 P/E × 1.5 | XLK P/E × 1.3 | Live SPY/XLK data |
+
+**Rate regime effect on INFLATED mode:**
+- HIGH rate regime: P/E multiplier compresses to 1.2× (vs 1.5× in NORMAL)
+- HIGH rate regime: REIT yield floor tightens (0.95× vs 0.85×)
+- HIGH rate regime: bond spread demand increases (0.90× vs 0.80×)
+
+| Signal | Meaning |
+|---|---|
+| ✅ Strong Buy | Passes both fundamental AND inflated gates |
+| ⚡ Momentum | Passes inflated, holds fundamentally |
+| ⚠️ Speculation | Passes inflated, fails fundamental |
+| 🔄 Neutral | Hold territory in one or both lenses |
+| ❌ Avoid | Fails both |
+
+---
+
+## ScoringConfig Key Values
+
+`src/config/ScoringConfig.js` — single source of truth for all gates, weights, thresholds.
+
+**STOCK base gates (Fundamental mode):**
+- `maxPERatio: 15` — Graham's actual rule (trailing P/E)
+- `maxPegGate: 1.0` — Lynch standard: PEG > 1.0 means paying full price
+- `maxDebtToEquity: 1.5` — most distress starts above 2x
+- `minQuickRatio: 0.8` — below this signals real liquidity stress
+
+**Sector overrides** (structural — apply in both modes):
+
+| Sector | Key difference |
+|---|---|
+| TECHNOLOGY | D/E up to 2.0, P/E up to 35x, FCF weight raised |
+| REIT | P/E and PEG disabled (9999), scored on yield + P/FFO |
+| FINANCIAL | D/E disabled, scored on ROE + P/B, maxPriceToBook 1.5x |
+| ENERGY | FCF weight 4, yield weight 3, opMargin primary |
+| HEALTHCARE | Revenue growth primary, P/E up to 25x |
+| COMMUNICATION | FCF weight 4, P/E up to 25x (META, GOOGL, NFLX) |
+| CONSUMER_STAPLES | Margin/ROE focus, low revenue growth expectations |
+| CONSUMER_DISCRETIONARY | Revenue growth primary, P/E up to 25x |
+
+**ETF gates:**
+- `maxExpenseRatio: 0.2%` — hard gate
+- `minFiveYearReturn: 8.0%` — S&P long-run floor
+- `minVolume: 1,000,000` ADV
+
+**BOND gates:**
+- `minCreditRating: 7` (BBB = investment-grade floor)
+- `minSpread: 1.5%` above risk-free
+- `maxDuration: 7` years
+
+---
+
+## MarketRegime (INFLATED overrides)
+
+`src/market/MarketRegime.js` derives gate overrides from live benchmarks and current rate regime:
+
+| Gate | Formula (NORMAL rates) | Formula (HIGH rates) |
+|---|---|---|
+| Stock maxPERatio | SPY trailing P/E × 1.5 | SPY trailing P/E × 1.2 |
+| Tech maxPERatio | XLK P/E × 1.3 | XLK P/E × 1.3 |
+| Tech maxPegGate | XLK P/E ÷ 15 | XLK P/E ÷ 15 |
+| REIT minYield | XLRE yield × 0.85 | XLRE yield × 0.95 |
+| Bond minSpread | LQD−TNX × 0.80 | LQD−TNX × 0.90 |
+| ETF maxExpenseRatio | 0.75% | 0.75% |
+
+---
+
+## Sector Detection
+
+`Stock._mapToStandardSector()` maps Yahoo Finance `sector`/`industry` strings to internal constants.
+Order matters — more specific matches first:
+
+```
+TECHNOLOGY   → "technology", "electronic", "semiconductor", "software"
+REIT         → "real estate", "reit"
+FINANCIAL    → "financial", "bank", "insurance", "asset management"
+ENERGY       → "energy", "oil", "gas", "petroleum"
+HEALTHCARE   → "health", "biotech", "pharmaceutical", "medical"
+COMMUNICATION→ "communication", "media", "entertainment", "telecom"
+CONSUMER_STAPLES → "consumer defensive", "consumer staples", "household", "beverage", "food"
+CONSUMER_DISCRETIONARY → "consumer cyclical", "consumer discretionary", "retail", "apparel", "auto"
+GENERAL      → fallback
+```
+
+---
+
+## DataMapper Notes
+
+- **peRatio**: prefers `trailingPE` (audited) over `forwardPE` (analyst estimate, ~10-15% optimistic)
+- **FCF yield**: `freeCashflow !== 0` (not `> 0`) — negative FCF preserved so cash-burning companies fail the gate, not silently skip it
+- **Bond duration**: inferred from category string ("Short-Term" → 2y, "Intermediate" → 5y, "Long" → 18y, default 6y). Yahoo does not expose effective duration in the modules we fetch.
+- **ETF volume**: `summaryDetail.averageVolume` — was missing before, causing the `-2` liquidity penalty on every ETF
+
+---
+
+## Missing Data Convention
+
+- Missing metrics use `null` (not `0`) in `_sanitize`. Gate checks skip `null` rather than auto-failing.
+- `pegRatio` falls back to `trailingPE / earningsGrowth` when Yahoo doesn't provide it.
+- `quickRatio` falls back to `currentRatio` when missing.
+
+---
+
+## Logger Injection Pattern
+
+Classes that produce output accept an optional `{ logger }` constructor option so they work cleanly in server context:
+
+```js
+// CLI (default) — writes to stdout
+new ScreenerEngine()
+
+// Server — fully silent
+new ScreenerEngine({ logger: { write: () => {}, log: () => {}, warn: () => {} } })
+```
+
+Affected: `ScreenerEngine`, `BenchmarkProvider`, `CatalystAnalyst`, `SimpleFINClient`, `LLMAnalyst`.
+
+---
+
+## Reporter Pattern
+
+Both reporters have two methods:
+
+```js
+reporter.render(...)     // → HTML string (use in server route responses)
+reporter.generate(...)   // → writes file to disk, returns path (use in CLI)
+```
+
+---
+
+## SimpleFIN Auth Flow
+
+1. User gets a Setup Token from https://beta-bridge.simplefin.org
+2. `SimpleFINClient.init()` base64-decodes it → POSTs once to claim Access URL
+3. `onAccessUrlClaimed` callback is called with the URL — CLI uses `saveAccessUrlToEnv()`, server stores elsewhere
+4. All subsequent requests use Access URL with `Authorization: Basic` header (not embedded in URL)
+
+---
+
+## portfolio.json Format
+
+```json
+{
+  "holdings": [
+    { "ticker": "AAPL",    "shares": 10,   "costBasis": 150.00, "source": "Robinhood", "type": "stock"  },
+    { "ticker": "VOO",     "shares": 8,    "costBasis": 380.00, "source": "Vanguard",  "type": "etf"    },
+    { "ticker": "BTC-USD", "shares": 0.25, "costBasis": 45000,  "source": "Coinbase",  "type": "crypto" }
+  ]
+}
+```
+
+`type` values: `stock`, `etf`, `crypto`. Crypto is priced via Yahoo (BTC-USD style) but not fundamentally scored.
+
+---
+
+## Tests
+
+Uses Node's built-in test runner (`node:test` + `node:assert/strict`) — no test framework needed.
+
+```
+tests/
+  ScoringConfig.test.js    ← gate values (P/E 15x, PEG 1.0, QuickRatio 0.8), sector overrides
+  RuleMerger.test.js       ← FUNDAMENTAL vs INFLATED modes, sector merging
+  MarketRegime.test.js     ← inflated overrides including HIGH/NORMAL rate regime variants
+  StockScorer.test.js      ← gate failures, scoring labels, risk flags
+  EtfScorer.test.js        ← expense gate, volume penalty, 5Y return scoring
+  BondScorer.test.js       ← credit gate, spread/duration scoring, unit handling
+  DataMapper.test.js       ← type detection, PEG computation, trailing PE preference,
+                             negative FCF, ETF volume, bond duration inference
+  PortfolioAdvisor.test.js ← _position gain/loss calc, _advice signal mapping, BRK.B dot-notation normalisation
+  LLMAnalyst.test.js       ← markdown fence stripping, JSON parse correctness
+```
+
+Pre-commit hook runs `lint-staged` (Prettier) then `npm test`. Pre-push hook runs `npm test`.
+Test output: silent on pass, shows only failures + one summary line (`scripts/summary-reporter.js`).
+
+**Key unit:** `ytm` in `Bond.metrics` is stored as a percentage (e.g. `6.5` = 6.5%). `BondScorer._sanitize` divides by 100 before spread calculation.
+
+---
+
+## Conventions
+
+- Asset `type` (uppercased) is the routing key across DataMapper, asset classes, `SCORERS` map, and ScoringRules.
+- Prefer adjusting `ScoringConfig` or `MarketRegime` over hardcoding numbers in scorers.
+- BenchmarkProvider caches for 1 hour — restart the server to force a fresh fetch.
+- All entry points live in `bin/`. Do not add logic to entry points — they call into `src/`.
+- `bin/server.js` starts Fastify; `src/server/` contains all route logic.
+- **Never** call `process.exit()` inside `src/` — only `bin/` may do that.
+- Class instances don't survive `JSON.stringify`. Call `getDisplayMetrics()` server-side before returning from API routes (see `src/server/routes/screener.js` `serializeAssets()`).
+
+---
+
+## Architecture Roadmap
+
+Planned improvements in priority order. Do not start a later phase before completing earlier ones.
+
+### Phase 1 — Cleanup ✅ COMPLETE
+All items completed. Additional features delivered alongside cleanup:
+
+**Cleanup done:**
+- Deleted root-level `finance.js`, `import-portfolio.js`, `markdown.md`
+- Deleted `src/server/routes/analyze.js` (orphaned route file)
+- Removed dead `analysis` state, `analysisOpen` state, and "🤖 AI Market Analysis" panel from `+page.svelte`
+- Fixed `.gitignore` — `portfolio.json`, `market-calls.json`, `.env` are now excluded from git
+
+**Features added during Phase 1:**
+- `POST /api/analyze` — per-tab LLM analysis with sidebar (✦ Analyze button on each asset section)
+- `POST /api/finance/holdings` + `DELETE /api/finance/holdings/:ticker` — add/edit/delete holdings via UI
+- Portfolio page: inline row editing, optimistic UI updates, sortable columns, collapsible market context with tooltips, P&L summary card tooltips
+- Holdings can be added/edited/deleted via the portfolio UI (manual entry replaces CSV importer)
+- `BRK.B` dot-notation tickers now normalised to Yahoo Finance format (`BRK.B → BRK-B`)
+- Market graph drawing-line animation replaces generic spinner (lg/md); dot-pulse for sm (buttons)
+- Portfolio page loads client-side (`$effect`) to avoid blocking navigation
+- Catalyst page auto-loads on mount; LLM analysis only runs on explicit ✦ Analyze click
+
+**Pending (deferred to later):**
+- LLM Analysis button on portfolio page (analyse holdings against current news)
+
+### Phase 2 — Extract Shared Utilities
+- Create `ui/src/lib/utils.ts` with all pure functions currently duplicated across pages: `sigOrd`, `sorted`, `verdictShort`, `vClass`, `fmtPE`, `fmt`, `fmtShort`, `glClass`
+- Create `src/server/utils/logger.js` with shared `noopLogger` constant (currently copy-pasted in `screener.js` and `app.js`)
+
+### Phase 3 — Rename `src/` → `server/`
+- Rename the directory and update all import paths in `bin/`, internal routes, and `CLAUDE.md`
+- Makes the API layer unambiguous — `src/` conventionally implies "all project source"
+
+### Phase 4 — SCSS Migration
+Replace per-component `<style>` blocks with a shared token system in `ui/src/styles/`:
+```
+_tokens.scss     ← all color variables, spacing, font-size scale
+_reset.scss      ← current 3-line app.css
+_layout.scss     ← shell, nav, main (from +layout.svelte)
+_table.scss      ← shared table/thead/.ticker/.num styles (used across 4 pages)
+_buttons.scss    ← btn-primary, btn-ghost, btn-analyze, btn-catalyst
+_badges.scss     ← verdict-pill, tag, sentiment-pill (resolves verdict-pill vs vpill inconsistency)
+_section.scss    ← section card + section-header pattern
+app.scss         ← root file, @uses all partials
+```
+Component `<style>` blocks should only contain styles genuinely unique to that component.
+
+### Phase 5 — Decompose `+page.svelte`
+At 962 lines it is the biggest maintenance liability. Extract into:
+- `AssetTable.svelte` — accepts `type`, `rows`, `mode`; renders STOCK/ETF/BOND table with mode tabs and Analyze button
+- `AnalysisSidebar.svelte` — owns open/close/loading state, takes `type` + `onAnalyze` as props
+- `VerdictPill.svelte` — single component replacing both `.verdict-pill` and `.vpill` usages
+- `MarketContextStrip.svelte` — consolidates the inline version in `+page.svelte` with the existing `MarketContext.svelte` in `$lib/`
+
+Also: split `loadCatalysts()` into two functions (fetch tickers / screen tickers). Replace the `_booted` / `$effect` hack with a proper `+page.js` load function.
+
+### Phase 6 — TypeScript
+Convert server first (no framework coupling), then `$lib/utils`, then Svelte components.
+
+Define shared types first:
+```ts
+type Signal    = '✅ Strong Buy' | '⚡ Momentum' | '🔄 Neutral' | '⚠️ Speculation' | '❌ Avoid'
+type AssetType = 'STOCK' | 'ETF' | 'BOND'
+type ScoreMode = 'inflated' | 'fundamental'
+
+interface ScreenerResult  { STOCK, ETF, BOND, ERROR, marketContext }
+interface MarketContext   { sp500Price, riskFreeRate, vixLevel, rateRegime, benchmarks }
+interface LLMAnalysis     { summary, sentiment, affectedIndustries, relatedTickers }
+interface MarketCall      { id, title, quarter, date, thesis, tickers, snapshot }
+interface PortfolioHolding { ticker, shares, costBasis, source, type }
+```
+
+SvelteKit supports TypeScript natively — components just need `<script lang="ts">`.
+
+### Not Planned
+- **npm workspaces / monorepo** — current `ui/` subdirectory structure works; high friction for low gain at this scale
+- **Database** — JSON files are sufficient at current portfolio size; Yahoo Finance rate limiting is the real bottleneck, not storage. Revisit with SQLite only if portfolio grows to 500+ holdings with frequent concurrent reads.
+
+---
+
+## Adding a New Asset Type
+
+1. Create a subclass of `Asset` in `src/screener/assets/` with a flat `metrics` object and `getDisplayMetrics()`.
+2. Add a per-type entry (`gates` / `weights` / `thresholds`) to `ScoringRules` in `ScoringConfig.js`.
+3. Add inflated overrides in `MarketRegime.getInflatedOverrides()`.
+4. Create a Scorer in `src/screener/scorers/` exposing `score(metrics, rules, marketContext)`.
+5. Add a mapper in `DataMapper.js`.
+6. Wire into `ScreenerEngine`: add `case` in `_buildAsset`, entry in `SCORERS` map.
+7. Add the new type to `serializeAssets()` handling in `src/server/routes/screener.js`.