Files
market_screener/CLAUDE.md
T
2026-06-04 11:24:08 -04:00

471 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
Guidance for working in this repository.
## Overview
`market-screener` is a Node.js project with two modes:
1. **CLI** — screens stocks, ETFs, and bonds via `npm start`, generates HTML reports
2. **Fastify API server** — powers the SvelteKit dashboard in the `ui/` subdirectory
Every asset is scored under two lenses:
- **Market-Adjusted** — gates derived from live Yahoo benchmarks (SPY P/E, XLK P/E, XLRE yield, LQD spread). Reflects what is acceptable in today's market.
- **Fundamental** — strict Graham/value-investing gates from `ScoringConfig`. Reflects genuine value regardless of market conditions.
The comparison produces a **Signal** (Strong Buy / Momentum / Speculation / Neutral / Avoid).
ES module project (`"type": "module"`); use `import`/`export`, not `require`.
---
## Commands
```bash
npm install # install dependencies
npm run dev # start API server (port 3000) + SvelteKit UI (port 5173) together
npm run server # API server only (port 3000)
npm start # CLI: Yahoo news → catalyst tickers → screener-report.html
npm start -- watch # CLI: default watchlist
npm start -- AAPL MSFT VOO # CLI: specific tickers
npm run finance # CLI: portfolio advice + SimpleFIN → finance-report.html
npm test # run all unit tests (node:test, zero external deps)
npm run test:watch # watch mode — uses verbose spec reporter
npm run format # format all server/bin/tests with Prettier
npm run format:check # check formatting without writing (used in CI/pre-commit)
npm run ui:install # install UI dependencies (ui/ subdirectory)
```
`npm run dev` runs both the API server and the SvelteKit UI (in `ui/`) concurrently. Run `npm run ui:install` once before first use.
---
## Project Structure
```
bin/
screen.js ← CLI screener entry point
finance.js ← CLI personal finance entry point
import-portfolio.js ← broker CSV importer
server.js ← Fastify API server entry point
scripts/
summary-reporter.js ← custom node:test reporter (silent on pass, summary line at end)
prompts/
catalyst-analysis.md ← daily catalyst analysis playbook (LLM prompt + workflow)
server/
config/
ScoringConfig.js ← CREDIT_RATING_SCALE + ScoringRules (single source of truth)
constants.js ← SIGNAL, ASSET_TYPE, SECTOR, SCORE_MODE, REGIME, SIGNAL_ORDER
market/ ← Yahoo Finance data layer
YahooClient.js ← wraps yahoo-finance2 v3, retry + backoff
BenchmarkProvider.js ← fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD → marketContext
MarketRegime.js ← derives INFLATED gate overrides from live benchmarks + rate regime
screener/ ← core screening domain
ScreenerEngine.js ← orchestrates: fetch → score × 2. Methods: screenTickers() (pure data),
screenWithProgress() (CLI with stdout). Accepts { logger } option.
DataMapper.js ← normalises Yahoo payload → flat asset data object
NOTE: uses trailingPE (not forwardPE). Preserves negative FCF.
Infers bond duration from category string. Maps ETF volume.
RuleMerger.js ← merges base rules + sector overrides + MarketRegime (INFLATED mode)
Chunker.js ← splits ticker list into batches
assets/
Asset.js ← abstract base: ticker, currentPrice, type, formatting helpers
Stock.js ← metrics + _mapToStandardSector (8 sectors detected)
Etf.js ← metrics: expenseRatio, yield, volume, fiveYearReturn, totalAssets
Bond.js ← metrics: ytm, duration, creditRating, creditRatingNumeric
scorers/
StockScorer.js ← gate checks + weighted registry (ROE, opMargin, margin, peg, rev, fcf)
EtfScorer.js ← expense gate + registry (cost, yield, volume, fiveYearReturn)
BondScorer.js ← credit gate + spread/duration scoring
analyst/
CatalystAnalyst.js ← fetches Yahoo Finance news, extracts relatedTickers. Accepts { logger }.
LLMAnalyst.js ← uses Claude Haiku (ANTHROPIC_API_KEY) to analyze headlines → summary,
sentiment (BULLISH/NEUTRAL/BEARISH), affectedIndustries, relatedTickers.
Returns null gracefully if API key is not set. Accepts { logger }.
calls/
MarketCallStore.js ← persists quarterly market thesis entries to market-calls.json.
Each call stores: title, quarter, date, thesis, tickers[], snapshot{}
(price + signal per ticker at creation time). CRUD: list/get/create/delete.
finance/
clients/
SimpleFINClient.js ← claims setup token → access URL, fetches /accounts via Basic Auth header
(NOT embedded credentials in URL). Accepts { logger, onAccessUrlClaimed }.
PersonalFinanceAnalyzer.js ← net worth, cash vs investments, spending by category
PortfolioAdvisor.js ← cross-references holdings with screener signals → hold/sell/add advice
reporters/
HtmlReporter.js ← render() → HTML string (server), generate() → writes file (CLI)
FinanceReporter.js ← render() → HTML string (server), generate() → writes file (CLI)
server/
app.js ← Fastify app factory (buildApp). Registers CORS + routes.
routes/
screener.js ← POST /api/screen, GET /api/screen/catalysts
Serializes asset.getDisplayMetrics() before JSON response.
finance.js ← GET /api/finance/portfolio, GET /api/finance/market-context
calls.js ← CRUD for market calls + GET /api/calls/calendar (earnings/dividend events)
ui/ ← SvelteKit dashboard (lives inside this repo, not a separate repo)
src/routes/
+page.svelte ← main screener UI
calls/ ← market calls list + detail views
portfolio/ ← portfolio advice view
safe-buys/ ← filtered strong-buy view
market-calls.json ← persisted market thesis calls (written by MarketCallStore)
portfolio.json ← user's holdings: ticker, shares, costBasis, source, type
.env ← SIMPLEFIN_ACCESS_URL or SIMPLEFIN_SETUP_TOKEN, ANTHROPIC_API_KEY
```
---
## Data Flow
```
Yahoo Finance API
BenchmarkProvider — fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD
builds marketContext { sp500Price, riskFreeRate, vixLevel,
rateRegime, volatilityRegime, benchmarks { marketPE, techPE, reitYield, igSpread } }
DataMapper — normalises raw Yahoo payload → flat data object with type (STOCK/ETF/BOND)
uses trailingPE as primary; preserves negative FCF yield; infers bond duration
Asset subclass — Stock / Etf / Bond holds metrics + getDisplayMetrics()
RuleMerger × 2 — FUNDAMENTAL mode: ScoringConfig as-is (Graham-style)
INFLATED mode: sector override + MarketRegime live gate overrides
Scorer × 2 — StockScorer / EtfScorer / BondScorer, fully stateless
ScreenerEngine — derives Signal from comparing both verdicts
├── CLI path: screenWithProgress() → HtmlReporter.generate() → screener-report.html
└── API path: screenTickers() → JSON (with serialized displayMetrics) → SvelteKit UI
```
---
## API Routes (Fastify)
| Method | Path | Description |
|---|---|---|
| GET | `/health` | Health check |
| POST | `/api/screen` | Screen tickers. Body: `{ tickers: string[] }`. Returns `{ STOCK, ETF, BOND, ERROR, marketContext }` with `asset.displayMetrics` pre-serialized |
| GET | `/api/screen/catalysts` | Yahoo news → `{ tickers, stories }` |
| GET | `/api/finance/portfolio` | Portfolio advice + optional SimpleFIN data |
| GET | `/api/finance/market-context` | Live benchmark data only |
| GET | `/api/calls` | List all market calls (newest first) |
| GET | `/api/calls/:id` | Get one call + re-screened current prices for comparison |
| POST | `/api/analyze` | Fetch Yahoo news for specific tickers + run LLM analysis. Body: `{ tickers: string[] }`. Returns `{ analysis }` |
| POST | `/api/calls` | Create a market call; snapshots current prices. Body: `{ title, quarter, thesis, tickers[], date? }` |
| DELETE | `/api/calls/:id` | Delete a market call |
| GET | `/api/calls/calendar` | Earnings + dividend calendar. Query: `?tickers=AAPL,MSFT` (omit for all call tickers) |
CORS is configured for `CLIENT_ORIGIN` env var (default `http://localhost:5173`).
---
## Scoring Modes
| Mode | P/E Gate (general) | P/E Gate (tech) | Source |
|---|---|---|---|
| FUNDAMENTAL | 15x | 35x | ScoringConfig (true Graham) |
| INFLATED | S&P 500 P/E × 1.5 | XLK P/E × 1.3 | Live SPY/XLK data |
**Rate regime effect on INFLATED mode:**
- HIGH rate regime: P/E multiplier compresses to 1.2× (vs 1.5× in NORMAL)
- HIGH rate regime: REIT yield floor tightens (0.95× vs 0.85×)
- HIGH rate regime: bond spread demand increases (0.90× vs 0.80×)
| Signal | Meaning |
|---|---|
| ✅ Strong Buy | Passes both fundamental AND inflated gates |
| ⚡ Momentum | Passes inflated, holds fundamentally |
| ⚠️ Speculation | Passes inflated, fails fundamental |
| 🔄 Neutral | Hold territory in one or both lenses |
| ❌ Avoid | Fails both |
---
## ScoringConfig Key Values
`server/config/ScoringConfig.js` — single source of truth for all gates, weights, thresholds.
**STOCK base gates (Fundamental mode):**
- `maxPERatio: 15` — Graham's actual rule (trailing P/E)
- `maxPegGate: 1.0` — Lynch standard: PEG > 1.0 means paying full price
- `maxDebtToEquity: 1.5` — most distress starts above 2x
- `minQuickRatio: 0.8` — below this signals real liquidity stress
**Sector overrides** (structural — apply in both modes):
| Sector | Key difference |
|---|---|
| TECHNOLOGY | D/E up to 2.0, P/E up to 35x, FCF weight raised |
| REIT | P/E and PEG disabled (9999), scored on yield + P/FFO |
| FINANCIAL | D/E disabled, scored on ROE + P/B, maxPriceToBook 1.5x |
| ENERGY | FCF weight 4, yield weight 3, opMargin primary |
| HEALTHCARE | Revenue growth primary, P/E up to 25x |
| COMMUNICATION | FCF weight 4, P/E up to 25x (META, GOOGL, NFLX) |
| CONSUMER_STAPLES | Margin/ROE focus, low revenue growth expectations |
| CONSUMER_DISCRETIONARY | Revenue growth primary, P/E up to 25x |
**ETF gates:**
- `maxExpenseRatio: 0.2%` — hard gate
- `minFiveYearReturn: 8.0%` — S&P long-run floor
- `minVolume: 1,000,000` ADV
**BOND gates:**
- `minCreditRating: 7` (BBB = investment-grade floor)
- `minSpread: 1.5%` above risk-free
- `maxDuration: 7` years
---
## MarketRegime (INFLATED overrides)
`server/market/MarketRegime.js` derives gate overrides from live benchmarks and current rate regime:
| Gate | Formula (NORMAL rates) | Formula (HIGH rates) |
|---|---|---|
| Stock maxPERatio | SPY trailing P/E × 1.5 | SPY trailing P/E × 1.2 |
| Tech maxPERatio | XLK P/E × 1.3 | XLK P/E × 1.3 |
| Tech maxPegGate | XLK P/E ÷ 15 | XLK P/E ÷ 15 |
| REIT minYield | XLRE yield × 0.85 | XLRE yield × 0.95 |
| Bond minSpread | LQDTNX × 0.80 | LQDTNX × 0.90 |
| ETF maxExpenseRatio | 0.75% | 0.75% |
---
## Sector Detection
`Stock._mapToStandardSector()` maps Yahoo Finance `sector`/`industry` strings to internal constants.
Order matters — more specific matches first:
```
TECHNOLOGY → "technology", "electronic", "semiconductor", "software"
REIT → "real estate", "reit"
FINANCIAL → "financial", "bank", "insurance", "asset management"
ENERGY → "energy", "oil", "gas", "petroleum"
HEALTHCARE → "health", "biotech", "pharmaceutical", "medical"
COMMUNICATION→ "communication", "media", "entertainment", "telecom"
CONSUMER_STAPLES → "consumer defensive", "consumer staples", "household", "beverage", "food"
CONSUMER_DISCRETIONARY → "consumer cyclical", "consumer discretionary", "retail", "apparel", "auto"
GENERAL → fallback
```
---
## DataMapper Notes
- **peRatio**: prefers `trailingPE` (audited) over `forwardPE` (analyst estimate, ~10-15% optimistic)
- **FCF yield**: `freeCashflow !== 0` (not `> 0`) — negative FCF preserved so cash-burning companies fail the gate, not silently skip it
- **Bond duration**: inferred from category string ("Short-Term" → 2y, "Intermediate" → 5y, "Long" → 18y, default 6y). Yahoo does not expose effective duration in the modules we fetch.
- **ETF volume**: `summaryDetail.averageVolume` — was missing before, causing the `-2` liquidity penalty on every ETF
---
## Missing Data Convention
- Missing metrics use `null` (not `0`) in `_sanitize`. Gate checks skip `null` rather than auto-failing.
- `pegRatio` falls back to `trailingPE / earningsGrowth` when Yahoo doesn't provide it.
- `quickRatio` falls back to `currentRatio` when missing.
---
## Logger Injection Pattern
Classes that produce output accept an optional `{ logger }` constructor option so they work cleanly in server context:
```js
// CLI (default) — writes to stdout
new ScreenerEngine()
// Server — fully silent
new ScreenerEngine({ logger: { write: () => {}, log: () => {}, warn: () => {} } })
```
Affected: `ScreenerEngine`, `BenchmarkProvider`, `CatalystAnalyst`, `SimpleFINClient`, `LLMAnalyst`.
---
## Reporter Pattern
Both reporters have two methods:
```js
reporter.render(...) // → HTML string (use in server route responses)
reporter.generate(...) // → writes file to disk, returns path (use in CLI)
```
---
## SimpleFIN Auth Flow
1. User gets a Setup Token from https://beta-bridge.simplefin.org
2. `SimpleFINClient.init()` base64-decodes it → POSTs once to claim Access URL
3. `onAccessUrlClaimed` callback is called with the URL — CLI uses `saveAccessUrlToEnv()`, server stores elsewhere
4. All subsequent requests use Access URL with `Authorization: Basic` header (not embedded in URL)
---
## portfolio.json Format
```json
{
"holdings": [
{ "ticker": "AAPL", "shares": 10, "costBasis": 150.00, "source": "Robinhood", "type": "stock" },
{ "ticker": "VOO", "shares": 8, "costBasis": 380.00, "source": "Vanguard", "type": "etf" },
{ "ticker": "BTC-USD", "shares": 0.25, "costBasis": 45000, "source": "Coinbase", "type": "crypto" }
]
}
```
`type` values: `stock`, `etf`, `crypto`. Crypto is priced via Yahoo (BTC-USD style) but not fundamentally scored.
---
## Tests
Uses Node's built-in test runner (`node:test` + `node:assert/strict`) — no test framework needed.
```
tests/
ScoringConfig.test.js ← gate values (P/E 15x, PEG 1.0, QuickRatio 0.8), sector overrides
RuleMerger.test.js ← FUNDAMENTAL vs INFLATED modes, sector merging
MarketRegime.test.js ← inflated overrides including HIGH/NORMAL rate regime variants
StockScorer.test.js ← gate failures, scoring labels, risk flags
EtfScorer.test.js ← expense gate, volume penalty, 5Y return scoring
BondScorer.test.js ← credit gate, spread/duration scoring, unit handling
DataMapper.test.js ← type detection, PEG computation, trailing PE preference,
negative FCF, ETF volume, bond duration inference
PortfolioAdvisor.test.js ← _position gain/loss calc, _advice signal mapping, BRK.B dot-notation normalisation
LLMAnalyst.test.js ← markdown fence stripping, JSON parse correctness
```
Pre-commit hook runs `lint-staged` (Prettier) then `npm test`. Pre-push hook runs `npm test`.
Test output: silent on pass, shows only failures + one summary line (`scripts/summary-reporter.js`).
**Key unit:** `ytm` in `Bond.metrics` is stored as a percentage (e.g. `6.5` = 6.5%). `BondScorer._sanitize` divides by 100 before spread calculation.
---
## Conventions
- Asset `type` (uppercased) is the routing key across DataMapper, asset classes, `SCORERS` map, and ScoringRules.
- Prefer adjusting `ScoringConfig` or `MarketRegime` over hardcoding numbers in scorers.
- BenchmarkProvider caches for 1 hour — restart the server to force a fresh fetch.
- All entry points live in `bin/`. Do not add logic to entry points — they call into `server/`.
- `bin/server.js` starts Fastify; `server/server/` contains all route logic.
- **Never** call `process.exit()` inside `server/` — only `bin/` may do that.
- Class instances don't survive `JSON.stringify`. Call `getDisplayMetrics()` server-side before returning from API routes (see `server/server/routes/screener.js` `serializeAssets()`).
---
## Architecture Roadmap
Planned improvements in priority order. Do not start a later phase before completing earlier ones.
### Phase 1 — Cleanup ✅ COMPLETE
All items completed. Additional features delivered alongside cleanup:
**Cleanup done:**
- Deleted root-level `finance.js`, `import-portfolio.js`, `markdown.md`
- Deleted `server/server/routes/analyze.js` (orphaned route file)
- Removed dead `analysis` state, `analysisOpen` state, and "🤖 AI Market Analysis" panel from `+page.svelte`
- Fixed `.gitignore``portfolio.json`, `market-calls.json`, `.env` are now excluded from git
**Features added during Phase 1:**
- `POST /api/analyze` — per-tab LLM analysis with sidebar (✦ Analyze button on each asset section)
- `POST /api/finance/holdings` + `DELETE /api/finance/holdings/:ticker` — add/edit/delete holdings via UI
- Portfolio page: inline row editing, optimistic UI updates, sortable columns, collapsible market context with tooltips, P&L summary card tooltips
- Holdings can be added/edited/deleted via the portfolio UI (manual entry replaces CSV importer)
- `BRK.B` dot-notation tickers now normalised to Yahoo Finance format (`BRK.B → BRK-B`)
- Market graph drawing-line animation replaces generic spinner (lg/md); dot-pulse for sm (buttons)
- Portfolio page loads client-side (`$effect`) to avoid blocking navigation
- Catalyst page auto-loads on mount; LLM analysis only runs on explicit ✦ Analyze click
**Pending (deferred to later):**
- LLM Analysis button on portfolio page (analyse holdings against current news)
### Phase 2 — Extract Shared Utilities ✅ COMPLETE
**Done:**
- Created `ui/src/lib/utils.ts` — typed shared pure functions: `sigOrd`, `sorted`, `verdictShort`, `vClass`, `fmtPE`, `fmt`, `fmtShort`, `glClass`, `advClass`. Exports `Signal` type.
- Created `server/server/utils/logger.js` — shared `noopLogger` constant, imported by `screener.js`, `app.js`, `finance.js`, and `calls.js`
- Added TypeScript support to `ui/``tsconfig.json` extending SvelteKit's generated config, `typescript` and `svelte-check` added as dev dependencies
- All three pages (`+page.svelte`, `safe-buys/+page.svelte`, `portfolio/+page.svelte`) now import from `$lib/utils.js` instead of duplicating logic
### Phase 3 — Rename `src/` → `server/` ✅ COMPLETE
**Done:**
- Renamed `src/` to `server/``src/server/` is now `server/server/`
- Updated all import paths in `bin/`, `tests/`, and `CLAUDE.md`
### Phase 4 — SCSS Migration
Replace per-component `<style>` blocks with a shared token system in `ui/src/styles/`:
```
_tokens.scss ← all color variables, spacing, font-size scale
_reset.scss ← current 3-line app.css
_layout.scss ← shell, nav, main (from +layout.svelte)
_table.scss ← shared table/thead/.ticker/.num styles (used across 4 pages)
_buttons.scss ← btn-primary, btn-ghost, btn-analyze, btn-catalyst
_badges.scss ← verdict-pill, tag, sentiment-pill (resolves verdict-pill vs vpill inconsistency)
_section.scss ← section card + section-header pattern
app.scss ← root file, @uses all partials
```
Component `<style>` blocks should only contain styles genuinely unique to that component.
### Phase 5 — Decompose `+page.svelte`
At 962 lines it is the biggest maintenance liability. Extract into:
- `AssetTable.svelte` — accepts `type`, `rows`, `mode`; renders STOCK/ETF/BOND table with mode tabs and Analyze button
- `AnalysisSidebar.svelte` — owns open/close/loading state, takes `type` + `onAnalyze` as props
- `VerdictPill.svelte` — single component replacing both `.verdict-pill` and `.vpill` usages
- `MarketContextStrip.svelte` — consolidates the inline version in `+page.svelte` with the existing `MarketContext.svelte` in `$lib/`
Also: split `loadCatalysts()` into two functions (fetch tickers / screen tickers). Replace the `_booted` / `$effect` hack with a proper `+page.js` load function.
### Phase 6 — TypeScript
Convert server first (no framework coupling), then `$lib/utils`, then Svelte components.
Define shared types first:
```ts
type Signal = '✅ Strong Buy' | '⚡ Momentum' | '🔄 Neutral' | '⚠️ Speculation' | '❌ Avoid'
type AssetType = 'STOCK' | 'ETF' | 'BOND'
type ScoreMode = 'inflated' | 'fundamental'
interface ScreenerResult { STOCK, ETF, BOND, ERROR, marketContext }
interface MarketContext { sp500Price, riskFreeRate, vixLevel, rateRegime, benchmarks }
interface LLMAnalysis { summary, sentiment, affectedIndustries, relatedTickers }
interface MarketCall { id, title, quarter, date, thesis, tickers, snapshot }
interface PortfolioHolding { ticker, shares, costBasis, source, type }
```
SvelteKit supports TypeScript natively — components just need `<script lang="ts">`.
### Not Planned
- **npm workspaces / monorepo** — current `ui/` subdirectory structure works; high friction for low gain at this scale
- **Database** — JSON files are sufficient at current portfolio size; Yahoo Finance rate limiting is the real bottleneck, not storage. Revisit with SQLite only if portfolio grows to 500+ holdings with frequent concurrent reads.
---
## Adding a New Asset Type
1. Create a subclass of `Asset` in `server/screener/assets/` with a flat `metrics` object and `getDisplayMetrics()`.
2. Add a per-type entry (`gates` / `weights` / `thresholds`) to `ScoringRules` in `ScoringConfig.js`.
3. Add inflated overrides in `MarketRegime.getInflatedOverrides()`.
4. Create a Scorer in `server/screener/scorers/` exposing `score(metrics, rules, marketContext)`.
5. Add a mapper in `DataMapper.js`.
6. Wire into `ScreenerEngine`: add `case` in `_buildAsset`, entry in `SCORERS` map.
7. Add the new type to `serializeAssets()` handling in `server/server/routes/screener.js`.