# CLAUDE.md Guidance for working in this repository. **πŸ“‹ See [`PHASES.md`](./PHASES.md) for the complete Phase 9-16+ roadmap, architecture summaries, and production readiness checklists.** ## Overview `market-screener` is a Node.js project consisting of a Fastify API server that powers the SvelteKit dashboard in the `ui/` subdirectory. **Evolved to support day trading**: real-time news webhooks, LLM-driven stock analysis with prompt caching, multi-user authentication, and Discord alerts for price movements. --- ## βœ… Status Update β€” Shipped June 2026 (post-Phase-10 sprint) See **PRODUCT.md** (priorities P0–P3) and **FREE-DATA-STACK.md** (zero-cost data architecture) for design rationale. Shipped since the Phase 9/10 reports below: **Correctness / foundations (PRODUCT.md P0):** - Structured verdict tiers β€” scorers return `{ tier: PASS|HOLD|REJECT, score }`; `signal()` no longer string-matches emoji (P0.3) - Signal snapshot ledger β€” `signal_snapshots` table, written on every screen; `GET /api/screen/history/:ticker`; `npm run screen:daily` (P0.1) - Rate-regime hysteresis (Β±0.25% band, survives restarts) (P0.5) - Data-sanity sentinel β€” `dataHealth` on `/api/screen` + UI banner when >30% of stocks return null fundamentals (P0.4) - Bug-fix pass: ETF null-handling (missing data no longer auto-REJECTs), dividend-ETF↔bond classification via `fundProfile.categoryName`, zero-as-null sanitization, no-data verdicts labeled honestly, coverage field in every audit **Free-tier news pipeline (`server/domains/news/` β€” Phase-12-lite, $0):** - EDGAR poller (8-K, SC 13D, S-4, DEFM14A; CIKβ†’ticker map) + PR-wire RSS poller (GlobeNewswire, PR Newswire; exchange-tag ticker extraction) - Shared pipeline: universe filter β†’ noise blocklist β†’ dedupe β†’ keyword catalyst classifier β†’ `news_articles` / `ticker_catalysts` tables; retention jobs - In-server scheduler (EDGAR 10 min, PR 15 min; `NEWS_POLL=off` to disable) + `npm run news:poll` for cron - `GET /api/news/:ticker` (stored + live Yahoo merge), `GET /api/news/recent` **Daily change digest (`server/domains/digest/` β€” PRODUCT P1.1, partial Phase 14):** - Diffs today's snapshots vs previous, attaches news catalysts, M&A always surfaced - `GET /api/digest`, `npm run digest:daily`, Discord webhook (forum-channel aware), `npm run discord:test` **UI (screener page):** - Market Pulse header band β€” full-bleed sector bubbles (SPDR ETFs, 15-min cache), leader headline, loading/unavailable states; `GET /api/screen/sectors` - Sector drill-down panel β€” top-10 ETF holdings screened on demand, Today/1Y gain sort, 3-day sector news; `GET /api/screen/sector/:sector` - Ticker modal β€” company profile, range-switchable chart (1D…5Y, hover crosshair), Yahoo analyst target bar + Zacks link, latest news; `GET /api/screen/profile/:ticker`, `GET /api/screen/chart/:ticker?range=` - Plain-language advice layer (`adviceFor`) β€” "Buy β€” stable growth" / "Buy, but expect dips" on Strong Buys + no-data honesty; full text in modal - β†— Turnaround watch + πŸ’Ž Quality dips filters (with live counts + self-explaining empty states) in the STOCK table header - Score cell: negative-score fix, coverage chip, "No data" state **New env vars:** `EDGAR_USER_AGENT` (recommended), `DISCORD_WEBHOOK_URL`, `NEWS_PRWIRE_FEEDS`, `NEWS_POLL`. **Pages NOT yet touched in this sprint:** Portfolio, Market Calls, Safe Buys (still at their Phase 7 state β€” see realigned roadmap in PHASES.md). ### Two Scoring Lenses (Original) Every asset is scored under two lenses: - **Market-Adjusted** β€” gates derived from live Yahoo benchmarks (SPY P/E, XLK P/E, XLRE yield, LQD spread). Reflects what is acceptable in today's market. - **Fundamental** β€” strict Graham/value-investing gates from `ScoringConfig`. Reflects genuine value regardless of market conditions. The comparison produces a **Signal** (Strong Buy / Momentum / Speculation / Neutral / Avoid). ### Day Trading Mode (New) The app now supports real-time trading workflows: - **News webhooks** (Polygon.io) β€” ingest market news instantly - **Price monitoring** (Alpaca/Interactive Brokers) β€” detect 5%+ dips - **LLM analysis** with prompt caching β€” analyze opportunities in real-time - **Multi-user auth** β€” JWT-based, role-based portfolio access - **Discord notifications** β€” alerts for trading signals - **Trade journal** β€” log decisions + outcomes for performance analysis ES module project (`"type": "module"`); use `import`/`export`, not `require`. --- ## Commands ```bash npm install # install dependencies npm run dev # start API server (port 3000) + SvelteKit UI (port 5173) together npm run server # API server only (port 3000) npm test # run all unit tests (node:test, zero external deps) npm run test:watch # watch mode β€” uses verbose spec reporter npm run format # format all server/bin/tests with Prettier npm run format:check # check formatting without writing (used in CI/pre-commit) npm run ui:install # install UI dependencies (ui/ subdirectory) ``` `npm run dev` runs both the API server and the SvelteKit UI (in `ui/`) concurrently. Run `npm run ui:install` once before first use. **Day trading features** (when added): ```bash npm run queue:worker # start BullMQ workers (job processing) npm run webhook:test # test webhook signature validation locally npm run migrate:db # run schema migrations (when switching SQLite β†’ Postgres) ``` --- ## Project Structure (Phase 9: Domain-Driven) ``` bin/ server.ts ← Fastify API server entry point (imports buildApp from server/app.ts) prompts/ catalyst-analysis.md ← daily catalyst analysis playbook (LLM prompt + workflow) server/ app.ts ← Fastify app factory (buildApp). Registers CORS + all controllers. types.ts ← Barrel export: export * from domains/shared/types domains/ ← Domain-driven architecture (Phase 9+) shared/ ← Infrastructure & cross-domain utilities adapters/ YahooFinanceClient.ts AnthropicClient.ts SimpleFINClient.ts services/ BenchmarkProvider.ts ← fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD β†’ marketContext CatalystAnalyst.ts ← fetches Yahoo Finance news, extracts relatedTickers LLMAnalyst.ts ← analyzes headlines β†’ summary, sentiment, industries, tickers CatalystCache.ts ← 15-minute cache for catalyst analysis (Phase 8j) entities/ Asset.ts, Stock.ts, Etf.ts, Bond.ts persistence/ MarketCallRepository.ts PortfolioRepository.ts config/ constants.ts ← SIGNAL, SCORE_MODE, ASSET_TYPE, REGIME, CAP_CATEGORY, etc. scoring/ ScoringConfig.ts ← gates, weights, thresholds (single source of truth) MarketRegime.ts ← derives INFLATED overrides from live benchmarks db/ DatabaseConnection.ts ← SQLite wrapper with audit logging DatabaseInitializer.ts ← schema, migrations, legacy JSON migration QueryAudit.ts queries.constant.ts utils/ logger.ts Chunker.ts QueryBuilder.ts sanitizer.ts types/ *.model.ts ← all TypeScript types (asset, market, portfolio, finance, etc.) index.ts ← public API barrel screener/ ← Stock/ETF/Bond filtering & scoring screener.controller.ts ← POST /api/screen, GET /api/screen/catalysts analyze.controller.ts ← POST /api/analyze (LLM analysis) ScreenerEngine.ts ← orchestrates: fetch β†’ score Γ— 2 PersonalFinanceAnalyzer.ts ← net worth, cash vs investments analysis scorers/ StockScorer.ts EtfScorer.ts BondScorer.ts transform/ DataMapper.ts ← normalises Yahoo payload β†’ flat asset data RuleMerger.ts ← merges base rules + sector overrides index.ts portfolio/ ← Holdings management & investment advice finance.controller.ts ← GET /api/finance/portfolio, POST|DELETE /api/finance/holdings PortfolioAdvisor.ts ← cross-references holdings with signals index.ts calls/ ← Market call tracking & earnings calendar calls.controller.ts ← CRUD for market calls + GET /api/calls/calendar CalendarService.ts ← earnings calendar logic index.ts finance/ ← Portfolio reporting finance.controller.ts ← portfolio metrics endpoint index.ts ``` clients/ ← external API connectors, one class per third-party system YahooFinanceClient.ts ← wraps yahoo-finance2 v3, retry + backoff. Methods: fetchSummary, fetchCalendarEvents, search. Typed via YahooFinanceLib interface. SimpleFINClient.ts ← claims setup token β†’ access URL, fetches /accounts via Basic Auth. AnthropicClient.ts ← wraps Anthropic SDK. complete(system, user) β†’ raw text response. models/ ← domain entity classes with metrics + display logic Asset.ts ← abstract base: ticker, currentPrice, type, formatting helpers Stock.ts ← metrics + _mapToStandardSector (8 sectors) + _classifyMarketCap (Mega/Large/Mid/Small/Micro) + _classifyGrowth (style classification). Holds: valuation, quality, risk, 52W movement, analyst consensus, DCF. Etf.ts ← metrics: expenseRatio, yield, volume, fiveYearReturn, totalAssets Bond.ts ← metrics: ytm, duration, creditRating, creditRatingNumeric scorers/ ← stateless pure scoring functions, no I/O StockScorer.ts ← gate checks + weighted registry: core: ROE, opMargin, margin, peg, revenue, fcf expert: analyst consensus (inverted Yahoo 1–5 scale), DCF margin of safety riskFlags: beta, 52W position, 52W momentum, analyst divergence, DCF divergence EtfScorer.ts ← expense gate + registry (cost, yield, volume, fiveYearReturn) BondScorer.ts ← credit gate + spread/duration scoring config/ ScoringConfig.ts ← CREDIT_RATING_SCALE + ScoringRules (single source of truth for all gates, weights, thresholds including analyst and dcf weights) constants.ts ← SIGNAL, ASSET_TYPE, SECTOR, SCORE_MODE, REGIME, SIGNAL_ORDER, CAP_CATEGORY (Mega/Large/Mid/Small/Micro), GROWTH_CATEGORY (High Growth/Growth/Stable/Value/Turnaround/Declining) types/ ← all domain types, one file per domain asset.model.ts ← Signal, AssetType, ScoreMode, ScoringRules, ScoreResult, AssetResult, ScreenerResult, GateSet, WeightSet, ThresholdSet, RuleBlock, StockRules market.model.ts ← RateRegime, VolatilityRegime, Benchmarks, MarketContext portfolio.model.ts ← HoldingType, PortfolioHolding, PortfolioAdvice, AdviceRow calls.model.ts ← TickerSnapshot, MarketCall, SnapshotEntry, CreateCallInput finance.model.ts ← LLMAnalysis, CatalystStory, CalendarEvent, SimpleFINAccount, SimpleFINTransaction, SimpleFINData, YahooNewsItem, YahooFinanceLib logger.model.ts ← Logger models.model.ts ← AssetData, StockData, StockMetrics, EtfData, EtfMetrics, BondData, BondMetrics repositories.model.ts ← StoreData, PortfolioData scorers.model.ts ← NumVal, SanitizedMetrics, SanitizedBondMetrics services.model.ts ← BenchmarkProviderOptions, InflatedOverrides, ErrorResult, RuleSet, ScreenerEngineOptions, CatalystResult, MappedData, and other service shapes schemas.ts ← Fastify JSON Schema objects for request body validation index.ts ← re-exports all of the above (use this barrel for imports) types.ts ← thin barrel: export type * from './types/index.js' utils/ Chunker.ts ← splits ticker list into batches logger.ts ← noopLogger constant for silencing output in server context ui/ ← SvelteKit dashboard (lives inside this repo, not a separate repo) src/ styles/ ← global SCSS design-token system app.scss ← root file β€” @use all partials _tokens.scss ← CSS custom properties generated from SCSS maps ($bg, $text, $blues, $signals…) _reset.scss ← box-sizing reset + body base _layout.scss ← shell, nav, .links, main, nav-progress, .loading-area _section.scss ← .section, .section-header, .count, .mode-tabs, .error-banner _table.scss ← table, thead/tbody, .table-wrap, .col-ticker, .ticker, .num, .tag _buttons.scss ← button base, .btn-primary, .btn-catalyst, .btn-ghost, .btn-screen, .btn-analyze _badges.scss ← .verdict-pill, .sentiment-pill, .text-* helpers (SCSS @each maps) lib/ api.ts ← backward-compat shim; re-exports from api/index.ts api/ screener.ts ← screenTickers, fetchCatalysts, analyzeTickers finance.ts ← fetchPortfolio, addHolding, removeHolding, fetchMarketContext calls.ts ← fetchCalls, fetchCall, createCall, deleteCall, fetchCallsCalendar index.ts ← barrel re-export types.ts ← re-exports shared types from $types (server/types/); UI-only types defined here utils.ts ← shared pure functions: sigOrd, sorted, verdictShort, vClass, fmtPE, fmt… MarketContext.svelte ← collapsible card-grid context (used in portfolio + safe-buys) MarketContextStrip.svelte ← horizontal chip strip (used in screener) AssetTable.svelte ← STOCK/ETF/BOND section: mode tabs + Analyze + table AnalysisSidebar.svelte ← LLM analysis slide-over panel VerdictPill.svelte ← verdict-pill span; props: label SignalBadge.svelte ← signal emoji + label badge Spinner.svelte ← sm: dot-pulse | md/lg: chart-line animation routes/ +page.ts ← SvelteKit load (ssr:false) β€” fetches catalysts + screens on mount +page.svelte ← main screener UI +layout.svelte ← shell, nav, nav-progress bar, nav-overlay with Spinner calls/ ← market calls list + detail views portfolio/ ← portfolio advice view safe-buys/ ← filtered strong-buy view market-screener.db ← SQLite database (created on first boot). Contains holdings + market_calls tables. Legacy portfolio.json / market-calls.json are auto-migrated on first boot and renamed to *.json.migrated. .env ← SIMPLEFIN_ACCESS_URL or SIMPLEFIN_SETUP_TOKEN, ANTHROPIC_API_KEY, API_KEY (optional β€” enables Bearer auth on all routes) ``` --- ## Data Flow ``` Yahoo Finance API ↓ BenchmarkProvider β€” fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD builds marketContext { sp500Price, riskFreeRate, vixLevel, rateRegime, volatilityRegime, benchmarks { marketPE, techPE, reitYield, igSpread } } ↓ DataMapper β€” normalises raw Yahoo payload β†’ flat data object with type (STOCK/ETF/BOND) uses trailingPE as primary; preserves negative FCF yield; infers bond duration computes: DCF intrinsic value, analyst upside, 52W movement, grossMargin, marketCap ↓ Asset subclass β€” Stock / Etf / Bond holds metrics + getDisplayMetrics() Stock also derives capCategory (Mega/Large/Mid/Small/Micro) and growthCategory (High Growth/Growth/Stable/Value/Turnaround/Declining) ↓ RuleMerger Γ— 2 β€” FUNDAMENTAL mode: ScoringConfig as-is (Graham-style) INFLATED mode: sector override + MarketRegime live gate overrides ↓ Scorer Γ— 2 β€” StockScorer / EtfScorer / BondScorer, fully stateless ↓ ScreenerEngine β€” derives Signal from comparing both verdicts ↓ └── API path: screenTickers() β†’ JSON (with serialized displayMetrics) β†’ SvelteKit UI ``` --- ## API Routes (Fastify) | Method | Path | Description | |---|---|---| | GET | `/health` | Health check | | POST | `/api/screen` | Screen tickers. Body: `{ tickers: string[] }` (max 50). Returns `{ STOCK, ETF, BOND, ERROR, marketContext }` with `asset.displayMetrics` pre-serialized | | GET | `/api/screen/catalysts` | Yahoo news β†’ `{ tickers, stories }` | | GET | `/api/finance/portfolio` | Portfolio advice + optional SimpleFIN data | | GET | `/api/finance/market-context` | Live benchmark data only | | GET | `/api/calls` | List all market calls (newest first) | | GET | `/api/calls/:id` | Get one call + re-screened current prices for comparison | | POST | `/api/analyze` | Fetch Yahoo news for specific tickers + run LLM analysis. Body: `{ tickers: string[] }`. Returns `{ analysis }` | | POST | `/api/calls` | Create a market call; snapshots current prices. Body: `{ title, quarter, thesis, tickers[], date? }` | | DELETE | `/api/calls/:id` | Delete a market call | | GET | `/api/calls/calendar` | Earnings + dividend calendar. Query: `?tickers=AAPL,MSFT` (omit for all call tickers) | CORS is configured for `CLIENT_ORIGIN` env var (default `http://localhost:5173`). Request body validation uses Fastify JSON Schema, defined in `server/types/schemas.ts`. --- ## Scoring Modes | Mode | P/E Gate (general) | P/E Gate (tech) | Source | |---|---|---|---| | FUNDAMENTAL | 15x | 35x | ScoringConfig (true Graham) | | INFLATED | S&P 500 P/E Γ— 1.5 | XLK P/E Γ— 1.3 | Live SPY/XLK data | **Rate regime effect on INFLATED mode:** - HIGH rate regime: P/E multiplier compresses to 1.2Γ— (vs 1.5Γ— in NORMAL) - HIGH rate regime: REIT yield floor tightens (0.95Γ— vs 0.85Γ—) - HIGH rate regime: bond spread demand increases (0.90Γ— vs 0.80Γ—) | Signal | Meaning | |---|---| | βœ… Strong Buy | Passes both fundamental AND inflated gates | | ⚑ Momentum | Passes inflated, holds fundamentally | | ⚠️ Speculation | Passes inflated, fails fundamental | | πŸ”„ Neutral | Hold territory in one or both lenses | | ❌ Avoid | Fails both | --- ## ScoringConfig Key Values `server/config/ScoringConfig.ts` β€” single source of truth for all gates, weights, thresholds. **STOCK base gates (Fundamental mode):** - `maxPERatio: 15` β€” Graham's actual rule (trailing P/E) - `maxPegGate: 1.0` β€” Lynch standard: PEG > 1.0 means paying full price - `maxDebtToEquity: 1.5` β€” most distress starts above 2x - `minQuickRatio: 0.8` β€” below this signals real liquidity stress **STOCK base weights:** - `roe: 3`, `fcf: 3` β€” primary quality signals - `opMargin: 2`, `margin: 2`, `peg: 2`, `revenue: 2` β€” secondary factors - `analyst: 2` β€” Wall Street consensus (Yahoo 1–5 scale inverted; requires β‰₯3 analysts) - `dcf: 2` β€” DCF margin of safety (positive = undervalued; only fires when FCF > 0) **Sector overrides** (structural β€” apply in both modes): | Sector | Key difference | |---|---| | TECHNOLOGY | D/E up to 2.0, P/E up to 35x, FCF weight raised | | REIT | P/E and PEG disabled (9999), scored on yield + P/FFO | | FINANCIAL | D/E disabled, scored on ROE + P/B, maxPriceToBook 1.5x | | ENERGY | FCF weight 4, yield weight 3, opMargin primary | | HEALTHCARE | Revenue growth primary, P/E up to 25x | | COMMUNICATION | FCF weight 4, P/E up to 25x (META, GOOGL, NFLX) | | CONSUMER_STAPLES | Margin/ROE focus, low revenue growth expectations | | CONSUMER_DISCRETIONARY | Revenue growth primary, P/E up to 25x | **ETF gates:** - `maxExpenseRatio: 0.2%` β€” hard gate - `minFiveYearReturn: 8.0%` β€” S&P long-run floor - `minVolume: 1,000,000` ADV **BOND gates:** - `minCreditRating: 7` (BBB = investment-grade floor) - `minSpread: 1.5%` above risk-free - `maxDuration: 7` years --- ## MarketRegime (INFLATED overrides) `server/services/MarketRegime.ts` derives gate overrides from live benchmarks and current rate regime: | Gate | Formula (NORMAL rates) | Formula (HIGH rates) | |---|---|---| | Stock maxPERatio | SPY trailing P/E Γ— 1.5 | SPY trailing P/E Γ— 1.2 | | Tech maxPERatio | XLK P/E Γ— 1.3 | XLK P/E Γ— 1.3 | | Tech maxPegGate | XLK P/E Γ· 15 | XLK P/E Γ· 15 | | REIT minYield | XLRE yield Γ— 0.85 | XLRE yield Γ— 0.95 | | Bond minSpread | LQDβˆ’TNX Γ— 0.80 | LQDβˆ’TNX Γ— 0.90 | | ETF maxExpenseRatio | 0.75% | 0.75% | Rate regime thresholds: `< 2%` = LOW, `2–5%` = NORMAL, `> 5%` = HIGH (10Y Treasury yield). **Known issue**: sharp break at 5% can flip the scoring regime between two back-to-back requests when the 10Y hovers near the threshold. Consider smoothing or making the threshold configurable via env var. --- ## Expert Scoring Features (Stock) Added to `DataMapper.ts` (extraction) and `StockScorer.ts` (scoring). All data comes from Yahoo's existing modules β€” no extra API calls required. ### DCF Intrinsic Value Two-stage discounted cash flow model computed per stock with positive TTM FCF: - **Stage 1**: FCF per share grows at the earnings/revenue growth rate for 5 years, discounted at 9.5% (4% risk-free + 5.5% equity risk premium) - **Stage 2**: Terminal value at 2.5% perpetuity growth (Gordon Growth Model) - Growth rate capped at 30%, floored at -5% - Returns `dcfIntrinsicValue` ($ per share) and `dcfMarginOfSafety` (% undervaluation) Scoring: β‰₯20% margin of safety β†’ +dcf weight; 0–20% β†’ +1; -20% to 0 β†’ -1; < -20% β†’ -dcf weight. Only fires when FCF > 0. Risk flags trigger at Β±30% divergence. ### Analyst Consensus From `financialData.recommendationMean` (Yahoo scale: 1.0 = Strong Buy, 5.0 = Strong Sell): - Requires β‰₯3 analysts to avoid noise from thin coverage - ≀2.0 β†’ full weight; ≀3.0 β†’ +1; ≀4.0 β†’ -1; >4.0 β†’ -full weight - `analystTargetPrice`, `analystUpside` (% to target), `numberOfAnalysts` surfaced in display - Risk flags trigger at β‰₯25% upside or ≀-15% downside vs analyst target ### 52-Week Movement Three fields replace the single `52W Pos` position metric: | Field | Meaning | |---|---| | `52W Chg` | Total % price return over last 52 weeks (`ks['52WeekChange']`) | | `From High` | % current price is below 52-week high (negative = below peak) | | `From Low` | % current price is above 52-week low (positive = recovered) | Risk flags: strong uptrend (β‰₯+50%), significant drawdown (≀-30%), >20% off 52W high. ### Market Cap Segmentation `Stock._classifyMarketCap()` derives `capCategory` from `price.marketCap`: | Tier | Threshold | |---|---| | Mega Cap | > $200B | | Large Cap | $10B – $200B | | Mid Cap | $2B – $10B | | Small Cap | $300M – $2B | | Micro Cap | < $300M | ### Growth / Style Classification `Stock._classifyGrowth()` derives `growthCategory` from revenue growth, earnings growth, and dividend yield: | Category | Condition | |---|---| | High Growth | revenueGrowth β‰₯ 15% OR earningsGrowth β‰₯ 20% | | Growth | revenueGrowth 5–15% | | Value | revenueGrowth < 5% AND dividendYield β‰₯ 3% | | Stable | Low growth, modest or no dividend | | Turnaround | earningsGrowth < 0% AND revenueGrowth β‰₯ 0% | | Declining | revenueGrowth < -5% | Both `Cap Tier` and `Style` appear in `getDisplayMetrics()` and the screener table. --- ## Sector Detection `Stock._mapToStandardSector()` maps Yahoo Finance `sector`/`industry` strings to internal constants. Order matters β€” more specific matches first: ``` TECHNOLOGY β†’ "technology", "electronic", "semiconductor", "software" REIT β†’ "real estate", "reit" FINANCIAL β†’ "financial", "bank", "insurance", "asset management" ENERGY β†’ "energy", "oil", "gas", "petroleum" HEALTHCARE β†’ "health", "biotech", "pharmaceutical", "medical" COMMUNICATIONβ†’ "communication", "media", "entertainment", "telecom" CONSUMER_STAPLES β†’ "consumer defensive", "consumer staples", "household", "beverage", "food" CONSUMER_DISCRETIONARY β†’ "consumer cyclical", "consumer discretionary", "retail", "apparel", "auto" GENERAL β†’ fallback ``` --- ## DataMapper Notes - **peRatio**: prefers `trailingPE` (audited) over `forwardPE` (analyst estimate, ~10-15% optimistic) - **FCF yield**: `freeCashflow !== 0` (not `> 0`) β€” negative FCF preserved so cash-burning companies fail the gate, not silently skip it - **grossMargin**: `financialData.grossMargins * 100` β€” exposed in display; not yet a scoring factor - **marketCap**: `price.marketCap` β€” used for cap tier classification - **analystRating**: `financialData.recommendationMean` (1=Strong Buy, 5=Strong Sell). `targetMeanPrice` and `numberOfAnalystOpinions` also extracted - **52W movement**: `defaultKeyStatistics['52WeekChange']` for annual return; `From High`/`From Low` computed from `fiftyTwoWeekHigh`/`fiftyTwoWeekLow` - **DCF growth rate**: uses `earningsGrowth` (TTM decimal) first, falls back to `revenueGrowth * 0.7`. Capped at 30%, floored at -5% - **Bond duration**: inferred from category string ("Short-Term" β†’ 2y, "Intermediate" β†’ 5y, "Long" β†’ 18y, default 6y). Yahoo does not expose effective duration in the modules we fetch - **D/E ratio**: Yahoo returns `financialData.debtToEquity` as a percentage (e.g. 175.56 for 1.7556Γ—); dividing by 100 normalises to the standard ratio - **Quick ratio**: falls back to `currentRatio` when missing β€” known methodological issue; current ratio includes inventory and can overstate liquidity for retailers/manufacturers --- ## Missing Data Convention - Missing metrics use `null` (not `0`) in `_sanitize`. Gate checks skip `null` rather than auto-failing. - `pegRatio` falls back to `trailingPE / earningsGrowth` when Yahoo doesn't provide it. - DCF, analyst, and 52W scoring factors are all optional β€” they activate only when the underlying data is non-null. --- ## Logger Injection Pattern Classes that produce output accept an optional `{ logger }` constructor option so they work cleanly in server context. Pass `noopLogger` from `server/utils/logger.ts` to silence all output. Affected: `ScreenerEngine`, `BenchmarkProvider`, `CatalystAnalyst`, `SimpleFINClient`, `LLMAnalyst`. ```ts // CLI (default) β€” writes to stdout new ScreenerEngine() // Server β€” fully silent new ScreenerEngine({ logger: noopLogger }) ``` --- ## SimpleFIN Auth Flow 1. User gets a Setup Token from https://beta-bridge.simplefin.org 2. `SimpleFINClient.init()` base64-decodes it β†’ POSTs once to claim Access URL 3. `onAccessUrlClaimed` callback is called with the URL β€” CLI uses `saveAccessUrlToEnv()`, server stores elsewhere 4. All subsequent requests use Access URL with `Authorization: Basic` header (not embedded in URL) --- ## Holdings Format Holdings are stored in the `holdings` table in `market-screener.db`. To seed initial data, add holdings via the Portfolio UI or by inserting into the database directly. `type` values: `stock`, `etf`, `bond`, `crypto`. Crypto is priced via Yahoo (BTC-USD style) but not fundamentally scored. If you have an existing `portfolio.json`, it will be auto-migrated to SQLite on first boot and renamed to `portfolio.json.migrated`. --- ## Tests Uses Node's built-in test runner (`node:test` + `node:assert/strict`) β€” no test framework needed. ``` tests/ ScoringConfig.test.js ← gate values (P/E 15x, PEG 1.0, QuickRatio 0.8), sector overrides RuleMerger.test.js ← FUNDAMENTAL vs INFLATED modes, sector merging MarketRegime.test.js ← inflated overrides including HIGH/NORMAL rate regime variants StockScorer.test.js ← gate failures, scoring labels, risk flags EtfScorer.test.js ← expense gate, volume penalty, 5Y return scoring BondScorer.test.js ← credit gate, spread/duration scoring, unit handling DataMapper.test.js ← type detection, PEG computation, trailing PE preference, negative FCF, ETF volume, bond duration inference PortfolioAdvisor.test.js ← position gain/loss calc, advice signal mapping, BRK.B normalisation LLMAnalyst.test.js ← markdown fence stripping, JSON parse correctness ``` Pre-commit hook runs `lint-staged` (Prettier) then `npm test`. Pre-push hook runs `npm test`. Test output uses the built-in `spec` reporter. **Key unit:** `ytm` in `Bond.metrics` is stored as a percentage (e.g. `6.5` = 6.5%). `BondScorer._sanitize` divides by 100 before spread calculation. **Coverage gaps (known):** - `MarketCallRepository.ts` β€” covered by `tests/MarketCallRepository.test.ts` using in-memory SQLite - `LLMAnalyst.test.js` β€” tests a local copy of the fence-stripping regex rather than importing from source; will silently drift if the regex changes - API controllers (`server/controllers/`) β€” no integration tests; covered implicitly by manual testing only - Expert scoring features (analyst, DCF, 52W) β€” not yet covered in `StockScorer.test.js` - UI components β€” not tested at the unit level --- ## Architecture Guide This section is the single reference for where code lives and how to add features. Read this before touching any file. ### Server layer map | Folder | Role | Rule | |---|---|---| | `server/app.ts` | Fastify bootstrap β€” wires DI, registers controllers | No business logic | | `server/controllers/` | HTTP only β€” parse request, call service, return response | No business logic, no file I/O, no external API calls | | `server/services/` | Business logic and orchestration | No HTTP concerns (`req`/`reply`), no direct file I/O | | `server/repositories/` | Data persistence β€” JSON file read/write | No business logic; one class per data file | | `server/clients/` | External API connectors β€” one class per third-party system | No business logic; only I/O and protocol handling | | `server/models/` | Domain entity classes β€” hold metrics and `getDisplayMetrics()` | No I/O; pure data + formatting | | `server/scorers/` | Stateless pure scoring functions | No I/O, no state; `score(metrics, rules, marketContext)` only | | `server/config/` | Constants and scoring gates/weights | No logic; change numbers here, not in scorers | | `server/types/` | TypeScript interfaces and types | No logic; one `*.model.ts` per domain | | `server/utils/` | Shared pure utilities | No domain knowledge | | `bin/` | CLI entry points | Call into services only; the only place `process.exit()` is allowed | ### UI layer map | Path | Role | |---|---| | `ui/src/routes/*/+page.ts` | SvelteKit load functions β€” fetch data for the page | | `ui/src/routes/*/+page.svelte` | Route component β€” composition only; minimal logic | | `ui/src/lib/api/screener.ts` | Fetch wrappers for `/api/screen*` | | `ui/src/lib/api/finance.ts` | Fetch wrappers for `/api/finance/*` | | `ui/src/lib/api/calls.ts` | Fetch wrappers for `/api/calls/*` | | `ui/src/lib/stores/` | Reactive state shared across components (Svelte 5 runes) | | `ui/src/lib/components/` | Generic shared components (VerdictPill, SignalBadge, Spinner) | | `ui/src/lib/portfolio/` | Portfolio-specific components | | `ui/src/lib/calls/` | Calls-specific components | | `ui/src/lib/types.ts` | Re-exports server types via `$types` alias; UI-only types defined here | | `ui/src/lib/utils.ts` | Pure formatting and sorting helpers | | `ui/src/styles/` | SCSS partials β€” global design tokens, layout, tables, forms | ### Where to put a new type - Shared domain type (used by server + UI): `server/types/.model.ts` and re-export from `server/types/index.ts` - UI-only type (component state, display shape): `ui/src/lib/types.ts` - Private implementation detail used only within one file: inline in that file ### Where to put new code β€” decision table | What you're adding | Where it goes | |---|---| | New API endpoint | `server/controllers/.controller.ts` + register in `server/app.ts` | | Business logic for that endpoint | New method in `server/services/.ts` | | Call to a new external API | New class in `server/clients/Client.ts` | | New data stored in a JSON file | New class in `server/repositories/Repository.ts` | | New scoring rule or gate value | `server/config/ScoringConfig.ts` | | New market regime override | `server/services/MarketRegime.ts` β†’ `getInflatedOverrides()` | | New stock metric (mapped from Yahoo) | `server/services/DataMapper.ts` β†’ `mapStockData()` + `StockData`/`StockMetrics` interfaces in `server/types/models.model.ts` + `server/models/Stock.ts` constructor + `getDisplayMetrics()` | | New scoring factor | `server/config/ScoringConfig.ts` (add weight + threshold) + `server/scorers/StockScorer.ts` (add factor to array) | | New UI page | `ui/src/routes//+page.ts` + `+page.svelte` | | New UI fetch call | `ui/src/lib/api/.ts` + re-export from `api/index.ts` | | Reactive state shared by >1 component | `ui/src/lib/stores/.store.ts` | | New shared UI component | `ui/src/lib/components/` (generic) or domain subfolder | | New global style | `ui/src/styles/_.scss` + `@use` in `app.scss` | ### Conventions - Asset `type` (uppercased: `STOCK` / `ETF` / `BOND`) is the routing key across `DataMapper`, model classes, `SCORERS` map, and `ScoringRules`. Keep it consistent everywhere. - Prefer adjusting `ScoringConfig` or `MarketRegime` over hardcoding numbers in scorers. - `BenchmarkProvider` caches for 1 hour in memory β€” cache is lost on server restart. A persistent cache is planned (see Phase 8). - All entry points live in `bin/`. Do not add logic there β€” they call into `services/` and controllers. - **Never** call `process.exit()` inside `server/` β€” only `bin/` may do that. - Class instances don't survive `JSON.stringify`. Call `getDisplayMetrics()` server-side before returning from API routes (see `serializeAssets()` in `screener.controller.ts`). - Controllers use constructor injection β€” dependencies are wired in `server/app.ts`, not created inside handlers. - The `$types` alias in the UI resolves to `server/types/` β€” use it instead of duplicating type definitions. - Ticker normalisation (`BRK.B β†’ BRK-B`) happens in `YahooFinanceClient.normalise()` and applies to all callers via `fetchSummary()` and `fetchCalendarEvents()`. ### Adding a new scoring metric β€” step-by-step When adding a new data point that flows from Yahoo β†’ scoring: 1. Extract field in `DataMapper.mapStockData()` from the Yahoo payload 2. Add field to `StockData` and `StockMetrics` interfaces in `server/types/models.model.ts` 3. Add field to `Stock` constructor assignment and `getDisplayMetrics()` in `server/models/Stock.ts` 4. Add weight + threshold to `ScoringConfig.ts` base weights/thresholds (and any relevant sector overrides) 5. Add the field to `SanitizedMetrics` and `_sanitize()` in `StockScorer.ts` 6. Add a factor entry to the `factors` array in `StockScorer.score()` 7. Add a test case to `StockScorer.test.js` **Warning**: The `scorer.score(asset.metrics as never, ...)` cast in `ScreenerEngine._process()` bypasses TypeScript for the scorer dispatch. If you add a field to `StockMetrics` but forget to add it to `SanitizedMetrics`, the compiler will not catch it β€” the scorer will silently receive `undefined`. Always update `_sanitize()` when adding metrics. --- ## Architecture Roadmap ### Phases 1–7 βœ… COMPLETE All phases of the original roadmap are done. Summary of what was delivered: - Phase 1–4: CLI cleanup, shared utilities, TypeScript migration, SCSS design tokens - Phase 5: `+page.svelte` decomposed into `AssetTable`, `AnalysisSidebar`, `MarketContextStrip`, `VerdictPill` - Phase 6: Full TypeScript conversion across `server/` and `bin/` - Phase 7a–7b: Type domain split, API module split - Phase 7f: Server layer restructured to layered architecture (controllers / services / repositories / clients / models / scorers) - Phase 7g: Controllers converted to classes with DI, types moved to domain files, `YahooFinanceClient` properly typed **Pending UI work (Phase 7c–7e):** - 7c: Decompose `portfolio/+page.svelte` (751 lines) into `AddHoldingForm`, `InlineEditRow`, `AdviceTable`, `AccountsTable`; decompose `calls/+page.svelte` (385 lines) into `CallForm`, `CallCard`, `CalendarSection` - 7d: Add `ui/src/lib/stores/` layer β€” `screener.store.ts`, `portfolio.store.ts` - 7e: Extract inline `