Sai Kiran Vella
742 lines
39 KiB
Markdown
742 lines
39 KiB
Markdown
# CLAUDE.md
|
||
|
||
Guidance for working in this repository.
|
||
|
||
## Overview
|
||
|
||
`market-screener` is a Node.js project consisting of a Fastify API server that powers the SvelteKit dashboard in the `ui/` subdirectory.
|
||
|
||
Every asset is scored under two lenses:
|
||
|
||
- **Market-Adjusted** — gates derived from live Yahoo benchmarks (SPY P/E, XLK P/E, XLRE yield, LQD spread). Reflects what is acceptable in today's market.
|
||
- **Fundamental** — strict Graham/value-investing gates from `ScoringConfig`. Reflects genuine value regardless of market conditions.
|
||
|
||
The comparison produces a **Signal** (Strong Buy / Momentum / Speculation / Neutral / Avoid).
|
||
|
||
ES module project (`"type": "module"`); use `import`/`export`, not `require`.
|
||
|
||
---
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
npm install # install dependencies
|
||
npm run dev # start API server (port 3000) + SvelteKit UI (port 5173) together
|
||
npm run server # API server only (port 3000)
|
||
npm test # run all unit tests (node:test, zero external deps)
|
||
npm run test:watch # watch mode — uses verbose spec reporter
|
||
npm run format # format all server/bin/tests with Prettier
|
||
npm run format:check # check formatting without writing (used in CI/pre-commit)
|
||
npm run ui:install # install UI dependencies (ui/ subdirectory)
|
||
```
|
||
|
||
`npm run dev` runs both the API server and the SvelteKit UI (in `ui/`) concurrently. Run `npm run ui:install` once before first use.
|
||
|
||
---
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
bin/
|
||
server.ts ← Fastify API server entry point (imports buildApp from server/app.ts)
|
||
|
||
prompts/
|
||
catalyst-analysis.md ← daily catalyst analysis playbook (LLM prompt + workflow)
|
||
|
||
server/
|
||
app.ts ← Fastify app factory (buildApp). Registers CORS + all controllers.
|
||
NOTE: lives at server/app.ts, NOT inside server/controllers/.
|
||
|
||
controllers/ ← HTTP only: parse request, call service, return response
|
||
screener.controller.ts ← POST /api/screen, GET /api/screen/catalysts
|
||
finance.controller.ts ← GET /api/finance/portfolio, POST|DELETE /api/finance/holdings
|
||
calls.controller.ts ← CRUD for market calls + GET /api/calls/calendar
|
||
analyze.controller.ts ← POST /api/analyze (LLM analysis for a ticker set)
|
||
|
||
services/ ← business logic, no HTTP or I/O concerns
|
||
ScreenerEngine.ts ← orchestrates: fetch → score × 2. Method: screenTickers() → ScreenerResult.
|
||
Accepts injected YahooFinanceClient + BenchmarkProvider + { logger } option.
|
||
DataMapper.ts ← normalises Yahoo payload → flat asset data object.
|
||
Computes: DCF intrinsic value, analyst upside, 52W movement fields,
|
||
grossMargin, marketCap. Uses trailingPE. Preserves negative FCF.
|
||
RuleMerger.ts ← merges base rules + sector overrides + MarketRegime (INFLATED mode)
|
||
BenchmarkProvider.ts ← fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD → marketContext.
|
||
In-memory cache: 1 hr TTL. Resets on server restart.
|
||
MarketRegime.ts ← derives INFLATED gate overrides from live benchmarks + rate regime
|
||
CatalystAnalyst.ts ← fetches Yahoo Finance news, extracts relatedTickers. Accepts { logger }.
|
||
LLMAnalyst.ts ← uses AnthropicClient to analyze headlines → summary, sentiment,
|
||
affectedIndustries, relatedTickers. Returns null if API key not set.
|
||
PersonalFinanceAnalyzer.ts ← net worth, cash vs investments, spending by category
|
||
PortfolioAdvisor.ts ← cross-references holdings with screener signals → hold/sell/add advice
|
||
index.ts ← barrel re-export (import services from here, not individual files)
|
||
|
||
repositories/ ← data persistence only (JSON file read/write)
|
||
MarketCallRepository.ts ← persists market thesis entries to market-calls.json.
|
||
CRUD: list/get/create/delete.
|
||
PortfolioRepository.ts ← read/write portfolio.json. Methods: read, upsert, remove.
|
||
|
||
clients/ ← external API connectors, one class per third-party system
|
||
YahooFinanceClient.ts ← wraps yahoo-finance2 v3, retry + backoff. Methods: fetchSummary,
|
||
fetchCalendarEvents, search. Typed via YahooFinanceLib interface.
|
||
SimpleFINClient.ts ← claims setup token → access URL, fetches /accounts via Basic Auth.
|
||
AnthropicClient.ts ← wraps Anthropic SDK. complete(system, user) → raw text response.
|
||
|
||
models/ ← domain entity classes with metrics + display logic
|
||
Asset.ts ← abstract base: ticker, currentPrice, type, formatting helpers
|
||
Stock.ts ← metrics + _mapToStandardSector (8 sectors) + _classifyMarketCap
|
||
(Mega/Large/Mid/Small/Micro) + _classifyGrowth (style classification).
|
||
Holds: valuation, quality, risk, 52W movement, analyst consensus, DCF.
|
||
Etf.ts ← metrics: expenseRatio, yield, volume, fiveYearReturn, totalAssets
|
||
Bond.ts ← metrics: ytm, duration, creditRating, creditRatingNumeric
|
||
|
||
scorers/ ← stateless pure scoring functions, no I/O
|
||
StockScorer.ts ← gate checks + weighted registry:
|
||
core: ROE, opMargin, margin, peg, revenue, fcf
|
||
expert: analyst consensus (inverted Yahoo 1–5 scale), DCF margin of safety
|
||
riskFlags: beta, 52W position, 52W momentum, analyst divergence, DCF divergence
|
||
EtfScorer.ts ← expense gate + registry (cost, yield, volume, fiveYearReturn)
|
||
BondScorer.ts ← credit gate + spread/duration scoring
|
||
|
||
config/
|
||
ScoringConfig.ts ← CREDIT_RATING_SCALE + ScoringRules (single source of truth for all
|
||
gates, weights, thresholds including analyst and dcf weights)
|
||
constants.ts ← SIGNAL, ASSET_TYPE, SECTOR, SCORE_MODE, REGIME, SIGNAL_ORDER,
|
||
CAP_CATEGORY (Mega/Large/Mid/Small/Micro),
|
||
GROWTH_CATEGORY (High Growth/Growth/Stable/Value/Turnaround/Declining)
|
||
|
||
types/ ← all domain types, one file per domain
|
||
asset.model.ts ← Signal, AssetType, ScoreMode, ScoringRules, ScoreResult, AssetResult,
|
||
ScreenerResult, GateSet, WeightSet, ThresholdSet, RuleBlock, StockRules
|
||
market.model.ts ← RateRegime, VolatilityRegime, Benchmarks, MarketContext
|
||
portfolio.model.ts ← HoldingType, PortfolioHolding, PortfolioAdvice, AdviceRow
|
||
calls.model.ts ← TickerSnapshot, MarketCall, SnapshotEntry, CreateCallInput
|
||
finance.model.ts ← LLMAnalysis, CatalystStory, CalendarEvent, SimpleFINAccount,
|
||
SimpleFINTransaction, SimpleFINData, YahooNewsItem, YahooFinanceLib
|
||
logger.model.ts ← Logger
|
||
models.model.ts ← AssetData, StockData, StockMetrics, EtfData, EtfMetrics, BondData, BondMetrics
|
||
repositories.model.ts ← StoreData, PortfolioData
|
||
scorers.model.ts ← NumVal, SanitizedMetrics, SanitizedBondMetrics
|
||
services.model.ts ← BenchmarkProviderOptions, InflatedOverrides, ErrorResult, RuleSet,
|
||
ScreenerEngineOptions, CatalystResult, MappedData, and other service shapes
|
||
schemas.ts ← Fastify JSON Schema objects for request body validation
|
||
index.ts ← re-exports all of the above (use this barrel for imports)
|
||
types.ts ← thin barrel: export type * from './types/index.js'
|
||
|
||
utils/
|
||
Chunker.ts ← splits ticker list into batches
|
||
logger.ts ← noopLogger constant for silencing output in server context
|
||
|
||
ui/ ← SvelteKit dashboard (lives inside this repo, not a separate repo)
|
||
src/
|
||
styles/ ← global SCSS design-token system
|
||
app.scss ← root file — @use all partials
|
||
_tokens.scss ← CSS custom properties generated from SCSS maps ($bg, $text, $blues, $signals…)
|
||
_reset.scss ← box-sizing reset + body base
|
||
_layout.scss ← shell, nav, .links, main, nav-progress, .loading-area
|
||
_section.scss ← .section, .section-header, .count, .mode-tabs, .error-banner
|
||
_table.scss ← table, thead/tbody, .table-wrap, .col-ticker, .ticker, .num, .tag
|
||
_buttons.scss ← button base, .btn-primary, .btn-catalyst, .btn-ghost, .btn-screen, .btn-analyze
|
||
_badges.scss ← .verdict-pill, .sentiment-pill, .text-* helpers (SCSS @each maps)
|
||
lib/
|
||
api.ts ← backward-compat shim; re-exports from api/index.ts
|
||
api/
|
||
screener.ts ← screenTickers, fetchCatalysts, analyzeTickers
|
||
finance.ts ← fetchPortfolio, addHolding, removeHolding, fetchMarketContext
|
||
calls.ts ← fetchCalls, fetchCall, createCall, deleteCall, fetchCallsCalendar
|
||
index.ts ← barrel re-export
|
||
types.ts ← re-exports shared types from $types (server/types/); UI-only types defined here
|
||
utils.ts ← shared pure functions: sigOrd, sorted, verdictShort, vClass, fmtPE, fmt…
|
||
MarketContext.svelte ← collapsible card-grid context (used in portfolio + safe-buys)
|
||
MarketContextStrip.svelte ← horizontal chip strip (used in screener)
|
||
AssetTable.svelte ← STOCK/ETF/BOND section: mode tabs + Analyze + table
|
||
AnalysisSidebar.svelte ← LLM analysis slide-over panel
|
||
VerdictPill.svelte ← verdict-pill span; props: label
|
||
SignalBadge.svelte ← signal emoji + label badge
|
||
Spinner.svelte ← sm: dot-pulse | md/lg: chart-line animation
|
||
routes/
|
||
+page.ts ← SvelteKit load (ssr:false) — fetches catalysts + screens on mount
|
||
+page.svelte ← main screener UI
|
||
+layout.svelte ← shell, nav, nav-progress bar, nav-overlay with Spinner
|
||
calls/ ← market calls list + detail views
|
||
portfolio/ ← portfolio advice view
|
||
safe-buys/ ← filtered strong-buy view
|
||
|
||
market-calls.json ← persisted market thesis calls (written by MarketCallRepository)
|
||
portfolio.json ← user's holdings: ticker, shares, costBasis, source, type
|
||
.env ← SIMPLEFIN_ACCESS_URL or SIMPLEFIN_SETUP_TOKEN, ANTHROPIC_API_KEY
|
||
```
|
||
|
||
---
|
||
|
||
## Data Flow
|
||
|
||
```
|
||
Yahoo Finance API
|
||
↓
|
||
BenchmarkProvider — fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD
|
||
builds marketContext { sp500Price, riskFreeRate, vixLevel,
|
||
rateRegime, volatilityRegime, benchmarks { marketPE, techPE, reitYield, igSpread } }
|
||
↓
|
||
DataMapper — normalises raw Yahoo payload → flat data object with type (STOCK/ETF/BOND)
|
||
uses trailingPE as primary; preserves negative FCF yield; infers bond duration
|
||
computes: DCF intrinsic value, analyst upside, 52W movement, grossMargin, marketCap
|
||
↓
|
||
Asset subclass — Stock / Etf / Bond holds metrics + getDisplayMetrics()
|
||
Stock also derives capCategory (Mega/Large/Mid/Small/Micro)
|
||
and growthCategory (High Growth/Growth/Stable/Value/Turnaround/Declining)
|
||
↓
|
||
RuleMerger × 2 — FUNDAMENTAL mode: ScoringConfig as-is (Graham-style)
|
||
INFLATED mode: sector override + MarketRegime live gate overrides
|
||
↓
|
||
Scorer × 2 — StockScorer / EtfScorer / BondScorer, fully stateless
|
||
↓
|
||
ScreenerEngine — derives Signal from comparing both verdicts
|
||
↓
|
||
└── API path: screenTickers() → JSON (with serialized displayMetrics) → SvelteKit UI
|
||
```
|
||
|
||
---
|
||
|
||
## API Routes (Fastify)
|
||
|
||
| Method | Path | Description |
|
||
|---|---|---|
|
||
| GET | `/health` | Health check |
|
||
| POST | `/api/screen` | Screen tickers. Body: `{ tickers: string[] }` (max 50). Returns `{ STOCK, ETF, BOND, ERROR, marketContext }` with `asset.displayMetrics` pre-serialized |
|
||
| GET | `/api/screen/catalysts` | Yahoo news → `{ tickers, stories }` |
|
||
| GET | `/api/finance/portfolio` | Portfolio advice + optional SimpleFIN data |
|
||
| GET | `/api/finance/market-context` | Live benchmark data only |
|
||
| GET | `/api/calls` | List all market calls (newest first) |
|
||
| GET | `/api/calls/:id` | Get one call + re-screened current prices for comparison |
|
||
| POST | `/api/analyze` | Fetch Yahoo news for specific tickers + run LLM analysis. Body: `{ tickers: string[] }`. Returns `{ analysis }` |
|
||
| POST | `/api/calls` | Create a market call; snapshots current prices. Body: `{ title, quarter, thesis, tickers[], date? }` |
|
||
| DELETE | `/api/calls/:id` | Delete a market call |
|
||
| GET | `/api/calls/calendar` | Earnings + dividend calendar. Query: `?tickers=AAPL,MSFT` (omit for all call tickers) |
|
||
|
||
CORS is configured for `CLIENT_ORIGIN` env var (default `http://localhost:5173`).
|
||
|
||
Request body validation uses Fastify JSON Schema, defined in `server/types/schemas.ts`.
|
||
|
||
---
|
||
|
||
## Scoring Modes
|
||
|
||
| Mode | P/E Gate (general) | P/E Gate (tech) | Source |
|
||
|---|---|---|---|
|
||
| FUNDAMENTAL | 15x | 35x | ScoringConfig (true Graham) |
|
||
| INFLATED | S&P 500 P/E × 1.5 | XLK P/E × 1.3 | Live SPY/XLK data |
|
||
|
||
**Rate regime effect on INFLATED mode:**
|
||
- HIGH rate regime: P/E multiplier compresses to 1.2× (vs 1.5× in NORMAL)
|
||
- HIGH rate regime: REIT yield floor tightens (0.95× vs 0.85×)
|
||
- HIGH rate regime: bond spread demand increases (0.90× vs 0.80×)
|
||
|
||
| Signal | Meaning |
|
||
|---|---|
|
||
| ✅ Strong Buy | Passes both fundamental AND inflated gates |
|
||
| ⚡ Momentum | Passes inflated, holds fundamentally |
|
||
| ⚠️ Speculation | Passes inflated, fails fundamental |
|
||
| 🔄 Neutral | Hold territory in one or both lenses |
|
||
| ❌ Avoid | Fails both |
|
||
|
||
---
|
||
|
||
## ScoringConfig Key Values
|
||
|
||
`server/config/ScoringConfig.ts` — single source of truth for all gates, weights, thresholds.
|
||
|
||
**STOCK base gates (Fundamental mode):**
|
||
- `maxPERatio: 15` — Graham's actual rule (trailing P/E)
|
||
- `maxPegGate: 1.0` — Lynch standard: PEG > 1.0 means paying full price
|
||
- `maxDebtToEquity: 1.5` — most distress starts above 2x
|
||
- `minQuickRatio: 0.8` — below this signals real liquidity stress
|
||
|
||
**STOCK base weights:**
|
||
- `roe: 3`, `fcf: 3` — primary quality signals
|
||
- `opMargin: 2`, `margin: 2`, `peg: 2`, `revenue: 2` — secondary factors
|
||
- `analyst: 2` — Wall Street consensus (Yahoo 1–5 scale inverted; requires ≥3 analysts)
|
||
- `dcf: 2` — DCF margin of safety (positive = undervalued; only fires when FCF > 0)
|
||
|
||
**Sector overrides** (structural — apply in both modes):
|
||
|
||
| Sector | Key difference |
|
||
|---|---|
|
||
| TECHNOLOGY | D/E up to 2.0, P/E up to 35x, FCF weight raised |
|
||
| REIT | P/E and PEG disabled (9999), scored on yield + P/FFO |
|
||
| FINANCIAL | D/E disabled, scored on ROE + P/B, maxPriceToBook 1.5x |
|
||
| ENERGY | FCF weight 4, yield weight 3, opMargin primary |
|
||
| HEALTHCARE | Revenue growth primary, P/E up to 25x |
|
||
| COMMUNICATION | FCF weight 4, P/E up to 25x (META, GOOGL, NFLX) |
|
||
| CONSUMER_STAPLES | Margin/ROE focus, low revenue growth expectations |
|
||
| CONSUMER_DISCRETIONARY | Revenue growth primary, P/E up to 25x |
|
||
|
||
**ETF gates:**
|
||
- `maxExpenseRatio: 0.2%` — hard gate
|
||
- `minFiveYearReturn: 8.0%` — S&P long-run floor
|
||
- `minVolume: 1,000,000` ADV
|
||
|
||
**BOND gates:**
|
||
- `minCreditRating: 7` (BBB = investment-grade floor)
|
||
- `minSpread: 1.5%` above risk-free
|
||
- `maxDuration: 7` years
|
||
|
||
---
|
||
|
||
## MarketRegime (INFLATED overrides)
|
||
|
||
`server/services/MarketRegime.ts` derives gate overrides from live benchmarks and current rate regime:
|
||
|
||
| Gate | Formula (NORMAL rates) | Formula (HIGH rates) |
|
||
|---|---|---|
|
||
| Stock maxPERatio | SPY trailing P/E × 1.5 | SPY trailing P/E × 1.2 |
|
||
| Tech maxPERatio | XLK P/E × 1.3 | XLK P/E × 1.3 |
|
||
| Tech maxPegGate | XLK P/E ÷ 15 | XLK P/E ÷ 15 |
|
||
| REIT minYield | XLRE yield × 0.85 | XLRE yield × 0.95 |
|
||
| Bond minSpread | LQD−TNX × 0.80 | LQD−TNX × 0.90 |
|
||
| ETF maxExpenseRatio | 0.75% | 0.75% |
|
||
|
||
Rate regime thresholds: `< 2%` = LOW, `2–5%` = NORMAL, `> 5%` = HIGH (10Y Treasury yield).
|
||
**Known issue**: sharp break at 5% can flip the scoring regime between two back-to-back requests when the 10Y hovers near the threshold. Consider smoothing or making the threshold configurable via env var.
|
||
|
||
---
|
||
|
||
## Expert Scoring Features (Stock)
|
||
|
||
Added to `DataMapper.ts` (extraction) and `StockScorer.ts` (scoring).
|
||
All data comes from Yahoo's existing modules — no extra API calls required.
|
||
|
||
### DCF Intrinsic Value
|
||
|
||
Two-stage discounted cash flow model computed per stock with positive TTM FCF:
|
||
|
||
- **Stage 1**: FCF per share grows at the earnings/revenue growth rate for 5 years, discounted at 9.5% (4% risk-free + 5.5% equity risk premium)
|
||
- **Stage 2**: Terminal value at 2.5% perpetuity growth (Gordon Growth Model)
|
||
- Growth rate capped at 30%, floored at -5%
|
||
- Returns `dcfIntrinsicValue` ($ per share) and `dcfMarginOfSafety` (% undervaluation)
|
||
|
||
Scoring: ≥20% margin of safety → +dcf weight; 0–20% → +1; -20% to 0 → -1; < -20% → -dcf weight.
|
||
Only fires when FCF > 0. Risk flags trigger at ±30% divergence.
|
||
|
||
### Analyst Consensus
|
||
|
||
From `financialData.recommendationMean` (Yahoo scale: 1.0 = Strong Buy, 5.0 = Strong Sell):
|
||
|
||
- Requires ≥3 analysts to avoid noise from thin coverage
|
||
- ≤2.0 → full weight; ≤3.0 → +1; ≤4.0 → -1; >4.0 → -full weight
|
||
- `analystTargetPrice`, `analystUpside` (% to target), `numberOfAnalysts` surfaced in display
|
||
- Risk flags trigger at ≥25% upside or ≤-15% downside vs analyst target
|
||
|
||
### 52-Week Movement
|
||
|
||
Three fields replace the single `52W Pos` position metric:
|
||
|
||
| Field | Meaning |
|
||
|---|---|
|
||
| `52W Chg` | Total % price return over last 52 weeks (`ks['52WeekChange']`) |
|
||
| `From High` | % current price is below 52-week high (negative = below peak) |
|
||
| `From Low` | % current price is above 52-week low (positive = recovered) |
|
||
|
||
Risk flags: strong uptrend (≥+50%), significant drawdown (≤-30%), >20% off 52W high.
|
||
|
||
### Market Cap Segmentation
|
||
|
||
`Stock._classifyMarketCap()` derives `capCategory` from `price.marketCap`:
|
||
|
||
| Tier | Threshold |
|
||
|---|---|
|
||
| Mega Cap | > $200B |
|
||
| Large Cap | $10B – $200B |
|
||
| Mid Cap | $2B – $10B |
|
||
| Small Cap | $300M – $2B |
|
||
| Micro Cap | < $300M |
|
||
|
||
### Growth / Style Classification
|
||
|
||
`Stock._classifyGrowth()` derives `growthCategory` from revenue growth, earnings growth, and dividend yield:
|
||
|
||
| Category | Condition |
|
||
|---|---|
|
||
| High Growth | revenueGrowth ≥ 15% OR earningsGrowth ≥ 20% |
|
||
| Growth | revenueGrowth 5–15% |
|
||
| Value | revenueGrowth < 5% AND dividendYield ≥ 3% |
|
||
| Stable | Low growth, modest or no dividend |
|
||
| Turnaround | earningsGrowth < 0% AND revenueGrowth ≥ 0% |
|
||
| Declining | revenueGrowth < -5% |
|
||
|
||
Both `Cap Tier` and `Style` appear in `getDisplayMetrics()` and the screener table.
|
||
|
||
---
|
||
|
||
## Sector Detection
|
||
|
||
`Stock._mapToStandardSector()` maps Yahoo Finance `sector`/`industry` strings to internal constants.
|
||
Order matters — more specific matches first:
|
||
|
||
```
|
||
TECHNOLOGY → "technology", "electronic", "semiconductor", "software"
|
||
REIT → "real estate", "reit"
|
||
FINANCIAL → "financial", "bank", "insurance", "asset management"
|
||
ENERGY → "energy", "oil", "gas", "petroleum"
|
||
HEALTHCARE → "health", "biotech", "pharmaceutical", "medical"
|
||
COMMUNICATION→ "communication", "media", "entertainment", "telecom"
|
||
CONSUMER_STAPLES → "consumer defensive", "consumer staples", "household", "beverage", "food"
|
||
CONSUMER_DISCRETIONARY → "consumer cyclical", "consumer discretionary", "retail", "apparel", "auto"
|
||
GENERAL → fallback
|
||
```
|
||
|
||
---
|
||
|
||
## DataMapper Notes
|
||
|
||
- **peRatio**: prefers `trailingPE` (audited) over `forwardPE` (analyst estimate, ~10-15% optimistic)
|
||
- **FCF yield**: `freeCashflow !== 0` (not `> 0`) — negative FCF preserved so cash-burning companies fail the gate, not silently skip it
|
||
- **grossMargin**: `financialData.grossMargins * 100` — exposed in display; not yet a scoring factor
|
||
- **marketCap**: `price.marketCap` — used for cap tier classification
|
||
- **analystRating**: `financialData.recommendationMean` (1=Strong Buy, 5=Strong Sell). `targetMeanPrice` and `numberOfAnalystOpinions` also extracted
|
||
- **52W movement**: `defaultKeyStatistics['52WeekChange']` for annual return; `From High`/`From Low` computed from `fiftyTwoWeekHigh`/`fiftyTwoWeekLow`
|
||
- **DCF growth rate**: uses `earningsGrowth` (TTM decimal) first, falls back to `revenueGrowth * 0.7`. Capped at 30%, floored at -5%
|
||
- **Bond duration**: inferred from category string ("Short-Term" → 2y, "Intermediate" → 5y, "Long" → 18y, default 6y). Yahoo does not expose effective duration in the modules we fetch
|
||
- **D/E ratio**: Yahoo returns `financialData.debtToEquity` as a percentage (e.g. 175.56 for 1.7556×); dividing by 100 normalises to the standard ratio
|
||
- **Quick ratio**: falls back to `currentRatio` when missing — known methodological issue; current ratio includes inventory and can overstate liquidity for retailers/manufacturers
|
||
|
||
---
|
||
|
||
## Missing Data Convention
|
||
|
||
- Missing metrics use `null` (not `0`) in `_sanitize`. Gate checks skip `null` rather than auto-failing.
|
||
- `pegRatio` falls back to `trailingPE / earningsGrowth` when Yahoo doesn't provide it.
|
||
- DCF, analyst, and 52W scoring factors are all optional — they activate only when the underlying data is non-null.
|
||
|
||
---
|
||
|
||
## Logger Injection Pattern
|
||
|
||
Classes that produce output accept an optional `{ logger }` constructor option so they work cleanly in server context. Pass `noopLogger` from `server/utils/logger.ts` to silence all output.
|
||
|
||
Affected: `ScreenerEngine`, `BenchmarkProvider`, `CatalystAnalyst`, `SimpleFINClient`, `LLMAnalyst`.
|
||
|
||
```ts
|
||
// CLI (default) — writes to stdout
|
||
new ScreenerEngine()
|
||
|
||
// Server — fully silent
|
||
new ScreenerEngine({ logger: noopLogger })
|
||
```
|
||
|
||
---
|
||
|
||
## SimpleFIN Auth Flow
|
||
|
||
1. User gets a Setup Token from https://beta-bridge.simplefin.org
|
||
2. `SimpleFINClient.init()` base64-decodes it → POSTs once to claim Access URL
|
||
3. `onAccessUrlClaimed` callback is called with the URL — CLI uses `saveAccessUrlToEnv()`, server stores elsewhere
|
||
4. All subsequent requests use Access URL with `Authorization: Basic` header (not embedded in URL)
|
||
|
||
---
|
||
|
||
## portfolio.json Format
|
||
|
||
```json
|
||
{
|
||
"holdings": [
|
||
{ "ticker": "AAPL", "shares": 10, "costBasis": 150.00, "source": "Robinhood", "type": "stock" },
|
||
{ "ticker": "VOO", "shares": 8, "costBasis": 380.00, "source": "Vanguard", "type": "etf" },
|
||
{ "ticker": "BTC-USD", "shares": 0.25, "costBasis": 45000, "source": "Coinbase", "type": "crypto" }
|
||
]
|
||
}
|
||
```
|
||
|
||
`type` values: `stock`, `etf`, `crypto`. Crypto is priced via Yahoo (BTC-USD style) but not fundamentally scored.
|
||
|
||
---
|
||
|
||
## Tests
|
||
|
||
Uses Node's built-in test runner (`node:test` + `node:assert/strict`) — no test framework needed.
|
||
|
||
```
|
||
tests/
|
||
ScoringConfig.test.js ← gate values (P/E 15x, PEG 1.0, QuickRatio 0.8), sector overrides
|
||
RuleMerger.test.js ← FUNDAMENTAL vs INFLATED modes, sector merging
|
||
MarketRegime.test.js ← inflated overrides including HIGH/NORMAL rate regime variants
|
||
StockScorer.test.js ← gate failures, scoring labels, risk flags
|
||
EtfScorer.test.js ← expense gate, volume penalty, 5Y return scoring
|
||
BondScorer.test.js ← credit gate, spread/duration scoring, unit handling
|
||
DataMapper.test.js ← type detection, PEG computation, trailing PE preference,
|
||
negative FCF, ETF volume, bond duration inference
|
||
PortfolioAdvisor.test.js ← position gain/loss calc, advice signal mapping, BRK.B normalisation
|
||
LLMAnalyst.test.js ← markdown fence stripping, JSON parse correctness
|
||
```
|
||
|
||
Pre-commit hook runs `lint-staged` (Prettier) then `npm test`. Pre-push hook runs `npm test`.
|
||
Test output uses the built-in `spec` reporter.
|
||
|
||
**Key unit:** `ytm` in `Bond.metrics` is stored as a percentage (e.g. `6.5` = 6.5%). `BondScorer._sanitize` divides by 100 before spread calculation.
|
||
|
||
**Coverage gaps (known):**
|
||
- `MarketCallRepository.ts` — no tests; CRUD against `market-calls.json` is untested
|
||
- `LLMAnalyst.test.js` — tests a local copy of the fence-stripping regex rather than importing from source; will silently drift if the regex changes
|
||
- API controllers (`server/controllers/`) — no integration tests; covered implicitly by manual testing only
|
||
- Expert scoring features (analyst, DCF, 52W) — not yet covered in `StockScorer.test.js`
|
||
- UI components — not tested at the unit level
|
||
|
||
---
|
||
|
||
## Architecture Guide
|
||
|
||
This section is the single reference for where code lives and how to add features. Read this before touching any file.
|
||
|
||
### Server layer map
|
||
|
||
| Folder | Role | Rule |
|
||
|---|---|---|
|
||
| `server/app.ts` | Fastify bootstrap — wires DI, registers controllers | No business logic |
|
||
| `server/controllers/` | HTTP only — parse request, call service, return response | No business logic, no file I/O, no external API calls |
|
||
| `server/services/` | Business logic and orchestration | No HTTP concerns (`req`/`reply`), no direct file I/O |
|
||
| `server/repositories/` | Data persistence — JSON file read/write | No business logic; one class per data file |
|
||
| `server/clients/` | External API connectors — one class per third-party system | No business logic; only I/O and protocol handling |
|
||
| `server/models/` | Domain entity classes — hold metrics and `getDisplayMetrics()` | No I/O; pure data + formatting |
|
||
| `server/scorers/` | Stateless pure scoring functions | No I/O, no state; `score(metrics, rules, marketContext)` only |
|
||
| `server/config/` | Constants and scoring gates/weights | No logic; change numbers here, not in scorers |
|
||
| `server/types/` | TypeScript interfaces and types | No logic; one `*.model.ts` per domain |
|
||
| `server/utils/` | Shared pure utilities | No domain knowledge |
|
||
| `bin/` | CLI entry points | Call into services only; the only place `process.exit()` is allowed |
|
||
|
||
### UI layer map
|
||
|
||
| Path | Role |
|
||
|---|---|
|
||
| `ui/src/routes/*/+page.ts` | SvelteKit load functions — fetch data for the page |
|
||
| `ui/src/routes/*/+page.svelte` | Route component — composition only; minimal logic |
|
||
| `ui/src/lib/api/screener.ts` | Fetch wrappers for `/api/screen*` |
|
||
| `ui/src/lib/api/finance.ts` | Fetch wrappers for `/api/finance/*` |
|
||
| `ui/src/lib/api/calls.ts` | Fetch wrappers for `/api/calls/*` |
|
||
| `ui/src/lib/stores/` | Reactive state shared across components (Svelte 5 runes) |
|
||
| `ui/src/lib/components/` | Generic shared components (VerdictPill, SignalBadge, Spinner) |
|
||
| `ui/src/lib/portfolio/` | Portfolio-specific components |
|
||
| `ui/src/lib/calls/` | Calls-specific components |
|
||
| `ui/src/lib/types.ts` | Re-exports server types via `$types` alias; UI-only types defined here |
|
||
| `ui/src/lib/utils.ts` | Pure formatting and sorting helpers |
|
||
| `ui/src/styles/` | SCSS partials — global design tokens, layout, tables, forms |
|
||
|
||
### Where to put a new type
|
||
|
||
- Shared domain type (used by server + UI): `server/types/<domain>.model.ts` and re-export from `server/types/index.ts`
|
||
- UI-only type (component state, display shape): `ui/src/lib/types.ts`
|
||
- Private implementation detail used only within one file: inline in that file
|
||
|
||
### Where to put new code — decision table
|
||
|
||
| What you're adding | Where it goes |
|
||
|---|---|
|
||
| New API endpoint | `server/controllers/<domain>.controller.ts` + register in `server/app.ts` |
|
||
| Business logic for that endpoint | New method in `server/services/<Domain>.ts` |
|
||
| Call to a new external API | New class in `server/clients/<Service>Client.ts` |
|
||
| New data stored in a JSON file | New class in `server/repositories/<Domain>Repository.ts` |
|
||
| New scoring rule or gate value | `server/config/ScoringConfig.ts` |
|
||
| New market regime override | `server/services/MarketRegime.ts` → `getInflatedOverrides()` |
|
||
| New stock metric (mapped from Yahoo) | `server/services/DataMapper.ts` → `mapStockData()` + `StockData`/`StockMetrics` interfaces in `server/types/models.model.ts` + `server/models/Stock.ts` constructor + `getDisplayMetrics()` |
|
||
| New scoring factor | `server/config/ScoringConfig.ts` (add weight + threshold) + `server/scorers/StockScorer.ts` (add factor to array) |
|
||
| New UI page | `ui/src/routes/<name>/+page.ts` + `+page.svelte` |
|
||
| New UI fetch call | `ui/src/lib/api/<domain>.ts` + re-export from `api/index.ts` |
|
||
| Reactive state shared by >1 component | `ui/src/lib/stores/<domain>.store.ts` |
|
||
| New shared UI component | `ui/src/lib/components/` (generic) or domain subfolder |
|
||
| New global style | `ui/src/styles/_<domain>.scss` + `@use` in `app.scss` |
|
||
|
||
### Conventions
|
||
|
||
- Asset `type` (uppercased: `STOCK` / `ETF` / `BOND`) is the routing key across `DataMapper`, model classes, `SCORERS` map, and `ScoringRules`. Keep it consistent everywhere.
|
||
- Prefer adjusting `ScoringConfig` or `MarketRegime` over hardcoding numbers in scorers.
|
||
- `BenchmarkProvider` caches for 1 hour in memory — cache is lost on server restart. A persistent cache is planned (see Phase 8).
|
||
- All entry points live in `bin/`. Do not add logic there — they call into `services/` and controllers.
|
||
- **Never** call `process.exit()` inside `server/` — only `bin/` may do that.
|
||
- Class instances don't survive `JSON.stringify`. Call `getDisplayMetrics()` server-side before returning from API routes (see `serializeAssets()` in `screener.controller.ts`).
|
||
- Controllers use constructor injection — dependencies are wired in `server/app.ts`, not created inside handlers.
|
||
- The `$types` alias in the UI resolves to `server/types/` — use it instead of duplicating type definitions.
|
||
- Ticker normalisation (`BRK.B → BRK-B`) happens in `YahooFinanceClient.normalise()` and applies to all callers via `fetchSummary()` and `fetchCalendarEvents()`.
|
||
|
||
### Adding a new scoring metric — step-by-step
|
||
|
||
When adding a new data point that flows from Yahoo → scoring:
|
||
|
||
1. Extract field in `DataMapper.mapStockData()` from the Yahoo payload
|
||
2. Add field to `StockData` and `StockMetrics` interfaces in `server/types/models.model.ts`
|
||
3. Add field to `Stock` constructor assignment and `getDisplayMetrics()` in `server/models/Stock.ts`
|
||
4. Add weight + threshold to `ScoringConfig.ts` base weights/thresholds (and any relevant sector overrides)
|
||
5. Add the field to `SanitizedMetrics` and `_sanitize()` in `StockScorer.ts`
|
||
6. Add a factor entry to the `factors` array in `StockScorer.score()`
|
||
7. Add a test case to `StockScorer.test.js`
|
||
|
||
**Warning**: The `scorer.score(asset.metrics as never, ...)` cast in `ScreenerEngine._process()` bypasses TypeScript for the scorer dispatch. If you add a field to `StockMetrics` but forget to add it to `SanitizedMetrics`, the compiler will not catch it — the scorer will silently receive `undefined`. Always update `_sanitize()` when adding metrics.
|
||
|
||
---
|
||
|
||
## Architecture Roadmap
|
||
|
||
### Phases 1–7 ✅ COMPLETE
|
||
|
||
All phases of the original roadmap are done. Summary of what was delivered:
|
||
- Phase 1–4: CLI cleanup, shared utilities, TypeScript migration, SCSS design tokens
|
||
- Phase 5: `+page.svelte` decomposed into `AssetTable`, `AnalysisSidebar`, `MarketContextStrip`, `VerdictPill`
|
||
- Phase 6: Full TypeScript conversion across `server/` and `bin/`
|
||
- Phase 7a–7b: Type domain split, API module split
|
||
- Phase 7f: Server layer restructured to layered architecture (controllers / services / repositories / clients / models / scorers)
|
||
- Phase 7g: Controllers converted to classes with DI, types moved to domain files, `YahooFinanceClient` properly typed
|
||
|
||
**Pending UI work (Phase 7c–7e):**
|
||
- 7c: Decompose `portfolio/+page.svelte` (751 lines) into `AddHoldingForm`, `InlineEditRow`, `AdviceTable`, `AccountsTable`; decompose `calls/+page.svelte` (385 lines) into `CallForm`, `CallCard`, `CalendarSection`
|
||
- 7d: Add `ui/src/lib/stores/` layer — `screener.store.ts`, `portfolio.store.ts`
|
||
- 7e: Extract inline `<style>` blocks from portfolio, calls, and AnalysisSidebar into `_forms.scss`, `_sidebar.scss`, `_calls.scss`, `_portfolio.scss`
|
||
|
||
---
|
||
|
||
### Phase 8 — Server Hardening & Quality
|
||
|
||
Priority order. Complete earlier items before starting later ones.
|
||
|
||
#### 8a — Fix `as never` scorer dispatch in `ScreenerEngine`
|
||
|
||
`scorer.score(asset.metrics as never, ...)` bypasses TypeScript. Replace with a properly typed discriminated union or per-type dispatch so the compiler can verify that `StockMetrics` matches `StockScorer.score()`'s signature. This is the highest-risk cast in the scoring pipeline.
|
||
|
||
#### 8b — Inject dependencies into `ScreenerEngine` and `PortfolioAdvisor`
|
||
|
||
Both classes self-construct their `YahooFinanceClient` and `BenchmarkProvider` in the constructor, making unit testing impossible without monkey-patching. Target:
|
||
|
||
```ts
|
||
export class ScreenerEngine {
|
||
constructor(
|
||
private readonly client: YahooFinanceClient,
|
||
private readonly benchmarkProvider: BenchmarkProvider,
|
||
{ logger }: ScreenerEngineOptions = {},
|
||
) {}
|
||
}
|
||
```
|
||
|
||
Wire in `server/app.ts`. This unblocks proper service-layer unit tests.
|
||
|
||
#### 8c — Controller integration tests
|
||
|
||
Add one Fastify `inject()` smoke test per route using a fixture for `ScreenerEngine.screenTickers()`. Catches schema validation regressions and response shape changes without needing live Yahoo access. Target: `tests/screener.controller.test.js`, `tests/calls.controller.test.js`.
|
||
|
||
#### 8d — Repository tests
|
||
|
||
`MarketCallRepository` has zero test coverage. Add `tests/MarketCallRepository.test.js` using a temp file path (inject via constructor or env var) to test `list`, `create`, `delete`, and concurrent-write safety.
|
||
|
||
#### 8f — Persistent benchmark cache
|
||
|
||
`BenchmarkProvider`'s 1-hour cache is in-memory only — cold start after every restart adds 2–4s latency to the first request. Write the cached `MarketContext` to `.benchmark-cache.json` (or a single-row SQLite table). Read it on boot; only re-fetch if stale.
|
||
|
||
#### 8g — Rate limiting + API key auth
|
||
|
||
Add `@fastify/rate-limit` on `/api/screen` and `/api/analyze` (e.g. 10 req/min per IP). Add a simple `Authorization: Bearer <key>` check against an `API_KEY` env var as middleware in `server/app.ts`. Both are single-digit line additions.
|
||
|
||
#### 8h — Extract `CalendarService`
|
||
|
||
`CallsController.calendar()` is 80+ lines of inline event construction, date parsing, and sorting — all inside a controller method. Extract to `server/services/CalendarService.ts` to make it testable and keep the controller under 50 lines.
|
||
|
||
#### 8i — SQLite migration for repositories
|
||
|
||
Both `market-calls.json` and `portfolio.json` use `writeFileSync` with no concurrency guard. Two concurrent writes within the same event loop tick will lose one write. Replace with `better-sqlite3` for both repositories: concurrent-write safe, atomic transactions, no extra infrastructure. At current portfolio sizes the footprint is trivial.
|
||
|
||
#### 8j — Cache `CatalystAnalyst` results
|
||
|
||
`CatalystAnalyst.run()` fires fresh Yahoo news queries on every `/api/screen/catalysts` call. Cache the result for 15 minutes. A new `CatalystAnalyst` instance is also created on each call inside `ScreenerController.catalysts()` — hoist it to a class-level singleton wired in `server/app.ts`.
|
||
|
||
#### 8k — Add expert scoring tests
|
||
|
||
Update `StockScorer.test.js` to cover the three new scoring factors: analyst consensus scoring (including the `numberOfAnalysts < 3` guard), DCF margin of safety scoring (positive/negative/null cases), and the new 52W risk flags.
|
||
|
||
#### 8l — Anthropic prompt caching for LLMAnalyst
|
||
|
||
`LLMAnalyst.analyze()` sends a large system prompt on every `/api/analyze` call. Enabling Anthropic prompt caching would cache the static system prompt across calls, reducing latency and token costs significantly.
|
||
|
||
Target: add `cache_control: { type: 'ephemeral' }` to the system prompt message block in `AnthropicClient.complete()` (or in `LLMAnalyst.analyze()` if the system prompt is built there). Use the `anthropic-beta: prompt-caching-2024-07-31` header. The cache has a 5-minute TTL and applies to the longest common prefix of consecutive requests — ideal for the static analysis instructions that never change between calls.
|
||
|
||
See: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
|
||
|
||
---
|
||
|
||
## Adding a New Asset Type
|
||
|
||
1. Create a subclass of `Asset` in `server/models/` with a flat `metrics` object and `getDisplayMetrics()`.
|
||
2. Add a per-type entry (`gates` / `weights` / `thresholds`) to `ScoringRules` in `server/config/ScoringConfig.ts`.
|
||
3. Add inflated overrides in `server/services/MarketRegime.ts` → `getInflatedOverrides()`.
|
||
4. Create a Scorer in `server/scorers/` exposing `score(metrics, rules, marketContext)`.
|
||
5. Add a mapper branch in `server/services/DataMapper.ts`.
|
||
6. Wire into `server/services/ScreenerEngine.ts`: add `case` in `_buildAsset`, entry in `SCORERS` map.
|
||
7. Add the new type to `serializeAssets()` in `server/controllers/screener.controller.ts`.
|
||
|
||
---
|
||
|
||
## Clean Architecture Pattern (Server-Side)
|
||
|
||
### Core Principles
|
||
|
||
1. **Types in `server/types/`** — Domain shapes only; one `*.model.ts` per domain
|
||
2. **Schemas in `server/types/schemas.ts`** — Fastify JSON Schema objects for request body validation
|
||
3. **Class-Based Implementation** — All `.ts` files (except types/schemas) are classes
|
||
4. **No Inline Interfaces** — All types in `server/types/`; private-only shapes may stay inline
|
||
5. **Module-Level Cleanliness** — No module-level constants or functions outside classes
|
||
6. **Direct Imports** — Import directly from source files; use `server/types/index.ts` barrel for types
|
||
7. **Service barrel** — `server/services/index.ts` re-exports all services so controllers import from one path
|
||
|
||
### Implementation Pattern
|
||
|
||
#### Controllers (Classes with DI)
|
||
```typescript
|
||
export class ScreenerController {
|
||
constructor(private readonly engine: ScreenerEngine) {}
|
||
|
||
register(app: FastifyInstance): void {
|
||
app.post('/api/screen', { schema: screenSchema }, this.screen.bind(this));
|
||
}
|
||
|
||
private async screen(req: FastifyRequest) {
|
||
const tickers = (req.body as { tickers: string[] }).tickers.map(t => t.toUpperCase());
|
||
return this.engine.screenTickers(tickers);
|
||
}
|
||
}
|
||
```
|
||
|
||
#### Services (Instance methods, static helpers)
|
||
```typescript
|
||
export class BenchmarkProvider {
|
||
private static readonly TTL_MS = 60 * 60 * 1000;
|
||
|
||
async getMarketContext(): Promise<MarketContext> { ... }
|
||
|
||
private static rateRegime(rate: number): string {
|
||
return rate < 2 ? 'LOW' : rate <= 5 ? 'NORMAL' : 'HIGH';
|
||
}
|
||
}
|
||
```
|
||
|
||
#### Scorers (Static-only classes)
|
||
```typescript
|
||
export class StockScorer {
|
||
static score(metrics: StockMetrics, rules: ScoringRules): ScoreResult { ... }
|
||
private static _sanitize(m: StockMetrics): SanitizedMetrics { ... }
|
||
}
|
||
```
|
||
|
||
### Import Guidelines
|
||
|
||
```typescript
|
||
// ✅ CORRECT
|
||
import { ScreenerEngine } from '../services/ScreenerEngine'; // direct for implementations
|
||
import { ScreenerEngine } from '../services'; // via barrel also acceptable
|
||
import type { StockMetrics } from '../types'; // always use types barrel
|
||
import { screenSchema } from '../types/schemas'; // direct for schemas
|
||
|
||
// ❌ WRONG
|
||
import type { StockMetrics } from '../types/models.model'; // use barrel instead
|
||
```
|
||
|
||
### Adding a New API Endpoint
|
||
|
||
1. Define types → `server/types/<domain>.model.ts`
|
||
2. Define schema → `server/types/schemas.ts`
|
||
3. Create service → `server/services/<Domain>Service.ts`
|
||
4. Wire controller → `server/controllers/<domain>.controller.ts`
|
||
5. Register → `server/app.ts`
|