refactor: restructure to clean architecture
fix: restore ScoringConfig improvements lost in refactor commit docs: rewrite README and CLAUDE.md to reflect current architecture code-format code fixes
This commit is contained in:
Kazuma
@@ -0,0 +1,231 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Guidance for working in this repository.
|
||||
|
||||
## Overview
|
||||
|
||||
`market-screener` is a Node.js CLI tool that screens stocks, ETFs, and bonds by fetching live data from Yahoo Finance and scoring each asset under two lenses:
|
||||
|
||||
- **Market-Adjusted** — gates derived from live Yahoo benchmarks (SPY P/E, XLK P/E, XLRE yield, LQD spread). Reflects what is acceptable in today's inflated market.
|
||||
- **Fundamental** — strict Graham/value-investing style gates from `ScoringConfig`. Reflects genuine value regardless of market conditions.
|
||||
|
||||
The comparison produces a **Signal** (Strong Buy / Momentum / Speculation / Neutral / Avoid).
|
||||
|
||||
ES module project (`"type": "module"`); use `import`/`export`, not `require`.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
npm install # install dependencies (yahoo-finance2, dotenv, prettier)
|
||||
npm start # Yahoo news → catalyst tickers → screener-report.html
|
||||
npm start -- watch # default watchlist
|
||||
npm start -- AAPL MSFT VOO # specific tickers
|
||||
npm run finance # portfolio advice + SimpleFIN → finance-report.html
|
||||
npm run import-portfolio -- holdings.csv # import Robinhood/Vanguard CSV into portfolio.json
|
||||
npm test # run all unit tests (node:test, zero deps)
|
||||
npm run test:watch # watch mode during development
|
||||
npm run format # format all src/bin/tests with Prettier
|
||||
npm run format:check # check formatting without writing (CI)
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
bin/
|
||||
screen.js ← market screener entry point
|
||||
finance.js ← personal finance entry point
|
||||
import-portfolio.js ← broker CSV importer
|
||||
|
||||
prompts/
|
||||
catalyst-analysis.md ← daily catalyst analysis playbook (LLM prompt + workflow)
|
||||
|
||||
src/
|
||||
config/
|
||||
ScoringConfig.js ← CREDIT_RATING_SCALE + ScoringRules (single source of truth)
|
||||
|
||||
market/ ← Yahoo Finance data layer
|
||||
YahooClient.js ← wraps yahoo-finance2 v3, retry + backoff
|
||||
BenchmarkProvider.js ← fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD → marketContext
|
||||
MarketRegime.js ← derives INFLATED gate overrides from live benchmarks
|
||||
|
||||
screener/ ← core screening domain
|
||||
ScreenerEngine.js ← orchestrates: fetch → score × 2 → HTML report
|
||||
DataMapper.js ← normalises Yahoo payload → flat asset data object
|
||||
RuleMerger.js ← merges base rules + sector overrides + MarketRegime (INFLATED mode)
|
||||
Chunker.js ← splits ticker list into batches
|
||||
assets/
|
||||
Asset.js ← abstract base: ticker, currentPrice, type, formatting helpers
|
||||
Stock.js ← metrics: peRatio, pegRatio, priceToBook, ROE, opMargin, etc.
|
||||
Etf.js ← metrics: expenseRatio, yield, totalAssets
|
||||
Bond.js ← metrics: ytm, duration, creditRating, creditRatingNumeric
|
||||
scorers/
|
||||
StockScorer.js ← gate checks + weighted registry (ROE, opMargin, margin, peg, rev, fcf)
|
||||
EtfScorer.js ← expense gate + registry (cost, yield, volume)
|
||||
BondScorer.js ← credit gate + spread/duration scoring
|
||||
|
||||
analyst/
|
||||
CatalystAnalyst.js ← fetches Yahoo Finance news, extracts relatedTickers
|
||||
|
||||
finance/
|
||||
clients/
|
||||
SimpleFINClient.js ← claims setup token → access URL, fetches /accounts
|
||||
PersonalFinanceAnalyzer.js ← net worth, cash vs investments, spending by category
|
||||
PortfolioAdvisor.js ← cross-references holdings with screener signals → hold/sell/add advice
|
||||
PortfolioImporter.js ← parses Robinhood/Vanguard/Fidelity CSV → merges into portfolio.json
|
||||
|
||||
reporters/
|
||||
HtmlReporter.js ← generates screener-report.html (tabbed inflated/fundamental views)
|
||||
FinanceReporter.js ← generates finance-report.html (net worth, portfolio, spending)
|
||||
|
||||
portfolio.json ← user's holdings: ticker, shares, costBasis, source, type
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
Yahoo Finance API
|
||||
↓
|
||||
BenchmarkProvider — fetches ^GSPC, ^TNX, ^VIX, SPY, XLK, XLRE, LQD
|
||||
builds marketContext { sp500Price, riskFreeRate, vixLevel,
|
||||
rateRegime, volatilityRegime, benchmarks { marketPE, techPE, reitYield, igSpread } }
|
||||
↓
|
||||
DataMapper — normalises raw Yahoo payload → flat data object with type (STOCK/ETF/BOND)
|
||||
computes: pFFO proxy, FCF yield, computed PEG, 52-week position
|
||||
↓
|
||||
Asset subclass — Stock / Etf / Bond holds metrics, formats display
|
||||
↓
|
||||
RuleMerger × 2 — FUNDAMENTAL mode: ScoringConfig as-is (Graham-style)
|
||||
INFLATED mode: sector override + MarketRegime live gate overrides
|
||||
↓
|
||||
Scorer × 2 — StockScorer / EtfScorer / BondScorer, fully stateless
|
||||
↓
|
||||
ScreenerEngine — derives Signal from comparing both verdicts
|
||||
↓
|
||||
HtmlReporter — screener-report.html: signal summary + two tabbed tables per asset class
|
||||
```
|
||||
|
||||
## Scoring Modes
|
||||
|
||||
| Mode | P/E Gate (general) | Source |
|
||||
|---|---|---|
|
||||
| FUNDAMENTAL | 20x | ScoringConfig (Graham) |
|
||||
| INFLATED | S&P 500 P/E × 1.5 | Live SPY data |
|
||||
|
||||
| Signal | Meaning |
|
||||
|---|---|
|
||||
| ✅ Strong Buy | Passes both fundamental AND inflated gates |
|
||||
| ⚡ Momentum | Passes inflated, holds fundamentally |
|
||||
| ⚠️ Speculation | Passes inflated, fails fundamental |
|
||||
| 🔄 Neutral | Hold territory in one or both lenses |
|
||||
| ❌ Avoid | Fails both |
|
||||
|
||||
## ScoringConfig Structure
|
||||
|
||||
`src/config/ScoringConfig.js` exports two things:
|
||||
|
||||
- `CREDIT_RATING_SCALE` — `{ AAA: 10, AA: 9, ..., D: 1 }`. Used by Bond.js and BondScorer.
|
||||
- `ScoringRules` — per-type `{ gates, weights, thresholds }` + `SECTOR_OVERRIDE` map for STOCK.
|
||||
|
||||
Sector overrides are structural (apply in both modes). MarketRegime overrides valuation gates in INFLATED mode only.
|
||||
|
||||
## Sector Notes
|
||||
|
||||
- **TECHNOLOGY** — `maxDebtToEquity: 2.0` (mega-cap tech borrows for buybacks), `minQuickRatio: 0.8` (AAPL runs ~0.9). P/E and PEG gates inflated from XLK live data.
|
||||
- **REIT** — P/E and PEG disabled (9999). Scored on dividend yield and P/FFO proxy. All base weights (margin, peg, revenue) explicitly zeroed out.
|
||||
- **FINANCIAL** — P/E, PEG, D/E disabled. Scored on ROE + Price-to-Book. Quick ratio gate lowered (0.1).
|
||||
|
||||
## MarketRegime (INFLATED overrides)
|
||||
|
||||
`src/market/MarketRegime.js` derives gate overrides from live benchmarks:
|
||||
|
||||
| Gate | Formula |
|
||||
|---|---|
|
||||
| Stock maxPERatio | SPY trailing P/E × 1.5 |
|
||||
| Tech maxPERatio | XLK P/E × 1.3 |
|
||||
| Tech maxPegGate | XLK P/E ÷ 15 |
|
||||
| REIT minYield | XLRE dividend yield × 0.85 |
|
||||
| Bond minSpread | LQD−TNX spread × 0.80 |
|
||||
| ETF maxExpenseRatio | 0.75% (structural loosening) |
|
||||
|
||||
## Missing Data Convention
|
||||
|
||||
- Missing metrics use `null` (not `0`) in `_sanitize`. Gate checks skip `null` values rather than auto-failing.
|
||||
- `pegRatio` falls back to `trailingPE / earningsGrowth` when Yahoo doesn't provide it.
|
||||
- `quickRatio` falls back to `currentRatio` when missing.
|
||||
|
||||
## SimpleFIN Auth Flow
|
||||
|
||||
1. User gets a Setup Token from https://beta-bridge.simplefin.org
|
||||
2. `SimpleFINClient.init()` base64-decodes it → POSTs once to claim URL → receives Access URL
|
||||
3. Access URL is auto-appended to `.env` as `SIMPLEFIN_ACCESS_URL`
|
||||
4. All subsequent requests use Access URL directly (setup token is one-time use)
|
||||
|
||||
## portfolio.json Format
|
||||
|
||||
```json
|
||||
{
|
||||
"holdings": [
|
||||
{ "ticker": "AAPL", "shares": 10, "costBasis": 150.00, "source": "Robinhood", "type": "stock" },
|
||||
{ "ticker": "VOO", "shares": 8, "costBasis": 380.00, "source": "Vanguard", "type": "etf" },
|
||||
{ "ticker": "BTC-USD", "shares": 0.25, "costBasis": 45000, "source": "Coinbase", "type": "crypto" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`type` values: `stock`, `etf`, `crypto`. Crypto is priced via Yahoo (BTC-USD style) but not fundamentally scored.
|
||||
|
||||
## Tests
|
||||
|
||||
Uses Node's built-in test runner (`node:test` + `node:assert/strict`) — no test framework to install.
|
||||
|
||||
```
|
||||
tests/
|
||||
ScoringConfig.test.js ← CREDIT_RATING_SCALE, gate values, sector overrides
|
||||
RuleMerger.test.js ← FUNDAMENTAL vs INFLATED modes, sector merging
|
||||
MarketRegime.test.js ← inflated override formulas per asset type and sector
|
||||
StockScorer.test.js ← gate failures, scoring labels, risk flags
|
||||
EtfScorer.test.js ← expense gate, score tiers
|
||||
BondScorer.test.js ← credit gate, spread/duration scoring, unit handling
|
||||
DataMapper.test.js ← type detection, PEG computation, null convention
|
||||
PortfolioAdvisor.test.js ← _position gain/loss calc, _advice signal mapping
|
||||
```
|
||||
|
||||
**Key unit to remember:** `ytm` in `Bond.metrics` is stored as a percentage (e.g. `6.5` = 6.5%). `BondScorer._sanitize` divides by 100 before spread calculation. `minSpread` in `ScoringConfig` is also in percentage form (e.g. `1.0` = 1%).
|
||||
|
||||
## Conventions
|
||||
|
||||
- Asset `type` (uppercased) is the routing key across DataMapper, asset classes, `SCORERS` map in ScreenerEngine, and ScoringRules.
|
||||
- Prefer adjusting `ScoringConfig` or `MarketRegime` over hardcoding numbers in scorers.
|
||||
- BenchmarkProvider caches for 1 hour — restart the process to force a fresh fetch during development.
|
||||
- All entry points live in `bin/`. Do not add logic to entry points — they call into `src/`.
|
||||
|
||||
## Planned Enhancements
|
||||
|
||||
### 1. Rate Sensitivity Flag (no new fetches needed)
|
||||
Flag assets exposed to rate changes using `riskFreeRate` and `rateRegime`:
|
||||
- Long-duration bonds (duration > 7) in HIGH rate regime → warn
|
||||
- REITs with high D/E in HIGH rate regime → warn
|
||||
- Growth stocks with negative FCF in HIGH rate regime → warn
|
||||
|
||||
### 2. 52-Week Positioning (data already mapped)
|
||||
`week52High` and `week52Low` are in asset metrics. Add interpretation:
|
||||
- `> 85%` position + high P/E = crowded/momentum trade
|
||||
- `< 15%` position + passing fundamental gates = potential opportunity
|
||||
|
||||
### 3. Sector Rotation Cues (3–4 new fetches)
|
||||
Add XLF, XLE, XLV, XLI to BenchmarkProvider. Compare each sector ETF's 1-year return to S&P 500 to identify leading/lagging sectors.
|
||||
|
||||
### 4. Client/Server Architecture
|
||||
Planned split into:
|
||||
- **Server** (Fastify + BullMQ) — API endpoints, webhook-driven news monitoring, WebSocket for live updates
|
||||
- **Client** (SvelteKit) — interactive dashboard replacing the HTML reports
|
||||
- LLM integration (Claude Haiku) wired into the server for deeper catalyst analysis
|
||||
|
||||
## Adding a New Asset Type
|
||||
|
||||
1. Create a subclass of `Asset` in `src/screener/assets/` with a flat `metrics` object and `getDisplayMetrics()`.
|
||||
2. Add a per-type entry (`gates` / `weights` / `thresholds`) to `ScoringRules` in `ScoringConfig.js`.
|
||||
3. Add inflated overrides in `MarketRegime.getInflatedOverrides()`.
|
||||
4. Create a Scorer in `src/screener/scorers/` exposing `score(metrics, rules, marketContext)`.
|
||||
5. Add a mapper in `DataMapper.js`.
|
||||
6. Wire into `ScreenerEngine`: add `case` in `_buildAsset`, entry in `SCORERS` map.
|
||||
Reference in New Issue
Block a user