diff --git a/starters/quant-research-loop/AGENTS.md b/starters/quant-research-loop/AGENTS.md new file mode 100644 index 0000000..91168dd --- /dev/null +++ b/starters/quant-research-loop/AGENTS.md @@ -0,0 +1,29 @@ +# AGENTS.md — Quant Research Loop + +Conventions for agents (and humans) working in this starter. + +## Build & test + +- Pure Python **stdlib** — no third-party dependencies, no install step. +- Run the suite: `python3 test_engine.py` (also works under `pytest`). +- Run the app / loop: `python3 -m engine.loop --help`, `python3 -m engine.forward_paper --run`, + `python3 -m engine.service` (always-on tracker). + +## Layout + +- `engine/` — data adapters (`data`, `coinbase`, `multi_data`), strategies (`strategy`, + `xsectional`), verifier + walk-forward + lockbox + quarantine, blotter, forward tracker, + and the Railway `service`. +- `sample-data/` — committed price snapshots (offline seed / fallback). +- `forward-registration.json` — the FROZEN, write-once strategy contract. + +## Review norms + +- **Never re-optimize a frozen strategy.** A new thesis = a new name with a new start date. +- **Verification is mandatory** before any "approved" claim: out-of-sample, walk-forward, + survivorship-corrected. No survivor-only backtests. +- **Everything reconciles.** Blotter per-trade PnL reconciles to backtest equity to 1e-9; + per-coin contribution reconciles to portfolio returns. Keep it that way. +- **Degrees-of-freedom discipline.** Small param grids; honor the enforced trial counter and + `--trial-budget` auto-halt. Hand-sweeping params is uncounted multiple testing. +- **Paper only.** No live order path. Going live is a separate, human-gated project. diff --git a/starters/quant-research-loop/DEPLOY-RAILWAY.md b/starters/quant-research-loop/DEPLOY-RAILWAY.md index 5a600c6..a996687 100644 --- a/starters/quant-research-loop/DEPLOY-RAILWAY.md +++ b/starters/quant-research-loop/DEPLOY-RAILWAY.md @@ -37,13 +37,20 @@ for the machine-readable version; `GET /health` for the healthcheck. > you're on a commit that predates `requirements.txt`/`Procfile`. Redeploy the > latest `main` (Railway → Deployments → redeploy, or push a new commit). -## The data caveat (read this) +## The data feed -The default price source is the free Coin Metrics community dataset, which can lag -by days–weeks. Forward rows only appear once it publishes bars **after** the -registration date. For same-day tracking, wire a real-time feed (Binance.US / -Coinbase) into `data.py` / `multi_data.py` — Railway has open internet, so exchange -APIs that are blocked in some sandboxes work there. +- **`regime-trend` uses a live Coinbase feed** (`source: coinbase`, daily candles, + public API — no key). On Railway (open internet) it pulls **current** prices, so + its forward record starts filling in as soon as there are bars after the + registration date. In a sandbox that blocks exchange egress it falls back to the + committed snapshot and reads "awaiting data." +- **`xsectional-momentum-riskoff`** still uses the Coin Metrics panel (its + survivorship-corrected universe includes delisted coins an exchange won't serve), + which can lag — so it may read "awaiting data" longer. Forward trading can only + hold *listed* coins anyway; a live basket feed is a later refinement. + +Granular (hourly) data is available via `engine/coinbase.py` for the next class of +strategies; the daily strategies fetch daily candles. ## Adding a new thesis (the iterate loop) diff --git a/starters/quant-research-loop/LOOP.md b/starters/quant-research-loop/LOOP.md index 81a85af..e94a181 100644 --- a/starters/quant-research-loop/LOOP.md +++ b/starters/quant-research-loop/LOOP.md @@ -23,6 +23,17 @@ Binance.US/Coinbase locally), or `--csv` your own OHLCV. | Execute | Connector | per verified signal | **L1 paper-only** | `engine/paper_broker.py` | | Risk | Kill switch | every cycle | always on | `engine/risk.py` | +## Primitives + +- **Worktrees:** research experiments that mutate files (new strategy variants, + parameter studies) run in an isolated git worktree per attempt, discarded on reject — + so parallel experiments never collide. The frozen forward strategies are never edited + in place (write-once registration). +- **MCP / connectors:** not required for this loop — data comes from read-only public + price feeds (Coinbase, Coin Metrics). Any future connector is scoped read-only until trusted. +- **Safety & budget:** see [docs/safety.md](docs/safety.md) and `loop-budget.md` + (token/compute caps, kill switches, trial-budget auto-halt). + ## Human Gates - **Live trading is NOT wired and will not be added without explicit sign-off.** diff --git a/starters/quant-research-loop/README.md b/starters/quant-research-loop/README.md index 025a194..67af5db 100644 --- a/starters/quant-research-loop/README.md +++ b/starters/quant-research-loop/README.md @@ -58,6 +58,15 @@ python3 -m engine.loop --search --source live --symbol BTCUSDT --timeframe 1d Three real-data notes: +- **`coinbase`** pulls live OHLCV candles at any granularity (1m…1d) from the + public Coinbase Exchange API — no key. This is the **current, granular** feed: + it unblocks the forward tracker (the daily/lagging Coin Metrics feed can't) and + is the foundation for intraday strategies. Blocked in some sandboxes (exchange + egress) but works on Railway / your machine. + ```bash + python3 -m engine.coinbase --product BTC-USD --timeframe 1h --limit 500 + python3 -m engine.coinbase --product BTC-USD --timeframe 1h --daily # resample to daily + ``` - **`coinmetrics`** pulls a daily *reference price* (close), not OHLC, so the breakout runs as **Donchian-on-close** — a standard daily variant. It works even behind a restrictive network policy (`raw.githubusercontent.com`). @@ -225,6 +234,7 @@ further searches halt and point you to forward-testing or new data. | `engine/blotter.py` | Per-trade blotter — single-asset round-trips + per-coin basket contribution | | `engine/forward_paper.py` | Frozen-strategy forward paper trade + registry (write-once) | | `engine/service.py` | Always-on tracker service (Railway) — scheduler + scoreboard | +| `engine/coinbase.py` | Live Coinbase candles (hourly+), pagination + daily resample | | `engine/split.py` | Three-way train/validation/lockbox split | | `engine/ledger.py` | Trial counter + budget + write-once lockbox/forward ledger | | `engine/stats.py` | Overfitting-aware metrics (no numpy/scipy) | diff --git a/starters/quant-research-loop/STATE.md b/starters/quant-research-loop/STATE.md new file mode 100644 index 0000000..cf72e64 --- /dev/null +++ b/starters/quant-research-loop/STATE.md @@ -0,0 +1,30 @@ +# Loop State — Quant Research Loop + +Last updated: pre-registration baseline (forward tracker awaiting live data past 2026-05-23) + +## Active loops + +- **Forward paper trade** (`engine/service.py`, daily): two FROZEN strategies + (`regime-trend`, `xsectional-momentum-riskoff`) marked to market. Status: + `regime-trend` now sources a live Coinbase feed; `xsectional` still on the + lagging Coin Metrics panel. Forward equity is the verdict. +- **Research gauntlet** (`engine/loop.py`): on demand — enforced trial counting, + walk-forward, write-once lockbox, forward quarantine. No strategy approved. + +## High priority (loop acting / waiting on human) + +- Confirm `regime-trend` forward record populates from the live Coinbase feed on deploy. +- Wire a live basket feed for `xsectional` (survivorship-corrected panel can't come + from a single exchange — forward can only hold listed coins). + +## Watch list + +- Modern-era edge is thin / regime-dependent (2021+ ≈ breakeven) — temper expectations. +- Next thesis: an intraday (hourly) strategy on the new Coinbase feed, or funding-rate carry. + +## Recent + +- Added live Coinbase (hourly+) data adapter; `regime-trend` sources it forward. + +--- +Run log: `quant-forward-log.md` + `loop-run-log.md` · budget: `loop-budget.md` · gates: `LOOP.md` diff --git a/starters/quant-research-loop/docs/safety.md b/starters/quant-research-loop/docs/safety.md new file mode 100644 index 0000000..190ffcc --- /dev/null +++ b/starters/quant-research-loop/docs/safety.md @@ -0,0 +1,17 @@ +# Safety — Quant Research Loop + +- **Paper only.** No live order execution is wired anywhere. The forward tracker marks + strategies to market on paper; there is no exchange trading path and no API keys. +- **Kill switches.** Drawdown breaker (`engine/risk.py`); a forward mandate breach flags the + thesis on the scoreboard; research trial-budget auto-halt (`engine/ledger.py`) stops the + search before it overfits the data. +- **No re-optimization / no goalpost-moving.** Frozen strategies are write-once + (`forward-registration.json`). Changing a thesis means a NEW name with a NEW start date — + never editing an existing one after seeing results. +- **Data & egress.** Read-only public price feeds only (Coinbase public candles, Coin Metrics + community CSV). No secrets, no write access to any venue. +- **MCP / connectors.** Not required for this loop. If added later, scope any connector to + read-only until trusted. +- **Going live is out of scope** and would require, separately: an order connector behind an + allowlist, position + notional caps, a live kill-switch process, and a human approving the + switch. Treat that as a different project. diff --git a/starters/quant-research-loop/engine/coinbase.py b/starters/quant-research-loop/engine/coinbase.py new file mode 100644 index 0000000..a8a602f --- /dev/null +++ b/starters/quant-research-loop/engine/coinbase.py @@ -0,0 +1,104 @@ +"""Coinbase Exchange public candles adapter — granular (hourly+) live prices. + +Public endpoint, no API key required. Blocked in some sandboxes (exchange egress +policy) but works on open internet (Railway, your machine). Gives CURRENT data, so +it unblocks the forward tracker that the daily/lagging Coin Metrics feed cannot. + +Coinbase candle row: [time, low, high, open, close, volume], time in unix seconds, +returned most-recent-first, max 300 per request — so we paginate backward. +""" +from __future__ import annotations + +import json +import time +import urllib.request +from datetime import datetime + +from .data import Bar + + +BASE = "https://api.exchange.coinbase.com" +GRANULARITY = {"1m": 60, "5m": 300, "15m": 900, "1h": 3600, "6h": 21600, "1d": 86400} +MAX_PER_REQ = 300 + + +def parse_candles(rows: list) -> list[Bar]: + """Coinbase [time, low, high, open, close, volume] → sorted ascending Bars.""" + out = [Bar(ts=int(r[0]), open=float(r[3]), high=float(r[2]), low=float(r[1]), + close=float(r[4]), volume=float(r[5])) for r in rows] + out.sort(key=lambda b: b.ts) + return out + + +def _iso(ts: int) -> str: + return datetime.utcfromtimestamp(ts).isoformat() + + +def _fetch_page(product: str, gran: int, start: int, end: int, timeout: int = 20) -> list[Bar]: + url = (f"{BASE}/products/{product}/candles?granularity={gran}" + f"&start={_iso(start)}&end={_iso(end)}") + req = urllib.request.Request(url, headers={"User-Agent": "quant-research-loop"}) + with urllib.request.urlopen(req, timeout=timeout) as resp: + return parse_candles(json.loads(resp.read().decode())) + + +def fetch_candles(product: str = "BTC-USD", timeframe: str = "1h", limit: int = 2000, + now_ts: int | None = None, pause_s: float = 0.25) -> list[Bar]: + """Fetch up to `limit` candles at `timeframe`, paginating backward from now. + Returns Bars ascending by time. Deduplicates on timestamp.""" + if timeframe not in GRANULARITY: + raise ValueError(f"timeframe must be one of {list(GRANULARITY)}") + gran = GRANULARITY[timeframe] + end = now_ts if now_ts is not None else int(datetime.utcnow().timestamp()) + by_ts: dict[int, Bar] = {} + while len(by_ts) < limit: + start = end - MAX_PER_REQ * gran + page = _fetch_page(product, gran, start, end) + if not page: + break + for b in page: + by_ts[b.ts] = b + end = page[0].ts - gran # oldest bar returned, step back + if len(page) < MAX_PER_REQ: + break + if pause_s: + time.sleep(pause_s) # be polite to the public endpoint + bars = sorted(by_ts.values(), key=lambda b: b.ts) + return bars[-limit:] + + +def to_daily(bars: list[Bar]) -> list[Bar]: + """Resample intraday bars to daily (UTC): open=first, high=max, low=min, + close=last, volume=sum. Lets daily strategies run off an hourly feed.""" + buckets: dict[str, list[Bar]] = {} + for b in bars: + day = datetime.utcfromtimestamp(b.ts).strftime("%Y-%m-%d") + buckets.setdefault(day, []).append(b) + out: list[Bar] = [] + for day in sorted(buckets): + bs = sorted(buckets[day], key=lambda x: x.ts) + midnight = int(datetime.strptime(day, "%Y-%m-%d").timestamp()) + out.append(Bar(ts=midnight, open=bs[0].open, high=max(x.high for x in bs), + low=min(x.low for x in bs), close=bs[-1].close, + volume=sum(x.volume for x in bs))) + return out + + +if __name__ == "__main__": + import argparse + p = argparse.ArgumentParser(description="Fetch Coinbase candles (public, no key).") + p.add_argument("--product", default="BTC-USD") + p.add_argument("--timeframe", default="1h", choices=list(GRANULARITY)) + p.add_argument("--limit", type=int, default=500) + p.add_argument("--daily", action="store_true", help="resample to daily bars") + a = p.parse_args() + bars = fetch_candles(a.product, a.timeframe, a.limit) + if a.daily: + bars = to_daily(bars) + if bars: + print(f"{len(bars)} bars · {datetime.utcfromtimestamp(bars[0].ts).date()} " + f"→ {datetime.utcfromtimestamp(bars[-1].ts).date()}") + b = bars[-1] + print(f"latest: {datetime.utcfromtimestamp(b.ts)} O{b.open} H{b.high} L{b.low} C{b.close}") + else: + print("no bars returned") diff --git a/starters/quant-research-loop/engine/data.py b/starters/quant-research-loop/engine/data.py index e83b3a3..71c3f3f 100644 --- a/starters/quant-research-loop/engine/data.py +++ b/starters/quant-research-loop/engine/data.py @@ -213,6 +213,16 @@ def get_ohlcv(source: str = "synthetic", *, csv_path: str | None = None, return from_live(symbol, interval, min(limit, 1000)), f"live:binance:{symbol}:{interval}" except Exception as exc: # network blocked / rate limited → fall back loudly return synthetic(n=limit, seed=seed), f"SYNTHETIC(live-failed:{type(exc).__name__})" + if source == "coinbase": + from . import coinbase # local import: exchange egress may be blocked in some envs + product = symbol if "-" in symbol else "BTC-USD" + try: + bars = coinbase.fetch_candles(product, timeframe=interval, limit=limit or 2000) + if interval != "1d" and interval in coinbase.GRANULARITY: + pass # caller may resample via coinbase.to_daily if they want daily + return bars, f"coinbase:{product}:{interval}" + except Exception as exc: + return synthetic(n=limit or 1500, seed=seed), f"SYNTHETIC(coinbase-failed:{type(exc).__name__})" if source == "coinmetrics": # Coin Metrics keys by bare asset ("btc"), not a pair — strip the quote. asset = symbol.lower() diff --git a/starters/quant-research-loop/engine/forward_paper.py b/starters/quant-research-loop/engine/forward_paper.py index 431b836..da9222c 100644 --- a/starters/quant-research-loop/engine/forward_paper.py +++ b/starters/quant-research-loop/engine/forward_paper.py @@ -66,7 +66,10 @@ "regime-trend": { "kind": "single_asset", "description": "BTC long/flat trend gated by a calm-volatility regime, vol targeting", - "data": "btc_1d_coinmetrics.csv", + "data": "btc_1d_coinmetrics.csv", # committed snapshot (seed / offline fallback) + "source": "coinbase", # live feed for the forward record (current prices) + "product": "BTC-USD", + "timeframe": "1d", # daily strategy → daily candles "config": {"strategy": "regime", "trend_lookback": 100, "vol_regime_lookback": 30, "vol_target": True, "target_vol": 0.40, "vol_lookback": 30, "max_leverage": 1.0}, "mandate_max_drawdown": 0.40, @@ -142,7 +145,12 @@ def _refresh_data(entry: dict) -> str: md.panel_to_csv(dates, series, [a for a in entry["universe"] if a in series], path) else: path = os.path.join(cache_dir, "btc_live.csv") - bars = data_mod.from_coinmetrics(asset="btc") + if entry.get("source") == "coinbase": + from . import coinbase + bars = coinbase.fetch_candles(entry.get("product", "BTC-USD"), + timeframe=entry.get("timeframe", "1d"), limit=1200) + else: + bars = data_mod.from_coinmetrics(asset="btc") data_mod.to_csv(bars, path) return path except Exception as exc: # network blocked / source down → use the snapshot diff --git a/starters/quant-research-loop/engine/loop.py b/starters/quant-research-loop/engine/loop.py index 00e3103..9ac2852 100644 --- a/starters/quant-research-loop/engine/loop.py +++ b/starters/quant-research-loop/engine/loop.py @@ -648,9 +648,10 @@ def build_parser() -> argparse.ArgumentParser: p.add_argument("--strategy", default="donchian", choices=STRATEGY_NAMES, help="hypothesis: donchian|tsmom|meanrev|regime|mvrv|trendval") p.add_argument("--source", default="synthetic", - choices=["synthetic", "live", "coinmetrics"], - help="live=Binance OHLCV (US users: see README); " - "coinmetrics=real daily close history (works behind egress policy)") + choices=["synthetic", "live", "coinmetrics", "coinbase"], + help="coinbase=live hourly/daily candles (public, no key); " + "coinmetrics=daily close history (works behind egress policy); " + "live=Binance OHLCV (US users: see README)") p.add_argument("--csv", default=None, help="path to OHLCV csv (overrides --source)") p.add_argument("--symbol", default="BTCUSDT", help="e.g. BTCUSDT (spot)") p.add_argument("--timeframe", default="1d", choices=list(PERIODS_PER_YEAR), diff --git a/starters/quant-research-loop/loop-budget.md b/starters/quant-research-loop/loop-budget.md new file mode 100644 index 0000000..77ffa7d --- /dev/null +++ b/starters/quant-research-loop/loop-budget.md @@ -0,0 +1,19 @@ +# Loop Budget — Quant Research Loop + +## Compute / cost + +- The forward tracker (`engine/service.py`) is pure stdlib. Cost per cycle ≈ one price + fetch per strategy (Coinbase / Coin Metrics, free public endpoints) + a little CPU. + Negligible; no paid APIs, no LLM calls at runtime. +- Research runs (walk-forward, cross-sectional) are CPU-bound minutes. + +## Caps & kill switch + +- **Check cadence:** `CHECK_INTERVAL_SECONDS` (default `86400` = daily) caps how often the + tracker fetches prices. +- **Research budget:** enforced trial counting + `--trial-budget N` auto-halt + (`engine/ledger.py`) stops searching once cumulative trials hit the cap — the + alpha-spending cap that prevents grinding the data to dust. +- **Risk kill switch:** drawdown breaker (`engine/risk.py`); a forward mandate breach flags + the thesis on the scoreboard. +- **Financial kill switch:** paper-only — no capital at risk, no live order path wired. diff --git a/starters/quant-research-loop/loop-run-log.md b/starters/quant-research-loop/loop-run-log.md new file mode 100644 index 0000000..8913070 --- /dev/null +++ b/starters/quant-research-loop/loop-run-log.md @@ -0,0 +1,10 @@ +# Loop Run Log — Quant Research Loop + +One line per notable loop run / milestone. The forward tracker also appends +`quant-forward-log.md` automatically on each check that has forward data. + +| date | event | +|------|-------| +| 2026-05-23 | Registered `regime-trend` + `xsectional-momentum-riskoff` (frozen, write-once). | +| 2026-05-23 | Forward tracker + Railway service deployed; awaiting live data past registration. | +| 2026-05-23 | Added live Coinbase (hourly+) feed; `regime-trend` sources it forward. | diff --git a/starters/quant-research-loop/test_engine.py b/starters/quant-research-loop/test_engine.py index c698152..e89c87c 100644 --- a/starters/quant-research-loop/test_engine.py +++ b/starters/quant-research-loop/test_engine.py @@ -464,6 +464,52 @@ def test_service_scoreboard_scoring(): assert board["strategies"]["good-one"]["status"] == "breached" +def test_coinbase_parse_candles(): + from engine import coinbase + # Coinbase row order: [time, low, high, open, close, volume], newest first + rows = [[1700003600, 9, 12, 10, 11, 100], [1700000000, 8, 11, 9, 10, 50]] + bars = coinbase.parse_candles(rows) + assert [b.ts for b in bars] == [1700000000, 1700003600], "must sort ascending" + b = bars[0] + assert (b.open, b.high, b.low, b.close, b.volume) == (9.0, 11.0, 8.0, 10.0, 50.0) + + +def test_coinbase_to_daily_resample(): + import calendar + import datetime as _dt + from engine import coinbase + from engine.data import Bar + day = calendar.timegm(_dt.datetime(2024, 1, 1, 0).timetuple()) + hrs = [Bar(day, 10, 10, 10, 10, 1), Bar(day + 3600, 11, 15, 9, 12, 2), + Bar(day + 86400, 20, 20, 20, 20, 3)] + d = coinbase.to_daily(hrs) + assert len(d) == 2 + a = d[0] # open=first, high=max, low=min, close=last, volume=sum + assert (a.open, a.high, a.low, a.close, a.volume) == (10, 15, 9, 12, 3) + + +def test_coinbase_fetch_pagination(): + from engine import coinbase + from engine.data import Bar + orig, gran = coinbase._fetch_page, 3600 + calls = {"n": 0} + + def fake(product, g, start, end, timeout=20): + calls["n"] += 1 + if calls["n"] > 3: + return [] + page = [Bar(ts=end - i * gran, open=1, high=1, low=1, close=1, volume=1) for i in range(300)] + page.sort(key=lambda b: b.ts) # ascending, like real parse_candles + return page + coinbase._fetch_page = fake + try: + bars = coinbase.fetch_candles("BTC-USD", "1h", limit=500, now_ts=1_700_000_000, pause_s=0) + finally: + coinbase._fetch_page = orig + assert len(bars) == 500, "must cap at limit" + assert all(bars[i].ts < bars[i + 1].ts for i in range(len(bars) - 1)), "ascending, deduped" + + def _run_all(): fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] passed = 0