feat(quant-research-loop): live Coinbase feed + granular (hourly) dat…

starters/quant-research-loop/AGENTS.md

@@ -0,0 +1,29 @@
+# AGENTS.md — Quant Research Loop
+
+Conventions for agents (and humans) working in this starter.
+
+## Build & test
+
+- Pure Python **stdlib** — no third-party dependencies, no install step.
+- Run the suite: `python3 test_engine.py` (also works under `pytest`).
+- Run the app / loop: `python3 -m engine.loop --help`, `python3 -m engine.forward_paper --run`,
+  `python3 -m engine.service` (always-on tracker).
+
+## Layout
+
+- `engine/` — data adapters (`data`, `coinbase`, `multi_data`), strategies (`strategy`,
+  `xsectional`), verifier + walk-forward + lockbox + quarantine, blotter, forward tracker,
+  and the Railway `service`.
+- `sample-data/` — committed price snapshots (offline seed / fallback).
+- `forward-registration.json` — the FROZEN, write-once strategy contract.
+
+## Review norms
+
+- **Never re-optimize a frozen strategy.** A new thesis = a new name with a new start date.
+- **Verification is mandatory** before any "approved" claim: out-of-sample, walk-forward,
+  survivorship-corrected. No survivor-only backtests.
+- **Everything reconciles.** Blotter per-trade PnL reconciles to backtest equity to 1e-9;
+  per-coin contribution reconciles to portfolio returns. Keep it that way.
+- **Degrees-of-freedom discipline.** Small param grids; honor the enforced trial counter and
+  `--trial-budget` auto-halt. Hand-sweeping params is uncounted multiple testing.
+- **Paper only.** No live order path. Going live is a separate, human-gated project.

starters/quant-research-loop/DEPLOY-RAILWAY.md

@@ -37,13 +37,20 @@ for the machine-readable version; `GET /health` for the healthcheck.
 > you're on a commit that predates `requirements.txt`/`Procfile`. Redeploy the
 > latest `main` (Railway → Deployments → redeploy, or push a new commit).
 
-## The data caveat (read this)
+## The data feed
 
-The default price source is the free Coin Metrics community dataset, which can lag
-by days–weeks. Forward rows only appear once it publishes bars **after** the
-registration date. For same-day tracking, wire a real-time feed (Binance.US /
-Coinbase) into `data.py` / `multi_data.py` — Railway has open internet, so exchange
-APIs that are blocked in some sandboxes work there.
+- **`regime-trend` uses a live Coinbase feed** (`source: coinbase`, daily candles,
+  public API — no key). On Railway (open internet) it pulls **current** prices, so
+  its forward record starts filling in as soon as there are bars after the
+  registration date. In a sandbox that blocks exchange egress it falls back to the
+  committed snapshot and reads "awaiting data."
+- **`xsectional-momentum-riskoff`** still uses the Coin Metrics panel (its
+  survivorship-corrected universe includes delisted coins an exchange won't serve),
+  which can lag — so it may read "awaiting data" longer. Forward trading can only
+  hold *listed* coins anyway; a live basket feed is a later refinement.
+
+Granular (hourly) data is available via `engine/coinbase.py` for the next class of
+strategies; the daily strategies fetch daily candles.
 
 ## Adding a new thesis (the iterate loop)
 

starters/quant-research-loop/LOOP.md

@@ -23,6 +23,17 @@ Binance.US/Coinbase locally), or `--csv` your own OHLCV.
 | Execute | Connector | per verified signal | **L1 paper-only** | `engine/paper_broker.py` |
 | Risk | Kill switch | every cycle | always on | `engine/risk.py` |
 
+## Primitives
+
+- **Worktrees:** research experiments that mutate files (new strategy variants,
+  parameter studies) run in an isolated git worktree per attempt, discarded on reject —
+  so parallel experiments never collide. The frozen forward strategies are never edited
+  in place (write-once registration).
+- **MCP / connectors:** not required for this loop — data comes from read-only public
+  price feeds (Coinbase, Coin Metrics). Any future connector is scoped read-only until trusted.
+- **Safety & budget:** see [docs/safety.md](docs/safety.md) and `loop-budget.md`
+  (token/compute caps, kill switches, trial-budget auto-halt).
+
 ## Human Gates
 
 - **Live trading is NOT wired and will not be added without explicit sign-off.**

starters/quant-research-loop/README.md

@@ -58,6 +58,15 @@ python3 -m engine.loop --search --source live --symbol BTCUSDT --timeframe 1d
 
 Three real-data notes:
 
+- **`coinbase`** pulls live OHLCV candles at any granularity (1m…1d) from the
+  public Coinbase Exchange API — no key. This is the **current, granular** feed:
+  it unblocks the forward tracker (the daily/lagging Coin Metrics feed can't) and
+  is the foundation for intraday strategies. Blocked in some sandboxes (exchange
+  egress) but works on Railway / your machine.
+  ```bash
+  python3 -m engine.coinbase --product BTC-USD --timeframe 1h --limit 500
+  python3 -m engine.coinbase --product BTC-USD --timeframe 1h --daily   # resample to daily
+  ```
 - **`coinmetrics`** pulls a daily *reference price* (close), not OHLC, so the
   breakout runs as **Donchian-on-close** — a standard daily variant. It works
   even behind a restrictive network policy (`raw.githubusercontent.com`).
@@ -225,6 +234,7 @@ further searches halt and point you to forward-testing or new data.
 | `engine/blotter.py` | Per-trade blotter — single-asset round-trips + per-coin basket contribution |
 | `engine/forward_paper.py` | Frozen-strategy forward paper trade + registry (write-once) |
 | `engine/service.py` | Always-on tracker service (Railway) — scheduler + scoreboard |
+| `engine/coinbase.py` | Live Coinbase candles (hourly+), pagination + daily resample |
 | `engine/split.py` | Three-way train/validation/lockbox split |
 | `engine/ledger.py` | Trial counter + budget + write-once lockbox/forward ledger |
 | `engine/stats.py` | Overfitting-aware metrics (no numpy/scipy) |

starters/quant-research-loop/STATE.md

@@ -0,0 +1,30 @@
+# Loop State — Quant Research Loop
+
+Last updated: pre-registration baseline (forward tracker awaiting live data past 2026-05-23)
+
+## Active loops
+
+- **Forward paper trade** (`engine/service.py`, daily): two FROZEN strategies
+  (`regime-trend`, `xsectional-momentum-riskoff`) marked to market. Status:
+  `regime-trend` now sources a live Coinbase feed; `xsectional` still on the
+  lagging Coin Metrics panel. Forward equity is the verdict.
+- **Research gauntlet** (`engine/loop.py`): on demand — enforced trial counting,
+  walk-forward, write-once lockbox, forward quarantine. No strategy approved.
+
+## High priority (loop acting / waiting on human)
+
+- Confirm `regime-trend` forward record populates from the live Coinbase feed on deploy.
+- Wire a live basket feed for `xsectional` (survivorship-corrected panel can't come
+  from a single exchange — forward can only hold listed coins).
+
+## Watch list
+
+- Modern-era edge is thin / regime-dependent (2021+ ≈ breakeven) — temper expectations.
+- Next thesis: an intraday (hourly) strategy on the new Coinbase feed, or funding-rate carry.
+
+## Recent
+
+- Added live Coinbase (hourly+) data adapter; `regime-trend` sources it forward.
+
+---
+Run log: `quant-forward-log.md` + `loop-run-log.md` · budget: `loop-budget.md` · gates: `LOOP.md`

starters/quant-research-loop/docs/safety.md

@@ -0,0 +1,17 @@
+# Safety — Quant Research Loop
+
+- **Paper only.** No live order execution is wired anywhere. The forward tracker marks
+  strategies to market on paper; there is no exchange trading path and no API keys.
+- **Kill switches.** Drawdown breaker (`engine/risk.py`); a forward mandate breach flags the
+  thesis on the scoreboard; research trial-budget auto-halt (`engine/ledger.py`) stops the
+  search before it overfits the data.
+- **No re-optimization / no goalpost-moving.** Frozen strategies are write-once
+  (`forward-registration.json`). Changing a thesis means a NEW name with a NEW start date —
+  never editing an existing one after seeing results.
+- **Data & egress.** Read-only public price feeds only (Coinbase public candles, Coin Metrics
+  community CSV). No secrets, no write access to any venue.
+- **MCP / connectors.** Not required for this loop. If added later, scope any connector to
+  read-only until trusted.
+- **Going live is out of scope** and would require, separately: an order connector behind an
+  allowlist, position + notional caps, a live kill-switch process, and a human approving the
+  switch. Treat that as a different project.

starters/quant-research-loop/engine/coinbase.py

@@ -0,0 +1,104 @@
+"""Coinbase Exchange public candles adapter — granular (hourly+) live prices.
+
+Public endpoint, no API key required. Blocked in some sandboxes (exchange egress
+policy) but works on open internet (Railway, your machine). Gives CURRENT data, so
+it unblocks the forward tracker that the daily/lagging Coin Metrics feed cannot.
+
+Coinbase candle row: [time, low, high, open, close, volume], time in unix seconds,
+returned most-recent-first, max 300 per request — so we paginate backward.
+"""
+from __future__ import annotations
+
+import json
+import time
+import urllib.request
+from datetime import datetime
+
+from .data import Bar
+
+
+BASE = "https://api.exchange.coinbase.com"
+GRANULARITY = {"1m": 60, "5m": 300, "15m": 900, "1h": 3600, "6h": 21600, "1d": 86400}
+MAX_PER_REQ = 300
+
+
+def parse_candles(rows: list) -> list[Bar]:
+    """Coinbase [time, low, high, open, close, volume] → sorted ascending Bars."""
+    out = [Bar(ts=int(r[0]), open=float(r[3]), high=float(r[2]), low=float(r[1]),
+               close=float(r[4]), volume=float(r[5])) for r in rows]
+    out.sort(key=lambda b: b.ts)
+    return out
+
+
+def _iso(ts: int) -> str:
+    return datetime.utcfromtimestamp(ts).isoformat()
+
+
+def _fetch_page(product: str, gran: int, start: int, end: int, timeout: int = 20) -> list[Bar]:
+    url = (f"{BASE}/products/{product}/candles?granularity={gran}"
+           f"&start={_iso(start)}&end={_iso(end)}")
+    req = urllib.request.Request(url, headers={"User-Agent": "quant-research-loop"})
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        return parse_candles(json.loads(resp.read().decode()))
+
+
+def fetch_candles(product: str = "BTC-USD", timeframe: str = "1h", limit: int = 2000,
+                  now_ts: int | None = None, pause_s: float = 0.25) -> list[Bar]:
+    """Fetch up to `limit` candles at `timeframe`, paginating backward from now.
+    Returns Bars ascending by time. Deduplicates on timestamp."""
+    if timeframe not in GRANULARITY:
+        raise ValueError(f"timeframe must be one of {list(GRANULARITY)}")
+    gran = GRANULARITY[timeframe]
+    end = now_ts if now_ts is not None else int(datetime.utcnow().timestamp())
+    by_ts: dict[int, Bar] = {}
+    while len(by_ts) < limit:
+        start = end - MAX_PER_REQ * gran
+        page = _fetch_page(product, gran, start, end)
+        if not page:
+            break
+        for b in page:
+            by_ts[b.ts] = b
+        end = page[0].ts - gran  # oldest bar returned, step back
+        if len(page) < MAX_PER_REQ:
+            break
+        if pause_s:
+            time.sleep(pause_s)  # be polite to the public endpoint
+    bars = sorted(by_ts.values(), key=lambda b: b.ts)
+    return bars[-limit:]
+
+
+def to_daily(bars: list[Bar]) -> list[Bar]:
+    """Resample intraday bars to daily (UTC): open=first, high=max, low=min,
+    close=last, volume=sum. Lets daily strategies run off an hourly feed."""
+    buckets: dict[str, list[Bar]] = {}
+    for b in bars:
+        day = datetime.utcfromtimestamp(b.ts).strftime("%Y-%m-%d")
+        buckets.setdefault(day, []).append(b)
+    out: list[Bar] = []
+    for day in sorted(buckets):
+        bs = sorted(buckets[day], key=lambda x: x.ts)
+        midnight = int(datetime.strptime(day, "%Y-%m-%d").timestamp())
+        out.append(Bar(ts=midnight, open=bs[0].open, high=max(x.high for x in bs),
+                       low=min(x.low for x in bs), close=bs[-1].close,
+                       volume=sum(x.volume for x in bs)))
+    return out
+
+
+if __name__ == "__main__":
+    import argparse
+    p = argparse.ArgumentParser(description="Fetch Coinbase candles (public, no key).")
+    p.add_argument("--product", default="BTC-USD")
+    p.add_argument("--timeframe", default="1h", choices=list(GRANULARITY))
+    p.add_argument("--limit", type=int, default=500)
+    p.add_argument("--daily", action="store_true", help="resample to daily bars")
+    a = p.parse_args()
+    bars = fetch_candles(a.product, a.timeframe, a.limit)
+    if a.daily:
+        bars = to_daily(bars)
+    if bars:
+        print(f"{len(bars)} bars · {datetime.utcfromtimestamp(bars[0].ts).date()} "
+              f"→ {datetime.utcfromtimestamp(bars[-1].ts).date()}")
+        b = bars[-1]
+        print(f"latest: {datetime.utcfromtimestamp(b.ts)} O{b.open} H{b.high} L{b.low} C{b.close}")
+    else:
+        print("no bars returned")

starters/quant-research-loop/engine/data.py

@@ -213,6 +213,16 @@ def get_ohlcv(source: str = "synthetic", *, csv_path: str | None = None,
             return from_live(symbol, interval, min(limit, 1000)), f"live:binance:{symbol}:{interval}"
         except Exception as exc:  # network blocked / rate limited → fall back loudly
             return synthetic(n=limit, seed=seed), f"SYNTHETIC(live-failed:{type(exc).__name__})"
+    if source == "coinbase":
+        from . import coinbase  # local import: exchange egress may be blocked in some envs
+        product = symbol if "-" in symbol else "BTC-USD"
+        try:
+            bars = coinbase.fetch_candles(product, timeframe=interval, limit=limit or 2000)
+            if interval != "1d" and interval in coinbase.GRANULARITY:
+                pass  # caller may resample via coinbase.to_daily if they want daily
+            return bars, f"coinbase:{product}:{interval}"
+        except Exception as exc:
+            return synthetic(n=limit or 1500, seed=seed), f"SYNTHETIC(coinbase-failed:{type(exc).__name__})"
     if source == "coinmetrics":
         # Coin Metrics keys by bare asset ("btc"), not a pair — strip the quote.
         asset = symbol.lower()

starters/quant-research-loop/engine/forward_paper.py

@@ -66,7 +66,10 @@
     "regime-trend": {
         "kind": "single_asset",
         "description": "BTC long/flat trend gated by a calm-volatility regime, vol targeting",
-        "data": "btc_1d_coinmetrics.csv",
+        "data": "btc_1d_coinmetrics.csv",   # committed snapshot (seed / offline fallback)
+        "source": "coinbase",                # live feed for the forward record (current prices)
+        "product": "BTC-USD",
+        "timeframe": "1d",                   # daily strategy → daily candles
         "config": {"strategy": "regime", "trend_lookback": 100, "vol_regime_lookback": 30,
                    "vol_target": True, "target_vol": 0.40, "vol_lookback": 30, "max_leverage": 1.0},
         "mandate_max_drawdown": 0.40,
@@ -142,7 +145,12 @@ def _refresh_data(entry: dict) -> str:
             md.panel_to_csv(dates, series, [a for a in entry["universe"] if a in series], path)
         else:
             path = os.path.join(cache_dir, "btc_live.csv")
-            bars = data_mod.from_coinmetrics(asset="btc")
+            if entry.get("source") == "coinbase":
+                from . import coinbase
+                bars = coinbase.fetch_candles(entry.get("product", "BTC-USD"),
+                                              timeframe=entry.get("timeframe", "1d"), limit=1200)
+            else:
+                bars = data_mod.from_coinmetrics(asset="btc")
             data_mod.to_csv(bars, path)
         return path
     except Exception as exc:  # network blocked / source down → use the snapshot

starters/quant-research-loop/engine/loop.py

@@ -648,9 +648,10 @@ def build_parser() -> argparse.ArgumentParser:
     p.add_argument("--strategy", default="donchian", choices=STRATEGY_NAMES,
                    help="hypothesis: donchian|tsmom|meanrev|regime|mvrv|trendval")
     p.add_argument("--source", default="synthetic",
-                   choices=["synthetic", "live", "coinmetrics"],
-                   help="live=Binance OHLCV (US users: see README); "
-                        "coinmetrics=real daily close history (works behind egress policy)")
+                   choices=["synthetic", "live", "coinmetrics", "coinbase"],
+                   help="coinbase=live hourly/daily candles (public, no key); "
+                        "coinmetrics=daily close history (works behind egress policy); "
+                        "live=Binance OHLCV (US users: see README)")
     p.add_argument("--csv", default=None, help="path to OHLCV csv (overrides --source)")
     p.add_argument("--symbol", default="BTCUSDT", help="e.g. BTCUSDT (spot)")
     p.add_argument("--timeframe", default="1d", choices=list(PERIODS_PER_YEAR),

starters/quant-research-loop/loop-budget.md

@@ -0,0 +1,19 @@
+# Loop Budget — Quant Research Loop
+
+## Compute / cost
+
+- The forward tracker (`engine/service.py`) is pure stdlib. Cost per cycle ≈ one price
+  fetch per strategy (Coinbase / Coin Metrics, free public endpoints) + a little CPU.
+  Negligible; no paid APIs, no LLM calls at runtime.
+- Research runs (walk-forward, cross-sectional) are CPU-bound minutes.
+
+## Caps & kill switch
+
+- **Check cadence:** `CHECK_INTERVAL_SECONDS` (default `86400` = daily) caps how often the
+  tracker fetches prices.
+- **Research budget:** enforced trial counting + `--trial-budget N` auto-halt
+  (`engine/ledger.py`) stops searching once cumulative trials hit the cap — the
+  alpha-spending cap that prevents grinding the data to dust.
+- **Risk kill switch:** drawdown breaker (`engine/risk.py`); a forward mandate breach flags
+  the thesis on the scoreboard.
+- **Financial kill switch:** paper-only — no capital at risk, no live order path wired.

starters/quant-research-loop/loop-run-log.md

@@ -0,0 +1,10 @@
+# Loop Run Log — Quant Research Loop
+
+One line per notable loop run / milestone. The forward tracker also appends
+`quant-forward-log.md` automatically on each check that has forward data.
+
+| date | event |
+|------|-------|
+| 2026-05-23 | Registered `regime-trend` + `xsectional-momentum-riskoff` (frozen, write-once). |
+| 2026-05-23 | Forward tracker + Railway service deployed; awaiting live data past registration. |
+| 2026-05-23 | Added live Coinbase (hourly+) feed; `regime-trend` sources it forward. |

starters/quant-research-loop/test_engine.py

@@ -464,6 +464,52 @@ def test_service_scoreboard_scoring():
     assert board["strategies"]["good-one"]["status"] == "breached"
 
 
+def test_coinbase_parse_candles():
+    from engine import coinbase
+    # Coinbase row order: [time, low, high, open, close, volume], newest first
+    rows = [[1700003600, 9, 12, 10, 11, 100], [1700000000, 8, 11, 9, 10, 50]]
+    bars = coinbase.parse_candles(rows)
+    assert [b.ts for b in bars] == [1700000000, 1700003600], "must sort ascending"
+    b = bars[0]
+    assert (b.open, b.high, b.low, b.close, b.volume) == (9.0, 11.0, 8.0, 10.0, 50.0)
+
+
+def test_coinbase_to_daily_resample():
+    import calendar
+    import datetime as _dt
+    from engine import coinbase
+    from engine.data import Bar
+    day = calendar.timegm(_dt.datetime(2024, 1, 1, 0).timetuple())
+    hrs = [Bar(day, 10, 10, 10, 10, 1), Bar(day + 3600, 11, 15, 9, 12, 2),
+           Bar(day + 86400, 20, 20, 20, 20, 3)]
+    d = coinbase.to_daily(hrs)
+    assert len(d) == 2
+    a = d[0]  # open=first, high=max, low=min, close=last, volume=sum
+    assert (a.open, a.high, a.low, a.close, a.volume) == (10, 15, 9, 12, 3)
+
+
+def test_coinbase_fetch_pagination():
+    from engine import coinbase
+    from engine.data import Bar
+    orig, gran = coinbase._fetch_page, 3600
+    calls = {"n": 0}
+
+    def fake(product, g, start, end, timeout=20):
+        calls["n"] += 1
+        if calls["n"] > 3:
+            return []
+        page = [Bar(ts=end - i * gran, open=1, high=1, low=1, close=1, volume=1) for i in range(300)]
+        page.sort(key=lambda b: b.ts)  # ascending, like real parse_candles
+        return page
+    coinbase._fetch_page = fake
+    try:
+        bars = coinbase.fetch_candles("BTC-USD", "1h", limit=500, now_ts=1_700_000_000, pause_s=0)
+    finally:
+        coinbase._fetch_page = orig
+    assert len(bars) == 500, "must cap at limit"
+    assert all(bars[i].ts < bars[i + 1].ts for i in range(len(bars) - 1)), "ascending, deduped"
+
+
 def _run_all():
     fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
     passed = 0