0.7 release

.cursor/rules/dataframe-serialization.mdc

@@ -0,0 +1,48 @@
+# DataFrame Serialization
+
+All DataFrame-to-records conversion for API responses, streaming events, or
+frontend-visible data MUST use the centralized helpers in
+`data_formulator.datalake.parquet_utils`:
+
+| Source type | Helper |
+|---|---|
+| `pd.DataFrame` | `df_to_safe_records(df)` |
+| `pa.Table` (Arrow) | `get_sample_rows_from_arrow(table)` |
+
+## Why
+
+`pandas.DataFrame.to_json(orient='records')` defaults to `date_format='epoch'`,
+which serializes datetime columns as **epoch milliseconds** (e.g. `1773532800000`).
+The frontend interprets these as plain numbers and renders them with commas
+(`1,773,532,800,000`) instead of formatted dates.
+
+`df_to_safe_records` enforces `date_format='iso'` and `default_handler=str`,
+ensuring datetimes become ISO-8601 strings and exotic types degrade gracefully.
+
+## Banned Patterns
+
+```python
+# BAD — missing date_format, datetimes become epoch numbers
+json.loads(df.to_json(orient='records'))
+
+# BAD — to_dict returns Timestamp objects, not JSON-safe values
+df.to_dict(orient='records')
+
+# ACCEPTABLE but should be unified for consistency
+json.loads(df.to_json(orient='records', date_format='iso'))
+```
+
+## Correct Pattern
+
+```python
+from data_formulator.datalake.parquet_utils import df_to_safe_records
+
+rows = df_to_safe_records(df)
+preview = df_to_safe_records(df.head(5))
+```
+
+## Exceptions
+
+Internal data processing that never reaches the frontend or JSON serialization
+(e.g. Kusto SDK metadata parsing, Vega-Lite spec construction) may use
+`to_dict(orient='records')` directly.

.cursor/rules/dev-guides-first.mdc

@@ -9,21 +9,23 @@ alwaysApply: true
 ## Before Starting Any Development
 
 Before implementing or designing a new feature, module, or significant change, **always
-read the relevant documents in `dev-guides/`** to understand existing conventions:
+read the relevant documents in `docs/dev-guides/`** to understand existing conventions:
 
 | Guide | When to Read |
 |-------|-------------|
-| `dev-guides/1-streaming-protocol.md` | Any work on streaming endpoints or NDJSON protocol |
-| `dev-guides/2-log-sanitization.md` | Any work involving logging, credentials, external services, or DataLoaders |
-| `dev-guides/3-data-loader-development.md` | Any work on ExternalDataLoader, DataConnector, or connector routes |
-| `dev-guides/4-authentication-oidc-tokenstore.md` | Any work on OIDC, TokenStore, AUTH_MODE, or SSO flows |
-| `dev-guides/6-i18n-language-injection.md` | Any work on Agent prompts, Agent routes, backend user-visible messages, or frontend i18n |
-| `dev-guides/7-unified-error-handling.md` | Any work on API errors, frontend API calls, streaming error events, or error tests |
-| `dev-guides/8-path-safety.md` | Any work on backend file access, downloads, Workspace paths, Agent tools, DataLoaders, or sandbox config |
-| `dev-guides/10-agent-knowledge-reasoning-log.md` | Any work on Agent knowledge injection, KnowledgeStore, reasoning logs, or experience distillation |
-| `dev-guides/11-catalog-metadata-sync.md` | Any work on catalog sync, catalog_cache, catalog_annotations, metadata merge, Agent catalog tools, or frontend catalog browsing |
-| `dev-guides/12-sandbox-session.md` | Any work on sandbox execution, Agent tool-calling loops, explore/execute_python code execution, or namespace management |
-| `dev-guides/13-unified-row-limits.md` | Any work on row limits, data loading size caps, frontendRowLimit, MAX_IMPORT_ROWS, or DataLoader size parameter |
+| `docs/dev-guides/1-streaming-protocol.md` | Any work on streaming endpoints or NDJSON protocol |
+| `docs/dev-guides/2-log-sanitization.md` | Any work involving logging, credentials, external services, or DataLoaders |
+| `docs/dev-guides/3-data-loader-development.md` | Any work on ExternalDataLoader, DataConnector, or connector routes |
+| `docs/dev-guides/4-authentication-oidc-tokenstore.md` | Any work on OIDC, TokenStore, AUTH_MODE, or SSO flows |
+| `docs/dev-guides/6-i18n-language-injection.md` | Any work on Agent prompts, Agent routes, backend user-visible messages, or frontend i18n |
+| `docs/dev-guides/7-unified-error-handling.md` | Any work on API errors, frontend API calls, streaming error events, or error tests |
+| `docs/dev-guides/8-path-safety.md` | Any work on backend file access, downloads, Workspace paths, Agent tools, DataLoaders, or sandbox config |
+| `docs/dev-guides/10-agent-knowledge-reasoning-log.md` | Any work on Agent knowledge injection, KnowledgeStore, reasoning logs, or experience distillation |
+| `docs/dev-guides/11-catalog-metadata-sync.md` | Any work on catalog sync, catalog_cache, catalog_annotations, metadata merge, Agent catalog tools, or frontend catalog browsing |
+| `docs/dev-guides/12-sandbox-session.md` | Any work on sandbox execution, Agent tool-calling loops, explore/execute_python code execution, or namespace management |
+| `docs/dev-guides/13-unified-row-limits.md` | Any work on row limits, data loading size caps, frontendRowLimit, MAX_IMPORT_ROWS, or DataLoader size parameter |
+| `docs/dev-guides/14-model-capability-runtime-degradation.md` | Any work on LLM Client calls, Agent LLM invocations, model capability checks, reasoning_effort, or image/vision degradation |
+| `docs/dev-guides/15-dataframe-serialization.md` | Any work on DataFrame→JSON serialization, Agent result rows, DataLoader sample_rows, or table API responses |
 | `.cursor/rules/i18n-no-hardcoded-strings.mdc` | Any work adding or changing user-visible strings in `src/` |
 
 Also check `.cursor/rules/` and `.cursor/skills/` for related coding conventions.
@@ -37,7 +39,7 @@ decision, you MUST update documentation before considering the task complete:
    (e.g. new sensitive key type → update `2-log-sanitization.md`)
 
 2. **Create new dev-guide** — if your change introduces a new cross-cutting convention
-   that future developers must follow. Place in `dev-guides/` with the next number prefix.
+   that future developers must follow. Place in `docs/dev-guides/` with the next number prefix.
 
 3. **Update existing SKILL** — if your change affects how an existing skill works
    (e.g. new error handling pattern → update `error-handling/SKILL.md`)
@@ -55,7 +57,7 @@ decision, you MUST update documentation before considering the task complete:
 
 For every PR that introduces new patterns:
 
-- [ ] Searched `dev-guides/` for related existing docs
+- [ ] Searched `docs/dev-guides/` for related existing docs
 - [ ] Updated or created dev-guide if introducing cross-cutting conventions
 - [ ] Updated or created `.cursor/skills/` if introducing reusable workflows
 - [ ] Updated or created `.cursor/rules/` if introducing file-scoped constraints

.cursor/rules/error-response-safety.mdc

@@ -10,7 +10,7 @@ Never return raw exception text (`str(e)`, `f"...{e}"`) directly in HTTP respons
 Python exceptions may contain stack traces, file paths, database connection strings,
 API keys, or internal IP addresses — all of which are security risks (CWE-209).
 
-See `dev-guides/7-unified-error-handling.md` for the full error handling contract.
+See `docs/dev-guides/7-unified-error-handling.md` for the full error handling contract.
 
 ## Unified Error System
 

.cursor/rules/language-injection-conventions.mdc

@@ -40,4 +40,4 @@ yield {
 yield {"message": translate_in_python("empty_df", lang)}
 ```
 
-For detailed architecture and anti-pattern explanations, see `dev-guides/6-i18n-language-injection.md`.
+For detailed architecture and anti-pattern explanations, see `docs/dev-guides/6-i18n-language-injection.md`.

.cursor/rules/path-safety.mdc

@@ -6,7 +6,7 @@ alwaysApply: false
 
 # 路径安全编码规范
 
-编辑此目录下的文件时，请遵守以下安全规则（详见 `dev-guides/8-path-safety.md` 和 `.cursor/skills/path-safety/SKILL.md`）：
+编辑此目录下的文件时，请遵守以下安全规则（详见 `docs/dev-guides/8-path-safety.md` 和 `.cursor/skills/path-safety/SKILL.md`）：
 
 ## ConfinedDir 是唯一的路径约束原语
 

.cursor/rules/unified-error-protocol.mdc

@@ -6,7 +6,7 @@ alwaysApply: false
 
 # Unified Error Protocol
 
-See `dev-guides/7-unified-error-handling.md` for the full developer guide.
+See `docs/dev-guides/7-unified-error-handling.md` for the full developer guide.
 
 ## HTTP Status Code Policy
 

.cursor/skills/error-handling/SKILL.md

@@ -7,8 +7,8 @@ description: 统一错误处理系统。在添加 API 端点、修改错误处
 
 Unified error handling system for DF. Use when adding API endpoints, modifying error handling, or adding frontend API calls.
 
-> **Prerequisites**: Read `dev-guides/7-unified-error-handling.md` before changing API error behavior.
-> Read `dev-guides/2-log-sanitization.md` when the work involves logging, credentials, external services, or DataLoaders.
+> **Prerequisites**: Read `docs/dev-guides/7-unified-error-handling.md` before changing API error behavior.
+> Read `docs/dev-guides/2-log-sanitization.md` when the work involves logging, credentials, external services, or DataLoaders.
 > If your work introduces new error handling patterns or conventions, update this file and related dev-guides accordingly.
 
 ## Architecture Overview

.cursor/skills/language-injection/SKILL.md

@@ -5,9 +5,9 @@ description: LLM Agent 多语言注入规范。在修改 Agent 提示词、添
 
 # Language Injection for Agent Prompts
 
-Authoritative developer guide: `dev-guides/6-i18n-language-injection.md`.
+Authoritative developer guide: `docs/dev-guides/6-i18n-language-injection.md`.
 
-> **Prerequisites**: Read `dev-guides/6-i18n-language-injection.md` before changing Agent prompts, Agent routes, backend user-visible messages, or frontend i18n strings.
+> **Prerequisites**: Read `docs/dev-guides/6-i18n-language-injection.md` before changing Agent prompts, Agent routes, backend user-visible messages, or frontend i18n strings.
 > If your work introduces new language injection patterns or conventions, update this file and related dev-guides accordingly.
 
 ## Architecture

.cursor/skills/path-safety/SKILL.md

@@ -5,7 +5,7 @@ description: 服务端路径安全与文件访问编码规范。在编写文件
 
 # Path Safety — 服务端安全编码规范
 
-> **来源**：`dev-guides/8-path-safety.md`（正式开发规范）+ `design-docs/issues/002-arbitrary-file-read-audit.md`（安全审计复核）。
+> **来源**：`docs/dev-guides/8-path-safety.md`（正式开发规范）+ `design-docs/issues/002-arbitrary-file-read-audit.md`（安全审计复核）。
 > 本文档提炼了 6 条必须遵守的编码规范。违反任一条即可能引入路径穿越（LFI）漏洞。
 
 ---
@@ -154,7 +154,7 @@ def _enforce_deployment_restrictions():
 
 ## 参考文档
 
-- `dev-guides/8-path-safety.md` — 服务端路径安全开发规范
+- `docs/dev-guides/8-path-safety.md` — 服务端路径安全开发规范
 - `design-docs/6-path-safety-confined-dir.md` — 剩余未完成实现项状态页
 - `design-docs/issues/002-arbitrary-file-read-audit.md` — 安全审计复核报告
 - `py-src/data_formulator/security/path_safety.py` — ConfinedDir 源码

DEVELOPMENT.md

@@ -527,7 +527,7 @@ cp -r /backup/workspaces $DATA_FORMULATOR_HOME/workspaces
 
 > **Tip:** If you forgot to back up `FLASK_SECRET_KEY` and it was auto-generated, there is no way to recover it. Users will need to log in again, and any chart with a cached code signature will need to be re-executed by the Agent.
 
-> **中文版:** 详细的迁移操作指南见 [docs-cn/7-server-migration-guide.md](docs-cn/7-server-migration-guide.md)。
+> **中文版:** 详细的迁移操作指南见 [docs/docs-cn/7-server-migration-guide.md](docs/docs-cn/7-server-migration-guide.md)。
 
 ## Authentication Architecture
 

README.md

@@ -26,21 +26,35 @@
 <!-- 
 https://github.com/user-attachments/assets/8ca57b68-4d7a-42cb-bcce-43f8b1681ce2 -->
 
+
+## Why Data Formulator?
+
+Your data lives everywhere — databases, warehouses, BI tools, files. Coding agents can help, but only after someone wires them up, and answers come back as walls of code or text that are hard to follow, refine, or share.
+
+Data Formulator makes it simple: **connect any data, ask anything, get charts you can edit, branch, and share** — all on one interactive, visual canvas.
+
+- **Data & platform teams**: wire up your databases, warehouses, and BI sources once, and give the whole org an AI-powered data exploration layer.
+- **Analysts & users**: ask, edit, branch, share. It's so easy to get insights from good-looking charts.
+
 <kbd>
   <img src="https://github.com/user-attachments/assets/3ffb15aa-93ce-42b8-92cf-aaf321f9a06a">
 </kbd>
 
-
 ## News 🔥🔥🔥
 
-[05-11-2026] **Data Formulator 0.7 (alpha 2)** — A new chapter for AI-powered data exploration
--  🔌 **Data connectors** — first-class persistent connection to Superset, Kusto, Cosmos DB, MySQL, PostgreSQL, MSSQL, S3, Azure Blob, BigQuery, and more, with SSO, lazy catalog loading, search, and smart filters.
--  💬 **Conversational agent with thread memory** — a unified `DataAgent` that weaves explanation, exploration, visualization, and recommendation into one fluid conversation, carrying context across turns so the agent stays in sync with your train of thought.
--  🗂️ **Persistent session & workspace management** — identity-isolated workspaces with local and Azure Blob backends; sessions persist across restarts with timestamps and sort.
--  📊 **Expressive visualization** — 30+ chart types via a new semantic chart engine (area, streamgraph, candlestick, pie, radar, maps, …), plus a chart style-refinement agent that turns rough charts into presentation-ready visuals: restyle in one click, refine typography, color, layout, and annotations through natural language.
--  📚 **Knowledge distillation (experimental)** — agents distill reusable skills and experiences from your sessions into a shared knowledge library that informs future sessions.
+[05-28-2026] **Data Formulator 0.7** — turn ANY data into insights in five easy steps:
+
+1.  **Connect.** Governed, reusable connections to databases, warehouses, BI systems, object stores, and files (Superset, Kusto, Cosmos DB, MySQL, PostgreSQL, MSSQL, BigQuery, S3, Azure Blob, …). Need a custom source? Point your coding agent at the [data loader plugin guide](examples/plugins/README.md).
+2.  **Load.** Ask the **data-loading agent** to find tables from connected databases, or extract data from Excel files, images, websites, and text.
+3.  **Explore.** A unified **Data Agent** with thread memory inspects data, runs sandboxed code, and weaves explanation, exploration, and recommendation into one fluid conversation — grounded in your context. The **Data Thread** keeps questions, intermediate results, and charts navigable: revisit earlier steps, branch into alternatives, and compare side by side.
+4.  **Refine.** 30+ chart types (area, streamgraph, candlestick, radar, maps, KPI, …) via a new semantic chart engine, plus a **style-refinement agent** that turns rough charts into presentation-ready visuals through natural language.
+5.  **Share.** Build reports and export as image or PDF to tell the story.
+
+➕ **Persistent sessions & workspaces** — identity-isolated, saved across restarts. Data Formulator is your de facto data analysis pane.
 
-> Install the pre-release with `pip install --pre data-formulator` or pin `==0.7.0a2`.
+**Multilingual UI** — Data Formulator now speaks Chinese in addition to English (没错，DF现在会说中文了！). More languages on the way — [contributions welcome](src/i18n/TRANSLATION_GUIDE.md).
+
+> Install with `pip install data_formulator` or run instantly with `uvx data_formulator`.
 
 > [!TIP]
 > **Are you a developer?** Join us to shape the future of AI-powered data exploration!
@@ -50,6 +64,7 @@ https://github.com/user-attachments/assets/8ca57b68-4d7a-42cb-bcce-43f8b1681ce2
 ## Previous Updates
 
 Here are milestones that lead to the current design:
+- **v0.7 alpha 2** (05-11-2026): Early preview of data connectors, the unified `DataAgent` with thread memory, persistent workspaces, the semantic chart engine, and experimental knowledge distillation.
 - **v0.6** ([Demo](https://github.com/microsoft/data-formulator/releases/tag/0.6)): Real-time insights from live data — connect to URLs and databases with automatic refresh
 - **uv support**: Faster installation with [uv](https://docs.astral.sh/uv/) — `uvx data_formulator` or `uv pip install data_formulator`
 - **v0.5.1** ([Demo](https://github.com/microsoft/data-formulator/pull/200#issue-3635408217)): Community data loaders, US Map & Pie Chart, editable reports, snappier UI
@@ -67,9 +82,7 @@ Here are milestones that lead to the current design:
 
 ## Overview
 
-**Data Formulator** is a Microsoft Research prototype for data exploration with visualizations powered by AI agents.
-
-Data Formulator enables analysts to explore data with visualizations. Started with data in any format (screenshot, text, csv, or database), you can work with AI agents with a novel blended interface that combines *user interface interactions (UI)* and *natural language (NL) inputs* to communicate their intents, control branching exploration directions, and create reports to share their insights. 
+**Data Formulator** is a Microsoft Research project for data exploration with visualizations powered by AI agents. It combines *UI interactions* with *natural language* so analysts can communicate intent, branch into alternative analyses, and share results — starting from any data format (screenshot, text, CSV, or database).
 
 ## Get Started
 

dev-guides/1-streaming-protocol.md → docs/dev-guides/1-streaming-protocol.md

@@ -34,7 +34,6 @@
 | `get-recommendation-questions` | `"question"` | 探索建议问题 |
 | `generate-report-chat` | `"text_delta"`, `"embed_chart"`, `"embed_table"` | 报告生成流 |
 | `data-loading-chat` | `"text_delta"`, `"tool_call"`, `"tool_result"`, `"done"` | 数据加载对话 |
-| `clean-data-stream` | 各种 agent 事件 | 数据清洗流 |
 | （跨端点通用） | `"thinking_text"` | Agent 推理/思考过程文本（参见 2.4） |
 
 `data-agent-streaming` 的 `result.type === "clarify"` 使用结构化多问题格式。后端和前端都以
@@ -177,7 +176,7 @@ Agent 在执行过程中产生的推理/思考文本。前端应实时展示为
 
 1. **Agent 层面的 think tool**：`DataAgent` 使用 `think` 工具时，将 tool message 以 `thinking_text` 事件输出。
 2. **LLM 伴随内容**：当 LLM 在 tool_calls 旁返回文本 content 时，Route 层将其作为 `thinking_text` 事件输出。
-3. **（Phase 2/3）模型原生推理**：Anthropic extended thinking 或 OpenAI reasoning tokens（`reasoning_content` 字段），由 `client_utils.py` 解析后输出为 `thinking_text` 事件。
+3. **模型原生推理链保持**：部分推理模型（当前为 DeepSeek V4）在响应中返回 `reasoning_content` 字段，该字段在多轮对话中必须回传至 assistant 消息。已通过 `agent_utils.attach_reasoning_content()` 和 `accumulate_reasoning_content()` 在所有 tool-loop agent 中统一处理。未来 Anthropic extended thinking 或 OpenAI reasoning tokens 如需类似处理，可复用同一机制。
 
 **后端生成**：
 
@@ -272,7 +271,6 @@ if (parsed.text) { ... }
 | `/get-recommendation-questions` | `x-ndjson` | route 累积碎片 → `_try_parse_explore_line` | `stream_error_event` | ✅ `_with_warnings` |
 | `/generate-report-chat` | `x-ndjson` | route `json.dumps(event)` | `stream_error_event` | ✅ `_with_warnings` |
 | `/data-loading-chat` | `x-ndjson` | route `json.dumps(event)` | `stream_error_event` | ✅ `_with_warnings` |
-| `/clean-data-stream` | `x-ndjson` | agent 直接 yield | `stream_error_event` | ✅ `_with_warnings` |
 
 > **注意**: `/refine-data` 曾出现在此表中，但实际实现为普通 JSON endpoint（`jsonify` 返回），不使用 NDJSON 流。已于 2026-04-30 Phase 0 盘点中确认并移除。详见 `design-docs/20` 附录 A.10。