Skip to content

feat(antd): data streaming download — daemon wiring + REST SDK fan-out#196

Draft
Nic-dorman wants to merge 8 commits into
mainfrom
nic/v2-289-antd-stream-wiring
Draft

feat(antd): data streaming download — daemon wiring + REST SDK fan-out#196
Nic-dorman wants to merge 8 commits into
mainfrom
nic/v2-289-antd-stream-wiring

Conversation

@Nic-dorman

@Nic-dorman Nic-dorman commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What

Adds constant-memory streaming download for data, on the daemon + REST SDKs:

  • antd daemon: private POST /v1/data/stream + public GET /v1/data/public/{addr}/stream (REST), plus the gRPC DataService.Stream (private) / StreamPublic (public) service methods. A shared helper drives ant-core's file_download_to_sender; REST sets Content-Length from the DataMap so a short read signals failure (chunked transfer can't error after the 200 headers are sent).
  • REST SDK fan-out (all 16 SDKs): data_stream / data_stream_public (idiomatic streaming type per language) alongside the unchanged buffered data_get / data_get_public.

ant-core pin

Previously blocked on upstream WithAutonomi/ant-client#111 (Client::file_download_to_sender). That has merged into rc-2026.6.2, so antd/Cargo.toml now pins ant-core = 0.2.8-rc.1 (rev cd0beb4) and the dev-only [patch] override is dropped. cargo check --release is green against the published ref. Repin to the ant-cli-v0.2.8 tag once it is cut.

Validation (local + dev2 devnet)

  • Daemon E2E on a live 10-node devnet: 1 MiB public and private upload → stream back → byte-for-byte match.
  • antd-go SDK round-trip against the live daemon: public + private streams, exact byte match.
  • All 16 SDKs build + unit-test green. swift uses a URLSessionDataDelegate → AsyncThrowingStream because Darwin's URLSession.bytes is unavailable on Linux corelibs-foundation.

Follow-up

  • The gRPC SDK client fan-out (proto regen + per-language wrappers + the Swift core gRPC client) is split into a stacked PR on top of this branch.

🤖 Generated with Claude Code

Nic-dorman and others added 6 commits June 10, 2026 09:54
@Nic-dorman @claude
feat(antd): wire data streaming download — private + public, REST + g…
7c09fce 
…RPC (V2-289)

Replaces the stubbed stream endpoints with real constant-memory streaming
backed by ant-core's file_download_to_sender. Private streams from a
caller-held DataMap (the primitive); public resolves the address via
data_map_fetch then streams from the DataMap (wraps the private path).

- proto: new private rpc Stream(StreamDataRequest) alongside StreamPublic.
- gRPC: shared spawn_data_chunk_stream helper drives the download on a
  background task into a ReceiverStream<DataChunk>; terminal errors reuse
  the daemon's from_core -> Status mapping, after any chunks already sent.
- REST: POST /v1/data/stream (private) + GET /v1/data/public/{addr}/stream
  (public), both via Body::from_stream with Content-Length from
  DataMap::original_file_size() — a short read is the failure signal since
  chunked transfer can't error after the 200 headers.

Smoke-tested locally: both REST routes are live and validate (bad hex/addr
-> 400, valid addr reaches data_map_fetch), no longer 501 stubs.

Depends on ant-client#111 (file_download_to_sender); built locally via a
dev-only [patch] override that is intentionally NOT committed. CI stays red
until #111 merges upstream and antd's ant-core pin is bumped. See V2-289.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman @claude
feat(sdk): data streaming download in antd-go + antd-py REST (V2-289 …
2b0abac 
…fan-out)

Canonical references for the streaming fan-out, mirroring the daemon's new
private + public stream endpoints:

- antd-go: DataStream(dataMap) + DataStreamPublic(addr) returning io.ReadCloser
  (idiomatic streaming; caller reads + Closes). Shared doStream helper parses
  the JSON error body on non-2xx.
- antd-py: data_stream / data_stream_public on both RestClient and
  AsyncRestClient as (async) context managers yielding byte-chunk iterators,
  via httpx .stream(). _check_streamed reads the body before parsing errors.

Streaming is opt-in alongside the unchanged buffered data_get/data_get_public.
SDK clients have no ant-core dependency, so these build independently of #111;
only live integration tests need the streaming daemon (running on dev2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman @claude
feat(sdk): REST data streaming fan-out across remaining 14 SDKs (V2-2…
9237ad6 
…89 Phase 1)

Adds data_stream (private) + data_stream_public (public) — streaming
counterparts to the buffered data_get/data_get_public — to every remaining
SDK's REST client, each returning the language's idiomatic streaming type and
reusing existing HTTP plumbing + error mapping. Additive only; buffered methods
unchanged. Mirrors the antd-go/antd-py references.

Return types: rust impl Stream<Bytes>; js ReadableStream<Uint8Array>; csharp
Task<Stream>; kotlin okhttp ResponseBody; java InputStream; swift
URLSession.AsyncBytes; php PSR-7 StreamInterface; ruby block/Enumerator; dart
Stream<List<int>>; elixir Req into:self Stream; lua LTN12 sink callback; cpp
DataSink chunk callback; zig writer-based. antd-mcp adds a client-side
stream_download_file tool (MCP tools can't return live streams).

Locally built+tested green: rust, js, csharp, kotlin, java, php, ruby, dart,
cpp, mcp. Pending Linux (dev2) build: swift, elixir, zig, lua (toolchains not
on Windows).

Notes: streaming methods kept off shared transport interfaces (csharp/kotlin/
swift) to avoid forcing a gRPC impl — gRPC is Phase 2 (V2-499). antd-rust
grpc_tests.rs gains a no-op Stream mock because rust regenerates from the
daemon proto, which now carries the Stream rpc. SDK clients have no ant-core
dep, so this builds independently of ant-client#111.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman @claude
fix(antd-swift): cross-platform streaming via delegate AsyncThrowingS…
06254c0 
…tream

URLSession.bytes/AsyncBytes are Darwin-only and fail to compile on Linux
swift-corelibs-foundation. Rework dataStream/dataStreamPublic to a
URLSessionDataDelegate-driven AsyncThrowingStream<Data, Error> using
dataTask + resume, which compiles on both Darwin and Linux. Status check +
{"error"} envelope mapping preserved; buffered methods untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman @claude
chore(antd): repin ant-core to rc-2026.6.2 (0.2.8-rc.1) — drops the #111
8df835d 
 dev patch

ant-client #111 (file_download_to_sender, the streaming-download primitive the
V2-289 daemon wiring needs) is now merged into the rc-2026.6.2 branch. Repins
ant-core from the ant-cli-v0.2.6 tag to rev cd0beb4 (rc-2026.6.2) and removes
the local-path [patch] DO-NOT-MERGE override, so the daemon builds against a
published ref. self_encryption 0.36 stays compatible; cargo check --release
green. Repin to the ant-cli-v0.2.8 tag once it's cut.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman @claude
style(antd): apply cargo fmt to streaming handler
b54c8aa 
Wrap the shared stream_response signature per rustfmt — fixes the cargo fmt --all --check CI gate for the daemon streaming wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman Nic-dorman force-pushed the nic/v2-289-antd-stream-wiring branch from 706118f to b54c8aa Compare June 11, 2026 13:44
@Nic-dorman Nic-dorman changed the title [BLOCKED on ant-client#111] feat: data streaming download — daemon wiring + REST SDK fan-out feat(antd): data streaming download — daemon wiring + REST SDK fan-out Jun 11, 2026
@Nic-dorman Nic-dorman mentioned this pull request Jun 11, 2026
Open
Nic-dorman and others added 2 commits June 11, 2026 15:09
@Nic-dorman @claude
feat(antd): return chunks_stored + payment_mode_used from gRPC data put
24803b2 
The gRPC DataService put/put_public responses omitted these two fields, so SDK gRPC clients saw chunks_stored=0 / payment_mode_used="" even though REST data put and the gRPC file service both populate them from the same data_upload_with_mode result. Add the fields to PutDataResponse/PutPublicDataResponse and wire them through, mirroring the file-service handlers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman @claude
test(antd-rust): add new put fields to gRPC mock for proto change
f74dbcb 
antd-rust compiles the canonical antd/proto via build.rs, so the PutDataResponse/PutPublicDataResponse field additions in this PR make the exhaustive struct literals in the test mock require the two new fields. Client population of the fields lands with the rest of the SDK fan-out (stacked PR).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Nic-dorman Nic-dorman force-pushed the nic/v2-289-antd-stream-wiring branch from a2aa15a to f74dbcb Compare June 11, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Nic-dorman