Skip to content

Release Miles sync port claims#22

Open
TianyeGGBond wants to merge 1 commit into
rlops:zhenyu/miles-mvp-e2efrom
TianyeGGBond:codex/miles-release-port-claim
Open

Release Miles sync port claims#22
TianyeGGBond wants to merge 1 commit into
rlops:zhenyu/miles-mvp-e2efrom
TianyeGGBond:codex/miles-release-port-claim

Conversation

@TianyeGGBond

@TianyeGGBond TianyeGGBond commented Jun 20, 2026

Copy link
Copy Markdown
Collaborator

Context

MilesModelUpdateService claims NCCL rendezvous ports in SharedStorage while building sync plans. The MILES path recorded the claim but never released it after the cache-owner transport completed, so successful syncs could leave stale MASTER_ADDR_PORT:* keys behind over a long-running job.

@TianyeGGBond TianyeGGBond force-pushed the codex/miles-release-port-claim branch from 343a6c8 to f5c8829 Compare June 20, 2026 23:14
@TianyeGGBond TianyeGGBond force-pushed the codex/miles-release-port-claim branch from f5c8829 to 74f5152 Compare June 20, 2026 23:18
@TianyeGGBond TianyeGGBond marked this pull request as ready for review June 20, 2026 23:20
@TianyeGGBond TianyeGGBond changed the title [codex] Release Miles sync port claims Release Miles sync port claims Jun 20, 2026
@JunzheJoe

JunzheJoe commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Reviewed this as part of the F1-F12 code review — supportive. Two notes before merge:

  1. The failure/timeout path still leaks the claim. This PR releases on the successful-transport path (_release_port_claim fires right after run_sync_session returns — correct placement, since the port is only needed for the NCCL rendezvous during transport). But if run_sync_session raises, or the atomic unit's asyncio.wait_for cancels mid-flight (_cancel_inflight fires ray.cancel on the inflight refs but never releases the port), _release_port_claim is skipped and the claim lingers until orchestrator.kill_pipeline's delete_port_claims. Since every sync claims a fresh MASTER_ADDR_PORT:{addr}:{port} key, repeated failures still grow the claim table, and OS port reuse can collide with stale claims and burn the 8-attempt retry budget. Suggestion: keep the current post-transport release, and additionally release in the timeout/cancellation handler (or restructure as try/finally around the transport step).

  2. The claim currently serves a dead path. master_addr/port is only consumed by the NCCL broadcast transport, and that path is double-guarded with NotImplementedError (service-side sync_selected_workers L177-190, miles-side _dispatch_nccl_broadcast P1-8 self-guard). cpu_serialize never touches the port. An alternative to hardening the release: skip claiming entirely until broadcast lands — which would also intersect with delete unimplemented broadcast transport path [need discussion] #24's discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants