Windows: engine silently dies ("engine gone - run was interrupted") - save_state os.replace() WinError 5 sharing violation with TUI reader

[dracic]

Summary

On Windows a long-running engine can die silently mid-run. The TUI then shows "engine gone — run was interrupted — press e to resume", the run is frozen at phase=dev-running, and state.json has crashed=false / stopped=false (no crash recorded). Root cause: save_state()'s atomic write os.replace(state.json.tmp → state.json) raises PermissionError: [WinError 5] Access is denied because the TUI poller holds an open read handle on state.json while the engine tries to rename over it.

This is a Windows-only defect — it does not reproduce on Linux/WSL/macOS.

Environment

• Windows 11, Python 3.x, native-Windows multiplexer + WindowsProcessHost
• Reproduced against a live bmad-dev-auto run

Root cause

src/bmad_loop/journal.py:

def save_state(run_dir: Path, state: RunState) -> None:
    ...
    tmp.write_text(json.dumps(state.to_dict(), indent=2), encoding="utf-8")
    os.replace(tmp, target)          # <-- WinError 5 here

load_state() reads state.json via Path.read_text(), i.e. open() without FILE_SHARE_DELETE. On Windows MoveFileExW (what os.replace calls) fails with ERROR_ACCESS_DENIED when another handle on the target is open. The TUI dashboard polls state.json on every tick via load_state(), so writer and reader collide stochastically.

On POSIX rename(2) over an open fd always succeeds (the reader keeps the old inode), which is why this only bites on Windows.

How it was found

1. A frozen run showed a dead engine pid but state.json had just been written (mtime matched the last Stop event) — yet journal.jsonl had no session-end line, and crashed=false.

2. That pinpointed the death to the window in _run_session between the durability self._save() (engine.py:~1994) and the journal.append("session-end") a few lines later — the code comment there already anticipates a host kill in that window.

3. First occurrence was silent (uncaught); on resume the same collision surfaced as a real crash:

   PermissionError: [WinError 5] Access is denied:
   '...\runs\<id>\state.json.tmp' -> '...\runs\<id>\state.json'
   

   which confirmed the os.replace sharing violation as the mechanism.

Broader hazard class

This is one instance of "POSIX-safe idiom, Windows-unsafe". A short research pass surfaced a ranked checklist we intend to work through (each with a grep detection heuristic + fix pattern):

• H1 — atomic replace over a concurrently-read file (this issue). pip wraps its wheel replace() in retries; npm graceful-fs backs off up to 60s for EACCES/EPERM/EBUSY.
• H2 — os.kill(pid, 0) is a destructive CTRL_C_EVENT on Windows (it maps to GenerateConsoleCtrlEvent, not an existence probe). Already mitigated by WindowsProcessHost (psutil); worth a guard against stray callers.
• H3 — console close/logoff/shutdown events are not deliverable to Python signal handlers → silent engine death; only resumable checkpoints save you.
• H4 — taskkill /F /T tree blast radius + Windows PID reuse (guard force-kills with a create-time identity match).
• H5 — CREATE_NEW_PROCESS_GROUP / CTRL_C_EVENT cannot target a single process (hits the whole console).
• H6 — AV/Defender/Search-indexer transient handle holds widen the H1 window beyond just the TUI reader.
• H7 — long paths (>260), UTF-8 default encoding, CRLF, reserved filenames.

Proposed fix

Route save_state through a retry helper — retry os.replace on PermissionError with short jittered backoff, gated to sys.platform == "win32" so a genuine POSIX EACCES/EPERM still surfaces immediately:

def _atomic_replace(tmp: Path, target: Path) -> None:
    for attempt in range(_REPLACE_RETRIES):
        try:
            os.replace(tmp, target)
            return
        except PermissionError:
            if sys.platform != "win32" or attempt == _REPLACE_RETRIES - 1:
                raise
            time.sleep(_REPLACE_BACKOFF_S * (attempt + 1))

Follow-ups: factor this into one shared atomic_replace for every os.replace/rename-over site (audit needed), widen the backoff to survive AV/indexer holds (H6), and add guards/tests for H2/H4/H5.

Plan / offer

I have this implemented and unit-tested locally and am happy to open a PR (the save_state retry first, then the broader audit as separate PRs). Self-assigning to track it.

Repro-once check: keep the TUI open on a running story on Windows; the collision is timing-dependent but recurs within a few state writes under active polling.

[coderabbitai]

🔗 Related PRs

#53 - fix(adapters): complete dev sessions only on Stop or window death [merged]
#54 - feat(scm): bounded retention for attempt-preserve-dirty worktree-snapshot refs [merged]
#62 - fix(engine): session-end durability + raw KeyboardInterrupt handling [merged]
#63 - fix(engine): close resume-replay edges from #62 (final-cycle replay, stale result_json, record bloat) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!