fix(upload): refresh Entra token per request + bounded retry for large backfills (#338)

[colombod]

Summary

Fixes microsoft-amplifier/amplifier-support#338. The batch uploader
(context-intelligence-upload) was not robust for large Entra/APIM-protected
backfills for two independent reasons: (1) the Entra bearer token was captured once
and baked into the httpx.Client, so it was never refreshed — every POST past the
~67 min token lifetime returned 401 and the run aborted; and (2) the POST loop
returned failure on the first httpx.HTTPError or any non-2xx, so a single
transient blip aborted an entire multi-thousand-event run. Uploads are idempotent
(SHA-256 key + Neo4j MERGE), so this is purely a robustness/operability fix.

The fix fetches the auth header per request (so EntraTokenAuth's existing
cache-and-refresh actually fires mid-run) and wraps each POST in a bounded
exponential backoff. Transient = 5xx / 429 / connection-reset / timeout;
permanent = 4xx other than 429, 3xx, and DNS/TLS transport failures — fail fast,
loud. Two small operability follow-ups are included: --timeout to tune the
read/write timeout, and cause-classification of ConnectError so DNS/TLS faults
fail fast instead of burning the retry budget on a dead host.

Scope / guardrails

• Change is confined to ONE module: modules/tool-context-intelligence-upload
  (uploader.py + cli.py) plus its tests. context_intelligence/auth.py is
  untouched — the fix simply exercises its existing refresh path.
• The write-side hook fan-out (fanout.py / _DestinationDispatcher /
  logging_handler) is untouched. The uploader deliberately adapts (does not
  import) the hook's _classify_http_outcome: the uploader must treat 401 as
  PERMANENT (per-request refresh owns token expiry), whereas the hook treats it
  as transient. Copying it verbatim would silently reintroduce the bug.
• Static/API-key path preserved: ApiKeyAuth.headers() is constant, so per-request
  fetch is identical to the old baked-once behavior.
• New CLI flags are additive with safe defaults — existing invocations behave
  identically except that transient failures now retry instead of aborting.
• No bundle structure changed.

Verification

• Module tests pass — modules/tool-context-intelligence-upload: 233 passed
• Top-level tests pass — tests/: 753 passed
• ruff check + ruff format --check clean (19 files already formatted)
• pyright clean (0 errors)
• Full bundle validation — N/A: no bundle structure changed (Python-only
  change in one module; bundle.dot/bundle.png unaffected, BUNDLE_DOT_STALE
  not triggered).

Real evidence on seams (not mock-only)

The client↔server boundary is the seam this change touches. Proven with a LIVE
run of the modified CLI against the team-shared Entra/APIM Context Intelligence
server (--auth-mode entra, real az-backed token, --max-retries 3), uploading a
real session — run twice to prove idempotent re-run (no duplicates):

run 1: {"status":"completed","sessions_uploaded":1,"events_uploaded":3}
run 2: {"status":"completed","sessions_uploaded":1,"events_uploaded":3}

Both bug paths are additionally proven by deterministic unit tests: mid-run token
rollover (a near-expiry credential re-fetched per POST → later POST carries the new
bearer), transient-retry-then-succeed, persistent-transient fails after exactly
max_retries+1, permanent 4xx / 401 / DNS / TLS fail fast with zero retries, and
headers() raising fails loud instead of crashing the run.

• Seam(s) crossed by this change are proven against real behavior (live Entra/APIM upload, idempotent re-run)

Docs & diagrams

• bundle.dot / bundle.png — N/A, bundle structure unchanged
• README / SKILLs / agent files — N/A, no tool/skill contract changed (two
  additive optional CLI flags with safe defaults: --max-retries, --timeout)
• Convention files — N/A, no lasting repo-level lesson surfaced

Notes / follow-ups

• Non-blocking: honor the HTTP-date form of Retry-After (the numeric-seconds form
  is honored today); server-side hardening of the lone-child placeholder-parent path
  (the "minor" item in #338, server-side, out of scope for this client change).

[colombod]

Additional evidence — large-session live upload (long-running backfill)

Following the smaller live upload in the PR body, I ran the fixed CLI against the
largest root session on disk to prove the fix holds at real backfill scale over a
long, token-lifetime-spanning run — the exact failure case of Bug 1.

Target: session 36ea1b8e-0704-4576-9304-c6a40e5b6599 — 11,110 events / ~685 MB

Command: modified context-intelligence-upload against the team-shared Entra/APIM
server, exercising both new flags:
--auth-mode entra --max-retries 5 --timeout 60

Run: ~30 minutes of continuous uploading (spans well past a token-lifetime window,
i.e. the scenario the old baked-once header died on). Progress from the job tracker:

events sent: 270 → 2656 → 4310 → 5955 → 7637 → 9268 → 10689 → 11110 ✓ session completed
failed_at:   null throughout
transient retries triggered: 0     warnings/errors: 0

The 11,110-event root uploaded to completion and the run advanced into its session
tree before I stopped it (idempotent — safe to stop/resume).

What each layer of evidence covers:

Evidence	Proves
Small session, run twice (PR body)	Entra path + idempotent re-run
This 11,110-event / ~30-min run	Per-request token stays valid across a long token-lifetime-spanning backfill at scale — Bug 1's failure case, now clean
Unit tests (233 module + 753 top-level)	The failure branches a healthy live network won't trigger: mid-run token rollover (credential re-fetched → new bearer on a later POST), transient-retry-then-succeed, persistent-transient fail-after-N, permanent 4xx/401/DNS/TLS fail-fast, headers()-raises fail-loud

The live network was healthy (0 transient blips), so the retry path was not naturally
exercised here — that half is carried by the deterministic unit tests. Together: real
scale + real Entra auth from the live run, all failure branches from the tests.