Fix intermittent Windows ESP Account setup hang (user-scope 405 race)

[lukeheath]

Related issue: Resolves #49134

Background for reviewers new to Windows MDM

Expand for a plain-language primer on ESP, node scopes, and SyncML Add/Replace

What's the ESP? When a Windows device is set up through Autopilot (the out-of-box "Account setup" flow you see on a brand-new work laptop), Windows shows an Enrollment Status Page (ESP) — the "Working on it…" spinner — and blocks the user from reaching the desktop until the MDM server (Fleet) says setup is done. There are two sub-phases: Device setup (before anyone logs in) and Account setup (after the user's profile starts loading).

How does Fleet tell Windows "you're done"? Windows MDM is a protocol called SyncML/OMA-DM. The device holds a tree of configuration nodes (think of a filesystem of settings). Fleet sends commands to that tree:

• Add = create a node (like mkdir).
• Replace = set a node's value (like writing to a file).

To release the ESP, Fleet sets a node called ServerHasFinishedProvisioning to true. Crucially, this node exists in two separate places in the tree:

• Device scope (./Device/...) — completes the "Device setup" phase. This node exists by default.
• User scope (./User/...) — completes the "Account setup" phase. This node does not exist by default; Fleet has to create it first with Add.

What went wrong? Fleet created the user-scope node only during an early "hold" phase, and only while it hadn't yet matched the device to a Fleet host record (host_uuid). Once fleetd installs during setup, that match happens — sometimes before the hold phase ever ran. In that case the Add was skipped, the user-scope node was never created, and when Fleet later sent the Replace to release the device, it hit a node that doesn't exist. The device answered with SyncML status 405 ("not allowed" / no such target), the "Account setup" phase never completed, and the machine sat on "Working on it…" until a 3-hour timeout. Because it was a timing race, it happened only sometimes — which is what made it hard to pin down.

The fix in one sentence: send the Add again as part of the release itself, right before the Replace, so the node is guaranteed to exist regardless of timing. Re-adding a node that already exists is harmless — the device just replies 418 ("Already Exists"), which SyncML ignores.

Summary

Windows Autopilot / Entra OOBE enrollments intermittently hung on the Enrollment Status Page at "Account setup → Working on it…" until the 3-hour ESP timeout. Fleet had run the ESP lifecycle to completion, but the user-scope release command (./User/.../DMClient/Provider/Fleet/FirstSyncStatus/ServerHasFinishedProvisioning) failed on-device with SyncML 405, so the user ("Account setup") phase never got its completion signal.

Root cause (a race, latent since the ESP feature shipped in 4.86.0): the user-scope DMClient Provider node is created by Add commands sent only while host_uuid == "" in the Pending/hold phase (handleESPHoldOrTransition). The release phase (buildESPReleaseCommands) then writes the user-scope Replace unconditionally, with no Add. When host_uuid is linked — by orbit enroll, the DevDetail/SMBIOS reply, or the osquery backstop — before the first Pending-phase hold session runs, the Adds are skipped, the user node is never created, and the release Replace lands on a nonexistent node → 405 → hang.

The intermittency ("sometimes a host goes through, sometimes it doesn't") is whichever wins between the ~1-minute ESP hold poll and host_uuid linking. The window has been progressively narrowed by changes that link host_uuid earlier (#46268 DevDetail-at-first-session linking in 4.87.0; fleetd install-ordering changes in the 4.89.0 RC), which is why it now surfaces as a P1.

Fix

• buildESPReleaseCommands: prepend two idempotent Adds for the user-scope Provider node and its FirstSyncStatus child, ordered before the user-scope Replace. SyncML processes commands in message order, so the node exists by the time the Replace runs; when it already exists the Adds return a harmless 418 (Already Exists). This makes the release self-sufficient regardless of who wins the host_uuid linking race.
• handleResendingAlreadyExistsCommands: skip container-node Adds (format=node, no <Data>) in the 418→Replace rewrite. The release now persists two such Adds for its dropped-response retry; without this guard, the common-case 418 (node already exists) would rewrite them into undefined container Replace commands that the device rejects, emitting spurious failures post-ESP. Leaf-value Adds (profiles, DMClient settings) still convert to Replace.

Deferred as a follow-up (#49252): gating awaiting_configuration = None on the user-scope Replace acking 200, so an unrelated future failure of that command can't silently strand a device. Not required for this P1.

Checklist for submitter

• Changes file added for user-visible changes in changes/.

• Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.

• Timeouts are implemented and retries are limited to avoid infinite loops

Testing

• Added/updated automated tests
  • TestPBT_HandleESPRelease (property-based): asserts the release path includes the user-scope node Adds and that they are ordered before the user-scope ServerHasFinishedProvisioning Replace.
  • TestHandleResendingAlreadyExistsCommands (unit): container-node Adds are skipped in the 418 rewrite while leaf-value Adds still convert to Replace.
  • TestWindows...ESP integration test (already simulates the host_uuid-linked-before-hold race): asserts the node Adds reach the wire in the release session.
• Where appropriate, automated tests simulate multiple hosts and test for host isolation.
• QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

• Confirmed that the fix is not expected to adversely impact load test results (adds two idempotent SyncML commands to the one-time ESP release message; no new DB calls, no steady-state impact)
• Alerted the release DRI if additional load testing is needed

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.08%. Comparing base (aba0c2d) to head (5b6dbf2).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #49250      +/-   ##
==========================================
- Coverage   68.08%   68.08%   -0.01%     
==========================================
  Files        3784     3784              
  Lines      239240   239244       +4     
  Branches    12575    12575              
==========================================
- Hits       162893   162892       -1     
  Misses      61571    61571              
- Partials    14776    14781       +5     

Flag	Coverage Δ
backend	69.56% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.