bundle: add DATABRICKS_BUNDLE_MANAGED_STATE gate + DMS-backed deployment lock#5331
Draft
shreyas-goenka wants to merge 1 commit into
Draft
Conversation
…ent lock Wires up a new env-var gate (DATABRICKS_BUNDLE_MANAGED_STATE) that switches the deployment lock from the workspace filesystem to the bundle deployment metadata service (DMS). The env var accepts the usual boolean spellings (true/false, 1/0, yes/no, on/off); the historical filesystem lock is the default. The DMS-backed lock uses the SDK's databricks-sdk-go/service/bundle client (merged via #5311, available since v0.135.0) — no hand-rolled DMS client. Acquire calls CreateDeployment / CreateVersion and starts a background heartbeat goroutine; Release stops the heartbeat and calls CompleteVersion (plus DeleteDeployment on successful destroy). Bind and Unbind are not yet supported under DMS and return an error at lock construction. Deployment ID persistence lives inside the lock package for now (managed_service.json in the workspace state dir). In step 4 of the DMS split it will move to bundle/statemgmt so it can be shared with the state-from-DMS path. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 3 of the DMS split (see PR #4856 for the full kitchen-sink change).
DATABRICKS_BUNDLE_MANAGED_STATEenvironment variable(
bundle/env/deployment_metadata.go). The value is parsed with the usualboolean tolerance —
true/false,1/0,yes/no,on/off(case-insensitive). Default behavior (var unset) is unchanged.
DeploymentLockimplementation that calls the bundleservice through the SDK client. Acquire =
CreateDeployment(for freshdeployments) /
GetDeployment+CreateVersion, plus a backgroundheartbeat goroutine that renews the lock lease. Release =
CompleteVersion, plusDeleteDeploymenton successful destroy.lock.NewDeploymentLockto branch onenv.IsManagedState(ctx)and return the DMS impl, falling back to the existing workspace
filesystem lock otherwise. The factory now takes
ctxso it can readthe env var — phases call sites updated to pass it.
libs/testserver(modeled on the existingapps.gopattern) and registers the routes inhandlers.go.acceptance/bundle/dms/release-lock-erroracceptance testthat exercises a
CompleteVersionfailure end-to-end.Dependencies
DeploymentLockinterface + filesystem impl extraction).Retarget to
mainonce that lands.databricks-sdk-go/service/bundleintroduced in v0.135.0 / bumpedto v0.136.0 in Bump databricks-sdk-go to v0.136.0 #5311 — no
libs/tmpdmsshim.Notes vs the kitchen-sink branch (
shreyas-goenka/deployment-metadata-service)libs/tmpdms(hand-rolled DMS typeswritten before the SDK had the bundle service). This PR uses the real
SDK package instead. The on-the-wire schema matches — request/response
bodies and URL shapes are identical — but Go type names differ
(
tmpdms.CreateDeploymentRequest.DeploymentID→ SDKbundle.CreateDeploymentRequest.DeploymentId; enum names switched fromtmpdms.VersionTypeDeployto SDKbundle.VersionTypeVersionTypeDeploy, etc.).deployment_metadata_service.goreads/writesbundle/statemgmt.ManagedServiceJSON/ManagedServiceFileName. Thoselive in
statemgmt/managed_service_json.goin the kitchen-sink branch— that file is step 4 of the split. To keep this PR scoped to the lock,
the deployment-ID persistence (struct + filename) is kept as
package-private inside
bundle/deploy/lockand will be promoted tostatemgmtwhen step 4 lands andstate_pull.go/state_push.goalsoneed it.
async_reporter.gofrom the kitchen-sink branch is intentionally notincluded here. It exists to feed operation events into the DMS, which
requires the
direct.OperationReporterhook onDeploymentBundlethat lives in step 5. The lock works end-to-endwithout it; the heartbeat goroutine that the original task description
conflated with
async_reporter.golives indeployment_metadata_service.go(startHeartbeat).env.IsManagedState. The kitchen-sink branch usesstatemgmt.IsDmsActive, which also factors in server-side opt-in. Thatpredicate is part of step 4; promoting the gate to it is an open item
for the next PR.
Test plan
go build ./...clean.go vet ./...clean (ignoring pre-existing JSON tag warnings).go test ./bundle/... ./libs/testserver/... ./cmd/...— all green.go test ./acceptance -run "TestAccept/bundle/dms/release-lock-error"passes without
-update../task lintclean.IsManagedStatereturns the right bool fortrue,false,TRUE,yes,1,0,no,off,garbage, and unset.(
acceptance/bundle/deploy/empty-bundle,bundle/deploy/files,bundle/destroy) still pass — the defaultDATABRICKS_BUNDLE_MANAGED_STATEpath is the workspace filesystem lock, unchanged from PR bundle: extract DeploymentLock interface + workspace filesystem impl #5314.This pull request and its description were written by Isaac.