feat: optional AWS Lambda MicroVM stateful sandbox backend

[danny-avila]

What

Adds AWS Lambda MicroVMs as an optional, config-gated stateful execution backend for the Code Interpreter, giving perceived-indefinite statefulness (a warm per-session workspace plus checkpoint/restore across the VM's 8h lifetime) without changing the legacy semi-stateless HTTP path.

Design

• A SandboxBackend seam is extracted from the worker dispatch. The http backend is byte-identical to today. CODEAPI_SANDBOX_BACKEND=lambda-microvm opts in, and @aws-sdk/* is lazily imported so http deployments never load it.
• Server-derived runtime_session_id = hash(storageNamespace, canonicalUserId, hint) plus a Redis registry (SET NX locks, Lua CAS release, generation fencing) pins one VM per session. CODEAPI_RUNTIME_SESSION_MODE=stateless|affinity|strict.
• Persistent session workspace: a session-bound VM reuses one /mnt/data and pinned UID across calls (prime dedup and output diffing), gated by SANDBOX_SESSION_WORKSPACE_ENABLED (Lambda image only) plus a per-request signal.
• Checkpoint/restore of the whole workspace (tar.gz over the authed proxy, control plane owns S3) restores a session into a replacement VM after expiry. That is the differentiator over the file-ref system.
• Native idlePolicy (autoResume) handles idle suspend/resume, so the sweeper shrinks to registry hygiene.

Hookless session binding

Session mode is delivered per request via an X-Runtime-Session-Id header rather than a /run lifecycle hook. Lambda's image build hooks only route on the snapshot-compatible base container image, and enabling any runtime hook forces the /ready build hook, which never reaches a stock container's listener (builds then fail at the ready timeout). Hookless image builds are reliable; checkpoint/restore already runs over plain HTTP endpoints, and idle suspend/resume is native.

Proven live

Real MicroVM: launch ~3s, suspend/resume ~2.2s with process continuity, auto-resume ~1.2s. Two-turn E2E on a real hookless MicroVM: with the header, /mnt/data persists across separate /execute calls (42 read back); without it, fresh-per-job (ABSENT).

Tests

~630 api and service tests (ioredis-mock, fake AWS client, and Bun.serve fakes).

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[macroscopeapp]

🟡 Medium api/lifecycle.ts:48

applyRunHook permanently locks in the very first /run body, even when it is malformed or missing both microvmId and runHookPayload. Line 48 stores { microvmId: undefined, runHookPayload: undefined } in runContext, so any later retry with the real values falls through the else if branch (both ids are undefined, so they don't mismatch) and never re-initializes or binds the session workspace. A transient bad first payload thus leaves the VM with an empty runContext for its whole lifetime. Consider gating the else if so retries with real values overwrite an uninitialized runContext — e.g. only treat the existing context as locked when it has a non-null microvmId.

-  if (runContext == null) {
+  if (runContext == null || runContext.microvmId == null) {

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @api/src/api/lifecycle.ts around line 48:

`applyRunHook` permanently locks in the very first `/run` body, even when it is malformed or missing both `microvmId` and `runHookPayload`. Line 48 stores `{ microvmId: undefined, runHookPayload: undefined }` in `runContext`, so any later retry with the real values falls through the `else if` branch (both ids are `undefined`, so they don't mismatch) and never re-initializes or binds the session workspace. A transient bad first payload thus leaves the VM with an empty `runContext` for its whole lifetime. Consider gating the `else if` so retries with real values overwrite an uninitialized `runContext` — e.g. only treat the existing context as locked when it has a non-null `microvmId`.

[macroscopeapp]

🟠 High src/session-workspace.ts:172

getBoundSessionWorkspace() always returns the previously bound global session, even for /execute requests that do not carry a valid X-Runtime-Session-Id. After one sessioned request binds a workspace on the runner, any later headerless or malformed-header request still runs inside that persistent workspace, reusing prior files and checkpoint state instead of the documented fresh-per-job path. That can leak one session's data into another request and breaks the per-request opt-in contract. Consider gating the return on whether the current request actually opted in, or clearing boundSession when the header is absent or malformed.

Also found in 2 other location(s)

api/src/job.ts:803

prime() now switches into persistent-workspace mode whenever this.session is set, but getJob() populates that field from getBoundSessionWorkspace() even when the current request does not send X-Runtime-Session-Id. Once a VM has been bound once, a later /execute without the header will still hit this branch and reuse the previous session's workspace/UID, leaking files and state across requests instead of falling back to a fresh per-job sandbox.

api/src/api/v2.ts:208

getJob() now passes session: getBoundSessionWorkspace() ?? null even when the current request did not include X-Runtime-Session-Id. Because bindSessionWorkspace(undefined) is never called and getBoundSessionWorkspace() returns the previous global boundSession, any non-session /execute that follows a session-bound request will incorrectly reuse that prior session workspace and pinned UID. This leaks state across requests and can let one caller read/write another caller's persistent workspace.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @api/src/session-workspace.ts around line 172:

`getBoundSessionWorkspace()` always returns the previously bound global session, even for `/execute` requests that do not carry a valid `X-Runtime-Session-Id`. After one sessioned request binds a workspace on the runner, any later headerless or malformed-header request still runs inside that persistent workspace, reusing prior files and checkpoint state instead of the documented fresh-per-job path. That can leak one session's data into another request and breaks the per-request opt-in contract. Consider gating the return on whether the current request actually opted in, or clearing `boundSession` when the header is absent or malformed.

Also found in 2 other location(s):
- api/src/job.ts:803 -- `prime()` now switches into persistent-workspace mode whenever `this.session` is set, but `getJob()` populates that field from `getBoundSessionWorkspace()` even when the current request does not send `X-Runtime-Session-Id`. Once a VM has been bound once, a later `/execute` without the header will still hit this branch and reuse the previous session's workspace/UID, leaking files and state across requests instead of falling back to a fresh per-job sandbox.
- api/src/api/v2.ts:208 -- `getJob()` now passes `session: getBoundSessionWorkspace() ?? null` even when the current request did not include `X-Runtime-Session-Id`. Because `bindSessionWorkspace(undefined)` is never called and `getBoundSessionWorkspace()` returns the previous global `boundSession`, any non-session `/execute` that follows a session-bound request will incorrectly reuse that prior session workspace and pinned UID. This leaks state across requests and can let one caller read/write another caller's persistent workspace.

[macroscopeapp]

🟠 High src/session-workspace.ts:160

When bindSessionWorkspace() is called with a different runtimeSessionId, it fires boundSession.reset() in the background and immediately assigns boundSession to a new SessionWorkspace. The old session's reset() calls resetSessionWorkspace(), which wipes the fixed session workspace directory and returns the pinned UID to the pool. If the new session calls acquire() before that background reset finishes, the old reset deletes the new session's live workspace and releases its UID slot while both are still in use. The race window is open as soon as a second binding arrives while the first session is still active. Await boundSession.reset() before constructing the replacement so the teardown fully completes before the new session can acquire the workspace.

Suggested change

	export function bindSessionWorkspace(binding: SessionBinding | undefined): SessionWorkspace | undefined {
	if (!binding) return boundSession;
	if (boundSession && boundSession.runtimeSessionId === binding.runtimeSessionId) {
	return boundSession;
	}
	if (boundSession) {
	void boundSession.reset().catch((err) => logger.error({ err }, 'Failed to reset superseded session workspace'));
	}
	boundSession = new SessionWorkspace(binding);
	return boundSession;
	export async function bindSessionWorkspace(binding: SessionBinding | undefined): Promise<SessionWorkspace | undefined> {
	if (!binding) return boundSession;
	if (boundSession && boundSession.runtimeSessionId === binding.runtimeSessionId) {
	return boundSession;
	}
	if (boundSession) {
	await boundSession.reset().catch((err) => logger.error({ err }, 'Failed to reset superseded session workspace'));
	}
	boundSession = new SessionWorkspace(binding);
	return boundSession;
	}

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @api/src/session-workspace.ts around lines 160-169:

When `bindSessionWorkspace()` is called with a different `runtimeSessionId`, it fires `boundSession.reset()` in the background and immediately assigns `boundSession` to a new `SessionWorkspace`. The old session's `reset()` calls `resetSessionWorkspace()`, which wipes the fixed session workspace directory and returns the pinned UID to the pool. If the new session calls `acquire()` before that background reset finishes, the old reset deletes the new session's live workspace and releases its UID slot while both are still in use. The race window is open as soon as a second binding arrives while the first session is still active. Await `boundSession.reset()` before constructing the replacement so the teardown fully completes before the new session can acquire the workspace.

[macroscopeapp]

🟡 Medium runtime-session/registry.ts:184

touchRuntimeSessionActive adds the session id to the rtsx:active zset without any expiry, so when a session record naturally expires (its rtsx:sess:<id> key times out) and no explicit removeRuntimeSession or forgetRuntimeSessionActive runs, the zset entry is left behind permanently. As a result, listIdleRuntimeSessions() keeps returning orphaned session ids and countActiveRuntimeSessions() keeps overcounting, so active-session bookkeeping becomes permanently wrong after normal TTL expiry. The session record keys use EX TTLs but Redis sorted sets have no per-member expiry, so nothing prunes these entries unless an explicit removal path runs. Consider coupling zset cleanup to record expiry (e.g. scheduling a delayed zrem, having the sweeper cross-check against the record key, or documenting that an external reaper is required).

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @service/src/runtime-session/registry.ts around line 184:

`touchRuntimeSessionActive` adds the session id to the `rtsx:active` zset without any expiry, so when a session record naturally expires (its `rtsx:sess:<id>` key times out) and no explicit `removeRuntimeSession` or `forgetRuntimeSessionActive` runs, the zset entry is left behind permanently. As a result, `listIdleRuntimeSessions()` keeps returning orphaned session ids and `countActiveRuntimeSessions()` keeps overcounting, so active-session bookkeeping becomes permanently wrong after normal TTL expiry. The session record keys use `EX` TTLs but Redis sorted sets have no per-member expiry, so nothing prunes these entries unless an explicit removal path runs. Consider coupling zset cleanup to record expiry (e.g. scheduling a delayed `zrem`, having the sweeper cross-check against the record key, or documenting that an external reaper is required).

[macroscopeapp]

🟡 Medium runtime-session/registry.ts:144

releaseRuntimeSessionLock swallows every Redis/script failure and returns as if the lock was released. When the DEL script fails (e.g. Redis temporarily unavailable), the lock key remains set for the full RUNTIME_SESSION_LOCK_TTL_MS, but callers proceed as if the session was unlocked. Later requests for the same runtime_session_id will then see the session as busy and 409 (strict) or fall back to stateless mode (affinity) until the stale lock finally expires. Consider rethrowing the error so callers can treat the session as still locked, or documenting why silent failure is acceptable here.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @service/src/runtime-session/registry.ts around line 144:

`releaseRuntimeSessionLock` swallows every Redis/script failure and returns as if the lock was released. When the `DEL` script fails (e.g. Redis temporarily unavailable), the lock key remains set for the full `RUNTIME_SESSION_LOCK_TTL_MS`, but callers proceed as if the session was unlocked. Later requests for the same `runtime_session_id` will then see the session as busy and 409 (`strict`) or fall back to stateless mode (`affinity`) until the stale lock finally expires. Consider rethrowing the error so callers can treat the session as still locked, or documenting why silent failure is acceptable here.

[danny-avila]

Addressed the review findings in 3c7e9d3.

Fixed:

• registry.ts lock TTL — now derived from the actual launch + health + execute + checkpoint budgets (was a 60s placeholder that could expire mid-work even at defaults).
• registry.ts readRuntimeSessionRecord — JSON.parse guarded; a corrupt key now reads as missing instead of throwing.
• checkpoint.ts — fenced record write now happens before the deterministic-key S3 put, so a lock-expired caller can't clobber a newer blob; restore caps size against maxBytes before buffering.
• session-checkpoint.ts — lchown + never follow symlinks when chowning a restored (untrusted) checkpoint.
• lambda-client-aws.ts — throw on a missing microvmId instead of returning '' (avoids orphaning a billable VM on a partial RunMicrovm).
• router.ts — skip hint validation in stateless mode (the field is ignored there, so a bad hint no longer 400s).
• v2.ts — only run in the persistent workspace when the current request carried a valid X-Runtime-Session-Id; a headerless request never inherits a prior session.
• entrypoint.sh — RLIMIT_NOFILE only ever raised, never clamped down.
• secure-startup.ts — warn on affinity+http instead of silently no-op'ing.

Deferred, with reasoning:

• registry.ts rtsx:active zset orphan — valid, but nothing consumes that zset in this PR; it's handled by the (separate) sweeper PR that owns idle-session bookkeeping.
• session-workspace.ts bindSessionWorkspace rebind race — only reachable if two different runtime_session_ids bind the same VM, which the one-VM-per-session invariant prevents (each VM is registry-pinned to a single session; the backend only ever sends that session's id).
• registry.ts releaseRuntimeSessionLock swallowing errors — the lock has a TTL that self-heals; rethrowing would mostly mask the real error in the caller's finally.
• lifecycle.ts applyRunHook empty-context lock-in — the /run lifecycle hook is inert in the shipping (hookless) design; session mode is delivered per request via the header.
• lifecycle.ts startupApiOnly policy placement — a real split-deployment question (should API-only pods validate the sandbox backend?) worth deciding deliberately rather than dropping the check.

[danny-avila]

@codex review

[danny-avila]

@codex review

[macroscopeapp]

🟡 Medium terraform/main.tf:234

The checkpoint_access IAM policy is documented as attachable to the CodeAPI task role, but in that deployment path every checkpoint read/write fails with empty credentials. The runtime checkpoint client constructs MinioClient using only MINIO_ACCESS_KEY/MINIO_SECRET_KEY (defaulting both to empty strings) and never loads IAM role credentials the way file-server.ts does, so when the policy is attached to the task role instead of minting static keys for the optional IAM user, all put/get calls authenticate with empty strings and session checkpoint/restore is broken. Either add IRSA/task-role credential loading to the checkpoint client or update the documentation to state that only the static-key user path is supported.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @docs/lambda-microvm/terraform/main.tf around line 234:

The `checkpoint_access` IAM policy is documented as attachable to the CodeAPI task role, but in that deployment path every checkpoint read/write fails with empty credentials. The runtime checkpoint client constructs `MinioClient` using only `MINIO_ACCESS_KEY`/`MINIO_SECRET_KEY` (defaulting both to empty strings) and never loads IAM role credentials the way `file-server.ts` does, so when the policy is attached to the task role instead of minting static keys for the optional IAM user, all `put`/`get` calls authenticate with empty strings and session checkpoint/restore is broken. Either add IRSA/task-role credential loading to the checkpoint client or update the documentation to state that only the static-key user path is supported.

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e2da55797

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bind the session before restoring checkpoints

When a checkpointed session is relaunched, the backend calls /api/v2/session/restore before the first /execute, and this hookless MicroVM path does not run a /run hook to bind the workspace. At this check getBoundSessionWorkspace() is therefore undefined, so every real restore returns 409 and the control plane silently continues with an empty workspace; any state that only existed in the checkpoint is lost after VM expiry/rollover. Bind the workspace from the restore request/runtime session id before rejecting it.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in daa43b0. Real regression from the hookless pivot — /session/restore runs before the first /execute, so nothing had bound the workspace and every restore 409'd (state lost across expiry). The checkpoint/restore routes now bind from X-Runtime-Session-Id (parseSessionBindingFromHeader → bindSessionWorkspace) before the handler, and the backend's pullCheckpoint/pushRestore send that header. Added header assertions to the checkpoint + restore tests.

[chatgpt-codex-connector]

Drop stale session VM records on reuse failure

When a session VM is auto-terminated by idlePolicy after being suspended, or otherwise evicted before the hard deadline, proxyExecute fails here but the finally only releases the lock. The registry record still says RUNNING, so subsequent requests keep reusing the same dead MicroVM and never reach the relaunch/restore path until the deadline/TTL expires. Remove or mark the record on reuse failure so the next call can launch a replacement.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in daa43b0. New executeOnSessionVm helper: on a reuse failure where the runner is unreachable (health/connection error — e.g. idlePolicy auto-terminated the suspended VM), it terminates the VM and removeRuntimeSession, so the next call relaunches + restores instead of reusing a dead VM still recorded RUNNING. A live-but-non-200 response leaves the warm VM and its record intact.

[chatgpt-codex-connector]

Mark session outputs surfaced only after upload

In session mode this marks a generated file as already surfaced during directory scanning, before /execute uploads it and before the route prunes failed uploads from the response. If a transient file-server or egress failure drops the upload, the next turn sees the unchanged file in the persistent workspace and suppresses it as already surfaced, so the user never receives an addressable file unless it changes. Move this bookkeeping to after successful upload/response emission.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in daa43b0. The surfaced-mark is now recorded per-file (id → name+signature) during the scan and committed to session.markSurfaced only for files that upload successfully, in uploadGeneratedFiles. A dropped upload leaves the file eligible to surface next turn.

[chatgpt-codex-connector]

Pass runner file-egress env into MicroVM images

The documented helper creates/updates the MicroVM image with only SANDBOX_USE_CGROUPV2, and the backend's RunMicrovm request does not inject runner environment later. Images built this way keep the sandbox defaults with FILE_SERVER_URL/EGRESS_GATEWAY_URL empty, so any Lambda execution that needs to download input files or upload generated files builds invalid /sessions/... URLs inside the runner. Include the file/egress and manifest environment in the image config, or pass it at launch, instead of setting only this key.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in daa43b0. create-microvm-image now takes --env-json / MICROVM_IMAGE_ENV_JSON, merged over SANDBOX_USE_CGROUPV2, so operators bake FILE_SERVER_URL / EGRESS_GATEWAY_URL / manifest env into the image at build time. README updated with the required keys and the "invalid /sessions/... URLs" rationale.

[chatgpt-codex-connector]

Keep the session locked after aborted executes

If ctx.signal aborts while a session /execute is in flight, Axios rejects and this finally releases the runtime-session mutex even though the sandbox runner can keep NsJail running after the response closes (the runner route explicitly waits to clean up until the child exits). A follow-up request can then acquire the same session and mutate the persistent workspace concurrently with the timed-out run; terminate/quarantine the VM or otherwise wait for runner completion before releasing this lock on abort.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in daa43b0 (same executeOnSessionVm helper). On ctx.signal abort we terminate the session VM before releasing the lock, so a follow-up request can't acquire the session and mutate the workspace while the aborted run's NsJail child is still exiting after the socket closed.

[chatgpt-codex-connector]

Don't include checkpoint time in the client job timeout

This awaits the full post-run checkpoint before completing the BullMQ job, but /exec waits only JOB_TIMEOUT for the job result while the checkpoint can consume CODEAPI_CHECKPOINT_TIMEOUT_MS (and the S3 putObject has no matching timeout). For executions that finish near the normal timeout or produce a large workspace, user code can succeed and still time out at the API because the worker is checkpointing; complete the job before best-effort checkpointing or include that budget in the public wait timeout.

Useful? React with 👍 / 👎.

[danny-avila]

Deferred — valid, but the fix is a design choice. checkpointUnderLock is already best-effort (never throws), so a checkpoint failure doesn't lose the result, but its latency still counts against JOB_TIMEOUT. Decoupling it (complete the job then checkpoint, or include the checkpoint budget in the public wait) needs a call on where checkpointing sits relative to job completion — raised with the maintainer. Not a correctness issue.

[chatgpt-codex-connector]

Enforce checkpoint size before buffering S3 objects

restoreSession checks data.length against CODEAPI_CHECKPOINT_MAX_BYTES, but this implementation has already read the entire S3 object into memory. If a checkpoint was written under a larger prior limit or otherwise appears in the shared/versioned bucket, relaunching that session can allocate an arbitrarily large buffer and OOM the worker before the intended guard runs; stream with a byte cap or inspect object size before Buffer.concat.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in daa43b0. CheckpointStore.get now takes maxBytes and stats the S3 object size before downloading (throws CheckpointTooLargeError if over), so we never Buffer.concat an oversized object. Removed the now-redundant post-buffer guard in restoreSession; added a MemoryCheckpointStore oversize test.

[danny-avila]

@codex review

[danny-avila]

Addressed the review of daa43b0 in 59f6fae.

Fixed:

• lambda-microvm.ts executeOnSessionVm (High) — good catch, real regression: axios throws an AxiosError with .response on a non-2xx, so the message === 'Error from sandbox' check tore down a healthy VM on a runner-level error. Now keeps the VM when the runner responded (isAxiosError && error.response), terminates only on no-response (connection/timeout) or abort. Added a regression test.
• lambda-microvm.ts reuse guard — now compares image_arn / image_version / port against config; a mismatch relaunches on the current config instead of running the old image / health-checking the wrong port.
• checkpoint-store.ts — cap accumulated bytes during the download loop too (not just statObject), so a concurrent grow can't OOM.
• v2.ts checkpoint/restore — fail closed (409) on a missing/malformed X-Runtime-Session-Id instead of acting on a stale bound session, and .catch(next) so a rejected handler promise reaches Express 4's error handler.

Not taking — job.ts:1553 size+mtime signature: valid in theory, but size+mtime is the standard cheap change-detection heuristic (rsync defaults to it). Content-hashing every output file on every workspace scan defeats the point of the diffing optimization, and the trigger is an adversarial edge (rewriting a file to the exact same size and mtime, e.g. touch -r). Leaving it as a documented limitation rather than paying the per-scan hashing cost; happy to revisit if it bites in practice.

[danny-avila]

@codex review

[macroscopeapp]

🟡 Medium sandbox-backend/lambda-microvm.ts:280

When findOrLaunchSession relaunches because the existing record no longer matches (image, port, or deadline), it overwrites the registry with the new VM but never terminates the old record.microvm_id. The previous VM is orphaned and keeps running — and billing — until AWS's idle policy eventually reaps it. Deploys or deadline rollovers can temporarily double the fleet and exhaust MicroVM quota. Consider terminating record.microvm_id before or after launching the replacement.

+    if (reusable && record && record.microvm_id) {
+      const oldMicrovmId = record.microvm_id;
       const launchedAt = Date.now();
       const vm = await this.launch(client, ctx, {
         clientToken: `sess-${runtimeSessionId}-${generation}`,
         idlePolicy: {
           maxIdleSeconds: this.config.idleSeconds,
           suspendedSeconds: this.config.suspendedSeconds,
           autoResume: true,
         },
         maxDurationSeconds: this.config.maxDurationSeconds,
       });
+      if (oldMicrovmId && oldMicrovmId !== vm.microvmId) {
+        await this.terminate(client, oldMicrovmId, 'rollover').catch(() => {});
+      }

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @service/src/sandbox-backend/lambda-microvm.ts around lines 280-289:

When `findOrLaunchSession` relaunches because the existing record no longer matches (image, port, or deadline), it overwrites the registry with the new VM but never terminates the old `record.microvm_id`. The previous VM is orphaned and keeps running — and billing — until AWS's idle policy eventually reaps it. Deploys or deadline rollovers can temporarily double the fleet and exhaust MicroVM quota. Consider terminating `record.microvm_id` before or after launching the replacement.

[macroscopeapp]

🟡 Medium sandbox-backend/lambda-microvm.ts:233

When removeRuntimeSession fails (e.g. Redis transiently unavailable) during cleanup after a timeout or unreachable VM, the stale RUNNING record is left in place. Every later request for that session then keeps reusing the dead microvm_id/endpoint and fails again until the record expires — a one-off cleanup error wedges the session. The .catch(() => {}) swallows the Redis failure so the caller never sees it. Consider retrying the removal, logging the failure, or marking the record as non-reusable so the next request relaunches instead of reusing a dead VM.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @service/src/sandbox-backend/lambda-microvm.ts around line 233:

When `removeRuntimeSession` fails (e.g. Redis transiently unavailable) during cleanup after a timeout or unreachable VM, the stale `RUNNING` record is left in place. Every later request for that session then keeps reusing the dead `microvm_id`/`endpoint` and fails again until the record expires — a one-off cleanup error wedges the session. The `.catch(() => {})` swallows the Redis failure so the caller never sees it. Consider retrying the removal, logging the failure, or marking the record as non-reusable so the next request relaunches instead of reusing a dead VM.

[macroscopeapp]

🟡 Medium sandbox-backend/lambda-microvm.ts:211

When a reused session VM is unreachable (e.g. AWS auto-terminated it while the record still says RUNNING), executeOnSessionVm terminates the VM, drops the registry record, and rethrows — surfacing MICROVM_UNHEALTHY/connection failure to the user instead of recovering on this call. The backend has the lock, the checkpoint store, and the session id in hand at that point, so it could relaunch + restore a replacement VM and retry the execute. Instead the first request after an auto-termination fails and only the next retry succeeds. Consider catching the unreachable-VM case inside the session execute path and falling through to findOrLaunchSession + retry before giving up.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @service/src/sandbox-backend/lambda-microvm.ts around line 211:

When a reused session VM is unreachable (e.g. AWS auto-terminated it while the record still says `RUNNING`), `executeOnSessionVm` terminates the VM, drops the registry record, and rethrows — surfacing `MICROVM_UNHEALTHY`/connection failure to the user instead of recovering on this call. The backend has the lock, the checkpoint store, and the session id in hand at that point, so it could relaunch + restore a replacement VM and retry the execute. Instead the first request after an auto-termination fails and only the next retry succeeds. Consider catching the unreachable-VM case inside the session execute path and falling through to `findOrLaunchSession` + retry before giving up.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: daa43b06e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Budget the session lock for restore and launch wait

When a session has to launch or relaunch, the critical section can spend up to launchTimeoutMs waiting in acquireOpBudget, another launchTimeoutMs polling in waitUntilRunning, and, for checkpointed sessions, CHECKPOINT_TIMEOUT_MS restoring before the execute even starts. This TTL only budgets one launch timeout and one checkpoint timeout, so under allowed slow launch/restore conditions the Redis lock can expire while the first worker is still executing or checkpointing, letting a second worker acquire the same session and send concurrent /execute requests to the same persistent workspace.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in e0dd912. RUNTIME_SESSION_LOCK_TTL_MS now budgets 2× LAUNCH_TIMEOUT_MS (the throttle wait in acquireOpBudget + the poll to RUNNING) and 2× CHECKPOINT_TIMEOUT_MS (restore-before-exec + post-exec checkpoint), plus health + JOB_TIMEOUT + headroom, so a slow relaunch+restore can't expire the lock mid-critical-section.

[chatgpt-codex-connector]

Preserve sandbox HTTP responses before recycling sessions

For a live session VM that returns an intentional non-2xx sandbox response, Axios rejects here before the later response.status !== 200 check can throw Error from sandbox; executeOnSessionVm therefore treats the AxiosError as a dead VM and removes the runtime-session record. This means user/request validation failures from /api/v2/execute unnecessarily terminate the warm session and can drop state when checkpoints are disabled; configure validateStatus or classify Axios responses so only transport/health failures recycle the VM.

Useful? React with 👍 / 👎.

[danny-avila]

Already fixed in 59f6fae (before this review's commit), and refined further in e0dd912: executeOnSessionVm now terminates only on abort or a transport-level failure (axios.isAxiosError(error) && error.response == null). A non-2xx sandbox response (AxiosError with .response) keeps the warm VM. Regression test added.

[chatgpt-codex-connector]

Re-download read-only inputs before reusing workspace copies

With session workspaces, this reuse path also applies to read_only inputs such as skill files. A sandbox process can unlink a root-owned 0444 file because the parent directory is writable, replace it, and the next turn will skip the download, hash the replacement as the new baseline, and never reapply the file-server's X-Read-Only protection, so the tampered infrastructure file persists across turns.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in e0dd912. A read_only prime is now recorded as such and primedInputId reports it as not-primed, so the file always re-downloads — restoring pristine content and re-applying the file-server's 0444/root protection that an unlink+replace via the writable parent dir would have stripped. Test added.

[chatgpt-codex-connector]

Hash session outputs before suppressing uploads

In a stateful session, user code can rewrite a previously surfaced output with different bytes while preserving the same size and mtime, for example by calling os.utime after writing. This size/mtime signature then suppresses the file and the client keeps the old file id/content, so non-input outputs in persistent workspaces need a content hash or other non-forgeable change detector before skipping upload.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in e0dd912 — reversing my earlier push-back now that both you and Macroscope flagged it. Output-diffing suppresses a surfaced output only when its content hash matches, not size+mtime (spoofable via os.utime / touch -r). The hash is computed once per session/input file and reused for input-modification detection, so it's a single hash per file rather than a separate pass.

[chatgpt-codex-connector]

Skip optional checkpoints when the job budget is exhausted

With checkpoints enabled, a successful /execute result is not returned to the worker until this post-run checkpoint completes. If the sandbox execution consumes most of JOB_TIMEOUT, the optional cache write can still take up to CHECKPOINT_TIMEOUT_MS, causing the router's job.waitUntilFinished(..., env.JOB_TIMEOUT) to time out a run that already succeeded; skip or background the checkpoint when there is no remaining job budget.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in e0dd912. The post-run checkpoint is skipped (liveness-only record update) when the remaining JOB_TIMEOUT budget can't fit a full CHECKPOINT_TIMEOUT_MS, so a run that already succeeded isn't timed out at the router by checkpoint latency. The next relaunch restores the prior checkpoint (one exec staler).

[chatgpt-codex-connector]

Relaunch expired idle sessions before proxying

When a session is idle longer than idleSeconds + suspendedSeconds (35 minutes with the defaults), AWS can terminate the suspended VM while this registry record still looks RUNNING until the 8-hour hard deadline. This branch reuses the stale endpoint based only on hard_deadline_at; the health check then fails, removes the record, and returns a 503, so the first request after idle expiry fails instead of relaunching and restoring the checkpoint.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in e0dd912. The reuse guard now treats a record whose last_seen_at is older than idleSeconds + suspendedSeconds as non-reusable, so the first request after idle-expiry relaunches + restores instead of reusing the stale RUNNING endpoint, failing the health check, and 503ing. Test added (backdated last_seen → second execute relaunches).

[chatgpt-codex-connector]

Preserve sessions on auth-token throttles

If CreateMicrovmAuthToken is throttled or has a transient control-plane failure for a warm session, this await throws before any sandbox request is made; executeOnSessionVm then treats it as a non-sandbox failure and terminates/removes the session, losing warm state when checkpoints are off and forcing the current request to fail. Token creation should use the existing token budget/retry path and should not mark the VM dirty on transient token failures.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in e0dd912 via the same classification change: teardown now happens only on abort or a transport-level axios failure. A throttled/transient CreateMicrovmAuthToken is an SDK error (not an axios error), so it no longer marks the VM dirty — the warm session survives and the request just fails. (The token budget/retry path is a reasonable follow-up, but not recycling the VM is the key harm addressed.)

[chatgpt-codex-connector]

Pass the CloudWatch logging config when launching

The Terraform provisions a runtime log group and execution role, and the repo docs note that runtime stdout needs a RunMicrovm CloudWatch logging config, but this RunMicrovmCommand never sends logging. In deployments relying on LAMBDA_MICROVM_EXECUTION_ROLE_ARN, MicroVM stdout/stderr will not reach CloudWatch, leaving sandbox startup and checkpoint/restore failures opaque despite the role being configured.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in e0dd912. RunMicrovm now sends logging: { cloudWatch: { logGroup } } from a new LAMBDA_MICROVM_LOG_GROUP (threaded config → backend → client), so VM stdout/stderr reaches CloudWatch. Correct — the execution role alone wasn't enough; the runbook + README now note both are required and Terraform outputs runtime_log_group to set it.

[danny-avila]

@codex review

[macroscopeapp]

🟡 Medium src/session-workspace.ts:150

SessionWorkspace.reset() ignores the false return value from resetSessionWorkspace(). When the underlying fsp.rm() fails, the session directory is only quarantined and its contents are left on disk, but reset() still clears this.surfaced, this.primed, and this.lease, and releases the pinned UID. The next session reuses that identity slot, calls ensureSessionWorkspace(), hits the EEXIST path, and inherits the previous session's leftover files instead of a clean workspace. Consider checking the boolean result and throwing (or skipping state cleanup) when resetSessionWorkspace() returns false.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @api/src/session-workspace.ts around line 150:

`SessionWorkspace.reset()` ignores the `false` return value from `resetSessionWorkspace()`. When the underlying `fsp.rm()` fails, the session directory is only quarantined and its contents are left on disk, but `reset()` still clears `this.surfaced`, `this.primed`, and `this.lease`, and releases the pinned UID. The next session reuses that identity slot, calls `ensureSessionWorkspace()`, hits the `EEXIST` path, and inherits the previous session's leftover files instead of a clean workspace. Consider checking the boolean result and throwing (or skipping state cleanup) when `resetSessionWorkspace()` returns `false`.

[macroscopeapp]

🟠 High runtime-session/lambda-client-aws.ts:131

resumeMicrovm calls toDescription() on the ResumeMicrovmCommand response, but the API returns an empty object — no microvmId field. toDescription() throws 'Lambda MicroVM response omitted microvmId' whenever microvmId is missing, so every successful resume against the real AWS API throws an error even though the MicroVM was resumed. Consider returning void (matching the interface) or fetching the VM via getMicrovm after a successful resume.

  async resumeMicrovm(microvmId: string): Promise<MicrovmDescription> {
-    const response = await this.send<Parameters<typeof toDescription>[0]>(
-      'ResumeMicrovm',
-      new ResumeMicrovmCommand({ microvmIdentifier: microvmId }),
-    );
-    return toDescription(response);
+    await this.send('ResumeMicrovm', new ResumeMicrovmCommand({ microvmIdentifier: microvmId }));
+    return this.getMicrovm(microvmId);

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @service/src/runtime-session/lambda-client-aws.ts around lines 131-137:

`resumeMicrovm` calls `toDescription()` on the `ResumeMicrovmCommand` response, but the API returns an empty object — no `microvmId` field. `toDescription()` throws `'Lambda MicroVM response omitted microvmId'` whenever `microvmId` is missing, so every successful resume against the real AWS API throws an error even though the MicroVM was resumed. Consider returning `void` (matching the interface) or fetching the VM via `getMicrovm` after a successful resume.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e0dd9121b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Drop stale session records on health failures

When a warm session VM fails the pre-execute health check, assertHealthy wraps the Axios connection/timeout/non-200 error in a SandboxBackendError. This predicate only recognizes a top-level AxiosError, so transportFailure is false and the RUNNING registry record is left intact; subsequent executions for that runtime_session_id keep reusing the same dead endpoint until idle/hard deadline instead of relaunching/restoring.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 06bbb17. assertHealthy now wraps a failed pre-execute health check as SandboxBackendError('MICROVM_UNHEALTHY'), and executeOnSessionVm classifies that alongside transport failures: on either it terminates the VM ('error') and calls removeRuntimeSession(runtimeSessionId, lockToken) so the next call relaunches a fresh VM and re-primes from file refs instead of pinning a dead microvm_id. Covered by a new health-fail teardown assertion in lambda-microvm.test.ts.

[chatgpt-codex-connector]

Restore checkpoints even when Redis records are gone

This gate skips restore whenever readRuntimeSessionRecord returns null, but checkpoints are stored under a deterministic runtimeSessionId key and can outlive or repair a lost/expired Redis record. In that case a relaunch with the same session ID starts with an empty workspace despite an S3 checkpoint being present; restoreSession already treats missing objects as absent, so gating on record != null drops recoverable state.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 06bbb17. The restore gate is now if (this.checkpointStore && this.checkpointsActive()) — the record != null precondition is gone. So a relaunch onto a fresh VM restores from the last checkpoint even when the Redis record has aged out (TTL) or was dropped by a prior teardown. restoreSession still degrades to a fresh workspace on absent/failed.

[chatgpt-codex-connector]

Bound object-store checkpoint calls by the timeout

When S3/MinIO stalls during the optional post-exec checkpoint, this store.put is not covered by CHECKPOINT_TIMEOUT_MS (that timeout only applies to the HTTP pull from the VM). A slow object-store write can therefore hold the session lock and delay a successful execution past JOB_TIMEOUT; if it outlives the lock TTL, the stale writer can also overwrite the deterministic checkpoint key after a newer execution has checkpointed.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 06bbb17. The store.put is now wrapped in withTimeout(store.put(...), config.timeoutMs, 'checkpoint store.put'), so a stalled S3/MinIO write rejects at the checkpoint timeout instead of holding the session lock past JOB_TIMEOUT (and past the lock TTL, where it could clobber a newer checkpoint). The underlying put isn't cancellable, but the caller stops waiting.

[chatgpt-codex-connector]

Clean partial restores before continuing

If a checkpoint object is corrupt or the restore upload is cut off after tar has extracted some members, this catch returns 500 but leaves those partially restored files in /mnt/data/session. The control plane treats restore failures as non-fatal and proceeds with the execution as a fresh workspace, so that request can run against a mix of stale checkpoint files instead of an empty workspace.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 06bbb17. The restore catch now rm -rfs and recreates the workspace dir before returning, so a corrupt archive or a cut-off upload leaves an empty workspace rather than a half-extracted tar. The control plane already treats restore failure as non-fatal and runs the job anyway, so a clean slate is correct ('a relaunched VM must be correct').

[chatgpt-codex-connector]

Throttle auth-token creation across workers

Every health check and execute mints a MicroVM auth token, but this path bypasses the Redis control-plane budget used for RunMicrovm; under many concurrent warm-session executions the fleet can exceed the CreateMicrovmAuthToken TPS limit and surface unclassified LambdaMicrovmApiError failures to users. Reserve a shared token budget before this call, just as launch does for run, so concurrency is queued instead of burst-throttled by AWS.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 06bbb17. Added a private mintAuthToken that reserves a shared per-second token budget via acquireOpBudget('token', { limitPerSecond: tokenTps, budgetMs: launchTimeoutMs }) before createMicrovmAuthToken, throttling to LAMBDA_MICROVM_TOKEN_TPS (default 8) and emitting microvm_throttle_events{op=token} — mirroring launch's run budget. proxyExecute mints once and reuses that token for both the health check and the execute, so concurrent warm-session executes queue instead of bursting past the AWS CreateMicrovmAuthToken limit. The checkpoint-path mints stay direct since they're serialized under the per-session lock and can't burst.

[danny-avila]

@codex review

[macroscopeapp]

🟠 High sandbox-backend/lambda-microvm.ts:405

When client.createMicrovmAuthToken(...) throws a LambdaMicrovmApiError with kind throttled, mintAuthToken rethrows it raw instead of mapping it to SandboxBackendError('MICROVM_LAUNCH_THROTTLED') and calling poisonOpBucket('token'). The raw error escapes the backend, so the worker can't classify it and other workers keep hammering CreateMicrovmAuthToken without backoff. Wrap the call the same way launch handles runMicrovm.

-    return client.createMicrovmAuthToken({
-      microvmId,
-      port: this.config.port,
-      ttlSeconds: this.config.authTokenTtlSeconds,
-    });
+    try {
+      return await client.createMicrovmAuthToken({
+        microvmId,
+        port: this.config.port,
+        ttlSeconds: this.config.authTokenTtlSeconds,
+      });
+    } catch (error) {
+      if (error instanceof LambdaMicrovmApiError && error.kind === 'throttled') {
+        await poisonOpBucket('token');
+        microvmThrottleEvents.inc({ op: 'token' });
+        throw new SandboxBackendError('MICROVM_LAUNCH_THROTTLED', error.message, error);
+      }
+      throw error;
+    }

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @service/src/sandbox-backend/lambda-microvm.ts around lines 405-409:

When `client.createMicrovmAuthToken(...)` throws a `LambdaMicrovmApiError` with kind `throttled`, `mintAuthToken` rethrows it raw instead of mapping it to `SandboxBackendError('MICROVM_LAUNCH_THROTTLED')` and calling `poisonOpBucket('token')`. The raw error escapes the backend, so the worker can't classify it and other workers keep hammering `CreateMicrovmAuthToken` without backoff. Wrap the call the same way `launch` handles `runMicrovm`.

[danny-avila]

Fixed in 78673a2 (also flagged by Codex). mintAuthToken now wraps createMicrovmAuthToken: throttled -> poisonOpBucket('token') + microvm_throttle_events{op=token} + MICROVM_LAUNCH_THROTTLED, matching how launch handles runMicrovm. not_found -> MICROVM_UNHEALTHY (drop record + relaunch), other -> MICROVM_LAUNCH_FAILED.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06bbb17f5d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve primed inputs across checkpoint restore

On a relaunched MicroVM the /session/restore path only restores files; the new SessionWorkspace has an empty primed map. This branch therefore falls through to downloadAndWriteFile() for every file ref on the first post-restore execute. If a prior turn modified an input in place and the client keeps sending the original file ref, the restored modified file is overwritten with the original before user code runs, so checkpoint recovery loses state that a warm VM would have preserved. Persist/rebuild the primed metadata or avoid overwriting existing restored files for matching refs.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 78673a2. The checkpoint now carries a sidecar (SessionWorkspace.snapshotMeta() -> .codeapi-session-meta.json, written under the held lock and removed before any execute) that restoreSessionCheckpoint reads back via loadMeta(). So a relaunched VM rebuilds the primed (and surfaced) maps and reusePrimedInput keeps a restored in-place-modified input instead of re-downloading the original. Read-only flags round-trip too. Covered by a new snapshotMeta/loadMeta round-trip test.

[chatgpt-codex-connector]

Prevent timed-out checkpoint puts from winning later

withTimeout stops waiting for store.put, but the helper explicitly does not cancel the underlying object-store write. If this put stalls past the timeout/lock lifetime and later succeeds after a newer execution has checkpointed the same deterministic key, the older workspace can overwrite the newer checkpoint; restores then read by runtimeSessionId and can roll the session back to stale state. Use an abortable upload/versioned key or otherwise fence the object write itself.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 78673a2. Each checkpoint now writes a distinct, strictly-increasing object key (<prefix>/<runtimeSessionId>/<paddedSequence>.tar.gz, sequence from allocateCheckpointSequence INCR), and restore reads the highest sequence. A put that stalled past the timeout and lands late writes an OLDER key and can never overwrite the newer one. Restore lists the session prefix, so it still works with no Redis record (preserves the round-3 fix). Prune only removes strictly-older keys. New test asserts a late lower-sequence put does not win.

[chatgpt-codex-connector]

Terminate superseded session VMs before relaunch

When an existing RUNNING record is rejected because the image/version/port changed or the hard deadline is too close, execution falls through to launch a replacement and overwrites the registry record without terminating the previous microvm_id. During a rollout or max-duration rollover, each active session can leave its old VM running/suspended and consuming quota/cost until idle/max-duration expiry. Terminate the non-reusable recorded VM before replacing the record, except when you have positive evidence it is already gone.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 78673a2. findOrLaunchSession now terminates record.microvm_id (reason superseded) before overwriting the record and relaunching, except when likelyIdleTerminated (AWS already auto-terminated it) — positive evidence it's gone. New test: an image-version bump terminates the old VM before the replacement launches.

[chatgpt-codex-connector]

Handle auth-token API failures as backend failures

If CreateMicrovmAuthToken returns a LambdaMicrovmApiError here, it escapes raw: the worker only converts SandboxBackendError/Axios errors into public failures, and executeOnSessionVm does not remove the session record because this is neither an Axios transport error nor MICROVM_UNHEALTHY. For a reused session whose VM was evicted or manually terminated, token minting can keep failing with the registry still pointing at the dead VM, producing generic 500s until the hard deadline. Wrap token API failures and drop/relaunch on not-found the same way the health path does.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 78673a2. mintAuthToken now wraps createMicrovmAuthToken: not_found maps to SandboxBackendError('MICROVM_UNHEALTHY'), which executeOnSessionVm already treats as a teardown signal (terminate + removeRuntimeSession), so a reused session whose VM was evicted drops the stale record and the next call relaunches instead of 500ing to the hard deadline. throttled -> poison + MICROVM_LAUNCH_THROTTLED; other API errors -> MICROVM_LAUNCH_FAILED. New test covers the not_found teardown path.

[chatgpt-codex-connector]

Throttle checkpoint auth-token creation

The execute path budgets CreateMicrovmAuthToken calls via mintAuthToken, but checkpoint/restore mints tokens directly here and in pushRestore. With checkpointing enabled, every successful session execution and every relaunch can add unbudgeted token calls, so a burst of concurrent sessions can exceed the AWS token TPS limit even though normal executes are throttled; checkpoint failures are nonfatal, but restore failures start the replacement VM with a fresh workspace. Route these token mints through the same shared token budget.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 78673a2. pullCheckpoint/pushRestore no longer mint tokens directly — they take a mintToken callback, and the backend passes mintAuthToken, so checkpoint/restore mints go through the same shared token TPS budget as the execute path. A burst of concurrent sessions now queues instead of bypassing the cap.

[chatgpt-codex-connector]

Bound checkpoint fetch time during restore

Restore treats checkpoint recovery as nonfatal, but the S3/MinIO fetch is awaited without the checkpoint timeout or the worker abort signal. If statObject/getObject stalls while a session is relaunching, the worker holds the session lock before any execute can run, so the request can time out and subsequent strict calls see the session as busy while affinity calls fall back stateless. Wrap store.get with the same timeout policy used for store.put.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 78673a2. restoreSession now wraps store.get in withTimeout(config.timeoutMs), symmetric with the store.put bound, so a stalled S3/MinIO fetch during relaunch can't hold the session lock past the checkpoint timeout.

[chatgpt-codex-connector]

Prune active session zset entries

Every successful session execution adds its id to rtsx:active, but there is no production caller pruning this zset when the corresponding session record expires; forgetRuntimeSessionActive/listIdleRuntimeSessions are only exercised by tests. In a long-running deployment with many distinct session hints, these members accumulate indefinitely and countActiveRuntimeSessions/the active-session metric overcount dead sessions. Add a sweeper or expiry-backed cleanup for this zset.

Useful? React with 👍 / 👎.

[danny-avila]

Valid, deferring to the sweeper PR (plan PR 7 = registry hygiene + ZSET repair) rather than fixing here. removeRuntimeSession already ZREMs on every teardown (unhealthy/transport/abort); the residual gap is only records that TTL-expire without an explicit removal, and there's no cheap per-member TTL on a zset — it needs the periodic sweeper that forgetRuntimeSessionActive/listIdleRuntimeSessions were built for. Tracking it there so this PR's scope stays the session mechanics.

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 78673a2fa8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Refuse symlinked checkpoint sidecar paths

When a session checkpoint runs after sandboxed code has created .codeapi-session-meta.json as a symlink, this privileged writeFile follows the link and overwrites the target outside /tmp/sandbox/session before tar starts. In the Lambda runner image the API process runs as root, so a user can turn the automatic post-exec checkpoint into an arbitrary-file overwrite; create this sidecar with no-follow/exclusive regular-file semantics or keep the metadata outside the user-writable workspace.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 2f75eff. The sidecar is no longer written with a plain root writeFile: streamSessionCheckpoint now fsp.rm(metaPath, {force:true}) first (unlink never follows a symlink) then writeFile(..., {flag:'wx'}) (O_CREAT|O_EXCL, which won't follow an existing link), so a squatted .codeapi-session-meta.json symlink can't redirect the write outside the workspace. On the restore side applyRestoredMeta now lstats and only reads a regular file, so a symlinked sidecar in an untrusted archive is never followed either. No concurrent user code runs during checkpoint (session lock held), so there's no TOCTOU between the unlink and the write.

[chatgpt-codex-connector]

Route non-default MicroVM ports with the proxy header

When LAMBDA_MICROVM_PORT is set to anything other than 8080, these proxied requests still omit X-aws-proxy-port; AWS documents that MicroVM endpoint traffic defaults to port 8080 and requires that header to route to a different target port (https://docs.aws.amazon.com/lambda/latest/dg/microvms-launching.html). The token is minted for this.config.port, but health/execute/checkpoint traffic will be sent to 8080, so non-default-port images fail health checks or hit the wrong service.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 2f75eff. Added microvmPortHeaders(port) in lambda-client.ts — it emits X-aws-proxy-port: <port> only when the port isn't the 8080 default — and applied it to every proxied request: the health check and execute in proxyExecute, plus pullCheckpoint/pushRestore. So a non-default LAMBDA_MICROVM_PORT now routes to the minted port instead of silently hitting 8080. New test asserts the header is present for port 9090 and absent for 8080.

[chatgpt-codex-connector]

Reserve the whole checkpoint budget before starting it

When an execution finishes with only a little more than CHECKPOINT_TIMEOUT_MS left, this condition still starts a checkpoint even though checkpointUnderLock can spend a full GET timeout and a full object-store put timeout (plus token-budget wait) before returning. With the defaults, a successful run with ~90s remaining can still block for up to 120s+ in the optional checkpoint and cause waitUntilFinished(JOB_TIMEOUT) to time out after the sandbox work already succeeded; require enough budget for the complete checkpoint path or skip it.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 2f75eff. The skip guard now reserves the full checkpoint path, not one timeout: worstCaseCheckpointMs = launchTimeoutMs + 2 * checkpoint.timeoutMs (token-budget wait + the checkpoint GET + the object-store put), and only checkpoints when remainingBudgetMs exceeds that. So a run that finishes with barely more than a single timeout left now skips the optional checkpoint instead of risking a waitUntilFinished(JOB_TIMEOUT) timeout after the sandbox work already succeeded.

[chatgpt-codex-connector]

Don't trust mutated cached inputs as pristine

For writable inputs in a stateful session, this marks the storage id as reusable before knowing whether the job mutates the file. If that path is modified and a later turn supplies the same original file id again (for example after the modified output upload was pruned or the caller intentionally reuses the original ref), reusePrimedInput skips the download and records the mutated on-disk bytes as the baseline, so user code runs against stale modified content and the walker can echo the original ref as unchanged. Re-download after a writable input was modified, or retain the original hash and verify it before reusing.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 2f75eff. The primed map now retains the ORIGINAL upload hash (recorded at prime time, round-trips through the checkpoint sidecar), and reusePrimedInput baselines against session.primedHash(name) instead of re-hashing the on-disk copy. So a writable input mutated in place by a prior turn is detected as modified-from-original (the walker no longer echoes the original ref as unchanged) — without re-downloading, which would regress the round-4 fix that intentionally persists in-place mutations across a warm reuse / restore. The output-diff surfaced map still prevents a redundant re-upload when the mutated content was already surfaced.

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2f75eff7fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Avoid worker-only backend validation in API startup

In split deployments (api-server.ts without workers), the API process only authenticates/enqueues jobs and derives runtimeSessionId; it never creates the Lambda MicroVM backend or checkpoint store. Calling validateSandboxBackendPolicy() here makes an API pod using runtime sessions fail startup unless it also has worker-only settings such as LAMBDA_MICROVM_IMAGE_ARN and the MINIO_* checkpoint credentials, effectively forcing backend secrets/config into API-only pods. Keep Lambda backend validation in worker/combined startup, or split the policy so API-only startup validates only the routing pieces it actually uses.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 299a77b. Dropped validateSandboxBackendPolicy() from startupApiOnly — an API-only pod authenticates + enqueues and never constructs the Lambda backend or checkpoint store, so validating that policy (which after the round-4 MinIO fail-fast also requires the MINIO_* creds) forced worker-only config into API pods just to boot. startupWorkerOnly and the combined gracefulStartup still run it, so the backend config is validated where the backend actually lives.

[chatgpt-codex-connector]

Handle MicroVM poll failures as backend errors

If the follow-up GetMicrovm call is throttled or briefly returns a control-plane error while a VM is still PENDING, the LambdaMicrovmApiError from this line escapes unchanged. The worker only maps SandboxBackendError and Axios errors to public responses, so these launch failures become generic 500s instead of the intended MICROVM_LAUNCH_* 503/throttle handling. Wrap/classify polling errors the same way runMicrovm errors are handled.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 299a77b. launch's catch around waitUntilRunning no longer rethrows raw: a GetMicrovm poll that throws a LambdaMicrovmApiError is now mapped like runMicrovm — throttled -> poisonOpBucket('run') + MICROVM_LAUNCH_THROTTLED, anything else -> MICROVM_LAUNCH_FAILED — while waitUntilRunning's own SandboxBackendErrors pass through unchanged. So a transient control-plane error during the poll surfaces as a public 503/throttle instead of a generic 500.

[chatgpt-codex-connector]

Suppress previously primed session inputs

In session mode, an input downloaded in an earlier turn remains in the persistent workspace; if a later request omits that file, it has no current inputFileHashes or inputByName entry, and this branch only suppresses paths marked as surfaced outputs. The unchanged old input then falls through to the generated-file upload path and is returned as a new output on unrelated executions. Use the session priming metadata to skip or classify unchanged previously primed inputs before treating them as generated files.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 299a77b. Added SessionWorkspace.isPrimedInput(relPath) and a skip branch in handleRegularFile: a file the session primed as an input on an earlier turn (which persists in the workspace) is no longer echoed as a newly generated output when a later turn omits it and it has no inputFileHashes/surfaced entry. It's treated as the input it is, not a fresh output. Covered by an isPrimedInput unit test.

[chatgpt-codex-connector]

Remove squatted checkpoint metadata directories

If sandboxed code creates a directory named .codeapi-session-meta.json in the persistent workspace, this non-recursive rm rejects with EISDIR, so every post-run checkpoint for that session fails before the archive is streamed. Because the directory persists across turns, workspace recovery after MicroVM expiry is disabled until the user removes that reserved path. Treat this sidecar name as reserved and remove directories at this path safely before writing the metadata file.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 299a77b. The round-5 no-follow rm was non-recursive, so a squatted .codeapi-session-meta.json DIRECTORY threw EISDIR and broke every subsequent checkpoint. It's now fsp.rm(metaPath, { force: true, recursive: true }) — recursive clears a directory, force ignores absence, and neither follows a symlink — then the exclusive wx write creates a fresh regular file. So a squatted directory (or symlink, or file) can no longer wedge checkpoints.

[chatgpt-codex-connector]

Keep checkpoint sequence ahead of retained objects

When a session is idle long enough for this Redis sequence key to expire but its checkpoint objects are still retained in S3/MinIO, the next checkpoint starts again at sequence 1. Because restore selects the lexicographically greatest object under the prefix, any older object with a higher previous sequence remains the “latest” checkpoint, so subsequent relaunches restore stale workspace state even after newer checkpoints are written. Either keep the sequence at least as long as checkpoint retention, seed it from the latest object, or include a non-resetting epoch in the key.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 299a77b by removing the resettable counter entirely. Checkpoint objects are now keyed by the wall-clock time the checkpoint was taken (<prefix>/<paddedTakenAtMs>.tar.gz), captured at checkpoint start, instead of a Redis ckptseq that carried the record TTL. A never-resetting timestamp always sorts above a retained older object after a long idle gap (so restore, which reads the lexicographically-greatest key, no longer picks stale state, and prune drops the old object); and a put that stalled and lands late still loses because it carries its earlier start time. Dropped allocateCheckpointSequence/rtsx:ckptseq. New checkpoint-store test asserts a later timestamp wins over a stale earlier one. (Caveat: cross-worker clock skew within one session could momentarily pick a slightly older checkpoint — a self-healing 'restore one exec staler' at worst, versus the current stale-forever bug.)

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 299a77b474

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use the documented build log path

When this Terraform role is used to create or update a MicroVM image, the Lambda builder writes build diagnostics to /aws/lambda/microvms/<image-name> per the AWS MicroVM image docs, but these permissions only match /aws/lambda-microvms/*. The logs:CreateLogGroup/CreateLogStream/PutLogEvents calls for the actual group will not match this policy, so image builds can fail or at least lose the logs needed to debug failures; the pre-created aws_cloudwatch_log_group.build path above should be updated to the same slash-separated path too.

Useful? React with 👍 / 👎.

[danny-avila]

Pushing back — the hyphen path is correct and the AWS docs cited here are wrong. This is a documented spike gotcha, and I just re-confirmed it live: the MicroVM image builder writes to /aws/lambda-microvms/<name> (hyphen), and every build log group in the account is at that path, including the one from a build I ran today:

/aws/lambda-microvms/codeapi-session
/aws/lambda-microvms/codeapi-session-r4
/aws/lambda-microvms/codeapi-diag...  (x7)
/aws/lambda-microvms/codeapi-spike

There is NO /aws/lambda/microvms/ (slash) group in the account. Changing the policy + pre-created log group to the docs' slash path would stop matching the group the builder actually uses and lose the build logs. The main.tf comment already flags this discrepancy; leaving both the log group and the policy on the hyphen path.

[chatgpt-codex-connector]

Don't hide changed primed inputs

In session mode, when a file was primed as an input on an earlier turn but this request does not re-send that input ref, this unconditional skip suppresses the file even if the sandbox has since modified its bytes. This also defeats the new upload-failure retry path for modified inputs: if the modified upload is pruned from the response, the next turn has no inputFileInfo and the changed workspace file is silently dropped because it is still marked primed. Compare the current hash to the stored primed hash (or clear primed state once a primed path is modified) before skipping.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 11bc26c. The round-6 skip is no longer unconditional: it suppresses a previously-primed input only when the current on-disk hash matches the stored primed baseline (session.primedHash), or the input is read-only (edits dropped by contract). A writable primed input the sandbox modified now falls through and surfaces as an output — which also restores the upload-retry path you flagged (a modified input whose earlier upload was pruned is re-surfaced next turn instead of silently dropped).

[chatgpt-codex-connector]

Re-prime session inputs without following old symlinks

This branch reuses a workspace that prior sandbox code can modify, so a previous turn can leave foo or a parent like dir as a symlink and a later request that supplies inline content or a downloaded file at that path will prime through the existing writeFile/rename paths as the root API process. In that same-session scenario the symlink is followed outside /tmp/sandbox/session, letting untrusted code clobber arbitrary files in the MicroVM on the next prime. Before writing into a persistent workspace, remove/recreate the destination and validate each ancestor with no-follow semantics, or use an O_NOFOLLOW temp+rename flow that rejects symlink parents.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 11bc26c (P1). secureAncestors now lstats each ancestor (no follow) and throws on a symlinked component, so priming can't chmod/traverse a symlink a prior turn planted. Both write paths also clear a squatted target first: inline writeFile does fsp.rm(filePath, {force:true,recursive:true}) before writing (unlink never follows a link; recursive clears a squatted dir), and the download path clears finalPath before the rename. So a same-session symlink at a file path or a parent dir can no longer redirect a root write outside /tmp/sandbox/session. Gated on session mode; a fresh per-job workspace has nothing to squat.

[chatgpt-codex-connector]

Don't time out normal auto-resumes as unhealthy

When a session VM is suspended, the first request to its endpoint is held while Lambda auto-resumes it, and that resume latency grows with the suspended state size. This health probe uses the short healthTimeoutMs budget (5s by default), so a valid but slow auto-resume is classified as MICROVM_UNHEALTHY; executeOnSessionVm then terminates the VM, drops the record, and returns a 503 instead of letting the execute request complete. Use a resume-aware timeout/budget here or skip the preflight health check for reusable suspended sessions.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 11bc26c. findOrLaunchSession now reports whether the VM was reused, and proxyExecute skips the preflight health check for reused session VMs. So a suspended VM auto-resumes on the execute itself under the full job budget instead of being killed when a large-state resume exceeds the 5s healthTimeoutMs. Dead-VM detection is unaffected: an evicted VM fails token minting with not_found (mapped to MICROVM_UNHEALTHY -> teardown), and a freshly-launched/relaunched VM still gets the readiness probe. New test asserts a reused VM issues no /health preflight.

[danny-avila]

@codex review

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61ba32e852

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve liveness after stored checkpoints

When checkpoints are enabled and checkpointSession returns stored, this returns the record re-read from Redis instead of merging in base. That record was written from the pre-execute record inside checkpointSession, so it drops the last_seen_at: now update; because findOrLaunchSession bases idle-expiry on record.last_seen_at, an actively used session with successful checkpoints will look idle after idleSeconds + suspendedSeconds, causing an unnecessary relaunch and skipping termination of the still-live VM.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in a53ff5f. checkpointUnderLock now re-applies last_seen_at: now to the record it re-reads after a stored checkpoint (that record was built from the pre-execute snapshot, so it carried a stale timestamp). Liveness now advances on every checkpointed execute, so an actively-used session no longer drifts toward the idle-expiry threshold.

[chatgpt-codex-connector]

Budget the lock for all checkpoint I/O

The session lock is held through restore, execute, and post-run checkpointing, but this TTL only reserves 2 * CHECKPOINT_TIMEOUT_MS. On a relaunch with checkpoints enabled, restoreSession can spend one timeout in store.get and another in pushRestore, while checkpointSession can spend one in pullCheckpoint and another in store.put; under slow object store or runner responses near the configured timeouts, the Redis lock can expire while the first worker is still operating on the VM, allowing a second worker to acquire the same runtime session and run concurrently/fence the first holder.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in a53ff5f. Bumped the lock TTL from 2 * CHECKPOINT_TIMEOUT_MS to 4 *. As you note, a relaunch-with-checkpoint call holds the lock through restore (store.get + pushRestore = 2 timeout-bounded I/Os) and the post-run checkpoint (pullCheckpoint + store.put = 2 more), so 4x is the correct reservation; the comment now spells out both pairs.

[chatgpt-codex-connector]

Include connector drift in session reuse

When LAMBDA_MICROVM_EGRESS_CONNECTOR_ARNS or ingress connector config changes, this still treats existing records as reusable because the match only includes image/version/port. Those connector ARNs are only applied in RunMicrovm when a VM is launched, so hardened deployments that tighten egress can keep reusing already-running sessions with the previous, broader network policy until idle/deadline expiry.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in a53ff5f (P1). The record now stores a connectors fingerprint (order-independent JSON of the ingress+egress ARNs) written at launch, and configMatches compares it. So changing LAMBDA_MICROVM_EGRESS_CONNECTOR_ARNS (or ingress) makes existing sessions non-reusable — they relaunch onto the new network policy instead of serving on the old broader one until idle/deadline. New test: a tightened egress config forces a relaunch + terminates the old VM.

[chatgpt-codex-connector]

Wait for runner health before restoring

On a fresh relaunch with an existing checkpoint, restore is attempted immediately after the control plane reports the VM as RUNNING, before the normal runner health check. If the endpoint is allocated but the API listener is not ready yet, pushRestore fails and restoreSession intentionally treats it as non-fatal, so the subsequent health-checked execute runs against an empty workspace and silently loses the checkpoint for that session turn.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in a53ff5f. Restore is now gated on runner readiness: findOrLaunchSession polls the health endpoint (bounded by the launch timeout) before calling restoreSession, so pushRestore no longer races a booting app listener and drops the checkpoint onto an empty workspace. If the runner never becomes ready, the VM is torn down + the record dropped so the next call relaunches rather than reusing a dead endpoint.

[chatgpt-codex-connector]

Treat an empty checkpoint bucket as unset

If deployments leave CODEAPI_CHECKPOINT_BUCKET defined as an empty string while relying on MINIO_BUCKET, startup validation accepts the config because MINIO_BUCKET is non-empty, but this nullish fallback still selects the empty string. The store then calls S3/MinIO with bucket === '', causing every checkpoint and restore operation to fail even though the documented fallback bucket is configured.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in a53ff5f. Bucket selection is now CODEAPI_CHECKPOINT_BUCKET || MINIO_BUCKET || 'test-bucket' (was ??), so an empty-string checkpoint bucket falls through to MINIO_BUCKET — matching what the startup fail-fast accepts — instead of selecting '' and failing every S3 call.

[chatgpt-codex-connector]

Avoid deleting user files for checkpoint metadata

In session mode, if user code creates a legitimate workspace file named .codeapi-session-meta.json, every checkpoint removes it and replaces it with the internal sidecar before deleting the sidecar after tar finishes. That makes the checkpoint path corrupt user-visible workspace state for this filename and the file will be missing on the next execution/restore.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in a53ff5f. The sidecar path is now collision-safe: on checkpoint, if a regular user file already holds .codeapi-session-meta.json we log and skip metadata persistence for that turn (the user's file tars as normal workspace content) rather than deleting it; a symlink/dir squat is still removed (the round-6/7 security fix). On restore, applyRestoredMeta only deletes the sidecar when it parses as our own metadata, so a user file sharing the name is left intact.

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a53ff5f5f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not reuse a VM after a failed restore

When a checkpointed session relaunches and POST /session/restore times out or fails after the runner has begun handling the archive (for example a large/slow or corrupt checkpoint), restoreSession returns 'failed' but this result is ignored here and the same VM is immediately used for /execute. The runner-side restore handler can still be extracting, chowning, or wiping the session workspace after the client aborts, so the next execution can run against a partially restored workspace instead of a clean or consistent one; treat a failed push restore as a VM recycle/remove-record condition or otherwise wait for cleanup before executing.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 7871287. restoreSession now returns fetch_failed (store.get failed before the runner was touched — the fresh workspace is clean, safe to execute) vs push_failed (pushRestore failed after the runner may have begun extracting/wiping). On push_failed, findOrLaunchSession terminates the VM + drops the record and throws MICROVM_UNHEALTHY, so the next call relaunches a clean VM instead of executing against a possibly-partial workspace.

[chatgpt-codex-connector]

Include token mint waits in the session lock TTL

In lambda-microvm session mode under CreateMicrovmAuthToken throttling, each mintAuthToken call can wait up to LAMBDA_MICROVM_LAUNCH_TIMEOUT_MS while the runtime-session lock is held; a relaunch with readiness probing, restore, execute, and post-checkpoint can mint several tokens, but this TTL only budgets two launch-timeout waits total. With default timeouts and a full-length execution after worst-case token waits, the lock can expire before the first worker finishes, letting another worker acquire the same runtime session and mutate the workspace concurrently; include the token waits here or renew the lock while executing.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 7871287 (P1). Rather than keep growing the TTL sum (this is the third time it's been undersized — token-mint throttle waits push the true worst case to ~7x launch + 4x checkpoint, which is unworkable), the lock now RENEWS on a heartbeat: executeSession runs a fenced renewRuntimeSessionLock every TTL/3 while it holds the lock, so an arbitrarily long critical path can't outlive the TTL and let a second worker fence it. A fenced renew (another worker owns the lock) stops itself. The TTL is now just a comfortable base.

[chatgpt-codex-connector]

Do not trust user files as checkpoint metadata

If user code creates a root-level .codeapi-session-meta.json containing JSON with primed and surfaced arrays, checkpointing deliberately preserves it as a user file, but restore treats the same shape as internal metadata, loads it into the session, and deletes it. After VM expiry this can silently remove a legitimate file and poison the restored priming/output-diff maps, causing later files to be suppressed or treated as inherited; add an authenticity marker or store metadata outside the user workspace before calling loadMeta.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 7871287. The sidecar now carries an authenticity marker (SESSION_META_MARKER), and applyRestoredMeta only calls loadMeta + deletes when parsed.marker matches. So a user file that shares the reserved name is never loaded as metadata (no priming-map poisoning) nor deleted, even if it happens to contain primed/surfaced arrays.

[chatgpt-codex-connector]

Mark inline source files as session inputs

In session mode this branch keeps the workspace between jobs, but inline source files written by writeFile() are only tracked in the current job's inputFileHashes, not in SessionWorkspace state. If a session switches entrypoint names, such as Python main.py followed by Bash script.sh, the old main.py is no longer an input for the second job and is uploaded as a generated output even though it was just prior request source; persist/suppress inline inputs across session turns like downloaded inputs.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 7871287. writeFile now calls session.markPrimed(name, ..., hash) for inline source files (previously only downloaded inputs were tracked in SessionWorkspace). So after switching entrypoint (e.g. main.py -> script.sh) the prior turn's source is recognized as a primed input and suppressed from the output scan (via the round-6/7 isPrimedInput path with the round-7 hash-compare) instead of being echoed as a generated file.

[chatgpt-codex-connector]

Check symlink ancestors before recursive mkdir

For persistent sessions, this lstat check runs only after the priming paths have already called fsp.mkdir(parent, { recursive: true }). If a previous turn leaves a as a symlink and the next request primes a/b/file.py, the root runner follows the symlink during mkdir -p and creates directories outside the workspace before this check throws; validate each existing ancestor with lstat before creating descendants, or create directories one component at a time without following symlinks.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 7871287. Added ensureDirNoFollow which builds the workspace path one component at a time, lstat-checking each existing component and refusing to descend through a symlink, and use it in place of fsp.mkdir(parent, {recursive:true}) for session-mode primes (both the inline and download paths). So mkdir -p can no longer follow a prior turn's symlinked ancestor and create directories outside the workspace before the check ran. Fresh per-job workspaces keep plain mkdir -p.

[chatgpt-codex-connector]

Recycle sessions after auto-resume 502s

When a reused session VM is suspended and Lambda cannot auto-resume it, AWS returns a 502 to the proxied request, which Axios exposes with error.response; this classification treats that as an ordinary sandbox response instead of an unreachable/dirty VM. In that case the stale RUNNING registry record is kept, so every later request for the same session keeps reusing the VM and failing until idle expiry rather than relaunching from the checkpoint; treat MicroVM proxy 502/5xx on reused sessions as a recycle/remove-record condition.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 7871287. executeOnSessionVm now treats an Axios error with response status 502/503/504 on a REUSED VM as unreachable (the AWS proxy couldn't reach/resume the VM) and recycles it (terminate + remove record), distinct from a runner 500 (kept — the VM is alive, the request failed). New test asserts a reused VM's 502 is recycled while the existing round-3 test still keeps the VM on a 500.

[chatgpt-codex-connector]

Use a monotonic checkpoint sequence

Checkpoint ordering now depends on each worker's local Date.now(), but runtime-session locks can move between pods whose clocks are skewed or step backwards. If a later execution checkpoints on a worker with an earlier clock, it writes a lexicographically smaller key, latestKey() restores the older checkpoint after a relaunch, and the newer workspace state is lost; use a Redis-side monotonic sequence/time source or include the session generation instead of local wall-clock time.

Useful? React with 👍 / 👎.

[danny-avila]

Fixed in 7871287. Reverted the wall-clock key to a store-seeded monotonic sequence: allocateCheckpointSequence is a pure Redis INCR (no wall clock -> no cross-pod skew), and on a cold/TTL-reset counter (INCR==1) checkpointSession seeds it above store.latestSequence() so it can't fall below retained objects (the round-6 reset bug). Checkpoints serialize on the (now heartbeat-renewed) session lock, so the read-then-seed has no concurrent writer. This closes both the skew and the reset failure modes.

[chatgpt-codex-connector]

Prune inactive session ids from the active zset

In affinity/strict mode this adds every successful runtimeSessionId to rtsx:active, but the record TTL does not expire zset members and the only non-test removal path is explicit removeRuntimeSession() on errors. Deployments with many one-off session hints will accumulate stale members indefinitely after normal idle termination, inflating Redis memory and any active-session metrics; add a sweeper or remove entries when their session record is gone.

Useful? React with 👍 / 👎.

[danny-avila]

Valid, and it's the same accumulation as the deferred sweeper work (rtsx:active grows because record TTL doesn't expire zset members and only removeRuntimeSession prunes them). Tracking it with the sweeper PR (plan PR 7 = registry hygiene: drop zset members whose record is gone, plus the rtsx:ckptseq/rtsx:gen counters, which have the same idle-leak shape). Keeping it out of this PR's scope so the session mechanics stay reviewable; it's a periodic-cleanup job, not a change to the execute path.

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7871287687

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not trust user-restored session metadata

When sandbox code creates .codeapi-session-meta.json with the public SESSION_META_MARKER before a checkpoint, streamSessionCheckpoint treats it as a regular user file and archives it instead of writing the control-plane sidecar; on the next relaunch this branch trusts that user-controlled file and calls session.loadMeta(parsed). That lets a restored session forge primed entries, including marking read-only/infrastructure file refs as non-read-only and already present, so a later execute can skip re-downloading a trusted read-only file and run the attacker-modified workspace copy. The restore path needs an unforgeable sidecar or must avoid loading metadata that could have come from the workspace.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve read-only ownership after restore

When a checkpoint contains read-only inputs, this restore step chowns the entire workspace to the sandbox UID, undoing the applyReadOnlyInputPermissions protection that initially kept those files root-owned and only readable. If the same read-only file is not re-sent on the first post-restore execute, it remains in the persistent workspace owned by the sandbox user, so user code can chmod and mutate infrastructure/read-only content that was meant to stay immutable. Restore should reapply the read-only ownership from the restored metadata or avoid blanket-chowning those paths.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Poll fresh MicroVM readiness before failing

For a newly launched VM this performs exactly one health request bounded only by healthTimeoutMs (5s by default). The restore path below already accounts for the endpoint being available before the runner listener is ready by polling with waitForRunnerReady, but stateless executions and fresh session launches without checkpoints still take this single probe; if the app boots a little slower than the health timeout, the VM is marked MICROVM_UNHEALTHY and torn down even though it could have become ready within launchTimeoutMs. Fresh launches should use the same readiness polling before executing.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Record the resolved image version for reuse checks

When LAMBDA_MICROVM_IMAGE_VERSION is left unset (the documented “latest” mode), this stores undefined instead of the imageVersion returned for the launched VM. After publishing a new version under the same image ARN, old session records still compare undefined to undefined in the reuse check, so warm sessions keep running the previous image until idle or hard expiry and miss deploys/security fixes. Persist the resolved version or require pinning so unpinned latest updates cause a relaunch.

Useful? React with 👍 / 👎.