Skip to content

prime images push-bulk#772

Merged
kcoopermiller merged 4 commits into
mainfrom
feature/bulk-image-push
Jul 2, 2026
Merged

prime images push-bulk#772
kcoopermiller merged 4 commits into
mainfrom
feature/bulk-image-push

Conversation

@kcoopermiller

@kcoopermiller kcoopermiller commented Jul 1, 2026

Copy link
Copy Markdown
Member

Adds a CLI command for pushing many images at once, from either of two inputs:

  • JSONL manifest (--manifest builds.jsonl): one build per line — {"image": "name:tag", "context": "./dir", "dockerfile"?, "platform"?}, paths resolved relative to the manifest file.
  • Harbor tasks (--harbor <dir>): discovers task directories (task.toml + environment/) and uses each task's environment/ folder as the build context. Images are named <task-dir-name>:<tag>, overridable via --name-template with {dir}/{name} placeholders. Tasks with a prebuilt docker_image or no environment/Dockerfile are skipped with a warning.

Each build then goes through the existing single-push flow with a sliding window of at most --concurrency (default 64) builds in flight server-side. Build status is polled via GET /images/build/{build_id} every 10s; when a build reaches a terminal state its slot is immediately given to the next queued build, so one slow build never blocks the rest. A per-build --build-timeout (default 30 min) prevents a stuck build from wedging the queue.

Rate-limit 429s (the initiate/start endpoints allow 150 req/min) are retried with exponential backoff; wallet-quota 429s ("Image limit exceeded") abort submission immediately since retrying can't succeed — in-flight builds are still polled to completion.

Any build that doesn't complete is written to push-bulk-failures.jsonl (a valid manifest with absolute paths), so retrying is just prime images push-bulk --manifest push-bulk-failures.jsonl. Exit code is non-zero if anything failed


Note

Medium Risk
New client-side orchestration drives many authenticated image build/upload API calls with concurrency and quota edge cases; mistakes could spam the API or mis-handle limits, but it does not change core auth or server behavior.

Overview
Adds prime images push-bulk, which builds and pushes many images in one run from either a JSONL manifest (--manifest) or Harbor task trees (--harbor), with up-front validation before any API calls.

Bulk execution reuses the single-push flow (package context → initiate → upload → start), keeps a sliding window of in-flight builds (--concurrency, default 64), polls GET /images/build/{id} until terminal status or --build-timeout, retries rate-limit 429s with backoff, and stops new submissions on wallet quota 429s while finishing builds already started. Failures are written to push-bulk-failures.jsonl for easy retry.

package_build_context (tar + .dockerignore handling) is moved into images_bulk.py and shared with prime images push. ImageClient / AsyncImageClient gain get_build_status. Tests cover manifest/Harbor validation, concurrency, retries, quota, and failure manifests.

Reviewed by Cursor Bugbot for commit a2109b8. Bugbot is set up for automated code reviews on this repo. Configure here.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0eddb82aa2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/prime/src/prime_cli/commands/images.py Outdated
Comment thread packages/prime/src/prime_cli/commands/images.py Outdated
@kcoopermiller

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9dd9e394b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/prime/src/prime_cli/commands/images.py Outdated
@kcoopermiller kcoopermiller force-pushed the feature/bulk-image-push branch from 9dd9e39 to 5e830e4 Compare July 1, 2026 20:06
@kcoopermiller kcoopermiller changed the title prime images push-many prime images push-bulk Jul 1, 2026
@kcoopermiller

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ab2f2a093

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/prime/src/prime_cli/commands/images_bulk.py
Comment thread packages/prime/src/prime_cli/commands/images_bulk.py
@kcoopermiller

Copy link
Copy Markdown
Member Author

@codex review

Comment thread packages/prime/src/prime_cli/commands/images_bulk.py
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: a2109b8e04

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kcoopermiller

Copy link
Copy Markdown
Member Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a2109b8. Configure here.

@kcoopermiller kcoopermiller merged commit 1d6b645 into main Jul 2, 2026
18 checks passed
@kcoopermiller kcoopermiller deleted the feature/bulk-image-push branch July 2, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants