Skip to content

fix: allow auto-install of private envs#1852

Draft
mikasenghaas wants to merge 1 commit into
mainfrom
fix/hub-install-auth
Draft

fix: allow auto-install of private envs#1852
mikasenghaas wants to merge 1 commit into
mainfrom
fix/hub-install-auth

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Jun 24, 2026

Copy link
Copy Markdown
Member

Summary

  • install_from_hub fetched env metadata with an unauthenticated requests.get, so the Hub returned 404 for any private env — the metadata endpoint requires auth even for public envs the caller could otherwise list. The v1 eval <org/name> auto-install (and vf-install/prime-driven installs) therefore failed for private environments before any rollout.
  • It also bailed out of private envs by design ("Cannot install private environment, use prime env pull"), because private envs publish only a source archive (package_url) — no wheel_url/simple_index_url.

This PR makes the verifiers Hub-install path authenticate and handle private envs, mirroring what prime env install already does:

  • Authenticate every Hub request — send Authorization: Bearer <token> + X-Prime-Team-ID, resolving credentials via PRIME_API_KEY~/.prime/config.json (the same resolution verifiers/utils/client_utils.py already uses for the eval client). New load_prime_config / prime_auth_headers in install_utils.py.
  • Install private envs from source — when the metadata exposes only a presigned package_url, download the archive, extract it, and uv pip install the source tree (build-from-source, the path prime env install takes for private envs).
  • Single source of truthload_prime_config now lives in install_utils and is re-exported from client_utils (no duplication, no import cycle).

Verification

Tested with a real private env (mikasenghaas/vf-v1-test) and a fresh uv venv that did not have the env installed.

Before (private env, unauthenticated metadata fetch):

Failed to fetch environment details: 404 Client Error: Not Found for url:
  https://api.primeintellect.ai/api/v1/environmentshub/mikasenghaas/vf-v1-test/@latest
ModuleNotFoundError: could not install 'mikasenghaas/vf-v1-test' from the environments hub

After (same private env, fresh venv):

Fetching environment details for mikasenghaas/vf-v1-test@latest...
Installing mikasenghaas/vf-v1-test from source archive...
Successfully installed mikasenghaas/vf-v1-test

End-to-end eval mikasenghaas/vf-v1-test -n 2 -r 1 then auto-installs the private env and runs: both rollouts reward=1.000, no errors.

Also confirmed:

  • No public-env regression — public auto-install still works through the wheel_url/simple_index_url path with the added auth header.
  • tests/test_install_utils.py — all 27 pass; ruff check / ruff format clean.

Repro setup

# fresh venv with v1-capable verifiers, env NOT installed
uv venv && uv pip install -e /path/to/verifiers
# private env on the hub; PRIME_API_KEY (or ~/.prime/config.json) available
eval mikasenghaas/vf-v1-test -n 2 -r 1 --rich False

🤖 Generated with Claude Code

Note

Fix auto-install of private Hub environments via source archive fallback

  • Adds prime_auth_headers() in install_utils.py that reads PRIME_API_KEY/PRIME_TEAM_ID env vars (falling back to ~/.prime/config.json) and injects them into Hub metadata requests.
  • When a Hub package provides neither a wheel nor a simple index URL, install_from_hub now falls back to downloading and installing a source tarball via package_url instead of failing.
  • The new _install_from_source_archive helper extracts the tarball to a temp directory and runs uv pip install --upgrade on the source path.
  • load_prime_config is moved from client_utils.py to install_utils.py and re-exported; client_utils now declares __all__ = ["load_prime_config"].
  • Risk: star-imports from verifiers.utils.client_utils will now only expose load_prime_config instead of all previously public names.
📊 Macroscope summarized 03422cb. 2 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

`install_from_hub` fetched env metadata with an unauthenticated
`requests.get`, so the Hub returned 404 for any private env (the endpoint
requires auth even for public envs the caller can otherwise list). It also
bailed out for private envs entirely, since those publish only a source
archive (no wheel/simple index).

- Send `Authorization: Bearer <token>` + `X-Prime-Team-ID` on every Hub
  request, resolving creds via `PRIME_API_KEY` / `~/.prime/config.json`
  (the same resolution the eval client uses).
- Install private envs from their presigned `package_url` source archive:
  download, extract, and `uv pip install` — the source-pull-and-build path
  `prime env install` takes.
- Move `load_prime_config` into `install_utils` and re-export from
  `client_utils` so there is a single definition.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikasenghaas mikasenghaas changed the title fix: authenticate Environments Hub installs and support private envs fix: allow auto-install of private envs Jun 24, 2026
if isinstance(data, dict):
return data
logger.warning("Invalid prime config: expected dict")
except (RuntimeError, json.JSONDecodeError, OSError) as e:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low utils/install_utils.py:35

config_file.read_text() raises UnicodeDecodeError when ~/.prime/config.json contains invalid UTF-8, but the except tuple only catches RuntimeError, json.JSONDecodeError, and OSError. Because UnicodeDecodeError is a subclass of ValueError, it escapes the handler and propagates as an unhandled exception instead of falling back to {}. Consider adding UnicodeDecodeError to the exception tuple, or catching ValueError to cover both decoding and JSON parse errors.

Suggested change
except (RuntimeError, json.JSONDecodeError, OSError) as e:
except (RuntimeError, json.JSONDecodeError, OSError, UnicodeDecodeError) as e:
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/utils/install_utils.py around line 35:

`config_file.read_text()` raises `UnicodeDecodeError` when `~/.prime/config.json` contains invalid UTF-8, but the `except` tuple only catches `RuntimeError`, `json.JSONDecodeError`, and `OSError`. Because `UnicodeDecodeError` is a subclass of `ValueError`, it escapes the handler and propagates as an unhandled exception instead of falling back to `{}`. Consider adding `UnicodeDecodeError` to the exception tuple, or catching `ValueError` to cover both decoding and JSON parse errors.

Evidence trail:
verifiers/utils/install_utils.py lines 26-37 at REVIEWED_COMMIT: shows the except clause on line 35 catches `(RuntimeError, json.JSONDecodeError, OSError)`. Python docs (https://docs.python.org/3/library/exceptions.html) confirm UnicodeDecodeError MRO: UnicodeDecodeError → UnicodeError → ValueError → Exception — not a subclass of any of the three caught exceptions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant