fix: allow auto-install of private envs#1852
Draft
mikasenghaas wants to merge 1 commit into
Draft
Conversation
`install_from_hub` fetched env metadata with an unauthenticated `requests.get`, so the Hub returned 404 for any private env (the endpoint requires auth even for public envs the caller can otherwise list). It also bailed out for private envs entirely, since those publish only a source archive (no wheel/simple index). - Send `Authorization: Bearer <token>` + `X-Prime-Team-ID` on every Hub request, resolving creds via `PRIME_API_KEY` / `~/.prime/config.json` (the same resolution the eval client uses). - Install private envs from their presigned `package_url` source archive: download, extract, and `uv pip install` — the source-pull-and-build path `prime env install` takes. - Move `load_prime_config` into `install_utils` and re-export from `client_utils` so there is a single definition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| if isinstance(data, dict): | ||
| return data | ||
| logger.warning("Invalid prime config: expected dict") | ||
| except (RuntimeError, json.JSONDecodeError, OSError) as e: |
There was a problem hiding this comment.
🟢 Low utils/install_utils.py:35
config_file.read_text() raises UnicodeDecodeError when ~/.prime/config.json contains invalid UTF-8, but the except tuple only catches RuntimeError, json.JSONDecodeError, and OSError. Because UnicodeDecodeError is a subclass of ValueError, it escapes the handler and propagates as an unhandled exception instead of falling back to {}. Consider adding UnicodeDecodeError to the exception tuple, or catching ValueError to cover both decoding and JSON parse errors.
Suggested change
| except (RuntimeError, json.JSONDecodeError, OSError) as e: | |
| except (RuntimeError, json.JSONDecodeError, OSError, UnicodeDecodeError) as e: |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/utils/install_utils.py around line 35:
`config_file.read_text()` raises `UnicodeDecodeError` when `~/.prime/config.json` contains invalid UTF-8, but the `except` tuple only catches `RuntimeError`, `json.JSONDecodeError`, and `OSError`. Because `UnicodeDecodeError` is a subclass of `ValueError`, it escapes the handler and propagates as an unhandled exception instead of falling back to `{}`. Consider adding `UnicodeDecodeError` to the exception tuple, or catching `ValueError` to cover both decoding and JSON parse errors.
Evidence trail:
verifiers/utils/install_utils.py lines 26-37 at REVIEWED_COMMIT: shows the except clause on line 35 catches `(RuntimeError, json.JSONDecodeError, OSError)`. Python docs (https://docs.python.org/3/library/exceptions.html) confirm UnicodeDecodeError MRO: UnicodeDecodeError → UnicodeError → ValueError → Exception — not a subclass of any of the three caught exceptions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
install_from_hubfetched env metadata with an unauthenticatedrequests.get, so the Hub returned 404 for any private env — the metadata endpoint requires auth even for public envs the caller could otherwise list. The v1eval <org/name>auto-install (andvf-install/prime-driven installs) therefore failed for private environments before any rollout."Cannot install private environment, use prime env pull"), because private envs publish only a source archive (package_url) — nowheel_url/simple_index_url.This PR makes the verifiers Hub-install path authenticate and handle private envs, mirroring what
prime env installalready does:Authorization: Bearer <token>+X-Prime-Team-ID, resolving credentials viaPRIME_API_KEY→~/.prime/config.json(the same resolutionverifiers/utils/client_utils.pyalready uses for the eval client). Newload_prime_config/prime_auth_headersininstall_utils.py.package_url, download the archive, extract it, anduv pip installthe source tree (build-from-source, the pathprime env installtakes for private envs).load_prime_confignow lives ininstall_utilsand is re-exported fromclient_utils(no duplication, no import cycle).Verification
Tested with a real private env (
mikasenghaas/vf-v1-test) and a fresh uv venv that did not have the env installed.Before (private env, unauthenticated metadata fetch):
After (same private env, fresh venv):
End-to-end
eval mikasenghaas/vf-v1-test -n 2 -r 1then auto-installs the private env and runs: both rolloutsreward=1.000, no errors.Also confirmed:
wheel_url/simple_index_urlpath with the added auth header.tests/test_install_utils.py— all 27 pass;ruff check/ruff formatclean.Repro setup
🤖 Generated with Claude Code
Note
Fix auto-install of private Hub environments via source archive fallback
prime_auth_headers()in install_utils.py that readsPRIME_API_KEY/PRIME_TEAM_IDenv vars (falling back to~/.prime/config.json) and injects them into Hub metadata requests.install_from_hubnow falls back to downloading and installing a source tarball viapackage_urlinstead of failing._install_from_source_archivehelper extracts the tarball to a temp directory and runsuv pip install --upgradeon the source path.load_prime_configis moved from client_utils.py to install_utils.py and re-exported;client_utilsnow declares__all__ = ["load_prime_config"].verifiers.utils.client_utilswill now only exposeload_prime_configinstead of all previously public names.📊 Macroscope summarized 03422cb. 2 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.