Fix red CI on master: plugin deps (aiohttp/scipy) + flytectl install#3448
Merged
Conversation
Signed-off-by: 1fanwang <1fannnw@gmail.com>
…ter -U Both plugin builds are red on master, so every PR inherits two failing checks. onnx-scikitlearn: scikit-learn (via skl2onnx) pulls scipy 1.18, which requires numpy>=2 and uses np.long; under CI's forced numpy<2 it crashes at import and surfaces as 'sklearn is not a package'. Cap scipy to its last numpy<2 line. spark: the '-U flytekit' step re-resolves aiohttp back to 3.14, which aioresponses cannot build a mocked ClientResponse against. Re-pin aiohttp<3.14 after that step. This single pin clears all 18 spark failures: the broken aioresponses patches were leaking a current_context Mock and a dirty Spark session into later tests, which is why the databricks, autospec, and real-Spark (pyspark 4) tests all failed. The plugin itself works on pyspark 4 (every real-Spark test passes on 4.1.2), so no pyspark cap. See flyteorg/flyte#7604. Signed-off-by: 1fanwang <1fannnw@gmail.com>
1fanwang
requested review from
cosmicBboy,
davidmirror-ops,
kumare3,
machichima,
pingsutw,
samhita-alla and
wild-endeavor
as code owners
The integration job's unionai-oss/flytectl-setup-action flaked resolving the
latest flytectl release from the GitHub API ("Cannot read properties of undefined
(reading 'assets')"); it is unrelated to this change and passes on re-run.
Signed-off-by: 1fanwang <1fannnw@gmail.com>
unionai-oss/flytectl-setup-action resolves the latest flytectl from the first page of flyteorg/flyte releases. That page is now entirely Flyte 2.0 (v2.0.x) tags with no flytectl/ release, so the action crashes with "Cannot read properties of undefined (reading 'assets')" and reds the integration jobs on master and every PR. Download the last flytectl release (v0.9.8) from its asset URL instead. Signed-off-by: 1fanwang <1fannnw@gmail.com>
3 tasks
1fanwang
changed the title
machichima
approved these changes
Jun 30, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Two unrelated failures are reding CI on
master(and every PR):Plugin builds —
flytekit-sparkandflytekit-onnx-scikitlearn. A-Udependency step re-resolvesaiohttpto>=3.14(whichaioresponsescan't mock) and pullsscipy>=1.18for onnx. See flytekit plugin CI red on master: flytekit-spark and flytekit-onnx-scikitlearn builds failing flyte#7604.Integration jobs — both
integration_test_*jobs die at Install FlyteCTL:unionai-oss/flytectl-setup-action@masterresolves "latest" from the first page offlyteorg/flytereleases and readsi[0].assets. That page is now entirely Flyte 2.0 (v2.0.x) tags — noflytectl/release on it — so the list is empty and it throws.What changes were proposed in this pull request?
aiohttp<3.14(spark) andscipy<1.18(onnx) after the-Ustep.flytectl/v0.9.8) directly from its asset URL instead of the action.The flytectl root cause is in
unionai-oss/flytectl-setup-action(it reads only the first page offlyteorg/flytereleases, now all Flyte 2.0 tags). Fixing the action would be the cleaner fix for everyone, but flytectl is frozen atv0.9.8, so pinning it directly is deterministic and avoids depending on that action. Happy to pursue a fix in the action instead if you'd prefer.How was this patch tested?
CI on this PR. The plugin builds are green, and the integration jobs now get past Install FlyteCTL and Setup Flyte Sandbox (
flytectl demo start) — both confirmed green on a prior run of the flytectl change — into the integration tests themselves.Check all the applicable boxes