fix: return flat namespace properties from BigQueryMetastoreCatalog.load_namespace_properties#3650
Open
anxkhn wants to merge 1 commit into
Open
Conversation
…oad_namespace_properties
BigQueryMetastoreCatalog.create_namespace stores user properties in the
ExternalCatalogDatasetOptions.parameters map, but load_namespace_properties
returned external_catalog_dataset_options.to_api_repr(). That API
representation is a nested dict that buries the properties under a
"parameters" key and adds "defaultStorageLocationUri", so a round-trip of
create_namespace(ns, {"owner": "..."}) followed by
load_namespace_properties(ns)["owner"] raised KeyError.
Return the flat parameters map instead, matching the base MetastoreCatalog
contract and the Glue and DynamoDB catalogs. Guard the None case (parameters
is unset by default) so a namespace with no properties yields an empty dict.
Add regression tests covering both the populated round-trip and the
no-parameters case.
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The base
MetastoreCatalogcontract types this method as-> Properties(a flatDict[str, str]), and the Glue and DynamoDB catalogs both return the flat map(
dict(database.get("Parameters", {}))). BigQuery was the outlier. This changesit to return the flat
parametersmap, using the same.parametersaccessoralready used on the table path in this file. The
parametersfield isNonewhen unset, so it is guarded to yield an empty dict for a namespace with no
properties.
Are these changes tested?
Yes. Two regression tests were added to
tests/catalog/test_bigquery_metastore.py:test_load_namespace_properties_returns_flat_parameters: creates a namespacewith
{"owner": ..., "comment": ...}, then assertsload_namespace_propertiesreturns exactly that flat map (the mock feeds the persisted
Datasetbackthrough
get_dataset). This fails on the oldto_api_repr()code (returns thenested dict) and passes with the fix.
test_load_namespace_properties_without_parameters: asserts a dataset whoseoptions carry no parameters yields
{}rather thanNoneor a nested dict.python -m pytest tests/catalog/test_bigquery_metastore.py-> 7 passed. Lint(
ruff,ruff-format,mypy,pydocstyle,codespell) passes on both files.The BigQuery integration tests need Docker + Spark, which were not available in
this environment; the behavior is covered by the unit tests above.
Are there any user-facing changes?
Yes.
BigQueryMetastoreCatalog.load_namespace_propertiesnow returns the flatnamespace property map, consistent with the other catalogs, instead of the nested
BigQuery API representation. Code that relied on the previous nested shape
(reading
["parameters"][...]or["defaultStorageLocationUri"]) would need toread the properties directly; the previous shape did not match the documented
Propertiesreturn type.