feat: Type-aware Durable Functions payload serialization#343
Conversation
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…taples/azure-functions-python-library into andystaples/df-strict-type
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
| class_ = getattr(module, class_name) | ||
| # Resolve the class from already-loaded modules; this function | ||
| # does not import modules on demand. | ||
| module = sys.modules.get(module_name) |
There was a problem hiding this comment.
Are we sure the module will always be in sysmodules? The import_module is removed, so how is this getting added to sys_modules? Shoud we run import_module(module_name) before?
There was a problem hiding this comment.
This is the main update. For majority of the cases, the module will be in sys.modules - it'll be added when the worker indexes the function app file.
| "This message will not be repeated." | ||
| ) | ||
| logger.info(msg) | ||
| warnings.warn(msg, DeprecationWarning, stacklevel=2) |
There was a problem hiding this comment.
ActivityTriggerConverter.decode calls df_loads(data.value) without expected_type, unconditionally tripping _notify_no_expected_type(). The warning tells users to "pass the destination type," but the converter framework doesn't forward annotations so users can't act on it. Should we skip the warning for internal callers?
There was a problem hiding this comment.
That should be caught with the first few lines of df_loads
if expected_type is not None:
return _loads_with_expected_type(s, expected_type)
This method will be called if expected_type is None, which is what we expect
Adds an opt-in, type-aware JSON codec for Durable Functions payloads in
azure.functions._durable_functions, routesActivityTriggerConverterthrough it, and replaces the on-demandimportlib.import_modulecall in_deserialize_custom_objectwith asys.moduleslookup.What changed
azure/functions/_durable_functions.pyNew public API:
df_dumps(value) -> str— JSON-encode a value using the Durable Functions convention.df_loads(s, expected_type=None) -> Any— JSON-decode a string, optionally validating against an expected Python type and using it to reconstruct custom objects.New helper exposed for callers that build their own
json.dumpsinvocations (e.g. orchestrator state encoders):_get_serialize_default()— returns thedefault=callback to pass tojson.dumpsunder the active typing mode.Two operating modes, selected at runtime via the environment variable
AZURE_FUNCTIONS_DURABLE_STRICT_TYPING(truthy values:1,true,yes, case-insensitive):df_dumpsisjson.dumps(value, default=_serialize_custom_object).df_loadsruns the existingobject_hookpath so nested custom objects continue to be reconstructed automatically. Passingexpected_typeadds a class/module check that logs a warning on mismatch but otherwise preserves the legacy decode path.df_dumpsonly wraps the top-level custom object —__data__is serialized as plain JSON with nodefault=hook, soto_json()is responsible for serializing nested custom objects explicitly.df_loadsparses without anobject_hook; deserializing a custom-object envelope requires the caller to passexpected_type, which is then used to callexpected_type.from_json(__data__)directly. Type mismatches raiseTypeError._serialize_custom_objectis unchanged._deserialize_custom_objectnow resolves the declared class viasys.modules.get(...)instead ofimportlib.import_module(...).azure/functions/durable_functions.pyActivityTriggerConverter:decodenow calls_durable_functions.df_loads(data.value)in place of the inlinejson.loads(..., object_hook=...).encodenow calls_durable_functions.df_dumps(obj)in place of the inlinejson.dumps(..., default=...).expected_typebecauseInConverter.decodedoesn't receive the function's parameter type annotation from the worker — flagged as a follow-up.Other converters (
OrchestrationTriggerConverter,EnitityTriggerConverter,DurableClientConverter) are not touched.Behavior changes for existing callers
The wire format is unchanged. Default
df_dumpsoutput equals whatjson.dumps(value, default=_serialize_custom_object)produced before. The behavior deltas are:_deserialize_custom_object(and thereforedf_loads/json.loads(..., object_hook=...)) now resolves the declared class viasys.modules.get(module_name)instead ofimportlib.import_module(module_name). A custom-object envelope whose module has not already been imported by the host process can no longer be deserialized.import_modulechanges a couple of error types raised from the loose-mode decode path:sys.modules→ValueError(previouslyImportError/ModuleNotFoundError).AttributeError(preserved from the oldgetattrbehavior; no change).from_json→TypeError(no change).AZURE_FUNCTIONS_DURABLE_STRICT_TYPINGnow influences this module. Hosts that happen to set it for unrelated reasons will get strict-mode behavior.Strict mode is opt-in. With
AZURE_FUNCTIONS_DURABLE_STRICT_TYPINGunset, behavior matches today's contract apart from items (1) and (2) above.Follow-up
ActivityTriggerConverter.decodecannot currently forward anexpected_typetodf_loadsbecause the worker'sInConverter.decodedispatch doesn't pass the function's parameter type annotation. Consequence: Custom objects are not supported at all in strict-type mode. Adding that plumbing inazure-functions-python-workerwould let strict-mode activity decode fully reconstruct custom objects viafrom_json. Flagged inline in the converter for the library/worker owners' input.