Summary
Today env_params keys must name a top-level cmd_args field. They cannot address nested attributes (cmd_args.foo.bar) or other CLI argument bags some workloads expose (e.g. vllm's bench_cmd_args). This makes the env_params path semantics inconsistent with the rest of the parsing surface.
Proposed by @podkidyshev in review of #901 (thread). Explicitly out of scope for #901; tracked here as a follow-up.
Current behavior
TestDefinition.validate_env_params rejects any key that is not a top-level cmd_args field:
unknown = sorted(k for k in self.env_params if k not in cmd_args_fields)
if unknown:
raise ValueError(f"env_params keys {unknown} are not cmd_args fields on ...")
and is_env_sampled / is_dse_job match on the bare key, so nested and non-cmd_args paths are unreachable.
Proposed behavior
Let an env_params key be a full dot-path rooted at TestDefinition, so it can target any nested attribute consistently:
num_nodes = 2
[cmd_args]
foo = 1
[cmd_args.args]
bar = [1, 2, 3]
[env_params]
"cmd_args.args.bar" = { weights = [0.1, 0.1, 0.8] }
This makes env_params paths consistent with how other features address nested config, and unblocks workloads whose sweepable knobs live outside the root cmd_args (e.g. bench_cmd_args).
Scope notes
- Path resolution must be shared with
is_env_sampled / is_dse_job / param_space / apply_params_set so all of them agree on what a key addresses.
- Keep the existing leaf/list/weights validation (a key must resolve to a leaf candidate list).
- Backward compatibility: bare top-level keys should keep working (or a clear migration).
cc @podkidyshev (offered to take this).
Summary
Today
env_paramskeys must name a top-levelcmd_argsfield. They cannot address nested attributes (cmd_args.foo.bar) or other CLI argument bags some workloads expose (e.g. vllm'sbench_cmd_args). This makes theenv_paramspath semantics inconsistent with the rest of the parsing surface.Proposed by @podkidyshev in review of #901 (thread). Explicitly out of scope for #901; tracked here as a follow-up.
Current behavior
TestDefinition.validate_env_paramsrejects any key that is not a top-levelcmd_argsfield:and
is_env_sampled/is_dse_jobmatch on the bare key, so nested and non-cmd_argspaths are unreachable.Proposed behavior
Let an
env_paramskey be a full dot-path rooted atTestDefinition, so it can target any nested attribute consistently:This makes
env_paramspaths consistent with how other features address nested config, and unblocks workloads whose sweepable knobs live outside the rootcmd_args(e.g.bench_cmd_args).Scope notes
is_env_sampled/is_dse_job/param_space/apply_params_setso all of them agree on what a key addresses.cc @podkidyshev (offered to take this).