working on the generator code

AGENTS.md

@@ -18,110 +18,86 @@ Keep this section as a Markdown table. When updating entries, maintain the table
 | `src/private/mx/utility/`       | Shared helpers (string, parsing, file system utilities)                              |
 | `src/private/mxtest/`           | Test suite (api, core, file, api import, impl, control, core roundtrip)              |
 | `src/private/cpul/`             | Catch-based unit-test harness and test runner main                                   |
-| `gen/version-a/`                | Historical Ruby/shell scripts from the original brute-force code gen                 |
-| `gen/version-b/`                | Active Rust code generator for reproducing MusicXML 4.0 element classes              |
+| `gen/`                          | Python code generator: `generate.py` (main), `parse.py`, `ids.py`, `quality.py`      |
 | `data/`                         | MusicXML test files and expected-output suites for api import / core roundtrip tests |
-| `docs/musicxml.xsd`             | MusicXML specification (reference)                                                   |
-| `docs/ai/project/`              | AI-assisted project planning and codegen design documents                            |
+| `docs/musicxml.xsd`             | The currently live MusicXML XSD specification                                        |
+| `docs/ai/projects/gen/`         | Active codegen project: plans, state, design docs                                    |
 | `Makefile`                      | Primary build-and-test entry point (wraps CMake; `make help` lists targets)          |
 | `CMakeLists.txt`                | CMake build configuration                                                            |
 
-## Code Generation - Historical Context
+## Code Generation
 
-`src/private/mx/core/` and `src/private/mx/core/elements/` was originally "hand-generated" by human
-brute-force using Ruby scripts which can still be found in `./gen/version-a`. This was never a
-one-shot solution to generating the code from the XSD spec. Rather, it was an iterative process,
-solving problems encountered one-at-a-time until the XSD spec was entirely covered. As such, it is
-not viable for re-use at this time, but can be used to understand the historical nature of how the
-types were first generated.
+Much of the code in `src/private/mx/core/` and `src/private/mx/core/elements/` was originally
+generated by hand-written Ruby scripts in a sort of brute force methodology. These scripts have
+since bin deleted. This was never a one-shot XSD codegen exercise. Many decisions in the original
+gen process were taken by hand — hand-rolled types, non-strict spec interpretations, special-cased
+elements. These choices exist throughout `mx/core`. Some of these decisions should be preserved,
+while others should be reconsidered. The user will decide.
 
-At some point, I tried to create a Rust based codegenerator in order to be able to regenerate
-`mx/core`. However, this devolved and failed. It is kept in `gen/version-b` for historical
-curiosity, but it never worked.
+The active generator is Python-based and lives in `gen/`. Entry point: `python3 gen/generate.py`. It
+reads `docs/musicxml.xsd` and emits C++ into `src/private/mx/core/elements/`. Current target is
+MusicXML 3.0; upgrading to 4.0 is in progress (see Current Project below).
 
-### The Problem
+## Roundtrip Test Suites
 
-We are stuck somewhere around MusicXML 3.1 (or maybe 3.0) because we cannot reliably re-generate the
-types from a newer version of the specification. MusicxML 4.0 has been out for a long time, and we
-want to support it. But we need to write new code-gen tooling to reproduce the emission of the core
-types and then expose the new features in `src/include/mx/api/`.
+Two distinct test suites exercise MusicXML round-tripping. Always use the qualified name; never say
+"roundtrip" unqualified.
 
-Furthermore, many decisions in the original gen process were taken by hand. Using a hand-rolled
-type, for example, instead of what would have strictly implemented the spec. There are human choices
-throughout `mx/core` that will need to be preserved with future code generating efforts.
+- API_IMPORT:
+  - `src/private/mxtest/import/`
+  - `mx::api::DocumentManager`
+  - Exercises: `mx/core`, `mx/impl`, `mx/api`
+  - Use: `make test`
+  - Purpose: test API
 
-Code generation was never, and should not in the future, be designed to generate any valid XSD
-specification. Rather, the goal of code generation is bespoke, to produce what is needed for the
-`mx` library from the MusicXML specification.
-
-## Terminology: Roundtrip Suites
-
-Two distinct test suites exercise MusicXML round-tripping. Always use the qualified name; never
-say "roundtrip" unqualified.
-
-- **api import** (`API_IMPORT`): the existing suite under `src/private/mxtest/import/`. Imports
-  a file through `mx::api::DocumentManager`, which exercises `mx/core` + `mx/impl` + `mx/api`,
-  then compares against a pre-generated expected XML file under `data/expected/`.
-- **core roundtrip** (`CORE_ROUNDTRIP`, `CORE_RT`): the suite under
-  `src/private/mxtest/roundtrip/`. Round-trips a file through `mx::core::Document` only
-  (`fromXDoc` → `toXDoc`), no api/impl involvement, comparing against the normalized input
-  computed in-memory.
+- CORE_ROUNDTRIP (`corert`):
+  - `src/private/mxtest/corert/`
+  - `mx::core::Document`
+  - Exercises: `mx/core` only
+  - Use: `make test-core-dev`
+  - Purpose: test `mx/core`
 
 ## Quality Gates
 
 Always run `make fmt` after modifying code under `src/`.
 
 To see whether your code changes are sound, follow that with:
 - for changes in `src/private/mx/core/*`: `make test-all` (very slow, can be more than 10 minutes)
-- for changes not in `src/private/mx/core/*`: `make test` (faster, can take a couple minues)
+- for changes not in `src/private/mx/core/*`: `make test` (faster, can take a few minues)
 
 Check for warnings with: `make check`.
 
 CI will run all of these plus the `xcode` targets.
 
-## Core Development Mode (Codegen)
+## Codegen Development Mode (core-dev)
 
-Use core-dev when modifying files under `src/private/mx/core/` and you do not need `mx/api`
-or `mx/impl` to compile. Trimmed build for codegen iteration: only `mx/core` + `mx/ezxml` +
-`mx/utility` compile.
+When working on `gen/`, it is permissable to skip the compilation of `mx/api` and `mx/impl` until
+some very final step. The idea is that coding agents will have freedom to innovate and produce
+better generated code without constantly dealing with symbol breakages at the higher levels. Then,
+when reshaping of `mx/core` has reached a desireable state, the upper levels can be fixed and
+tested.
 
-### In-mode iteration
+To skip upper-level compilation, simply run the following:
 
 ```
-make core-dev          # configure + build (fast: skips api/impl)
-make test-core-dev     # run all core roundtrip cases
-make test-core-dev ARGS='[core-roundtrip] lysuite/*'   # subset
+make core-dev                                          # build fast
+make test-core-dev                                     # run core roundtrip tests
+make test-core-dev ARGS='[core-roundtrip] lysuite/*'   # run specific core roundtrip tests
 ```
 
-In-mode gate: `make fmt && make check-core-dev && make test-core-dev`. `fmt` and
-`check-core-dev` run in Docker; `test-core-dev` runs natively.
+Always use `make fmt` though.
 
-### Full pre-merge gate
+### Full Merge Gate
 
-Before merging core changes, run the full gate. Changes under `mx/core/` require
-`test-all`, which exercises per-element core unit tests, api import, and core
-roundtrip together:
+Before PRs run the full gate (unless the user says to skip it):
 
 ```
-make fmt && make check && make test-all
+make fmt
+make check-all
+make test-all
 ```
 
-### What core-dev tests
-
-Backed by the **core roundtrip** suite (`CORE_ROUNDTRIP`, `CORE_RT`) under
-`src/private/mxtest/corert/`. Each `*.xml` / `*.musicxml` file under `data/` (excluding
-`data/expected/`, `data/testOutput/`, `data/generalxml/`, `data/smufl/`) is one Catch2
-test case round-tripping through `mx::core::Document` (`fromXDoc` → `toXDoc`) against a
-normalized input computed in-memory.
-
-The same suite runs inside the normal `mxtest` binary during `make test-all`, gated on
-`MX_BUILD_CORE_TESTS=ON`. Distinct from the **api import** (`API_IMPORT`) suite in
-`src/private/mxtest/import/`, which exercises the full `mx::api` stack against pre-generated
-expected files under `data/expected/`.
-
-For design details see `docs/ai/projects/core-dev/design/design.md`.
-
 ## Current Project
 
-We are working on reverse engineering a new codegen system to regenerate mx/core for MusicXML 4.0.
-See the project directory `./docs/ai/project/gen`.
+A Python-based codegen system is being built to regenerate `mx/core` and eventually target MusicXML
+4.0. See the project directory `./docs/ai/projects/gen` for status.

Dockerfile

@@ -28,6 +28,10 @@ RUN python3 -m venv /opt/quality-venv \
         pylint==4.0.5 \
         cognitive_complexity==1.3.0
 
+RUN python3 -m venv /opt/gen-venv \
+    && /opt/gen-venv/bin/pip install --no-cache-dir \
+        Jinja2==3.1.6
+
 # Unversioned name so the Makefile invokes the pinned formatter without
 # knowing the version suffix.
 RUN ln -sf /usr/bin/clang-format-18 /usr/local/bin/clang-format

Makefile

@@ -75,6 +75,7 @@ GCOV    ?= gcov-14
 # complexity); gen-lint enforces genuine lint defects. Pinned analyzers live in
 # the mx-sdk venv (see Dockerfile). GEN_PY is every gen/*.py except the measurer.
 QUALITY_VENV      := /opt/quality-venv
+GEN_VENV          := /opt/gen-venv
 QUALITY_DIR       := data/testOutput/gen-quality
 GEN_PY            := $(filter-out gen/quality.py,$(wildcard gen/*.py))
 GEN_QUALITY_FLOOR ?= 37.7
@@ -153,7 +154,7 @@ endef
 .DEFAULT_GOAL := help
 .PHONY: help lib dev core test test-all examples-run all clean clean-docker \
         check-docker docker-volume fmt check core-dev check-core-dev \
-        test-core-dev coverage-core-dev gen-quality gen-lint \
+        test-core-dev coverage-core-dev generate gen-quality gen-lint \
         xcode-gen xcode-build xcode-test
 
 help:
@@ -162,6 +163,7 @@ help:
 	@echo 'Quality gates (run in docker):'
 	@echo '  make fmt            Format all C++ files under src/.'
 	@echo '  make check          fmt-check + warning-free build.'
+	@echo '  make generate       Regenerate C++ from the XSD (gen/generate.py).'
 	@echo '  make gen-quality    Score gen/ design quality; fail below the floor.'
 	@echo '  make gen-lint       Lint gen/ with pylint; fail below the floor.'
 	@echo ''
@@ -371,6 +373,9 @@ coverage-core-dev:
 		$(call mode_dir,cov-core-dev) | tee $(COV_DIR)/summary.txt
 	@echo "=== coverage written to $(COV_DIR)/ ==="
 
+generate:
+	$(GEN_VENV)/bin/python gen/generate.py
+
 # Static analysis of the gen/ generator (in-container branch). quality.py
 # measures and writes the report tree to the workspace mount; the floor check
 # below is the gate. See docs/ai/projects/gen.
@@ -434,6 +439,9 @@ coverage-core-dev: $(DOCKER_STAMP) docker-volume
 	$(DOCKER_RUN) make coverage-core-dev
 	@echo "Coverage written to $(COV_DIR)/ (open $(COV_DIR)/index.html)"
 
+generate: $(DOCKER_STAMP)
+	$(DOCKER_RUN) make generate
+
 # Static analysis gates. Pure Python -- no C++ build -- so they only need the
 # image. The report tree is written through the workspace mount to
 # ./data/testOutput/gen-quality. Same commands run identically in CI.