Skip to content

test: filter generation compatibility models#1352

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
axelray-dev:axelray-dev/fix-transformerlens-1328-generation-test-filter
Jun 1, 2026
Merged

test: filter generation compatibility models#1352
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
axelray-dev:axelray-dev/fix-transformerlens-1328-generation-test-filter

Conversation

@axelray-dev
Copy link
Copy Markdown

Fixes #1328

What changed

Replaced the runtime try/except + pytest.skip pattern in test_generation_compatibility with a dedicated generating_model fixture that only includes generation-capable models.

  • Added GENERATING_MODELS class constant (currently ["gpt2"])
  • Added generating_model_name and generating_model fixtures
  • Changed test_generation_compatibility to use the new generating_model fixture
  • Removed the inline except (AttributeError, RuntimeError) + pytest.skip

Why filtering is better than runtime skip

The previous approach silently skipped the generation test when AttributeError or RuntimeError was raised, hiding the fact that some models don't support generation. By filtering at the fixture level:

  • Generation support is explicit in the test parameterization
  • Tests that should run actually run (no silent skips)
  • Adding a non-generating model to the main model_name fixture won't accidentally suppress the generation test
  • Future contributors can see which models are expected to support generation by looking at GENERATING_MODELS

Checks run

uv run pytest tests/unit/model_bridge/compatibility/test_utils.py -v -x
# 14 passed in 77.38s (including test_generation_compatibility[gpt2])

Skipped checks

  • Full test suite not run (would take significantly longer and download many models)
  • No production code changed, so only the targeted test file was verified

@axelray-dev axelray-dev marked this pull request as ready for review June 1, 2026 15:36
@jlarson4 jlarson4 changed the base branch from main to dev June 1, 2026 16:03
@jlarson4
Copy link
Copy Markdown
Collaborator

jlarson4 commented Jun 1, 2026

Hi @axelray-dev!

A few requests:

  1. Could you rebase this to origin/dev to clean up the diff? There are old commits that are leaking in because your code was based on an outdated version of origin/main
  2. I'd like to adjust the approach here. The GENERATING_MODELS allowlist solution here is functional, but it creates another static list of models that will need to be maintained and has the potential to be fragile if forgotten. Could we instead update the ArchitectureAdapter base class to add a supports_generation flag that defaults to true. We can then set it to false false on models that don't support generation. The test suite can check the flag for the skip, rather than managing an allowlist. It also allows us access to the supports_generation information if we need it for future features.

Replace the GENERATING_MODELS allowlist with a supports_generation flag
on ArchitectureAdapter. The flag defaults to True and is set to False on
encoder-only adapters (BERT, HuBERT) that don't support text generation.

The test now checks model.adapter.supports_generation at runtime and
skips via pytest.skip instead of maintaining a static model list.

Fixes TransformerLensOrg#1328
@axelray-dev axelray-dev force-pushed the axelray-dev/fix-transformerlens-1328-generation-test-filter branch from 945f507 to e7117a9 Compare June 1, 2026 20:43
@axelray-dev
Copy link
Copy Markdown
Author

Updated. Here is what changed:

  1. Rebased onto origin/dev. The diff is now clean (4 files, +12/-8).
  2. Added supports_generation: bool = True as a class attribute on ArchitectureAdapter, next to the existing applicable_phases flag.
  3. Set supports_generation = False on BertArchitectureAdapter and HubertArchitectureAdapter (encoder-only models).
  4. Replaced the GENERATING_MODELS allowlist and separate fixtures in the test with a runtime pytest.skip check on model.adapter.supports_generation.

Tests run:

uv run pytest tests/unit/model_bridge/compatibility/test_utils.py -v

All 14 tests passed. The diff is additions-only for the adapter and test files (no unrelated changes).

@jlarson4
Copy link
Copy Markdown
Collaborator

jlarson4 commented Jun 1, 2026

Excellent work, thank you @axelray-dev

@jlarson4 jlarson4 merged commit 73e3228 into TransformerLensOrg:dev Jun 1, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Test Coverage] Generation compatibility for TransformerBridge models lacking native generate()

2 participants