Skip to content

Dedupe the lexer numeric-suffix list#75

Merged
StreamDemon merged 1 commit into
feature/parser-enhancements-basefrom
feature/parser-enh-suffix-dedup
Jul 2, 2026
Merged

Dedupe the lexer numeric-suffix list#75
StreamDemon merged 1 commit into
feature/parser-enhancements-basefrom
feature/parser-enh-suffix-dedup

Conversation

@StreamDemon

@StreamDemon StreamDemon commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

  • Hoists the numeric-suffix table — previously duplicated verbatim in numeric_suffix and validate_numeric_body — into a single shared NUMERIC_SUFFIXES const in sploosh-lexer.
  • numeric_suffix now returns the matched &'static str instead of allocating a String per scan.
  • Adds a table-driven test that lexes 1<suffix> for every entry in the shared list (kind + lexeme + span asserted), so future suffix additions are covered automatically.

First sub-PR of the parser enhancement wave (targets feature/parser-enhancements-base, see PR #71 roadmap: "Dedupe the numeric-suffix list in the lexer"). No behavior change.

Related Issue

None (PR #71 roadmap item).

Spec Sections Affected

None — implementation quality only.

Checklist

  • Code follows the Sploosh design principles (one way to do it, explicit over implicit, etc.)
  • Documentation updated in relevant docs/ pages — N/A, no behavior change
  • Tests added or updated
  • All build targets still compile (if applicable)
  • Spec-only PR (skip Build Targets section if checked)

Build Targets Tested

  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace all green locally (49 tests).

Test Plan

  • New every_numeric_suffix_lexes test iterates NUMERIC_SUFFIXES and asserts token kind (IntLit/FloatLit), lexeme, and span for each suffix.
  • Existing suffix/separator tests (lexes_suffixed_numbers, rejects_int_suffix_on_float_literals, rejects_float_suffix_on_based_literals, separator_validation_is_base_aware) continue to pass unchanged.

Summary by cubic

Deduplicates the lexer numeric-suffix list by moving it into a shared NUMERIC_SUFFIXES const used by both suffix scanning and separator validation, and returns &'static str to avoid per-scan allocation. Adds a table-driven test that lexes 1<suffix> for every entry; behavior is unchanged.

Written for commit e31a214. Summary will update on new commits.

Review in cubic

`numeric_suffix` and `validate_numeric_body` each carried their own copy
of the 13-entry suffix table, so any future suffix change had to be made
twice or the two paths would drift apart. Hoist the table into a shared
`NUMERIC_SUFFIXES` const and return the matched `&'static str` instead
of allocating a fresh `String` on every suffix scan.

A new test iterates the shared table and lexes `1<suffix>` for every
entry, so additions to the list are covered automatically.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant Lexer as sploosh-lexer Lexer
    participant Suffixes as NUMERIC_SUFFIXES const
    participant Token as Token

    Note over Lexer,Token: Numeric literal lexing (unchanged flow, source of truth deduped)

    Lexer->>Lexer: scan_numeric_literal()
    Lexer->>Suffixes: CHANGED: iterate &NUMERIC_SUFFIXES
    Suffixes-->>Lexer: &'static str suffix match (was String allocation)
    alt Suffix found
        Lexer->>Lexer: advance pos by suffix.len()
        Lexer-->>Token: produce IntLit / FloatLit
    else No suffix
        Lexer-->>Token: produce numeric literal without suffix
    end

    Note over Lexer,Suffixes: Separator validation also uses same const

    Lexer->>Lexer: validate_numeric_body(start, base)
    Lexer->>Suffixes: NEW: strip suffix via &NUMERIC_SUFFIXES
    Suffixes-->>Lexer: body without suffix

    Note over Lexer,Token: Table-driven test covers all suffixes

    participant Test as every_numeric_suffix_lexes test

    Test->>Suffixes: iterate NUMERIC_SUFFIXES
    loop for each suffix
        Test->>Lexer: lex("1{suffix}")
        Lexer-->>Test: tokens[0] (kind, lexeme, span)
        Test->>Test: assert kind (IntLit / FloatLit)
        Test->>Test: assert lexeme == "1{suffix}"
        Test->>Test: assert span correct
    end
Loading

Auto-approved: Refactors numeric suffix list into shared constant and returns &'static str instead of allocating, with new table-driven tests. No behavior change, low risk.

Re-trigger cubic

@StreamDemon StreamDemon merged commit 9a8eab2 into feature/parser-enhancements-base Jul 2, 2026
2 checks passed
@StreamDemon StreamDemon deleted the feature/parser-enh-suffix-dedup branch July 2, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant