Parser enhancements wave: integration base

[StreamDemon]

Summary

• Integration base branch for the parser enhancement wave — quality/optimization sub-PRs (no behavior fixes) squash-merged into this branch; it lands in main with a true merge commit. Same stacked-PR pattern as the correctness wave (PR Parser correctness wave: integration base #69) and the v0.5.10 cleanup (PR Spec v0.5.10: Cleanup batch (integration) #40).

Wave complete — all six roadmap items shipped:

• Binary/Unary op: String → dedicated operator enums (Replace operator strings with UnaryOp and BinaryOp enums #79)
• Token.lexeme: String → span-based slicing (Token { kind, span } + Token::text(source); slicing chosen over interning — the token stream never outlives the source buffer in the bootstrap pipeline) (Store spans instead of lexeme strings on tokens #76)
• Preserve attribute arguments in the AST (Attribute.args/.span per §16 attr_args; actor handlers are now Handler { attrs, function }) (Preserve attribute arguments and actor-handler attrs in the AST #80)
• Dedupe the numeric-suffix list in the lexer (shared NUMERIC_SUFFIXES const, &'static str return) (Dedupe the lexer numeric-suffix list #75)
• at/eat/expect take &TokenKind Re-sliced: Token/TokenKind derive Copy, which removes every parse-loop .clone() with by-value call sites instead of reference threading — same goal, simpler result; only possible once Store spans instead of lexeme strings on tokens #76 removed the owned lexeme (Make Token and TokenKind Copy, drop parse-loop clones #77)
• Corpus expansion for all accepted shapes — six new fixtures (extern/onchain, casts + literal zoo, async/.await, modules/use, attributes, struct-literal shapes) and auto-discovery of tests/corpus/*.sp in the harness (Expand the corpus to cover every accepted grammar shape #81)

Ride-along: issue #78 (spurious "expected identifier" for non-ident-headed type args like Vec<&str>) was found while designing the attribute-argument parser, verified with a probe, and filed for a separate fix — it is a behavior change, so it deliberately does not ride this wave.

Related Issue

None (review-driven; companion to PR #69).

Spec Sections Affected

None — implementation quality only; no grammar or behavior changes.

Checklist

• Code follows the Sploosh design principles (one way to do it, explicit over implicit, etc.)
• Documentation updated in relevant docs/ pages — N/A, no behavior change
• Tests added or updated
• All build targets still compile (if applicable)
• Spec-only PR (skip Build Targets section if checked)

Build Targets Tested

• cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace green on the assembled base (54 tests, 13 corpus fixtures). Every sub-PR was individually CI-green and cubic-reviewed before merging.

Test Plan

• Each sub-PR carried its own tests: table-driven suffix coverage (Dedupe the lexer numeric-suffix list #75), Token::text span assertions (Store spans instead of lexeme strings on tokens #76), structural operator assertions (Replace operator strings with UnaryOp and BinaryOp enums #79), five attribute-shape/span tests (Preserve attribute arguments and actor-handler attrs in the AST #80), and 13 auto-discovered corpus fixtures (Expand the corpus to cover every accepted grammar shape #81).
• Full workspace suite green at every merge point and on the final base.

[cubic-dev-ai]

No issues found across 1 file

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant L as Lexer
    participant Src as Source Text
    participant NS as NUMERIC_SUFFIXES const
    participant Tok as Token

    Note over L,Tok: Numeric literal scanning with shared suffix const

    L->>Src: rest = &source[pos..]
    L->>L: scan numeric digits & separators
    L->>NS: iterate over suffixes for match
    NS-->>L: suffix (e.g., "i32") as &'static str
    alt suffix found
        L->>Src: advance pos by suffix.len()
        L->>Tok: create token with &'static suffix (no String alloc)
    else no suffix
        L->>Tok: create token without suffix
    end

    Note over L: Later, validate_numeric_body uses same const
    L->>NS: find_map(f: strip suffix from body)
    NS-->>L: suffix to strip (if present)
    L->>L: validate separators in remainder

    Note over L,Tok: NEW: test covers all suffixes
    Test->>L: lex("1<i32>")
    L->>NS: iterate suffixes
    NS-->>L: "i32"
    L-->>Test: Token(IntLit, "1i32")
    Test->>Test: assert kind and lexeme

Loading

<sub>Auto-approved: Refactors numeric suffix handling into a shared const and eliminates per-scan allocations; adds a test covering all suffixes.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Requires human review: Refactors token representation in lexer and parser to use span-based text slicing; moderate risk due to core data structure change.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Requires human review: This PR refactors the lexer to store tokens as span-based slices and the parser to use them, along with other internal improvements. Although no behavior changes are intended, such refactors have a broad impact across core data structures and code paths, making human review necessary.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Requires human review: Introduces structural changes in AST, lexer, and parser (enums, span-based tokens, Copy); risk of subtle breakage across multiple crates.

Re-trigger cubic</sub>