Skip to content

fix(codegen): remove blank line at EOF in antlr4-rust-gen output#51

Open
Copilot wants to merge 2 commits into
mainfrom
copilot/fix-antlr4-rust-gen-eof
Open

fix(codegen): remove blank line at EOF in antlr4-rust-gen output#51
Copilot wants to merge 2 commits into
mainfrom
copilot/fix-antlr4-rust-gen-eof

Conversation

Copilot AI commented Jun 23, 2026

Copy link
Copy Markdown

Generated .rs files ended with two trailing newlines — a blank line at EOF — causing git diff --check and similar whitespace gates to fail in downstream consumers.

Root cause

GENERATED_MODULE_FOOTER already ends with \n. Both the lexer and parser format strings had an extra \n between {generated_footer} and the closing "#, producing a double newline:

// Before
{generated_footer}
"#

// After
{generated_footer}"#

Changes

  • render_lexer — remove trailing \n before "# in the lexer format string
  • render_parser_with_options — same fix for the parser format string
  • generated_modules_start_with_file_level_header test — tighten assertion from trim_end().ends_with(...) (which masked the bug) to ends_with("pub use self::__antlr4_rust_generated::*;\n"), enforcing exactly one trailing newline

Copilot AI changed the title [WIP] Fix antlr4-rust-gen to not emit a blank line at EOF fix(codegen): remove blank line at EOF in antlr4-rust-gen output Jun 23, 2026
Copilot AI requested a review from tinovyatkin June 23, 2026 19:59
@tinovyatkin tinovyatkin marked this pull request as ready for review June 23, 2026 20:21
@github-actions

Copy link
Copy Markdown

Copy/Paste Detection

Found 9 duplication(s) across 1 changed Rust file(s) (threshold: 100 tokens).

Show duplications

Found a 70 line (285 tokens) duplication in the following files:

  • Starting at line 4885 of src/bin/antlr4-rust-gen.rs
  • Starting at line 5061 of src/bin/antlr4-rust-gen.rs
        let size = ch.len_utf8();
        if line_comment {
            line_comment = ch != '\n';
            index += size;
            continue;
        }
        if block_comment {
            if source.as_bytes().get(index..index + 2) == Some(b"*/") {
                block_comment = false;
                index += 2;
            } else {
                index += size;
            }
            continue;
        }
        if char_set {
            match ch {
                _ if escaped => escaped = false,
                '\\' => escaped = true,
                ']' => char_set = false,
                _ => {}
            }
            index += size;
            continue;
        }
        if escaped {
            escaped = false;
            index += size;
            continue;
        }
        if single_quoted {
            match ch {
                '\\' => escaped = true,
                '\'' => single_quoted = false,
                _ => {}
            }
            index += size;
            continue;
        }
        if double_quoted {
            match ch {
                '\\' => escaped = true,
                '"' => double_quoted = false,
                _ => {}
            }
            index += size;
            continue;
        }
        match ch {
            '/' if source.as_bytes().get(index..index + 2) == Some(b"//") => {
                line_comment = true;
                index += 2;
            }
            '/' if source.as_bytes().get(index..index + 2) == Some(b"/*") => {
                block_comment = true;
                index += 2;
            }
            '\'' => {
                single_quoted = true;
                index += size;
            }
            '"' => {
                double_quoted = true;
                index += size;
            }
            '[' => {
                char_set = true;
                index += size;
            }
            '{' => return Some(index),

Found a 26 line (168 tokens) duplication in the following files:

  • Starting at line 10446 of src/bin/antlr4-rust-gen.rs
  • Starting at line 10570 of src/bin/antlr4-rust-gen.rs
        atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
        atn.state_mut(0)
            .expect("state 0")
            .add_transition(Transition::Epsilon { target: 1 });
        atn.state_mut(1)
            .expect("state 1")
            .add_transition(Transition::Epsilon { target: 2 });
        atn.state_mut(1)
            .expect("state 1")
            .add_transition(Transition::Epsilon { target: 3 });
        atn.state_mut(2)
            .expect("state 2")
            .add_transition(Transition::Atom {
                target: 4,
                label: 1,
            });
        atn.state_mut(3)
            .expect("state 3")
            .add_transition(Transition::Atom {
                target: 4,
                label: 2,
            });
        atn.state_mut(4)
            .expect("state 4")
            .add_transition(Transition::Epsilon { target: 5 });
        atn.add_decision_state(1);

Found a 43 line (157 tokens) duplication in the following files:

  • Starting at line 2092 of src/bin/antlr4-rust-gen.rs
  • Starting at line 2256 of src/bin/antlr4-rust-gen.rs
    )
    .expect("writing to a string cannot fail");
    // Capture the rule start AFTER `enter_rule`, which advances the cursor past any
    // leading hidden-channel tokens to the first visible token. Capturing before
    // would make `$start`/`$text` in generated actions include a leading hidden
    // prefix (e.g. whitespace), diverging from ANTLR and the rule context start.
    writeln!(
        out,
        "        let __rule_start = antlr4_runtime::IntStream::index(self.base.input());"
    )
    .expect("writing to a string cannot fail");
    // Member-setting `@init` runs on rule entry (before the body) so same-rule
    // predicates and actions observe the state it sets.
    render_generated_init_action_entry(
        out,
        index,
        step_render_context.init_entry_action_statements,
        2,
    );
    // Queue the `@init` action event before the body steps so the buffered replay
    // (`run_generated_action`) runs it ahead of body actions, matching ANTLR's
    // "init before body" order. It sits after `__generated_action_marker`, so a
    // fatal-sync abort that truncates back to the marker discards it too.
    render_generated_init_action(out, index, entry_state, init_action_statements, 2);
    writeln!(out, "        let mut __consumed_eof = false;")
        .expect("writing to a string cannot fail");
    writeln!(
        out,
        "        let mut __sync_error: Option<antlr4_runtime::AntlrError> = None;"
    )
    .expect("writing to a string cannot fail");
    writeln!(
        out,
        "        let __result = (|| -> Result<(), antlr4_runtime::AntlrError> {{"
    )
    .expect("writing to a string cannot fail");
    render_generated_steps(out, &rule.steps, 3, step_render_context);
    writeln!(out, "            Ok(())").expect("writing to a string cannot fail");
    writeln!(out, "        }})();").expect("writing to a string cannot fail");
    writeln!(out, "        match __result {{").expect("writing to a string cannot fail");
    writeln!(out, "            Ok(()) => {{").expect("writing to a string cannot fail");
    writeln!(
        out,

Found a 18 line (134 tokens) duplication in the following files:

  • Starting at line 10445 of src/bin/antlr4-rust-gen.rs
  • Starting at line 10485 of src/bin/antlr4-rust-gen.rs
        atn.add_state(AtnState::new(4, AtnStateKind::BlockEnd).with_rule_index(0));
        atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
        atn.state_mut(0)
            .expect("state 0")
            .add_transition(Transition::Epsilon { target: 1 });
        atn.state_mut(1)
            .expect("state 1")
            .add_transition(Transition::Epsilon { target: 2 });
        atn.state_mut(1)
            .expect("state 1")
            .add_transition(Transition::Epsilon { target: 3 });
        atn.state_mut(2)
            .expect("state 2")
            .add_transition(Transition::Atom {
                target: 4,
                label: 1,
            });
        atn.state_mut(3)

Found a 17 line (117 tokens) duplication in the following files:

  • Starting at line 10486 of src/bin/antlr4-rust-gen.rs
  • Starting at line 10570 of src/bin/antlr4-rust-gen.rs
        atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
        atn.state_mut(0)
            .expect("state 0")
            .add_transition(Transition::Epsilon { target: 1 });
        atn.state_mut(1)
            .expect("state 1")
            .add_transition(Transition::Epsilon { target: 2 });
        atn.state_mut(1)
            .expect("state 1")
            .add_transition(Transition::Epsilon { target: 3 });
        atn.state_mut(2)
            .expect("state 2")
            .add_transition(Transition::Atom {
                target: 4,
                label: 1,
            });
        atn.state_mut(4)

Found a 20 line (109 tokens) duplication in the following files:

  • Starting at line 8277 of src/bin/antlr4-rust-gen.rs
  • Starting at line 8421 of src/bin/antlr4-rust-gen.rs
            [GeneratedParserStep::Decision {
                state: 1,
                decision: 0,
                track_alt_number: true,
                allow_semantic_context: false,
                force_context: false,
                fast_path: Some(GeneratedDecisionFastPath {
                    arms: vec![
                        GeneratedDecisionFastArm {
                            alt: 1,
                            intervals: vec![(1, 1)],
                        },
                        GeneratedDecisionFastArm {
                            alt: 2,
                            intervals: vec![(2, 2)],
                        },
                    ],
                }),
                alts: vec![vec![mt(1, 4)], vec![mt(2, 4)]],
            }]

Found a 13 line (105 tokens) duplication in the following files:

  • Starting at line 1592 of src/bin/antlr4-rust-gen.rs
  • Starting at line 1666 of src/bin/antlr4-rust-gen.rs
fn compile_generated_parser_star_loop(
    context: &GeneratedParserCompileContext<'_>,
    state: &antlr4_runtime::atn::AtnState,
    decision: usize,
    stop_state: usize,
    visited: &mut BTreeSet<usize>,
) -> Option<Vec<GeneratedParserStep>> {
    let mut enter = None;
    let mut exit = None;
    for (index, transition) in state.transitions.iter().enumerate() {
        let alt = index + 1;
        let target = transition.target();
        let target_state = context.atn.state(target)?;

Found a 17 line (105 tokens) duplication in the following files:

  • Starting at line 6314 of src/bin/antlr4-rust-gen.rs
  • Starting at line 6474 of src/bin/antlr4-rust-gen.rs
        }
        ActionTemplate::Noop
        | ActionTemplate::Text { .. }
        | ActionTemplate::TextWithPrefix { .. }
        | ActionTemplate::RuleTextWithPrefix { .. }
        | ActionTemplate::StringTree { .. }
        | ActionTemplate::RuleInvocationStack { .. }
        | ActionTemplate::ListenerWalk { .. }
        | ActionTemplate::RuleValue { .. }
        | ActionTemplate::RuleReturnValue { .. }
        | ActionTemplate::SetIntReturn { .. }
        | ActionTemplate::TokenText { .. }
        | ActionTemplate::TokenTextWithPrefix { .. }
        | ActionTemplate::TokenDisplay { .. }
        | ActionTemplate::ExpectedTokenNames { .. }
        | ActionTemplate::Literal { .. }
        | ActionTemplate::MemberValue { .. }

Found a 35 line (103 tokens) duplication in the following files:

  • Starting at line 2156 of src/bin/antlr4-rust-gen.rs
  • Starting at line 2320 of src/bin/antlr4-rust-gen.rs
    writeln!(out, "                        self.base.exit_rule();")
        .expect("writing to a string cannot fail");
    writeln!(
        out,
        "                        self.generated_actions.truncate(__generated_action_marker);"
    )
    .expect("writing to a string cannot fail");
    writeln!(
        out,
        "                        self.base.restore_int_members(__generated_member_checkpoint);"
    )
    .expect("writing to a string cannot fail");
    writeln!(
        out,
        "                        self.base.restore_generated_diagnostics(__generated_diagnostic_marker);"
    )
    .expect("writing to a string cannot fail");
    writeln!(
        out,
        "                        self.base.record_generated_syntax_error();"
    )
    .expect("writing to a string cannot fail");
    writeln!(
        out,
        "                        return Err(GeneratedRuleError::Fatal(__error));"
    )
    .expect("writing to a string cannot fail");
    writeln!(out, "                    }}").expect("writing to a string cannot fail");
    writeln!(
        out,
        "                    self.base.recover_generated_rule(&mut __ctx, atn(), __error);"
    )
    .expect("writing to a string cannot fail");
    writeln!(
        out,

@greptile-apps

greptile-apps Bot commented Jun 23, 2026

Copy link
Copy Markdown

Greptile Summary

This PR removes a trailing blank line at EOF from generated .rs files by trimming an extra before the raw string terminator "# in both the lexer and parser format strings, and tightens the corresponding test to enforce exactly one trailing newline.

  • Lexer fix (render_lexer, line 462): {generated_footer} "#{generated_footer}"#, preventing the double newline appended after GENERATED_MODULE_FOOTER's own trailing .
  • Parser fix (render_parser_with_options, line 4218): identical change in the parser format string.
  • Test hardening (generated_modules_start_with_file_level_header): assertion changed from trim_end().ends_with(...) (which silently masked the double-newline) to a strict ends_with("pub use self::__antlr4_rust_generated::*; ") with a descriptive failure message.

Confidence Score: 5/5

Safe to merge — the change is a minimal two-character removal in two raw string literals, correctly eliminating the double trailing newline in generated files.

Both hunks are identical single-character deletions ( before "#) in isolated raw string literals. GENERATED_MODULE_FOOTER already ends with a newline, so removing the extra one is precisely correct. The test change closes the loophole that allowed the double-newline to go undetected, and no other render functions produce top-level module files, so there are no missed call sites.

No files require special attention.

Important Files Changed

Filename Overview
src/bin/antlr4-rust-gen.rs Removes extra \n before "# in both lexer and parser raw format strings, and tightens the EOF test assertion to prevent regression.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[render_lexer / render_parser_with_options] --> B[Format string with generated_footer placeholder]
    B --> C[GENERATED_MODULE_FOOTER ends with single newline]
    C --> D{Extra newline before closing quote?}
    D -->|Before fix: yes| E[Output ends with double newline - blank line at EOF]
    D -->|After fix: no| F[Output ends with single newline from GENERATED_MODULE_FOOTER]
    E --> G[git diff check fails]
    F --> H[Test assertion passes - ends_with single newline]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[render_lexer / render_parser_with_options] --> B[Format string with generated_footer placeholder]
    B --> C[GENERATED_MODULE_FOOTER ends with single newline]
    C --> D{Extra newline before closing quote?}
    D -->|Before fix: yes| E[Output ends with double newline - blank line at EOF]
    D -->|After fix: no| F[Output ends with single newline from GENERATED_MODULE_FOOTER]
    E --> G[git diff check fails]
    F --> H[Test assertion passes - ends_with single newline]
Loading

Reviews (1): Last reviewed commit: "fix: antlr4-rust-gen no longer emits bla..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants