fix(codegen): remove blank line at EOF in antlr4-rust-gen output#51
Open
Copilot wants to merge 2 commits into
Open
fix(codegen): remove blank line at EOF in antlr4-rust-gen output#51Copilot wants to merge 2 commits into
Copilot wants to merge 2 commits into
Conversation
Copilot
AI
changed the title
[WIP] Fix antlr4-rust-gen to not emit a blank line at EOF
fix(codegen): remove blank line at EOF in antlr4-rust-gen output
Jun 23, 2026
Copy/Paste DetectionFound 9 duplication(s) across 1 changed Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 70 line (285 tokens) duplication in the following files:
let size = ch.len_utf8();
if line_comment {
line_comment = ch != '\n';
index += size;
continue;
}
if block_comment {
if source.as_bytes().get(index..index + 2) == Some(b"*/") {
block_comment = false;
index += 2;
} else {
index += size;
}
continue;
}
if char_set {
match ch {
_ if escaped => escaped = false,
'\\' => escaped = true,
']' => char_set = false,
_ => {}
}
index += size;
continue;
}
if escaped {
escaped = false;
index += size;
continue;
}
if single_quoted {
match ch {
'\\' => escaped = true,
'\'' => single_quoted = false,
_ => {}
}
index += size;
continue;
}
if double_quoted {
match ch {
'\\' => escaped = true,
'"' => double_quoted = false,
_ => {}
}
index += size;
continue;
}
match ch {
'/' if source.as_bytes().get(index..index + 2) == Some(b"//") => {
line_comment = true;
index += 2;
}
'/' if source.as_bytes().get(index..index + 2) == Some(b"/*") => {
block_comment = true;
index += 2;
}
'\'' => {
single_quoted = true;
index += size;
}
'"' => {
double_quoted = true;
index += size;
}
'[' => {
char_set = true;
index += size;
}
'{' => return Some(index),Found a 26 line (168 tokens) duplication in the following files:
atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
atn.state_mut(0)
.expect("state 0")
.add_transition(Transition::Epsilon { target: 1 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 2 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 3 });
atn.state_mut(2)
.expect("state 2")
.add_transition(Transition::Atom {
target: 4,
label: 1,
});
atn.state_mut(3)
.expect("state 3")
.add_transition(Transition::Atom {
target: 4,
label: 2,
});
atn.state_mut(4)
.expect("state 4")
.add_transition(Transition::Epsilon { target: 5 });
atn.add_decision_state(1);Found a 43 line (157 tokens) duplication in the following files:
)
.expect("writing to a string cannot fail");
// Capture the rule start AFTER `enter_rule`, which advances the cursor past any
// leading hidden-channel tokens to the first visible token. Capturing before
// would make `$start`/`$text` in generated actions include a leading hidden
// prefix (e.g. whitespace), diverging from ANTLR and the rule context start.
writeln!(
out,
" let __rule_start = antlr4_runtime::IntStream::index(self.base.input());"
)
.expect("writing to a string cannot fail");
// Member-setting `@init` runs on rule entry (before the body) so same-rule
// predicates and actions observe the state it sets.
render_generated_init_action_entry(
out,
index,
step_render_context.init_entry_action_statements,
2,
);
// Queue the `@init` action event before the body steps so the buffered replay
// (`run_generated_action`) runs it ahead of body actions, matching ANTLR's
// "init before body" order. It sits after `__generated_action_marker`, so a
// fatal-sync abort that truncates back to the marker discards it too.
render_generated_init_action(out, index, entry_state, init_action_statements, 2);
writeln!(out, " let mut __consumed_eof = false;")
.expect("writing to a string cannot fail");
writeln!(
out,
" let mut __sync_error: Option<antlr4_runtime::AntlrError> = None;"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" let __result = (|| -> Result<(), antlr4_runtime::AntlrError> {{"
)
.expect("writing to a string cannot fail");
render_generated_steps(out, &rule.steps, 3, step_render_context);
writeln!(out, " Ok(())").expect("writing to a string cannot fail");
writeln!(out, " }})();").expect("writing to a string cannot fail");
writeln!(out, " match __result {{").expect("writing to a string cannot fail");
writeln!(out, " Ok(()) => {{").expect("writing to a string cannot fail");
writeln!(
out,Found a 18 line (134 tokens) duplication in the following files:
atn.add_state(AtnState::new(4, AtnStateKind::BlockEnd).with_rule_index(0));
atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
atn.state_mut(0)
.expect("state 0")
.add_transition(Transition::Epsilon { target: 1 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 2 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 3 });
atn.state_mut(2)
.expect("state 2")
.add_transition(Transition::Atom {
target: 4,
label: 1,
});
atn.state_mut(3)Found a 17 line (117 tokens) duplication in the following files:
atn.add_state(AtnState::new(5, AtnStateKind::RuleStop).with_rule_index(0));
atn.state_mut(0)
.expect("state 0")
.add_transition(Transition::Epsilon { target: 1 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 2 });
atn.state_mut(1)
.expect("state 1")
.add_transition(Transition::Epsilon { target: 3 });
atn.state_mut(2)
.expect("state 2")
.add_transition(Transition::Atom {
target: 4,
label: 1,
});
atn.state_mut(4)Found a 20 line (109 tokens) duplication in the following files:
[GeneratedParserStep::Decision {
state: 1,
decision: 0,
track_alt_number: true,
allow_semantic_context: false,
force_context: false,
fast_path: Some(GeneratedDecisionFastPath {
arms: vec![
GeneratedDecisionFastArm {
alt: 1,
intervals: vec![(1, 1)],
},
GeneratedDecisionFastArm {
alt: 2,
intervals: vec![(2, 2)],
},
],
}),
alts: vec![vec![mt(1, 4)], vec![mt(2, 4)]],
}]Found a 13 line (105 tokens) duplication in the following files:
fn compile_generated_parser_star_loop(
context: &GeneratedParserCompileContext<'_>,
state: &antlr4_runtime::atn::AtnState,
decision: usize,
stop_state: usize,
visited: &mut BTreeSet<usize>,
) -> Option<Vec<GeneratedParserStep>> {
let mut enter = None;
let mut exit = None;
for (index, transition) in state.transitions.iter().enumerate() {
let alt = index + 1;
let target = transition.target();
let target_state = context.atn.state(target)?;Found a 17 line (105 tokens) duplication in the following files:
}
ActionTemplate::Noop
| ActionTemplate::Text { .. }
| ActionTemplate::TextWithPrefix { .. }
| ActionTemplate::RuleTextWithPrefix { .. }
| ActionTemplate::StringTree { .. }
| ActionTemplate::RuleInvocationStack { .. }
| ActionTemplate::ListenerWalk { .. }
| ActionTemplate::RuleValue { .. }
| ActionTemplate::RuleReturnValue { .. }
| ActionTemplate::SetIntReturn { .. }
| ActionTemplate::TokenText { .. }
| ActionTemplate::TokenTextWithPrefix { .. }
| ActionTemplate::TokenDisplay { .. }
| ActionTemplate::ExpectedTokenNames { .. }
| ActionTemplate::Literal { .. }
| ActionTemplate::MemberValue { .. }Found a 35 line (103 tokens) duplication in the following files:
writeln!(out, " self.base.exit_rule();")
.expect("writing to a string cannot fail");
writeln!(
out,
" self.generated_actions.truncate(__generated_action_marker);"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" self.base.restore_int_members(__generated_member_checkpoint);"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" self.base.restore_generated_diagnostics(__generated_diagnostic_marker);"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" self.base.record_generated_syntax_error();"
)
.expect("writing to a string cannot fail");
writeln!(
out,
" return Err(GeneratedRuleError::Fatal(__error));"
)
.expect("writing to a string cannot fail");
writeln!(out, " }}").expect("writing to a string cannot fail");
writeln!(
out,
" self.base.recover_generated_rule(&mut __ctx, atn(), __error);"
)
.expect("writing to a string cannot fail");
writeln!(
out, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated
.rsfiles ended with two trailing newlines — a blank line at EOF — causinggit diff --checkand similar whitespace gates to fail in downstream consumers.Root cause
GENERATED_MODULE_FOOTERalready ends with\n. Both the lexer and parser format strings had an extra\nbetween{generated_footer}and the closing"#, producing a double newline:Changes
render_lexer— remove trailing\nbefore"#in the lexer format stringrender_parser_with_options— same fix for the parser format stringgenerated_modules_start_with_file_level_headertest — tighten assertion fromtrim_end().ends_with(...)(which masked the bug) toends_with("pub use self::__antlr4_rust_generated::*;\n"), enforcing exactly one trailing newline