Skip to content

perf: optimize regexp_match for literal pattern usage (20% faster)#23547

Open
andygrove wants to merge 3 commits into
apache:mainfrom
andygrove:auto-opt/regexp_match-datafusion-20260713-235232
Open

perf: optimize regexp_match for literal pattern usage (20% faster)#23547
andygrove wants to merge 3 commits into
apache:mainfrom
andygrove:auto-opt/regexp_match-datafusion-20260713-235232

Conversation

@andygrove

@andygrove andygrove commented Jul 14, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

Optimize existing expressions

What changes are included in this PR?

Pass a literal regexp pattern/flags to arrow's kernel as scalar Datums so the regex is compiled once per batch, instead of expanding the literal to a full array and forcing a per-row HashMap<String, Regex> cache lookup (plus a per-row format! allocation when flags are present).

Are these changes tested?

Existing tests + new unit tests.

Benchmark (criterion):

  • regexp_match_1000 literal pattern utf8view: 21.394% faster (base 249069ns -> cand 195784ns)
  • regexp_match_1000 pattern array: 2.996% faster (base 190020ns -> cand 184327ns)
  • regexp_match_1000 literal pattern: 23.964% faster (base 261795ns -> cand 199059ns)
  • regexp_match_1000 literal pattern and flags: 37.227% faster (base 280047ns -> cand 175794ns)

Full criterion output:

regexp_match_1000 literal pattern
                        time:   [198.39 µs 198.66 µs 199.03 µs]
                        change: [−24.244% −23.964% −23.600%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  4 (4.00%) high severe

regexp_match_1000 literal pattern and flags
                        time:   [175.59 µs 175.68 µs 175.81 µs]
                        change: [−37.342% −37.227% −37.124%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

regexp_match_1000 literal pattern utf8view
                        time:   [195.75 µs 195.85 µs 195.98 µs]
                        change: [−21.511% −21.394% −21.298%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

regexp_match_1000 pattern array
                        time:   [184.11 µs 184.22 µs 184.37 µs]
                        change: [−3.1007% −2.9962% −2.8581%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

Are there any user-facing changes?

@github-actions github-actions Bot added the functions Changes to functions implementation label Jul 14, 2026
@andygrove andygrove changed the title perf: optimize regexp_match in datafusion-functions perf: optimize regexp_match (20% faster) Jul 14, 2026
@andygrove andygrove changed the title perf: optimize regexp_match (20% faster) perf: optimize regexp_match for scalar pattern usage (20% faster) Jul 14, 2026
@andygrove andygrove changed the title perf: optimize regexp_match for scalar pattern usage (20% faster) perf: optimize regexp_match for literal pattern usage (20% faster) Jul 14, 2026
@andygrove andygrove marked this pull request as ready for review July 14, 2026 15:54
@andygrove andygrove requested review from Jefffrey and comphead July 14, 2026 15:54
Comment on lines +179 to +199
let flags = match flags {
None => None,
Some(ColumnarValue::Scalar(flags)) => Some(flags),
// An array of flags has to be zipped with the values row by row.
Some(ColumnarValue::Array(_)) => return Ok(None),
};

if !matches!(pattern.try_as_str(), Some(Some(_)))
|| flags.is_some_and(|flags| flags.try_as_str() == Some(Some("g")))
{
return Ok(None);
}

// The kernel requires the values, the pattern and the flags to share one
// string type.
let value_type = values.data_type();
if &pattern.data_type() != value_type
|| flags.is_some_and(|flags| &flags.data_type() != value_type)
{
return Ok(None);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let flags = match flags {
None => None,
Some(ColumnarValue::Scalar(flags)) => Some(flags),
// An array of flags has to be zipped with the values row by row.
Some(ColumnarValue::Array(_)) => return Ok(None),
};
if !matches!(pattern.try_as_str(), Some(Some(_)))
|| flags.is_some_and(|flags| flags.try_as_str() == Some(Some("g")))
{
return Ok(None);
}
// The kernel requires the values, the pattern and the flags to share one
// string type.
let value_type = values.data_type();
if &pattern.data_type() != value_type
|| flags.is_some_and(|flags| &flags.data_type() != value_type)
{
return Ok(None);
}
let flags = match flags {
Some(ColumnarValue::Array(_)) => return Ok(None),
Some(ColumnarValue::Scalar(flags)) => Some(flags),
None => None,
};
let value_type = values.data_type();
if !matches!(pattern.try_as_str(), Some(Some(_)))
|| &pattern.data_type() != value_type
|| flags.is_some_and(|flags| {
flags.try_as_str() == Some(Some("g")) || &flags.data_type() != value_type
})
{
return Ok(None);
}

more concise IMO

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants