Skip to content

perf: paginate combined events with DB-level UNION ALL instead of in-memory sort#2666

Open
mroderick wants to merge 1 commit into
bullet-n-plus-one-fixesfrom
arel-union-pagination
Open

perf: paginate combined events with DB-level UNION ALL instead of in-memory sort#2666
mroderick wants to merge 1 commit into
bullet-n-plus-one-fixesfrom
arel-union-pagination

Conversation

@mroderick

@mroderick mroderick commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

Past events page loads ALL workshops (~1100), meetings, and events into ActiveRecord objects, then sorts and paginates to 20 in Ruby via #sort_by(&:date_and_time).reverse + pagy(sorted, items: 20). The remaining ~1080 rows are instantiated and immediately discarded.

Same pattern exists for upcoming events, though less impactful at lower volumes.

This cost grows linearly with total records regardless of page size.

Solution

Replace the in-memory merge with a UNION ALL at the database level:

  1. paginated_events(upcoming:) — builds three Arel subqueries (workshops x chapters.active, meetings, events), combines them with UNION ALL, runs a COUNT(*) for Pagy, then fetches only 20 (id, event_type) rows via LIMIT/OFFSET.
  2. load_events(rows) — dispatches the 20 IDs to their respective model queries with full eager loading (includes), preserving the UNION's sort order via filter_map.

Pure Arel AST nodes (not extracted .arel from scopes) to avoid bind parameter numbering issues in UNION context.

Both fetch_upcoming_events and fetch_past_events share the same machinery; the upcoming: flag controls sort direction and date comparison.

Benchmark

Tested on GET /events/past (1056 past workshops in DB):

Metric Master (in-memory) UNION ALL (cached)
Total time ~10s 1.3s
DB runtime ~230ms 201ms
View runtime ~1.8s 893ms
Queries 119 118
Allocations ~25M 2.9M

The DB runtime is stable because LIMIT 20 fetches 20 rows regardless of total data volume — this scales to 10k, 100k, or more.

The remaining ~118 queries come from per-workshop N+1s in the view layer (sponsors, host, permissions, organisers) — unchanged between both versions, so comparison is apples-to-apples.

Tech Notes

  • Uses Pagy::Offset.new (Pagy v43.5 API) with a manual Pagy::Request for URL generation
  • Removed :host from Workshop includes (Bullet AVOID warning); uses :workshop_host instead
  • Rebased on origin/bullet-n-plus-one-fixes (PR Fix N+1 queries detected by Bullet #2665) for fair benchmarking — excludes N+1 query noise from the comparison

…memory sort

Extract fetch_upcoming_events and fetch_past_events into two new methods:
- paginated_events(upcoming:) — builds a UNION ALL across workshops,
  meetings, and events with LIMIT/OFFSET at the database level
- load_events(rows) — fetches only the 20 visible rows with full
  eager loading, preserving UNION sort order

Previously, all past workshops (~1100) were loaded into AR objects and
sorted/paginated in Ruby. Now only the current page's 20 rows ever
leave the database.

Benchmark (first load / cached):
  Past events page: 11.3s → 1.3s
  DB queries count: ~119 → 118 (same — view N+1s unchanged)
  Allocations: 25M → 2.9M (cached)
@mroderick mroderick marked this pull request as ready for review June 23, 2026 08:24
@mroderick

Copy link
Copy Markdown
Collaborator Author

The numbers listed above are from local development. production has far more past events, so it's be even more important there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant