BE-596: Materialize entity aggregates into entity_edition_cache#8854
BE-596: Materialize entity aggregates into entity_edition_cache#8854TimDiekmann wants to merge 6 commits into
entity_edition_cache#8854Conversation
Replaces the seven per-row aggregate views (labels, type titles, type IDs) with a denormalized edition-keyed cache table. Arrays cover all inheritance depths in canonical order (depth, title, base URL, version DESC) with the direct types as prefix (direct_types); labels resolve per direct type through the closed schema (nearest ancestor first). Populated transactionally on entity writes via a shared statement template, rebuilt by reindex_entity_cache (now also chained to reindex_entity_type_cache), and backfilled in V51. The query compiler, projection decode, and HashQL lateral read the cache; descending sorts reuse the first element instead of flipping to last_* views. Nullable sort keys (entity label) surface through the cursor decode as the IS-NULL sentinel.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
entity_edition_cache
entity_edition_cacheentity_edition_cache
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8854 +/- ##
==========================================
+ Coverage 59.13% 59.17% +0.04%
==========================================
Files 1346 1346
Lines 130096 130111 +15
Branches 5883 5885 +2
==========================================
+ Hits 76931 76995 +64
+ Misses 52259 52210 -49
Partials 906 906
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Convert the SQL template into insert_entity_edition_cache_statement(scoped: bool) to inline scoped filters for types and labels. Add unit tests to verify scoped/unscoped occurrence counts and that the jsonb text-extraction operator's braces survive format! escaping. Also consolidate duplicate match arms for TypeBaseUrls/TypeVersionedUrls into a single vector type
PR SummaryHigh Risk Overview The cache stores The query compiler and HashQL now read Behaviour change: Reviewed by Cursor Bugbot for commit db769d6. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 72a957f. Configure here.
There was a problem hiding this comment.
Pull request overview
This PR introduces a denormalized entity_edition_cache table to materialize per-entity-edition type/label aggregates on the write path, replacing several expensive aggregate views to significantly reduce queryEntitySubgraph latency.
Changes:
- Add
entity_edition_cache(migrations + write-path population + reindex support) and drop legacy aggregate views. - Update read/query compilation paths to source type IDs, type-title/label sort keys, and type containment filtering from the cache (including array element access and ordinality-restricted unnesting).
- Update Postgres query cursor decoding to tolerate nullable sort keys (e.g. unlabeled entities).
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/@local/hashql/eval/tests/ui/postgres/filter/provides_drives_select_and_joins.snap | Snapshot update: HashQL SQL now joins entity_edition_cache and restricts unnested rows via ordinality/direct_types. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/property_mask.snap | Snapshot update: same cache-based join/unnest changes. |
| libs/@local/hashql/eval/tests/ui/postgres/filter/data_island_provides_without_lateral.snap | Snapshot update: same cache-based join/unnest changes. |
| libs/@local/hashql/eval/tests/ui/postgres/entity-type-ids-lateral.stdout | Snapshot update: lateral type-id projection now uses cache + ordinality filter. |
| libs/@local/hashql/eval/src/postgres/projections.rs | Update HashQL entity_type_ids lateral projection to source from cache and filter to direct-type prefix. |
| libs/@local/graph/store/src/filter/protection.rs | Adjust protected-path collection for removed paths and new cache-backed query paths. |
| libs/@local/graph/store/src/entity/query.rs | Add TypeVersionedUrls/DirectTypeCount, remove descending “Last*” variants, and update sorting behavior. |
| libs/@local/graph/postgres-store/src/store/postgres/query/table.rs | Add EntityEditionCache table/columns and remove legacy view-backed tables/relations. |
| libs/@local/graph/postgres-store/src/store/postgres/query/statement/select.rs | Add compilation test for sorting by cached label/type title; update masking join expectation. |
| libs/@local/graph/postgres-store/src/store/postgres/query/mod.rs | Decode nullable sort keys as Json(Null) sentinel for cursor continuation logic. |
| libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs | Add Expression::ArrayElement for (<expr>)[idx] SQL emission. |
| libs/@local/graph/postgres-store/src/store/postgres/query/entity.rs | Route entity query paths to cache relations/columns; use array subscript for first label/title. |
| libs/@local/graph/postgres-store/src/store/postgres/query/compile.rs | Support JsonField::ArrayElement compilation and ensure filter parameter typing treats it as text. |
| libs/@local/graph/postgres-store/src/store/postgres/ontology/entity_type.rs | Trigger cache reindex after entity-type schema reindex to avoid stale cached titles/labels. |
| libs/@local/graph/postgres-store/src/store/postgres/knowledge/entity/query.rs | Decode entity type IDs from cached versioned_urls using direct_types prefix length. |
| libs/@local/graph/postgres-store/src/store/postgres/knowledge/entity/mod.rs | Populate cache on write paths; implement cache rebuild during reindex_entity_cache. |
| libs/@local/graph/postgres-store/src/snapshot/entity/batch.rs | Reindex cache after snapshot restore updates that may change label-derived cache entries. |
| libs/@local/graph/postgres-store/postgres_migrations/V51__entity_edition_cache.sql | Production migration: create/backfill cache + indexes; drop legacy aggregate views. |
| libs/@local/graph/migrations/graph-migrations/v009__entities/up.sql | Test migration: add cache table/indexes; remove legacy aggregate views. |
| libs/@local/graph/migrations/graph-migrations/v009__entities/down.sql | Test migration rollback: drop cache table and remove obsolete view drops. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1001 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1526 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 51 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 107 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 25 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |

🌟 What is the purpose of this PR?
The entity-type aggregate views dominated
queryEntitySubgraphruntime: every type filter/sort multiplied rows throughentity_is_of_type⋈ ontology joins and forcedDISTINCT ON. This PR materializes the per-edition aggregates into a newentity_edition_cachetable maintained on the write path. Entities-table page query on a 1.9M-edition dataset: ~130s → ~14s.🔗 Related links
🔍 What does this change?
entity_edition_cachetable:direct_typescount + parallel arrayslabels,type_titles,base_urls,versions,versioned_urls(all inheritance depths, canonical ordering; direct types as array prefix), GIN onbase_urls/versioned_urlsV51(create + backfill + drop the 7 aggregate views), test-onlyv009updated inlinereindex_entity_cache)Relation::EntityEditionCache,TypeVersionedUrls/DirectTypeCountpaths,(labels)[1]viaExpression::ArrayElement; HashQL projections unnest from the cacheOption<CursorField>;Last*DESC-flip variants removedtypeBaseUrls(and the email-masking rule built on it) is now inheritance-aware — the droppedentity_is_of_type_idsview only listed direct types. Entities whose direct type inherits fromusernow get email masking applied; previously their email was exposed.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
V51backfill and the code going live miss the cache — runreindex-cache --entitiesonce post-deploy (noted in the migration)🐾 Next steps
🛡 What tests cover this?
❓ How to test this?
migrateagainst an existing database — the backfill populatesentity_edition_cacheincludeTypeIds— identical results, no aggregate-view joins in the compiled SQLcd tests/graph/http && sh test.sh