feat(collectionView): add Index Management tab

docs/ai-and-plans/PRs/732-index-dashboard/documentdb-supported-indexes.md

@@ -0,0 +1,95 @@
+# DocumentDB-supported indexes (from documentation scraping)
+
+**Source:** [MicrosoftDocs/nosql-docs](https://github.com/MicrosoftDocs/nosql-docs) →
+`azure/documentdb/compatibility-query-language.md`
+([rendered](https://learn.microsoft.com/en-us/documentdb/compatibility-query-language))
+**Scraped by:** `@documentdb-js/operator-registry` →
+`resources/scraped/index-reference.md` → generated `src/indexReference.ts`
+(`INDEX_TYPES`, `INDEX_PROPERTIES`)
+**Last scraped:** 2026-07-07
+
+This file summarizes what we currently know, from the official documentation,
+about the indexes DocumentDB supports over its MongoDB-compatible (MQL) wire
+protocol. It is derived data — re-run `npm run scrape` in the operator-registry
+package to refresh it.
+
+---
+
+## Index types
+
+All eight documented index types are marked **supported**.
+
+| Type             | Description                                      | Type-column badge?                     |
+| ---------------- | ------------------------------------------------ | -------------------------------------- |
+| **Single Field** | Indexes a single field for faster lookups.       | ✅ (field count = 1)                   |
+| **Compound**     | Indexes multiple fields in one index.            | ✅ (field count > 1)                   |
+| **Multikey**     | Indexes array fields by indexing each element.   | ❌ needs runtime array data            |
+| **Text**         | Supports text search on string fields.           | ✅ (`text` key direction)              |
+| **Wildcard**     | Dynamically indexes all or selected fields.      | ✅ (`$**` in a key field)              |
+| **Geospatial**   | Supports spatial queries on GeoJSON data.        | ✅ (`2dsphere` / `2d` / `geoHaystack`) |
+| **Hashed**       | Indexes hashed field values, often for sharding. | ✅ (`hashed` key direction)            |
+| **Vector**       | Enables similarity search on vector data.        | ❌ special key form; DocumentDB-only   |
+
+> The **Type-column badge** column notes how the Index Management webview
+> classifies an _existing_ index from its key spec (see
+> `src/webviews/documentdb/indexView/utils/indexType.ts`). Two additional
+> display-only badges exist that are not index _types_ per se: `Default` (the
+> immutable `_id_` index) and `ObjectId` (a single `_id` field).
+
+### Notes per type
+
+- **Vector** is documented as _"only in DocumentDB"_ and links out to the
+  dedicated vector-search docs. It is the one type with no direct MongoDB
+  equivalent.
+- **Multikey** is not a user-declared type — any index automatically becomes
+  multikey when it covers an array field, so it cannot be detected from the key
+  spec alone.
+- **Hashed** and **Wildcard** are detectable from the key spec and were the two
+  types previously mislabeled by the UI (now fixed).
+
+---
+
+## Index properties
+
+All six documented index properties are marked **supported**. Properties are
+modifiers that can be combined with the index types above.
+
+| Property               | Description                                                                   |
+| ---------------------- | ----------------------------------------------------------------------------- |
+| **TTL** (time-to-live) | Automatically deletes documents after a specified time-to-live period.        |
+| **Unique**             | Ensures that all values in the indexed field are unique.                      |
+| **Partial**            | Indexes only documents that match a specified filter condition.               |
+| **Case Insensitive**   | Supports case-insensitive indexing for string fields.                         |
+| **Sparse**             | Indexes only documents that contain the indexed field.                        |
+| **Background**         | Allows the index to be created in the background without blocking operations. |
+
+> Documentation caveat (from the same page): creating a **unique index** takes
+> an exclusive lock on the collection for the whole build and blocks reads and
+> writes until it completes.
+
+---
+
+## How this maps to the extension today
+
+- **Reference data:** available programmatically via
+  `import { INDEX_TYPES, INDEX_PROPERTIES } from '@documentdb-js/operator-registry';`
+- **Existing-index classification (Type column):** Default, ObjectId,
+  Single Field, Compound, Text, Geospatial, **Hashed**, **Wildcard**.
+- **Creatable index types (Create Index dialog):** currently
+  `asc / desc / ttl / geospatial / text` only. Wildcard, Hashed, and Vector are
+  documented and supported by the service but are **not yet offered in the
+  create UI** — that requires backend create-index key-spec support and a UX
+  decision.
+
+---
+
+## Refreshing this data
+
+```bash
+npm run scrape   --workspace=@documentdb-js/operator-registry   # re-scrape docs
+npm run generate --workspace=@documentdb-js/operator-registry   # regenerate TS
+```
+
+If the counts here (8 types / 6 properties) change after a scrape, the upstream
+compatibility page has been updated — reconcile the webview classification and
+this summary accordingly.

docs/ai-and-plans/PRs/732-index-dashboard/operator-registry-scraper-updates.md

@@ -0,0 +1,134 @@
+# PR #732: Operator-registry scraper migration + index metadata
+
+**Branch:** `dev/khelanmodi/index-management-ui`
+**Base:** `main`
+**Date:** 2026-07-07
+**PR URL:** https://github.com/microsoft/vscode-documentdb/pull/732
+**Commits (this workstream):** 3 on top of the Index Management tab work
+
+> This note covers the changes to the **`@documentdb-js/operator-registry`**
+> constants package (and the small webview follow-on that consumes them). The
+> broader PR #732 adds the Index Management tab; these commits keep the
+> package's generated data current and extend it to cover indexes.
+
+---
+
+## Why
+
+The operator-registry package is auto-generated by scraping the official
+DocumentDB documentation. The scraper had silently broken: the docs were moved
+off `MicrosoftDocs/azure-databases-docs` (that repo now only hosts
+postgresql/horizondb content), so **every scraper URL returned 404** and the
+generated data was ~5 months stale (`Last scraped: 2026-02-20`).
+
+At the same time, the new Index Management tab hardcoded its knowledge of index
+types/properties, and the docs already publish that knowledge in machine-
+readable tables — so it made sense to scrape it into the package rather than
+maintain a second, hand-written copy.
+
+---
+
+## What was done
+
+### Commit 1 — `fix(operator-registry): migrate scraper to nosql-docs public repo`
+
+The docs now live in the public **`MicrosoftDocs/nosql-docs`** repo (branch
+`main`). Updated every source location:
+
+| Constant                       | New value                                                                |
+| ------------------------------ | ------------------------------------------------------------------------ |
+| `COMPAT_PAGE_URL`              | `.../nosql-docs/main/azure/documentdb/compatibility-query-language.md`   |
+| `OPERATOR_DOC_BASE`            | `.../nosql-docs/main/documentdb/query/operators`                         |
+| `DOC_LINK_BASE`                | `https://learn.microsoft.com/en-us/documentdb/query/operators`           |
+| GitHub API base (global index) | `.../repos/MicrosoftDocs/nosql-docs/contents/documentdb/query/operators` |
+| `docLinks.ts` `DOC_BASE`       | `https://learn.microsoft.com/en-us/documentdb/query/operators`           |
+
+Notable structural changes handled:
+
+- Operator pages moved from `articles/documentdb/operators/**` to
+  `documentdb/query/operators/**` (no `azure/` prefix), while the compatibility
+  page sits under `azure/documentdb/...`.
+- Type-expression pages are now flat under `aggregation/` (was
+  `aggregation/type-expression/`), so `expr:type` was remapped accordingly.
+- The compatibility page newly lists **Projection `$meta`** and **Text
+  Expression `$meta`** as supported. These are the same `$meta` operator, so the
+  generator/evaluator/tests map `Text Expression Operator` to the projection
+  meta and deduplicate; the query-operator count invariant moved 43 → 44.
+
+Regenerated `operator-reference.md` and all `src/*.ts` data files, and updated
+`docLinks.test.ts` / `operatorReference.test.ts` / `structuralInvariants.test.ts`
+expectations.
+
+**Operator-count impact — discovery did _not_ regress:**
+
+| Metric                | Before | After   |
+| --------------------- | ------ | ------- |
+| Listed operators      | 308    | **310** |
+| Total in compat table | 324    | 324     |
+| Description coverage  | 100%   | 100%    |
+
+Phase 2 fetched 254/303 operator sub-pages; the ~49 misses are upstream pages
+(string/trig expression + a few aggregation stages) that have **not been
+migrated to `nosql-docs` yet** and 404. Those operators are still _listed_
+(Phase 1), and the hand-maintained overrides keep description coverage at 100%.
+
+### Commit 2 — `feat(operator-registry): scrape & expose index types and properties`
+
+Extended the pipeline to also capture index knowledge from the compatibility
+page's `## Index types` and `## Index properties` tables:
+
+- **Scraper:** new `parseIndexTables()` (Phase 1b) parses both 3-column tables
+  and writes `resources/scraped/index-reference.md`. Handles the upstream
+  quirk where the Vector row is wrapped across two physical lines.
+- **Generator:** parses that dump and emits `src/indexReference.ts` exporting
+  `INDEX_TYPES` (8) and `INDEX_PROPERTIES` (6) as `IndexReferenceEntry`
+  (`{ name, description, supported }`).
+- **Public API:** `index.ts` now exports `INDEX_TYPES`, `INDEX_PROPERTIES`, and
+  the `IndexReferenceEntry` type.
+- Added `indexReference.test.ts` and documented the new outputs in
+  `scripts/README.md`.
+
+### Commit 3 — `feat(indexView): classify Wildcard and Hashed indexes`
+
+Display-only follow-on that uses the newly-scraped knowledge that Wildcard and
+Hashed are supported types. The Index Management **Type** column previously
+mislabeled them as `Single Field` / `Compound`. Added detection from the key
+spec (`'hashed'` direction → **Hashed**; `$**` in a key field → **Wildcard**)
+plus the two badge categories and colours. No create-index changes.
+
+---
+
+## Files changed (package + webview)
+
+- `scripts/scrape-operator-docs.ts` — URL migration + index-table scraping
+- `scripts/generate-from-reference.ts` — index generation + `$meta` mapping
+- `scripts/evaluate-overrides.ts` — `$meta` category mapping
+- `src/docLinks.ts` (+ `docLinks.test.ts`) — new doc base + `expr:type` dir
+- `src/index.ts`, `src/types.ts` — index exports + `IndexReferenceEntry`
+- `src/indexReference.ts` (generated) + `src/indexReference.test.ts`
+- Regenerated `src/{queryOperators,updateOperators,expressionOperators,accumulators,windowOperators,stages}.ts`
+- `resources/scraped/{operator-reference,index-reference}.md`
+- `src/webviews/documentdb/indexView/{constants,types,utils/indexType}.ts`, `components/IndexTypeBadgeView.tsx`
+
+---
+
+## Verification
+
+- Package: `npm run build` ✅, 74 unit tests ✅
+- Repo: `npm run build` ✅, `npm run lint` ✅, full suite **2672 tests / 160 suites** ✅
+- No new user-facing l10n strings (badge labels follow the existing
+  non-localized `{type}` pattern).
+
+---
+
+## Follow-ups (deliberately out of scope)
+
+- **Creatable index types.** The Create Index dialog still offers only
+  `asc / desc / ttl / geospatial / text`. Adding Wildcard/Hashed/Vector requires
+  backend create-index key-spec support and a UX decision.
+- **Multikey / Vector badges.** Not reliably detectable from the key spec alone
+  (Multikey depends on runtime array data; Vector uses a special key form), so
+  they are intentionally not classified in the Type column.
+- **Upstream doc gaps.** ~49 operator pages are not yet migrated to `nosql-docs`
+  and return 404; re-run `npm run scrape` once they land to restore scraped
+  descriptions for those operators.