Skip to content

fix: Update capability checks and enhance search functionality in Semantic Search#604

Draft
surajkumar-08 wants to merge 1 commit into
microsoft:mainfrom
surajkumar-08:update-acs-sample
Draft

fix: Update capability checks and enhance search functionality in Semantic Search#604
surajkumar-08 wants to merge 1 commit into
microsoft:mainfrom
surajkumar-08:update-acs-sample

Conversation

@surajkumar-08

Copy link
Copy Markdown

Updates the WCRAPIs Application Content Search (ACS) sample to align with the latest AppContentIndexer IDL and apispec. Adds live text + image suggestions, renders region-of-interest overlays on image matches, fixes a mis-mapped capability check, and resolves several crash / NRE bugs surfaced during testing.

Changes

AIDevGallery/Samples/WCRAPIs/AppIndexCapability.xaml.cs (+2 / −2)

  • Fixed swapped capability checks: the OCR row was reading IndexCapability.ImageSemantic and the Semantic-Image row was reading IndexCapability.ImageOcr. Now each row reflects its intended IndexCapability value.

AIDevGallery/Samples/WCRAPIs/SemanticSearch.xaml (+57 / −16)

  • SearchBox: added TextChanged handler, TextMemberPath="QueryText", and an ItemTemplate that renders live suggestion text with wrapping.
  • ImageResultsBox: new item template renders each result image at scaled display size with a caption and an accent-colored region-of-interest overlay (Canvas + Border) when RegionOfInterest is present.

AIDevGallery/Samples/WCRAPIs/SemanticSearch.xaml.cs (+654 / −107)

Live suggestions (text + image)

  • New session fields: _suggestionTextQuerySession (AppIndexTextQuerySession) and _suggestionImageQuerySession (AppIndexImageQuerySession).
  • Options-signature tracking (_suggestionTextQueryOptionsSignature, _suggestionImageQueryOptionsSignature, _currentSuggestionQueryText) so sessions are only restarted when the relevant options actually change.
  • Per-stream result caches _lastTextSuggestionItems and _lastImageSuggestionItems, merged into the AutoSuggestBox via UpdateCombinedSuggestions.
  • New SearchBox_TextChanged handler creates / updates both sessions via GetOrCreateSuggestionTextQuerySession and GetOrCreateSuggestionImageQuerySession, subscribing to ResultChanged for incremental updates.
  • New session-lifecycle helpers: StopSuggestionTextQuerySession, StopSuggestionImageQuerySession, StopSuggestionQuerySession, plus GetTextMatchSuggestionItems, GetImageMatchSuggestionItems, GetTextQueryOptionsSignature,GetImageQueryOptionsSignature.

Image results with region-of-interest overlay

  • New view-model ImageDisplayItem (ImageSource, Caption, DisplayWidth/Height, RegionLeft/Top/Width/Height, RegionVisibility).
  • GetImageMatchDisplayItems consumes AppManagedImageQueryMatch.RegionOfInterest (IReference<Rect>Rect?) and computes display-coordinate overlays scaled to a max width.
  • _imageDimensions dictionary captures original PixelWidth/PixelHeight at index time inside IndexImageData, so overlays scale correctly regardless of display size.

Bug fixes

  • Search crash on Enter ("Failed to search indexed content.") in SearchBox_QuerySubmitted:
    • Removed StopSuggestionQuerySession() before Task.Run — stopping the image session immediately before invoking indexer.CreateImageQuery(...) on the same indexer was the crash trigger. Sessions now stay alive and are recycledby the next TextChanged / CleanUp(), matching the apispec's interactive-search pattern (sessions only, no mixing one-shots on the same indexer mid-flight).
    • Materialized IVectorView<TextQueryMatch> / IVectorView<ImageQueryMatch> into List<T> inside Task.Run to avoid cross-apartment COM enumeration on the UI thread.
    • Catch block now calls ShowException(ex) (no optionalMessage) and writes ex.Message into ResultStatusTextBlock so the real failure surfaces instead of a generic message.
  • NRE in indexing path: replaced await _indexer?.WaitForIndexingIdleAsync(...) (which becomes await null ⇒ NRE) with a local capture + null-check in TextData_TextChanged, ImageData_ImageOpened, and IndexAll.
  • Misleading "No results found." dropdown: removed the SearchSuggestionItem.NoResults placeholder injection; QuerySubmitted now ignores IsPlaceholder chosen suggestions.

Cleanup

  • CleanUp() now stops both suggestion sessions (text + image) and clears suggestion caches.

Validation

  • dotnet build AIDevGallery -r win-x64 -f net9.0-windows10.0.26100.0 … → 0 errors.
  • ✅ Self-contained MSIX rebuilt, signed, installed on a clean machine — search no longer crashes; image matches show ROI overlays; capability page correctly identifies OCR vs Semantic-Image availability.
  • ✅ Verified text + image suggestions appear as the user types and that pressing Enter returns results (previously crashed).

References

  • IDL: Microsoft.Windows.Search-Lib/AppContentIndex/AppContentIndexer.idl
    • CreateTextQuery / CreateImageQuery are synchronous, no cancellation (lines 218-235).
    • AppIndexImageQuerySession mirrors AppIndexTextQuerySession (lines 900-942).
    • IndexCapability enum (lines 1122-1142).
  • API spec: ApplicationContentIndexer-apispec.md §1627-1690 — canonical interactive-search pattern (sessions only).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant