Move check for "running reindex" in the data store

[SergeyGaluzo]

Fixes https://microsofthealth.visualstudio.com/Health/_workitems/edit/186796/

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@7ebcc52). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5648   +/-   ##
=======================================
  Coverage        ?   76.92%           
=======================================
  Files           ?     1004           
  Lines           ?    36760           
  Branches        ?     5562           
=======================================
  Hits            ?    28276           
  Misses          ?     7134           
  Partials        ?     1350           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rajithaalurims]

Backward- and forward-compatible schema changes
The old code works with the new schema changes as expected, which covers our rollback scenario.
However, new code with the old schema changes will silently skip the Reindex check.
Since our rolling release leaves us in a state where pods running new code need to work with the old schema, I think we need version-gated behavior for this change.

Change in customer visible error message:
Before PR: Changes to search parameters is not allowed while a reindex job is ongoing.  Wait for the reindex job with Id: {jobId} to finish, or cancel it.
After PR : Reindex job is in progress. Job Id=12345
Could we replace the SQL message with the resource string so the error message stays the same?

Change in when customer receives the error:
For example when customer runs Bulk delete, it used to fail with 409 at the controller level and return 409 right away. Now it starts the job (returns 202 Accepted), and the job itself fails later with 409 status and a conflict issue in the OperationOutcome. This changes when the error surfaces. Do we need to announce the behavior change? What if the customer was only listening to the bulk update API response and not depending on the actual operation outcome?

[SergeyGaluzo]

Backward- and forward-compatible schema changes The old code works with the new schema changes as expected, which covers our rollback scenario. However, new code with the old schema changes will silently skip the Reindex check. Since our rolling release leaves us in a state where pods running new code need to work with the old schema, I think we need version-gated behavior for this change.

Change in customer visible error message: Before PR: Changes to search parameters is not allowed while a reindex job is ongoing.  Wait for the reindex job with Id: {jobId} to finish, or cancel it. After PR : Reindex job is in progress. Job Id=12345 Could we replace the SQL message with the resource string so the error message stays the same?

Change in when customer receives the error: For example when customer runs Bulk delete, it used to fail with 409 at the controller level and return 409 right away. Now it starts the job (returns 202 Accepted), and the job itself fails later with 409 status and a conflict issue in the OperationOutcome. This changes when the error surfaces. Do we need to announce the behavior change? What if the customer was only listening to the bulk update API response and not depending on the actual operation outcome?

1. Compatibility. Old schema is also checking for reindex conflicts, so we should be covered. New schema is accepting 0 reindex id and does the checks too.
2. I will replace the message with old one.
3. I doubt that there are real customers, who use bulk delete to delete search parameters. If there are, we should announce before, if not - we can just update in release notes. I might need some help in determining that.