Multiple scan XY API Change#51
Open
CSSFrancis wants to merge 15 commits intodirectelectron:mainfrom
Open
Conversation
… validation and initial delay handling
… sphinx-tabs for improved layout
…out client.stop_acquisition in test setup
…alidate single frame result
Member
Author
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
Member
Author
|
pre-commit.ci autofix |
for more information, see https://pre-commit.ci
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Pull request overview
This PR updates the deapi client protocol and test suite to better support advanced scanning workflows, most notably multiple XY scan patterns and (commandVer 16) virtual image buffer retrieval, along with documentation and test reliability improvements.
Changes:
- Extend the client scan-pattern API (
set_xy_array) to accept multiple patterns and broaden input handling/validation. - Add client + simulated server support for virtual image buffer metadata/data retrieval (GET_VIRTUAL_IMAGE_INFO / GET_VIRTUAL_IMAGE) and expose
VirtualImageInfo. - Update docs and tests (Sphinx tabs, new scan design doc, fixture scoping, reduced sleeps, skip flaky/slow file tests).
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 13 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Add Windows-only dependency and adjust optional test/doc extras. |
| doc/reference/scan_design.rst | New design document describing scan/XY pattern behavior and usage examples. |
| doc/reference/index.rst | Include design document section in reference toctree. |
| doc/conf.py | Enable sphinx_tabs extension for tabbed code examples. |
| deapi/version.py | Change commandVersion definition (now hard-coded). |
| deapi/tests/test_utils/test_utils.py | Refactor gain-reference setup into a session fixture; reduce sleeps/poll instead. |
| deapi/tests/test_scanning/test_continual_scanning.py | Add comprehensive “continual scanning” server tests for repeated/multi-pattern scans and timing edge cases. |
| deapi/tests/test_scanning/init.py | Package marker for scanning tests. |
| deapi/tests/test_file_saving/test_scan_pattern_saving.py | Skip slow/broken scan saving test. |
| deapi/tests/test_file_saving/test_file_loading_rsciio.py | Skip slow/broken RSCIO loading test; remove an extra sleep. |
| deapi/tests/test_file_saving/test_file_loading_libertem.py | Move libertem import into test function. |
| deapi/tests/test_client.py | Adjust fixtures/state handling; update timings and a few property interactions. |
| deapi/tests/conftest.py | Make client fixture session-scoped and remove teardown sleep. |
| deapi/simulated_server/fake_server.py | Implement fake handlers for virtual image buffer info/data commands. |
| deapi/data_types.py | Add VirtualImageInfo type for virtual image buffer metadata/dtype mapping. |
| deapi/client.py | Add virtual image buffer APIs, queue flags in start_acquisition, boolean property normalization, enhance set_xy_array, default pixel_format to AUTO. |
| deapi/init.py | Export VirtualImageInfo from package root. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1270
to
1288
| if isinstance(queue_virtual_buffers, bool): | ||
| vb = [queue_virtual_buffers] * 5 | ||
| else: | ||
| if len(queue_virtual_buffers) != 5: | ||
| raise ValueError( | ||
| f"queue_virtual_buffers must be a list of exactly 5 booleans, " | ||
| f"got {len(queue_virtual_buffers)}." | ||
| ) | ||
| vb = list(queue_virtual_buffers) | ||
|
|
||
| if self.width * self.height == 0: | ||
| log.error(" Image size is 0! ") | ||
| else: | ||
| bytesize = 0 | ||
| command = self._addSingleCommand( | ||
| self.START_ACQUISITION, | ||
| None, | ||
| [number_of_acquisitions, request_movie_buffer], | ||
| [number_of_acquisitions, request_movie_buffer] + vb, | ||
| ) |
Comment on lines
+1444
to
+1445
| positions[i] = pos.astype(np.int32) | ||
| elif pos.ndim != 2 or pos.shape[1] != 2: |
| num_positions.append(len(pos)) | ||
|
|
||
| vals_to_send = [int(new_width), int(new_height)] + num_positions | ||
| print("Vals to send:", vals_to_send) |
| Default is 5000. | ||
| virtual_image_info : VirtualImageInfo, optional | ||
| Pre-fetched virtual image metadata (width, height, data type). If | ||
| ``None`` (default), :meth:`get_virtual_image_info` is called |
| @@ -1,3 +1,3 @@ | |||
| version = "5.3.0" | |||
| versionInfo = list(map(int, version.split("."))) | |||
Comment on lines
+248
to
+255
| client["Scan - Size X"] = 4 | ||
| client["Scan - Size Y"] = 4 | ||
| client["Scan - Repeats"] = 2 | ||
| client["Scan - Camera Frames Per Point"] = 8 | ||
| client["Scan - Enable"] = True | ||
| client["Test Pattern"] = "SW Frame Number" | ||
| client["Use DE Camera"] = "Use Frame Time" | ||
|
|
| client["Autosave Movie"] = "On" | ||
| client["Autosave Virtual Image 0"] = "On" | ||
| client["Autosave 4D File Format"] = "MRC" | ||
| client["Use DE Camera"] = "Use Frame Time" |
Comment on lines
+1454
to
+1460
| if positions.ndim > 3: | ||
| log.error( | ||
| "Positions must be a 2D array of shape (N, 2) or 3D array of shape (M, N, 2)" | ||
| ) | ||
| return False | ||
| elif positions.ndim == 2: | ||
| positions = positions[np.newaxis, :, :] |
Comment on lines
+1996
to
+2017
| def get_virtual_image_buffer_info(self) -> VirtualImageInfo: | ||
| """ | ||
| Get information about the virtual image buffer from DE-Server. | ||
|
|
||
| Returns the buffer size, dimensions, and data type for the virtual image | ||
| produced by the active virtual detector configuration. | ||
|
|
||
| Returns | ||
| ------- | ||
| VirtualImageInfo | ||
| Object containing: | ||
| - ``buffer_size`` : total bytes of one virtual image frame | ||
| - ``width`` : image width in pixels | ||
| - ``height`` : image height in pixels | ||
| - ``data_type`` : :class:`~deapi.DataType` of each pixel | ||
| """ | ||
| command = self._addSingleCommand(self.GET_VIRTUAL_IMAGE_INFO, None, None) | ||
| response = self._sendCommand(command) | ||
|
|
||
| info = VirtualImageInfo() | ||
| if response: | ||
| values = self.__getParameters(response.acknowledge[0]) |
Comment on lines
38
to
45
| ) # MRC file loading in LiberTEM is broken! | ||
| @pytest.mark.server | ||
| def test_save_4DSTEM(self, client, file_format): | ||
| import libertem.api as lt | ||
|
|
||
| if not os.path.exists("D:\Temp"): | ||
| os.mkdir("D:\Temp") | ||
| temp_dir = "D:\Temp" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and bug fixes to the
deapiclient and its test suite. The most significant changes are focused on enhancing the flexibility and robustness of scan pattern handling, improving test reliability and performance, and refining property and acquisition management. Below is a summary of the most important changes, grouped by theme.Scan Pattern Handling and Client Improvements:
set_xy_arrayindeapi/client.pyto support lists of numpy arrays, handle both 2D and 3D input, ensure integer types, validate shapes, and correctly compute scan dimensions. This improves flexibility and error handling for scan pattern input.pixel_formatinget_resultto"AUTO"for better compatibility.Test Suite Reliability and Performance:
scope="session"for theclientfixture, reducing setup/teardown overhead and improving test performance.time.sleepcalls and replaced them with more robust idle-waiting logic (e.g.,wait_for_idle(client)), leading to more reliable and faster tests. [1] [2] [3]Test Coverage and Robustness:
File Saving and Loading Tests:
@pytest.mark.skip(reason="Slow and broken")to improve test suite reliability and focus on stable tests. [1] [2]libertem.apiinto the test function to avoid unnecessary imports and potential issues when the library is not available.