diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..b7067a7 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,28 @@ +name: Deploy Docs + +on: + release: + types: [published] + +permissions: + contents: write + +jobs: + deploy: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Set up uv + uses: astral-sh/setup-uv@v6 + with: + enable-cache: true + + - name: Install dependencies + run: uv sync --frozen --extra dev + + - name: Deploy docs to GitHub Pages + run: uv run mkdocs gh-deploy --force --clean diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml new file mode 100644 index 0000000..c80d1ea --- /dev/null +++ b/.github/workflows/publish.yml @@ -0,0 +1,42 @@ +name: Publish to PyPI + +on: + release: + types: [published] + +jobs: + build: + name: Build distribution + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v6 + + - name: Build package + run: uv build + + - name: Upload dist artifacts + uses: actions/upload-artifact@v4 + with: + name: dist + path: dist/ + + publish: + name: Publish to PyPI + needs: build + runs-on: ubuntu-latest + environment: pypi-nely + permissions: + id-token: write # required for OIDC trusted publisher + + steps: + - name: Download dist artifacts + uses: actions/download-artifact@v4 + with: + name: dist + path: dist/ + + - name: Publish to PyPI + uses: pypa/gh-action-pypi-publish@release/v1 diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..6ec5227 --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,33 @@ +name: Tests + +on: + push: + branches: ["main", "dev-*"] + pull_request: + branches: ["main"] + +jobs: + test: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"] + + steps: + - uses: actions/checkout@v4 + + - name: Install FFmpeg + run: sudo apt-get update && sudo apt-get install -y ffmpeg + + - name: Set up uv + uses: astral-sh/setup-uv@v6 + with: + enable-cache: true + python-version: ${{ matrix.python-version }} + + - name: Install dependencies + run: uv sync --frozen --extra dev + + - name: Run tests + run: uv run pytest tests diff --git a/.gitignore b/.gitignore index 270bbbf..5d4edf6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,4 @@ -example_output/ +output/ ###################################################################### diff --git a/README.md b/README.md index 94b4e83..67a23c8 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ Renders matplotlib animations by: 1. Creating a bunch of worker processes, and creating matplotlib resources (plt.Figure, plt.Axes, artists, etc.) once per worker 2. Distributing frames across workers via a dynamic queue 3. Rendering the assigned frames from each worker, but updating the data only (without redrawing the whole plot from scratch) -4. Encoding frames to video with PyAV (very efficient FFmpeg under the hood) +4. Encoding frames to video with [parallel-video-io](https://github.com/sibocw/parallel-video-io) (FFmpeg under the hood, with automatic GPU/NVENC acceleration when available) **Key design: Figure reuse.** In each worker process, `setup()` runs once to create the figure, then `update()` modifies it repeatedly. This brings the best of: - Serial processing: avoids the overhead of recreating complex layouts for every frame @@ -37,7 +37,7 @@ from parallel_animate import Animator # Step 1: Create a child class of parallel_animate.Animator class WaveAnimation(Animator): - # Step 2: Define how the plot should be setup + # Step 2: Define how the plot should be set up def setup(self): fig, ax = plt.subplots() self.x = np.linspace(0, 4 * np.pi, 200) @@ -58,12 +58,12 @@ class WaveAnimation(Animator): # Step 4: Define a list of input parameters, one for each frame params = [{"phase": 2 * np.pi * i / 60} for i in range(60)] -# Step 5: Make video in parallel +# Step 5: Make the video in parallel anim = WaveAnimation() anim.make_video("wave.mp4", param_by_frame=params, fps=30, num_workers=4) ``` - + ## Usage This library has a single class: `parallel_animate.Animator`. To make an animation, you must create your own class inheriting from it and define the following methods: @@ -79,37 +79,41 @@ Once you have defined your animator class, there is a single method that you nee - `fps` (int): Frame rate of the output video - `n_frames` (int or None): Number of frames to render. If None, use the length of `param_by_frame`. If param_by_frame does not have `__len__` implemented and `n_frames` is None, the progress bar won't show completion percentage. - `num_workers` (int): Number of worker processes to be spawned. If -1, use all CPU cores. If -2, use all but one CPU cores, etc. If 1, no child process is created and the video is made in the main process itself. Default is -1. -- See the docstring for `parallel_animate.animator` directly for less commonly used, optional parameters. These control logging, rendering quality, etc. +- `video_mode` (str): Encoder selection passed to parallel-video-io: `"auto"` (default) uses the GPU encoder (FFmpeg/NVENC) when a CUDA device is available and falls back to CPU (libx264) otherwise; `"gpu"` forces NVENC and `"cpu"` forces libx264. Output is always an H.264 MP4. +- `video_quality` (int or None), `video_preset` (str or None), `video_extra_ffmpeg_params` (list of str or None): Optional encoding-quality controls forwarded to parallel-video-io. Leave as `None` to use its sensible defaults. +- See the docstring for `parallel_animate.animator` directly for the remaining less commonly used, optional parameters. These control logging, figure reuse, prefetching, etc. + +> **Note:** parallel-video-io is currently Linux-only. ### Special case: frame params arriving out-of-order in `param_by_frame` In some cases, frames in `param_by_frame` might be out of order. We can handle these scenarios by populating `param_by_frame` with a special `parallel_animate.IndexedFrameParams` dataclass, which specifies the frame index that overrides the ordering in `param_by_frame`. This can be useful when, for example, the animator needs to draw frames that are decoded from a video, and the dataloader for that video might return frames in nondeterministic order because it's parallelized. -See `src/parallel_animate/examples/nondeterministic_video_loader.py` for details. +See `examples/nondeterministic_video_loader.py` for details. ## Examples -See [`src/parallel_animate/examples/`](https://github.com/sibocw/parallel-matplotlib-animation/blob/main/src/parallel_animate/examples/): +See [`examples/`](https://github.com/sibocw/parallel-matplotlib-animation/blob/main/examples/). Run all of them (except the benchmark) with `./examples/run_all.sh`. `simple_wave_animation.py`: The example above `multi_panel_animation.py`: 5 subplots with different plot types - + `very_complex_animation.py`: 14 subplots with GridSpec layout - + `nondeterministic_video_loader.py`: handling frames that arrive out of order - + ## Performance test -A [strong scaling test](https://hpc-wiki.info/hpc/Scaling_tests#Strong_Scaling) is implemented in `src/parallel_animate/examples/scaling_test.py`. Here's the result on my 8-core (16-thread) Intel Core i9-11900K Processor: +A [strong scaling test](https://hpc-wiki.info/hpc/Scaling_tests#Strong_Scaling) is implemented in `examples/scaling_test.py`. Here's the result on my 8-core (16-thread) Intel Core i9-11900K Processor: -![](assets/scaling_graph.png) +See the [interactive scaling figure](https://sibocw.github.io/parallel-matplotlib-animation/benchmark/) on the documentation site. The left-most blue dot indicates serial processing with resources reuse. The black line indicates ideal scaling (zero overhead) if all frames are rendered completely independently in parallel (as is the case in all parallel matplotlib animation libraries I found). Blue dots at 1+ workers are what's implemented in this library. diff --git a/assets/multi_panel_animation.gif b/assets/multi_panel_animation.gif deleted file mode 100644 index 7bbec87..0000000 Binary files a/assets/multi_panel_animation.gif and /dev/null differ diff --git a/assets/nondeterministic_video_loader.gif b/assets/nondeterministic_video_loader.gif deleted file mode 100644 index 81111af..0000000 Binary files a/assets/nondeterministic_video_loader.gif and /dev/null differ diff --git a/assets/scaling_graph.png b/assets/scaling_graph.png deleted file mode 100644 index 4a618af..0000000 Binary files a/assets/scaling_graph.png and /dev/null differ diff --git a/assets/simple_wave_animation.gif b/assets/simple_wave_animation.gif deleted file mode 100644 index 8c4504b..0000000 Binary files a/assets/simple_wave_animation.gif and /dev/null differ diff --git a/assets/very_complex_animation.gif b/assets/very_complex_animation.gif deleted file mode 100644 index 2d1e995..0000000 Binary files a/assets/very_complex_animation.gif and /dev/null differ diff --git a/docs/api/animator.md b/docs/api/animator.md new file mode 100644 index 0000000..93d2be9 --- /dev/null +++ b/docs/api/animator.md @@ -0,0 +1,10 @@ +# `parallel_animate` — API Reference + +The public API consists of the `Animator` base class and the +`IndexedFrameParams` helper, both importable directly from `parallel_animate`. + +::: parallel_animate.animator + options: + members: + - Animator + - IndexedFrameParams diff --git a/docs/benchmark.md b/docs/benchmark.md new file mode 100644 index 0000000..5aca57c --- /dev/null +++ b/docs/benchmark.md @@ -0,0 +1,41 @@ +# Benchmark + +A [strong-scaling test](https://hpc-wiki.info/hpc/Scaling_tests#Strong_Scaling) +measures how render-to-video throughput scales with the number of worker +processes. It sweeps: + +- **worker count** — 1, 2, 4, 8, 16; +- **figure reuse** — with vs without (`reuse_figure_object`); and +- **encoding backend** — parallel-video-io's CPU (libx264) and GPU (NVENC) + encoders. The GPU backend is benchmarked only when a CUDA device is present. + +The animation rendered is +[`very_complex_animation.py`](examples.md#very_complex_animationpy), a 14-subplot +`GridSpec` figure that is expensive to build — exactly the case where the +setup-once / update-many design pays off. + +## How to run + +```bash +pip install -e '.[benchmark]' +python examples/scaling_test.py # full sweep +python examples/scaling_test.py --quick # small, fast smoke run +``` + +Results are written to `examples/output/scaling_test/` (`results.csv`, +`results.json`, and the interactive figure embedded below as +`scaling_graph.html`). + +## Result + +Speedup is normalised to the serial (one worker), no-reuse, CPU-encode baseline, +so the curves capture both the parallel speedup and the extra gains from figure +reuse and GPU encoding. The dashed black line is ideal (zero-overhead) linear +scaling. + +--8<-- "examples/output/scaling_test/scaling_graph.html" + +The left-most dark-blue point is serial processing with figure reuse. Points at +2+ workers show the parallel speedup; reuse curves (darker) sit above the +no-reuse curves (lighter) because the expensive layout is built once per worker +instead of once per frame. diff --git a/docs/developer.md b/docs/developer.md new file mode 100644 index 0000000..9e47038 --- /dev/null +++ b/docs/developer.md @@ -0,0 +1,49 @@ +# Developer Info + +Clone the repository and install the development extra: + +```bash +git clone https://github.com/sibocw/parallel-matplotlib-animation.git +cd parallel-matplotlib-animation +pip install -e '.[dev]' +``` + +The `dev` extra installs everything needed for testing, formatting, and building +the documentation site. + +## Testing + +The test suite uses pytest. Run it from the repository root: + +```bash +pytest +``` + +A few tests write small MP4 files via parallel-video-io / FFmpeg, so make sure +`ffmpeg` is on your `$PATH`. + +## Documentation + +The docs are built with [MkDocs](https://www.mkdocs.org/) and the Material theme, +with API pages generated from docstrings by +[mkdocstrings](https://mkdocstrings.github.io/). + +```bash +mkdocs serve # live-preview at http://127.0.0.1:8000 +mkdocs build # build the static site into site/ +mkdocs gh-deploy # build and push to the gh-pages branch +``` + +The [benchmark page](benchmark.md) embeds +`examples/output/scaling_test/scaling_graph.html`, produced +by `python examples/scaling_test.py`. If that file is missing at build time, a +placeholder is substituted by the `docs/gen_benchmark.py` hook so the build still +succeeds. + +## Code style + +- Follow the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html). +- Use ASCII-only characters in `.py` docstrings and comments. +- Format with [ruff](https://docs.astral.sh/ruff/). +- On a new GitHub release, the package is published to PyPI and the docs are + deployed to GitHub Pages automatically. diff --git a/docs/embed_example_videos.py b/docs/embed_example_videos.py new file mode 100644 index 0000000..a56c511 --- /dev/null +++ b/docs/embed_example_videos.py @@ -0,0 +1,40 @@ +"""MkDocs hook: publish the gallery example videos into the built site. + +The example scripts render their animations to ``examples/output/`` (git-ignored, +so the MP4s are not committed to ``main``/``dev`` branches). The docs embed them +via ``