Fix camera unprojection pixel-space mismatch (I1-A) + inference c= typo

[sibocw]

This work was implemented by Claude Code (Opus 4.8) upon user instruction.

Summary

The 3D-keypoints model emits pred_xy in the input-image pixel space, which at inference is inference_image_size = (256, 256) (the model is trained on 256-px labels). pose/camera.py::CameraToWorldMapper unprojects (pixel_xy, depth_mm) -> world_mm, and its intrinsics (focal length and principal point) are derived from the size it is built with — both scale linearly with that size.

The bug (I1-A): both the inference script and the production pipeline built the mapper with camera_rendering_size = (464, 464) (the simulation render size) while feeding it 256-px predictions. Because focal length and principal point scale with the build size, this injects an anisotropic distortion: the in-plane (x, y) coordinates are scaled by 256/464 ≈ 0.55 while depth is left exact. Measured downstream impact: a median inverse-kinematics joint-angle error of ~12° (CTr) / ~19° (FTi).

For 256-px predictions the mapper must be built at rendering_size = 256 — this is provably correct (focal & principal point scale linearly with sensor size; depth is unaffected).

Separate crash: run_keypoints3d_inference.py passed c=inference_image_size to run_keypoints3d_inference(...), whose parameter is named inference_image_size, raising TypeError. The script's __main__ was therefore dead.

Fix

• pose/keypoints3d/scripts/run_keypoints3d_inference.py: build the CameraToWorldMapper at inference_image_size (the pixel space pred_xy lives in) instead of camera_rendering_size. Added a clear comment explaining why, and a runtime note if camera_rendering_size != inference_image_size (the simulation render size is intentionally not used for unprojection). The camera_rendering_size parameter is kept for now.
• pose/keypoints3d/scripts/run_keypoints3d_inference.py: fixed c=inference_image_size → inference_image_size=inference_image_size (the crash typo at the __main__ call site).
• production/spotlight/keypoints3d.py: applied the identical one-line mapper-resolution fix — build the mapper at working_size (the prediction space) instead of camera_rendering_size. Production was touched only to apply this identical fix; it was otherwise out of audit scope.
• pose/camera.py: documented in CameraToWorldMapper.__call__ that xy must be expressed in rendering_size pixels (docstring-only; no behavior change), to guard the pixel-space assumption at the API level.

Tests

Added tests/test_camera_unprojection.py (camera.py imports only numpy/scipy, so it runs standalone):

• test_unprojection_inverts_projection[128/256/464] — builds a mapper at size S, forward-projects known world points with the mapper's own camera_matrix, unprojects via __call__, and asserts recovery to atol=1e-6 (actual error ~1e-15).
• test_focal_scales_with_size_depth_does_not — asserts the in-plane focal scaling ratio is exactly 256/464 ≈ 0.55 while the depth axis of the focal transform is the identity regardless of size.
• test_same_input_different_mapper_size_anisotropic_distortion — unprojects the same 256-space (xy, depth) with a 256-mapper vs a 464-mapper, showing the world outputs differ, the camera-frame depth ratio is exactly 1.0, and the in-plane scaling is 256/464 ≈ 0.55 — confirming 256 is correct for 256-space inputs.

Result:

$ .venv/bin/python -m pytest tests/test_camera_unprojection.py -q
.....                                                                    [100%]
5 passed in 0.25s

Notes

Addresses I1-A and the c= crash from #48. Refs #48 (using "Refs", not "Closes").

🤖 Generated with Claude Code