JPSS CRIS-FSR metadata fix

[aayushg55]

Earth2Studio Pull Request

Description

Problem

JPSS_CRIS treated monotonic IET microseconds as elapsed UTC, truncated source
time to milliseconds, published satellite zenith as scan_angle, discarded
scan/FOR/FOV identity, and named a post-Planck brightness-temperature matrix
radiance. Its rounded time/location deduplication could also collapse
distinct source footprints.

Change

• Convert each FORTime from IET with the paired UTC/IET beginning anchor in
  the same GEO granule:
  utc = beginning_utc + (FORTime - beginning_iet).
• Preserve source microseconds through filtering and publish them losslessly in
  the existing Arrow timestamp[ns] field.
• Add one-based scan_line, field_of_regard, and field_of_view metadata.
• Publish the nominal CrIS look angle, including the rotating 3x3 detector
  offset, as scan_angle; keep source SatelliteZenithAngle in
  satellite_za.
• Rename the private post-Planck matrix to brightness_temperature.
• Keep optional sensor_indices projection in source order after full-band
  apodization, but before Planck inversion and long-format expansion.
• Deduplicate only exact repeated source footprints using source granule,
  platform, variable, scan line, FOR, and FOV. No rounded geolocation is used.

Source Semantics

• Native footprint domain: scan line x 30 FOR x 9 detector FOV.
• scan_line is one-based and local to the source granule.
• field_of_regard is the one-based telescope position across a scan, 1--30.
• field_of_view is the one-based detector position within a FOR, 1--9.
• satellite_za remains the measured GEO satellite zenith angle.
• scan_angle is the nominal instrument look angle derived from FOR/FOV using
  the CrIS detector geometry documented in the
  NOAA CrIS SDR ATBD
  (successor to JPSS 474-00032) and implemented by
  NCEP GSI read_cris.f90.
• Public observation remains float32 brightness temperature in kelvin.

Compatibility

• sensor_indices=None is an appended optional constructor argument.
• The schema adds scan_line, field_of_regard, and field_of_view.
• scan_angle keeps its existing nullable float32 type but now carries the
  correct nominal angle instead of satellite zenith.
• The raw-HDF cache format is unchanged.
• Radiometry and default channel populations are unchanged.

Numerical Validation

Four real NOAA granules cover S-NPP, NOAA-20, and NOAA-21 across 2024--2025.
Each case projects channels [1, 714, 1579, 2211] before long expansion.

Gate	Result
Old direct-IET time error	exactly +37.0 s in all four cases
Corrected first FOR vs exact HDF UTC anchor	0 µs in all four cases
BT change in isolated time-conversion replay	0.0 K maximum absolute difference
Duplicate source identities	0 across 900 footprints
UFS scan-angle oracle	270/270 FOR/FOV values
Scan-angle RMS difference	0.272 µdeg
Scan-angle maximum absolute difference	3.815 µdeg

The scan-angle oracle is
diag_cris-fsr_n20_ges.2024010100_control.nc4. Comparison uses the complete
set of 270 unique (Scan_Position, Scan_Angle) values, sorted within each FOR;
there is no spatial or nearest-neighbor matching.

Visual Validation

Time

Scan Angle

# Evidence Boundary

This PR proves the metadata and API behavior it changes: IET-to-UTC time,
microsecond preservation, footprint identity, source-order projection, and
nominal scan geometry. The isolated old/new time replay proves that the time
fix does not alter sampled brightness temperatures; focused tests separately
pin projection values and ordering. It does not claim full native-to-NCEP
CrIS radiance parity, active-channel parity, QC, cloud scoring, or thinning.
Those require separate aggregate-source and selection references.

Tests

• 28 passed, 7 opt-in slow/network deselected
• source-order projection on apodized and unapodized paths
• duplicate-task collapse and cross-granule identity preservation
• Ruff check and format
• four real granules plus independent UFS diagnostic geometry
• evidence contract distinguishes unit, real-file, and external-oracle gates

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.
• The CHANGELOG.md is up to date with these changes.
• An issue is linked to this pull request.
• Assess and address Greptile feedback (AI code review bot for guidance; use discretion, addressing all feedback is not required).

Dependencies

[greptile-apps]

Greptile Summary

This PR fixes multiple metadata errors in the JPSS_CRIS CrIS FSR data source: incorrect IET-to-UTC time conversion (off by exactly 37 s on all tested granules), satellite zenith angle mistakenly published as scan_angle, millisecond time truncation losing sub-millisecond precision, and rounded geolocation-based deduplication that could collapse distinct adjacent FOVs.

• Time conversion: replaced the naive IET-epoch-offset approach with an anchor-based conversion using the paired Beginning_Date/Beginning_Time/N_Beginning_Time_IET granule attributes, preserving microsecond fidelity through to the Arrow timestamp[ns] output column.
• Scan geometry: computes and publishes the nominal CrIS look angle (FOR center + rotating 3×3 FOV offset) as scan_angle, moves the original SatelliteZenithAngle to satellite_za, and adds one-based scan_line, field_of_regard, and field_of_view identity columns to the output schema.
• Channel projection (sensor_indices): optional constructor argument that projects after full-band Hamming apodization but before Planck inversion, with exact source-order output; deduplication now keys on (source_uri, satellite, variable, scan_line, FOR, FOV) rather than rounded lat/lon/time.

Confidence Score: 4/5

The changes are a well-validated correctness fix — the old code was demonstrably wrong (37 s time error, wrong angle semantics) — but the schema additions and scan_angle semantic change are meaningful for downstream consumers to verify.

The implementation is thorough: four real NOAA granules validate the time fix, 270 UFS oracle pairs validate the scan-angle formula to sub-arcsecond precision, and mock-based unit tests cover footprint identity, source-order projection, cross-granule dedup, and invalid-argument rejection. No logic errors were found in the IET-to-UTC anchor conversion, the CrIS look-angle geometry, the deduplication keying, or the channel-projection ordering. The main consideration is that scan_angle values will differ from the previous release (satellite zenith replaced by nominal look angle), and the three new schema columns need to be consumed correctly by any downstream pipeline that ingests CRIS data.

No files require special attention beyond standard review; earth2studio/data/jpss_cris.py carries the complete change surface and its test coverage in test/data/test_jpss_cris.py is comprehensive.

Important Files Changed

Filename	Overview
earth2studio/data/jpss_cris.py	Core logic overhaul: IET→UTC anchor-based time conversion, nominal scan-angle geometry, new footprint-identity fields, sensor_indices projection, and exact-identity deduplication — all well-structured with no identified logic errors.
test/data/test_jpss_cris.py	Adds unit tests for IET conversion, scan-angle geometry, footprint identity propagation, source-order projection on both spectral paths, cross-granule dedup non-aliasing, and invalid sensor_indices rejection; mock HDF5 fixtures updated to carry correct anchor attributes.

_{Reviews (1): Last reviewed commit: "refactor: finalize JPSS CrIS metadata ha..." | Re-trigger Greptile}

[aayushg55]

Supporting Radiometric Invariance

Radiometry is unchanged by this PR, but the existing native value path was
checked independently through the NCEP product stages:

Boundary	Compared values	Result
Native Hamming-apodized SDR -> NCEP C0431 SRAD	3,491,100	2.89e-5 mW m^-2 sr^-1 cm RMS; BUFR quantization floor
Aggregate C0431 -> NNJA-AI, January/July	19,171,818 finite radiances	3.63e-18 / 4.92e-18 W m^-2 sr^-1 cm RMS; exact masks and 50,688/50,688 footprints
Aggregate C0431 + replay SpcCoeff -> UFS ncdiag BT	15,000	5.64e-6 K RMS, 1.53e-5 K maximum
Earth2Studio fixed-grid Planck -> UFS ncdiag BT	15,000	2.67e-4 K RMS, 3.14e-4 K maximum

The NNJA-AI comparison isolates two intentional representation differences:
NNJA-AI stores whole-second timestamps, dropping up to 0.984 s of source
fractional time, and its message numbering satisfies
source_message_index = MSG_IDX + 4. Geometry and quality metadata otherwise
agree exactly or at binary roundoff.

The replay CrIS SpcCoeff uses the same channel-center wavenumber grid and
identity band coefficients (Band_C1=0, Band_C2=1). Its small improvement
comes from pinned Planck constants. The file is static experiment
configuration, not cycle-varying state.