GH-184: Harden Arrow C Data Interface validation#606
Draft
samtalki wants to merge 5 commits into
Draft
Conversation
samtalki
force-pushed
the
agent/cdata-hardening
branch
2 times, most recently
from
bffab43 to
2b4a826
Compare
samtalki
force-pushed
the
agent/cdata-hardening
branch
from
2b4a826 to
5a4a7ea
Compare
Add immutable ArrowSchema and ArrowArray ABI structs plus C Data import for null and primitive arrays. Validate pointer layout, child counts, buffer counts, offsets, known flag bits, dictionary pointer consistency, release idempotency, and reads after release. Trust declared null counts like Arrow C++ and arrow-rs, require a validity bitmap for unknown null counts like nanoarrow, and ignore reserved flag bits for forward compatibility. Move the base ArrowSchema and ArrowArray structures into Julia owned storage at import and mark the sources released without calling their callbacks, following the C Data Interface move semantics used by arrow-rs from_raw and nanoarrow ArrowArrayMove. Callers may free or reuse the passed structures once from_c_data returns; release_c_data or finalization releases the moved copies through the producer callbacks. Keep imported buffers rooted behind a shared release owner. Copy misaligned fixed width buffers into aligned Julia storage before typed access, and materialize Julia owned data from copy, collect, deepcopy, and serialize so no read path bypasses the release liveness gate. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add C Data import for struct arrays on top of the null and primitive array importer. Root structs are exposed as Tables.jl column tables with DataAPI metadata; nested structs import as NamedTuple element vectors with parent validity masking and parent offsets applied to the children. Parse field names and schema metadata from the C schema for the imported table. Validate child pointer layout, child counts against the schema, struct child lengths, name and metadata bounds, and nesting depth, bound validation with a total node budget so aliased or cyclic child pointers cannot make it explode combinatorially, and reject duplicate struct field names, which NamedTuple rows and column lookup cannot represent. Adjust dictionary encoding offsets for non 1-based reference pools so categorical pools with missing values keep correct dictionary indices. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add import support for boolean, string, binary, list, fixed size, temporal, timestamp, duration, interval, and decimal C Data formats. Extend malformed input, offset, null count, release, copy, collect, metadata, and GC rooting coverage for the broader importer. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add array and table export entry points with independent schema and array owners. Release children that were already exported when a later child fails to build, so partially constructed exports do not leak owner registry entries. Cover primitive and struct round trips, release order, moved child ownership, GC, metadata, malformed layout, unsupported export cases, and exported buffer layout checks for nested, string, and fixed size arrays. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
Add deterministic malformed import fuzzing, nested malformed input and export layout checks, a C producer smoke test, repeated GC stress, and an optional PyArrow C Data smoke test with skips when the external dependency is unavailable. Co-authored-by: Robert Buessow <robert.buessow@relational.ai> Co-authored-by: Olle Martensson <olle.martensson@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Generated-by: OpenAI Codex
samtalki
force-pushed
the
agent/cdata-hardening
branch
from
5a4a7ea to
5411397
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #184
This PR builds on #605 and adds hardening coverage for the C Data Interface work.
Stack
Base remains
mainbecause these cross fork PRs are not represented as true stacked bases on GitHub.Depends on #605, currently
c8d7f55.Incremental review link:
samtalki/arrow-julia@agent/cdata-export...agent/cdata-hardening
Size
Incremental over #605: 2 files changed, 310 insertions, 0 deletions.
Total against
main: 6 files changed, 4080 insertions, 4 deletions.Validation
git diff --check agent/cdata-export...HEAD: passed.AI Assistance
Co-authored-by: Robert Buessow robert.buessow@relational.ai
Co-authored-by: Olle Martensson olle.martensson@gmail.com
Generated-by: OpenAI Codex