Feature Request / Improvement
AvroFileHeader.get_schema() (pyiceberg/avro/file.py) runs the full avro_to_iceberg conversion every time an
Avro file is opened.
The inefficiency
- Every manifest under a spec embeds an identical Avro schema string.
- So during scan planning, that same conversion is repeated once per manifest.
- The cost grows with the manifest count, even though only a couple of distinct schemas are ever involved.
Proposed fix
- The conversion depends only on the schema string, so the result can be cached (keyed on that string).
Measured impact
scan().plan_files() on an unpartitioned 150-manifest table: ~86 ms → ~48 ms (~1.8× faster).
- The saving grows as the number of manifests increases.
I am willing to contribute for this improvement.
Feature Request / Improvement
AvroFileHeader.get_schema()(pyiceberg/avro/file.py) runs the fullavro_to_icebergconversion every time anAvro file is opened.
The inefficiency
Proposed fix
Measured impact
scan().plan_files()on an unpartitioned 150-manifest table: ~86 ms → ~48 ms (~1.8× faster).I am willing to contribute for this improvement.