Add missing referenced_data_file field to v2 manifests

[kevinjqliu]

The Iceberg spec allows v2 position-delete files to use referenced_data_file (field ID 143) when all deletes reference a single data file. PyIceberg currently includes this field in its v3 data-file schema, but not v2.

This PR adds the optional string field to the v2 schema, updates the existing Avro and REST manifest comparisons to retain it, and adds a focused non-null serialization test. The Avro tests explicitly declare the v3 in-memory record schema while writing v2 files. No production record-layout or helper changes are included.

Related: #3690 introduced the v3 manifest read projection.

[Copilot]

🟡 Changes recommended

The tests do not verify a non-null value and conceal a positional writer mismatch that drops it.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

🟢 Approval recommended

The schema change matches the specification and is covered by focused serialization and integration assertions.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[geruh]

lgtm

[kevinjqliu]

We need to declare the record schema as v3 here, even though the write schema is v2. Otherwise, referenced_data_file gets written as null. 😭

this is because DataFile.from_args() defaults to v3, but the writer uses position-based indexes unless we provide the record schema.

In V2 schema, referenced_data_file should be index 16 because first_row_id doesnt exist in V2.
In V3 schema, referenced_data_file is index 17.

Without record_schema, the writer reads index 16 (first_row_id) instead of 17. Declaring v3 lets the existing field-ID projection handle this mapping.

The getter also assumes index 17 and I dont want to add an if branch to the getter.

iceberg-python/pyiceberg/manifest.py

Lines 539 to 541 in 562d3af

	@property
	def referenced_data_file(self) -> str | None:
	return self._data[17]

This is an existing problem so I think the best way to resolve it is to have records retain their schema and use field IDs for access. On read, we should infer the schema from the file. On write, we should specify the version and let manifest IO handle the conversion.

We can do this as a follow up

[kevinjqliu]

Tracking in #3957