You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Python][Parquet] Cannot read parquet with duplicate column names #37428
Describe the bug, including details regarding any error messages, version, and platform.
If use_legacy_dataset=False in use_legacy_dataset then Exception is raised.
If we specify the schema, the read succeeds, but the values of the duplicate columns are null.
Using use_legacy_dataset=True there are no issues reading the single file (but we can only rename_columns after reading the whole file).
Alternatively the columns param of read_table should allow reading using index, not just column names.
Firstly, you'd better not using same name here. Parquet uses name or fieldId to detect a field. And arrow usally uses name. So firstly I think we'd better using name
Then whats the expected behavior here? Maybe finally we can achieve some read result if FIELD_ID is applied. (Though still I think same name is a bad idea...)
I agree that it's an annoying edge case. I'm not generating the files, I've just created the examples above so it's easier to reproduce the regression. Maybe it's a non-issue, but maybe we want to fix the above before removing the deprecated use_legacy_dataset flag. I don't have enough knowledge on this topic.
Naively I think using the column order indexes might be a faster and more simple approach than using field id or column name maps.
changed the title [-]Cannot read parquet with duplicate column names[/-][+][Python][Parquet] Cannot read parquet with duplicate column names[/+]on Aug 30, 2023
I think long term, ideally, the new dataset API based reading would also support duplicate column names. I assume it will be hard to fully support that throughout a fully query in Acero, but at least supporting it in a Scan node so you can rename them afterwards would be useful.
One thing to note is that even if we would remove the use_legacy_dataset=True option in the near future, you can still use the single-file pq.ParquetFile(..).read() interface that does support this.
The difference in support is between the pure Parquet reader vs the Parquet-format Dataset reader. The unfortunate aspect from a user point of view is that the most used pq.read_table function mixes both cases, and because it has historically supported reading multiple files through the legacy pq.ParquetDataset, we updated pq.read_table to read using the new dataset API. But for some cases, the single-file reader actually works better (and duplicate column names is one such example, but eg selecting fields of nested columns is another).
This still happens on 26.0.0: pq.read_table() on a file with duplicate column names fails with Multiple matches for FieldRef.Name(colA), while pq.ParquetFile(path).read() works. It is the same dataset limitation tracked in #24407 (see also #33552 for the scanner design question), so I'm closing this as a duplicate of that one.
The all-null result when passing schema= isn't specific to duplicate names: the dataset reader matches the schema to the file's columns by name, so columns named col1 and col2 that aren't in the file come back as nulls.
Activity
mapleFU commentedon Aug 29, 2023
Can you print the schema for Parquet file? Or just upload the file that I can debug?
alippai commentedon Aug 29, 2023
vs
And
alippai commentedon Aug 29, 2023
Not sure if this should raise, but also makes it impossible to filter on the duplicate column names:
vs
mapleFU commentedon Aug 29, 2023
Firstly, you'd better not using same name here. Parquet uses name or fieldId to detect a field. And arrow usally uses name. So firstly I think we'd better using name
Then whats the expected behavior here? Maybe finally we can achieve some read result if FIELD_ID is applied. (Though still I think same name is a bad idea...)
alippai commentedon Aug 29, 2023
I agree that it's an annoying edge case. I'm not generating the files, I've just created the examples above so it's easier to reproduce the regression. Maybe it's a non-issue, but maybe we want to fix the above before removing the deprecated
use_legacy_datasetflag. I don't have enough knowledge on this topic.Naively I think using the column order indexes might be a faster and more simple approach than using field id or column name maps.
[-]Cannot read parquet with duplicate column names[/-][+][Python][Parquet] Cannot read parquet with duplicate column names[/+]jorisvandenbossche commentedon Aug 31, 2023
I think long term, ideally, the new dataset API based reading would also support duplicate column names. I assume it will be hard to fully support that throughout a fully query in Acero, but at least supporting it in a Scan node so you can rename them afterwards would be useful.
One thing to note is that even if we would remove the
use_legacy_dataset=Trueoption in the near future, you can still use the single-filepq.ParquetFile(..).read()interface that does support this.The difference in support is between the pure Parquet reader vs the Parquet-format Dataset reader. The unfortunate aspect from a user point of view is that the most used
pq.read_tablefunction mixes both cases, and because it has historically supported reading multiple files through the legacy pq.ParquetDataset, we updatedpq.read_tableto read using the new dataset API. But for some cases, the single-file reader actually works better (and duplicate column names is one such example, but eg selecting fields of nested columns is another).thisisnic commentedon Nov 20, 2026
This still happens on 26.0.0:
pq.read_table()on a file with duplicate column names fails withMultiple matches for FieldRef.Name(colA), whilepq.ParquetFile(path).read()works. It is the same dataset limitation tracked in #24407 (see also #33552 for the scanner design question), so I'm closing this as a duplicate of that one.The all-null result when passing
schema=isn't specific to duplicate names: the dataset reader matches the schema to the file's columns by name, so columns namedcol1andcol2that aren't in the file come back as nulls.