Skip to content

[Python][Parquet] Cannot read parquet with duplicate column names #37428

Closed as duplicate

Description

@alippai

Describe the bug, including details regarding any error messages, version, and platform.

If use_legacy_dataset=False in use_legacy_dataset then Exception is raised.
If we specify the schema, the read succeeds, but the values of the duplicate columns are null.

Using use_legacy_dataset=True there are no issues reading the single file (but we can only rename_columns after reading the whole file).

Alternatively the columns param of read_table should allow reading using index, not just column names.

Component(s)

Parquet, Python

Activity

  1. mapleFU commented on Aug 29, 2023

    @mapleFU
    Member

    Can you print the schema for Parquet file? Or just upload the file that I can debug?

  2. alippai commented on Aug 29, 2023

    @alippai
    ContributorAuthor
    import pyarrow as pa
    import pyarrow.parquet as pq
    
    t = pa.Table.from_arrays([[1,2],[3,4]], names=['colA', 'colA']) 
    pq.write_table(t,'/tmp/a.parquet', store_schema=False)
    pq.read_table('/tmp/a.parquet', use_legacy_dataset=True) # success

    vs

    import pyarrow as pa
    import pyarrow.parquet as pq
    
    t = pa.Table.from_arrays([[1,2],[3,4]], names=['colA', 'colA']) 
    pq.write_table(t,'/tmp/a.parquet', store_schema=False)
    pq.read_table('/tmp/a.parquet') # exception

    And

    import pyarrow as pa
    import pyarrow.parquet as pq
    
    t = pa.Table.from_arrays([[1,2],[3,4]], names=['colA', 'colA']) 
    pq.write_table(t,'/tmp/a.parquet', store_schema=False)
    t2 = pq.read_table('/tmp/a.parquet', schema=pa.schema([pa.field('col1', pa.int64()),pa.field('col2', pa.int64())]))
    print(t2) # no error, all null
  3. alippai commented on Aug 29, 2023

    @alippai
    ContributorAuthor

    Not sure if this should raise, but also makes it impossible to filter on the duplicate column names:

    import pyarrow as pa
    import pyarrow.parquet as pq
    
    t = pa.Table.from_arrays([[1,2],[3,4],[5,6]], names=['colA', 'colA', 'colB']) 
    pq.write_table(t,'/tmp/a.parquet', store_schema=False)
    t2 = pq.read_table('/tmp/a.parquet', schema=pa.schema([pa.field('col1', pa.int64()),pa.field('col2', pa.int64()),pa.field('col3', pa.int64())]), filters=[('col3', '=', 5)])
    print(t2) # no error, empty result

    vs

    import pyarrow as pa
    import pyarrow.parquet as pq
    
    t = pa.Table.from_arrays([[1,2],[3,4],[5,6]], names=['colA', 'colA', 'colB']) 
    pq.write_table(t,'/tmp/a.parquet', store_schema=False)
    t2 = pq.read_table('/tmp/a.parquet', schema=pa.schema([pa.field('col1', pa.int64()),pa.field('col2', pa.int64()),pa.field('colB', pa.int64())]), filters=[('colB', '=', 5)])
    print(t2) # filter applied
  4. mapleFU commented on Aug 29, 2023

    @mapleFU
    Member

    Firstly, you'd better not using same name here. Parquet uses name or fieldId to detect a field. And arrow usally uses name. So firstly I think we'd better using name

    Then whats the expected behavior here? Maybe finally we can achieve some read result if FIELD_ID is applied. (Though still I think same name is a bad idea...)

  5. alippai commented on Aug 29, 2023

    @alippai
    ContributorAuthor

    I agree that it's an annoying edge case. I'm not generating the files, I've just created the examples above so it's easier to reproduce the regression. Maybe it's a non-issue, but maybe we want to fix the above before removing the deprecated use_legacy_dataset flag. I don't have enough knowledge on this topic.

    Naively I think using the column order indexes might be a faster and more simple approach than using field id or column name maps.

  6. changed the title [-]Cannot read parquet with duplicate column names[/-] [+][Python][Parquet] Cannot read parquet with duplicate column names[/+] on Aug 30, 2023
  7. jorisvandenbossche commented on Aug 31, 2023

    @jorisvandenbossche
    Member

    I think long term, ideally, the new dataset API based reading would also support duplicate column names. I assume it will be hard to fully support that throughout a fully query in Acero, but at least supporting it in a Scan node so you can rename them afterwards would be useful.

    One thing to note is that even if we would remove the use_legacy_dataset=True option in the near future, you can still use the single-file pq.ParquetFile(..).read() interface that does support this.
    The difference in support is between the pure Parquet reader vs the Parquet-format Dataset reader. The unfortunate aspect from a user point of view is that the most used pq.read_table function mixes both cases, and because it has historically supported reading multiple files through the legacy pq.ParquetDataset, we updated pq.read_table to read using the new dataset API. But for some cases, the single-file reader actually works better (and duplicate column names is one such example, but eg selecting fields of nested columns is another).

  8. thisisnic commented on Nov 20, 2026

    @thisisnic
    Member

    This still happens on 26.0.0: pq.read_table() on a file with duplicate column names fails with Multiple matches for FieldRef.Name(colA), while pq.ParquetFile(path).read() works. It is the same dataset limitation tracked in #24407 (see also #33552 for the scanner design question), so I'm closing this as a duplicate of that one.

    The all-null result when passing schema= isn't specific to duplicate names: the dataset reader matches the schema to the file's columns by name, so columns named col1 and col2 that aren't in the file come back as nulls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions