Skip to content

[Python] Pandas dataframe.loc[bool_array] causes "offset overflow while concatenating arrays" #41890

Closed as duplicate

Description

@stevendavis

Describe the bug, including details regarding any error messages, version, and platform.

Tested with :

  • pyarrow 11, pandas 1.5
  • pyarrow 16, pandas 2.0

This Python code reproduces the problem:

import random
import pandas as pd

NUM_ROWS = 1_000_000
MAX_WORDS_PER_SENTENCE = 1000
NUM_PRE_DEFINED_SENTENCES = 1000

WORDS = """
story where fish and children are alarmed when thing one and thing two
and the cat in the big red hat wreck the house while mother is away
""".split()

def sentence(num_words):
    return " ".join(random.choices(WORDS, k=num_words))

PRE_DEFINED_SENTENCES = [sentence(random.randint(1, MAX_WORDS_PER_SENTENCE)) for s in range(NUM_PRE_DEFINED_SENTENCES)]

text_list = random.choices(PRE_DEFINED_SENTENCES, k=NUM_ROWS)

df = pd.DataFrame({"text": text_list})
df["len"] = df.text.str.len()
print(df)
print("\nMemory usage:")
print(df.memory_usage(deep=True))

df.text = df.text.astype("string[pyarrow]")

contains_fish = df.text.str.contains("fish")

df_fish = df.loc[contains_fish]

This is the resulting stack trace:

Traceback (most recent call last):
  File "test_arrow.py", line 30, in <module>
    df_fish = df.loc[contains_fish]
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/indexing.py", line 1292, in _getitem_axis
    return self._getbool_axis(key, axis=axis)
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/indexing.py", line 1093, in _getbool_axis
    return self.obj._take_with_is_copy(inds, axis=axis)
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/generic.py", line 3902, in _take_with_is_copy
    result = self._take(indices=indices, axis=axis)
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/generic.py", line 3886, in _take
    new_data = self._mgr.take(
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 978, in take
    return self.reindex_indexer(
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 751, in reindex_indexer
    new_blocks = [
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 752, in <listcomp>
    blk.take_nd(
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1775, in take_nd
    new_values = self.values.take(indexer, fill_value=fill_value, allow_fill=True)
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pandas/core/arrays/arrow/array.py", line 725, in take
    return type(self)(self._data.take(indices))
  File "pyarrow/table.pxi", line 1001, in pyarrow.lib.ChunkedArray.take
  File "/app/conda_envs/myvenv/lib/python3.8/site-packages/pyarrow/compute.py", line 473, in take
    return call_function('take', [data, indices], options, memory_pool)
  File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

Component(s)

Python

Activity

  1. changed the title [-]Pandas dataframe.loc[bool_array] causes "offset overflow while concatenating arrays"[/-] [+][Python] Pandas dataframe.loc[bool_array] causes "offset overflow while concatenating arrays"[/+] on May 30, 2024
  2. thisisnic commented on Nov 20, 2026

    @thisisnic
    Member

    The reprex above no longer fails with current pandas: since pandas 2.2.0 the string[pyarrow] dtype is stored as large_string, so the offsets can't overflow.

    The underlying Arrow call does still fail on 26.0.0:

    import numpy as np
    import pyarrow as pa
    
    chunk = pa.array(["x" * 1000] * 250_000)
    ca = pa.chunked_array([chunk] * 11)  # 2.57 GiB of string data
    ca.take(pa.array([0, len(ca) - 1]))
    # ArrowInvalid: offset overflow while concatenating arrays, consider casting input from `string` to `large_string` first.

    take on a chunked array concatenates the chunks into a single array first, which overflows the 32-bit offsets of string once there is more than 2 GiB of data. That is tracked in #25822, so I'm closing this as a duplicate of that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions