Skip to content

Not able to import pyarrow in aws lambda function. #39465

Closed as not planned
Closed as not planned

Description

@JainamRaval

I am trying to create a lambda function in aws, which requires pyarrow as a dependency. i have installed pyarrow along with other dependency and created a zip folder in my local machine. after that i have created layer on aws with that zip folder. but when i am trying to run the lambda function, it is giving me this below error.

{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'pyarrow.lib'",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "4eb1a49d-751f-4e11-ad1e-2fd75319ccfb",
  "stackTrace": []
}

Import statements in my lambda function looks like this,

import boto3
import pyarrow as pa
import pyarrow.parquet as pq
import redis

can anyone help me on this?? although i am able to import other dependencies like redis but not pyarrow.

Component(s)

Parquet

Activity

  1. WillAyd commented on Jan 9, 2025

    @WillAyd
    Contributor

    Are you uploading the PyArrow source or a pre-compiled version to lambda?

    FWIW AWS distributes a Lambda Layer that already has pyarrow in it. Any reason why you don't want to use that?

    https://aws-sdk-pandas.readthedocs.io/en/stable/layers.html

  2. dixithak commented on Jun 7, 2025

    @dixithak

    @JainamRaval were you able to resolve this? I tried this via cdk and docker, but didn't work

  3. thisisnic commented on Nov 20, 2026

    @thisisnic
    Member

    No module named 'pyarrow.lib' means the compiled part of pyarrow in the layer could not be loaded, which usually happens when the zip was built for a different OS, architecture or Python version than the Lambda runtime. I get the same error locally by importing a pyarrow wheel built for Python 3.11 from Python 3.12. Building the layer for the same platform and Python version as the function, or using the AWS SDK for pandas layer mentioned above, should resolve it.

    As this is a packaging question rather than a bug in Arrow and there has been no follow-up, I'm closing this; the user mailing list or GitHub Discussions are the best place for further help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions