Papers Offline

This multi-source scraper downloads content from Project Gutenberg, the Open Textbook Library, Wikisource, USAID DEC Archive, Bibliothèque numérique romande and Littérature jeunesse libre and packages it into a ZIM file, a clean and user-friendly format for storing content for offline usage.

The ZIM file includes a modern, responsive Vue.js interface with features like:

• Browse books by title, author, or source-specific categories such as Library of Congress Classification (LCC) shelves or Open Textbook Library subjects
• Advanced filtering by language, format, and more
• Built-in EPUB and PDF readers
• Full-text search across all content
• Multilingual support with automatic language detection
• Responsive design that works on desktop and mobile devices
• No-JavaScript fallback for maximum compatibility

Note

This scraper used to live at https://github.com/openzim/gutenberg ; it has been renamed in Sept 2026

Getting Started

The recommended way to use the scraper is with Docker, which includes all dependencies pre-installed.

Using Docker (Recommended)

Run the scraper:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers papers2zim

The -v $(pwd)/output:/output option mounts your local output folder to save the ZIM file.

Note: On Windows PowerShell, replace $(pwd) with ${PWD}. Alternatively, use the full path: -v C:\Users\YourName\output:/output

View available options:

docker run ghcr.io/openzim/papers papers2zim --help

Example with custom options:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim -l en,fr -f pdf --books 100-200 --lcc-shelves all

Using PyPI

Alternatively, install from PyPI:

pip install papers2zim
papers2zim --help

Note: You'll need to install system dependencies (zim-tools) separately. See CONTRIBUTING.md for details.

Scraping Sources

The scraper supports several sources, select a source with --source:

--source=<source> Source slug or short name: gutenberg (PG), onemorelibrary (OML), opentextbooks (OTL), wikisource (WS), usaiddec (USAID), bibliothequenumeriqueromande (BNR) or litteraturejeunesselibre (LJL) Default: gutenberg

Project Gutenberg

Select Project Gutenberg with --source=gutenberg (or --source=PG).

Gutenberg's catalog is not limited to books: entries typed something else than Text — audio recordings, images, datasets — are skipped when the catalog is loaded, since there is no text to export for them.

For example:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=gutenberg -l en,fr -f pdf --books=100-200 --lcc-shelves=all

The Project Gutenberg-specific option is:

--lcc-shelves=<shelves>    LCC shelf codes (comma-separated or 'all')

For example:

papers2zim --source=gutenberg --lcc-shelves=P,PR,Q

One More Library

Select One More Library with --source=onemorelibrary (or --source=OML).

Books are public-domain works with a per-book Creative Commons license; EPUB and PDF editions are downloaded and made available to the built-in readers.

To scrape one or more categories (given by name or by slug):

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=onemorelibrary --categories=Art,Science

To select specific books by ID:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=onemorelibrary --oml-ids=<id1>,<id2>,<id3>

The One More Library-specific options are:

--categories=<categories>  Comma-separated One More Library categories
                           (names or slugs)
--oml-ids=<ids>            Exact One More Library book IDs
--list-categories          List One More Library categories and exit
--refresh-oml-catalog      Refresh the One More Library catalog and exit

--oml-ids and --books cannot be used together. --refresh-oml-catalog requires --cache-dir. Selecting a whole library crawls the entire catalog; use --cache-dir to cache it across runs.

USAID DEC Archive

Select the USAID DEC Archive with --source=usaiddec (or --source=USAID).

The source packages USAID's Development Experience Clearinghouse evaluation reports, preserved and published by AidData with permission, from the AidData dataset catalog.

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=usaiddec -f pdf

To select specific reports by Formatted Document ID:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=usaiddec --usaid-ids=PA00JMRT,PA00ZH1K

The USAID DEC Archive-specific options are:

--usaid-ids=<ids>          Exact USAID DEC report IDs (Formatted Document IDs)
--refresh-usaid-catalog    Refresh the USAID DEC dataset catalog and exit

--usaid-ids and --books cannot be used together. --refresh-usaid-catalog requires --cache-dir.

Open Textbook Library

Select the Open Textbook Library with --source=opentextbooks (or --source=OTL).

To scrape books from one or more subjects:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=opentextbooks --subjects=Business,Mathematics

To select specific OTL records:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=opentextbooks --otl-ids=<id1>,<id2>,<id3>

The Open Textbook Library-specific options are:

--subjects=<subjects>      Comma-separated Open Textbook Library subjects
--otl-ids=<ids>            Exact Open Textbook Library record IDs
--list-subjects            List Open Textbook Library subjects and exit
--refresh-catalog          Refresh the Open Textbook Library CSV catalog and exit

--otl-ids and --books cannot be used together.

Wikisource

Warning

Wikisource support is experimental at this stage, there are known issues.

Select Wikisource with --source=wikisource (or --source=WS).

Wikisource is organised per language, so --languages is required: the scraper reads the "ready for export" OPDS feeds published by ws-export for each requested language and downloads each book as an EPUB. The supported language codes are the ones listed on the ws-export OPDS index; pass them to --languages.

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=wikisource -l en,fr

To select specific books, pass their page names as they appear in the Wikisource URL (https://en.wikisource.org/wiki/The_Time_Machine → The_Time_Machine):

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=wikisource -l en --wikisource-pages=The_Time_Machine

Spaces and underscores are interchangeable, and percent-encoded names are accepted. A comma inside a page name must be written as %2C. Only books in the ws-export feed of the requested languages can be selected. --books also works but selects by position in the feed, which changes from day to day; --wikisource-pages and --books cannot be used together.

ws-export publishes one feed per language and names it in that language (Ready_for_export.xml on en, Bon_pour_export.xml on fr), so the scraper reads the single feed it finds in the language directory. Should a directory ever advertise several, the scraper cannot tell which one holds the ready-for-export books from their localized names, and reading them all would silently mix in unrelated works, so it stops with an error listing them. --wikisource-feed then picks the feed to read, by file name or by URL. It only disambiguates: a feed name belonging to one language does not select another language's feed, so each language keeps using the feed advertised under it.

Bibliothèque numérique romande

Select the Bibliothèque numérique romande with --source=bibliothequenumeriqueromande (or --source=BNR).

BNR publishes its French-language catalogue as OPDS feeds and offers EPUB only. A scrape selects at most one filter family at a time: series, authors, categories, or title prefixes (letters or numbers). Categories are the broad collections derived from BNR tags, and the available filter values can be listed with the --list-bnr-* options.

For example:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=bibliothequenumeriqueromande --bnr-categories=Conte

To select specific books by UUID:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=bibliothequenumeriqueromande --bnr-ids=<uuid1>,<uuid2>

The Bibliothèque numérique romande-specific options are:

--bnr-series=<series>          Comma-separated BNR series names
--bnr-authors=<authors>        Comma-separated BNR author names
--bnr-categories=<categories>  Comma-separated BNR category names
--bnr-titles=<titles>          Comma-separated BNR title prefixes
                               (letters or numbers)
--bnr-ids=<ids>                Comma-separated BNR book UUIDs
--list-bnr-series              List BNR series and exit
--list-bnr-authors             List BNR authors and exit
--list-bnr-categories          List BNR categories and exit
--list-bnr-titles              List BNR title prefixes and exit

Only one of --bnr-series, --bnr-authors, --bnr-categories and --bnr-titles can be used at a time. --books is not supported.

Littérature jeunesse libre

Select Littérature jeunesse libre with --source=litteraturejeunesselibre (or --source=LJL).

LJL publishes its French-language children's catalogue as OPDS feeds and offers EPUB and PDF. A scrape selects at most one filter family at a time: series, authors, categories, or title prefixes (letters or numbers). Categories are the broad collections derived from LJL tags, and the available filter values can be listed with the --list-ljl-* options.

For example:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=litteraturejeunesselibre --ljl-categories=Animaux

To select specific books by UUID:

docker run -v $(pwd)/output:/output ghcr.io/openzim/papers \
  papers2zim --source=litteraturejeunesselibre --ljl-ids=<uuid1>,<uuid2>

The Littérature jeunesse libre-specific options are:

--ljl-series=<series>          Comma-separated LJL series names
--ljl-authors=<authors>        Comma-separated LJL author names
--ljl-categories=<categories>  Comma-separated LJL category names
--ljl-titles=<titles>          Comma-separated LJL title prefixes
                               (letters or numbers)
--ljl-ids=<ids>                Comma-separated LJL book UUIDs
--list-ljl-series              List LJL series and exit
--list-ljl-authors             List LJL authors and exit
--list-ljl-categories          List LJL categories and exit
--list-ljl-titles              List LJL title prefixes and exit

Only one of --ljl-series, --ljl-authors, --ljl-categories and --ljl-titles can be used at a time. --books is not supported.

Command-Line Options

Source Selection

--source=<source>                  Source slug or short name:
                                   gutenberg (PG), onemorelibrary (OML),
                                   opentextbooks (OTL), wikisource (WS),
                                   usaiddec (USAID), bibliothequenumeriqueromande
                                   (BNR) or litteraturejeunesselibre (LJL)
                                   Default: gutenberg

Common Options

-h --help                          Display this help message
--overwrite                        Overwrite existing ZIM file

-l --languages=<list>              Comma-separated language codes (ISO 639-1 or ISO 639-3)
-f --formats=<list>                Comma-separated formats (epub, html, pdf, all)

-z --zim-file=<file>               ZIM file output path
--zim-name=<name>                  ZIM name (metadata)
-t --zim-title=<title>             ZIM title
-n --zim-desc=<description>        ZIM description
-L --zim-long-desc=<description>   ZIM long description
--zim-languages=<languages>        ZIM language metadata

-b --books=<ids>                   Source catalog positions/IDs
-c --concurrency=<nb>              Number of concurrent workers (default: 16)

--no-index                          Skip full-text index creation
--index-format=<format>             Format to index book text from (epub, html, pdf;
                                    one of --formats; default: html, then epub,
                                    then pdf)
--title-search                      Enable title-based search
--stats-filename=<filename>        Statistics output file

--publisher=<publisher>            Custom publisher name
--mirror-url=<mirror_url>          Custom source mirror URL
--output=<output_folder>            Output folder
--cache-dir=<cache_folder>          Optional persistent metadata and catalog cache

--ui-dist=<ui_dist>                 Built UI distribution directory
--debug                             Enable verbose output

Project Gutenberg Options

--lcc-shelves=<shelves>             LCC shelf codes (comma-separated or 'all')

Open Textbook Library Options

--subjects=<subjects>               Comma-separated Open Textbook Library subjects
--otl-ids=<ids>                     Exact Open Textbook Library record IDs
--list-subjects                     List Open Textbook Library subjects and exit
--refresh-catalog                   Refresh the Open Textbook Library CSV catalog and exit

Wikisource Options

--wikisource-pages=<pages>          Wikisource page names, as in the page URL
--wikisource-feed=<feed>            OPDS feed to use when a language has several

USAID DEC Archive Options

--usaid-ids=<ids>                   Exact USAID DEC report IDs (Formatted Document IDs)
--refresh-usaid-catalog             Refresh the USAID DEC dataset catalog and exit

Bibliothèque numérique romande Options

--bnr-series=<series>               Comma-separated BNR series names
--bnr-authors=<authors>             Comma-separated BNR author names
--bnr-categories=<categories>       Comma-separated BNR category names
--bnr-titles=<titles>               Comma-separated BNR title prefixes
                                    (letters or numbers)
--bnr-ids=<ids>                     Comma-separated BNR book UUIDs
--list-bnr-series                   List BNR series and exit
--list-bnr-authors                  List BNR authors and exit
--list-bnr-categories               List BNR categories and exit
--list-bnr-titles                   List BNR title prefixes and exit

Littérature jeunesse libre Options

--ljl-series=<series>               Comma-separated LJL series names
--ljl-authors=<authors>             Comma-separated LJL author names
--ljl-categories=<categories>       Comma-separated LJL category names
--ljl-titles=<titles>               Comma-separated LJL title prefixes
                                    (letters or numbers)
--ljl-ids=<ids>                     Comma-separated LJL book UUIDs
--list-ljl-series                   List LJL series and exit
--list-ljl-authors                  List LJL authors and exit
--list-ljl-categories               List LJL categories and exit
--list-ljl-titles                   List LJL title prefixes and exit

Caching

Caching is opt-in. Pass --cache-dir to persist source metadata and catalog data. To reuse cached data in a later run, pass the same directory again; without --cache-dir, the scraper performs a fresh run and does not create a persistent cache.

Features

User Interface

• Modern Web Interface: Fast, responsive single-page application with smooth navigation
• Multiple View Modes: Switch between grid and list views for books
• Responsive Design: Optimized for desktop, tablet, and mobile devices
• Dark/Light Theme: Automatic theme switching based on system preferences
• Customizable Colors: Configure primary and secondary brand colors

Content Organization

• Browse by Books: View all books with cover images, titles, and authors
• Browse by Authors: Explore authors with their complete bibliographies
• Source-specific Categories: Browse Project Gutenberg books by LCC shelves or Open Textbook Library books by subjects
• Smart Pagination: Efficient navigation through large collections

Search & Discovery

• Full-Text Search: Search across all books, authors, and source-specific categories
• Quick Filters: Find authors by name or source-specific category
• Rich Search Results: Search results include descriptions and metadata

Filtering & Sorting

• Language Filter: Filter books by language
• Format Filter: Filter by available formats
• Sort Options: Sort by popularity (where available) or title
• Sort Order: Toggle between ascending and descending order

Book Details

• Comprehensive Metadata: Title, subtitle, author, description, languages, license
• Author Information: Author name with birth/death years where available
• Popularity Rating: Star rating based on download statistics where available
• Download Counts: Formatted download statistics where available
• Source-specific Metadata: Display source-specific information such as LCC classification
• Multiple Formats: Download books in available formats
• Cover Images: High-quality book cover images where available

Internationalization

• Multiple Languages: Full UI translations for many languages
• Automatic Detection: Detects browser language and sets UI accordingly
• Language Switcher: Easy language selection from header menu
• RTL Support: Right-to-left layout support for Arabic, Hebrew, etc.

Accessibility

• No-JavaScript Fallback: Complete HTML-only version for browsers without JavaScript
• Semantic HTML: Proper heading hierarchy and ARIA labels
• Keyboard Navigation: Full keyboard accessibility
• Screen Reader Support: ARIA labels and descriptions throughout
• High Contrast: Readable text with proper color contrast ratios

Technical Features

• ZIM Format: Compressed, indexed format for offline usage
• Full-Text Indexing: Optional full-text search index within ZIM
• Concurrent Processing: Multi-threaded book processing for faster scraping
• Custom Mirrors: Support for custom source mirror URLs
• Docker Support: Pre-built Docker images with all dependencies

Architecture

The scraper separates source-specific logic from the shared core engine and the web UI.

Sources

Source implementations live under scraper/src/papers2zim/sources/.

Currently supported sources are:

• Project Gutenberg (gutenberg / PG)
• One More Library (onemorelibrary / OML)
• Open Textbook Library (opentextbooks / OTL)
• Bibliothèque numérique romande (bibliothequenumeriqueromande / BNR)
• Littérature jeunesse libre (litteraturejeunesselibre / LJL)

The source registry connects each source implementation to the common pipeline.

Core Engine

The scraper/src/papers2zim/core/ package is source-independent. It defines common interfaces (ports) and shared processing for catalog access, metadata, format resolution, rewriting, exporting, and indexing.

The core engine therefore does not need to contain source-specific scraping logic.

Web UI

The Vue.js UI is maintained separately from the scraper sources. The scraper builds the UI and includes the resulting distribution in the generated ZIM.

This separation means that source implementations, the shared scraping/ZIM pipeline, and the presentation layer can evolve independently.

Contributing

We welcome contributions! Whether you want to:

• Add or improve UI translations
• Fix bugs or add features
• Improve documentation
• Develop the Vue.js interface

Please see CONTRIBUTING.md for detailed guidelines on setting up the development environment, code style, testing, and the pull request process.

Main coding guidelines follow the openZIM Wiki.

Screenshots

Project Gutenberg (PG) — Dark Mode	Open Textbook Library (OTL) — Light Mode
Home Page	Home Page
Book Page	Book Page
LCC Shelves	Subjects

Mobile View

Dark Mode	Light Mode

License

GPLv3 or later, see LICENSE for more details.