[Mohamed Emad (TheGeekyM)]

Summary

PdfConverter.convert could run pdfminer.high_level.extract_text twice on the same bytes in one conversion. This happened whenever the first pdfminer pass returned empty text. This PR records whether pdfminer has already run, and only uses the final "still empty" fallback when it has not. That is the case where pdfplumber found form-style pages but got no text from them.

Motivation

convert (packages/markitdown/src/markitdown/converters/_pdf_converter.py) has three places that call pdfminer:

1. When form_page_count == 0, it runs pdfminer on the whole document, which gives better spacing for prose.
2. When pdfplumber raises, the except branch runs pdfminer.
3. At the end, if not markdown: runs pdfminer again.

The last check did not know whether pdfminer had already run. If step 1 or step 2 returned an empty string, step 3 parsed the whole document again with the same library and the same input. pdfminer is deterministic, so the second pass always returned the same empty result. The only effect was a second full parse.

The most common case is a scanned or image-only PDF. pdfplumber finds no form pages, so step 1 runs pdfminer, gets no text, and step 3 runs it again. The repo's own fixture MEDRPT-2024-PAT-3847_medical_report_scan.pdf follows this path. Anyone converting scanned documents paid for two full parses per file, and on large scans that cost is the whole pdfminer runtime a second time. Batch jobs and server use, where many scans go through one process, pay it on every file.

What changed

• packages/markitdown/src/markitdown/converters/_pdf_converter.py
  • Added a local used_pdfminer = False, set right after the input is buffered.
  • Set it to True just before the pdfminer call in the form_page_count == 0 branch and in the except branch.
  • Changed the final fallback condition from if not markdown: to if not markdown and not used_pdfminer: and updated the comment. The fallback now only runs when pdfplumber found form-style pages and every chunk came back empty. That is the one case where pdfminer has not been tried yet and could still return text.
• packages/markitdown/tests/test_pdf.py
  • Added TestPdfminerFallbackNotRepeated, which covers each path into pdfminer and checks it runs exactly once (details under Tests).

How it works

pdfminer can be reached in two ways: as the main extractor (no form pages, or pdfplumber failed) or as a last resort (form pages found but empty). The flag tells the final check which way the code went. If pdfminer was the main extractor, its result is final, even an empty one. If it was not, the last-resort call works as before.

The flag is set before the call, not after it. If pdfminer raises inside the try block, the except branch still runs and sets the flag itself, so the final check always sees the right state.

Alternatives I considered:

• Return early from each pdfminer branch. This would skip _merge_partial_numbering_lines, or require calling it in several places. A single flag keeps one exit path and one post-processing step.
• Restructure the function so pdfminer is called in exactly one place. This might be cleaner in the long run, but it makes a larger diff in a function that also handles the per-page page.close() memory behaviour covered by TestPdfMemoryOptimization. I kept this change small so it is easy to review.
• Change the condition to if not markdown and form_page_count > 0. form_page_count is not defined if pdfplumber.open itself raises, so this would need extra guarding. The explicit flag is clearer.

Tests

All new cases are in packages/markitdown/tests/test_pdf.py, in a new class TestPdfminerFallbackNotRepeated. It sits next to TestPdfMemoryOptimization and reuses its pdf_activity fixture and _convert_cleanup_fixture helper.

• test_scanned_pdf_runs_pdfminer_once: converts the real scanned medical-report fixture. It checks that the output is empty, that no page was classified as a form, and that pdf_activity.count(("pdfminer", None)) == 1. This is the real-world case that motivated the change.
• test_plain_pdf_with_empty_pdfminer_result_not_retried: patches extract_text to return "" on pdf_cleanup_plain.pdf and checks it was called exactly once and the result is "". This covers the form_page_count == 0 branch.
• test_pdfplumber_failure_with_empty_pdfminer_result_not_retried: makes pdfplumber.open raise and extract_text return "" on pdf_cleanup_form.pdf. It checks there is one call and the result is empty. This covers the except branch.
• test_pdfplumber_failure_uses_pdfminer_text: makes pdfplumber.open raise, with the real pdfminer. It checks that the real text comes through and pdfminer ran once, so the exception fallback still produces output.
• test_empty_form_content_still_falls_back_to_pdfminer: patches _extract_form_content_from_words to return "", so every page counts as a form page with no content. It checks that the final fallback still runs pdfminer once and recovers all three copies of the expected text. This protects the one path where the last-resort call is still needed.

To run them:

(cd 'packages/markitdown' && python3 -m unittest discover -s tests -p 'test_pdf.py')

The tests use pytest fixtures (monkeypatch, pdf_activity), so hatch test tests/test_pdf.py from packages/markitdown is the intended runner.

Compatibility & risk

• Output: no change. The skipped call ran the same deterministic function on the same bytes as a call that had just returned empty, so it could only ever return empty again. Every path that used to produce text still does.
• Public API: no change. The flag is local to convert.
• Performance: removes one full pdfminer parse for every PDF where pdfminer was the main extractor and returned no text. In practice that is mostly scanned and image-only PDFs. Nothing else is affected.
• Migration: none.
• What to review closely: that the final fallback is still reachable when form pages exist but are all empty (covered by test_empty_form_content_still_falls_back_to_pdfminer). One related case is out of scope and unchanged: if pdfminer itself raises inside the form_page_count == 0 branch, the except branch still calls it a second time. That is existing error-handling behaviour, not the empty-result retry this PR fixes.

---

The tests are in a new TestPdfminerFallbackNotRepeated class, not inside TestPdfMemoryOptimization as the original idea said, so the description lists them that way. I haven't run the tests or pre-commit, so the description doesn't claim they pass.

[microsoft-github-policy-service]

Mohamed Emad (@TheGeekyM) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.