# Extract text and tables from any PDF URL with Python (pdfplumber + pypdf)

*I wrote this with help from an AI assistant. I ran every script below myself on Nov 2, 2026, against the public PDFs linked in the post, and the output shown is copied from those runs.*

PDFs are where a lot of useful data goes to hide: government forms, annual reports, research results, price lists. Getting the text out is easy. Getting it out in a shape you can actually use, with tables that are still tables, is the hard part.

In this post we'll build one small Python script that takes a PDF URL and returns:

- the document's metadata (page count, title, author, producer, creation date),
- the plain text of each page, **without** the table contents duplicated in it,
- every table on each page as clean Markdown,
- and a flag for pages that are scanned images and need OCR.

We'll use three libraries: `requests` to download, `pypdf` for metadata, and `pdfplumber` for text and tables.

```bash
pip install requests pdfplumber pypdf
```

I tested with Python 3.13, pdfplumber 0.11.10 and pypdf 6.21.4.

## Step 1: download safely

Two things go wrong when you download "a PDF" from the internet: the file is huge, or it isn't a PDF at all (a login page, a 404 page served with status 200, a cookie wall). So we stream the download, stop at 50 MB, and check the magic bytes. Every real PDF starts with `%PDF`.

```python
MAX_BYTES = 50 * 1024 * 1024

def fetch_pdf(url: str) -> bytes:
    with requests.get(url, stream=True, timeout=30,
                      headers={"User-Agent": "pdf-to-text-tutorial/1.0"}) as r:
        r.raise_for_status()
        buf = io.BytesIO()
        for chunk in r.iter_content(64 * 1024):
            buf.write(chunk)
            if buf.tell() > MAX_BYTES:
                raise ValueError("PDF is larger than 50 MB")
    data = buf.getvalue()
    if not data.startswith(b"%PDF"):
        raise ValueError(f"Not a PDF (starts with {data[:8]!r})")
    return data
```

Point it at a web page and you get a clear error instead of a confusing parser crash later:

```text
ValueError: Not a PDF (starts with b'<!doctyp')
```

## Step 2: metadata with pypdf

`pypdf` reads the document info dictionary in a couple of lines. Values can be missing, so every field may be `None`.

```python
def metadata(data: bytes) -> dict:
    reader = PdfReader(io.BytesIO(data))
    meta = reader.metadata or {}
    return {
        "pages": len(reader.pages),
        "encrypted": reader.is_encrypted,
        "title": meta.get("/Title"),
        "author": meta.get("/Author"),
        "producer": meta.get("/Producer"),
        "created": str(meta.get("/CreationDate") or "") or None,
    }
```

For the IRS's [2026 Form W-4](https://www.irs.gov/pub/irs-pdf/fw4.pdf) this returned:

```text
{'pages': 5, 'encrypted': False, 'title': '2026 Form W-4', 'author': 'C:DC:TS:CAR:MP',
 'producer': 'Designer 6.5', 'created': "D:20251211071541-06'00'"}
```

The date is in PDF format (`D:YYYYMMDDHHmmSS` plus an offset). It's easy to parse later if you need a real `datetime`.

## Step 3: tables, and why the first try looks terrible

`pdfplumber` finds tables by looking for the ruling lines drawn around cells. Let's try it on the W3C's [example PDF table](https://www.w3.org/WAI/WCAG21/working-examples/pdf-table/table.pdf), a small results table with merged header cells.

With default settings, `page.find_tables()` returned a table **18 columns wide** for what is visibly a 6-column table. Most cells were empty, and the header words ("Disability", "Category", "Ballots", "Completed"...) were scattered over seven separate rows.

Two things cause this:

1. **Lines that almost touch.** The cell borders in many PDFs are drawn as separate rectangles that are off by a point or two. pdfplumber treats each slightly offset edge as a new column boundary. Raising `snap_tolerance` and `join_tolerance` from the default 3 to 6 points merges them.
2. **Merged header cells.** A header like "Ballots / Incomplete / Terminated" spanning several drawn rows comes back as several sparse rows.

With `{"snap_tolerance": 6, "join_tolerance": 6}` the body rows were already perfect, but there were still empty columns and a four-row header. So we add two small cleanup steps.

First, normalise cells and drop rows and columns that are completely empty:

```python
def clean_table(rows):
    rows = [[(c or "").replace("\n", " ").strip() for c in row] for row in rows if row]
    rows = [r for r in rows if any(r)]
    if not rows:
        return []
    width = max(len(r) for r in rows)
    rows = [r + [""] * (width - len(r)) for r in rows]
    keep = [j for j in range(width) if any(r[j] for r in rows)]
    return [[r[j] for j in keep] for r in rows]
```

Second, collapse the multi-row header. The rule is simple: every leading row that still has an empty cell is part of the header, and the first fully filled row is data. Join the header rows column by column:

```python
def collapse_header(rows):
    if len(rows) < 2:
        return rows
    n = 0
    while n < len(rows) - 1 and not all(rows[n]):
        n += 1
    if n <= 1:
        return rows
    header = [" ".join(r[j] for r in rows[:n] if r[j]) for j in range(len(rows[0]))]
    return [header] + rows[n:]
```

Here's the W3C table after both steps, exactly as the script printed it:

```text
| Disability Category | Participants | Ballots Completed | Ballots Incomplete/ Terminated | Results Accuracy | Time to complete |
| --- | --- | --- | --- | --- | --- |
| Blind | 5 | 1 | 4 | 34.5%, n=1 | 1199 sec, n=1 |
| Low Vision | 5 | 2 | 3 | 98.3% n=2 (97.7%, n=3) | 1716 sec, n=3 (1934 sec, n=2) |
| Dexterity | 5 | 4 | 1 | 98.3%, n=4 | 1672.1 sec, n=4 |
| Mobility | 3 | 3 | 0 | 95.4%, n=3 | 1416 sec, n=3 |
```

Six columns, one header row, every value in the right place.

The header rule is a heuristic. If a real data row has an empty cell in its first rows, it would be pulled into the header, so check a few tables from your own source before trusting it blindly.

## Step 4: text without duplicated tables

If you call `page.extract_text()` on a page with a table, you get the table's cell text mixed into the paragraphs, and then again in your Markdown table. The fix is to filter out every character whose centre lies inside a table's bounding box before extracting text:

```python
boxes = [t.bbox for t in tables]

def outside(obj):
    cx = (obj["x0"] + obj["x1"]) / 2
    cy = (obj["top"] + obj["bottom"]) / 2
    return not any(x0 <= cx <= x1 and top <= cy <= bottom
                   for x0, top, x1, bottom in boxes)

text = page.filter(outside).extract_text() or ""
```

On the W3C file, the page text is now just the two lines outside the table:

```text
Example table
This is an example of a data table.
```

## Step 5: spot scanned pages

A scanned PDF is just pictures of pages. There are no characters to extract, so pdfplumber returns empty text, silently. Rather than shipping empty results, count the characters on each page and flag pages with zero:

```python
scanned = [p["page"] for p in result["pages"] if p["chars"] == 0]
if scanned:
    result["needs_ocr"] = scanned
```

To test this, I rendered the W3C page to an image and saved it as a new image-only PDF. The script reported `"needs_ocr": [1]` for it. Those pages need an OCR tool such as Tesseract, which is a topic of its own.

## The full script

```python
"""Download a PDF from a URL and turn it into page-by-page text,
Markdown tables and metadata. Uses requests + pdfplumber + pypdf."""
import io, json, sys
import requests
import pdfplumber
from pypdf import PdfReader

MAX_BYTES = 50 * 1024 * 1024  # refuse anything over 50 MB
# Snap/join nearly-touching ruling lines so merged cells don't split
# into phantom columns. pdfplumber's default tolerance is 3 points.
TABLE_SETTINGS = {"snap_tolerance": 6, "join_tolerance": 6}


def fetch_pdf(url: str) -> bytes:
    with requests.get(url, stream=True, timeout=30,
                      headers={"User-Agent": "pdf-to-text-tutorial/1.0"}) as r:
        r.raise_for_status()
        buf = io.BytesIO()
        for chunk in r.iter_content(64 * 1024):
            buf.write(chunk)
            if buf.tell() > MAX_BYTES:
                raise ValueError("PDF is larger than 50 MB")
    data = buf.getvalue()
    if not data.startswith(b"%PDF"):
        raise ValueError(f"Not a PDF (starts with {data[:8]!r})")
    return data


def metadata(data: bytes) -> dict:
    reader = PdfReader(io.BytesIO(data))
    meta = reader.metadata or {}
    return {
        "pages": len(reader.pages),
        "encrypted": reader.is_encrypted,
        "title": meta.get("/Title"),
        "author": meta.get("/Author"),
        "producer": meta.get("/Producer"),
        "created": str(meta.get("/CreationDate") or "") or None,
    }


def clean_table(rows: list[list]) -> list[list[str]]:
    """Normalise cells, then drop rows and columns that are entirely empty.
    Ruled tables often come back with many phantom columns (merged cells)."""
    rows = [[(c or "").replace("\n", " ").strip() for c in row] for row in rows if row]
    rows = [r for r in rows if any(r)]
    if not rows:
        return []
    width = max(len(r) for r in rows)
    rows = [r + [""] * (width - len(r)) for r in rows]
    keep = [j for j in range(width) if any(r[j] for r in rows)]
    return [[r[j] for j in keep] for r in rows]


def collapse_header(rows: list[list[str]]) -> list[list[str]]:
    """Multi-row headers (merged cells) arrive as several sparse rows.
    Treat every leading row that has an empty cell as part of the header
    and join those rows column by column into a single header row."""
    if len(rows) < 2:
        return rows
    n = 0
    while n < len(rows) - 1 and not all(rows[n]):
        n += 1
    if n <= 1:
        return rows
    header = [" ".join(r[j] for r in rows[:n] if r[j]) for j in range(len(rows[0]))]
    return [header] + rows[n:]


def table_to_markdown(rows: list[list]) -> str:
    rows = collapse_header(clean_table(rows))
    if not rows:
        return ""
    width = len(rows[0])
    out = ["| " + " | ".join(rows[0]) + " |", "|" + " --- |" * width]
    out += ["| " + " | ".join(r) + " |" for r in rows[1:]]
    return "\n".join(out)


def extract(data: bytes, max_pages: int | None = None) -> list[dict]:
    pages = []
    with pdfplumber.open(io.BytesIO(data)) as pdf:
        for i, page in enumerate(pdf.pages[:max_pages], start=1):
            tables = page.find_tables(TABLE_SETTINGS)
            # Drop the characters that sit inside tables so table cells
            # don't show up twice (once as text, once as Markdown).
            boxes = [t.bbox for t in tables]
            def outside(obj):
                cx = (obj["x0"] + obj["x1"]) / 2
                cy = (obj["top"] + obj["bottom"]) / 2
                return not any(x0 <= cx <= x1 and top <= cy <= bottom
                               for x0, top, x1, bottom in boxes)
            text = page.filter(outside).extract_text(layout=False) or ""
            pages.append({
                "page": i,
                "text": text.strip(),
                "tables": [table_to_markdown(t.extract()) for t in tables],
                "chars": len(page.chars),
            })
    return pages


if __name__ == "__main__":
    url = sys.argv[1]
    limit = int(sys.argv[2]) if len(sys.argv) > 2 else None
    data = fetch_pdf(url)
    result = {"url": url, "bytes": len(data), "meta": metadata(data),
              "pages": extract(data, limit)}
    scanned = [p["page"] for p in result["pages"] if p["chars"] == 0]
    if scanned:
        result["needs_ocr"] = scanned
    json.dump(result, sys.stdout, indent=2, ensure_ascii=False)
```

Run it with a URL and an optional page limit:

```bash
python pdf_to_text.py https://www.irs.gov/pub/irs-pdf/fw4.pdf > w4.json
```

For the 5-page, 209 KB Form W-4 it found 3 tables on page 1 and 3 on page 5, and around 1,500 to 7,300 characters of text per page.

## What it won't do well

I want to be honest about the limits, because PDF extraction is never perfect:

- **Tables without ruling lines.** pdfplumber's line strategy needs drawn borders. On page 5 of the W-4, the withholding table only draws lines between groups of rows, so three printed rows end up in a single cell (for example `$90 850 1,020`). For borderless tables, try `"horizontal_strategy": "text"`, but test it, because it can also split words across columns.
- **Scanned pages** need OCR, as above.
- **Multi-column layouts** (academic papers) can interleave columns in the text. Cropping each column with `page.crop(...)` before extracting helps.
- **Encrypted PDFs** with an owner password may still open; ones with a user password need it.

## If you'd rather not run it yourself

I also packaged this approach as a hosted tool, [PDF Text Extractor on Apify](https://apify.com/saadhazz/pdf-text-extractor). You give it a list of PDF URLs and get back page-by-page text, Markdown tables and metadata as JSON or CSV, without setting up Python. The script above is free to use however you like, though.

If you hit a PDF that breaks it, leave a comment with the link and I'll take a look.