Structured data from PDFs

Docling parses layout and tables, Pydantic AI fills a typed schema, and a golden set scores field accuracy.

Run it as a prompt

Paste this into any AI agent, or fetch it: curl -s https://uplift.page/api/v1/prompts/pdf-structured-extraction/raw

prompt.md

# Structured data from PDFs

Extract typed records (e.g. invoices: vendor, dates, line items, totals) from mixed digital and scanned PDFs with at least 95% field-level accuracy on a 50-document golden set, line items summing to the total within 0.01, and under 20 s per 10-page document on a CPU worker.

## Recommended stack

- **Docling 2.x (Python 3.11+, the floor for Pydantic AI 2.x)** - PDF parsing: layout, reading order, table structure, OCR. MIT-licensed, hosted by LF AI & Data, and keeps tables as cells instead of flattening them into word soup.
- **Pydantic AI 2.x** - Typed extraction with output_type and output validators. Returns a validated Pydantic model, and ModelRetry sends arithmetic or format errors back to the model for another attempt.
- **pydantic-evals 2.x** - Golden-set evaluation. Dataset, Case, and custom evaluators score each field and print a report per run.
- **Postgres (JSONB) or DuckDB** - Results store with source page references. Keeps extracted JSON next to file hashes so only changed files are reprocessed.

## Build steps

1. Define the schema as Pydantic models with field descriptions, Decimal for money, date types, enums, Optional for fields that may be absent, and a source_page per field group.
2. Collect 50 representative PDFs (digital, scanned, multi-page, rotated, multi-currency) and hand-label each as JSON; this is the golden set.
3. Convert with Docling: `docling sample.pdf --to md` to inspect, DocumentConverter in the pipeline; enable OCR only for pages without a text layer.
4. Pass Docling Markdown (tables intact) to a Pydantic AI agent with output_type=Invoice; add an @agent.output_validator that checks line items = subtotal and subtotal + tax = total and raises ModelRetry naming the gap.
5. Route failures: if validation still fails after retries or a required field is null, send the document to a human review queue instead of inserting it.
6. Build the eval with pydantic_evals.Dataset(name=..., cases=[...], evaluators=[...]): one Case per golden document and a custom Evaluator that returns {field: bool}, so `dataset.evaluate_sync(extract)` reports each field's pass rate.
7. Run as a batch worker: hash each file, skip unchanged ones, store JSON plus page references, and log per-document latency.

## Watch out for

- Plain text extraction per page flattens tables, so line items merge. Use a layout-aware parse.
- Floats for money cause rounding failures in total checks. Use Decimal end to end.
- A golden set of only clean digital PDFs hides OCR failures. Include your worst scans.
- The model fills missing fields with plausible guesses. Make them Optional, instruct null over guessing, and measure the invented-field rate.
- Borderless tables can still come out as flat text in Docling Markdown; put borderless invoices in the golden set and check the Markdown before blaming the model.

## Definition of done

- The eval command prints per-field accuracy with an overall score of 95% or higher.
- A test document with a deliberately altered total is caught by the validator and lands in review.
- A scanned PDF goes through the OCR path and its totals match the label.
- Re-running the batch on unchanged files makes 0 model calls.
- A 10-page scanned invoice converts and extracts in under 20 s on the CPU worker with a warm DocumentConverter (the log shows the time).

The full prompt

Recommended stack

Docling 2.x (Python 3.11+, the floor for Pydantic AI 2.x) - PDF parsing: layout, reading order, table structure, OCR. MIT-licensed, hosted by LF AI & Data, and keeps tables as cells instead of flattening them into word soup.
Pydantic AI 2.x - Typed extraction with output_type and output validators. Returns a validated Pydantic model, and ModelRetry sends arithmetic or format errors back to the model for another attempt.
pydantic-evals 2.x - Golden-set evaluation. Dataset, Case, and custom evaluators score each field and print a report per run.
Postgres (JSONB) or DuckDB - Results store with source page references. Keeps extracted JSON next to file hashes so only changed files are reprocessed.

Build steps

Define the schema as Pydantic models with field descriptions, Decimal for money, date types, enums, Optional for fields that may be absent, and a source_page per field group.
Collect 50 representative PDFs (digital, scanned, multi-page, rotated, multi-currency) and hand-label each as JSON; this is the golden set.
Convert with Docling: docling sample.pdf --to md to inspect, DocumentConverter in the pipeline; enable OCR only for pages without a text layer.
Pass Docling Markdown (tables intact) to a Pydantic AI agent with output_type=Invoice; add an @agent.output_validator that checks line items = subtotal and subtotal + tax = total and raises ModelRetry naming the gap.
Route failures: if validation still fails after retries or a required field is null, send the document to a human review queue instead of inserting it.
Build the eval with pydantic_evals.Dataset(name=..., cases=[...], evaluators=[...]): one Case per golden document and a custom Evaluator that returns {field: bool}, so dataset.evaluate_sync(extract) reports each field's pass rate.
Run as a batch worker: hash each file, skip unchanged ones, store JSON plus page references, and log per-document latency.

Watch out for

Plain text extraction per page flattens tables, so line items merge. Use a layout-aware parse.
Floats for money cause rounding failures in total checks. Use Decimal end to end.
A golden set of only clean digital PDFs hides OCR failures. Include your worst scans.
The model fills missing fields with plausible guesses. Make them Optional, instruct null over guessing, and measure the invented-field rate.
Borderless tables can still come out as flat text in Docling Markdown; put borderless invoices in the golden set and check the Markdown before blaming the model.

Definition of done

The eval command prints per-field accuracy with an overall score of 95% or higher.
A test document with a deliberately altered total is caught by the validator and lands in review.
A scanned PDF goes through the OCR path and its totals match the label.
Re-running the batch on unchanged files makes 0 model calls.
A 10-page scanned invoice converts and extracts in under 20 s on the CPU worker with a warm DocumentConverter (the log shows the time).

Served from the uplift.page library and refreshed within 5 minutes of every update.