Why Your AI Automations Fail on Real-World Data

Why an AI automation that works in a demo fails on your real data

Short answer: AI automations usually fail because a company’s real data (scanned PDFs, hand-built spreadsheets, free-text fields, duplicates) looks nothing like the clean samples used in the demo. The model is rarely the problem; the layer that prepares information for it almost always is.

The pattern is familiar. A flow that extracts supplier invoices or triages customer requests works on twenty hand-picked documents, then degrades within the first week of real use. This article covers the break points, how to diagnose them, and which fixes matter. It complements our piece on what breaks first when an AI agent goes to production, which covers orchestration; here we stay on the data.

What the published figures say

Two numbers recur in the literature. Gartner estimates that 57% of organisations consider their data unfit for AI, and forecasts that 60% of AI projects not backed by AI-ready data will be abandoned by 2026 (figures relayed by several vendors and consultancies; see sources below). These are analyst projections, not field measurements: treat them as an order of magnitude, not a law.

On the technical side, analyses of document pipelines converge on one point: most failures start in the extraction layer (character recognition, reading order, table structure), not in the language model. An extraction error raises no error message. It produces a wrong answer, delivered with confidence.

Six break points when real data arrives

1. Scanned documents and character recognition

A contract or invoice scanned at an angle, stamped, or photographed with a phone degrades character recognition. A “1” read as an “l”, a lost decimal comma: the model receives text that is already wrong and cannot guess.

2. Tables that lose their structure

In an invoice, the meaning of a number depends on its position: column, row, subtotal. A language model reads text; if extraction shifts a column by one, the result stays plausible yet corrupts every downstream calculation. This is the textbook silent failure.

3. Free-text fields and inconsistent formats

Dates written three ways, amounts with or without thousands separators, “CHF”, “Fr.” or nothing. In a small business, the same supplier often appears under three names across the CRM, the accounting tool and the tracking spreadsheet.

4. Duplicates and sources that contradict each other

When two systems hold two addresses or two prices for the same customer, the automation picks one without saying so. It does not flag the conflict, it absorbs it.

5. Context that only exists in people’s heads

“This client always pays at 45 days”, “this line is booked as accruals”. These rules appear in no document. The automation does not know them, and the demo samples did not contain them.

6. Rare cases that happen anyway

Credit notes, foreign-currency invoices, multi-page documents, attachments in the wrong order. Across twenty samples they never show up; across two thousand, they add up to a volume someone will have to handle by hand.

Diagnosing: is it the model or the data?

Before swapping models, separate the two layers. The method takes four steps:

1. Pull fifty real cases from the last month, including the ugly ones, without sorting them.
2. For each, read what the model received as input (the extracted text), not only what it answered.
3. Classify each error: wrong input, correct input misread, or missing business rule.
4. Count. If most errors come from wrong input, a better model will change almost nothing.

This triage follows the same logic as evaluating before the demo: the test set must contain the documents you fear, not the ones that reassure you.

Table: symptom, likely cause, fix

The fixes that matter, in order

• Validate before acting. Every output goes through deterministic checks (totals, formats, value ranges) before it writes to any system.
• Allow an “I don’t know” output. An automation that can never abstain will invent. A human review queue for uncertain cases costs less than an accounting error found at closing.
• Normalise upstream. A single reference list of suppliers, clients and items is worth more than any well-written prompt.
• Log the input, not only the output. Without the extracted text on file, you cannot tell afterwards where the error began.
• Measure on a monthly sample. Data drifts: a new supplier or a new invoice template, and accuracy drops quietly.

Illustrative example: sorting supplier invoices

Take a fictional but representative case: an accounting firm receives roughly two thousand invoices a month, some as native PDFs and the rest as scans. The automation reads each invoice, suggests the expense account and prepares the entry. On a demo sample of twenty native PDFs, everything passes.

In production, three things happen. Poor scans produce misread amounts. Multi-page invoices lose their total, which sits on the last page. Foreign suppliers use another date format. None of these cases is exotic; each is absent from the starting sample.

The fix is not a more powerful model. It is three simple mechanisms: a “line items sum to total” check that sends any inconsistent invoice to review, an enforced date format with rejection of ambiguous cases, and a scan-quality indicator that routes unreadable documents to a person at intake. The gain comes from pipeline discipline, not from model intelligence.

Building the human review queue without turning it into a bottleneck

Human review only makes sense if it stays light. Three rules help. First, send to review only cases under a confidence threshold or failing a check, never a random sample as a general precaution. Second, show the reviewer the source document next to the proposal, with doubtful fields highlighted: without that, review takes longer than manual entry. Third, treat each human correction as data: the same correction repeated ten times is a missing rule that needs writing down.

When this is not the right project

If volume is a few dozen documents a month, automation will probably not pay back: the time spent preparing data exceeds the time saved. Same verdict if the process itself is not stable: automating a rule nobody has written down freezes the ambiguity. In those cases, clarify the process first, perhaps with a simple shared spreadsheet, then come back to automation. For another angle on cost trade-offs, see the real cost of an AI feature in production.

What this means for a Swiss SME

The documents of an accounting firm, a real estate agency or a private clinic often stack the factors above: scans, French, German and Italian side by side, currencies and naming conventions specific to each client. The selection criterion for a provider is therefore not the announced model, but its willingness to work on a sample of your real documents before any commitment. That is the logic of the forward deployed engineer, embedded in your data rather than in a slide deck. To see how we scope this work, read our packages and pricing.

Key takeaways

• Most AI automations fail upstream of the model, in extraction and data quality.
• An extraction failure is silent: the answer looks plausible.
• Test on fifty unsorted real cases, and read the input as much as the output.
• Validate with deterministic checks and provide human review for uncertain cases.
• Below a few dozen documents a month, automation may not be worth it.