A CFO at a venture fund told us something in an interview that we've thought about more than anything else we've heard:

"A lot of people mention using AI to review capital statements against spreadsheets. Unclear if accurate."

That's the state of the category. Not "should we use AI" — that decision already got made, quietly, by individual CFOs at 9pm during close, one paste at a time. The open question is whether the answer coming back can be trusted, and nobody has published a straight answer.

Here's ours. We build extraction for exactly these documents — capital statements, K-1s, audited financials, LPAs — and we've spent a lot of engineering time on the gap between "the model produced numbers" and "the numbers are right." What follows is what actually breaks. Every one of these is a failure mode a general-purpose chat session will hit too. The difference is that we instrumented for them and it won't.

The headline, before the detail:

A general-purpose model will always give you an answer. That is the problem, not the feature.

1. When the data isn't there, it fills

This is the one that should worry you most, because it's invisible.

Ask a model to pull ending capital for twelve partners from a document that only states nine, and you will get twelve numbers. Not nine numbers and a note. Twelve. The three it couldn't find will be plausible — right order of magnitude, consistent with the others, formatted identically. Nothing in the output distinguishes them from the nine that are real.

And the invented rows are internally consistent. They foot. They look like every other run.

The fix isn't a better prompt. Asking a model nicely to say "I don't know" is not a control — it's a request. The fix is structural: the extraction has to be able to refuse, and refusal has to be a first-class outcome that the surrounding system records and shows you, not an absence it papers over. A row that couldn't be sourced has to come back marked as un-sourced, and the total has to be marked incomplete.

There is no way to get that behaviour out of a chat window. A chat window's entire design goal is to produce a response.

The check: for any figure a model gives you, ask what page and line it came from. If it can't point, it didn't read it.

2. Units are per-column, never a global convention

Fund documents mix scales inside a single table. A schedule of investments will show fair value in whole dollars and ownership in percent. A management-company P&L will show most lines in thousands and one line — often a pass-through or a syndication cost — in units. There's no header that announces this. A human reader gets it from context and never notices they did.

A model applies whatever convention it inferred first, to everything.

We shipped a bug where exactly this happened: one expense line was being read from the wrong column, producing a figure 137 times what the document said. It ran across 144 rows before it was caught. Every individual number looked like a number. The report rendered. Nothing errored.

A 137× error is easy to spot when it lands on a headline figure. It is nearly impossible to spot when it lands on line 40 of an expense schedule that nobody reads line by line — which is precisely where these land, because the headline figures are the ones humans check.

The check: pick one line in the middle of a long schedule — not the top, not the total — and verify it against the source by eye. Middle-of-schedule is where scale errors hide.

3. The file is not always what its name says

This one has nothing to do with the model and everything to do with the real world.

We hit a set of workbooks whose contents did not match their own filenames. Not corrupted — just mislabelled at some point in a chain of hands between administrator, auditor, and firm. The fund named in the file was not the fund inside the file.

If your process pairs a capital statement to an allocation schedule by filename, or by folder, or by anything other than what the documents actually say about themselves, you will confidently reconcile two unrelated funds. We did. A run that looked complete reported $1.66 million of discrepancies that did not exist in any source document, because it had compared one fund's capital statement against the other fund's allocation export. Every number on the page was real. The comparison was fiction.

Worth sitting with: no filename heuristic can save you here. The only defence is to read the identity out of the content — fund name, period, entity — and refuse to compare two documents that don't agree about who they are. That's a rule the system has to enforce, because a person under close deadline will not.

The check: before any comparison, confirm both documents independently state the same fund and the same period. Not the same folder. The same stated fund.

4. Two sources agreeing is not verification

We had an extraction validated against two independent sources. It matched 153 of 153 on the first, and 72 of 72 on the second. That is the kind of result that ends an investigation.

Then a third source arrived from a different fund administrator, and 144 of 145 rows were wrong.

The reason is worth understanding, because it generalises far past our system. The first two administrators used similar vocabulary — the same words for the same concepts, in the same places. The extraction had learned their dialect, not the domain. When it met an administrator who called a contribution something else and put it somewhere else, it failed almost completely, while remaining just as confident.

Two sources agreeing tells you the two sources are similar. It tells you nothing about the third.

The check: if you're evaluating any AI tool on your documents, test it on a package from a different administrator than the one you tested first. If a vendor's accuracy claims come from one firm's documents, that is a claim about one firm's documents.

5. A total can tie by coincidence

The most seductive validation in fund operations is the footing check: the parts sum to the whole, so the parts are right.

They aren't necessarily. Two errors of opposite sign inside a schedule net to zero at the total. A misallocation between two partners is invisible in the fund-level figure. A line moved from one category to another changes nothing about the sum.

We have watched a total tie perfectly across a set of rows where individual rows were wrong. The footing check passed. It was doing its job — it just isn't the job people think it is.

Footing tells you the arithmetic is internally consistent. It says nothing about whether the inputs were read correctly. Those are different questions and only one of them is answered by a total.

The check: tie at the row level, not just the total. If a tool reports a clean total without showing you row-level agreement, it has told you the smaller thing.

So what's the actual answer?

If you paste a capital statement into a chat window and ask it to check the numbers against your spreadsheet, here's what you get: a fast, articulate, well-formatted response that is probably mostly right, with no way to tell which parts aren't, and no record of what it looked at.

For a first read, that's genuinely useful. We're not going to pretend otherwise — the CFOs doing this are getting real value, and the alternative is often doing nothing.

For anything that goes to an LP, an auditor, or a tax filing, the problem isn't the error rate. It's the absence of a defensible trail. When your auditor asks why a figure is what it is, "the model said so" is not an answer, and you can't reconstruct the session.

What has to be true instead:

Every figure points to a document, page, and line
The system can refuse — and a refusal is visible, not silently backfilled
Identity is checked before comparison, from content and not filenames
Agreement is measured row by row, not just at the total
The same checks run on every document, every time, not when someone remembers

Those aren't AI features. They're the ordinary controls a fund CFO already applies to human work. The mistake the category is making is treating a model as an oracle that either is or isn't accurate, rather than as a worker whose output needs the same review any junior's would.

We built Fundry around that assumption. Every answer is cited to the source document. Nothing sends without a human approving it. The checks are deterministic and they run every time, including the boring ones.

Test it on your own document

We would rather you didn't take our word for any of this.

Fundry's free DDQ tool takes a real questionnaire and returns drafted answers, each with a confidence score attached. No account required. Use a real one — the generic ones prove nothing.

The thing to look at isn't the answers it gets right. It's how it scores the questions it is unsure of. That's where you find out whether a tool admits doubt or fills.