What has been measured
FinanceBench has 150 questions about real filings, and for each it marks the page that holds the evidence. Here, a plain keyword search (BM25) reads the question and ranks the pages of 360 filings (53,399 pages). The score is how often a marked page is among the first few results. No language model is involved.
Scope. The retrieval measurements below use no model. A separate real-model comparison of basic retrieval and the agent follows them. New to these terms? See what they mean.
Is the evidence page in the top 10 results?
Share of questions where it is. Choose how many results count:
13%
Search all 360 filings
Keywords only, no help.
31%
+ company and year filter
First pick the filings that fit the company and year in the question.
44%
Right filing given (a ceiling)
Told which filing to look in: the best this search can do.
In plain words. Given only the question, keyword search puts a page FinanceBench marks into the top 10 for 13% of questions. Picking the right company and year first lifts that to 31%. Even told the exact filing, it reaches only 44%, so a large part of the weakness is vocabulary: questions and filings use different words (more below).
By type of question
All questions 150 questions
- 13%, All filings
- 31%, With filter
- 44%, Right filing
domain-relevant 50 questions
- 12%, All filings
- 20%, With filter
- 26%, Right filing
metrics-generated 50 questions
- 0%, All filings
- 32%, With filter
- 34%, Right filing
novel-generated 50 questions
- 26%, All filings
- 40%, With filter
- 72%, Right filing
What to take from it
Plain search is weak. Over all filings it puts a marked page in the top 10 for 13% of questions. Even told the right filing, it manages 44%. Questions and filings use different words (a question says “capital expenditure”, the filing says “Purchases of property, plant and equipment”). The real-model comparison below tests an agent that can rewrite its search; embedding reranking remains unevaluated.
The company and year filter helps a lot: 31%, which closes 57% of the gap to the ceiling. It keeps the right filing for 148 of 150 questions and narrows to about 3 filings of 360.
It stops short of the ceiling, and not for lack of a year. 131 of the 150 questions name a year. A year does not pin down one filing: a company has a 10-K, 10-Qs, 8-Ks and earnings releases for the same year, so 90 of the 147 recognised questions still have more than one candidate. Of the 20 questions that the right filing alone gets right and the filter does not, 18 were crowded out by another filing of the same company.
The filter’s recognition is partly tuned on this benchmark. By company name alone it recognises 128 of 150 questions. I then added five nicknames (AMEX, JNJ, JPM, MGM, ULTA) after seeing which questions it missed, which brings it to 147. The honest figure for questions it has not seen is the first.
The page numbers are checked. Searching with the evidence text itself finds its own page at rank 1 for 97% of questions, so the low numbers above are not an indexing mistake. FinanceBench counts pages from 0, which the evaluation measured instead of assuming.
What the words mean
- FinanceBench
- A public set of 150 questions about real company filings. For each, experts marked the page that holds the evidence. That marked page is what this site tries to find.
- Filing
- A document a company publishes about its finances: a 10-K (annual report), a 10-Q (quarterly report), an 8-K (a report of one event) or an earnings release.
- Top 10
- Whether the marked page is among the first 10 pages the search returns. “Top 1” is the strictest: the very first result.
- Keyword search (BM25)
- Ranks pages by how well their words match the question's words, giving rare words more weight. It does not understand meaning.
- Company and year filter
- A rule that reads the company and year in the question and searches only the filings that fit, so a Best Buy question is not answered from an AMD page.
- Ceiling
- The score when the search is told the right filing. No setting without that help can beat it, so it shows how much of the difficulty is the search itself.
- MRR
- Mean reciprocal rank: 1 divided by the rank of the first marked page (1 for first, ½ for second, ⅓ for third), averaged over the questions. Higher is better, at most 1.
More on how each part works, and which parts have not been run, is on How it works.
Every number
The table is wider than the screen: swipe it sideways to see every column.
| Questions | Search | top 1 | top 3 | top 5 | top 10 | MRR |
|---|---|---|---|---|---|---|
| All questions150 questions | All filings | 7% | 8% | 10% | 13% | 0.080 |
| With filter | 9% | 14% | 18% | 31% | 0.135 | |
| Right filing | 14% | 23% | 29% | 44% | 0.209 | |
| domain-relevant50 questions | All filings | 2% | 2% | 6% | 12% | 0.037 |
| With filter | 6% | 10% | 12% | 20% | 0.091 | |
| Right filing | 6% | 10% | 16% | 26% | 0.109 | |
| metrics-generated50 questions | All filings | 0% | 0% | 0% | 0% | 0.000 |
| With filter | 2% | 10% | 16% | 32% | 0.083 | |
| Right filing | 4% | 16% | 18% | 34% | 0.110 | |
| novel-generated50 questions | All filings | 18% | 22% | 24% | 26% | 0.204 |
| With filter | 18% | 22% | 26% | 40% | 0.230 | |
| Right filing | 32% | 42% | 52% | 72% | 0.410 |
Question-type names are FinanceBench’s own. Full text of the run: results/ in the repository.
Beyond retrieval
The real-model comparison below evaluates basic retrieval and the bounded agent on the same fixed question sample. Closed-book and embedding-reranked comparisons remain planned. The browser search still returns source pages; the offline evaluation measures generated answers separately.
Real-model answers: retrieval versus the agent
Partial evaluation: 16 completed pairs from a planned stratified sample of 50 FinanceBench questions, searching 360 filings. C1 receives the top five keyword-search pages. C5 can search, read pages and calculate before answering, with citation checks enforced.
| Configuration | Questions | Correct | 95% interval | Refused | Median answer time |
|---|---|---|---|---|---|
| C1 · basic retrieval | 16 | 1 (6%) | 0% to 19% | 15 | 5.7 s |
| C5 · agent | 16 | 5 (31%) | 13% to 56% | 5 | 39.7 s |
Agent minus basic retrieval: 25.0 percentage points across 16 paired questions; 95% paired bootstrap interval 0.0 to 50.0 points. This sample does not establish an improvement.
Answer model: claude-sonnet-5. Judge: claude-haiku-4-5-20251001. Refusals count as unsuccessful answers. Unjudged answers remain visible in the downloadable metrics. Completed pairs by question type: domain-relevant: 8; novel-generated: 8.
Quote checks, usage and evaluation limits
- C1: all quotes found for 1 of 1 non-refused answers with checked citations; 5825 mean answer tokens; $0.414 API-equivalent estimate including judging.
- C5: all quotes found for 11 of 11 non-refused answers with checked citations; 18300 mean answer tokens; $1.544 API-equivalent estimate including judging.
- A fixed stratified sample, not the full benchmark; intervals reflect sample uncertainty.
- Tables compare only completed pairs. A quota-limited prefix of the planned stratified sample is not necessarily stratified itself.
- Quote checks verify text exists on a page, not that it supports every inference.
- A different model judges answers; this is automated grading, not human validation.
- C1 and C5 differ in filtering, page access, calculation and citation enforcement; this comparison does not isolate agent reasoning alone.
- CLI cannot set temperature or max output tokens; its tools and customizations are disabled.
- Costs are API-equivalent estimates reported by CLI, not subscription charges.
- Latency sums recorded answer-model calls; cached replays retain original call latency.