How it works
Footnote answers questions about company filings and shows the page behind every figure. The project is a measurement as much as a tool: each part is added one at a time and scored against the same 150 questions from FinanceBench, whose own README reports that GPT-4-Turbo with retrieval answered 81% of them wrongly or refused. Retrieval is measured on all questions; generated answers are evaluated on a fixed sample.
Stand-in. “Stand-in” means the code runs end to end in tests against a scripted fake of the model, which proves the plumbing and the rules but not how a real model behaves with these prompts. Nothing has been run against Gemini or Groq yet.
In short
What it is. A tool for checking a number in a company’s filing. You ask a question; it shows the pages most likely to hold the answer, so you can read the figure at its source.
What works today. Finding pages, in your browser, for 7 filings. It has been scored on all 150 FinanceBench questions (see Results).
Generated answers. A separate offline evaluation compares basic retrieval with the agent using real models. The Results page includes the sample size, errors, refusals and uncertainty. This browser interface continues to show source pages.
The path of a question
- Question
- Filter by company and year
- Rank pages
- Answer with citations
- Check the citations
The demo on the Search page does steps 1 to 3, the filled stops. Steps 4 and 5, the hollow ones, need a language model.
Each part, and how far it has been checked
The table is wider than the screen: swipe it sideways to read every column.
| Part | What it does | State |
|---|---|---|
| Reading the pages | Turns each filing's PDF into numbered pages. FinanceBench counts pages from 0; this was measured on all 183 evidence records, not assumed. | Checked against the real filings |
| Keyword search (BM25) | Ranks pages by how well they match the question's words. This site runs a TypeScript port that a test holds to the Python ranking, page for page. | Measured on all 150 questions |
| Company and year filter | Reads the company and year in the question and searches only the filings that fit. Rules, no model. | Measured on all 150 questions |
| Embedding rerank | Re-orders BM25's top 50 pages by meaning, using Gemini's free embeddings, and fuses the two rankings. | Tested with a stand-in model only |
| Typed, cited answers | A model must answer as structured data with a page and an exact quote for every figure. A rule checks each quote really appears on that page; an answer that fails gets one retry, then becomes a refusal. | Evaluated with real models on a fixed sample |
| Calculator | Ratios and changes go through a calculator that parses arithmetic and nothing else: no eval, no names, no function calls beyond abs, round, min and max. | Unit-tested, including refusals |
| Agent | Searches, reads a page, calculates, and finishes, within a budget of six tool calls. Rebuilds a compact prompt each turn so each call fits a free tier's per-minute limit. | Evaluated with real models on a fixed sample |
| Judge | A second model compares an answer with FinanceBench's reference answer, cross-checked by a rule that compares a single reference number. | Evaluated with real models on a fixed sample |
| Free-tier budgeting | Every model call is cached; per-minute limits are waited out; a spent daily limit moves to the next model or stops the run to resume tomorrow. | Tested with a stand-in model only |
| MCP server | The same four tools (list filings, search, get a page, calculate) for Claude Desktop or Claude Code, run as a real subprocess against the real filings. | Checked against the real filings |
Choices and limits
Free hosted models only. Gemini and Groq free tiers, no card, nothing that can bill anyone. Their daily limits decide how many questions can be answered, so every result will state its sample size.
What this site ships. The text of 7 filings (670 pages), published by the companies themselves (each card links to its filing), and 31 benchmark questions with the page FinanceBench points to. It does not ship FinanceBench’s reference answers or evidence text: the benchmark states no licence, so it is used to evaluate and not redistributed.
Not covered. Fine-tuning, local models, a hosted vector database, multi-agent setups.
Code, tests and the full write-up: github.com/f20230125-sudo/footnote.