Skip to content

fix: retain every document returned by batched fetches - #22

Open
rudycelekli wants to merge 2 commits into
firecrawl:mainfrom
rudycelekli:fix/retain-batched-fetch-documents
Open

rudycelekli wants to merge 2 commits into
firecrawl:mainfrom
rudycelekli:fix/retain-batched-fetch-documents

Conversation

@rudycelekli

Copy link
Copy Markdown

The committed Exa and Parallel fetch schemas support multiple URLs, but record_fetch calls a singular parser that unwraps only the first document. Later pages never reach the citation evidence store or reader provenance, although the model receives the whole response.

Harvest all structured fetch pages while keeping each URL/body pair together, including root-body/metadata-URL and excerpt formats. Preserve the singular helper API. A URL-less response is attributed only when exactly one URL was requested; an unstructured batch is not falsely attached to its first URL.

Validation:

  • Seventeen actual Controller/parser cases cover results/data/documents/pages/root arrays, excerpt content, metadata URLs, the legacy helper, bare single-page markdown, ambiguous batch prose, empty batches, MCP errors and JSON document bodies. Same tests on main:8 fail/9 pass; fixed full repository suite:149 pass.
  • Regression review additionally verified that raw JSON booleans/numbers/strings and unrecognized object/array bodies remain intact as page content; explicit empty batches still log zero documents.
  • Independent public Controller lifecycle check: two documents consume one read, zero searches, produce two reader refs and zero engine refs, and log both document bodies. Fetches remain outside search accounting.
  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest devdex/tests -q and git diff --check pass. Final suite runs env-clean with external sockets/DNS denied.

No model/provider session or score/ranking/denominator change is claimed. The oracle uses controlled valid MCP responses, not a live vendor result or full SDK agent run.

Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant