feat(rq): retry transient HTTP source fetch failures

[dan-m62]

Summary

On the RQ engine, an async conversion whose HTTP source URL is temporarily unavailable (for example a 503 from the upstream host) is neither retried nor reported as a failure — the task completes as a success and the per-document errors array comes back empty. This PR makes the RQ worker fetch HTTP sources itself so those failures can be retried when they are transient and surfaced with their real cause when they are not.

This follows up on docling-project/docling-serve#648 (enhancing 5xx handling for HTTP sources on the async engine), and adopts the same retry, classification and configuration patterns the Ray engine already uses, so the two engines behave consistently. The companion docling-serve change that exposes the new settings as environment variables is in docling-project/docling-serve#650

Background

On the RQ path, HttpSource URLs are handed to the converter as plain strings and docling fetches them internally. Since docling 2.102, a fetch error inside _DocumentConversionInput.docs() is caught and turned into an error-less "invalid" input document, so the exception never reaches the worker. The result is that a genuinely failed fetch looks like this:

// poll: /v1/status/poll/{task_id}
{ "task_status": "success", "error_message": null, "failure": null }

// result: /v1/result/{task_id}
{ "num_failed": 1, "documents": [ { "status": "failure", "errors": [] } ] }

No retry, and no indication of why the document failed (a 503 is indistinguishable from a 404 or a parse error).

What this changes

• The RQ worker now fetches HttpSource inputs before conversion, reusing the existing fetch_http_source_bytes_async from docling_jobkit/convert/materialization.py, so docling receives bytes (a DocumentStream) rather than a URL.
• Transient failures are retried. The retry decision reuses the shared classify_public_task_failure classifier: 429, 502, 503, 504 and connection/timeout errors are retryable; permanent failures (401/403/404/413/415/422, oversize) are not and fail immediately. Retries use a fixed delay, up to a configurable maximum.
• A source that still cannot be fetched becomes a document-level FAILURE carrying the real cause (built with build_public_error_item), so a batch reports partial_success and the other sources still convert, rather than the source being silently dropped.
• New knobs on RQOrchestratorConfig: max_task_retries (default 3) and retry_delay (default 5.0 seconds), matching the Ray engine's names and defaults.
• expand_task_sources gains an optional http_materializer callback and now also returns an origin-index list so callers can attribute conversion results back to the originating source. This incidentally fixes a latent mislabeling case where one source expands into several converter inputs (S3). The Ray and local callers are updated for the new return shape.

Files

• docling_jobkit/convert/http_retry.py — new: fetch_http_source_with_retry and build_http_failure_document.
• docling_jobkit/convert/source_expansion.py — optional http_materializer, origin indices.
• docling_jobkit/orchestrators/rq/worker.py — materialize HTTP sources, collect per-source failures, merge them into the results.
• docling_jobkit/orchestrators/rq/orchestrator.py — max_task_retries / retry_delay on RQOrchestratorConfig.
• docling_jobkit/orchestrators/ray/serve_deployment.py, docling_jobkit/orchestrators/local/worker.py — updated for the new expand_task_sources return shape.
• tests/test_http_retry.py — new tests; tests/test_rq_orchestrator.py, tests/test_s3_source_orchestrators.py — updated for the new signatures.

Consistency with the Ray engine

The intent is to reuse what the Ray engine already established rather than invent a parallel mechanism:

• Same classifier: classify_public_task_failure (which classify_ray_public_task_failure wraps) and the same retryable flag.
• Same retry shape as _run_with_retry: a fixed retry_delay between attempts, max_task_retries additional attempts after the first.
• Same per-document failure representation as _build_materialization_failure_result: an ExportableDocument with status=ConversionStatus.FAILURE, errors=[build_public_error_item(exc)] and source_uri=source_to_public_uri(source).
• Same config names and defaults as the Ray engine's max_task_retries / retry_delay.

The retry uses a fixed delay (not exponential back-off) deliberately, to match Ray's retry_delay semantics.

Behaviour change

Terminal HTTP fetch failures are now visible instead of silent. For a single-source task this means status: "failure" with the cause recorded; for a batch it means partial_success with a per-document failure. This is a deliberate improvement over the previous silent-success behaviour, but it is a behavioural change worth calling out for anyone currently relying on the old (empty-error) result.

Testing

Unit tests cover the retry matrix (retry-then-succeed, exhaustion, no-retry on 404/oversize, connection-error retry, zero-retries), the failure-document builder, the expand_task_sources seam, and the worker wiring (including a mixed good/bad batch producing a per-document failure).

Beyond the mocked unit tests, I validated the real fetch path against a local server (real httpx, real retry loop counting requests) and then ran the whole thing end-to-end against a running docling-serve API and RQ worker built from this branch. Against the same reproduction that produced the silent success above, the behaviour flips:

worker log:  retry 1/2 after 1.0s
             retry 2/2 after 1.0s

503 server:  3 GET attempts (1 initial + 2 retries)

result:      status "failure",
             errors[0] = { module_name: "SourceFetchError",
                           error_message: "... Server error '503 Service Unavailable' ...",
                           category: "source_unavailable" }

Related

• Original issue: RQ engine silently reports success for unfetchable HTTP sources (no error surfaced, no retry) docling-serve#648
• docling-serve companion PR: feat(rq): configurable HTTP source fetch retries docling-serve#650

[github-actions]

✅ DCO Check Passed

Thanks @dan-m62, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[codecov]

Codecov Report

❌ Patch coverage is 89.33333% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling_jobkit/orchestrators/rq/worker.py	76.92%	6 Missing ⚠️
docling_jobkit/convert/source_expansion.py	94.11%	1 Missing ⚠️
...cling_jobkit/orchestrators/ray/serve_deployment.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!