Skip to content

fix: count async filter-chain rejection once per URL - #2333

Open
workstonedai-collab wants to merge 1 commit into
unclecode:developfrom
workstonedai-collab:fix/filter-chain-rejection-stats
Open

workstonedai-collab wants to merge 1 commit into
unclecode:developfrom
workstonedai-collab:fix/filter-chain-rejection-stats

Conversation

@workstonedai-collab

Copy link
Copy Markdown

Summary

Count an asynchronously rejected URL once in FilterChain.stats, regardless of how many filters reject it.

The synchronous path counts rejected URLs, while the asynchronous path currently adds results.count(False). Two rejecting async filters therefore produce total_urls=1, rejected_urls=2. A falsy result such as None rejects the URL through all(results) but records no rejection at all.

The change increments the rejection counter once when the gathered results reject the URL. Filter execution and the chain's return value are unchanged.

Validation

  • Added three offline regression tests: multiple async rejections, a falsy async result, and passing/rejected calls preserving total = passed + rejected.
  • Against the untouched develop implementation: all three new tests fail.
  • With this patch: python -m pytest tests/deep_crawling/test_filter_chain_stats.py -q → 3 passed.
  • Black check for the added test and git diff --check passed.

PR #2305 concerns cleanup of pending async filters when a synchronous filter rejects. This patch addresses URL statistics after async evaluation.

Prepared with Codex and verified using actual filter execution; no browser, network crawling, credentials or private data are needed for the tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant