You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Disallow: /*? in robots.txt blocks the whole website. #2225
The rule in robots.txt disallows scraping urls with query params, the rest of the page works as expected.
Current Behavior
crawl4ai 0.9.x patches the stdlib urllib.robotparser to support wildcards, but it still matches rules against normalized paths (paths where the query string has already been stripped). Ecommerce sites commonly ship (this is advised by multiple SEO blogs online):
Disallow: /*?
The intent is to block only URLs carrying a query string, e.g. /?s=search. After normalization the ? is gone, so the rule collapses to Disallow: /* and the entire site reads as disallowed.
Is this reproducible?
Yes
Inputs Causing the Bug
- Disallow: /*?in robots.txt
When check_robots_txt=True
Steps to Reproduce
- Scrape any website (check_robots_txt=True) with Disallow: /*?in robots.txt (test webserver is sufficient in this case)
Activity
Nalhin commentedon Sep 5, 2026
Suggested fix: #2229. I'd love to contribute to getting this fixed ASAP :).
SohamKukreti commentedon Sep 15, 2026
Reviewed PR #2229 for this issue. Two small nits but otherwise LGTM 👍
fix(robots): don't rewrite bare-'?' rules on Python 3.14+ either
fix(robots): don't patch robotparser on Python 3.14+ (#2278)