Skip to content

feat(crawlers): Resume a paced crawl at the request manager's ready_at - #2393

Open
Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:crawler-run-wake
Open

Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:crawler-run-wake

Conversation

@Mantisus

@Mantisus Mantisus commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Description

When the request manager reports waiting with a ready_at, the crawler schedules a wake-up for that moment, so a paced crawl resumes on the manager's clock instead of on the autoscaled pool's next poll. This tightens the dispatch cadence for ThrottlingRequestManager cooldowns and crawl-delays. Only the earliest pending ready_at is kept, and a pending wake-up is dropped when the run ends.

Issues

✍️ Drafted by Claude Code

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 3, 2026
@Mantisus Mantisus self-assigned this Nov 3, 2026
@Mantisus
Mantisus requested a review from vdusek November 3, 2026 22:37
@Mantisus
Mantisus marked this pull request as ready for review November 3, 2026 22:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant