Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up new sync algorithm #2158

Open
shawn-zil opened this issue Jan 14, 2025 · 0 comments
Open

Speed up new sync algorithm #2158

shawn-zil opened this issue Jan 14, 2025 · 0 comments
Assignees
Labels
Agate Required for mainnet launch

Comments

@shawn-zil
Copy link
Contributor

shawn-zil commented Jan 14, 2025

Related to #2039
If we assume that Phase 2 saturates the CPU, then there is no speed-up necessary beyond the existing mechanism of firing a speculative retrieval request for blocks.

Phase 1 can be sped up differently for different versions:

  1. For V1, we can split the metadata into N macro-segments, and fire requests for one micro-segment to each N peer. Eventually, the responses should sync up to either history, or some other segment. Effectively, splitting up the entire history into N macro-segments and giving each one the same Phase 1 treatment. Speeds things up by exploiting more concurrency, while maintaining payload sizes. Requires at least N peers before starting.
  2. For V2, we retrieve a larger segment e.g. N * batch_size; and then split the retrieved metadata into N segments (with the same peer) into the metadata. Effectively requesting a much larger macro-segment at a time, and then splitting it into several micro-segments. Speeds things up by retrieving more metadata per request.
@shawn-zil shawn-zil self-assigned this Jan 14, 2025
@shawn-zil shawn-zil added the Agate Required for mainnet launch label Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Agate Required for mainnet launch
Projects
None yet
Development

No branches or pull requests

1 participant