Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Figure out and document upper limit for restore_intellient #19

Open
bethac07 opened this issue Apr 20, 2022 · 0 comments
Open

Figure out and document upper limit for restore_intellient #19

bethac07 opened this issue Apr 20, 2022 · 0 comments

Comments

@bethac07
Copy link
Contributor

From here(needs jump repo access)

Yeah looking at CPU utilization I think I'm right; even though AFAICT the workers call from the main API is being passed correctly into tqdm, it was not using 95 workers - look at the bump in CPU and network out in the last third there when I start two more tmux sessions with 7 total more calls to the python script.
We certainly may hit Amazon's API limit eventually, the not-as-ideal part with this script is that we won't know until the end of each one how many are errors (but we can scrape them from the CSV and try again), but I'm just glad that it doesn't seems like 500-700 API requests a minute is a hard cutoff!
image

I'm not sure if this is a tqdm hard cutoff or what; I don't think it's worth trying to increase if it's not SUPER easy to do, because it's easy to solve with parallel, but we should at least dig out the reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant