Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Comment out all references to 'run' queue #3

Closed
roddtalebi opened this issue Aug 9, 2018 · 3 comments
Closed

Comment out all references to 'run' queue #3

roddtalebi opened this issue Aug 9, 2018 · 3 comments

Comments

@roddtalebi
Copy link
Member

if args.run:

comment out any code dealing with the 'run' queue

Couldn't get the sql server to update quickly enough after one sql client adds a new point so that another sql client can change the state from 'queue' to 'running' after a worker picks it off the 'main' queue...couldn't invest enough time to resolve this issue so commenting out functionality

@roddtalebi
Copy link
Member Author

Commented out 'run' queue function in sql_client.py and the line that adds to the 'run' queue from worker.py
The rediswq.py still has the run queue active though; i'd rather have that available; might get too messy to comment it out

Thinking of keeping this issue open though so that it can potentially get addressed in the future

@roddtalebi roddtalebi added the help wanted Extra attention is needed label Aug 9, 2018
@roddtalebi
Copy link
Member Author

Future work:

Better understand Cloud SQL to see how much of a 'delay' we can expect before a second sql-client instance can read the point added by the first sql-client instance. From that, decide whether we can re-add the 'run' queue functionality or if it has to be tossed completely.

Problem:
The 'main' sql-client would add points to the table and to the 'main' queue; if a worker immediately picks it off the 'main' queue and adds it to the 'run' queue so it can change states on the table from state='queue' to ='running', often the 'run' sql-client won't see any point in the table with that hash...likely because it isn't seeing a 'new-enough' vision of the table.
We even added 'sleep' statements so that it would check the table again for the point after a quick rest, but that didn't improve. Something else is going on with how these instances communicate with the server and what view they see that we don't know about.
I suppose the easy solution would be to have the same 'main' client also be the 'run' client but with thousands of nodes, that might make a bottleneck.

@roddtalebi roddtalebi removed the help wanted Extra attention is needed label Aug 9, 2018
@roddtalebi
Copy link
Member Author

I created a new issue (#6) for fixing 'run' client since this current issue is more about removing all 'run' client references

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant