GSoC 2023: Dask fixes and development.

[adi611]

I tried running the example as mentioned in #663 on Dask, and I got the expected result: the required Output.txt and Image.png files. Here is my code:

import nest_asyncio
import pydra

nest_asyncio.apply()

if __name__ ==  '__main__':
    cmd1 = 'pdftotext'
    args = ['/mnt/c/Users/Aditya/OneDrive/Desktop/GSoC/pydra_codes/Week_1_2/content/Lorem_ipsum.pdf', '/mnt/c/Users/Aditya/OneDrive/Desktop/GSoC/pydra_codes/Week_1_2/content/Output.txt']
    
    shelly1 = pydra.ShellCommandTask(name='shelly1', executable=cmd1, args=args)
    print('cmndline = ', shelly1.cmdline)
    
    cmd2 = 'wordcloud_cli'
    args2 = ['--text', '/mnt/c/Users/Aditya/OneDrive/Desktop/GSoC/pydra_codes/Week_1_2/content/Output.txt', '--imagefile', '/mnt/c/Users/Aditya/OneDrive/Desktop/GSoC/pydra_codes/Week_1_2/content/Image.png']
    
    shelly2 = pydra.ShellCommandTask(name='shelly2', executable=cmd2, args=args2)
    print('cmndline = ', shelly2.cmdline)
    
    wf1 = pydra.Workflow(name = 'wf1', input_spec=['x'], x = 'dummy')
    
    wf1.add(shelly1)
    wf1.add(shelly2)
    
    wf1.set_output([('out_s1', wf1.shelly1.lzout.stdout), ('out_s2', wf1.shelly2.lzout.stdout)])
    with pydra.Submitter(plugin='dask') as sub:
        sub(wf1)

    wf1.result()

I worked on the code before checking the discussion #664 which mentioned using pytest and relative paths and hence I was unable to do it for this example.

[djarecka]

@adi611 - I've checked the pytest command after our discussion, and the option to pytest was correct --dask, but I just run it outside the pydra/engine directory (and didn't point it to it), and the --dask option is defined in conftest that is inside pydra/engine, so you can run pytest -v --dask pydra/engine.

you can also start from running for a specific test file, e.g. pytest --dask -v pydra/engine/tests/test_shelltask.py or for specific test.

I tried quckly, and some tests, as expected, are failing, so you should try to figure out how to fix the dask worker.

Let me know if you have questiosn

[adi611]

Thank you for your help with this. I have been running pytest -v --dask pydra/engine and I always get stuck at the same task pydra/engine/tests/test_node_task.py::test_task_state_comb_1[dask] after a RERUN. Please let me know if this is an issue in your system as well.

====================================================== short test summary info ======================================================
FAILED pydra/engine/tests/test_node_task.py::test_task_state_1[dask-list] - AttributeError: 'tuple' object has no attribute '_run'
FAILED pydra/engine/tests/test_node_task.py::test_task_state_1[dask-array] - AttributeError: 'tuple' object has no attribute '_run'
FAILED pydra/engine/tests/test_node_task.py::test_task_state_comb_1[cf] - OSError: [Errno 24] Too many open files
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/home/aditya/.local/lib/python3.8/site-packages/filelock/_api.py:199: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
=========================== 3 failed, 219 passed, 74 skipped, 22 warnings, 7 rerun in 1425.20s (0:23:45) ============================

Since this seems to be limited to only test_node_task.py, I tried running the files individually and so far I encountered no such issue. The pytest coverage for test_shelltask.py:

====================================================== short test summary info ======================================================
FAILED pydra/engine/tests/test_shelltask.py::test_shell_cmd_3[dask] - AttributeError: 'tuple' object has no attribute '_run'
============================ 1 failed, 36 passed, 124 skipped, 8 warnings, 2 rerun in 137.95s (0:02:17) =============================
[<AsyncProcess Dask Worker process (from Nanny)>] process 11372 exit status was already read will report exitcode 255
[<AsyncProcess Dask Worker process (from Nanny)>] process 11369 exit status was already read will report exitcode 255

So far it seems like the same error is present in the failing dask tests: AttributeError: 'tuple' object has no attribute '_run'. I will be trying this for all the files and try to come up with a solution for the same.

[djarecka]

yes, I was also getting the error, so the worker has to be updated

[adi611]

Result for test_submitter.py:

======================================= 7 passed, 17 skipped, 6 warnings in 99.28s (0:01:39) ========================================
[<AsyncProcess Dask Worker process (from Nanny)>] process 302 exit status was already read will report exitcode 255
Task was destroyed but it is pending!

Result for test_workflow.py, stuck at pydra/engine/tests/test_workflow.py::test_wf_3nd_ndst_1[dask]:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/home/aditya/.local/lib/python3.8/site-packages/_pytest/logging.py:708: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
====================================== 14 passed, 45 skipped, 19 warnings in 384.25s (0:06:24) ======================================

Result for rest of the tests:

================================ 414 passed, 81 skipped, 2 xfailed, 4 warnings in 140.84s (0:02:20) =================================

[adi611]

I added some fixes to the DaskWorker (based on the ConcurrentFuturesWorker) and now there is no AttributeError: 'tuple' object has no attribute '_run' but some tests are still getting stuck on reruns (for dask), like pydra/engine/tests/test_workflow.py::test_wf_3nd_ndst_1[dask] RERUN and pydra/engine/tests/test_shelltask.py::test_shell_cmd_3[dask] RERUN. I will look further into this.

[adi611]

@djarecka and @ghisvail - As discussed, I checked for all the tests for Dask and only test_wf_3nd_ndst_1 in test_workflow.py hangs (link to comment and PR) and I created another PR for adding GitHub Actions workflow for Dask Tests (link to PR).

Please let me know any further steps I should take. Thank you.

[adi611]

@ghisvail - Thanks for the help. Here are my observations as I tried the above testing workflow on:

1. The current DaskWorker - As expected many dask tests failed [link]

2. The modified DaskWorker Fix DaskWorker: AttributeError: 'tuple' object has no attribute '_run' #673 - All dask tests successfully passes for Python 3.9, but some fail for versions 3.7 and 3.8 [link]

3. The modified DaskWorker Fix DaskWorker: AttributeError: 'tuple' object has no attribute '_run' #673 (while commenting out test_wf_3nd_ndst_1 in test_workflow.py) - All dask tests successfully passes for Python versions 3.7 and 3.9, but some fail for version 3.8 [link]

I am confused as to why Workflow 2 is failing a test test_wf_3nd_st_1 in test_workflow.py for Python 3.7 while Workflow 3 is passing that test for Python 3.7. The only difference between the workflows is the commented out test_wf_3nd_ndst_1 in test_workflow.py.

[adi611]

The main takeaway is that the Dask tests do pass successfully using the modified DaskWorker for Python 3.9 even without any change to the tests themselves.

[djarecka]

I think python3.8 is old enough that we can simply ignore it. Perhaps we could test for python 3.10 and 3.11. Have you had a chance?

[adi611]

@djarecka - Yes I checked it with different versions of Python on Google Colab and the GA workflow works fine for the Dask Tests.

• for python 3.8: [github gist]
• for python 3.9: [github gist]
• for python 3.10: [github gist]

The issue with the specific test test_wf_3nd_ndst_1[dask] seems to be limited to my system only.

[djarecka]

ok, perfect!

[djarecka]

@adi611 - did you submit a PR with your new GA workflow? I can't find it. We should just test it with GA

[adi611]

@adi611 - did you submit a PR with your new GA workflow? I can't find it. We should just test it with GA

I was waiting for the approval, I will open the PR now.

[djarecka]

ok, thank you, no need to wait to open PR :)

[adi611]

Ok, I will keep that in mind. Meanwhile I have opened the PR for the GA workflow for Dask tests - #677.