[GSoC] Parallelisation of AnalysisBase with multiprocessing and dask#4162

Merged

orbeckst merged 288 commits intoMDAnalysis:developfrom marinegor:feature/dask-0

Aug 16, 2024

+2,373-362

This pull request is big! We're only showing the most recent 250 commits

Commits on Jun 15, 2023

Remove _scheduler attribute and make dask-based tests run properly
Egor Marin
committed

Commits on Jun 20, 2023

Refactor scheduler usage
Egor Marin
committed
Add multiple workers in dask for testing
Egor Marin
committed

Commits on Jun 21, 2023

Refactor _setup_bslices and add processes to dask scheduler kwargs
Egor Marin
committed

Commits on Jun 22, 2023

Create frame_indices and trajectory for each bslice during _setup_bslices
Egor Marin
committed
Use explicit initialisation of timeseries wiith zeros
Egor Marin
committed
Add non-trivial _parallel_conclude function
Egor Marin
committed
Fix tests for new dask fixture
Egor Marin
committed

Commits on Jun 23, 2023

Add type-matching _parallel_conclude
Egor Marin
committed

Commits on Jun 25, 2023

Add fixtures to test combinations of dask and multiprocessing
Egor Marin
committed
dask and multiprocessing works in test_atomicdistances.py
Egor Marin
committed
Fix bug in results is np.ndarray codepath
Egor Marin
committed
Add _setup_scheduler raising NotImplemented error in align.py::AverageStructure
Egor Marin
committed
dask and multiprocessing schedulers to test_align.py
Egor Marin
committed
dask scheduler for test_contacts.py and test for incompatibility with multiprocessing
Egor Marin
committed
dask and multiprocessing scheduler for test_density.py
Egor Marin
committed
Add _parallel_conclude implementation for dielectric
Egor Marin
committed
dask and multiprocessing schedulers for test_dielectric.py
Egor Marin
committed
dask and multiprocessing schedulers for test_diffusionmap.py
Egor Marin
committed
Add NotImplementedError for parallel schedulers in dihedrals.py
Egor Marin
committed
only current scheduler for test_dihedrals.py
Egor Marin
committed
dask and multiprocessing tests for test_encore.py -- but some fail because of RMSF module
Egor Marin
committed
Add NotImplementedError for _setup_scheduler in gnm.py
Egor Marin
committed
Add NotImplementedError for _setup_scheduler in helix_analysis.py
Egor Marin
committed
current process scheduler for test_helix_analysis.py
Egor Marin
committed
dask and multiprocessing schedulers for test_hole2.py
Egor Marin
committed
Add NotImplementedError in for not-None schedulers
Egor Marin
committed
current process scheduler and test for failing non-current ones in test_hydrogendbonds_analysis.py
Egor Marin
committed
current process only scheduler and failing test for others in test_lineardensity.py
Egor Marin
committed
Add NotImplementedError for non-current process schedulers
Egor Marin
committed
current process scheduler only and failing tests for non-current ones in test_msd.py
Egor Marin
committed
Add NotImplementedError for non-current process schedulers
Egor Marin
committed
Fix scope of fixtures
Egor Marin
committed
Add NotImplemented error for all non-current process schedulers
Egor Marin
committed
only current process scheduler and failing tests for test_nucleicacids.py
Egor Marin
committed
dask and multiprocessing schedulers for test_persistentlength.py
Egor Marin
committed
Add _parallel_conclude implementation
Egor Marin
committed
dask and multiprocessing schedulers for test_psa.py
Egor Marin
committed
Add _parallel_conclude implementation for RDF and RDF_S
Egor Marin
committed
dask and multiprocessing schedulers for test_rdf_s.py
Egor Marin
committed
dask and multiprocessing schedulers for test_rdf.py
Egor Marin
committed
Add NotImplementedError for RMSD and RMSF classes
Egor Marin
committed
only local process scheduler and failing tests for others for test_rms.py
Egor Marin
committed
current process scheduler only and failing test for others for test_wbridge.py
Egor Marin
committed
Add NotImplementedError in _setup_scheduler
Egor Marin
committed
Add more clear message during exception
Egor Marin
committed

Commits on Jun 29, 2023

Add timeseries aggregation function
Egor Marin
committed
dask and multiprocessing scheduler for most of the test_base.py testcases
Egor Marin
committed
dask and multiprocessing schedulers for test_rms.py::TestRMSD
Egor Marin
committed
Add NotImplementedError for pca and rms
Egor Marin
committed
dask and multiprocessing schedulers for test_bat
Egor Marin
committed
dcurrent process scheduler for test_gnm.py
Egor Marin
committed
dcurrent process scheduler for test_pca.py
Egor Marin
committed
Fix rmsf-related scheduler usage to only current process scheduler
Egor Marin
committed
remove fixme marks
Egor Marin
committed

Commits on Jun 30, 2023

Switch to enumerate in _compute main loop and fix code review comments
Egor Marin
committed

Commits on Jul 2, 2023

Add dask to CI setup actions
Egor Marin
committed
Remove local scheduler for progressbar test
Egor Marin
committed
Add installation with dask as asetup option
Egor Marin
committed
fix hole2 tests for -- implement only current scheduler and add failing test
Egor Marin
committed

Commits on Jul 3, 2023

fix progressbar test by changing order of ProgressBar and enumerate
Egor Marin
committed

Commits on Jul 4, 2023

use only frame indices and frames in _setup_bslices after writing a blogpost
Egor Marin
committed
Refactor _setup_bslices: move enumerate to numpy and fuse logic in defining type of input
Egor Marin
committed
Add documentation to AnalysisBase._parallel_conclude()
Egor Marin
committed

Commits on Jul 20, 2023

add functional-like interface draft
Egor Marin
committed

Commits on Jul 23, 2023

Implement proper Client class, separating computations from AnalysisBase
Egor Marin
committed
FINALLY implement working one-time dask cluster setup in kwargs of a client
Egor Marin
committed
Correct tests accordingly
Egor Marin
committed

Commits on Jul 24, 2023

Separately process case of only one remote worker
Egor Marin
committed
Add available_schedulers to AverageStructure
Egor Marin
committed
Use automatic fixture for AverageStructure
Egor Marin
committed
Add fixture for AverageStructure
Egor Marin
committed
Add fixture for AtomicDistances
Egor Marin
committed
Change default available_backends to all implemented in Client
Egor Marin
committed
Limit available backends for AverageStructure
Egor Marin
committed
Add fixture for BAT
Egor Marin
committed
Add fixture tests to Contacts
Egor Marin
committed
Fix n_workers check and boolean frames handling
Egor Marin
committed

Commits on Jul 27, 2023

Fix performance of backend="dask"
Egor Marin
committed
Add available_backends for Contacts
Egor Marin
committed
Remove _setup_scheduler
Egor Marin
committed
Use client fixture for Contacts
Egor Marin
committed
Use client fixture for RMSD/RMSF
Egor Marin
committed
Revert files to their state in develop
Egor Marin
committed
Delete files_for_undoing_changes.txt
Egor Marin
authored
Delete conftest.py
Egor Marin
authored
Delete parallel_analysis_demo.ipynb
Egor Marin
authored
Clean up notebook
Egor Marin
committed
remove notebook
Egor Marin
committed
Limit available schedulers in RMSF
Egor Marin
committed
Split test in two due to failing with "expectation" parametrization
Egor Marin
committed
Add fixture generator and fixtures for test_base and test_rms
Egor Marin
committed
Add dask to pyproject.toml
Egor Marin
committed
Return computation groups explicitly
Egor Marin
committed

Commits on Aug 1, 2023

Merge branch 'develop' of https://github.com/MDAnalysis/mdanalysis into feature/parallel_backend
Egor Marin
committed

Commits on Aug 2, 2023

Fix dask position in setup-deps/action.yaml
Egor Marin
committed
Add dask[distributed] to mdanalysis[parallel] installation
Egor Marin
committed
Undo autoformatter
Egor Marin
committed

Commits on Aug 6, 2023

Manually define available_backends for RMSD class
Egor Marin
committed
Create separate "parallel" entry
Egor Marin
committed
Add is_installed function to utils
Egor Marin
committed
Add dict-based validatdion and computation logic for ParallelExecutor
Egor Marin
committed
Add tests for ParallelExecutor
Egor Marin
committed
Add documentation for "apply" method of ParallelExecutor
Egor Marin
committed
Correct dask.distributed name
Egor Marin
committed
Use chunksize=1 instead of explicit Pool in _compute_with_dask
Egor Marin
committed
Remove unnecessary function in conftest
Egor Marin
committed
Fix function to retrieve dask client if dask is not installed
Egor Marin
committed
Fix base tests when dask is not installed
Egor Marin
committed

Commits on Aug 8, 2023

Use new LocalCluster every time
Egor Marin
committed
Fix client/backend logic
Egor Marin
committed
Add documentation to a silly square function
Egor Marin
committed
Switch to package-wise autouse fixture for dask.distributed.Client
Egor Marin
committed
Add explicit result() when computing with cluster
Egor Marin
committed

Commits on Aug 17, 2023

Fix codereview
Egor Marin
committed
Replace list with tuple in available_backends for RMSD
Egor Marin
committed
Remove unnecessary get_running_dask_client
Egor Marin
committed
Implement fixture injection for subclasses testing
Egor Marin
committed
Add warnings filters
Egor Marin
committed
Fix backend check when client is present
Egor Marin
committed
Return get_runnning_dask_client function
Egor Marin
committed

Commits on Aug 18, 2023

Change dask fixture scope
Egor Marin
committed
Close LocalCluster to avoid trillions of logs
Egor Marin
committed
Implement ResultsGroup based aggregation instead of type matching
Egor Marin
committed
Add non-default _get_aggregator() to RMS and Base classes
Egor Marin
committed

Commits on Aug 19, 2023

Mark test_multiprocessing.py::test_creating_multiple_universe_without_offset as skipped
Egor Marin
committed
Restore failing test
Egor Marin
committed

Commits on Aug 21, 2023

Make aggregation functions static methods of ResultsGroup
Egor Marin
committed
Merge branch 'feature/dask-0' of github.com:marinegor/mdanalysis into feature/parallel_backend
Egor Marin
committed
Remove test skip
Egor Marin
committed

Commits on Aug 24, 2023

Move parallel part into a separate file
Egor Marin
committed

Commits on Aug 31, 2023

Fix imports
Egor Marin
committed

Commits on Sep 5, 2023

Proof of concept for duck-typed backends
Egor Marin
committed
Remove unused code
Egor Marin
committed

Commits on Sep 12, 2023

Commits on Sep 13, 2023

Change typing to py3.9 compatible syntax
Egor Marin
committed
Add _is_parallelizable to AnalysisFromFunction
Egor Marin
committed
Remove dask[distributed] even as an optional dependency
Egor Marin
committed
Update documentation
Egor Marin
committed
Remove function to get running dask client
Egor Marin
committed
Remove unused code from analysis/conftest.py
Egor Marin
committed
Fix documentation and minor issues from codereview
Egor Marin
committed
Update package/MDAnalysis/analysis/rms.py

Egor Marin
and
IAlibay
authored
Merge branch 'feature/dask-0' of github.com:marinegor/mdanalysis into feature/dask-0
Egor Marin
committed
Add more backend validation tests and fix autoformatter issues
Egor Marin
committed

Commits on Sep 15, 2023

Start implementing correct result sizes in separate computation groups
Egor Marin
committed
Continue working: diffusionmap and PCA tests fail
Egor Marin
committed

Commits on Jan 10, 2024

Merge remote-tracking branch 'upstream/develop' into feature/appropriate-sized-results-in-parallel
marinegor
committed

Commits on Jan 12, 2024

Commits on Jan 21, 2024

Fix merge conflicts
marinegor
committed

Commits on Jan 22, 2024

Commits on Jan 24, 2024

Fix bug with default progressbar_kwargs being dict
marinegor
committed

Commits on Jan 30, 2024

Merge branch 'develop' into feature/dask-0
RMeli
authored

Commits on Jan 31, 2024

Apply suggestions from code review

authored

Commits on Feb 17, 2024

Merge remote-tracking branch 'origin/feature/dask-1' into feature/parallelization
marinegor
committed

Commits on Mar 4, 2024

Merge branch 'develop' into feature/dask-0
RMeli
authored

Commits on Mar 6, 2024

Apply suggestions from code review

Egor Marin
and
RMeli
authored
Apply suggestions from code review

Egor Marin
and
RMeli
authored

Commits on Apr 19, 2024

Commits on Apr 21, 2024

Update dask installation test after exception text changed
marinegor
committed

Commits on Apr 28, 2024

Merge branch 'develop' into feature/dask-0
yuxuanzhuang
authored

Commits on May 1, 2024

Merge branch 'feature/dask-0' of github.com:marinegor/mdanalysis into dask-0
orbeckst
committed

Commits on May 4, 2024

edited documentation for parallelization
orbeckst
committed

Commits on May 5, 2024

analysis top level docs fixes
orbeckst
committed

Commits on May 20, 2024

Commits on May 22, 2024

Commits on Jun 11, 2024

Commits on Jun 12, 2024

Add client_RMSD explanation
marinegor
authored

Commits on Jun 13, 2024

versioninformation markup fix in base.py
orbeckst
authored

Commits on Jun 18, 2024

Merge branch 'develop' into feature/dask-0
orbeckst
authored

Commits on Jul 29, 2024

Merge branch 'develop' into feature/dask-0
marinegor
authored

Commits on Aug 7, 2024

Update package/doc/sphinx/source/documentation_pages/analysis/parallelization.rst

orbeckst
and
IAlibay
authored

Commits on Aug 11, 2024

Update testsuite/MDAnalysisTests/analysis/test_backends.py

marinegor
and
RMeli
authored

Commits on Aug 15, 2024

minor text fixes
orbeckst
authored

Commits on Aug 16, 2024