Explore optimization of machine learning pipeline

[jessept]

The current classifier pipeline takes a long time to fit and may be fitting the same model multiple times. This should be looked at in hopes of finding some low-hanging fruit in performance gains.

[dhimmel]

The pipeline used to be much faster when we (incorrectly) did feature selection and standardization prior to cross validation (grid search). The issue is that sklearn reperforms these tranformations verbatim when they could be memoized: see scikit-learn/scikit-learn#7536 (comment).

There are two potential solutions:

1. Use grid search from dask-learn (dklearn). dask-learn uses dask in the background. I'm excited about dask, but dask-learn development appears to have stalled and the pull request to include dask-learn in dask petered out. However, it may still be functional.

2. @jnothman may know of a solution based on Finding which features are passed to the final estimator of an sklearn pipeline scikit-learn/scikit-learn#7536 (comment) where he said:

   we've seen a couple of attempted contributions, as well as my generic remember_model wrapper which I've never formally submitted as a PR

[jnothman]

Pipeline memoising has been implemented in:

• WIP allow Pipeline to memoize partial results scikit-learn/scikit-learn#2086
• ENH add joblib caching to Pipeline scikit-learn/scikit-learn#3951

My more generic model memoiser requires scikit-learn/scikit-learn#5080, which I may push for again at some point :)

[jnothman]

I think there's still potential for something like scikit-learn/scikit-learn#3951 to be merged.