Can we use more threads?

[charalamm]

Hello,

I am using stackstac for some simple index calculations and I have noticed that the most time-consuming part of the process is the image downloading, which as far as I know is a case that can be improved with threading.

At the same time I see that the whole processing takes more time with less CPU cores but these cores are used around at 20-30%, which again looks like the threading done is not efficient.

I know that the parallel processing is handled by dask but I have not found any solution for such cases from the dask docs and it looks like people using dask are implementing their own solutions (?).

Is there something that I can do as a user to improve performance?

If a PR could help I would be glad to work on it with some guidance bc I am not familiar with dask.

[charalamm]

The slowest step in my procedure was the downloading part and adding more num_workers with scheduler="threads" in .compute of dask helped a lot.

So I did something like:

dask.diagnostics.ProgressBar():
    ndvi = ndvi.compute(num_workers=50, scheduler="threads")