9.5.0 regression: no way to use a batch system + auto generated job store location

[mr-c]

Use case: running the CWL conformance tests in parallel and thus not able to specify a unique --jobStore for each invocation

Previous to toil 9.5.0, then toil-cwl-runner would create one automatically:

toil/src/toil/cwl/cwltoil.py

Lines 5044 to 5045 in a9681fe

	if options.jobStore is None:
	jobstore = cwltool.utils.create_tmp_dir(tmp_outdir_prefix)

using --tmp-outdir-prefix or --tmpdir-prefix or tempfile.gettempdir() (which uses the first available of $TMPDIR, $TEMP, $TMP, /tmp, /var/tmp, or /usr/tmp)

However, since #5520 this functionality has been removed:

...
Test Workflow without outputs
Returned non-zero
Test 196 failed: toil-cwl-runner --singularity --disableCaching --disableProgress --defaultMemory=2G --maxMemory=2G --cwl-min-ram=2G --outdir=/scratch/PROJECT/USER/tmp//d51af243-00dc-43d0-8ce8-b2c087adc93e/outdir --log-dir=/scratch/PROJECT/USER/tmp//d51af243-00dc-43d0-8ce8-b2c087adc93e/logdir --tmpdir-prefix=/scratch/PROJECT/USER/tmp//d51af243-00dc-43d0-8ce8-b2c087adc93e/tmpdir --tmp-outdir-prefix=/scratch/PROJECT/USER/tmp//d51af243-00dc-43d0-8ce8-b2c087adc93e/tmpoutdir --workDir=/scratch/PROJECT/USER/tmp//d51af243-00dc-43d0-8ce8-b2c087adc93e/workdir --coordinationDir=/scratch/PROJECT/USER/tmp//d51af243-00dc-43d0-8ce8-b2c087adc93e/coordination --batchSystem slurm --bypass-file-store --outdir=/scratch/PROJECT/USER/tmp/d51af243-00dc-43d0-8ce8-b2c087adc93e/e98296cb-3edf-4e4b-a1d9-0758f8cd164b/tmprnnur56_ --quiet v1.0/anon_enum_inside_array.cwl v1.0/anon_enum_inside_array.yml
Test an anonymous enum inside an array inside a record
Returned non-zero
Length of workDir path "/scratch/PROJECT/USER/tmp/d51af243-00dc-43d0-8ce8-b2c087adc93e/workdir" is 85 characters.  Consider setting a shorter path with --workPath or setting TMPDIR to something like "/tmp" to avoid overly long paths.
Traceback (most recent call last):
  File "/project/PROJECT/USER/cwl-mpi/cwl-conformance-tests/runs/venv-toil/bin/toil-cwl-runner", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File "/project/PROJECT/USER/cwl-mpi/cwl-conformance-tests/runs/venv-toil/lib/python3.13/site-packages/toil/cwl/cwltoil.py", line 4273, in main
    options.jobStore = generate_default_job_store(
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        options.batchSystem, options.provisioner, jobstore, "cwl"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/project/PROJECT/USER/cwl-mpi/cwl-conformance-tests/runs/venv-toil/lib/python3.13/site-packages/toil/jobStores/utils.py", line 140, in generate_default_job_store
    raise NoAvailableJobStoreException(
    ...<2 lines>...
    )
toil.jobStores.utils.NoAvailableJobStoreException: slurm batch system doesn't work with a default job store because it requires a shared filesystem. Please specify a file job store on shared storage with --jobStore.
Test 197 failed: toil-cwl-runner --singularity --disableCaching --disableProgress --defaultMemory=2G --maxMemory=2G --cwl-min-ram=2G --outdi...

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1867

[kinow]

Thank you @mr-c ! 🙏

The log is from my execution with cwltest on CSC LUMI. It's being launched from a login node, submitting jobs with -j 8 via cwltest.

cwltest launches 8 toil-cwl-runner ... --batchSystem slurm --bypass-file-store..., each running a different CWL conformance test.

Running things in serial mode works, and for v1.0 takes 1-2 hours. But for v1.2 it takes a really long time.

Cheers

[adamnovak]

I think we changed the behavior on purpose, because we didn't think either the temp outdir or the temp directory made sense as a place for a default job store on an HPC system, since they're unlikely to be shared. So we decided you ought to need to manually set a job store on HPC systems, because that was the hardest for the user to get wrong in a confusing way.

#5560 will add a new and better way to automatically pick a job store, by letting you set a directory for the job store to be created in.

[mr-c]

@adamnovak Okay, and I already see that a follow-up has been flagged to allow using the same --runDir multiple times simultaneously

Known follow-up: the job store path derived from --runDir is fixed, so concurrent workflows sharing one --runDir will collide. Flagged with TODOs pending a fix.

[kinow]

Maybe what would work is to have a way to say here's a location where we can use job stores, but I don't care the actual job store location... just don't use the file system, and please create one automatically with some random name, within this base run dir...