CWL job output files can end up in non-shared directories and cannot move between nodes with `--bypass-file-store`

[adamnovak]

@kinow reported some CWL test failures on an HPC system and attached a JUnit XLM log file.

Most of the problems were like (note that this is still XML):

Traceback (most recent call last):
  File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/toil/src/toil/cwl/cwltoil.py", line 1591, in visit
    st = os.lstat(deref)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp3cfsnkwd/foo'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/netapp2/Store_uni/home/ulc/cursos/[USER]/micromamba/envs/toil/bin/toil-cwl-runner", line 6, in <module>
    sys.exit(main())
             ~~~~^^
  File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/toil/src/toil/cwl/cwltoil.py", line 5351, in main
    toilStageFiles(
    ~~~~~~~~~~~~~~^
        toil,
        ^^^^^
    ...<3 lines>...
        log_level=logging.INFO,
        ^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/toil/src/toil/cwl/cwltoil.py", line 2653, in toilStageFiles
    pm = ToilPathMapper(
        jobfiles,
    ...<3 lines>...
        stage_listing=True,
    )
  File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/toil/src/toil/cwl/cwltoil.py", line 1327, in __init__
    super().__init__(referenced_files, basedir, stagedir, separateDirs=separateDirs)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/netapp2/Store_uni/home/ulc/cursos/[USER]/micromamba/envs/toil/lib/python3.13/site-packages/cwltool/pathmapper.py", line 88, in __init__
    self.setup(dedup(referenced_files), basedir)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/netapp2/Store_uni/home/ulc/cursos/[USER]/micromamba/envs/toil/lib/python3.13/site-packages/cwltool/pathmapper.py", line 196, in setup
    self.visit(
    ~~~~~~~~~~^
        fob,
        ^^^^
    ...<3 lines>...
        staged=True,
        ^^^^^^^^^^^^
    )
    ^
  File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/toil/src/toil/cwl/cwltoil.py", line 1568, in visit
    with SourceLine(
         ~~~~~~~~~~^
        obj,
        ^^^^
    ...<2 lines>...
        logger.isEnabledFor(logging.DEBUG),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/mnt/netapp2/Store_uni/home/ulc/cursos/[USER]/micromamba/envs/toil/lib/python3.13/site-packages/schema_salad/sourceline.py", line 255, in __exit__
    raise self.makeError(str(exc_value)) from exc_value
schema_salad.exceptions.ValidationException: [Errno 2] No such file or directory: '/tmp/tmp3cfsnkwd/foo'

This came from running:

toil-cwl-runner --clean=always --enable-dev --singularity --batchSystem=slurm --bypass-file-store --disableCaching --disableProgress --defaultMemory=2G --maxMemory=2G --cwl-min-ram=2G --jobStore=/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/common-workflow-language/d3dc3aa2-9075-4fdf-9282-0c8062635f44/ --outdir=/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/common-workflow-language/v1.0/4645115c-bc9d-43f2-a35b-3f67b4f4365f/ --log-dir=/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/common-workflow-language/v1.0/4645115c-bc9d-43f2-a35b-3f67b4f4365f/ --tmp-outdir-prefix=/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/common-workflow-language/v1.0/4645115c-bc9d-43f2-a35b-3f67b4f4365f/ --workDir=/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/common-workflow-language/v1.0/4645115c-bc9d-43f2-a35b-3f67b4f4365f/ --coordinationDir=/mnt/lustre/scratch/nlsas/home/ulc/cursos/[USER]/common-workflow-language/v1.0/4645115c-bc9d-43f2-a35b-3f67b4f4365f/ --outdir=/tmp/tmp3cfsnkwd --quiet v1.0/v1.0/record-output.cwl v1.0/v1.0/record-output-job.json

Note that we are bypassing the file store, and --outdir is set to the particular directory under /tmp where this file is not, and that we are failing to find it on the leader, I think when staging the output files for the workflow.

I suspect that some Toil jobs are trying to write to the workflow --outdir from the worker nodes, when the file store is being bypassed. Either we need to throw some kind of useful error when --bypass-file-store is combined with a non-shared --outdir, or we need to not quite bypass it all the way and make sure only the leader is responsible for shipping things from --tmp-outdir-prefix to --outdir.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1857

[kinow]

Thanks @adamnovak !