Add BlueBEAR cluster config #745
Conversation
There was a problem hiding this comment.
A couple of minor comments, if you disagree we can proceed.
Note, so you don't ave to wait for approval for the tests to run - consider joining the nf-core github organisation, instructions here: https://nf-co.re/join
conf/bluebear.config
Outdated
| // Cluster-specific params | ||
| project = slurm_job_account |
There was a problem hiding this comment.
Having non-pipeline parameters here may fail some (and increasingly more pipelines).
A better approach is to tell users to set a bash environment variable with the information needed, and yse that information (in this case down in your clusterOptions).
For example, see
configs/conf/pawsey_setonix.config
Line 26 in 2900904
There was a problem hiding this comment.
Thanks, that sounds good. In fact System.getenv("SLURM_JOB_ACCOUNT") could be substituted there, since it is unlikely someone would want individual processes to be submitted under a different account.
docs/bluebear.md
Outdated
| BlueBEAR comes with Apptainer installed for running processes inside containers. Our configuration makes use of this | ||
| when executing processes. | ||
|
|
||
| We advise you create a directory in which to cache images using the following environment variable. For example, |
There was a problem hiding this comment.
Maybe mention it's a good idea to put the export command in a .bashrc or similar?
docs/bluebear.md
Outdated
| Nextflow caches pipeline files in a work directory (default is `work`). This is useful if you need to resume a job | ||
| after a failure (using the `-resume` option). However, the `work` directory can fill up quickly. Regularly cleaning the | ||
| work directory avoids filling up your project quota. |
There was a problem hiding this comment.
You could consider by default adding the cleanup = true 1 option to have the emptying of the work/ happen by default, and to override this use the debug profile in addition to the institutional profile.
For examples: https://github.com/search?q=repo%3Anf-core%2Fconfigs%20cleanup%20%3D%20true&type=code
Footnotes
There was a problem hiding this comment.
I was a little concerned about the warning in the Nextflow docs about this. It suggests cleanup = true prevents use of -resume. If cleanup only occurs on a successful run, is -resume still possible after a failed run?
There was a problem hiding this comment.
It only prevents -resume if the pipeline completes with exit code 0 (so no errors).
If the pipeline hits an error (exit code >1), all the files in work are retained and will not be deleted
The only case that it might be problematic is if someone is playing with parameters in later stages of a pipeline, and want to resume from midway but with a different parameter, yes in this case this resume would not work (if that makes sense)
There was a problem hiding this comment.
Ah fantastic, I misunderstood the docs slightly. This sounds like a good addition, thanks!
|
Good to go @alexlyttle ! With your new found nf-core org membership, you may do the honours! (At least I think you should be able to merge yourself now 😅) |
name: BlueBEAR
about: A new cluster config
Please follow these steps before submitting your PR:
[WIP]in its titlemasterbranchSteps for adding a new config profile:
conf/directorydocs/directorynfcore_custom.configfile in the top-level directoryREADME.mdfile in the top-level directoryprofile:scope in.github/workflows/main.yml