error pulling singularity image #175
Comments
|
@selmapichot Thank you for this fuller bug report! I can now see the following:
We have found that occasionally Nextflow workflows using Singularity are unable to pull and convert images reliably. This "no descriptor found for reference" should be intermittent and eventually go away after a couple of retries. Please advise if you find that is not the case. Note that the human variation workflow will need to download a few images to be able to run, so you may be seeing this error for different images. |
|
Many thanks Sam for your reply. I will retry running the workflow. |
|
Hi again, |
|
Had the same issue. Re-running the workflow using nf did not work for me. It seems that Nextflow is not able to pull the images but if you pull them manually it works. I ran the pull command manually each time the workflow failed and then stored the Singularity inside $NXF_SINGULARITY_CACHEDIR dir and it eventually worked
|
|
Many thanks for your suggestion :) can you please share the command that you used to pull manually the singularity images ? Many thanks, |
|
Apologies for the delay. If you are still having trouble this may be
because Nextflow may attempt to pull multiple images at once with
Singularity which can occasionally lead to a race condition with
simultaneous writes to the Singularity cache. You may find that running the
workflow again may simply avoid the race condition and the image downloads
will proceed without trouble.
As that has not been the case for you, you may wish to try:
- Disabling the Singularity/Apptainer cache
For Singularity this can be achieved by setting the environment variable
SINGULARITY_DISABLE_CACHE=True.
- Pulling the images yourself to avoid the race condition
You can run the Singularity command indicated in the error message yourself
(removing the .pulling.NNNN extension), in the directory where you keep
your images (usually set with NXF_SINGULARITY_CACHEDIR). Note that if you
are not already setting NXF_SINGULARITY_CACHEDIR, I would strongly advise
you to do so, to avoid downloading images for each workflow.
The command is in the Caused by part of your error message, for example:
Caused by:
Failed to pull singularity image
command: singularity pull --name
ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img.pulling.1713840376147
docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9
/dev/null
You will want to run (note the pulling.1713840376147 removed) in your
NXF_SINGULARITY_CACHEDIR:
singularity pull --name
ontresearch-wf-common-sha1c5febff9f75143710826498b093d9769a5edbb9.img
docker://ontresearch/wf-common:sha1c5febff9f75143710826498b093d9769a5edbb9
…On Sat, 27 Apr 2024, 22:20 Selma_LP, ***@***.***> wrote:
Many thanks for your suggestion :) can you please share the command that
you used to pull manually the singularity images ?
Is it ok to use the same $NXF_SINGULARITY_CACHEDIR dir for multiple
samples ?
Many thanks,
S
—
Reply to this email directly, view it on GitHub
<#175 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAIN6OW3TMQMRFZ3IE3FTDLY7QJBHAVCNFSM6AAAAABGUNOZZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRGE4DKMZRGI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Operating System
Other Linux (please specify below)
Other Linux
rocky linux 8.9
Workflow Version
v2.0.0-g52e3698
Workflow Execution
Command line (Cluster)
Other workflow execution
No response
EPI2ME Version
No response
CLI command run
nextflow run epi2me-labs/wf-human-variation --bam "/rds/project/nanopore/TB23.00177/TB23.00177_T4/SLX-24559.NoIndex.PAW12603.s_1.bam" --ref "/rds/project/selma/reference/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna" --cnv --mod --sample_name "cnv_TB23_00177_T4" --out_dir "/rds/project/nanopore_pipelined/TB23_00177_t4" -profile singularity
Workflow Execution - CLI Execution Profile
singularity
What happened?
Process ended after a Run time of 00:01:25.
Error 255
Relevant log output
Application activity log entry
No response
Were you able to successfully run the latest version of the workflow with the demo data?
yes
Other demo data information
No response
The text was updated successfully, but these errors were encountered: