umi_gene_saturation error 137

[lizh0205]

Ask away!

Hi,

Thank you for providing this wonderful tool. I have run several nanopore sequencings with v2.0.3 wf-single-cell, and they all finished successfully. However, for one sample, I received the following error (log attached).

################
Sep-27 08:55:30.015 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=312; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Sep-27 08:55:30.025 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=pipeline:process_bams:umi_gene_saturation (1); work-dir=/home/bio/work/ec/8570ac4622af7c8ccf48df2e5b070e
error [nextflow.exception.ProcessFailedException]: Process pipeline:process_bams:umi_gene_saturation (1) terminated with an error exit status (137)
Sep-27 08:55:30.049 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'pipeline:process_bams:umi_gene_saturation (1)'
Caused by:
Process pipeline:process_bams:umi_gene_saturation (1) terminated with an error exit status (137)
Command executed:
export POLARS_MAX_THREADS=4
workflow-glue calc_saturation --output "saturation_curves.png" --read_tags read_tags.tsv
Command exit status:
137
Command output:
(empty)
#####################

I looked up the 137 error code and found it may be related to memory. I tried increasing the memory from "32G" to "64G" in "/home/bio/epi2melabs/workflows/epi2me-labs/wf-single-cell/subworkflows/process_bams.nf," but I still encountered this error. Please let me know if this is not the correct place to modify.

I run these jobs on a cluster, and I believe I have enough CPUs and memory for these jobs.
nextflow.log

Could you please give any suggestions about solving this issue?

Best regards,
Zhuo

[nrhorner]

Hi @lizh0205

Yes 137 is an out of memory error. Does this sample have more reads than the other samples?
Can you try running again using the -resume option to check whether it was an intermittent issue please?

Thanks,

Neil

[lizh0205]

Hi @lizh0205

Yes 137 is an out of memory error. Does this sample have more reads than the other samples? Can you try running again using the -resume option to check whether it was an intermittent issue please?

Thanks,

Neil

Dear Neil,

Thank you very much for your prompt reply.

Actually, I have successfully completed some jobs with more reads than this one. Additionally, I have tried to resume it using the -resume flag in Nextflow, but it stops at the same point each time.

[nrhorner]

Hi @lizh0205

Thanks for the update. I noticed you have --expected_cells set at 14,000. 10x currently recommends targeting a maximum of 10k cells with the Next GEM kits. On top of this, your sample may also contain more UMIs than normal, which could explain the out of memory error.

That being said, we should look at reducing memory usage of the process in question as 10x have increased the recommended number of target cells to 20k in their GEM-X chemistry.

Are you able to increase memory to this process further? You can do that like this:
create a file with the following contents called memory.config (for example)

process {
	 withName:umi_gene_saturation {
		memory = 128.GB
	 }
 }
Run you workflow with the following additions to the command. 

` -c memory.config -r resume`

Thanks,

Neil

[lizh0205]

Thank you for your suggestion, Neil.

I will give it a shot.

Best regards,
Zhuo

[nrhorner]

Hi @lizh0205 Did you get anywhere with this?