cooler cload error on pairs file from ONT reads

[hackkr]

Hi, I'm trying to convert a pairs file generated from long-read sequencing data into a cooler matrix and running into a similar error reported in #251. I've tried doing the suggested csort + cload pairix workaround, but the pairix file generated is empty. I'm wondering if it could have to do with the fact that my reference genome has a small, single chromosome? The .pairs file is generated from processed bam files coercing paired-end organization on to single-end ONT reads, but the format looks OK.

I'd appreciate any troubleshooting suggestions!

pairs file:

## pairs format v1.0.0						
#sorted: chr1-chr2-pos1-pos2						
#shape: whole matrix						
#chromsize: NC_017276.1 2522992						
#columns: readID chr1 pos1 chr2 pos2 strand1 strand2						
4b3af7b2-b367-4738-a93b-e46a6da4023e:109912	NC_017276.1	1	NC_017276.1	819	+	+
335d51dd-5978-4a06-8100-8366e4f70eff:865668	NC_017276.1	1	NC_017276.1	823	+	+
4b3af7b2-b367-4738-a93b-e46a6da4023e:109913	NC_017276.1	1	NC_017276.1	1034	+	+
591ecc9f-f3fd-4520-ac26-20f7364635ba:842730	NC_017276.1	1	NC_017276.1	128368	+	+
bf0f7128-51a4-4885-bb7c-6fea56a0f5df:156090	NC_017276.1	1	NC_017276.1	213988	+	-

cooler cload pairs returns:

INFO:cooler.create:Creating cooler at "/N/slate/rhackley/01_poreC_test/sup-output/paired_end/tmphd1l4ufk.multi.cool::/0"
INFO:cooler.create:Writing chroms
INFO:cooler.create:Writing bins
INFO:cooler.create:Writing pixels
INFO:cooler.create:Writing indexes
INFO:cooler.create:Writing info
INFO:cooler.create:Merging into test1_UU_1000.cool
INFO:cooler.create:Creating cooler at "/N/slate/rhackley/01_poreC_test/sup-output/paired_end/test1_UU_1000.cool::/"
INFO:cooler.create:Writing chroms
INFO:cooler.create:Writing bins
INFO:cooler.create:Writing pixels
INFO:cooler.reduce:n_merge_epochs: 1
INFO:cooler.reduce:nnzs: [0]
Traceback (most recent call last):
  File "/N/slate/rhackley/porec/bin/cooler", line 10, in <module>
    sys.exit(cli())
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/cli/cload.py", line 646, in pairs
    create_cooler(
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/create/_create.py", line 1091, in create_cooler
    create_from_unordered(
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/create/_create.py", line 808, in create_from_unordered
    create(cool_uri, bins, chunks, columns=columns, dtypes=dtypes, mode=mode, **kwargs)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/create/_create.py", line 681, in create
    nnz, ncontacts = write_pixels(
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/create/_create.py", line 235, in write_pixels
    for i, chunk in enumerate(iterable):
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/reduce.py", line 190, in __iter__
    combined = pd.concat(
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 382, in concat
    op = _Concatenator(
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 445, in __init__
    objs, keys = self._clean_keys_and_objs(objs, keys)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 507, in _clean_keys_and_objs
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate```

[nvictus]

Is it possible your chrom.sizes file is not matching the contig name in your pairs file? Also, make sure it is tab-separated (NC_017276.1\t2522992).

[hackkr]

Yes, there was an error in my sizes file. thanks!

I'm running into a subsequent error running cooler zoomify on the output matrix. Both balance and coarsen run without error on the same file.

[rhackley]$ cooler zoomify -r 4DN --balance ``--balance-args '--cis-only'`` test-simplify-1000.mcool 
INFO:cooler.cli.zoomify:Recursively aggregating "test-simplify-1000.mcool"
INFO:cooler.cli.zoomify:Writing to "test-simplify-1000.mcool"
INFO:cooler.reduce:Copying base matrices and producing 3 new zoom levels.
INFO:cooler.reduce:Bin size: 1000
Traceback (most recent call last):
  File "/N/slate/rhackley/porec/bin/cooler", line 10, in <module>
    sys.exit(cli())
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/cli/zoomify.py", line 233, in zoomify
    zoomify_cooler(
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/cooler/reduce.py", line 841, in zoomify_cooler
    h5py.File(outfile, "w") as dest:  # fmt: skip
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/h5py/_hl/files.py", line 562, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/N/slate/rhackley/porec/lib/python3.10/site-packages/h5py/_hl/files.py", line 241, in make_fid
    fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 122, in h5py.h5f.create
OSError: Unable to synchronously create file (unable to truncate a file which is already open)

[nvictus]

It looks like you named your initial file test-simplify-1000.mcool, when it should have been test-simplify-1000.cool. The command generates the output path by replacing the extension with .mcool by default, so in this case it ends up trying to write the output to the input file.

These are good edge cases to catch. What ended up being the issue with your chromsizes file?

[hackkr]

Ah, yes, that resolved the error. Thanks for your help answering beginner questions! The issue with my chromsize was the chromosome names didn't match: I was using RefSeq chr name, and my sizes file had Genbank.