Smoothxg error during poa alignment to subgraph

[brettChapman]

Hi Erik

I tried running smoothxg on my GFA files (edyeet params = p99/a99/s50k, p95/a95/s50k, p90/a90/s50k, p90/a90/s100k), from chromosome 2H alignments across 4 different genomes.

I found all but the p99/a99/s50k alignment failed.

Heres the p99/a99/s50k output:

[path sgd sort]: 86.67% progress: iteration: 26, eta: 1.46, delta_max: 173155956.54, number of updates: 26956363
[path sgd sort]: 90.00% progress: iteration: 27, eta: 0.42, delta_max: 59465276.20, number of updates: 26956248
[path sgd sort]: 93.33% progress: iteration: 28, eta: 0.12, delta_max: 20829314.71, number of updates: 26957441
[path sgd sort]: 96.67% progress: iteration: 29, eta: 0.03, delta_max: 4379940.05, number of updates: 26957158
[path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 1955241.69, number of updates: 26956240
topological sort 26819945 of 26819945 ~ 100.0000%
[smoothxg::smoothable_blocks] computing blocks
[smoothxg::smoothable_blocks] computing blocks 100.00%%
[smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000)
[smoothxg::break_blocks] cut 263466 blocks of which 2782 had repeats
[smoothxg::smooth_and_lace] applying abPOA (global alignment mode) to block 268333/268333 100.000%
[smoothxg::smooth_and_lace] sorting path_mappings
[smoothxg::smooth_and_lace] building final graph
[smoothxg::smooth_and_lace] adding graph 268333/268333 100.000%
[smoothxg::smooth_and_lace] embedding path fragment 290597/290597
[smoothxg::smooth_and_lace] verifying paths

The p95/a95/s50k output:

[path sgd sort]: 83.33% progress: iteration: 25, eta: 6.03, delta_max: 104674509.10, number of updates: 45525237
[path sgd sort]: 86.67% progress: iteration: 26, eta: 1.68, delta_max: 108659302.66, number of updates: 45523325
[path sgd sort]: 90.00% progress: iteration: 27, eta: 0.47, delta_max: 53257587.61, number of updates: 45529831
[path sgd sort]: 93.33% progress: iteration: 28, eta: 0.13, delta_max: 13299096.00, number of updates: 45531472
[path sgd sort]: 96.67% progress: iteration: 29, eta: 0.04, delta_max: 4763521.38, number of updates: 45531132
[path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 1273493.52, number of updates: 45524400
topological sort 27368108 of 27368108 ~ 100.0000%
[smoothxg::smoothable_blocks] computing blocks
[smoothxg::smoothable_blocks] computing blocks 100.00%%
[smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000)
[smoothxg::break_blocks] cut 142012 blocks of which 1789 had repeats
[simd_abpoa_align_sequence_to_subgraph1] Error in cg_backtrack. (4)to block 157649/409308 38.516%
[smoothxg::smooth_and_lace] applying srun: error: node-20: task 0: Exited with exit code 138.516%

The p90/a90/s50k output:

[path sgd sort]: 86.67% progress: iteration: 26, eta: 2.35, delta_max: 71882224.52, number of updates: 148980422
[path sgd sort]: 90.00% progress: iteration: 27, eta: 0.60, delta_max: 33787250.39, number of updates: 148982175
[path sgd sort]: 93.33% progress: iteration: 28, eta: 0.15, delta_max: 7108062.96, number of updates: 148978394
[path sgd sort]: 96.67% progress: iteration: 29, eta: 0.04, delta_max: 2098275.19, number of updates: 148980490
[path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 1218520.58, number of updates: 148979518
topological sort 41166023 of 41166023 ~ 100.0000%
[smoothxg::smoothable_blocks] computing blocks
[smoothxg::smoothable_blocks] computing blocks 100.00%%
[smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000)
[smoothxg::break_blocks] cut 57103 blocks of which 1000 had repeats
[simd_abpoa_align_sequence_to_subgraph1] Error in cg_backtrack. (4)to block 974/858858 0.113%
srun: error: node-2: task 0: Exited with exit code 1lignment mode) to block 1014/858858 0.118%

The p90/a90/s100k output:

[path sgd sort]: 86.67% progress: iteration: 26, eta: 2.34, delta_max: 45389367.41, number of updates: 145013869
[path sgd sort]: 90.00% progress: iteration: 27, eta: 0.60, delta_max: 17867948.24, number of updates: 145014736
[path sgd sort]: 93.33% progress: iteration: 28, eta: 0.15, delta_max: 8877562.74, number of updates: 145014706
[path sgd sort]: 96.67% progress: iteration: 29, eta: 0.04, delta_max: 2934047.45, number of updates: 145012933
[path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 405494.10, number of updates: 145014858
topological sort 40181737 of 40181737 ~ 100.0000%
[smoothxg::smoothable_blocks] computing blocks
[smoothxg::smoothable_blocks] computing blocks 100.00%%
[smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000)
[smoothxg::break_blocks] cut 73145 blocks of which 1151 had repeats
[simd_abpoa_align_sequence_to_subgraph1] Error in cg_backtrack. (4)to block 14442/806980 1.790%
srun: error: node-19: task 0: Exited with exit code 1ignment mode) to block 14470/806980 1.793%

I noticed last week you added a new parameter -l to specify the max-poa-length, so I updated to the latest version of Smoothxg on Friday (9th October).

Even though setting at a99/p99 didn't provide any problems for smoothxg, I think a90/p90 is where the alignments will work well. I compared gene level alignments using a99/p99 with a90/p90 and a95/p95 using edyeet and wfmash, and it looks like anything above 90% starts to filter too many sequences out, leaving a fragmented alignment across the gene region.

[ekg]

This is a known issue. yangao07/abPOA#9

…

On Mon, Oct 12, 2020, 07:01 Brett Chapman ***@***.***> wrote: Hi Erik I tried running smoothxg on my GFA files (edyeet params = p99/a99/s50k, p95/a95/s50k, p90/a90/s50k, p90/a90/s100k), from chromosome 2H alignments across 4 different genomes. I found all but the p99/a99/s50k alignment failed. Heres the p99/a99/s50k output: [path sgd sort]: 86.67% progress: iteration: 26, eta: 1.46, delta_max: 173155956.54, number of updates: 26956363 [path sgd sort]: 90.00% progress: iteration: 27, eta: 0.42, delta_max: 59465276.20, number of updates: 26956248 [path sgd sort]: 93.33% progress: iteration: 28, eta: 0.12, delta_max: 20829314.71, number of updates: 26957441 [path sgd sort]: 96.67% progress: iteration: 29, eta: 0.03, delta_max: 4379940.05, number of updates: 26957158 [path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 1955241.69, number of updates: 26956240 topological sort 26819945 of 26819945 ~ 100.0000% [smoothxg::smoothable_blocks] computing blocks [smoothxg::smoothable_blocks] computing blocks 100.00%% [smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000) [smoothxg::break_blocks] cut 263466 blocks of which 2782 had repeats [smoothxg::smooth_and_lace] applying abPOA (global alignment mode) to block 268333/268333 100.000% [smoothxg::smooth_and_lace] sorting path_mappings [smoothxg::smooth_and_lace] building final graph [smoothxg::smooth_and_lace] adding graph 268333/268333 100.000% [smoothxg::smooth_and_lace] embedding path fragment 290597/290597 [smoothxg::smooth_and_lace] verifying paths The p95/a95/s50k output: [path sgd sort]: 83.33% progress: iteration: 25, eta: 6.03, delta_max: 104674509.10, number of updates: 45525237 [path sgd sort]: 86.67% progress: iteration: 26, eta: 1.68, delta_max: 108659302.66, number of updates: 45523325 [path sgd sort]: 90.00% progress: iteration: 27, eta: 0.47, delta_max: 53257587.61, number of updates: 45529831 [path sgd sort]: 93.33% progress: iteration: 28, eta: 0.13, delta_max: 13299096.00, number of updates: 45531472 [path sgd sort]: 96.67% progress: iteration: 29, eta: 0.04, delta_max: 4763521.38, number of updates: 45531132 [path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 1273493.52, number of updates: 45524400 topological sort 27368108 of 27368108 ~ 100.0000% [smoothxg::smoothable_blocks] computing blocks [smoothxg::smoothable_blocks] computing blocks 100.00%% [smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000) [smoothxg::break_blocks] cut 142012 blocks of which 1789 had repeats [simd_abpoa_align_sequence_to_subgraph1] Error in cg_backtrack. (4)to block 157649/409308 38.516% [smoothxg::smooth_and_lace] applying srun: error: node-20: task 0: Exited with exit code 138.516% The p90/a90/s50k output: [path sgd sort]: 86.67% progress: iteration: 26, eta: 2.35, delta_max: 71882224.52, number of updates: 148980422 [path sgd sort]: 90.00% progress: iteration: 27, eta: 0.60, delta_max: 33787250.39, number of updates: 148982175 [path sgd sort]: 93.33% progress: iteration: 28, eta: 0.15, delta_max: 7108062.96, number of updates: 148978394 [path sgd sort]: 96.67% progress: iteration: 29, eta: 0.04, delta_max: 2098275.19, number of updates: 148980490 [path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 1218520.58, number of updates: 148979518 topological sort 41166023 of 41166023 ~ 100.0000% [smoothxg::smoothable_blocks] computing blocks [smoothxg::smoothable_blocks] computing blocks 100.00%% [smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000) [smoothxg::break_blocks] cut 57103 blocks of which 1000 had repeats [simd_abpoa_align_sequence_to_subgraph1] Error in cg_backtrack. (4)to block 974/858858 0.113% srun: error: node-2: task 0: Exited with exit code 1lignment mode) to block 1014/858858 0.118% The p90/a90/s100k output: [path sgd sort]: 86.67% progress: iteration: 26, eta: 2.34, delta_max: 45389367.41, number of updates: 145013869 [path sgd sort]: 90.00% progress: iteration: 27, eta: 0.60, delta_max: 17867948.24, number of updates: 145014736 [path sgd sort]: 93.33% progress: iteration: 28, eta: 0.15, delta_max: 8877562.74, number of updates: 145014706 [path sgd sort]: 96.67% progress: iteration: 29, eta: 0.04, delta_max: 2934047.45, number of updates: 145012933 [path sgd sort]: 100.00% progress: iteration: 30, eta: 0.01, delta_max: 405494.10, number of updates: 145014858 topological sort 40181737 of 40181737 ~ 100.0000% [smoothxg::smoothable_blocks] computing blocks [smoothxg::smoothable_blocks] computing blocks 100.00%% [smoothxg::break_blocks] cutting blocks that contain sequences longer than max-poa-length (10000) [smoothxg::break_blocks] cut 73145 blocks of which 1151 had repeats [simd_abpoa_align_sequence_to_subgraph1] Error in cg_backtrack. (4)to block 14442/806980 1.790% srun: error: node-19: task 0: Exited with exit code 1ignment mode) to block 14470/806980 1.793% I noticed last week you added a new parameter -l to specify the max-poa-length, so I updated to the latest version of Smoothxg on Friday (9th October). Even though setting at a99/p99 didn't provide any problems for smoothxg, I think a90/p90 is where the alignments will work well. I compared gene level alignments using a99/p99 with a90/p90 and a95/p95 using edyeet and wfmash, and it looks like anything above 90% starts to filter too many sequences out, leaving a fragmented alignment across the gene region. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDQEIGCMMA5PQCHM6K6ATSKKETJANCNFSM4SMJQZFQ> .

[brettChapman]

Thanks. Are the abPOA parameters for the b parameter embedded in the code or could I try smoothxg with the -Z parameter, which would use an different alignment mode?

[brettChapman]

Or use the -S parameter with smoothxg?

[ekg]

You can run with -S to use spoa. It will be around 10x slower, and use more memory.

…

On Mon, Oct 12, 2020, 07:41 Brett Chapman ***@***.***> wrote: Or use the -S parameter with smoothxg? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDQEKE35YW43LFE2LVYBLSKKJIPANCNFSM4SMJQZFQ> .