Add concat fastqs from SRA manifest

[lldelisle]

Here is a new workflow when the samples has been sequenced in different run.

Points to discuss:

• SRA list vs manifest etc... I need more columns than just SRA name
• The use of parallel download seems to fail, probably because there are too many levels of nested in lists.

[lldelisle]

Also I don't like this hard written column 6 buy I don't know how to do because it is a column parameter.

[wm75]

I've incorporated a couple of suggestions into https://usegalaxy.eu/u/wolfgang-maier/w/sralisttoconcatenatedfastqs-imported-from-url, specifically:

• perform splitting on the input dataset directly and make sure the header line gets propagated to each collection element
• the output at this stage will always be of tabular format ->change this to sra_manifest.tabular and feed it directly to the faster download tool (saves cut jobs on collection elements)
• I've also changed the tool for splitting to toolshed.g2.bx.psu.edu/repos/bgruening/split_file_to_collection/split_file_to_collection/0.5.0 because it's the more versatile tool and is also used in at least one other iwc workflow (covid-19 variation reporting) already, so we might want to promote use of it, but feel free to ignore me
• the Select last, Cut and Unique steps can all be replaced with a single Datamash step - just that this requires now hard-coding of the final sample name column instead of column 6, again because we cannot create a param of type data_column from a WF input parameter.

I can confirm the remaining problem besides the data_column issue: parallel faster download jobs don't seem to organize their outputs correctly, but produce empty nested list structures

[lldelisle]

Meanwhile, I was wondering if we should not start by using the 'cut' tool to keep only the first column with SRA and the column with IDs the user wants. Then we could set column 2. What do you think?
I will have a look at your workflow.

[lldelisle]

Datamash is indeed a good improvement.

[lldelisle]

@wm75 I have an issue with 'sra_manifest.tabular'. I do not manage to have this datatype out of 'split_file_to_collection':
In the workflow:

In the history:

And it is the same with the tool split_file_on_column...
I also tried to put a 'apply rule' step between the 'split_file_to_collection' and the 'fasterqdump' to change the datatype.
While in theory it worked:

It seems that something went wrong with the column name:

That's why I need to add a 'cut1' step after 'split_file_to_collection'.

[lldelisle]

The tests pass with the master branch of planemo, should we release planemo?

[lldelisle]

@PierreOsteil for your information

[wm75]

@wm75 I have an issue with 'sra_manifest.tabular'. I do not manage to have this datatype out of 'split_file_to_collection'

Yes, splitting to a collection will always produce tabular format. It's ok not to care about the input format until after that step, but then change the datatype of its output to sra_manifest.tabular.

[lldelisle]

@wm75 I have an issue with 'sra_manifest.tabular'. I do not manage to have this datatype out of 'split_file_to_collection'

Yes, splitting to a collection will always produce tabular format. It's ok not to care about the input format until after that step, but then change the datatype of its output to sra_manifest.tabular.

I don't understand. Do you mean this or something else:

[lldelisle]

The workflow is working. However, because one of the fasterq-dump output for an accession number or for a list of accession number is a list:paired, when we run this in parallel we get a list:list:paired and then we face galaxyproject/galaxy#16878.
A way to solve this would be to change the fasterq-dump wrapper to output a 'paired' when an accession number is given. @mvdbeek @wm75 what do you think?
Ideally, I would like to have this workflow in IWC for a training next Thursday...

[mvdbeek]

I'm pretty sure we can fix the Galaxy side before this ... and in either case I'm happy to merge the workflow now if you're happy with it.

[lldelisle]

We still need to solve an issue @wm75 identified which is to be sure to use identifier which passes the 'relabel' step.

[lldelisle]

I think I can include a fix into my awk step:
https://github.com/galaxyproject/galaxy/blob/d8d27e8e86ec56199bdfda00e81400c02a1776c2/lib/galaxy/tools/relabel_from_file.xml#L201C145-L201C145

[lldelisle]

@wm75, tell me if it is OK for you like this (keeping the awk step) or if you prefer that I change it.

[wm75]

Since https://github.com/galaxyproject/galaxy/pull/8757/files, the allowed chars for the Relabel tool also include ,. and space so no need to replace these with awk.

[wm75]

Let me check how complicated it would be to replace awk with Cut and Replace ...

[lldelisle]

Since https://github.com/galaxyproject/galaxy/pull/8757/files, the allowed chars for the Relabel tool also include ,. and space so no need to replace these with awk.

Good point. The documentation had not been updated. I've just wrote a PR to fix it.

Tell me if you manage with a cut replace.

[wm75]

I've adjusted the README a bit and I think we can handle hypothetical sample names with triple underscores in them correctly with just a minimal change to the APPLY_RULES regex (untested though).

[lldelisle]

Thanks. I updated the tests... If they pass, we are ready.

[lldelisle]

Youhou! May I click on merge?

[wm75]

Great work @lldelisle !

[lldelisle]

I would say. Great collaboration! Thanks @wm75

[wm75]

Hmm, the merge failed with a failing test now:

Output collection 'paired_output': failed to find identifier 'GSM461177-' in the tool generated elements []
Output collection 'single_output': failed to find identifier 'GSM461176.-' in the tool generated elements []

@mvdbeek any idea why the results would be different from the within-PR testing?

[lldelisle]

I've relaunched the CI and then I will check on eu if I can reproduce the error...

[lldelisle]

It seems fixed.

[mvdbeek]

Yes, I assume a temporary job error. We'll have to rework the planemo testing code a little so we always get a report ...