TrESFlow

TrESFlow is a Nextflow DSL2 pipeline for preprocessing TrES-seq RNA and DNA sequencing data, including barcode assignment, read splitting, alignment, duplicate handling, coverage tracks, and quality control.

Installation

Create and activate a conda/mamba/micromamba environment with the required tools:

micromamba create -n tres
micromamba activate tres

micromamba install \
  pandas polars ipython pysam pybedtools numpy matplotlib seaborn scipy \
  pyarrow upsetplot anndata scanpy matplotlib-venn leidenalg scikit-learn \
  snapatac2 screen samtools bwa-mem2 star fastqc multiqc trim-galore \
  deeptools parallel ucsc-bedGraphToBigWig nextflow git gatk4

Clone TrESFlow:

git clone git@github.com:CSOgroup/TrESFlow.git
cd TrESFlow

Install Codon in the same environment:

./scripts/install_codon_0.16.3.sh --prefix /path/to/env/prefix

Inputs

TrESFlow uses one hierarchical YAML samplesheet describing the runtime environment, references, biological groups, barcodes, and FASTQ inputs.

library_name: Isa

runtime:
  env_prefix: /path/to/env/prefix
  # tmpdir: /path/to/large/tmp

references:
  species: human
  root: /path/to/TrESFlow_References
  ligation_barcode_whitelist: /path/to/TrESFlow_References/ligation_barcode_whitelist.txt

  rna_ref_dir: /path/to/TrESFlow_References/rna/human/star

  dna_ref_dir: /path/to/TrESFlow_References/dna/human/bwa
  dna_blacklist_bed: /path/to/TrESFlow_References/dna/human/hg38-blacklist.v2.bed
  dna_chrom_sizes: /path/to/TrESFlow_References/dna/human/hg38.chrom.sizes
  dna_effective_genome_size: 2913022398

samples:
  sample_id:
    groups:
      group_a:
        sb_oligo_indices: ["01", "02"]
        mark_barcodes:
          H3K27me3: AGGCTATA
          H3K27ac: GCCTCTAT

      group_b:
        sb_oligo_indices: ["03", "04"]
        mark_barcodes:
          H3K27me3: AGGCTATA
          H3K9me3: GCCTCTAT

    rna:
      reads:
        i1: /path/to/sample_RNA_I1.fastq.gz
        r1: /path/to/sample_RNA_R1.fastq.gz
        r2: /path/to/sample_RNA_R2.fastq.gz

    dna:
      # Both `single` and `dual` tagmentation modes are supported.
      #
      # Single tagmentation uses 4 nt DNA sample barcodes.
      # Dual tagmentation uses 3 nt DNA sample barcodes.
      #
      # `reads.i2` is required for single tagmentation, not for dual.
      #
      # DNA ligation barcodes are read from i1 at positions:
      #   single: 15,53,91
      #   dual:   41,79,117
      tagmentation: dual
      reads:
        i1: /path/to/sample_DNA_I1.fastq.gz
        r1: /path/to/sample_DNA_R1.fastq.gz
        r2: /path/to/sample_DNA_R2.fastq.gz

Samplesheet notes

• A sample may contain RNA, DNA, or both.
• FASTQs are defined once at sample level. Groups define how reads are assigned downstream.
• Groups may independently participate in RNA and/or DNA.
• Use shared sb_oligo_indices for normal corresponding RNA/DNA selections. The physical lookup expands each selected chemistry automatically.
• Use rna_sb_oligo_indices and dna_sb_oligo_indices for independent selections or modality-specific groups. Each keeps its actual index in the cell identifier; differing selections warn and continue. List order never declares correspondence.
• Existing sb_barcodes, rna_sb_barcodes and dna_sb_barcodes sequence inputs retain their precedence and chemistry rules. Shared indices are exclusive with other selectors; modality indices conflict with sequence selectors that apply to that modality. Shared sb_barcodes applies to RNA and single DNA; dual DNA indices may coexist with that RNA input.
• DNA groups require mark_barcodes. Indices 13–16 exist for RNA/single DNA; their dual sequences are unavailable (-) and cannot be selected for dual DNA.
• The same DNA modality barcode may represent different marks in different groups.
• dna.reads.i2 is required for single tagmentation and optional for dual.
• FASTQ inputs may be a single path, a comma-separated list, or a YAML sequence. Multiple entries represent ordered technical FASTQ chunks from the same library, and corresponding read roles must contain the same number of entries.
• YAML sequences are recommended for multi-FASTQ inputs and are required when a path itself contains a comma. Relative paths resolve from the samplesheet directory.
• runtime.tmpdir is optional and defaults to --outdir.
• references.rna_ref_dir must point to a STAR index when RNA is present.
• DNA samples require a bwa-mem2 reference, blacklist, chromosome sizes, and effective genome size.

For an experimenter mismatch, exceptionally declare sb_oligo_pairings:

B:
  sb_oligo_pairings:
    - sb_index: "07"
      rna_oligo_index: "07"
      dna_oligo_index: "10"
  mark_barcodes:
    H3K27ac: GCCTCTAT

This explicitly declares RNA physical oligo 07 and DNA physical oligo 10 to represent one biological partition, labelled 07. Pairing changes identity labels while retaining actual chemistry-specific SB sequences. Each present modality requires exactly one physical index or singular sequence selector (rna_sb_barcode / dna_sb_barcode). Omitted modalities are permitted; at least one must be present. Pairing is exclusive with all other group selectors. Normal corresponding selections should use the normal interface above. Warnings appear before preflight on every launch, including resume, and are saved with the resolved mapping under pipeline_info. See the cell-identity contract, pairing example, and YAML samplesheet schema.

For multiple technical FASTQ chunks:

rna:
  reads:
    i1: [/run1/I1.fastq.gz, /run2/I1.fastq.gz]
    r1: [/run1/R1.fastq.gz, /run2/R1.fastq.gz]
    r2: [/run1/R2.fastq.gz, /run2/R2.fastq.gz]

An example samplesheet is available above and a template in: assets/samplesheet.template.yaml

Running TrESFlow

A typical run is:

NXF_OFFLINE=true nextflow run . \
  --samplesheet /path/to/samplesheet.yaml \
  --outdir /path/to/results \
  --max_cpus 80

Useful options include:

• --max_cpus: maximum CPU budget available to the pipeline.
• --cleanup_work false: disable early intermediate-FASTQ cleanup and retain successful Nextflow work directories when debugging or when -resume is important.
• --publish_split_fastqs true: also publish gzip-compressed per-group split FASTQs.
• --filter_dual_tag_artifacts false: disable the dual-tagmentation DNA artifact filter.

NXF_OFFLINE=true prevents loading optional remote nf-core configuration and is suitable for normal local TrESFlow runs.

Outputs

The main output structure is:

<outdir>/
├── rna_align/
│   ├── STARsolo outputs
│   ├── *.filtered_cells.bam
│   └── *.bw
│
├── dna_align/
│   ├── aligned BAMs
│   ├── *_MarkedDup.bam
│   ├── *_NoDup.bam
│   └── *_NoDup.bw
│
├── TrES_Stats/
│   ├── tres_report.html
│   ├── read_retention.tsv
│   ├── qc_metrics.tsv
│   ├── barcode_composition.tsv
│   ├── library_complexity.tsv
│   └── qc/
│       ├── fastqc/
│       ├── samtools/
│       └── multiqc/
│
└── pipeline_info/
    ├── execution_report.html
    ├── execution_timeline.html
    ├── execution_trace.tsv
    ├── flowchart.html
    └── derived_contract/

The main QC report is:

TrES_Stats/tres_report.html

It summarizes read retention, barcode composition, DNA duplicate/library-complexity metrics, and RNA sequencing metrics.

MultiQC is written to:

TrES_Stats/qc/multiqc/multiqc_report.html

When --publish_split_fastqs true is used, the pipeline additionally creates:

rna_split_fastqs/
dna_split_fastqs/

Important information

Work directory cleanup

--cleanup_work true is the default. Generated intermediate FASTQs are removed during the run as soon as every enabled consumer has succeeded. Samplesheet FASTQs and gzip-compressed split FASTQs being published are never early-cleanup targets. Nextflow also removes successful task work directories after the run finishes.

Use:

--cleanup_work false

when you need to preserve the work directory for debugging or reliable reuse with -resume. With cleanup enabled, a resumed run can recompute tasks whose cached FASTQ outputs were deliberately removed; cleanup tasks themselves are never restored from cache.

Temporary disk space

runtime.tmpdir defaults to --outdir. Large datasets can require substantial temporary disk space, so set runtime.tmpdir explicitly when a larger or faster filesystem is available.

Canonical chromosomes

TrESFlow derives canonical chromosomes directly from the supplied references. UCSC and Ensembl chromosome naming conventions are supported.

Final canonical RNA and DNA outputs exclude alternative loci, patches, random/unplaced contigs, decoys, and other noncanonical sequences.

DNA duplicates

DNA duplicate marking uses the corrected cell barcode and AVITI read coordinates. Read groups identify AVITI run, flowcell, and lane while retaining the same logical library across sequencing units. The default optical-duplicate distance is:

--aviti_optical_duplicate_distance 10

Final *_NoDup.bam files and DNA BigWig tracks exclude reads marked as duplicates.

Dual-tagmentation artifact filtering

Dual-tagmentation DNA libraries are filtered after trimming for known residual linker signatures. If either mate contains one of these signatures, the pair is discarded.

This filter:

• is enabled by default;
• applies only to dual DNA libraries;
• does not affect RNA or single DNA libraries.

Further documentation

More detailed documentation is available in:

• Usage documentation
• Pipeline architecture

Cell identity uses CB = XI = <sample>_<group>_<sb_index>_<L1L2L3>, where sb_index defaults to the physical oligo_index unless explicitly paired. Use group-level sb_oligo_indices for chemistry-independent sample partitions; legacy nucleotide sequence inputs remain supported. See the cell-identity contract and regeneration guidance.