SeqMatcher — High-Performance DNA Primer Matching & Variant Counting

Overview

What is SeqMatcher?

A fast, reproducible, single-binary tool for the most common bottleneck in amplicon sequencing pipelines: assigning reads to primer pairs and quantifying variants.

Given a table of primer pairs and a table of library variants, SeqMatcher streams one or more sequence files and produces two count matrices per input: how many reads matched each primer, and how many times each variant appeared under each primer. It is implemented as a fully self-contained Rust binary with no runtime dependencies, built around Rayon data parallelism and an Aho-Corasick automaton for multi-pattern variant search.

At a glance

Language	Rust (stable, edition 2021)
Version	2.1.0
License	MIT
Author	Jiwen Zhao
Repository	github.com/CropCoder/SeqMatcher

Key dependencies

clap 4	CLI argument parsing
csv 1.3	CSV reading & writing
rayon 1.8	Data parallelism
aho-corasick 1.1	Multi-pattern matching
anyhow / tracing	Error handling & logging

Features

Built for large sequence files

Every design decision targets throughput, memory efficiency, and reproducibility on real sequencing-scale inputs.

Aho-Corasick variant search

All library variants (original and reverse complement) are compiled into a single automaton once at startup, reducing per-variant contains() scans to one linear pass.

Rayon parallel processing

Sequences are streamed in configurable chunks and processed with lock-free per-thread accumulation, merged at chunk boundaries.

Compile-time reverse complement

A 128-byte const lookup table maps every IUPAC nucleotide (including degenerate bases) in a single indexed load.

Streaming I/O

Reads are streamed line-by-line rather than loaded into memory, so file size does not limit the input scale.

Real-time progress

An in-place progress bar reports completion percentage, throughput, and ETA without adding I/O overhead.

Reproducible by design

Every run emits run_summary.json and run_summary.txt documenting inputs, parameters, system info, and a reproduction command.

Installation

Install from source

SeqMatcher is a single static binary with no runtime dependencies. Building from source requires a Rust toolchain.

Clone the repository

Get the source from GitHub.

terminal

$ git clone https://github.com/CropCoder/SeqMatcher.git
$ cd SeqMatcher

Build in release mode

The release profile enables fat LTO, single codegen unit, and stripping for maximum performance.

terminal

$ cargo build --release
# Binary: target/release/seq_matcher

(Optional) Cross-compile for Linux

Produce a statically-linked ELF binary from macOS using the bundled helper script.

terminal

# Prerequisites: brew install musl-cross && rustup target add x86_64-unknown-linux-musl
$ ./build-linux.sh
# Produces: target/x86_64-unknown-linux-musl/release/seq_matcher

Quick Start

Match in minutes

Point SeqMatcher at a primer table, a library table, and one or more sequence files using the LABEL:PATH format.

terminal

$ ./target/release/seq_matcher \
    --primer-csv primers.csv \
    --library-csv library.csv \
    --seq a_11:data/11_seq.txt \
    --output-dir output

Example run

stdout

  Loading primers from: primers_list_all.csv
  Loaded 32 primers
  Loading library from: 80_full_library_2_12.csv
  Loaded 8192 library variants
  Total: ~1500000 (est.) | 2.7 MB | chunk: 100000 | primers: 32 | variants: 8192 | AC patterns: 16384
  [████████████████████░░░░░░░░░░░░░░░░]  55.0%  825000/1500000  45230 seq/s  ETA: 15s
  Complete: 1500000 sequences in 33.2s, 45181 seq/s
  Written: output/a_11_seq_matched_primers_count.csv, output/a_11_seq_matched_library_variant_count.csv
  All done.

Command-Line Reference

Every option explained

Both --primer-csv and --library-csv are required; all other options have sensible defaults.

usage

Usage: seq_matcher [OPTIONS] --primer-csv <PRIMER_CSV> --library-csv <LIBRARY_CSV>

Option	Description	Default
`-p, --primer-csv`	Primer CSV file path (columns: id, forward_seq, reverse_seq)	required
`-l, --library-csv`	Library variant CSV file path	required
`--library-seq-col`	Column name in library CSV containing the variant sequence	`single_degenerate_library_expanded_reference`
`-s, --seq`	Sequence files in `LABEL:PATH` format; repeatable for batch processing	—
`-o, --output-dir`	Output directory for result CSVs	`output`
`-c, --chunk-size`	Number of sequences per parallel processing chunk	`100000`
`-t, --threads`	Number of worker threads	all cores
`--dry-run`	Validate inputs and exit without processing	off
`-q, --quiet`	Suppress non-error output	off
`-v, --verbose`	Enable verbose debug output	off
`--timestamp-output`	Append a timestamp to output filenames	off
`-h, --help`	Print help information	—
`-V, --version`	Print version information	—

Input & Output

File formats

All tabular I/O uses CSV. Original columns are preserved, and matching counts are appended as new columns.

Primer CSV

Any number of columns. The first three are treated as primer ID, forward sequence, and reverse sequence; all remaining columns are passed through unchanged.

Library CSV

Contains a variant sequence column whose name is set via --library-seq-col. Empty sequences become "always-matched" variants for Python compatibility.

Sequence files

Plain text with one DNA sequence per line. Sequences are normalized to uppercase automatically; no preprocessing required.

Primer CSV example

primers.csv

primer_id,forward_seq,reverse_seq,extra_columns...
P001,ATCGGTACC,GCTATAGCA,(preserved)
P002,TGCACTGAC,CGTACGATG,(preserved)

Library CSV example

library.csv

variant_id,single_degenerate_library_expanded_reference,extra_columns...
V001,ATCGNNNTCGA,(preserved)

Output files

File	Contents
{LABEL}_seq_matched_primers_count.csv	Original primer table plus a `count_{LABEL}` column (sequences matched per primer).
{LABEL}_seq_matched_library_variant_count.csv	Original library table plus one column per primer (`{primer_id}_{LABEL}`) with per-variant counts.
run_summary.json	Machine-readable run report: inputs, parameters, system info.
run_summary.txt	Human-readable run report including a reproduction command.

How It Works

The matching pipeline

Eight steps take raw sequence files to count matrices, optimized for throughput at every stage.

Load and pre-compute

Primer and library tables are loaded; reverse complements are pre-computed for every sequence using a compile-time 128-byte lookup table.

Build the automaton

An Aho-Corasick automaton encodes all library variants (original plus reverse complement) as multi-pattern states. It is built once and shared via Arc across threads.

Estimate line count

Total lines are estimated from file size by sampling the first lines, avoiding a full pre-scan of the input.

Match primers

Each sequence is tested against primers with first-match-wins semantics using anchored prefix/suffix matching (PCR amplicon model).

Scan for variants

Matched sequences undergo a single Aho-Corasick scan to detect all variant hits (deduplicated), replacing per-variant contains() calls.

Accumulate lock-free

Each thread accumulates counts independently; results are merged only at chunk boundaries.

Report progress

A real-time progress bar shows completion percentage, throughput, and ETA.

Write outputs

Count matrices are written to CSV with buffered I/O, alongside reproducible run summaries.

Performance optimizations

Optimization	Implementation
Reverse complement	Compile-time `const` 128-byte LUT, single-cycle indexed mapping
Variant matching	Aho-Corasick multi-pattern search, O(L+M) instead of O(V·L)
Parallel processing	Rayon work-stealing, chunk-level parallelism
Cross-thread sharing	`Arc` zero-copy sharing of primers, library, and automaton
Output I/O	`BufWriter` buffered writes
Line counting	File-size-based estimation avoids full pre-scan I/O
Variant count output	Pre-built column vectors for O(1) lookups over O(P·V) HashMap access
Progress feedback	In-place progress bar, no extra I/O overhead

Best Practices

Recommendations

Practical guidance for getting correct, reproducible results on sequencing-scale data.

›

Validate before processing

Run with --dry-run first to confirm primers, library variants, and sequence files are all loaded and consistent before committing to a long run.

›

Keep primer IDs unique and non-empty

Duplicate or empty primer IDs are rejected at validation time. Each ID becomes an output column name, so keep them short and filesystem-safe.

›

Use meaningful sequence labels

The LABEL in LABEL:PATH drives output filenames and count column headers, so use descriptive labels (for example sample_a_rep1).

›

Tune chunk size for very large files

Larger chunks reduce merge synchronization at the cost of a little peak memory. The default 100000 is a good balance for most inputs.

›

Mind the empty variants

Empty library sequences are always counted as a match (Python-compatible behavior). Verify your library has no unintended empty rows if this surprises you.

›

Treat progress as an estimate

Total lines are extrapolated from file size, so the progress bar and ETA are estimates until the final chunk completes.

›

Pin down reproducibility

Use --timestamp-output and keep the generated run_summary.txt alongside your results; it records the exact command and environment used.

›

Scale threads to your workload

SeqMatcher defaults to all cores. For I/O-bound inputs on shared machines, limit concurrency with --threads.

Note

Primer matching is anchored: a read must start with one primer and end with the reverse complement of its partner. Internal or partial binding is intentionally not counted.

Citation

Cite SeqMatcher

If you use SeqMatcher in your research, please cite it. A CITATION.cff file is included for GitHub and Zenodo integration.

@software{seq_matcher,
  author = {Zhao, Jiwen},
  title = {SeqMatcher: High-performance DNA sequence primer matching and variant counting},
  year = {2026},
  version = {2.0.0},
  url = {https://github.com/CropCoder/SeqMatcher}
}