dedupe.sh from BBmap. Can be as simple as:
dedupe.sh in=<file or stdin> out=<file or stdout>
Description: Accepts one or more files containing sets of sequences (reads or scaffolds). Removes duplicate sequences, which may be specified to be exact matches, subsequences, or sequences within some percent identity. Can also find overlapping sequences and group them into clusters.