MergeSamFiles: Alignments added out of order in SAMFileWriterImpl.addAlignment for null

Question

[Ric]

Hi, I failed to merge sam files with Picard-tools:

**********    MergeSamFiles -USE_THREADING true -ASSUME_SORTED false -COMMENT true -VALIDATION_STRINGENCY SILENT -I sams/SRR1555764.fastp.sam -I sams/SRR1555765.fastp.sam -I sams/SRR6074038.fastp.sam -I sams/SRR6074039.fastp.sam -I sams/SRR935013.fastp.sam -I sams/SRR947064.fastp.sam -O sams/merged.sam
**********


23:40:39.731 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/lustre/work-lustre/waterhouse_team/miniconda2/envs/picard/share/picard-2.19.0-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Sat Aug 03 23:40:39 AEST 2019] MergeSamFiles INPUT=[sams/SRR1555764.fastp.sam, sams/SRR1555765.fastp.sam, sams/SRR6074038.fastp.sam, sams/SRR6074039.fastp.sam, sams/SRR935013.fastp.sam, sams/SRR947064.fastp.sam] OUTPUT=sams/merged.sam ASSUME_SORTED=false USE_THREADING=true COMMENT=[true] VALIDATION_STRINGENCY=SILENT    SORT_ORDER=coordinate MERGE_SEQUENCE_DICTIONARIES=false VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Sat Aug 03 23:40:39 AEST 2019] Executing as lorencm@cl4n008 on Linux 4.4.120-92.70-default amd64; OpenJDK 64-Bit Server VM 1.8.0_121-b15; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.19.0-SNAPSHOT
INFO    2019-08-03 23:40:39     MergeSamFiles   Input files are in same order as output so sorting to temp directory is not needed.
[Sat Aug 03 23:40:42 AEST 2019] picard.sam.MergeSamFiles done. Elapsed time: 0.05 minutes.
Runtime.totalMemory()=2595749888
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.IllegalArgumentException: Alignments added out of order in SAMFileWriterImpl.addAlignment for null. Sort order is coordinate. Offending records are at [NbV1Ch01:213308] and [NbV1Ch01:213305]
        at htsjdk.samtools.SAMFileWriterImpl.assertPresorted(SAMFileWriterImpl.java:197)
        at htsjdk.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:184)
        at htsjdk.samtools.AsyncSAMFileWriter.synchronouslyWrite(AsyncSAMFileWriter.java:36)
        at htsjdk.samtools.AsyncSAMFileWriter.synchronouslyWrite(AsyncSAMFileWriter.java:16)
        at htsjdk.samtools.util.AbstractAsyncWriter$WriterRunnable.run(AbstractAsyncWriter.java:123)
        at java.lang.Thread.run(Thread.java:745)

All input SAM files contain this header:

> samtools  view -H  *.sam
@HD VN:1.5  SO:coordinate
@SQ SN:NbV1Ch01 LN:187320599
@SQ SN:NbV1Ch02 LN:137376221
@SQ SN:NbV1Ch03 LN:137133353
@SQ SN:NbV1Ch04 LN:135864837
@SQ SN:NbV1Ch05 LN:131834845
@SQ SN:NbV1Ch06 LN:130086652
@SQ SN:NbV1Ch07 LN:129781819
@SQ SN:NbV1Ch08 LN:129244199
@SQ SN:NbV1Ch09 LN:126922077
@SQ SN:NbV1Ch10 LN:126859650
@SQ SN:NbV1Ch11 LN:110549029
@SQ SN:NbV1Ch12 LN:179405951
@SQ SN:NbV1Ch13 LN:177101848
@SQ SN:NbV1Ch14 LN:172641549
@SQ SN:NbV1Ch15 LN:154107235
@SQ SN:NbV1Ch16 LN:148804966
@SQ SN:NbV1Ch17 LN:145611047
@SQ SN:NbV1Ch18 LN:144482707
@SQ SN:NbV1Ch19 LN:137831541

However, the merged.sam contain additionally @CO true

> samtools  view -H  merged.sam 
@HD VN:1.6  GO:none SO:coordinate
@SQ SN:NbV1Ch01 LN:187320599
@SQ SN:NbV1Ch02 LN:137376221
@SQ SN:NbV1Ch03 LN:137133353
@SQ SN:NbV1Ch04 LN:135864837
@SQ SN:NbV1Ch05 LN:131834845
@SQ SN:NbV1Ch06 LN:130086652
@SQ SN:NbV1Ch07 LN:129781819
@SQ SN:NbV1Ch08 LN:129244199
@SQ SN:NbV1Ch09 LN:126922077
@SQ SN:NbV1Ch10 LN:126859650
@SQ SN:NbV1Ch11 LN:110549029
@SQ SN:NbV1Ch12 LN:179405951
@SQ SN:NbV1Ch13 LN:177101848
@SQ SN:NbV1Ch14 LN:172641549
@SQ SN:NbV1Ch15 LN:154107235
@SQ SN:NbV1Ch16 LN:148804966
@SQ SN:NbV1Ch17 LN:145611047
@SQ SN:NbV1Ch18 LN:144482707
@SQ SN:NbV1Ch19 LN:137831541
@CO true

What did I miss?

Thank you in advance,

[Pierre Lindenbaum]

and btw, you should be always working with BAM files: it's faster and it taks less space.

[Pierre Lindenbaum]

hum.. should work even if your sam are not sorted... what is the header of those sam files ?

[Ric]

I have just added the headers information into my question above.

[Pierre Lindenbaum]

yeah, your sam are sorted...

what would be the output of

grep -w -F NbV1Ch01 *.sam | grep -w -E '(213308|213305)'

[WouterDeCoster]

For what it's worth, you can also use samtools merge.