I had this problem before, this is a known bug in samtools > 1.0 ...see e.g. https://biostar.usegalaxy.org/p/7928/ and https://biostar.usegalaxy.org/p/9335/ ... What I did was to just install samtools < 1 and it was okay then.. maybe you can give that a try ?
I encountered the same error with samtools-1.2 but not with the current samtools-1.3. In 1.3 it works fine for me with:
- rmdup -s singleend_input.bam singleend_output.bam
- rmdup pairedend_input pairedend_output
Don't forget to sort your files prior to rmdup.