Hadoop InputFormat for FASTA files?

Question

[alex.rubinsteyn]

I'm interested in analyzing large FASTA files (like the human genome and proteome) in parallel using Spark or pydoop. Is there a library which implements FASTA parsing as a Hadoop InputFormat?

[Pierre Lindenbaum]

"Hadoop FASTA reader" at gist.github.com/jflatow/45551 ?

[alex.rubinsteyn]

This looks like it works well for a FASTA file with many small records (since it seeks locally on each worker). However, for a FASTA file with large contigs (like the genome) this wouldn't perform very well.