vgconstruct crashes

Question

[Uveyik]

I am running following command

 vg construct -C -r chr1.fasta -v  chr1.vcf.gz -t 1 -m 32 > human_v38.chr1.vg

with the following error

terminate called after throwing an instance of 'std::runtime_error'
  what():  io::MessageEmitter::write: message too large

Crash report for vg v1.62.0 Ranzano
Stack trace (most recent call last):
#18   Object /usr/local/bin/vg  at 0x62c1a4  in _start
#17   Object /usr/local/bin/vg  at 0x20fff36  in __libc_start_main
#16   Object /usr/local/bin/vg  at 0x20fe699  in __libc_start_call_main
#15   Object /usr/local/bin/vg  at 0xe22bab  in vg::subcommand::Subcommand::operator()(int  char**) const
#14   Object /usr/local/bin/vg  at 0xcc8aeb  in main_construct(int  char**)
#13   Object /usr/local/bin/vg  at 0xfb5f5e  in vg::Constructor::construct_graph(std::vector<std::__cxx11::basic_string<char  std::char_traits<char>  std::allocator<char> >  std::allocator<std::__cxx11::basic_string<char  std::char_traits<char>  std::allocator<char> > > > const&  std::vector<std::__cxx11::basic_string<char  std::char_traits<char>  std::allocator<char> >  std::allocator<std::__cxx11::basic_string<char  std::char_traits<char>  std::allocator<char> > > > const&  std::vector<std::__cxx11::basic_string<char  std::char_traits<char>  std::allocator<char> >  std::allocator<std::__cxx11::basic_string<char  std::char_traits<char>  std::allocator<char> > > > const&  std::function<void (vg::Graph&)> const&)
#12   Object /usr/local/bin/vg  at 0xfb5124  in vg::Constructor::construct_graph(std::vector<FastaReference*  std::allocator<FastaReference*> > const&  std::vector<vcflib::VariantCallFile*  std::allocator<vcflib::VariantCallFile*> > const&  std::vector<FastaReference*  std::allocator<FastaReference*> > const&  std::function<void (vg::Graph&)> const&)
#11   Object /usr/local/bin/vg  at 0xfb3cf3  in vg::Constructor::construct_graph(std::__cxx11::basic_string<char  std::char_traits<char>  std::allocator<char> >  FastaReference&  vg::VcfBuffer&  std::vector<FastaReference*  std::allocator<FastaReference*> > const&  std::function<void (vg::Graph&)> const&)
#10   Object /usr/local/bin/vg  at 0xcca190  in std::_Function_handler<void (vg::Graph&)  main_construct(int  char**)::{lambda(vg::Graph&)#1}>::_M_invoke(std::_Any_data const&  vg::Graph&)
#9    Object /usr/local/bin/vg  at 0xb545b0  in vg::io::ProtobufEmitter<vg::Graph>::write_copy(vg::Graph const&)
#8    Object /usr/local/bin/vg  at 0x20fbd9d  in _Unwind_Resume
#7    Object /usr/local/bin/vg  at 0x20fb21b  in _Unwind_RaiseException_Phase2
#6    Object /usr/local/bin/vg  at 0x2037a39  in gxx_personality_v0
#5    Object /usr/local/bin/vg  at 0x20d1ee8  in __cxa_call_terminate
#4    Object /usr/local/bin/vg  at 0x20382eb  in __cxxabiv1::__terminate(void (*)())
#3    Object /usr/local/bin/vg  at 0x5f2bcb  in __gnu_cxx::__verbose_terminate_handler() [clone .cold]
#2    Object /usr/local/bin/vg  at 0x5f5313  in abort
#1    Object /usr/local/bin/vg  at 0x2117115  in raise
#0    Object /usr/local/bin/vg  at 0x2143c6c  in __pthread_kill

What could be the reason and how to solve?

Answer 1

[anovak]

vg construct makes the graph in pieces called "chunks", which by default include about 1024 variants, and saves each chunk to a Protobuf message. Protobuf messages can only be so large, and when writing them vg tries to limit each message to 1 GB. If it would write one that's larger, it fails with this message.

Try reducing the number of variants per chunk by setting -z/--region-size to a value less than 1024 on the command line, and see if that helps.

It could also be that you have a large number of overlapping variants; each chunk needs to include a complete set of overlapped variants, so chunks will grow until no more variants overlap those already in the chunk. If you have so many overlapping variants that vg can't split them into manageable-sized chunks, vg might need to be modified to work on your data.