Tool:open-cravat: variant annotation tool

Question

[slcrick]

open-cravat is an open-source platform for rapidly developing, using, and disseminating variant annotation tools.

It can handle unlimited number of variants in VCF format input files as well as its own input format and produce tab-separated text output files and excel spreadsheets. It is command-line-based as well as GUI-based. Its web visualization tool gives interactive filtering and sorting and tables and "widgets" which are graphic components to show annotation results (network graphs and gene diagrams with your variants on, for example). It is written in Python and installed with a pip install and has its own module manager.

It is modular and open-source, and it is easy to write input and output format converters and custom report generators as well as custom annotators and gene mappers. You write only minimum parts specific to your module since heavy lifting is done by open-cravat. You can publish your converters, report generators, annotation modules, gene mappers, and etc with one command to cravat store for other people to install and use also with one command or one click. There are 44 annotation modules as of today and more are being added.

Its GUI components are being rapidly polished but its command-line interface is quite stable. Let me know if you have any question or suggestion about open-cravat.

[avelarbio46]

Hello slcrick!

I would like to know if it is possible to extract variants from selected variant annotators with custom information, for example, frequency and gene name. I know that I can download manually all databases, but it is simpler if I can use your tool because it is up-to-date and I don't want to download two times the same database (such as 1000 genomes)

[slcrick]

Yes, you can install OpenCRAVAT and install the annotation modules of your interest, and go to the /data folder of the modules and examine the databases manually with sqlite3. If you need more help, let me know.

[g.papp-co]

I would like to know which database OpenCRAVAT uses for transcript IDs when it maps the variant's coordinate into transcript coordinates by default. I mean for example knownGene(USCS) or Ensembl maybe? I would like to join the results with other bed files with scores but i need the correct transcript IDs. Thank You!

[Oak Bioinformatics]

OpenCRAVAT uses Gencode. Currently, Gencode V33 is being used.

[Zahra]

Dear Slcrick

I want to extract the databases of some modules and as you mentioned before, I go to the /data folder of the module and find it, but here is a question, how can I understand if this data is for the hg38 or hg19? (I need the hg19 data).

Thanks for any help

[Ryangguk Kim]

Hi, all genomic mapping data in OpenCRAVAT is for hg38. OpenCRAVAT converts hg19 input to hg38 using liftOver and calls annotator modules with the hg38 coordinates.

If you need hg19 data, what comes to my mind is applying liftOver to the data's positions.

[Liu]

Hi,

The [e-mail] is no longer reachable. Could you provide the email address to your support team? Also the import_module.py download link is no longer valid here https://open-cravat.readthedocs.io/en/latest/Cloud.html#microsoft-azure, I'm following the guide to setup OpenCRAVAT on Azure, could you also update that? Thanks!

[Ryangguk Kim]

Somehow cravat.us domain is not working at this moment. I can answer at [e-mail]. The Azure guide has been updated with a working link. Let me know if you still encounter problems.

[allison_cox]

Hello, I am trying to use open-cravat to add annotations to vcf files. I have set up a venv, activated, installed open-cravat, installed the base module, and installed a few databases. When I try to annotate a vcf, it's telling me that the sqlite3 database is closed. See history with traceback below. I have used fuser to see if the installed databases are in use, but there are no operations currently using the annotation databases. Thank you in advance for your help. Allison

(env) Allisons-MacBook-Pro:virtualenv yam$ oc run vcfs/HD829ITD_r220726_p221004.vcf -l hg19 -t text Input file(s): /Users/yam/Desktop/Software/virtualenv/vcfs/HD829ITD_r220726_p221004.vcf Genome assembly: hg19 Running converter... Converter (converter) finished in 1.749s Running gene mapper... finished in 3.428s Running annotators... annotator(s) finished in 2.077s Running aggregator... Variants finished in 0.063s Genes finished in 0.028s Samples finished in 0.028s Tags finished in 0.045s Indexing variant base__chrom finished in 0.002s variant base__coding finished in 0.002s variant base__so finished in 0.005s Running postaggregators... Tag Sampler (tagsampler) Traceback (most recent call last): File "/Users/yam/Desktop/Software/virtualenv/env/lib/python3.11/site-packages/cravat/cravat_class.py", line 592, in main self.run_postaggregators() File "/Users/yam/Desktop/Software/virtualenv/env/lib/python3.11/site-packages/cravat/cravat_class.py", line 1557, in run_postaggregators post_agg.run() File "/Users/yam/Desktop/Software/virtualenv/env/lib/python3.11/site-packages/cravat/base_postaggregator.py", line 163, in run self.base_cleanup() File "/Users/yam/Desktop/Software/virtualenv/env/lib/python3.11/site-packages/cravat/base_postaggregator.py", line 292, in base_cleanup self.cleanup() File "/Users/yam/Desktop/Software/virtualenv/env/lib/python3.11/site-packages/cravat/modules/postaggregators/tagsampler/tagsampler.py", line 29, in cleanup self.cursor.execute('pragma journal_mode=delete;') sqlite3.OperationalError: database is locked

Finished with an exception. Runtime: 8.039s Check /Users/yam/Desktop/Software/virtualenv/vcfs/HD829ITD_r220726_p221004.vcf.log

[Ryangguk Kim]

Hi, I am the original poster. I was the lead architect of open-cravat, but I am currently developing OakVar, a fork of OpenCRAVAT at https://github.com/rkimoakbioinformatics/oakvar after leaving the team. The issue may have been fixed at OakVar. If you are interested, please check it out. It is backward-compatible to open-cravat. If you still have the issue with OakVar, please let me know and I'll investigate.

Answer 1

[slcrick]

open-cravat 1.4.5 has been released. This version has the following updates (since 1.3.2):

1.4.5

July 16, 2019

• Fix for bug preventing submission of multiple input files.

1.4.4

June 17, 2019

Installers

• The Mac installer is now a signed package installer.

cravat core

• In Windows, Mac OS, and Linux, different default folders for modules, jobs, and configuration files are used to better suit their native folder architecture.
• Python requirement has been increased to Python 3.6 or higher.
• Log file has non-redundant exception messages for better readability.
• Record of annotation modules are correctly kept with multiple cravat runs on the same input.
• For a job with multiple input files, output file names start with the first input file name plus _and_x_files, where x is the number of input files minus 1.

cravat GUI

• Job list is automatically updated when there are running jobs.
• Running jobs can be cancelled from the GUI.
• Warns if a job has more input lines than specified in Settings.
• GUI remembers genome assembly selection.
• Variout user interface improvements
• Fixed various bugs.

Web Store

• Module installation can be cancelled from the GUI.
• Shows module group, a collection of the varieties of a module.
• Warns with total size of installation for collective installation or update.
• Module tile shows their module types if they are not annotation modules.

Result Viewer

• hg19 and tagsampler results show next to Variant Annotation columns.
• Less-informative widgets are hidden by default.

Modules

• Protein Diagram widget on gene tab shows all variants for a gene on a table, whose rows when hovered will highlight corresponding variants on the protein diagram.
• 1000 Genomes module group and CHASMplus module group have been published.
• Comma-delimited format reporter and tab-delimited format reporter have been published.

1.4.3

April 30, 2019

GUI

• Improved the launch speed.
• Detects the absence of the server and prevents further operation.
• Multiple open-cravat GUI browser tabs work properly with install/update.

Web Store

• Prevents installation/update of modules if free disk space is not enough.

1.4.2

April 19, 2019

Installers

• Releasing Windows and Mac installers. No more pip installation is needed if these installers are used.

cravat core

• cravat can process multiple vcf-format input files at once.
• cravat better handles status and error logging. *Annotation modules run in multiple execution of cravat with the same input are accummulatively logged in the job status file for the input.

cravat GUI

• open-cravat command-line terminal can be launched from the GUI.

Web Store

• Improved launch speed.
• Fixed alphabetical name sorting of modules.

Result Viewer

• Simplified and improved the default settings of the basic load-filter.
• Added context menu to the column groups for each module on the result tables.
• Improved the layout and user experience of the table columns and widgets.
• Improved launching speed.
• hg19 coordinates, samples, and tags appear right next to Variant Annotation columns.
• Summary widgets without any result will hide themselves.
• Fixed wrong drag-and-drop of column headers

Modules

• Improved the speed of REVEL module.
• Smarter detection of vcf input format (vcf-converter)

1.4.1

March 21, 2019

Result Viewer

• Revamped the Summary tab.
• Fixed the load filter for the cases where samples were searched with "not".
• Added context-menu to the result table columns.
• Result table cell value area can be expanded to display large text strings.
• Columns of the result table can no longer be nested in another column group.
• Added new help dialog functionality for each widget (e.g. IGV widget).
• Fixed issues with browser zoom.
• Bug fixes

CRAVAT web GUI

• Added functionality to navigate between different module dialogs in store using arrow keys.
• Decluttered the interface of the web store by completely hiding base components.
• Added separate input examples for hg18, hg19, and hg38.
• Module detail panel in web store describes the output columns of the shown module.
• Updated look and feel of the web submit and the web store.
• Bug fixes

cravat

• Dependencies (between modules and between open-cravat and modules) are resolved before module installation and update.
• Decluttering of the log file for better readability.
• Each run creates an .err file which explicitly reports each variant with an error.
• Added the capability of cravat-admin to privately publish modules.

Others

• Added web links to the ID columns of Clinvar, COSMIC, dbSNP, UniProt, and denovo-db.
• Improved the default column size and widget layout for several modules.

Answer 2

[slcrick]

open-cravat 1.3.2 has been released. This version has the following updates:

Updates on Interactive Result Viewer (cravat-view):

• Data loading indicator
• Local filter select box
• Show-all and hide-all buttons for widgets
• Load filter of cravat-view now has two modes, simple and advanced, and it shows only the filter operators relevant to the type of the filtered data.
• Widgets can be hidden by default and its show/hide status can be saved and loaded.

Updates on cravat web GUI (wcravat):

• Store now has a Front Page with Most Downloaded and Newest modules.
• Store shows annotator modules' source data version for provenance.
• Store can sort modules by their name, size, and date of post.

Updates on cravat:

• cravat detects input files’ encoding and reads them correctly and writes output files always in UTF-8.

Others:

• Module updates have been concurrently released.
• VCF format converter now supports VarScan VCF format.
• Fixed various bugs.

Answer 3

[slcrick]

OpenCRAVAT 1.5.1 has been released, which has the following updates.

cravat core

• cravat-admin info shows data source version for each module version, if available, as well as the current version.
• cravat options changed: --startat for setting the starting stage, --endat for setting the ending stage, --repeat for setting the stage(s) to repeat, and --skip for setting the stage(s) to skip.
• cravat can receive a job configuration file which can direct any argument which can be given with command-line.
• Revampled how column definitions are handled internally.
• Gene level annotation is added to variant level annotation.
• Gene level aggregation is done dynamically with filters.
• cravat --version and cravat-admin --version show the version of open-cravat.
• cravat-admin ls shows module titles.
• cravat-admin info shows the explanation on module output columns as well as release note.

cravat GUI

• Revampled the interface.
• Annotation modules can be viewed and selected in groups and categories.
• Connection to the server will not be lost even if the GUI browser tab is left open for a long time.
• Fixed minor bugs.

Web Store

• Revampled the interface.
• Module detail panel shows required modules.
• Fixed minor bugs.

Result Viewer

• Revampled the interface.
• Added Sample Filter which can filter with inclusion and exclusion of samples.
• Added Gene Filter which can accept a list of HUGO symbols and filter the result with it.
• Added Smart Filter which can filter multiple columns with one selection.
• Added Query Builder with which complex custom filters can be built.
• Widget content can be exported to a png file.
• Widgets can hide themselves if there is no data for them.
• Module group names have tooltips which explains the modules.
• Added module group context menu.
• Module groups are alphabetically sorted.
• Table export button will export what is shown and with load and table filter information.
• A module can have a default set of columns to show, and the Result Viewer has small buttons in the header for each module for expanding, collaping, and bringing back to the default of the columns of the module.
• Fixed minor bugs.

Modules

• Added VCF format reporter which can preserve the annotations in the input VCF format file.