Need Help : JGI IMG Bulk Download

Question

[Ananaya]

I'm Struggling to automate bulk downloads of ~36k+ metagenome assemblies from JGI's IMG database. The web interface works but isn't feasible at scale.

What I've Tried:

1. jgi-query (Python)- https://lnkd.in/erXHH5SM
2. get_jgi_genomes (Perl)- https://lnkd.in/e_5keBQx
3. Custom scripts using JGI's XML API

Issues I'm hitting:

"Portal not found" errors for many IMG IDs 
Authentication cookies expiring mid-download 
Tape storage timeouts 
Variable portal naming (IMG_, Ga0, Gp0*)

I know others have faced this (Biostars thread), but haven't found a reliable solution for 2025.

Has anyone:

• Successfully automated bulk IMG downloads recently?
• Found updated tools that work with current JGI API?

Any guidance appreciated!

[GenoMax]

Please post direct links to software you mention. Not sure why you are using LinkedIn short URL's.

Historically downloads from JGI have been riddled with problems as noted in prior posts.

Looks like there is a new data portal now online at JGI : https://data.jgi.doe.gov/ Try your luck there.

Answer 1

[Kevin Blighe]

Hey Ananaya,

The JGI Data Portal (https://data.jgi.doe.gov/) unifies access across portals (including IMG) and has an API for bulk automation: https://files.jgi.doe.gov/apidoc/. It handles auth via session tokens (no expiring cookies) and lets you stage files from tape storage in advance.

Try this Python script for bulk metagenome pulls—it uses the API, splits huge requests (>10TB), and generates curl/Globus commands: https://github.com/WrightonLabCSU/JGI-API-Request. Input your IMG OIDs in a TSV, get a request ID, and download via Globus to dodge timeouts.

jgi-query (https://github.com/glarue/jgi-query) is still solid for CLI but regex portal names (e.g., IMG_|Ga0|Gp0*) and add retries (-n 5). Perl script looks unmaintained.

This worked for me on 30 000+ assemblies last month - please message if stuck.

Kind regards, Kevin