Question: Using tabix remotely against S3
0
gravatar for andrewl
21 months ago by
andrewl10
andrewl10 wrote:

Hi everyone,

I have tabix working well locally but want to store my genomes on an S3 bucket (index files will likely stay local). I am having trouble getting this to work as advertised.

wget https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

Resolving s3-eu-west-1.amazonaws.com... ...

Connecting to s3-eu-west-1.amazonaws.com|...|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 443976024 (423M) [application/x-gzip]

Saving to: 'WGC069899D-hg38-sorted.vcf.gz.1'

So no (permission) problems with accessing the file from this bucket.

But when I try tabix I get:

tabix https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000

[E::hts_open_format] fail to open file 'https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz'

Could not read https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

When I do the same tbix on the file locally, it works fine?

What am I missing?

Thanks.

tabix • 961 views
ADD COMMENTlink modified 21 months ago by Santosh Anand4.6k • written 21 months ago by andrewl10

stupid questions:

ADD REPLYlink written 21 months ago by Pierre Lindenbaum117k

What happens if you instead tabix s3://*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000, where * is the bucket on S3? At least for bam/cram files that sort of thing works.

ADD REPLYlink written 21 months ago by Devon Ryan88k
2
gravatar for Santosh Anand
21 months ago by
Santosh Anand4.6k
Santosh Anand4.6k wrote:

The problem might be the https: protocol. See this https://github.com/samtools/htslib/issues/139

TL; DR: Check that your htslib is current, and it has been installed with libcurl and libcrypto libraries. See requirements at https://github.com/samtools/htslib/blob/develop/INSTALL

HTSlib uses the following external libraries. Building requires both the library itself, and include files needed to compile code that uses functions from the library. Note that some Linux distributions put include files in a development ('-dev' or '-devel') package separate from the main library.

libz       (required)
libbz2     (required, unless configured with --disable-bz2)
liblzma    (required, unless configured with --disable-lzma)
libcurl    (optional, but strongly recommended)
libcrypto  (optional for Amazon S3 support; not needed on MacOS)
  
ADD COMMENTlink modified 5 months ago • written 21 months ago by Santosh Anand4.6k
1

^This. You need to build tabix with libcurl+openssl.

ADD REPLYlink written 21 months ago by lh331k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1677 users visited in the last hour