Question: Using tabix remotely against S3
0
gravatar for andrewl
2.3 years ago by
andrewl10
andrewl10 wrote:

Hi everyone,

I have tabix working well locally but want to store my genomes on an S3 bucket (index files will likely stay local). I am having trouble getting this to work as advertised.

wget https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

Resolving s3-eu-west-1.amazonaws.com... ...

Connecting to s3-eu-west-1.amazonaws.com|...|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 443976024 (423M) [application/x-gzip]

Saving to: 'WGC069899D-hg38-sorted.vcf.gz.1'

So no (permission) problems with accessing the file from this bucket.

But when I try tabix I get:

tabix https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000

[E::hts_open_format] fail to open file 'https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz'

Could not read https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

When I do the same tbix on the file locally, it works fine?

What am I missing?

Thanks.

tabix • 1.2k views
ADD COMMENTlink modified 2.3 years ago by Santosh Anand4.9k • written 2.3 years ago by andrewl10

stupid questions:

ADD REPLYlink written 2.3 years ago by Pierre Lindenbaum122k

What happens if you instead tabix s3://*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000, where * is the bucket on S3? At least for bam/cram files that sort of thing works.

ADD REPLYlink written 2.3 years ago by Devon Ryan91k
2
gravatar for Santosh Anand
2.3 years ago by
Santosh Anand4.9k
Santosh Anand4.9k wrote:

The problem might be the https: protocol. See this https://github.com/samtools/htslib/issues/139

TL; DR: Check that your htslib is current, and it has been installed with libcurl and libcrypto libraries. See requirements at https://github.com/samtools/htslib/blob/develop/INSTALL

HTSlib uses the following external libraries. Building requires both the library itself, and include files needed to compile code that uses functions from the library. Note that some Linux distributions put include files in a development ('-dev' or '-devel') package separate from the main library.

libz       (required)
libbz2     (required, unless configured with --disable-bz2)
liblzma    (required, unless configured with --disable-lzma)
libcurl    (optional, but strongly recommended)
libcrypto  (optional for Amazon S3 support; not needed on MacOS)
  
ADD COMMENTlink modified 11 months ago • written 2.3 years ago by Santosh Anand4.9k
1

^This. You need to build tabix with libcurl+openssl.

ADD REPLYlink written 2.3 years ago by lh331k

hello. Can you help me with the same issue. I am new to this environment of tabix and s3

ADD REPLYlink written 7 days ago by shubhamnath50

hello. Can you help me with the same issue. I am new to this environment of tabix and s3

ADD REPLYlink written 7 days ago by shubhamnath50
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1471 users visited in the last hour