Using tabix remotely against S3
1
0
Entering edit mode
5.1 years ago
andrewl ▴ 10

Hi everyone,

I have tabix working well locally but want to store my genomes on an S3 bucket (index files will likely stay local). I am having trouble getting this to work as advertised.

wget https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

Resolving s3-eu-west-1.amazonaws.com... ...

Connecting to s3-eu-west-1.amazonaws.com|...|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 443976024 (423M) [application/x-gzip]

Saving to: 'WGC069899D-hg38-sorted.vcf.gz.1'

So no (permission) problems with accessing the file from this bucket.

But when I try tabix I get:

tabix https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000

[E::hts_open_format] fail to open file 'https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz'

Could not read https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

When I do the same tbix on the file locally, it works fine?

What am I missing?

Thanks.

tabix • 2.7k views
ADD COMMENT
0
Entering edit mode

stupid questions:

ADD REPLY
0
Entering edit mode

What happens if you instead tabix s3://*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000, where * is the bucket on S3? At least for bam/cram files that sort of thing works.

ADD REPLY
2
Entering edit mode
5.1 years ago

The problem might be the https: protocol. See this https://github.com/samtools/htslib/issues/139

TL; DR: Check that your htslib is current, and it has been installed with libcurl and libcrypto libraries. See requirements at https://github.com/samtools/htslib/blob/develop/INSTALL

HTSlib uses the following external libraries. Building requires both the library itself, and include files needed to compile code that uses functions from the library. Note that some Linux distributions put include files in a development ('-dev' or '-devel') package separate from the main library.

libz       (required)
libbz2     (required, unless configured with --disable-bz2)
liblzma    (required, unless configured with --disable-lzma)
libcurl    (optional, but strongly recommended)
libcrypto  (optional for Amazon S3 support; not needed on MacOS)
  
ADD COMMENT
1
Entering edit mode

^This. You need to build tabix with libcurl+openssl.

ADD REPLY
0
Entering edit mode

hello. Can you help me with the same issue. I am new to this environment of tabix and s3

ADD REPLY
0
Entering edit mode

hello. Can you help me with the same issue. I am new to this environment of tabix and s3

ADD REPLY

Login before adding your answer.

Traffic: 865 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6