Question: Question any limit number of posts to download from API?
gravatar for toshuta
9 months ago by
toshuta0 wrote:

I tried to download 50000 posts from API by Python requests command. Is it possible? any limit number posts I could download? Thanks.

software error • 167 views
ADD COMMENTlink written 9 months ago by toshuta0

Post lacks any details what you are talking about. Please add details: Which API, which posts, are you talking about Biostars? What are the errors you get?

ADD REPLYlink written 9 months ago by ATpoint36k

Biostars API. I don't get output. Maybe, there is limit to download posts?

#title & comment: n number posts from "{id}"
for n in range(400000, 403674):
    n = str(n)
    biostars_web = "" + n
    response4 = requests.get(biostars_web)
    response4_json = response4.json()
    for key in response4_json:
        if key == 'title':
           title1 += [response4_json[key]]
        if key == 'xhtml':
           comm = [response4_json[key]]       
           comm = str(comm).replace('[','').replace(']','').rejavascript:document.forms["comment-form"].submit()place('\n','')
           comm = re.sub(clean, '', comm)          

data_tuples = list(zip(title1, comment))
df = pd.DataFrame(data_tuples, columns = ['title', 'comment'])

#filter with 9 keywords
df_filter = df.comment.str.contains('SRA | BLAST | GeneBank | RefSeek | pubmed | PMC | CDD | PubChem | SRAtoolkit')
df_2 = df[df_filter]
ADD REPLYlink modified 9 months ago by genomax87k • written 9 months ago by toshuta0

please at least add one line of time.sleep(n) after each loop....

ADD REPLYlink written 9 months ago by shoujun.gu310
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1637 users visited in the last hour