Question: Question any limit number of posts to download from API?
0
gravatar for toshuta
27 days ago by
toshuta0
toshuta0 wrote:

I tried to download 50000 posts from API by Python requests command. Is it possible? any limit number posts I could download? Thanks.

software error • 108 views
ADD COMMENTlink written 27 days ago by toshuta0

Post lacks any details what you are talking about. Please add details: Which API, which posts, are you talking about Biostars? What are the errors you get?

ADD REPLYlink written 27 days ago by ATpoint25k

Biostars API. I don't get output. Maybe, there is limit to download posts?

#title & comment: n number posts from "https://www.biostars.org/api/post/{id}"
for n in range(400000, 403674):
    n = str(n)
    biostars_web = "https://www.biostars.org/api/post/" + n
    response4 = requests.get(biostars_web)
    response4_json = response4.json()
    for key in response4_json:
        if key == 'title':
           title1 += [response4_json[key]]
        if key == 'xhtml':
           comm = [response4_json[key]]       
           comm = str(comm).replace('[','').replace(']','').rejavascript:document.forms["comment-form"].submit()place('\n','')
           comm = re.sub(clean, '', comm)          
           comment.append(comm)

data_tuples = list(zip(title1, comment))
df = pd.DataFrame(data_tuples, columns = ['title', 'comment'])

#filter with 9 keywords
df_filter = df.comment.str.contains('SRA | BLAST | GeneBank | RefSeek | pubmed | PMC | CDD | PubChem | SRAtoolkit')
df_2 = df[df_filter]
df_2
ADD REPLYlink modified 27 days ago by genomax74k • written 27 days ago by toshuta0

please at least add one line of time.sleep(n) after each loop....

ADD REPLYlink written 27 days ago by shoujun.gu250
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 968 users visited in the last hour