Question: Question any limit number of posts to download from API?
0
gravatar for toshuta
9 months ago by
toshuta0
toshuta0 wrote:

I tried to download 50000 posts from API by Python requests command. Is it possible? any limit number posts I could download? Thanks.

software error • 167 views
ADD COMMENTlink written 9 months ago by toshuta0

Post lacks any details what you are talking about. Please add details: Which API, which posts, are you talking about Biostars? What are the errors you get?

ADD REPLYlink written 9 months ago by ATpoint36k

Biostars API. I don't get output. Maybe, there is limit to download posts?

#title & comment: n number posts from "https://www.biostars.org/api/post/{id}"
for n in range(400000, 403674):
    n = str(n)
    biostars_web = "https://www.biostars.org/api/post/" + n
    response4 = requests.get(biostars_web)
    response4_json = response4.json()
    for key in response4_json:
        if key == 'title':
           title1 += [response4_json[key]]
        if key == 'xhtml':
           comm = [response4_json[key]]       
           comm = str(comm).replace('[','').replace(']','').rejavascript:document.forms["comment-form"].submit()place('\n','')
           comm = re.sub(clean, '', comm)          
           comment.append(comm)

data_tuples = list(zip(title1, comment))
df = pd.DataFrame(data_tuples, columns = ['title', 'comment'])

#filter with 9 keywords
df_filter = df.comment.str.contains('SRA | BLAST | GeneBank | RefSeek | pubmed | PMC | CDD | PubChem | SRAtoolkit')
df_2 = df[df_filter]
df_2
ADD REPLYlink modified 9 months ago by genomax87k • written 9 months ago by toshuta0

please at least add one line of time.sleep(n) after each loop....

ADD REPLYlink written 9 months ago by shoujun.gu310
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1637 users visited in the last hour