Question any limit number of posts to download from API?
0
0
Entering edit mode
4.5 years ago
toshuta • 0

I tried to download 50000 posts from API by Python requests command. Is it possible? any limit number posts I could download? Thanks.

software error • 623 views
ADD COMMENT
0
Entering edit mode

Post lacks any details what you are talking about. Please add details: Which API, which posts, are you talking about Biostars? What are the errors you get?

ADD REPLY
0
Entering edit mode

Biostars API. I don't get output. Maybe, there is limit to download posts?

#title & comment: n number posts from "https://www.biostars.org/api/post/{id}"
for n in range(400000, 403674):
    n = str(n)
    biostars_web = "https://www.biostars.org/api/post/" + n
    response4 = requests.get(biostars_web)
    response4_json = response4.json()
    for key in response4_json:
        if key == 'title':
           title1 += [response4_json[key]]
        if key == 'xhtml':
           comm = [response4_json[key]]       
           comm = str(comm).replace('[','').replace(']','').rejavascript:document.forms["comment-form"].submit()place('\n','')
           comm = re.sub(clean, '', comm)          
           comment.append(comm)

data_tuples = list(zip(title1, comment))
df = pd.DataFrame(data_tuples, columns = ['title', 'comment'])

#filter with 9 keywords
df_filter = df.comment.str.contains('SRA | BLAST | GeneBank | RefSeek | pubmed | PMC | CDD | PubChem | SRAtoolkit')
df_2 = df[df_filter]
df_2
ADD REPLY
0
Entering edit mode

please at least add one line of time.sleep(n) after each loop....

ADD REPLY

Login before adding your answer.

Traffic: 2328 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6