Entering edit mode
6 months ago
I am conducting a study where I need access to data on this forum on how to improve genomic data visualization tools. I have tried scraping using the word "visualization" but the website does not allow scraping. I tried fetching the posts using the API on https://www.biostars.org/info/api/ but it does not fetch. Is there a way I can do this? or is there a way I can get the data?
Yeah the tag based fetch is broken. Scraping is bad but you can pull each post and check tags there - add an interval between each curl and grep for
tag_val:. It's extremely inefficient though; but scraping won't help you much - for each tag, the /tags/ page shows only the top 50 results, no matter how many hits it gets. The "next" buttons don't work.