Parsing Dates from Multiple Websites

Question

[rohitsatyam102]

Hi Everyone!!

I have 100 COVID databases from where I wish to enlist the date of update. Some are regularly updated but some aren't.

I am trying to do this in R. Example given below. I wish to write a function that I can run on those 100 links weekly to record when were those 100 databases updated.

library(stringr)
library(rvest)
library(lubridate)

html <- readLines("https://grafnet.kaust.edu.sa/assayM/")
t <- html[grep(pattern = "(?i)(last update | update | updated)", x = html, ignore.case = TRUE)]
cleanFun <- function(htmlString) {
  return(gsub("<.*?>", "", htmlString))
}
t <- cleanFun(t)

dmy(t)

I can't move beyond this step. I know that if the string gets smaller I can use dmy function of lubridate. But I am poor at regex. can someone help?

[Dunois]

Are you sure scraping the HTML yields the date of the last update for all the datasets listed there? Did you actually take a look at the page source first?

[rohitsatyam102]

Well, it does for most of them, if not all. I have manually sorted the ones that don't. The problem is I don't get the dates always. Often keywords such as Daily, Monthly, Weekly are also included.

Here are few of the databases out of those 100:

1. https://91-divoc.com/pages/covid-visualization/
2. https://covidibd.org/current-data/
3. https://cov3d.ibbr.umd.edu/

But for some it doesn't: For example: http://covdb.popgenetics.net/v3/index/update

[cpad0112]

what is the expected output?

[rohitsatyam102]

The expected output is a dataframe with database name which I have in my excel sheet, the link (again in my excel sheet), and a new column with update date:

assayM	https://grafnet.kaust.edu.sa/assayM/	23-Dec-2020
BioGRID COVID-19 Coronavirus Curation Project	https://thebiogrid.org/project/3/covid-19-coronavirus.html	30-Mar-2021
Contact-guided Iterative Threading ASSEmbly Refinement (C-I-TASSER)	https://zhanglab.ccmb.med.umich.edu/COVID-19/	15-Jan-2021

[steve]

I am not sure that HTML scraping & text parsing is the best method for this, obviously it seems to work for a lot of cases but the complexity involved it handling all the possible edge cases seems like a big headache to me. Is there not way to programmatically query the databases directly and check for such things? Or if these are file downloads instead of API's, could you just maintain a database of md5sum's for the files and re-download & compare them to determine if something has changed? You might consider a two-tiered approach where you use your current HTML scraping/parsing method for the 'easy' ones, and take more extreme methods such as actually downloading & checking md5 for the harder cases. In general, determining this from the HTML web pages alone seems not ideal if you care about accuracy, because the web page could change at any time and break you parsing script, better to go directly to the data source if you can.

[rohitsatyam102]

most of these servers are shinyapps/dashboards and therefore no API.

Answer 1

[cpad0112]

> source= "https://grafnet.kaust.edu.sa/assayM/"
> html <- readLines(source)
> t <- html[grep(pattern = "(?i)(last update | update | updated)", x = html, ignore.case = TRUE)]
> cleanFun <- function(x) {
+     return(str_split(t,"Updated | based|update")[[1]][2] %>% 
+                dmy(.))
+ }
> data.frame (source, "date"=cleanFun(t))
                                source       date
1 https://grafnet.kaust.edu.sa/assayM/ 2020-12-23

But this function is not going to work for every site. You may have to write better functions for extracting dates from each website. Btw, Where are you getting the "assayM" from ?

[cpad0112]

Well, there are several ways to skin a cat in R. Here is the code I would prefer to scrap the same information:

library(stringr)
library(rvest)
library(lubridate)
library(dplyr)
library(purrr)

source= "https://grafnet.kaust.edu.sa/assayM/"
html=read_html(source)
html %>% 
    html_elements("h5") %>% 
    html_text2() %>% 
    tibble() %>%  
    slice(1L) %>% 
    str_split(.,"Updated | based") %>% 
    pluck(1,2) %>% 
    dmy()

[1] "2020-12-23"