Hello everyone,
I am trying to scrape a table from the ClinicalTrial.gov website from the following link:
Under the "By Topic" tab is a table showing 25 results from a total of 388 results. I can't seem to get the table to load using the xml2 and rvest packages and so I tried loading the website using PhantomJS. The code for the js and R scripts are shown below. The table will only load the 25 results orginally showing on the website. Does anyone know how I can get all 388 results from the table? Any help would be greatly appreciated.
js:
// scraper.js
var page = require('webpage').create();
var fs = require('fs');
var url = 'https://www.clinicaltrials.gov/ct2/results/browse?recrs=abdefm&intr="ibuprofen"&titles="ibuprofen"&brwse=cond_alpha_all';
var path = 'scraper.html';
page.open(url, function (status) {
if(status !== 'success')
console.log('Connection failed, page was not loaded!');
else
var content = page.content;
fs.write(path,content,'w')
phantom.exit();
});
R:
library(dplyr)
library(xml2)
library(rvest)
system("./phantomjs-2.1.1-windows/bin/phantomjs scraper.js")
raw_tbl <- xml2::read_html("scraper.html") %>%
rvest::html_nodes(xpath='//*[@id="theDataTable"]') %>%
rvest::html_table() %>%
.[[1]]