How to automate a data scraping script crypto

Hi

I'm trying to get better at coding, I'd like to do with a practical case, making an algorithm that allows to scrape data.

I've been using cryptocurrencies, and I found an R code that allows to retrieve historical prices via an API.

rm(list = ls())  # reset global variables
#import the libraries we need
library(jsonlite)
library(glue)
# create a function to retrieve daily data
retreive_daily_data <- function(pair, filename) {
    url = glue("https://api.pro.coinbase.com/products/{pair}/candles?granularity=86400")
    columnNames <- c('unix', 'low', 'high', 'open', 'close', glue('{pair} volume'))
    mydata <- fromJSON(url)
    df <- as.data.frame(mydata)
    colnames(df) <- columnNames  # rename the columns

    write.csv(df, file = filename)
    }


newPair <- "BTC-USD" #Insert crypto name here followed by -USD
fileName <- glue("dailyData{newPair}.csv")
runFunc <- retreive_daily_data(newPair, filename = fileName)
runFunc

In the current code, you have to fill in the name of the crypto in line 21, then the script downloads the file in csv format that contains its information (price, volume, etc).

I would like to automate it by making a loop, which will automatically download all the CSV of the cryptos filled in a other database file "with all the crypto in the column named "NOM".


library(readxl)
database <- read_excel("database.xlsx")

for (i in 1:nrow(database)) {
    if(!is.na(database$NOM[i])){ #i think this line is useless but i also try without for the same result
        
        name <- database$NOM
        
        retreive_daily_data <- function(pair, filename) {
        url = glue("https://api.pro.coinbase.com/products/{pair}/candles?granularity=86400")
        columnNames <- c('unix', 'low', 'high', 'open', 'close', glue('{pair} volume'))
        mydata <- fromJSON(url)
        df <- as.data.frame(mydata)
        colnames(df) <- columnNames  # rename the columns

        write.csv(df, file = filename)
        }

        newPair <- "{name}-USD"
        fileName <- glue("dailyData{newPair}.csv")
        runFunc <- retreive_daily_data(newPair, filename = fileName)
        runFunc
    }
}

Unfortunately this only displays the following error message: Error in open.connection(con, "rb") : HTTP error 404.

In a second time I will make a loop that will import the CSV and extract the data I am interested in in a common database. Do you have any idea how to deal with that issue ?

Thanks

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.