Unzip using loop problems

[giooe]

Hi! I have problems as I want to unzip the files from HTTP, still, the unzip function does not work for me and I'm having a dead end now.

Any help will be much appreciated

   res  =GET("https://github.com/binance/binance-spot-api-docs/blob/master/rest-api.md"
          )
cont_raw <- httr:: content(res)
options(stringsAsFactors = FALSE)

url  <- "https://data.binance.vision"

path <- '?prefix=data/spot/monthly/klines/LTCBTC/30m/'
waha <-read.csv(unzip(rbind, lapply(2017:2021, function(n) {
  url.1 <- paste0("https://data.binance.vision",
  path.1<-paste0("?prefix=data/spot/monthly/klines/LTCBTC/30m/", n, ".zip"))   
}), unzip = 'onlyfan.csv'), header=T)

[pieterjanvc]

HI,

First of all, it seems that the link you are generating is not correct, as there are no files with just a year as name (all named "LTCBTC-30m-2020-04.zip" for example).

R cannot download, decompress and zip files all at the same time I think, so I suggest you download the zip as a temp, then read it. Here is an example (you'll need to add the loops for getting all data)

#BASE URL
url = "https://data.binance.vision/data/spot/monthly/klines/LTCBTC/30m"

#Create temp file
tempFile = tempfile(fileext = ".zip")

#Download zip
download.file(sprintf("%s/LTCBTC-30m-%s.zip", url, "2020-04"), tempFile)

#Read
readr::read_csv(tempFile, col_names = F)

Hope this helps,
PJ

[giooe]

Big thanks for your kind assistance!

[giooe]

I tried loop over but it did not work out, any chance you can help me with this?

res  =GET("https://github.com/binance/binance-spot-api-docs/blob/master/rest-api.md"
          )
cont_raw <- httr:: content(res)
options(stringsAsFactors = FALSE)

url  = "https://data.binance.vision"

path = '?prefix=data/spot/monthly/klines/LTCBTC/30m/'
yearzz = as.Date(2017:2021)
waha <- list()


waha <-download.file(rbind, lapply(2017:2021, function(n) {
  url.1 <- paste0("https://data.binance.vision/data/spot/monthly/klines/LTCBTC/30m/LTCBTC-30m", yearzz, 
  path.1<-paste0("yearzz", n, ".zip"))
}))  
waha <- download.file(url = url.1, path= path.1, "LTCBTC.zip", method="curl", mode = "wb")


for (i in length(years)) {waha [[i]] <- download.file(url = url.1, path = path.1, paste0(yearzz), .zip, method = "curl", mode = "wb")
  
}
waha

[pieterjanvc]

HI,

Here is a loop that will check for all files within date range, then download, unzip and merge them.

library(httr)

#File name looks like: LTCBTC-30m-2018-05.zip

#BASE URL
url = "https://data.binance.vision/data/spot/monthly/klines/LTCBTC/30m"

#Dates to use
dates = seq(as.Date("2017-01-01"), as.Date("2021-12-01"), by = "month") 
dates = format(dates, "%Y-%m")

result = data.frame()

for(date in dates){
  
  #Check if the file with that date exists
  if(GET(sprintf("%s/LTCBTC-30m-%s.zip", url, date))$status_code == 200){
    
    #Create temp file
    tempFile = tempfile(fileext = ".zip")

    #Download zip
    download.file(sprintf("%s/LTCBTC-30m-%s.zip", url, date), tempFile)
    
    #Read
    newData = readr::read_csv(tempFile, col_names = F)
    newData$date = date
    
    #Merge
    result = rbind(result, newData)
    
    #remove temp file (optional as it is temp)
    file.remove(tempFile)
    
  }
  
}

PJ