Loop to fetch data keeps taking more time

Hello guys

I have a cronjob that saves data on a web server (which basically every hour a new data is added). Currently, I am using a for-loop which reads the .csv from the URL does some calculations and stores the result in a vector and goes on to the next file and does the same. However, the computation time keeps getting longer and longer and I wonder whether there is a more efficient way to do it. Currently it looks something like this:

# x is a vector to store the time-identifier for the different datasets
for(i in 1:length(x)) {
  # read data
  table_1 <- read.csv(url(paste("some_url_table1", x[i], ".csv", sep="")))
  table_2 <- read.csv(url(paste("some_url_table2", x[i], ".csv", sep="")))
# do some calculations
store_data[i] = store some variable
store_data2[i] = store some other variable
}

Since this script should be used by several people it does not make sense to store data locally and do some tricks with that. Currently, this takes about 3-5 minutes. Is there a faster way?
Thanks!

for loops can be pernicious in R due to the time required to write and then garbage collect tmp files.

Either functionalize or write a shell script to traverse an argument list to pass to an R script.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.