Loop to fetch data keeps taking more time

Hello guys

I have a cronjob that saves data on a web server (which basically every hour a new data is added). Currently, I am using a for-loop which reads the .csv from the URL does some calculations and stores the result in a vector and goes on to the next file and does the same. However, the computation time keeps getting longer and longer and I wonder whether there is a more efficient way to do it. Currently it looks something like this:

# x is a vector to store the time-identifier for the different datasets
for(i in 1:length(x)) {
  # read data
  table_1 <- read.csv(url(paste("some_url_table1", x[i], ".csv", sep="")))
  table_2 <- read.csv(url(paste("some_url_table2", x[i], ".csv", sep="")))
# do some calculations
store_data[i] = store some variable
store_data2[i] = store some other variable

Since this script should be used by several people it does not make sense to store data locally and do some tricks with that. Currently, this takes about 3-5 minutes. Is there a faster way?

for loops can be pernicious in R due to the time required to write and then garbage collect tmp files.

Either functionalize or write a shell script to traverse an argument list to pass to an R script.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.