Hello guys
I have a cronjob that saves data on a web server (which basically every hour a new data is added). Currently, I am using a for-loop which reads the .csv from the URL does some calculations and stores the result in a vector and goes on to the next file and does the same. However, the computation time keeps getting longer and longer and I wonder whether there is a more efficient way to do it. Currently it looks something like this:
# x is a vector to store the time-identifier for the different datasets
for(i in 1:length(x)) {
# read data
table_1 <- read.csv(url(paste("some_url_table1", x[i], ".csv", sep="")))
table_2 <- read.csv(url(paste("some_url_table2", x[i], ".csv", sep="")))
# do some calculations
store_data[i] = store some variable
store_data2[i] = store some other variable
}
Since this script should be used by several people it does not make sense to store data locally and do some tricks with that. Currently, this takes about 3-5 minutes. Is there a faster way?
Thanks!