Parallel Process in Windows using doParallel & foreach


I need help in parallel processing, currently the sequential loop takes a lot of time to execute as the data is huge, the function refers multiple tables from sql database and create 3 dataframe in each loop , in each sequence the data get appended in the 3 dataframe. is it possible to do parallel processing ?

Here is the sample code

date <- read.csv("date_split_sample.csv")
start_date end_date
01-Jan-2021 30-jan-2021
01-Feb-2021 28-Feb-2021

numcores <- detectcores()
cl <- makecluster(numcores,type="PSOCK")

foreach (i=1:nrow(date),.combine='rbind') %dopar%{
a <- db.q("select * from abcd where date between start_date[i] and end_date[i] )
b <- db.q("select * from efgh where date between start_date[i] and end_date[i] )
c <- db.q("select * from ijkl where date between start_date[i] and end_date[i] )

#some process using above dataframe and 3 dataframe created in each loop
i want these 3 datafame in such a way that in each loop data gets appended

I donot know how to create multiple dataframe & append the records of each loop.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.