Read and Merge large files

[Atma_Deep_Dutta]

I am trying to read two files having more than 5 crore values. After the files are being read, I need to merge them and this process will be done for more 8 files having almost an equivalent of 40 crore values. The first file could be read easily but when I am trying to read the second file, it is showing me an error "Error: memory exhausted (limit reached?)
Error during wrapup: memory exhausted (limit reached?)". Please help me with the solution so that I can read and merge the files. I have used the following code

dat1 <- read.csv("R_T_P_1.csv")
dat2 <- read.csv("R_T_P_2.csv")
Error: memory exhausted (limit reached?)
Error during wrapup: memory exhausted (limit reached?)

[mishabalyasin]

First of all, what is "crore"?
Second, I'm not sure there is anything we can help you with. Since you are not even at the stage where you are merging files together, but simply reading them into memory, there is nothing much that can be done. Your memory is simply insufficient for this task. You can try making files smaller and/or getting bigger machine (e.g., with AWS, GCP or similar).

[Yarnabrina]

@mishabalyasin

First of all, what is "crore"?

Crore means 10 million, i.e. 1e+7.

[jdlong]

50m records merging with 400m records will be big. If you want to shove that into memory you'll need a big box. You'll probably be better off shoving this into a database, not using an in-memory solution like R. I'd recommend Postgres if you don't already have a database you know and love.