First of all, what is "crore"?
Second, I'm not sure there is anything we can help you with. Since you are not even at the stage where you are merging files together, but simply reading them into memory, there is nothing much that can be done. Your memory is simply insufficient for this task. You can try making files smaller and/or getting bigger machine (e.g., with AWS, GCP or similar).