start small. make a robust process. profile its timing, estimate its scalability.
These steps are all important on your way to addressing a potential monster task, and they might encourage you to abandon your dreams, or put a price on it. I.e. you might estimate how much cloud compute you might need to execute in reasonable time, and see how much that would cost on azure.
If you supply a small reprex, then the community on the forum could begin to attempt to consider how they might address your challenge.
FAQ: How to do a minimal reproducible example ( reprex ) for beginners