seems like an interesting and ambitious project !
shinyapps.io does list some ideas on permanent storage options, might make a useful read:
https://shiny.rstudio.com/articles/persistent-data-storage.html
I'm sure I don't have a good understanding of your requirements, but you might struggle to have very large in memory objects at once in the app. It may be that you have to limit your users to process no more than a certain volume based on the shiny app constraints. Though I think if thats the case, you would benefit from preprocessing your data so that its in roughly evenly sized chunks, to help you with predictability of performance etc.