What is the best way to run parallelized batch jobs in Amazon's RStudio Server Pro for AWS?

I currently have R code that can be run through a Slurm manager with a shell and batch script. Essentially, my shell script creates 1000 job arrays by calling the batch script 1000 times.

I am wondering how I can take this set-up and what might be the most efficient way to transfer it to RStudio Server Pro for AWS. I was able to familiarize myself with it, but it seems like an interactive panel, only designed for one job at a time.

Is there a way to take batch files or even shell files, to parallelize the running of jobs on Server Pro? Thanks.

It looks like this may be a good fit for what you're looking for, it's created by @davis:

https://twitter.com/dvaughan32/status/1002230212170584064

1 Like