No, you don't need to worry about concurrency limit. Each student will have their own account, so they will have personal RStudio Cloud all for themselves.
Only thing you need to keep in mind - there is 1 GB RAM cap, so if you are going to do something slightly heavy, it won't work. So it is best to go over everything you are going to do beforehand just to make sure.
Finally, what you can do to make onboarding even more smooth is to prepare base project that students will start from. This way they won't need to install dependencies and just R straight away.