Have you installed rmarkdown? Do you get any error message while installing?
Have in mind that you need to install rmarkdown system wide for shiny-server to be able to use it.
Also, please check if there are any logs available for that application at this folder /var/log/shiny-server/