I am investigating how to define a new language within RStudio. I have used Eclipse XText to develop it, so I even have generated an ACE editor.
To integrate this with RStudio, I can add my language to https://github.com/rstudio/rstudio/blob/5aa82fd564f356273b9c50f6cdc86a6f8d421abf/src/gwt/src/org/rstudio/studio/client/common/reditor/EditorLanguage.java
However, is this possible as some sort of plugin? Or will I have to build my seperate version of RStudio?
No, there's no way to do this with a plugin; RStudio's language system is currently not user extensible. You'll need to make a custom build of RStudio if you want to do this.
A couple of tips:
Building RStudio is not too hard. If you are making a build for Linux, and you have Docker installed, you can build RStudio with your change without installing anything else. See here for more: Building with Docker
We welcome pull requests! If you get it working with your language, you're welcome to submit a PR for consideration.