We've seen this occur when an R Markdown document contains a chunk which generates a very large amount of console output.
If you haven't already, you might also try updating to the latest version of RStudio (v1.2.5019) as it contains some mitigations for this issue (if it is indeed what you are seeing).