I'm not really sure where to start debugging this, but in the last year I routinely have to wait a few minutes before RStudio becomes responsive after executing code chunks. This is not while RStudio is running a chunk but after chunks complete. I have grown accustomed to running beepr::beep() in the console to signal when RStudio is going to become responsive again.
These delays only seem to happen when RStudio is connected to a database, but I'm not sure how to diagnose or even detect the source of delays as they seem to happen randomly. I am at least sure it's not do to memory issue as the delays happen regardless of how much data I have pulled into memory.
PS: The solution was to stop using SQL chunks in RStudio. The SQL chunks would query metadata from our very large database and that was bogging down RStudio. When I reverted to running SQL chunks using dbGetQuery() within R chunks the problem was solved. While SQL chunks have many nice features, the delays made them unusable.
a small code chunk that consistently reproduces the problem would be useful, along with any details of our environment; version of RSTudio, R, and details of the database connection if you feel that's related to the slowdown.
I typically encounter the freeze while connected to my company's AWS Athena database. I connect to Athena via dbConnect and RJDBC (athena_url is a connection string):
A typical code chunk like the following might freeze RStudio for 5 minutes, although it's usually the 3rd or 4th sql chunk that sets off the freeze. Mind you these are very small data being returned and the chunk is not executing during the freeze. It's after execution that RStudio becomes unresponsive.
```{sql, connection = athena}
SELECT *
FROM dlprd.SBG_PUBLISHED.qbcommerce_ongoing_usage
LIMIT 10
``` .
I am using RStudio Version 1.3.1093 and
> R.version
_
platform x86_64-apple-darwin17.0
arch x86_64
os darwin17.0
system x86_64, darwin17.0
status
major 4
minor 0.3
year 2020
month 10
day 10
svn rev 79318
language R
version.string R version 4.0.3 (2020-10-10)
nickname Bunny-Wunnies Freak Out
A colleague of mine has noticed that this freeze only occurs when RStudio is connected to our Athena database and that removing this athena connections via rm(athena) will prevent future freezes, but obviously inhibit a workflow where connection to an Athena database is central .
Yes I can run db code in my console, but my freeze is a different issue than what you linked to. The period of non-responsiveness does not happen while the sql executes. It can happen anytime while RStudio is connected to Athena. During this period I can type code in the console and in the Rmd document, but I cannot execute any of the code nor save my file. I have taken to typing beepr::beep() in my console and waiting for the beep to indicate that RStudio is responsive again. This non-responsive period typically lasts 5 - 30 minutes.
I am going to try to connect RStudio to Athena via ODBC rather than JDBC to see if this prevents the periods of non-responsiveness in RStudio.
@MikeBess I worked with some AWS engineers on this problem and they suspected that RStudio is intermittently pulling all the metadata from Athena (i.e., the name of every table in Athena), and this is bogging down RStudio and causing it to become unresponsive. Is there a way to deactivate this pulling of metadata by RStudio? Or do you suspect that pulling this metadata is driven by the driver? I am using the AthenaJDBC42.jar Driver.
I was already using the AthenaJDBC42.jar driver but have updated with the latest download of AthenaJDBC42.jar available here. We also removed the default catalog from the connection string for Athena and set UseResultsetStreaming=1 in that string. These changes seem to have prevented further sessions of unresponsiveness, but I will return if anything changes.
This sounds plausible. Beyond explicit user gestures in the Connections pane, most of the UI updates are driven by R itself (e.g. by the driver or by the driver interface package).
There's a couple things we can do to get more information.
The next time the hang occurs, try looking at Help -> Diagnostics -> Request Log. You'll be presented with a graph of some of the different operations RStudio has been doing. If you see a "long" bar (other than 'get_events'), that may be associated with the slowness you're seeing.
You could also try generating an R profile. Within R, execute:
utils::Rprof("~/rstudio-trace.Rprof")
to start the profiler. Then, work with the IDE until the next bit of slowness occurs. When that happens, you can stop the profiler with:
utils::Rprof(NULL)
Then, share the output of:
utils::summaryRprof("~/rstudio-trace.Rprof")
and that may point us towards the function(s) being invoked that are taking a long time to complete.
I ran Help -> Diagnostics -> Request Log and see several long bars besides "get_events", but I am not sure what remedial actions these bars would suggest I take.
I then began generating an R profile via utils::Rprof("~/rstudio-trace.Rprof") until another lag occurred, at which point I stopped the profiler and generated the output below. Something called .jcall seems to be the culprit.
Thanks! The output here makes it clear that the slowness is caused by RStudio's completion engine, which is trying to list tables on the completion engine:
@kevinushey I have disabled autocompletions as you advised, but I am still experiencing frequent lags. I reran the summary after another lag. Do you have any other suggestions? For instance is there a way to disable R from fetching metadata from Athena?
