Error encountered during execution. Retrying may solve the problem

RStudio IDE
1

Hi guys,

I'm running this code:

library(bigrquery)
project_id <- "projectid"
sql_string <- "SELECT * FROM mydb"
i_data <- query_exec(sql_string, project = project_id, useLegacySql = FALSE, destination_table = "projectid:myotherdb", allowLargeResults=TRUE, max_pages = Inf)

..on a pretty massive gcloud VM on windows (12core, 64gb mem).
It gets the db from googles bigquery db (about 50db data)

The issue is I get this error after 38% (only run it once as it took me about 8h to get this far) :
"Error encountered during execution. Retrying may solve the problem"

Any help appreciated!
Kaspar

2

Kaspar,

Are you trying to pull large amounts of data out of the cloud and onto your desktop? If so, would recommend a different approach. In my experience, extracting data via ODBC for large amounts of data is fraught with issues. Connections break, disks fill up, queuing can be problematic, etc.

Nathan

2 Likes
3

Rather than pulling all of the data using odbc, I suggest you use dplyr with the bigquery backend to pre-aggregate your data in the database (think group-by summaries), and then pull the summaries to your R session.

The bigrquery package was recently pushed to CRAN and will help with this:

For a really great introduction on using the package, see:

http://db.rstudio.com/databases/big-query/

1 Like