Error encountered during execution. Retrying may solve the problem


Hi guys,

I'm running this code:

project_id <- "projectid"
sql_string <- "SELECT * FROM mydb"
i_data <- query_exec(sql_string, project = project_id, useLegacySql = FALSE, destination_table = "projectid:myotherdb", allowLargeResults=TRUE, max_pages = Inf)

..on a pretty massive gcloud VM on windows (12core, 64gb mem).
It gets the db from googles bigquery db (about 50db data)

The issue is I get this error after 38% (only run it once as it took me about 8h to get this far) :
"Error encountered during execution. Retrying may solve the problem"

Any help appreciated!



Are you trying to pull large amounts of data out of the cloud and onto your desktop? If so, would recommend a different approach. In my experience, extracting data via ODBC for large amounts of data is fraught with issues. Connections break, disks fill up, queuing can be problematic, etc.



Rather than pulling all of the data using odbc, I suggest you use dplyr with the bigquery backend to pre-aggregate your data in the database (think group-by summaries), and then pull the summaries to your R session.

The bigrquery package was recently pushed to CRAN and will help with this:

For a really great introduction on using the package, see: