bq_table_download download expected number of rows but most columns for some random number of rows become blank

General
1

I am using bq_table_download from R package bigrquery (version 1.3.2) to download a few million rows of data with three columns (short string or number). The resulting data frame had the expected number of rows, but in some rows (seem to be random) the values are gone and become empty. A mock summary of the output is shown below . I expect to see the same counts for each sample, and no sample is blank as sample is a key for inner join.

SAMPLE	COUNT

1 36494
2 sample_1 451
3 sample_2 436
4 sample_3 458
5 sample_4 428
....
668 sample_668 418
669 sample_669 458
670 sample_670 437
671 sample_671 428
672 sample_672 476

The code looks like

sample_df <- bq_table_download(bq_dataset_query(
"mydatasets.x",
sample_sql,
destination_table = NULL,
billing = NULL,
quiet = NA
),page_size=400000,
bigint="integer64")

I tested on bigquery console, and I do see same number of rows for each sample.

Any help is appreciated!

2

It looks this happens when page_size is increased, but with no error or warning.

3

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.