Best practice to handle out-of-memory data

General
16

Thanks for your help. I am sorry about not using ```.

If I treat it as a query it works. For future reference:

  1. follow embedded instructions at: https://drill.apache.org/docs/drill-in-10-minutes/
    2)shell prompt: drill/bin/drill-embedded --verbose

3)To query file at /tmp/data.csv and have it process the headers

library(tidyverse)
library(sergeant)

drill_con <- drill_connection('localhost')
query <- "select * from table(dfs.`/tmp/data.csv`(type => 'text',fieldDelimiter => ',', extractHeader => true))"
drill_df <- drill_query(drill_con,query)
drill_df
A tibble: 3 x 3
fname score  lname
* <chr> <int>  <chr>
1  mark     1  smith
2 betty     2 wilson
3   jim     3  mccoy