Spark_read_csv making null some columns

Hi,

I am trying to load flights data following is code
if(!file.exists("./2003.csv.bz2")){
download.file("http://stat-computing.org/dataexpo/2009/2003.csv.bz2", "./2003.csv.bz2")

when I load in r data frame the it loads data
flights<- as.data.frame(read.csv("20031.csv"))

But when I am trying to load data in spark then it is making few last columns null
sp_flights <- spark_read_csv(sc,
name ="flights",
path = ".",
memory =FALSE,
#columns = file_columns,
infer_schema = FALSE)

what could be an issue? If it does make some columns null in spark data frame then it will be a big problem !

Please help !

best practice in tidy way:

flights <- readr::read_csv("20031.csv")
flights_tbl = flights %>% copy_to("flights")

@Suhasb11

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.