Spark_read_csv making null some columns


I am trying to load flights data following is code
download.file("", "./2003.csv.bz2")

when I load in r data frame the it loads data

But when I am trying to load data in spark then it is making few last columns null
sp_flights <- spark_read_csv(sc,
name ="flights",
path = ".",
memory =FALSE,
#columns = file_columns,
infer_schema = FALSE)

what could be an issue? If it does make some columns null in spark data frame then it will be a big problem !

Please help !

best practice in tidy way:

flights <- readr::read_csv("20031.csv")
flights_tbl = flights %>% copy_to("flights")


This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.