Hi,
I am trying to load flights data following is code
if(!file.exists("./2003.csv.bz2")){
download.file("http://stat-computing.org/dataexpo/2009/2003.csv.bz2", "./2003.csv.bz2")
when I load in r data frame the it loads data
flights<- as.data.frame(read.csv("20031.csv"))
But when I am trying to load data in spark then it is making few last columns null
sp_flights <- spark_read_csv(sc,
name ="flights",
path = ".",
memory =FALSE,
#columns = file_columns,
infer_schema = FALSE)
what could be an issue? If it does make some columns null in spark data frame then it will be a big problem !
Please help !