False Negative Error: org.apache.spark.sql.AnalysisException: cannot resolve 'A' given input columns

bhuvanesh1707 · September 12, 2022, 6:13am

I am facing a challenge to execute my R code in sparklyr (version 1.7.7) , where it throws an error "cannot resolve a 'column'" which means the column is not present but the column is present in my spark data frame. The same R code is working in previous sparklyr version (1.7.5).
I am guessing it fails to identify the columns when it goes beyond 100 columns or so.
Even the error is not constant, it either throws "column not found" or "invalid data type casting (decimal vs big decimal) or (date vs timestamp)".
Below is the link for code snippet and approaches taken to resolve it

github.com/sparklyr/sparklyr

False Negative Error: org.apache.spark.sql.AnalysisException: cannot resolve 'A' given input columns

opened 03:17PM - 16 Aug 22 UTC

closed 08:14AM - 26 Sep 22 UTC

bhuvanesh1707

bug

I am facing a challenge to execute my R code in sparklyr (version 1.7.7) and dbp…lyr (version 2.2.0), where it throws an error "cannot resolve a 'column'" which means the column is not present but the column is present. The same R code is working in previous sparklyr version (1.7.5) and dbplyr (2.1.1). I have created a dummy data with 1 row and 62 columns and did some calculation. I have column 'A' present in my dummy data but during the calculation it errors "cannot resolve column'A'" I am guessing it fails to identify the columns when it goes beyond 100 columns or so. ```r #Load libraries library(DBI) #1.1.2 library(sparklyr) #1.7.7 library(dplyr) #1.0.9 library(dbplyr) #2.2.0 #Dummy data frame with 1 row and 62 columns dummy_data <- data.frame( A = "XX" , B = 1L , U = 1L , D = T , A1 = as.Date("1900-01-01") , A2 = 1L , A3 = 1L , A4 = 1L , A5 = 1L , A6 = T , A7 = as.Date("1900-01-01") , A8 = T , A9 = T , A10 = "XX" , A11 = "XX" , A12 = T , A13 = T , A14 = as.Date("1900-01-01") , A15 = T , A16 = T , A17 = 1L , A18 = 1L , A19 = T , A20 = as.Date("1900-01-01") , A21 = as.Date("1900-01-01") , A22 = 1L , A23 = 1L , A24 = 1L , A25 = 1L , A26 = 1L , A27 = 1L , A28 = 1L , A29 = T , A30 = T , A31 = T , A32 = T , A33 = T , A34 = T , A35 = "XX" , A36 = "XX" , A37 = "XX" , A38 = "XX" , A39 = "XX" , A40 = "XX" , A41 = T , A42 = as.Date("1900-01-01") , A43 = T , A44 = 1L , A45 = T , A46 = 1L , A47 = as.Date("1900-01-01") , A48 = "XX" , A49 = as.Date("1900-01-01") , A50 = as.Date("1900-01-01") , A51 = as.Date("1900-01-01") , A52 = as.Date("1900-01-01") , A53 = as.Date("1900-01-01") , A54 = "XX" , A55 = as.Date("1900-01-01") , A56 = 1L , A57 = 1L , A58 = T ) #Create spark connection spark_disconnect_all() cn <- spark_connect(master = "yarn-client", app_name = "newJob") #Copying dummy_data data frame to spark data frame spark_dummy_data <- copy_to(cn, dummy_data, overwrite = T) %>% compute() #Execute code snippet A <- spark_dummy_data %>% mutate(excl = case_when( A %in% c("HHH","GGG") ~ (A17 == 1 & !A13) , A == "HHH" ~ T , T ~ (!A13) )) %>% filter(excl) %>% mutate( COL1 = case_when( A == "III" ~ B %in% c(3,72,70) , T ~ (B == 3) ) , COL2 = (B == 34) , COL3 = (B == 71) , COL4 = (B == 79) , COL5 = (B == 83) , COL6 = case_when(A == "PPP" ~ B %in% c(5,6), T ~ B == 6) , COL7 = case_when( A == "III" ~ (B == 5 | B == 6) , T ~ COL6 ) , COL8 = case_when( A == "III" ~ B %in% c(4,69,74) , T ~ (B == 4) ) , COL9 = (B == 4 & A31) , COL10 = (B == 25) , COL11 = (B ==5) , COL12 = case_when( A == "GGG" ~ F , T ~ T ) , COL13 = A15 , COL14 = B %in% c(34,35,21,9,37,38,39) , COL15 = case_when( A == "III" ~ (B != 21) , A == "CCC" ~ A30 , A == "ZZZ" ~ !COL14 , A == "PPP" ~ (B != 17 & B != 21) , T ~ (U < 12) ) , COL16 = case_when( A %in% c("III","ZZZ") ~ (B == 20) , A == "CCC" ~ (U == 15) , T ~ (U == 11) ) , COL17 = (B == 19) , COL18 = (B == 29) , COL19 = case_when(A == "PPP" ~ case_when(A18 > 60 & !is.na(A18) ~ TRUE , T ~ FALSE) , T ~ (B ==90)) , COL20 = case_when(A == "PPP" ~ case_when((D | COL18 | COL6 | COL1) ~ case_when((A35 == "M" & (A18) >= 37 & (A18) <= 60) | (A35 == "A" & (A18*12) >= 37 & (A18*12) <= 60) | (A35 == "B" & (A18/2) >= 37 & (A18/2) <= 60) | (A35 == "Q" & (A18*3) >= 37 & (A18*3) <= 60) | (A35 == "S" & (A18*6) >= 37 & (A18*6) <= 60) | (A35 == "W" & (A18/4) >= 37 & (A18/4) <= 60) ~ TRUE , T ~ FALSE) , T ~ FALSE) , T ~ ( B==91 )) , COL21 = case_when(A == "PPP" ~ case_when(D | COL18 | COL6 | COL1 ~ case_when((A35 == "M" & A18 == 1) | (A35 == "W" & (A18/4) == 1) ~ TRUE , T ~ FALSE) , T ~ FALSE) , T ~ ( B==33 )) , COL22 = case_when(A == "PPP" ~ case_when(D | COL18 | COL6 | COL1 ~ case_when((A35 == "M" & (A18) >= 2 & (A18) <= 36) | (A35 == "A" & (A18*12) >= 2 & (A18*12) <= 36) | (A35 == "B" & (A18/2) >= 2 & (A18/2) <= 36) | (A35 == "Q" & (A18*3) >= 2 & (A18*3) <= 36) | (A35 == "S" & (A18*6) >= 2 & (A18*6) <= 36) | (A35 == "W" & (A18/4) >= 2 & (A18/4) <= 36) ~ TRUE , T ~ FALSE) , T ~ FALSE) , T ~ ( B==92 )) , COL23 = (COL21 & U == 1) , COL24 = (COL22 & U == 1) , COL25 = (A15 & U == 1) , COL26 = (D & U == 2) , COL27 = (A15 & U == 2) , COL28 = (U == 2 & COL19) , COL29 = (U == 2 & COL20) , COL30= (U == 2 & COL21) , COL31 = (U == 2 & COL22) , COL32 = B ==35 , COL33 = F , COL34 = case_when( A == "GGG" ~ (B == 28) , A == "ZZZ" ~ F , T ~ (B == 59) ) , COL35 = case_when( A == "GGG" ~ as.integer(A37) %in% c(1, 2, 6, 8, 9) & as.integer(A38) %in% c(29, 62) , A == "ZZZ" ~ F ,T ~ (B == 13) ) , COL36 = case_when( A == "ZZZ" ~ F , T ~ (B == 8) ) , COL37 = (U == 3) , COL38 = case_when( A == "ZZZ" ~ F , T ~ (D & U != 1) ) ) %>% compute() ```

bhuvanesh1707 · September 26, 2022, 8:29am

I am satisfied with the solution provided by @edgararuiz after installing the dev version of the sparklyr my code executed successfully. Thanks.

system · November 7, 2022, 8:30am

This topic was automatically closed 42 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.