dplyr filter and NA

[richardprichard]

Hello - first time posting and I do so with some worry that I am just having a senior moment.
I have a dataframe with a bunch of values and a fair smattering of NA. I want to exclude a particular value without losing all the NAs. An equivalent process can be simulated

> mtcars_withNA <- mtcars %>% mutate(cyl = ifelse(cyl>4, NA, cyl))
> mtcars_withNA
                     mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           21.0  NA 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       21.0  NA 160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710          22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive      21.4  NA 258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout   18.7  NA 360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant             18.1  NA 225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360          14.3  NA 360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 240D           24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
Merc 230            22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
Merc 280            19.2  NA 167.6 123 3.92 3.440 18.30  1  0    4    4
Merc 280C           17.8  NA 167.6 123 3.92 3.440 18.90  1  0    4    4
Merc 450SE          16.4  NA 275.8 180 3.07 4.070 17.40  0  0    3    3
Merc 450SL          17.3  NA 275.8 180 3.07 3.730 17.60  0  0    3    3
Merc 450SLC         15.2  NA 275.8 180 3.07 3.780 18.00  0  0    3    3
Cadillac Fleetwood  10.4  NA 472.0 205 2.93 5.250 17.98  0  0    3    4
Lincoln Continental 10.4  NA 460.0 215 3.00 5.424 17.82  0  0    3    4
Chrysler Imperial   14.7  NA 440.0 230 3.23 5.345 17.42  0  0    3    4
Fiat 128            32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
Honda Civic         30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
Toyota Corolla      33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1
Toyota Corona       21.5   4 120.1  97 3.70 2.465 20.01  1  0    3    1
Dodge Challenger    15.5  NA 318.0 150 2.76 3.520 16.87  0  0    3    2
AMC Javelin         15.2  NA 304.0 150 3.15 3.435 17.30  0  0    3    2
Camaro Z28          13.3  NA 350.0 245 3.73 3.840 15.41  0  0    3    4
Pontiac Firebird    19.2  NA 400.0 175 3.08 3.845 17.05  0  0    3    2
Fiat X1-9           27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
Porsche 914-2       26.0   4 120.3  91 4.43 2.140 16.70  0  1    5    2
Lotus Europa        30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2
Ford Pantera L      15.8  NA 351.0 264 4.22 3.170 14.50  0  1    5    4
Ferrari Dino        19.7  NA 145.0 175 3.62 2.770 15.50  0  1    5    6
Maserati Bora       15.0  NA 301.0 335 3.54 3.570 14.60  0  1    5    8
Volvo 142E          21.4   4 121.0 109 4.11 2.780 18.60  1  1    4    2
> mtcars_withNA %>% filter(cyl != 4)
 [1] mpg  cyl  disp hp   drat wt   qsec vs   am   gear carb
<0 rows> (or 0-length row.names)

Why, when I filter to exclude the 11 rows with 4 cylinders do I also lose the ones with NA cylinders? Why doesn't the test evaluate to NA (and not the TRUE that I thought was necessary for the filter to work)?

If it helps / is relevant I know I can do

> mtcars_withNA %>% filter(!cyl %in% c(4))
                     mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           21.0  NA 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       21.0  NA 160.0 110 3.90 2.875 17.02  0  1    4    4
Hornet 4 Drive      21.4  NA 258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout   18.7  NA 360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant             18.1  NA 225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360          14.3  NA 360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 280            19.2  NA 167.6 123 3.92 3.440 18.30  1  0    4    4
Merc 280C           17.8  NA 167.6 123 3.92 3.440 18.90  1  0    4    4
Merc 450SE          16.4  NA 275.8 180 3.07 4.070 17.40  0  0    3    3
Merc 450SL          17.3  NA 275.8 180 3.07 3.730 17.60  0  0    3    3
Merc 450SLC         15.2  NA 275.8 180 3.07 3.780 18.00  0  0    3    3
Cadillac Fleetwood  10.4  NA 472.0 205 2.93 5.250 17.98  0  0    3    4
Lincoln Continental 10.4  NA 460.0 215 3.00 5.424 17.82  0  0    3    4
Chrysler Imperial   14.7  NA 440.0 230 3.23 5.345 17.42  0  0    3    4
Dodge Challenger    15.5  NA 318.0 150 2.76 3.520 16.87  0  0    3    2
AMC Javelin         15.2  NA 304.0 150 3.15 3.435 17.30  0  0    3    2
Camaro Z28          13.3  NA 350.0 245 3.73 3.840 15.41  0  0    3    4
Pontiac Firebird    19.2  NA 400.0 175 3.08 3.845 17.05  0  0    3    2
Ford Pantera L      15.8  NA 351.0 264 4.22 3.170 14.50  0  1    5    4
Ferrari Dino        19.7  NA 145.0 175 3.62 2.770 15.50  0  1    5    6
Maserati Bora       15.0  NA 301.0 335 3.54 3.570 14.60  0  1    5    8

But I worry I've never noticed this behaviour with != before and it might be lurking elsewhere in my code.

[GreyMerchant]

So there is a good example on what happens with NA here: 5 Data transformation | R for Data Science Essentially, R doesn't know how it should evaluate an NA. Even when comparing NA against NA will you not get a TRUE as R can't truly know those two to be identical.

# Let x be Mary's age. We don't know how old she is.
x <- NA

# Let y be John's age. We don't know how old he is.
y <- NA

# Are John and Mary the same age?
x == y
#> [1] NA
# We don't know!


NA == NA
#> [1] NA

NA > 5
#> [1] NA
10 == NA
#> [1] NA
NA + 10
#> [1] NA
NA / 2
#> [1] NA

filter() only includes rows where the condition is TRUE ; it excludes both FALSE and NA values. If you want to preserve missing values, ask for them explicitly:

df <- tibble(x = c(1, NA, 3))
filter(df, x > 1)
#> # A tibble: 1 x 1
#>       x
#>   <dbl>
#> 1     3
filter(df, is.na(x) | x > 1)
#> # A tibble: 2 x 1
#>       x
#>   <dbl>
#> 1    NA
#> 2     3

I hope this helps?

[williaml]

Also from the dplyr::filter() documentation:

Note that when a condition evaluates to NA the row will be dropped, unlike base subsetting with [ .

[richardprichard]

Thanks both - very helpful.

Somehow I had got to this point in my tidyverse life without noticing this behaviour of filter() - I'll whizz over the rest of the code to ensure I haven't inadvertently made the same mistake elsewhere.

[williaml]

I hadn't noticed it either until I read your post.