calculate new columns from existing column by group and assigned the calculated value for entire group (observations)

[dthumar16]

I am trying to calculate the mean, standard deviation and z score to calculate the outlier.
I need to remove outlier if my var2 is greater than or equal to 4.

My R code:

#var2 > = 4

ot_data = clean_df %>% group_by(var1) %>%

mutate(Mean = mean(var2), SD = sd(var2))%>%

mutate(lower_limit = -Inf) %>%

mutate(lower_limit = (ifelse(var2 >= 4, (Mean - 2*SD),-Inf)))%>%

mutate(upper_limit = Inf) %>%

mutate(upper_limit = (ifelse(var2 >= 4, (Mean + 2*SD),Inf)))%>%

mutate(z_score = (ifelse(var2 >=4, (var2-Mean)/SD, 0)))

#subset data and remove the outlier by using the standard deviation and mean (upper and lover limit)

ot1 = ot_data %>% group_by(var1) %>% filter(var2 <= upper_limit & var2 >= lower_limit)

[nirgrahamuk]

Hi!

To help us help you, could you please prepare a reproducible example (reprex) illustrating your issue?
In your case clean_df

Please have a look at this guide, to see how to create one:

FAQ: How to do a minimal reproducible example ( reprex ) for beginners

A minimal reproducible example consists of the following items: A minimal dataset, necessary to reproduce the issue The minimal runnable code necessary to reproduce the issue, which can be run on the given dataset, and including the necessary information on the used packages. Let's quickly go over each one of these with examples: Minimal Dataset (Sample Data) You need to provide a data frame that is small enough to be (reasonably) pasted on a post, but big enough to reproduce your issue. Let's say, as an example, that you are working with the iris data frame head(iris) #> Sepal.Length Sepal.Width Petal.Length Petal.Width Species #> 1 5.1 3.5 1.4 0.…

---

Short Version

You can share your data in a forum friendly way by passing the data to share to the dput() function.
If your data is too large you can use standard methods to reduce it before sending to dput().
When you come to share the dput() text that represents your data, please be sure to format your post with triple backticks on the line before your code begins to format it appropriately.

```
( example_df <- structure(list(Sepal.Length = c(5.1, 4.9, 4.7, 4.6, 5, 5.4, 4.6, 
5, 4.4, 4.9), Sepal.Width = c(3.5, 3, 3.2, 3.1, 3.6, 3.9, 3.4, 
3.4, 2.9, 3.1), Petal.Length = c(1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 
1.4, 1.5, 1.4, 1.5), Petal.Width = c(0.2, 0.2, 0.2, 0.2, 0.2, 
0.4, 0.3, 0.2, 0.2, 0.1), Species = structure(c(1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L), .Label = c("setosa", "versicolor", "virginica"
), class = "factor")), row.names = c(NA, -10L), class = c("tbl_df", 
"tbl", "data.frame")))
```

[dthumar16]

#dataframe create
var1 = c("A", "A", "A", "A", "A", "B", "B", "B", "B", "B", "C", "C", "C", "C", "C")
var2 = c(3, 2,1, 0, 0, 2, 0, 0, 5, 1, 2, 3, 1, 1,0)
clean_df = data.frame(var1, var2)

#verify the data
clean_df

#Calculate lower_limit, upper_limit, and z_score if var2 >= 4
#Assign lower_limit, upper_limit, and z_score to entire group if var2 >=4

ot_data = clean_df %>% group_by(var1) %>%
  mutate(Mean = mean(var2), SD = sd(var2)) %>%
  mutate(lower_limit = -Inf) %>%
  mutate(lower_limit = (ifelse(var2 >= 4, (Mean - 2*SD),-Inf)))%>%
  mutate(upper_limit = Inf) %>%
  mutate(upper_limit = (ifelse(var2 >= 4, (Mean + 2*SD),Inf))) %>%
  mutate(z_score = (ifelse(var2 >=4, (var2-Mean)/SD, 0)))

**#need to assigned conditional values to the group instead of the individual observation**
ot_data

[nirgrahamuk]

# Calculate lower_limit, upper_limit, and z_score 
# for all groups of var1 after first having removed var2 >= 4

(ot_data <- clean_df %>%
  filter(var2 < 4) %>%
  group_by(var1) %>%
  mutate(across(var2, 
                list(Mean = mean, SD = sd),
                .names = "{.fn}"),
    lower_limit = Mean - 2 * SD,
    upper_limit = Mean + 2 * SD,
    z_score = (var2 - Mean) / SD
  ))

[dthumar16]

I need to keep all observations for var2
When i remove filter(var2 < 4) from above code, I am getting an error.

[nirgrahamuk]

@dthumar16

I am getting an error.

please be precise and state what the error is.
I have tested to see if I get an error, and I don't ...

(ot_data <- clean_df %>%
  # filter(var2 < 4) %>%
  group_by(var1) %>%
  mutate(across(var2, 
                list(Mean = mean, SD = sd),
                .names = "{.fn}"),
         lower_limit = Mean - 2 * SD,
         upper_limit = Mean + 2 * SD,
         z_score = (var2 - Mean) / SD
  ))

[dthumar16]

It works!

I do need to assign NA (missing value) if var2 < 4 - So I just need to assign NA (missing value) if var2 < 4

Once I calculate outlier .... by using the Lower_limit and Upper_limit. I need to exclude observations if lower_limit > var2 or upper_limit < var2.