If you want your filtering to work you need to give it a valid condition. See below:
d <- d %>% filter(rank == 1 | rank == 2) %>%
mutate(diff = table - lag(table, order_by = rank))
No value will have 1 and 2 as rank at the same time so it will filter out everything.
Then in terms of your other code you will need to make a condition that color is only valid when it at least has 2 values associated with that color. Otherwise you will have a problem with some who only have single. You will also have unfortunate overlap so you should always calculate when it should occur.
A bit of code that might help you as well is something like this. You can essentially create a mutate with conditionals. If rank 1 or 2 then it will perform depth - lag(depth) for diff otherwise it will just have 0. You can adjust this accordingly to work for your problem (more than 2 levels etc).
d <- d %>% mutate(
diff = case_when(
rank == 1 | rank == 2 ~ depth - lag(depth),
TRUE ~ 0
)
)
Let us know if this helps 