how does the sequence of palette color in dyRibbon get decided?

You don't need 4 colors, you are passing 4 arbitrarily, the code works fine with just 3

It gets assigned in the same order each level is defined. I think your confusion comes from incorrectly scaling the data argument for the dyRibbon() function, it must be a numeric vector scaled from 0 to 1. Take a look at this example:

library(dplyr)
library(dygraphs)
library(RColorBrewer)
library(data.table)

set.seed(12)
# create data
dt1 <- data.table(t = seq(Sys.time(),by  = "1 sec", length.out = 10),
                  y = rnorm(10,20,5))
# add a column with 3 possible values to be used for dyRibbon
dt1[,flag1:=factor(dplyr::case_when(y>22 ~ "high",y>18 ~ "med", T ~ "low"))]

data_arg <- dt1 %>% 
    mutate(flagn = case_when(
        flag1 == "low" ~ 0,
        flag1 == "med" ~ 0.5,
        flag1 == "high" ~ 1)) %>% 
    pull(flagn)

dt1 %>% 
    select(t,y) %>%
    dygraph(main = "Reprex to show color sequence") %>% 
    dyOptions(drawPoints = T,pointSize = 5,drawGapEdgePoints = T,
              strokeWidth = 4,strokeBorderWidth = 1) %>% 
    dyRibbon(data = data_arg, palette = brewer.pal(3,"Set1"))