Really straightforward with base::grep:
sub(pattern = "((",
replacement = "[",
x = sub(pattern = "))",
replacement = "]",
x = "((hello))",
fixed = TRUE),
fixed = TRUE)
#> [1] "[hello]"
If you want stringr, this may be one option:
library(stringr)
"((hello))" %>%
str_replace(pattern = fixed(pattern = "(("),
replacement = "[") %>%
str_replace(pattern = fixed(pattern = "))"),
replacement = "]")
#> [1] "[hello]"