I suppose you can use CSS to limit the height. It might cause some weird interactions with the box, I haven't really tested it for longer than 10 seconds, but it might work for you.
library(shiny)
mycss <- "
#select ~ .selectize-control .selectize-input {
max-height: 100px;
overflow-y: auto;
}
"
ui <- fluidPage(
tags$style(mycss),
selectInput("select", "Select", rownames(mtcars), multiple = TRUE)
)
server <- function(input, output, session) {}
shinyApp(ui, server)
There might be better ways, that's just the first I could think of