I would like to help but without an example I can look into or reproduce, it is difficult for me.
Did you build the book using bookdown::render_book() or using the Build Book button in the IDE ?
In the resulting folder (by default, _book), there should be a search_index.json file. Is this present locally ?
When you run bookdown::serve_book() it should build the book and create this file. The same in the book you host in your Github server.
You can open in a browser the bookdown site and also look into the browser's inspector (usually by hitting F12) to see if there is an error while loading this json file.
The search engine needs this file in order for the search to work.
This is a guess of what could be happening. If you can share more information, I am happy to help further.