You are right, it's complicated, but I think the preset colours are used from the definition in the theme or in the css.
When you use another theme (e.g. from shinythemes) the colours can be different. Most often primary is blue, however but there are other possibilities as well.
I just checked my app, as I have include also some shinyWidgets elements, but I am just lucky that one of the colours (primary for me) matches the colour I want to have.
So if you find a solution - let me know!
I know that you can additionally edit the appearence of the sliders,
and then match the regular action buttons as described above.
And when you use your own css maybe this can help?
Matthias