Hiya @bragks! When you're using scale_*_viridis_d(), you can exert some limited control over the values it picks using the begin, end and direction arguments, which control the minimum and maximum values of the viridis range, as well as whether the values get progressively lighter or darker.
If you're looking for more specific control (ie. you want to pick out particular shades), I'd recommend retrieving them using viridis_pal and then passing the values on to scale_fill_manual using its values parameter (which take sa vector of colour names or hex values). Also, if you name the elements of values according to the values in your data, ggplot2 will match them up! 