Plot words and points with gradient, except for a few in black as specified in another column

[johnG]

I would like to plot some words with points, where the data are plotted according to a gradient except for those in the middle which are coloured black (with no gradient). See example data and plot below, where I would like the variable color_black to override the gradient for those words (and their points) with 0.

I suspect I would like to avoid adding another layer on-top as it might make the plot messy when more words are plotted on it? In below solution it is the black words that are overlapping the other word-labels more than the word-labels that has a gradient applied to them.

# Possibility to increase the data to test that there are no (only little) overlap of labels
n=5
#Labels plotted in figure
text <- c(rep("zero", n),
          rep("one", n),
          rep("two", n),
          rep("three", n),
          rep("four", n),
          rep("five", n),
          rep("six", n),
          rep("seven", n),
          rep("eight", n))

# Variable forming the color gradient (and position)
color_my <- c(rep(-4, n),
          rep(-3, n),
          rep(-2, n),
          rep(-1, n),
          rep(0, n),
          rep(1, n),
          rep(2, n),
          rep(3, n),
          rep(4, n))


# Words with "1" should be part of the gradient; 0 should just be "gray" with NO gradient
color_gray <- c(1, 1, 1, 0, 0, 0, 1, 1, 1)
color_gray <- c(rep(1, n),
              rep(1, n),
              rep(1, n),
              rep(0, n),
              rep(0, n),
              rep(0, n),
              rep(1, n),
              rep(1, n),
              rep(1, n))


data_test <- data.frame(text, color_my, color_gray)
data_test <- as_tibble(data_test)
colors_words_scale = c(-1, 0, 1)

library(ggplot2)
library(ggrepel)
library(scales)

plot <- ggplot(data_test, aes(color_my, y=1, label = text)) +  
  geom_text_repel(
    data=data_test[which(data_test$color_gray!=0),],
    aes(color = color_my)) +  
  geom_point(
    data=data_test[which(data_test$color_gray!=0),],
    aes(color = color_my)) +
  geom_text_repel(
    data=data_test[which(data_test$color_gray==0),],
    color='black') +
  geom_point(data=data_test[which(data_test$color_gray==0),],
             color='black') +
  scale_colour_gradientn(
    colours = c("blue", "lightblue", "black", "orange", "red"),
    values = rescale(colors_words_scale), aesthetics = "colour") +
  ggplot2::theme_minimal()

plot

Thank you for help and guidance!
John