I think the confusion here is that as.factor() only effects the specific column and converts it to a factor variable. It does not effect the data.frame as a whole. The kind of thing you'd be looking for is a grouped data.frame, such as through the group_by() function.
To create a table with the average of each group, you can use the following:
In order to do that, you need to apply the fill= aesthetic to the variable you want the colour to depend on, in this case Col1. These aesthetics are mappings from your data to aesthetic concepts, such as the x-axis (x=), the y-axis (y=), the fill of an object (fill=), it's size (size=), its shape (shape=) and many many more. So, in the aes() function in ggplot(), we use fill=Col1, rather than fill=moy. It's also useful, since you're using colour names as your factoring variable, to add the scale_fill_identity() function to the end, this will make the colours of your bars match the colours of your factor (try it without and see how confusing it is!)
ggplot(BDD, aes(x = Col1, fill = Col1)) +
geom_bar() +
xlab("Color") +
ylab("Average by color") +
scale_fill_identity()