Hi @Naarayanan777,
To increase the contrast between your categories of points you may like to specify the colours. We don't have your data, so here is an example using your modified code:
library(ggplot2)
library(tidyverse)
lm_fit_final2 <- lm(mpg ~ disp, data=mtcars)
resids <- lm_fit_final2$residuals
fitted <- fitted(lm_fit_final2)
category <- c(rep(1, 20), rep(2, 12)) # Dummy category
residual_Plot_2 <- data.frame(Predicted = fitted,
Residual = resids,
Category = factor(category))
my_med <- median(resids)
ggplot(residual_Plot_2, aes(Predicted, Residual)) +
xlab("Predicted") +
ylab("Residuals") +
geom_point(aes(shape = Category, color = Category), size= 3) +
geom_hline(yintercept = my_med) +
coord_equal() +
theme_classic() +
scale_color_manual(values=c("red", "blue"))

Created on 2020-05-25 by the reprex package (v0.3.0)
You also have a lot of points overlap: you may like to add an alpha = 0.2 value.
HTH