Move your library to a non synced folder and change the default path, maybe you will need to reinstall some packages too, like assertthat.
You can change the default library folder by setting your R_LIBS_SITE environmental variable on a .Reviron or .Rprofile file
# In a .Renviron file you can set it by adding this line
R_LIBS_SITE="C:\\Program Files\\R\\R-4.0.3\\library"
For a more detailed explanation, you can read this blog post