That is not what the error message says, you are misunderstanding it.
You are trying to install RTools into your package library folder and the installer is complaining because the folder is not empty. As I said, Rtools is not an R package so it doesn't need to (or should) be in your package library, just install it in the default location (i. e. the "Program Files" folder) like you would do with any other software.
After that, make sure to follow the rest of the installation instructions to properly configure RTools.
Putting Rtools on the PATH
After installation is complete, you need to perform one more step to be able to compile R packages: you need to put the location of the Rtools make utilities (bash, make, etc) on the PATH. The easiest way to do so is create a text file .Renviron in your Documents folder which contains the following line:
PATH="${RTOOLS40_HOME}\usr\bin;${PATH}"
You can do this with a text editor, or from R like so (note that in R code you need to escape backslashes):
writeLines('PATH="${RTOOLS40_HOME}\\usr\\bin;${PATH}"', con = "~/.Renviron")
Now restart R...