IMO, you should not include *.Rpoj in your .gitignore if nothing for the simple fact that you can set the project's options to never save/load the workspace on exit/startup, ensuring that you can share the project without worrying about an old image cluttering up your collaborator's workspace.
Moreover, it can really help with new collaborators who may be unfamiliar with R since you can tell them to download the project folder and double-click on the *.Rproj file to open it, setting them in the correct working directory every time.