You might need to add another line for sheets authorization as well
sheets_auth(path = "/path/to/my/service/account/token/file.json")
googlesheets4 is probably starting an OAuth2.0 dance because the sheets package doesn't see the service account token credentials, and instead is trying to create a new OAuth token for you. I have been able to use both packages successfully using this method of directly calling the service account location for both drive_auth and sheets_auth.
edited to add more detail and hopefully be more helpful.