You have not shown us how you are calling the function (i.e. what R code are you using)? Also, without the hmmerScan.exe file, it will be difficult for us to reproduce your issue. It looks like your options are being misinterpreted. Ensure that you can execute the command you are building from the terminal, and maybe from the RStudio terminal. Then, if you want to execute from R code, you can look into the help pages to system2 or system.
For instance, it is important to note that only the first part (hmmscan) is necessary for the command for system2 - everything else is in an arg character vector.
EDIT: Simple examples:
system2("echo","test")
system2("ping",c("-c 2","8.8.8.8"))