In general this is not possible. Many of the process names overlap between RStudio Open Source and RStudio Workbench, and the default ports are the same.
You may be able to run each product in a docker container on the same server, but you would have to be careful about how user home directories are shared. Both RStudio Open Source and RStudio Workbench save server state inside the user home directories, and you could run into conflicts.
Strictly speaking it may be possible, but it definitely is not recommended.