This is a great question! Sorry for the delayed response!
Are you trying to install this when deploying content to Connect? Debugging these system library dependencies can be a bit of a pain. One way to try interactively is to log into the server's terminal (ssh, etc.) and try installing the package in a session on the command line.
I have to say I am not super familiar with RQuantLib, but hopefully someone here is!
These are the only directions I have found for installation: https://github.com/eddelbuettel/rquantlib/wiki/RQuantLib
I do not recommend using apt-get install or yum install for the package. If you want Connect to be able to install the package, you will want to be sure that normal install.packages() works from a terminal.
You said you have installed libquantlib0-dev? Also, what linux distribution are you on?