The unofficial contract between package authors and users is that non-exported functions are liable to change without warning on notification. I can use that function, but it may change without warning.
Also if I used it in a package submitted to CRAN, it would bounce.
That is true indeed.
One option for you is that you could rerpoduce the source code in your own package (i.e. all exported and non exported function calls in lubridate that are those required for the format.Duration) and choose to export it in your package. Then you will be responsible not to alter its contracts.