Deriv does apply the chain rule, but we need a couple of tweaks to your code in order to get what you intended. In R, a function runs in its it own environment, so the m inside the function b is an argument that gets passed into b when you run b. It is not the function m that you defined outside of the function b.
First let's redefine our functions:
library(Deriv)
m <- function(x){
x^3
}
b <- function(x, f){
x^2 + f(x)^2
}
Notice that b has two arguments. x is a number and f is a function. For example, b(3, m) will pass the function m into b, giving 738 (3^2 + m(3)^2 = 3^2 + 3^6=738). b(3, cos) will pass the cosine function into b, giving 9.98 (3^2 + cos(3)^2 = 9.98).
Now, to run Deriv, we need to pass b(x, m) into Deriv as an unevaluated expression, which we can do in at least two ways (see code below). We also add the x="x" argument to let Deriv know we want only the derivative with respect to "x":
Deriv(~b(x, m), x="x")
Deriv(quote(b(x, m)), x="x")
Either way, we get:
x * (2 + 6 * (x * m(x)))
which is 2x + 6x^2m(x)=2x+6x^5.