It doesn't matter whether U is positive or negative, provided you're doing the right thing. Could you post a reprex?
One thing you can do is plot U against the parameters to see the shape of the surface.
Nelder-Mead has a tendency to get stuck in local minima, that's why SANN might be better, depending on the problem.
If the determinant of a matrix goes to zero, the matrix becomes singular, and it can't be inverted. If det(V)<0 then you will have a problem because you can't take the log.