Can you say something about the types of models you are building using keras or tensorflow? Not all operations are equally intensive in their utilization of GPU.
I have found that training an image recognition task, using many convolutional layers, consume a substantial amount of GPU, but other types of model doesn't necessarily do so.
I suggest you train a standard convolution task, e.g. the classical MNIST and observe the GPU in action while the model is training.
From your snippets above, it certainly seems like your GPU is configured correctly, but if you try the MNIST example and report back on what you find, it may help to confirm one way or the other.
Also, I don't think you need anaconda to do this, and I suggest you follow the instructions at https://tensorflow.rstudio.com/tools/local_gpu.html.