Tensorflow GPU only works when knitting

I am just beginning working neural networks, with keras and tensorflow. I wanted to use tensorflow gpu, so I got CUDA and cuDNN installed so i could run tensorflow on my gpu, and everything seemed to install properly, but when i run the model, and monitor my task manager, in Windows 10, I see that it's using the CPU and not the GPU. I tried everything i could to get it to work, but the "dedicated GPU memory usage" graph wouldnt move. I proceeded anyway with the model i was building, and an R markdown file, and at the end, knitted to PDF. It was only during the knit to pdf that I saw that the GPU was suddenly engaged, and the time for each epoch of my model was running at 1/6 the time it had been previously. So the tensorflow gpu is working, but only when I knit. Any ideas on why this is happening?

This is with:
R 4.0.5
CUDA 11.2.0
cuDNN 8.1.1.33
tensorflow 2.4

It's been extremely difficult finding anything online for R, everything is for strictly Python.