There are several R packages that can help you make tables. DT, gt, reactable to name a few popular ones.
For something like this, I'd encourage you to check out gt. The Introductory vignette has a nice walkthrough how to start customizing your table.
RStudio Community also hosts a nice table gallery, where folks in the community have shared first rate examples of great looking tables all made with R, and all including well documented and reproducible code.