Bingo! There appear to be some weird interactions between the RStudio Theme and the Editor Theme. For example, I was using the "Classic" RStudio theme (primarily dark on light) along with the "Material" editor theme (primarily light on dark). My guess is that Material specifies a white hyperlink color, which the help docs were picking up.
These types of interactions are unexpected, and I would encourage the RStudio team to look into how the Editor themes impact panels beyond the script editor.