Oops, somehow my previous searching overlooked this example, here's the way to format it - gt uses html instead!
cols_label(median_hourly_kwh_m2 = html("Median hourly kWh/m<sup>2</sup>")
)
I also noticed gt works as a tag on this forum, but throws an error message as a search term since it's too short, which could be confusing for newcomers.