[The original question title was "What function is called that creates the pretty format output of data.frames and tibbles?"]
I seem to recall that there's some special function that formats dataframes into the browsable ones you normally see in RMD documents in progress, like the one below.
Thanks for your willingness to help, but I think I need to clarify myself. I'm probably falling prey to the XY problem here.
My primary goal is instead of my custom classes showing up as <S3: myfhc> (see column foo), I want to specify how they should be seen when 'printed' in this context. I don't think I'll have any trouble writing up that method; but finding what method I need to specify is the tricky part. It doesn't appear that behavior is determined by the format generic or the print generic. (In fact, when I print this table in the console, I get a much more reasonable output than <S3: myfhc>, it actually includes values.) I'm dumbfounded on how to find the function that RStudio uses in this case.
I was hoping that finding that method would lead me to source code that shows the right method I need to implement, but that may be difficult too.
Pillar is used in the terminal-style output (fixed width, on the right of the photo above), and that was helpful to find, but I still can't find the right method that changes how custom types are displayed in the Rmd preview section.
I think I found it! The rendering method is rmarkdown::paged_table, but it doesn't appear that there's an easy way to specify how vectors should be converted to strings.