On Ubuntu 20.04, rsession links to the following ICU libraries:
kevin@ubuntu:/usr/lib/rstudio/bin
$ ldd rsession | grep icu
libicuuc.so.66 => /lib/x86_64-linux-gnu/libicuuc.so.66 (0x00007f916053c000)
libicui18n.so.66 => /lib/x86_64-linux-gnu/libicui18n.so.66 (0x00007f916023d000)
libicudata.so.66 => /lib/x86_64-linux-gnu/libicudata.so.66 (0x00007f915e5d2000)
which are normally provided by:
sudo apt install libicu-dev
Can you confirm that you downloaded the binary of RStudio built for Ubuntu 20.04, and not an older version of Ubuntu?