Rstudio server "cannot allocate memory..." though server has lot of memory

RStudio IDE
1

Hi, I am using R studio server in linux machine with the server memory 32GB and Rstudio available memory 6GB, but when I use memory.limit(12Gb) in R session to allocate it for my computation it's giving me the same error "cannot allocate vector of size 10Gb". I am working with a large file and whenever I am checking memory.limit() it is "inf". Please help!

:~$ ulimit -v
unlimited
ulimit -a
unlimited
> ulimit::memory_limit()
soft hard 
 Inf  Inf
> memory.size(max = NA)
[1] Inf