This issue was raised in https://github.com/tidyverse/dplyr/issues/6015, basically the way that the current CRAN version of dplyr accesses the lazy data of the second edition readr parser causes a slowdown you observe when using group_by().
We have fixed this in the development version of the vctrs package, which dplyr uses, but it is not yet on CRAN.
As a workaround you can use lazy = FALSE in your calls to read_csv() which will avoid the issue.