You can do it by passing a string of the standard type abbreviations to the col_types() parameter.
Use _ or - to skip columns you don't want. You can manually specify the type of each column that you want to read or use ? to allow the parser to guess.
Here's an example:
readr::read_csv("1,2,3,4\n5,6,7,8", col_names = FALSE, col_types = "?_?_")
#> # A tibble: 2 x 2
#> X1 X3
#> <dbl> <dbl>
#> 1 1 3
#> 2 5 7
Created on 2020-06-30 by the reprex package (v0.3.0)