This is because it's sorting alphabetically, and "alphabetically" speaking 1 comes before 3 (that's why people often prefix, say, numbered file names with zeroes: 01, 02, 03...—like you have for the study numbers).
It sounds like gtools' mixedorder() is working for you, so I can't think of a good reason not to use it.
Another approach with tidyr and dplyr would be to separate out the amounts from the groups and units so that you can properly sort them numerically. Note that there's the issue of precedence. So, for example, study 04 is coming before study 03 in group a because 300 is less than 10000.
library(tidyverse)
treatment = c("a 75 mg", 'p 0 mg', 'b 1 mg/kg', 'b 100 mg/kg',
'a 300 mg', 'b 0 mg/kg', 'a 1000 mg', 'a 300 mg')
study = c('01', '01', '02', '02', '04', '01', '03', '01')
patients = c(1, 10, 100, 3, 14, 5, 10, 3)
myData = data.frame(treatment, study, patients, stringsAsFactors = F)
myDat2 <- myData %>%
separate(treatment, into = c("group", "amt", "unit"), sep = " ")
myDat2 %>%
mutate(amt = as.numeric(amt)) %>%
group_by(group) %>%
arrange(group, amt, study, patients, .by_group = TRUE)
#> # A tibble: 8 x 5
#> # Groups: group [3]
#> group amt unit study patients
#> <chr> <dbl> <chr> <chr> <dbl>
#> 1 a 75 mg 01 1
#> 2 a 300 mg 01 3
#> 3 a 300 mg 04 14
#> 4 a 1000 mg 03 10
#> 5 b 0 mg/kg 01 5
#> 6 b 1 mg/kg 02 100
#> 7 b 100 mg/kg 02 3
#> 8 p 0 mg 01 10
Created on 2020-01-21 by the reprex package (v0.3.0.9001)