I´m trying to replicate your issue with dates stored as factors but I can´t, It just works as expected.
# Made up sample data
df <- data.frame(Time = c("06:20:00.0000000", "07:20:00.0000000", "08:20:00.0000000"))
#>  "factor"
mutate(Hour = as.factor(hour(hms(Time))))
#> Time Hour
#> 1 06:20:00.0000000 6
#> 2 07:20:00.0000000 7
#> 3 08:20:00.0000000 8
Could you ask this with a minimal REPRoducible EXample (reprex)? A reprex makes it much easier for others to understand your issue and figure out how to help.
If you've never heard of a reprex before, you might want to start by reading this FAQ: