Can you give an example of your current code?
This seems to work for me:
#Load dataframe with stringsAsFactors = F
test = data.frame(rowId = 1:5, customer_id = c("B32598", "254879", "?", "B12768", "?"), stringsAsFactors = F)
#Or convert afterwards
test = data.frame(rowId = 1:5, customer_id = c("B32598", "254879", "?", "B12768", "?"))
test$customer_id = as.character(test$customer_id)
#Subset
test = test[test$customer_id != "?",]