I think there are a few issues.
first 'if' is not vectorised so would only consider the first pair of trial values, you can use all to set each pair and assess a total matching, this will resolve to a true or false which if can then evaluate.
Also the trials pick the samples without ordering, so 1,2,3 will not match 3,2,1 but for a 'lottery' that wouldnt make sense to me, so add sorting before comparing. Furthermore if a lottery is physically based, there would usually not be replacement... finally, winner/success is not used but any purpose other than to pass if result into a position of list, so can be removed.
n <- 1000
list <- numeric(n)
for(i in 1:n){
trial1 <- sort(sample(0:9,3, replace = FALSE))
trial2 <- sort(sample(0:9,3, replace = FALSE))
list[i] <- if(all(trial1==trial2)) 500 else -1
}
mean(list)
Now the above is quite an explict way of working, and is arguably inelegant compared to other R approaches. Here is one.