I think this is on the right track, I just did a horrible job of explaining it.
Here's some more of the code to break it out a bit further:
df1$glm <- predict(fit.glm, df1) #Apply the GLM here
df1lupdate <- df1[glm == '1'] #subset any "matches"
df1lcheck <- df1[glm == '0'] #But I also want NO matches
df1insert <- subset(df1leadcheck, !(ID %in% df1leadupdate$ID)) #however it can't be in both. E.g. if ID 123 is a MATCH it can't be a non-match and vice versa
df1insert <- unique(df1insert, by = "ID") #Make sure that inserts will be unique
if(nrow(df1update)!=0) {
fwrite(df1insert, file=paste0("D:/nomatches/",row,".csv"))
} else {fwrite(df1lpdate,file=paste0("D:/matches/",row,".csv"))}
rm(df1)
rm(df1leadcheck)
rm(df1leadupdate)
}
So I'm taking my big DT, subsetting it into matches and non matches, and ensuring that if it's going into the insert table, it can't be in the update table and finally, insuring the insert records are unique.