Although you made a great effort to try and explain the issue, could you create a reprex for us where we can actually run the code with a minimal dataset you create. I still don't understand what you exactly mean by removing an insignificant factor (do you mean removing a whole feature i.e. column, or just one of the options within a factor..)