Can your lists be easily converted to matrices? What is their structure? What were the reasons they wound up as lists and not matrices up until now?
If you have matrix-appropriate data on which you are doing matrix multiplications, I can’t see a reason not to use matrices, but then I don’t know very much about what you’re doing. Is there a downside that you can see to switching to matrices?
I personally think broad statements about performance aren’t very convincing (the details matter), but the common advice is that the performance of matrices and the built-in matrix manipulation functions will be superior to trying to recreate the same thing using another data structure.