This is a fantastic question. And unfortunately one without a super clear answer. I was hopeful it would be obvious... but it wasn't. And I made some hopefully-not-too-damaging blunders before I found it 
If you open up the variant and then the "Input" pane, you should have a delete button in the bottom right corner. Does that work for you? I have put in some feedback - I definitely think this could have a better UX 