Once you've settled on your optimization method, check out CRAN Task View on Optimization.
I think you are jumping a couple of steps by wondering what package or solver to use.
First, figure out the formulation, then method, then package & solver.
What I see is a possible IP formulation
Minimize:
\sum d_{ij} x_{ij}
where d_{ij} are the distances, x_{ij} are binary decision variables (1 if you use that pairing), i indexes the A's, and j indexes the B's.
Constraints:
\sum_{j} x_{ij} = 1 \; \; \forall i so that A is paired but only once
\sum_{i} x_{ij} \le 1 \; \; \forall j so that B is only paired 0 or 1 time
assuming N < M where N, M are the index size of i,j respectively.
I did this very quickly so take this with a huge grain of salt.
I hope that gives you an idea on how to start the problem.