Hi
Yes, I am on M1 Mac Mini, with XCode installed, and R 4.1.0 for M1. Unfortunately, I also got fatal error. So I guess the issue is probably hardware related.
If you use R for M1 Arm, it might worth checking if it would work with R for Intel. If that still doesn't work, I think you probably need to find another hardware.