The best approach to these types of questions are with a spatial tree structure such as a Quad Tree. The QuadTree package provides access to a C implementation of this. While it doesn’t provide circular lookup it does provide a rectangular lookup which you can use to filter down the number of candidates. I would expect that this type of functionality is also present in sf or some of its related packages but I’m not sure. If it is it is likely to be more performant