Placing your library() calls as needed in your markdown chunks certainly fits the flow when initially writing your markdown documents. Consciously moving the library() calls together to the top of the document will make it easier for future users of the document to easily see what libraries are used.
The Google R style guide recommends placing your library calls together
General Layout and Ordering
If everyone uses the same general ordering, we'll be able to
read and understand each other's scripts faster and more easily.
1. Copyright statement comment
2. Author comment
3. File description comment, including purpose of
program, inputs, and outputs
4. source() and library() statements
5. Function definitions
6. Executed statements, if applicable (e.g.,
print, plot)