Apologies, I misread your post. But the general approach I've suggested still holds. You should read the Excel file in its entirety first. Then loop over the CSV file paths to read the contents into a single data frame (I'm assuming that all CSVs have the same structure). In R, this can be done by using lapply() or purrr::map(). Similar tools also exist in Python.
After importing the CSVs, you can filter for the values you want using regular expressions. For the final step, you'll have to join the Excel data frame to the result and then export it.