For the first time, I am working with processing a ndjson format data file in R. Currently I am working with sample data set for development but the data volumes will go very high with the production data. So I am trying to process a limited number of lines(JSON objects) per iteration. By looking at the documentation for jsonlite::stream_in(), pagesize parameter should limit the number of lines read but it doesn't work that way for me. Am I missing something here? Thank you for help!
test1<-jsonlite::stream_in(url("Actual URL"),pagesize = 10, verbose = TRUE)
I expect above line of code to return a list of 10 objects but I get all 50 objects that are in the web page.