Hi Mara,
Thanks for that link from Emily's website. That's a great article - I enjoyed reading it; in fact, I bookmarked it. It's a great case of getting rid of preconceived notions long enough to objectively assess the best way to get a job done.
Fair enough about blind gospel. I actually would have been inclined to believe as you do, except for a few things. I see other posts in other blogs that also suggest to the world that databases are slower than memory operations. Also, even more distressing, is that during my research I came across a few posts where the poster would support this idea with a sentence that starts with "Hadley says .." - which is really where the word "gospel" came to mind. This can make it difficult for skilled technical people to get their technical advice heard if their less technical managers read these kinds of blogs given that most of them won't have enough technical understanding to read between the lines, so to speak. It also can be confusing to those who are new to data science.
64 gigs is alot for a laptop, sure. But I have 64 gigs on my desktop and it really didn't cost me that much money, so my guess is that even if it's not common now, it will be soon enough. Also, as the use of server-side products like R-Shiny, R-Connect, and Microsoft ML-Server are becoming more prevalent, the decision to use a database for performance and resource optimization becomes a real concern. I also know that I'd find the same thing with a 16gb file and had the same results - I did this with the NYC Taxi Driver database.
I have too much respect for Hadley to muster the temerity I would need to force his hand in correcting his documentation for a product he invented and produced. I'm perfectly happy to leave that decision up to him. But I will step up when I run across fellow data scientists who might be accidentally misled into thinking that relational databases are to be avoided.
In any case, I'm very encouraged by the willingness of the community here to politely consider and even give credence to different points of view. Thanks again for your response, Mara.
Kind regards,
Claude