Package- SparkR

[tmmet]

Hi,
Could anyone please help me? I have RStudio in my local machine and running into memory issues when reading a huge data file. Planning to use SparkR. But, want to know on the below.
1.Is there any difference between Sparklyr and SparkR?
2. Since , both are used for Spark Integration from R, when to use which package?
3. Since, both have active community, trying to see which one is the best?
4. Also , are there any limitations or challenges on these two?

If anyone could provide details on this, that would be great.
Thanks!

[mara]

Things may have changed since May, but thought I'd point to @kevinushey's outline of differences/motivations here:

Issue: sparklyr and SparkR - the future?

[cderv]

@tmmet

I have RStudio in my local machine and running into memory issues when reading a huge data file.

About this, there another topic on the community that could have interesting information for you

Best practice to handle out-of-memory data

Hello, I am a data analyst using mainly R and SAS. While I generally prefer to use R and tidyverse tools for my data science and programming tasks, I miss SAS datasets whenever R data frames consume all the memory. I could use variety of R packages to handle large data (bigmemory, ff, dplyr interface to databases, etc.), but unified binary data format on disk as in SAS has several advantages: ease of data management, no need to learn additional syntax, being able to create many intermediate datasets without caring about RAM (which makes debugging easy). So, I want to ask R community: What are your ideas, techniques, workflows, and best practices to handle out-of-memory data in R ? (I …

And also the two topic are related, it is good to link them.

[ed_berry]

Hi all,

I recently wrote a blog post comparing sparklyr and SparkR on a range of criteria.

https://eddjberry.netlify.com/post/2017-12-05-sparkr-vs-sparklyr/

The table at the top of the post provides a rough overview of my conclusions. Essentially sparklyr is already nicer to use even at this early stage of its development (compared to SparkR).

[edgararuiz]

Great job on the article, I recently read it and shared it internally. Thanks!