Spark error while ingesting data from RDBMS to DataLake via SparklyR

Hi Team,
One of the data scientist is trying to move data from RDBMS/CSV to DataLake via SparklyRcode, he gets below error while. Can we join two different dataset in sparklyR and execute spark on cluster mode?

Test Case 1 :
Description : Trying to load a small mapping file into spark table. In our analysis we use few
One of the data scientist is trying to move data from RDBMS/CSV to DataLake via SparklyR mapping files which are in excel/csv which are required to be joined with rest of the data.
Result : The copy_to command works with max 15 records, beyond that throws exceptions – Fails

Hi! can you try copy_to(sc, as.data.frame(postalcodes)?

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.