Good day,
I am able to make a successful spark connection with the sparklyr package. I am able to make successful connections to a database of interest with the odbc package or the RODBC package.
I would like to read data from the database into a Spark data frame so that I can manipulate data with the dplyr package. How should I go about doing this? The information I have to work with is below, and the database is stored in Sybase IQ (made by SAP) and can be accessed with SQL or an ODBC method.
Please let me know if I can provide any additional information, and thank you for your help,
-Ted
####Spark Connection#####
library(sparklyr)
sc <- spark_connect(master = "http://<url>",
method = "livy", config = livy_config(
username = "UID",
password = rstudioapi::askForPassword("Livy password:")))
#####RODBC Database Connection#####
library(RODBC)
myConnection= odbcDriverConnect("Driver=/opt/app/SybaseIQ/IQ-16_1/lib64/libdbodbc17.so;host=<url:port>;ServerName=<name_of_server>;
DatabaseName=<name_of_database>;
UID=<userID>;
PWD=<password>",
believeNRows = FALSE
)
####odbc Database Connection####
require(odbc)
require(DBI)
con <- dbConnect(odbc::odbc(),
dsn = "<data_source_name>",
UID = "<userID>",
PWD = "<password>")